A fire infrared and visible light image fusion method, device and medium

Abstract

The present application relates to a kind of fire infrared light visible light image fusion method, device and medium, method includes the following steps: obtaining the infrared light image and visible light image of fire, infrared light image and visible light image are re-set size after being transmitted into fusion network, the fusion network includes encoder, reverse progressive pyramid module and decoder, the feature map of re-set size after infrared light image and visible light image is extracted using encoder;Feature map is input into reverse progressive pyramid module and is progressively fused, and after being binarized, mask image is obtained;New image is obtained by fusing strategy and fusing infrared light image, visible light image and mask image;New image is input into decoder, and image fusion result is output.Compared with prior art, the present application has the advantages that image main body is prominent, and environmental interference is small.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method, apparatus, and medium for fusing infrared and visible light images of a fire. Background Technology

[0002] Fire safety has always been a critical public safety issue. Effective fire detection and alarm systems are essential for reducing the damage and property loss caused by fires. With advancements in technology, particularly image processing and machine vision, the use of infrared and visible light fusion technology for fire detection has become a hot research topic. Infrared imaging technology can detect the thermal radiation of a fire, proving effective even in conditions of dense smoke or low visibility. Visible light imaging technology provides rich visual information, including smoke density and direction, as well as escape routes for people and important assets in the background. Therefore, fusing infrared and visible light images combines their advantages to improve the efficiency of fire emergency response.

[0003] In recent years, with the maturity of deep learning technology, researchers have proposed many deep fusion algorithms. Generally speaking, the main deep fusion methods can be divided into two categories: fusion methods based on convolutional neural networks (CNNs) and fusion methods based on generative adversarial networks (GANs). CNN-based fusion methods typically rely on the powerful fitting ability of neural networks to extract and reconstruct effective information under the guidance of a carefully designed loss function. GAN-based fusion methods establish an adversarial game between the fused image and the source images, enabling the fused image to approximate the desired probability distribution without supervision. However, some end-to-end models struggle to distinguish the importance of objects in different source images. Infrared images can effectively highlight the subject but cannot clearly express details in the background. Visible light images contain rich texture information but are very sensitive to light intensity and easily affected by obstacles. Therefore, during the fusion process, the background easily carries a large amount of redundant, irrelevant, or even detrimental information, making it challenging to obtain a clear overall image while effectively highlighting the subject using infrared images. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art, such as the lack of a prominent subject and susceptibility to influence, by providing a method, apparatus, and medium for fusing infrared and visible light images of fires.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A method for fusing infrared and visible light images of a fire, comprising the following steps:

[0007] The infrared and visible light images of the fire are acquired, and after being resized, they are fed into the fusion network. The fusion network includes an encoder, a backward progressive pyramid module, and a decoder. The encoder is used to extract feature maps from the resized infrared and visible light images.

[0008] The feature map is input into the reverse progressive pyramid module for progressive fusion, and then binarized to obtain the mask image;

[0009] A new image is obtained by fusing infrared light images, visible light images, and mask images using a fusion strategy.

[0010] The new image is input into the decoder, and the output is the image fusion result.

[0011] Furthermore, the encoder includes a second coding layer and multiple first coding layers connected in sequence, with the last first coding layer connected to the second coding layer. The first coding layer includes a convolutional layer and a max pooling layer. The input of the first coding layer is either the input of the fusion network or the output of the previous first coding layer. The output of the max pooling layer of the first coding layer is used as one of the feature maps extracted by the encoder, and also as the input of the next first coding layer or the second coding layer. The output of the second coding layer is used as one of the feature maps extracted by the encoder.

[0012] Furthermore, the progressive fusion process of the reverse progressive pyramid module is as follows: obtain the semantic information of each feature map of the infrared light image and the image texture information richness of each feature map of the infrared light image, fuse the feature map with the highest level of semantic information, and then fuse the remaining feature maps in turn according to the semantic information richness.

[0013] Furthermore, during the progressive fusion process, the dimensions of the feature maps are aligned, and downsampling and upsampling are performed respectively. Upsampling uses the CARAFE upsampling operator, and downsampling uses convolutions with a stride of 2×2, 4×4, and 8×8.

[0014] Furthermore, binarization specifically involves using the dynamic adaptive adjustment of the threshold during convolution to perform binarization.

[0015] Furthermore, the integration strategy specifically includes the following steps:

[0016] S31: Compare the infrared image and the visible light image with the mask image I respectively. mask Dot product yields the main images of infrared and visible light, then the infrared and visible light images are respectively multiplied by the inverse mask image (1-I). mask Dot multiplication yields the background images for infrared and visible light, expressed as:

[0017]

[0018] In the formula, These represent the main image and background image of visible light, respectively. These represent the main image and background image of the infrared light, respectively. I represents the dot product. mask Represents a mask image, 1-I mask This is an inverse mask image;

[0019] S32: After adding channel attention and spatial attention to both the main image of visible light and the main image of infrared light, fuse them to obtain the main image fused image. Simultaneously input the background image of visible light and the background image of infrared light into the RFN module for fusion to obtain the background image fused image.

[0020] S33: Add the main image fusion image and the background image fusion image together to obtain a new image.

[0021] Furthermore, in step S32, the RFN module fusion process includes:

[0022] Infrared and visible light images are each passed through a 3×3 convolutional layer and then stacked. After stacking, the images are passed through three 3×3 convolutional layers in sequence and then added to the skip-connected images as the output.

[0023] Furthermore, the decoder includes multiple rows of decoding blocks, each row of decoding blocks corresponds to a feature map, and decoding blocks in the same row are connected by short connections. Each decoding block consists of two 3×3 convolutional layers.

[0024] In a second aspect of the present invention, a fire infrared and visible light image fusion apparatus includes a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement any of the fire infrared and visible light image fusion methods described above.

[0025] A third aspect of the present invention is a storage medium having a program stored thereon, which, when executed, implements any of the above-mentioned methods for fusing infrared and visible light images of a fire.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1) This invention designs a new feature pyramid module for fire images. It prioritizes extracting the contour information of salient targets in infrared images to create a binarized mask for subsequent fusion strategies. This method can better separate the subject and background in the image, so that the subsequent fusion process can more comprehensively combine the information of the two source images, highlight the subject of the image and reduce the interference of the environment on the image.

[0028] 2) This invention can adaptively adjust the proportion of background information obtained from infrared and visible light images, taking into account both global and local information. Attached Figure Description

[0029] Figure 1 This is a network structure diagram of the present invention.

[0030] Figure 2 This is a diagram of the overall network structure of the present invention.

[0031] Figure 3 Structure diagram of the fusion strategy

[0032] Figure 4 Structure diagram of the RFN module

[0033] Figure 5 Figures showing the fusion results of different algorithms on the Corsican Fire dataset.

[0034] Figure 6 The flowchart of this invention Detailed Implementation

[0035] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0036] Example 1

[0037] This invention relates to a method, apparatus, and medium for fusing infrared and visible light images related to fire, such as... Figure 6 As shown, the method includes the following steps:

[0038] S1: Acquire infrared and visible light images of the fire, resize the infrared and visible light images and then feed them into the fusion network. The fusion network includes an encoder, an inverse progressive pyramid (IAFPN) module and a decoder. The encoder is used to extract feature maps of the resized infrared and visible light images.

[0039] S2: Input the feature map into the reverse progressive pyramid module for progressive fusion, and then binarize it to obtain the mask image;

[0040] S3: A new image is obtained by fusing infrared light image, visible light image and mask image through a fusion strategy;

[0041] S4: Input the new image into the decoder and output the image fusion result.

[0042] The specific steps are as follows:

[0043] S1: As Figure 1As shown, the network of this invention mainly comprises three parts: an encoder-decoder network, an IAFPN module, and a fusion strategy. The encoder-decoder network consists of convolutional blocks, max-pooling layers, and upsampling layers; the IAFPN module consists of the CARAFE operator, downsampling layers, and convolutional blocks; the fusion strategy includes an RFN module, channel and spatial attention modules; the encoder consists of eight 3×3 convolutional layers and three max-pooling layers. Every two 3×3 convolutions are followed by a max-pooling layer, whose output serves as one of the four feature maps of the encoder and also as the input to the next 3×3 convolution for further feature extraction. The last two 3×3 convolutional layers are not followed by a max-pooling layer, and their output is directly used as one of the encoder's outputs.

[0044] S2: As Figure 2 As shown, similar to the RFN fusion network, the fusion part uses four feature maps extracted by the encoder network (see...). Figure 2 However, only the feature maps of the infrared image are used as input to the IAFPN module, labeled ir0, ir1, ir2, and ir3 respectively. For feature extraction and fusion, the feature maps ir0 and ir1, rich in high-level semantic information, are first input into the feature pyramid, and then ir2 and ir3, which possess rich texture information, are gradually fused. This fusion process gradually integrates high-resolution features from adjacent layers, avoiding the adverse effects caused by significant semantic information differences resulting from directly mixing non-adjacent layers. While preserving the texture information in the high-resolution image, it recovers the resolution from the low-resolution image, enriches semantic information such as feature contours in the image, and enhances blurred details in the low-resolution feature maps, thus more completely preserving the information of the infrared image when generating the binary mask image.

[0045] During the complete fusion of multi-layer feature maps, the dimensions of the different original feature maps need to be aligned. For upsampling, the lightweight CARAFE upsampling operator is used, leveraging its larger receptive field and specific content-aware characteristics compared to bilinear interpolation to help extract salient targets in infrared images more accurately. For downsampling, 2x2 convolutions with a stride of 2 are used for double downsampling, 4x4 convolutions with a stride of 4 are used for quadruple downsampling, and 8x8 convolutions with a stride of 8 are used for eightruple downsampling.

[0046] Upsampling uses the CARAFE upsampling operator, while downsampling is achieved using 2×2 convolutions with a stride of 2, 4×4 convolutions with a stride of 4, and 8×8 convolutions with a stride of 8. The mask image is obtained through binarization by dynamically and adaptively adjusting the threshold using convolution, making salient targets in the image more complete.

[0047] S3: A new image is obtained by fusing infrared light images, visible light images, and mask images using a fusion strategy;

[0048] S31: Compare the infrared image and the visible light image with the mask image I respectively. mask Dot product yields the main images of infrared and visible light, then the infrared and visible light images are respectively multiplied by the inverse mask image (1-I). mask Dot multiplication yields the background images for infrared and visible light, expressed as:

[0049]

[0050] In the formula, These represent the main image and background image of visible light, respectively. These represent the main image and background image of the infrared light, respectively. I represents the dot product. mask Represents a mask image, 1-I mask This is the inverse mask image. In this operation, the feature values ​​at corresponding positions are multiplied to combine feature information from visible and infrared images. This fusion method helps the model comprehensively consider the characteristics of both modalities at each pixel location, thereby improving the accuracy of target recognition or scene understanding.

[0051] and, and These represent background images from infrared and visible light, respectively. This separation allows for enhancement of thermal information in the main subject areas of the infrared image during subsequent fusion, making these key targets visually more prominent. Simultaneously, it prioritizes preserving the texture and detail of the visible light image, reducing potential noise that the infrared image might introduce into the background.

[0052] S32: After adding channel attention and spatial attention to both the main image of visible light and the main image of infrared light, fuse them. Then, simultaneously input the background image of visible light and the background image of infrared light into the RFN module for fusion.

[0053] like Figure 4 As shown, the infrared and visible light images are each convolved by a 3×3 layer and then stacked. After three more 3×3 convolutions, they are added to the skip-connected image as the output.

[0054] Fusion is represented as:

[0055]

[0056] in, and The values ​​represent features filtered through channel attention and spatial attention, respectively. Channel attention consists of a global average pooling layer and a fully connected layer, while spatial attention consists of two 3×3 convolutional layers. The application of these two attention methods complements the information differences between the source images. Subsequently, background images from visible and infrared light are input into the RFN module (see...). Figure 4 The images are then blended to produce four resolution levels of background blended images: H×W (original resolution), H / 2×W / 2 (half resolution), H / 4×W / 4 (quarter resolution), and H / 8×W / 8 (eighth resolution).

[0057] like Figure 3 As shown, in the process of fusing the main images of infrared and visible light, parallel channel attention and spatial attention mechanisms are used to integrate complementary information, thereby highlighting salient targets and refining the main texture information that may be missed in the process of generating binary images.

[0058] S33: Add the two merged images together to obtain a new image.

[0059] The subject image and the background image are ultimately combined and used as input to the decoder. This integrated information helps the decoder to reconstruct and analyze the scene content more comprehensively and in detail. S It is the fused salient target image, F RFN This is the image obtained by fusing the visible light and infrared light background images in the RFN module. The two images are added together and used as the input to the decoder, as shown below:

[0060] F OUT =F S +F RFN

[0061] In the formula, F OUT It is the new image, which is the input to the decoder.

[0062] S4: To verify the network performance, this study selected the TNO dataset and the Corsican Fire dataset to test the IAFusion algorithm and compared its performance with a series of comparative algorithms, including GTF, DIDFusion, RFN-Nes, FusionGAN, TarDAL, UMF-CMGR, SeAFusion, SwinFusion, and U2Fusion.

[0063] The TNO dataset is the most commonly used dataset for validating the effectiveness of infrared and visible light image fusion. It contains dual-spectral images of natural and urban environments under different lighting conditions. The diversity of this dataset allows for a comprehensive evaluation of the performance of fusion algorithms under complex lighting conditions.

[0064] The Corsican Fire dataset focuses on fire scenes, collecting 648 pairs of infrared and visible light images from Corsica, France. These image pairs have low similarity and include frames extracted from video. This dataset was specifically used to validate the network's performance in image fusion, particularly its ability to highlight important targets when dealing with low-similarity images. Compared to the original RFN-Nest network, it better demonstrates IAFusion's improvements in enhancing salient target recognition and optimizing texture details. Through testing on these challenging datasets, this study comprehensively evaluates the superiority and robustness of the IAFusion algorithm in practical applications.

[0065] Sixty thousand pairs of infrared and visible light images were randomly selected from the KAIST pedestrian dataset as the training dataset for the network. These images were resized to 256×256 pixels and converted to grayscale. The network was trained for two epochs with a learning rate of 1e-3 and a batch size of 4. Subsequently, the network was validated on the TNO dataset, and the validation results are shown in Table 1.

[0066] Table 1. Validation results of the TNO dataset

[0067]

[0068] Based on the data in the table, compared to other algorithms tested on the TNO dataset, our model achieved the highest structural similarity score (SF), two second-highest scores (mean gradient AG and visual information fidelity VIF), and two third-highest scores (entropy EN and peak signal-to-noise ratio PSNR) in the objective evaluation metrics. The higher grayscale change rate and mean gradient indicate a significant advantage in image sharpness, information richness, and lower distortion. This effectively reduces the amount of noise common in infrared images, demonstrating the model's ability to generate high-quality fused images. The lower correlation coefficient (CC) value, representing lower linear similarity, may be due to insufficient preservation of details of non-critical targets in the infrared image background during the fusion process.

[0069] To verify the model's fusion performance on flame images and its ability to emphasize salient targets, we tested the RFN-Nest and IAFusion models on the Corsican Fire dataset. Both models were trained for 50 epochs using the training set before comparison. Table 2 shows the experimental results.

[0070] Table 2. Fusion effect of flame images

[0071]

[0072] As can be seen from Table 2, our model exhibits significant advantages in metrics such as average gradient, spatial frequency, and interactivity. Figure 5 As shown, the image clarity is higher and the fusion quality is better. Although there are slight deficiencies in entropy (EN) and peak signal-to-noise ratio (PSNR), this is mainly because each image in this fire dataset contains a distinct and salient object. During the convolution process, the threshold is adjusted to highlight the subject in the image. While this strategy results in some loss of texture details in the subject, it more effectively highlights important objects. Therefore, overall, this model performs excellently in emphasizing key information in fire images and improving visual resolvability.

[0073] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0074] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0075] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for fusing infrared and visible light images of a fire, characterized in that, Includes the following steps: The infrared and visible light images of the fire are acquired, and after being resized, they are fed into a fusion network. The fusion network includes an encoder, a reverse progressive pyramid module, and a decoder. The encoder is used to extract feature maps from the resized infrared and visible light images. The feature map is input into the reverse progressive pyramid module for progressive fusion, and then binarized to obtain the mask image; A new image is obtained by fusing infrared light images, visible light images, and mask images using a fusion strategy. The new image is input into the decoder, and the image fusion result is output. The encoder includes a second coding layer and multiple first coding layers connected in sequence, with the last first coding layer connected to the second coding layer. The first coding layer includes a convolutional layer and a max pooling layer. The input of the first coding layer is the input of the fusion network or the output of the previous first coding layer. The output of the max pooling layer of the first coding layer is used as one of the feature maps extracted by the encoder, and also as the input of the next first coding layer or the second coding layer. The output of the second coding layer is used as one of the feature maps extracted by the encoder. The progressive fusion process of the reverse progressive pyramid module is as follows: obtain the semantic information of each feature map of the infrared light image and the image texture information richness of each feature map of the infrared light image, fuse the feature map with the highest level of semantic information, and then fuse the remaining feature maps in turn according to the semantic information richness. The fusion strategy specifically includes the following steps: S31: Compare the infrared image and the visible light image with the mask image respectively. Dot product yields the main images of infrared and visible light, then the infrared and visible light images are respectively multiplied by the inverse mask image 1- Dot product yields the background images for infrared and visible light, expressed as: In the formula, , These represent the main image and background image of visible light, respectively. , These represent the main image and background image of the infrared light, respectively. Dot product, Represents a mask image, 1- This is an inverse mask image; S32: After adding channel attention and spatial attention to both the main image of visible light and the main image of infrared light, fuse them to obtain the main image fused image. Simultaneously input the background image of visible light and the background image of infrared light into the RFN module for fusion to obtain the background image fused image. S33: Add the main image fusion image and the background image fusion image together to obtain a new image.

2. The method for fusing infrared and visible light images of a fire according to claim 1, characterized in that, In the progressive fusion process, the dimensions of the feature maps are aligned, and downsampling and upsampling are performed respectively. The upsampling uses the CARAFE upsampling operator, and the downsampling uses convolutions with a stride of 2×2, 4×4, and 8×8.

3. The method for fusing infrared and visible light images of a fire according to claim 1, characterized in that, The binarization specifically involves using convolution to dynamically and adaptively adjust the threshold for binarization.

4. The method for fusing infrared and visible light images of a fire according to claim 1, characterized in that, In step S32, the RFN module fusion process includes: Infrared and visible light images are each passed through a 3×3 convolutional layer and then stacked. After stacking, the images are passed through three 3×3 convolutional layers in sequence and then added to the skip-connected images as the output.

5. The method for fusing infrared and visible light images of a fire according to claim 1, characterized in that, The decoder includes multiple rows of decoding blocks, each row of decoding blocks corresponds to a feature map, and decoding blocks in the same row are connected by short connections. Each decoding block consists of two 3×3 convolutional layers.

6. A fire infrared and visible light image fusion device, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements a fire infrared and visible light image fusion method as described in any one of claims 1-5.

7. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements a fire infrared and visible light image fusion method as described in any one of claims 1-5.

Images