An image fusion method and system with low-light enhancement and overexposure suppression

Abstract

The application discloses an image fusion method and system with low-light enhancement and overexposure inhibition, and belongs to the technical field of image processing. The method comprises the following steps: in the image fusion model training process, the reflectivity feature map, the visible light feature map and the dark channel feature map of a visible light image and the infrared feature map of an infrared image are extracted; an enhanced reflectivity image is obtained according to the reflectivity feature map; the reflectivity feature map, the visible light feature map and the infrared feature map are fused and then subjected to color space conversion to obtain a color fusion image; the overexposure area and the non-overexposure area of the dark channel feature map are determined; the overexposure area is replaced by the features in the infrared feature map, and the non-overexposure area is replaced by the features in the enhanced reflectivity image to obtain an overexposure-inhibited fusion image; and the overexposure-inhibited fusion image is used in model loss calculation. The quality of the color fusion image is improved, and the technical problem that the image fusion quality is affected by overexposure is solved.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to an image fusion method and system that combines low-light enhancement and overexposure suppression. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Currently, methods for fusing visible light and infrared images mainly fall into two categories: traditional image fusion methods and deep learning-based methods. However, neither traditional nor existing deep learning-based image fusion methods consider the problem of overexposure and flare caused by headlights at night when fusing visible light and infrared images in low-light scenes, resulting in low-quality fused images. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes an image fusion method and system that combines low-light enhancement and overexposure suppression. By suppressing overexposed areas in the image, the quality of the color fused image is improved.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, an image fusion method that combines low-light enhancement and overexposure suppression is proposed, including: Acquire multiple paired visible light and infrared images; The image fusion model is trained using paired visible light and infrared images. Once training is complete, a well-trained image fusion model is obtained. During the training of the image fusion model, reflectance feature maps, visible light feature maps, and dark channel feature maps of visible light images and infrared feature maps of infrared images are extracted. The reflectance feature maps are then decoded to obtain an enhanced reflectance image. The reflectance feature map, visible light feature map, and infrared feature map are fused to obtain a fused feature map; the fused feature map is then subjected to color space conversion to obtain a color fused image. Identify the overexposed and non-overexposed regions in the dark channel feature map; replace the overexposed regions in the dark channel feature map with features in the infrared feature map, and replace the non-overexposed regions with features in the enhanced reflectance image to obtain a fused image with overexposure suppression. The overexposure-suppressed fused image is included in the loss calculation of the image fusion model.

[0006] Furthermore, the global mean of the dark channel feature map pixels is calculated and a set multiple of this mean is used as the criterion for determining overexposed and non-overexposed areas. Regions in the dark channel feature map with pixels larger than the determination criteria are identified as overexposed regions. Regions in the dark channel feature map with pixels less than or equal to the determination criteria are identified as non-overexposed regions.

[0007] Furthermore, obtain the overexposed area mask of the visible light image; The R, G, and B channel features of the visible light image are separated by channel splitting operation, and the minimum value between the three channel features is calculated sequentially by pixel-by-pixel minimization operation to obtain the initial feature map of the dark channel. Noise suppression and dark region enhancement are performed on the initial feature map of the dark channel to output a coarse feature map of the dark channel. Using the fused feature map as a guide image, the coarse feature map of the dark channel is filtered to obtain the final dark channel feature map.

[0008] Furthermore, the process of extracting the reflectance feature map of a visible light image includes: The visible light image is convolved multiple times to obtain the initial visible light features; Extract initial reflectance features that are unaffected by changes in illumination from the initial visible light features; The reflectance feature map is obtained by multiplying the initial reflectance feature with the initial visible light feature.

[0009] Furthermore, during the training process of the image fusion model, the illuminance feature map of the visible light image is extracted and decoded to obtain an enhanced illuminance image. The enhanced illuminance image and the enhanced reflectance image are then used in the loss calculation of the image fusion model.

[0010] Furthermore, the process of fusing the reflectance feature map, visible light feature map, and infrared feature map to obtain the fused feature map includes: The reflectance feature map and the infrared feature map are multiplied and then subjected to dense convolution to obtain the densely concatenated reflectance feature map. The densely concatenated reflectance feature map is multiplied with the visible light feature map to obtain the fused feature map after the first injection. The fused feature map after the first injection is concatenated with the visible light feature map through an attention mechanism to obtain the fused feature map after the second injection. The fused feature map after the second injection is decoded to obtain the fused feature map.

[0011] Secondly, an image fusion system that combines low-light enhancement and overexposure suppression is proposed, including: The image acquisition unit is used to acquire multiple paired visible light images and infrared images; The model training unit is used to train the image fusion model using paired visible light and infrared images. Upon completion of training, a trained image fusion model is obtained. During the training process, reflectance feature maps, visible light feature maps, and dark channel feature maps from the visible light image, and infrared feature maps from the infrared image, are extracted. The reflectance feature maps are decoded to obtain an enhanced reflectance image. The reflectance feature maps, visible light feature maps, and infrared feature maps are fused to obtain a fused feature map. The fused feature map undergoes color space conversion to obtain a color fused image. Overexposed and non-overexposed regions in the dark channel feature map are identified. Overexposed regions in the dark channel feature map are replaced with features from the infrared feature map, and non-overexposed regions are replaced with features from the enhanced reflectance image to obtain an overexposed-suppressed fused image. This overexposed-suppressed fused image is then used in the loss calculation of the image fusion model.

[0012] Thirdly, a computer device is proposed, the device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements an image fusion method that combines low-light enhancement and overexposure suppression as proposed in the first aspect.

[0013] Fourthly, a computer-readable storage medium is proposed, which stores a computer program adapted to be loaded and executed by a processor, namely, an image fusion method that combines low-light enhancement and overexposure suppression as proposed in the first aspect.

[0014] Fifthly, a computer program product is proposed, which includes a computer program that, when executed by a processor, implements an image fusion method that combines low-light enhancement and overexposure suppression as proposed in the first aspect.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes an image fusion method and system that combines low-light enhancement and overexposure suppression. When training an image fusion model using paired visible light and infrared images, the method fuses reflectance, visible light, and infrared feature maps extracted from the visible light and infrared images to obtain a fused feature map. This fused feature map undergoes color space conversion to obtain a color fused image. Furthermore, by identifying overexposed and non-overexposed regions in the visible light feature map, and then replacing the features of the overexposed and non-overexposed regions in the dark channel feature map with features from the infrared and emissivity feature maps respectively, an overexposure-suppressed fused image is obtained. This overexposure-suppressed fused image is then incorporated into the loss calculation of the image fusion model to suppress overexposure in the color fused image, thereby improving the generation quality of the color fused image.

[0016] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0018] Figure 1 This is a flowchart of an image fusion method that combines low-light enhancement and overexposure suppression, as proposed in an embodiment of the present invention. Figure 2 This is a diagram illustrating the overall architecture of the image fusion model proposed in this embodiment of the invention. Figure 3 This is a diagram of the illumination adaptive correction network and decoder architecture proposed in an embodiment of the present invention; Figure 4 This is a diagram of the feature learning network architecture proposed in an embodiment of the present invention; Figure 5 This is a diagram of the window attention architecture proposed in an embodiment of the present invention; Figure 6 This is a diagram of the overexposure suppression branch architecture proposed in an embodiment of the present invention; Figure 7 This is a diagram of the lightweight, densely connected, converged network architecture proposed in an embodiment of the present invention; Figure 8 The image fusion method proposed in this embodiment of the invention and the fused image obtained from random test images by seven existing image fusion methods are shown. Figure 8 Image (a) is an infrared image from a random test image. Figure 8 (b) is a visible light image of a low-light scene from a random test image. Figure 8(c) shows the color fused image obtained by fusing random test images using the SeA image fusion method enhanced by the Zero-DCE low-light enhancement method. Figure 8 Image (d) shows the color fused image obtained by fusing random test images using the PIA image fusion method enhanced by the Zero-DCE low-light enhancement method. Figure 8 Image (e) shows the color fused image obtained by fusing random test images using the PIA image fusion method enhanced by the SCI low-light enhancement method. Figure 8 In the middle (f), the color fused image is obtained by fusing random test images using the DIVFusion image fusion method. Figure 8 In the middle (g), the color fused image is obtained after fusing the random test images using the DDBF image fusion method. Figure 8 Image (h) is the color fused image obtained by fusing random test images using the LENFusion image fusion method. Figure 8 In the middle (i), the color fused image is obtained by fusing random test images using the SDS-Fusion image fusion method. Figure 8 In the middle (j), the color fused image is obtained by fusing random test images using the DFVO image fusion method. Figure 8 In the middle (k), a color fused image is obtained by fusing a random test image using the method proposed in the embodiments of the present invention; Figure 9 The image fusion method proposed in this embodiment of the invention is compared with the fused image obtained by seven existing image fusion methods based on a generalization test image. Figure 9 Image (a) is the infrared image in the generalization test image. Figure 9 (b) is the visible light image of the low-light scene in the generalization test image. Figure 9 Image (c) shows the color fused image obtained by fusing the generalization test image using the SeA image fusion method enhanced by the Zero-DCE low-light enhancement method. Figure 9 (d) shows the color fused image obtained by fusing the generalization test image using the PIA image fusion method enhanced by the Zero-DCE low-light enhancement method. Figure 9 Image (e) shows the color fused image obtained after fusing the generalization test image using the PIA image fusion method enhanced by the SCI low-light enhancement method. Figure 9 In the middle (f), the color fused image is obtained after fusing the generalization test image using the DIVFusion image fusion method. Figure 9 In the middle (g), the color fused image is obtained after fusing the generalization test image using the DDBF image fusion method. Figure 9Image (h) is the color fused image obtained after fusing the generalization test image using the LENFusion image fusion method. Figure 9 In the middle (i), the color fused image is obtained after fusing the generalization test image using the SDSFusion image fusion method. Figure 9 Image (j) is the color fused image obtained after fusing the generalization test image using the DFVO image fusion method. Figure 9 In the image (k), the color fused image is obtained by fusing the generalized test image using the method proposed in the embodiments of the present invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0020] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0021] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0022] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0023] It should be noted that all data acquisition is conducted in accordance with laws and regulations and with user consent, and the data is used legally.

[0024] First, the application background of an image fusion method that combines low-light enhancement and overexposure suppression proposed in the embodiments of the present invention will be explained.

[0025] The present invention proposes an image fusion method that combines low-light enhancement and overexposure suppression, which is applied to the application scenario of visible light image and infrared image fusion.

[0026] Currently, methods for fusing visible light and infrared images mainly fall into two categories: traditional image fusion methods and deep learning-based methods. However, both traditional and existing deep learning-based image fusion methods still perform poorly in fusing visible light and infrared images in low-light scenes. This is primarily because, under low-light conditions, the texture details in visible light images are subject to degradation due to factors such as changes in illumination, including detail loss and noise interference. Furthermore, existing low-light image fusion methods do not adequately consider the impact of overexposure at night, such as flare caused by headlights. While infrared images can provide clear target outlines, they lack in texture and detail.

[0027] Therefore, in order to improve the quality of the fused image obtained by fusing visible light and infrared images, this invention proposes an image fusion method that combines low-light enhancement and overexposure suppression. By suppressing overexposure areas in the image, the quality of the color fused image is improved.

[0028] like Figures 1-9 As shown in the embodiment of the present invention, an image fusion method that combines low-light enhancement and overexposure suppression includes: Acquire multiple paired visible light and infrared images; The image fusion model is trained using paired visible light and infrared images. Once training is complete, a well-trained image fusion model is obtained. During the training of the image fusion model, reflectance feature maps, visible light feature maps, and dark channel feature maps of visible light images and infrared feature maps of infrared images are extracted. The reflectance feature maps are then decoded to obtain an enhanced reflectance image. The reflectance feature map, visible light feature map, and infrared feature map are fused to obtain a fused feature map; the fused feature map is then subjected to color space conversion to obtain a color fused image. Identify the overexposed and non-overexposed regions in the dark channel feature map; replace the overexposed regions in the dark channel feature map with features in the infrared feature map, and replace the non-overexposed regions with features in the enhanced reflectance image to obtain a fused image with overexposure suppression. The overexposure-suppressed fused image is included in the loss calculation of the image fusion model.

[0029] The embodiments of the present invention have pre-constructed such as Figure 2 The image fusion model shown is used to train the constructed image fusion model using training images. The training images contain multiple paired visible light images and infrared images, as well as overexposed area masks corresponding to the visible light images.

[0030] During the training process of the image fusion model, the illuminance feature map of the visible light image is also extracted and decoded to obtain the enhanced illuminance image. The enhanced illuminance image and the enhanced reflectance image are then used in the loss calculation of the image fusion model.

[0031] This image fusion model takes visible light and infrared images as input and a predicted color fused image as output. It includes an infrared information extraction branch, a reflectance and illuminance feature extraction branch, a feature learning network, an overexposure suppression branch, and a lightweight dense connection fusion network. The infrared information extraction branch extracts thermal radiation information from the infrared image to obtain an infrared feature map. φ ir The process involves: extracting reflectance and illuminance feature maps from visible light images that are unaffected by changes in illumination through a reflectance and illuminance feature extraction branch; extracting visible light feature maps from visible light images through a feature learning network; fusing the reflectance, visible light, and infrared feature maps through a strongly quantized densely connected fusion network to obtain a fused feature map; performing color space conversion on the fused feature map to obtain a color fused image; identifying overexposed and non-overexposed regions in the visible light feature map through an overexposure suppression branch; replacing overexposed regions in the visible light feature map with features from the infrared feature map, and replacing non-overexposed regions with features from the reflectance feature map, to obtain an overexposure-suppressed fused image.

[0032] The infrared information extraction branch is a convolutional module I, which includes sequentially connected convolutional layers and LReLU layers. This branch extracts thermal radiation information from the infrared image to obtain an infrared feature map. φ ir .

[0033] The process of extracting the reflectance feature map of a visible light image by the reflectance and illuminance feature extraction branch includes: The visible light image is convolved multiple times to obtain the initial visible light features; Extract initial reflectance features that are unaffected by changes in illumination from the initial visible light features; Multiply the initial reflectance feature by the initial visible light feature to obtain the reflectance feature map. φ rm .

[0034] Specifically, by sequentially performing multi-layer convolution, global average pooling, and function activation on the initial visible light features, the reflectivity initial features that are unaffected by changes in illumination are obtained.

[0035] The reflectance and illuminance feature extraction branch includes a visible light information extraction branch, an illumination adaptive correction network, and a decoder connected in sequence. The visible light information extraction branch includes four convolutional modules I connected in sequence. After the visible light image passes through the four convolutional modules I connected in sequence, the initial visible light features are obtained. Figure 3 As shown, the illumination adaptive correction network includes a reflectance feature extraction branch and a degradation feature extraction branch. The illumination adaptive correction network performs dual-channel decomposition and attention weighting on the initial visible light features output by the visible light information extraction branch, and obtains the reflectance feature map through the reflectance feature extraction branch and the degradation illuminance feature extraction branch. φ rm The illuminance feature map provides a good feature foundation for subsequent low-light detail recovery; the decoder uses the reflectance feature map output by the illumination adaptive correction network. φ rm Decoding the illuminance feature map yields an enhanced reflectance image. φ rl And enhanced illumination images.

[0036] The reflectance feature extraction branch and the degradation feature extraction branch are two parallel branches, both connected to the subsequent decoder. The reflectance feature extraction branch is used to obtain a reflectance feature map that is unaffected by changes in illumination, while the degradation feature extraction branch is used to obtain a degraded illumination feature map in low-light scenes and then remove it.

[0037] The reflectance feature extraction branch sequentially performs multi-layer convolution, global average pooling, and function activation on the initial visible light features to obtain the initial reflectance features that are unaffected by changes in illumination. The initial reflectance features are then multiplied with the initial visible light features to obtain the reflectance feature map.

[0038] The reflectance feature extraction branch includes the channel attention module CAM_E; the channel attention module CAM_E includes two sequentially connected convolutional modules II, an average pooling layer, and a softmax layer; the initial visible light features are processed twice by the two sequentially connected convolutional modules II, and the features after the two convolutional processes are processed by global average pooling through the average pooling layer to obtain the global average pooling features, and the global average pooling features are processed by function activation through the softmax layer to obtain the initial reflectance features.

[0039] The degradation feature extraction branch sequentially performs multi-layer convolution, global average pooling, and function activation on the initial visible light features to obtain the degradation initial illuminance features from the initial visible light features; the initial illuminance features are then multiplied with the initial visible light features to obtain the illuminance feature map.

[0040] The degradation feature extraction branch includes a channel attention module CAM_L; the channel attention module CAM_L includes two sequentially connected convolutional modules II, an average pooling layer, and a softmax layer; this degradation feature extraction branch performs two convolutional processes on the initial visible light features through the two sequentially connected convolutional modules II, and the features after the two convolutional processes are processed by global average pooling through the average pooling layer to obtain the features after global average pooling, and then the features after global average pooling are processed by function activation through the softmax layer to obtain the initial degradation illuminance features.

[0041] Each convolutional module II includes a convolutional layer, a batch normalization layer, and an LReLU layer connected in sequence.

[0042] Both reflectance and illuminance feature maps are input into the decoder, which includes a reflectance feature decoding sub-branch and a degraded illuminance feature decoding sub-branch. The reflectance feature decoding sub-branch consists of three convolutional modules I, convolutional layers, and a Sigmoid layer connected in sequence. The convolutional kernels in this sub-branch are 3x3, and each convolutional module I consists of a 3×3 convolution and LReLU. The number of channels in the four convolutional layers decreases sequentially from 32 to 16 to 8 to 4 to 1. The reflectance feature decoding sub-branch decodes the reflectance feature map step by step, and finally, after Sigmoid activation, an enhanced reflectance image is obtained. φ rl The degraded illumination feature decoding sub-branch includes three convolutional modules I, convolutional layers, and Sigmad layers. The output features of the first convolutional module I in the reflectance feature decoding sub-branch are concatenated with the output features of the second convolutional module I in the degraded illumination feature decoding sub-branch and then input into the second convolutional module I in the degraded illumination feature decoding sub-branch. The output features of the third convolutional module I in the degraded illumination feature decoding sub-branch are concatenated with the output features of the second convolutional module I in the degraded illumination feature decoding sub-branch and then input into the third convolutional module I in the degraded illumination feature decoding sub-branch. The output features of the third convolutional module I in the degraded illumination feature decoding sub-branch pass through the 3x3 convolutional layer and Sigmad layer of this branch in sequence, and then output an enhanced illumination image, thereby strengthening the inherent reflectance features of the object that are not affected by illumination and stripping away the degraded illumination features under low light.

[0043] This invention calculates the reconstruction loss using enhanced illuminance and enhanced reflectance images. This strips away the degenerated features.

[0044] Reconstruction loss It is constructed based on Retinex theory, as shown in equation (1): (1); in, For the i-th enhanced reflectance image, For the i-th enhanced illumination image, The original visible light image. This is the reconstructed i-th visible light image; For the batch size during training, this loss encourages the model to generate high-quality decomposition features while preserving information from the input image.

[0045] The process by which a feature learning network extracts visible light feature maps from a visible light image includes: Convolution processing is performed on the visible light image to obtain a shallow feature map of the visible light image; The shallow feature map is divided into windows to obtain the feature sequence; Attention features of the feature sequence are extracted using a window attention mechanism; Multi-layer perceptual processing is performed on attention features to obtain post-perceptual features; The features after perception are transposed, restored, and convolved, and then added to the feature sequence to obtain the additive features; Upsampling is performed on the additive features to obtain a visible light feature map.

[0046] like Figure 4 As shown, the feature learning network includes a first convolutional layer, a PatchEmbed module, a window attention module, an MLP module, a PatchUnEmbed module, and a second convolutional layer connected in sequence. The output of the PatchUnEmbed module is added to the output of the second convolutional layer to form a residual connection structure, which is connected to an upsampling module. The upsampling module outputs a visible light feature map. The first convolutional layer convolves the visible light image to obtain a shallow feature map. The PatchEmbed module performs a block embedding operation on the shallow feature map, converting spatial features into sequential features. The PatchEmbed module includes a window partitioning module, a flattening module, and a serialization transpose module. The window partitioning module divides the shallow feature map into non-overlapping patches according to the window size using an 8×8 window. The flattening module flattens the features within each patch output by the window partitioning module. The serialization transpose module transposes the spatial dimension of the features output by the flattening module to the sequence length dimension, thus obtaining a feature sequence.

[0047] The window attention module extracts attention features of the feature sequence through the window attention mechanism and models the spatial dependency of the sequence features. Then, the MLP module performs multilayer perceptual processing on the attention features output by the window attention module to obtain post-perceptual features. By performing nonlinear transformation on the attention features, the feature representation ability is enhanced. The MLP module consists of three parts connected in sequence: a linear layer, GELU activation, and a linear layer.

[0048] The PatchUnEmbed module transposes and restores the perceived features, converting them into spatial feature maps. Then, a second convolutional layer refines the spatial feature map output from the PatchUnEmbed module, enhancing detail representation. The residual connection unit fuses the feature map output from the second convolutional layer with the sequence features output from the PatchEmbed module, preventing feature degradation and obtaining additive features. These additive features are then upsampled by an upsampling module, ultimately outputting a visible light feature map with the same size as the input image and superior visual characteristics. The upsampling module consists of a 3×3 convolutional layer and a Sigmoid layer connected sequentially.

[0049] like Figure 4 As shown, the features of the input window attention module are first transformed by a linear layer to generate query (Q), key (K), and value (V) tensors. Then, the query tensor Q is scaled. Subsequently, the dot product of Q and K is calculated to obtain the original attention score. The attention score is then normalized by the Softmax function. Finally, the attention features after attention processing are obtained by projection through another linear layer, thereby capturing the spatial dependencies between elements in the feature sequence.

[0050] The lightweight, densely connected fusion network fuses reflectance feature maps, visible light feature maps, and infrared feature maps. The process of obtaining the fused feature map includes: The reflectance feature map and the infrared feature map are multiplied and then subjected to dense convolution to obtain the densely concatenated reflectance feature map. The densely concatenated reflectance feature map is multiplied with the visible light feature map to obtain the fused feature map after the first injection. The fused feature map after the first injection is concatenated with the visible light feature map through an attention mechanism to obtain the fused feature map after the second injection. The fused feature map after the second injection is decoded to obtain the fused feature map, which is the fused feature map of the Y channel.

[0051] A lightweight, densely connected fusion network is used to convert reflectance feature maps unaffected by changes in illumination. φ rm Refined visible light feature map φ re With infrared feature map φ ir Input a lightweight, densely connected fusion network to obtain a fused feature map of the Y channel.

[0052] like Figure 7As shown, the lightweight densely connected fusion network includes an infrared feature input branch, a visible light feature channel adjustment branch, a densely connected convolutional submodule, a shallow injection submodule for refined feature maps, a deep injection submodule, and a decoder submodule. The outputs of the infrared feature input branch and the visible light feature channel adjustment branch are connected via a concat layer and then fed into the input of the densely connected convolutional submodule. The densely connected convolutional submodule comprises a first convolutional module I, a second convolutional module I, a third convolutional module I, a fourth convolutional module I, and a fifth convolutional module I connected sequentially. The outputs of the first and second convolutional modules I are connected via a concat layer and then fed into the third convolutional module I. The outputs of the first, second, and third convolutional modules I are connected via a concat layer and then fed into the fourth convolutional module I. The outputs of the densely connected convolutional submodules and the outputs of the shallow injection submodule for refined feature maps are connected via element-wise multiplication to realize the visible light feature map. φ re The initial injection of the fused feature map yields the fused feature map after the initial injection. Simultaneously, the deep injection submodule is connected through an attention mechanism module to obtain the fused feature map after the second injection. The fused feature map after the second injection is connected to the input of the decoder submodule, and finally decoded to obtain the final fused feature map of the Y channel.

[0053] The infrared feature input branch is used to output infrared feature maps. φ ir It preserves and highlights key thermal radiation information in infrared images, while the visible light feature channel adjustment branch is used to process the input reflectance feature map. φ rm The channel dimension is adjusted to match the number of channels in the infrared feature map, outputting a channel-adapted reflectance feature map. In the densely connected convolutional submodule, the first convolutional module I performs preliminary feature fusion and enhancement on the concat multimodal initial features, generating the first-stage fused features. The second convolutional module I further extracts deep correlation information from the first-stage fused features, generating the second-stage fused features. The third convolutional module I fuses the first and second-stage fused features, integrating multi-scale information to enhance cross-modal feature interaction, obtaining the third-stage fused features. The fourth convolutional module I further fuses the first, second, and third-stage fused features, mining more complex complementary feature associations, obtaining the fourth-stage fused features. The fifth convolutional module I optimizes the channels and details of the fourth-stage fused features, outputting a feature map adapted for the injection operation, i.e., the reflectance densely connected features. The refined feature map shallow injection submodule is used to refine the input visible light feature map. φ reThe refined visible light feature map is processed by element-wise multiplication. φ re After injecting densely connected reflectance features, the initial enhancement of the fused features is achieved. The deep injection submodule, combined with the attention mechanism module, performs cross-layer information association and key feature enhancement on the fused features after the initial injection, achieving a secondary enhancement of key details and obtaining the fused features after the secondary injection. The decoder submodule is used to decode the fused features after the secondary injection to generate the final fused feature map of the Y channel. The combination of the above dense connections and multi-stage injection achieves efficient fusion and detail preservation of infrared and visible light cross-modal features.

[0054] The overexposure suppression branch identifies the overexposure and non-overexposure regions in the dark channel feature map; the overexposure regions in the dark channel feature map are replaced with features in the infrared feature map, and the non-overexposure regions are replaced with features in the enhanced reflectance image, to obtain an overexposure-suppressed fused image.

[0055] The process of determining overexposed and non-overexposed areas in the dark channel feature map includes: Calculate the global mean of the pixels in the dark channel feature map, and take a set multiple of this mean as the criterion for determining overexposed and non-overexposed areas; Regions in the dark channel feature map with pixels larger than the determination criteria are identified as overexposed regions. Regions in the dark channel feature map with pixels less than or equal to the determination criteria are identified as non-overexposed regions.

[0056] The process of extracting dark channel feature maps from visible light images includes: Obtain the mask of overexposed areas in a visible light image; The R, G, and B channel features of the visible light image are separated by channel splitting operation, and the minimum value between the three channel features is calculated sequentially by pixel-by-pixel minimization operation to obtain the initial feature map of the dark channel. Noise suppression and dark region enhancement are performed on the initial feature map of the dark channel to output a coarse feature map of the dark channel. Using the fused feature map as a guide image, the coarse feature map of the dark channel is filtered to obtain the final dark channel feature map.

[0057] like Figure 6As shown, the overexposure suppression branch includes a dark channel calculation sub-branch, a guided filter optimization sub-branch, and an overexposure region adaptive judgment sub-branch connected in sequence. The dark channel calculation sub-branch extracts dark channel features from the input visible light image by calculating the minimum values ​​of the R, G, and B channels and performing morphological erosion to obtain a coarse dark channel feature map reflecting the distribution of dark areas in the image. The guided filter optimization sub-branch calculates the mean of a local 30×30 window and calculates the offset based on the filter coefficients of covariance and variance, finally smoothing with the mean of the 30×30 window to generate a refined dark channel feature map. The overexposure region mask is a binary mask generated by text-guided ClipSeg semantic segmentation to mark overexposure regions. The overexposure region adaptive judgment sub-branch performs feature processing by applying differential losses to overexposure and non-overexposure regions based on the overexposure region mask. Overexposure regions are replaced with infrared feature maps, while non-overexposure regions retain enhanced reflectivity images, resulting in a fused image with suppressed overexposure regions and complete details, called the overexposure-suppressed fused image.

[0058] The dark channel calculation sub-branch separates the R, G, and B channel features of the original visible light image through channel splitting operation, and calculates the minimum value between the three channels sequentially by pixel-by-pixel minimization operation to obtain the initial feature map of the dark channel; then, a 10×10 local rectangular structuring element is constructed, and noise suppression and dark region enhancement are performed on the initial feature map of the dark channel through morphological erosion operation to output the coarse feature map of the dark channel.

[0059] The guided filtering optimization sub-branch uses the coarse feature map of the dark channel as the image to be filtered and the fused feature map of the Y channel as the guide image. It performs local mean calculation, and then smooths and outputs a refined dark channel feature map by calculating the filtering coefficients. φ dc_o .

[0060] The overexposure region adaptive decision sub-branch uses overexposure region masks and refined dark channel feature maps. φ dc_o Enhanced reflectance image φ rl Infrared feature map φ ir To guide the process, constraints are imposed on the Y-channel fused image. First, the global mean *m* of the dark channel feature map pixels is calculated, and a threshold coefficient of 7 is set. A judgment condition is established: when the pixel value in the refined dark channel image is greater than 7 × *m*, the region containing that pixel is determined to be an overexposed area, and the dark channel feature map is then... φ dc_o The visible light features of the overexposed area are replaced with infrared feature maps. φ irMedium feature; when the pixel value is less than or equal to 7×m, the region where the pixel is located is determined to be a non-overexposed region, and the dark channel feature map is... φ dc_o The features of overexposed areas in central and southern Africa were replaced with enhanced reflectance images. φ rl By analyzing the mid-range features, a fused image with overexposure suppression is obtained. φ R By zooming in on the fused feature map of the Y channel and... φ R and φ ir The distance is reduced, thus minimizing overexposure. An overexposure area mask can help distinguish between overexposed and non-overexposed areas. By applying differentiated losses to overexposed and non-overexposed areas, the overexposed areas are brought closer together. φ dc_o and φ ir The distance further suppresses overexposure of the fused feature map.

[0061] In this embodiment of the invention, the image fusion model is trained using training images, and the trained image fusion model is tested using test images. The trained image fusion model is then used to fuse visible light images and infrared images.

[0062] The training process of the image fusion model includes: S1. Obtain training and test images; This invention uses the LLVIP dataset as the core experimental dataset. This dataset contains a large number of paired infrared and visible light images in low-light nighttime environments, covering overexposure scenes caused by strong light sources such as vehicle headlights and streetlights. This invention selects and obtains 290 pairs of paired low-light visible light and infrared images from this dataset. 240 pairs are used as training images for training the image fusion model; the remaining 50 pairs are used as test images to verify the fusion performance of the image fusion model in low-light overexposure scenes.

[0063] In this embodiment of the invention, the LOL dataset is used as an auxiliary dataset. 485 pairs of visible light images under normal light conditions from the LOL dataset are introduced for training the feature learning network. This can provide a reference for the distribution of features in normal light images and guide the distribution alignment of visible light features under weak light conditions.

[0064] The MSRS dataset was used as the generalization test dataset. Thirty pairs of infrared and visible light images were randomly selected from it to verify the fusion generalization performance of the model in low-light overexposure scenarios.

[0065] The LLVIP, LOL, and MSRS datasets are all existing datasets. The LLVIP dataset can be obtained from: https: / / github.com / bupt-ai-cz / LLVIP; the LOL dataset can be obtained from: https: / / daooshee.github.io / BMVC2018website / ; and the MSRS dataset can be obtained from: https: / / github.com / Linfeng-Tang / MSRS.

[0066] S2. Train the constructed image fusion model using the training images. The specific steps are as follows: S2-1. Convert the visible light image in the low-light scene of the training image into a YCbCr image. Then separate the Y channel image, Cb channel image and Cr channel image of the visible light image to obtain the Y channel image, Cb channel image and Cr channel image of the visible light image. S2-2. Input the Y-channel image of the visible light image obtained in step S2-1 of the training set into the reflectance and illuminance feature extraction branch, the feature learning network, and the infrared information extraction branch, respectively. The reflectance and illuminance feature extraction branch outputs the reflectance feature map of the Y-channel image. φ rm Enhanced reflectance image φ rl The enhanced illumination image and the feature learning network output a refined visible light feature map. φ re The infrared information extraction branch outputs an infrared feature map containing thermal radiation information. φ ir ; S2-3, Adaptive loss correction based on illumination Calculate the loss value of the illumination adaptive correction network, optimize the loss gradient using the Adam optimizer and backpropagate to update the model parameters of the illumination adaptive correction network; Illumination adaptive correction loss Reconstruction loss including visible light low-light enhancement images Cross-correlation loss of constrained illumination images Illuminance smoothing loss And the perceptual loss of enhanced reflectivity images The calculation formula is shown in equation (2): (2); In equation (2), These are all hyperparameters, and they are set to... .

[0067] In equation (2), the reconstruction loss The calculation formula is shown in equation (3): (3); In equation (3), For the i-th enhanced reflectance image, For the i-th enhanced illumination image, The original visible light image, and Represents the height and width of the image; In equation (2), the cross-correlation loss The calculation formula is shown in equation (4): (4); In equation (4), , The parameter representing the intensity of exponential decay. An image for enhanced illumination.

[0068] In equation (2), the illuminance smoothing loss The calculation formula is shown in equation (5): (5); In equation (5), To enhance the illumination image, This indicates that the gradient direction is calculated along the horizontal and vertical directions of the image, respectively. This represents the Sobel operator, used to capture edge and structural information. This refers to a small constant that prevents division by zero, ensuring that the denominator is non-zero.

[0069] In equation (2), the perceptual loss of the enhanced reflectance image The calculation formula is shown in equation (6): (6); In formula (6) For enhanced reflectance images, the 3 represents three channels. This is the visible light image after histogram equalization. This represents the feature extraction function of the conv5_1 layer of VGG16. It is an L1 norm.

[0070] Loss based on feature learning Calculate the loss value of the feature learning network, optimize the loss gradient using the Adam optimizer, backpropagate, and update the model parameters of the feature learning network. Feature learning loss in the embodiments of the present invention The binary cross-entropy loss (BCELoss) is used for optimization, and the calculation formula is shown in equation (7): (7); In equation (7), This is a visible light feature map, used to force visible light images under dark conditions to learn the image distribution under normal light conditions. This is the original visible light image under low-light conditions.

[0071] The infrared image from the training images and the Y-channel image of the visible light image obtained in step S2-1 are input into the infrared information extraction branch, reflectance and illuminance feature extraction branch, and feature learning network of step S2 for forward propagation once to obtain the infrared feature map of the Y-channel image. φ ir Reflectance characteristic map φ rm And the refined visible light feature map φ re The reflectance feature map of the obtained Y channel image. φ rm Refined visible light feature map φ re and infrared feature map φ ir The input is propagated forward once into a lightweight, densely connected fusion network to obtain the fused feature map of the Y channel; the fusion loss is then constructed. And based on fusion loss Calculate the loss value of the lightweight densely connected fusion network, optimize the loss gradient using the Adam optimizer and backpropagate to update the model parameters of the entire network.

[0072] After obtaining the fusion feature map of the Y channel, the color space of the fusion feature map is converted. The fusion feature map of the Y channel is then stitched together with the Cb and Cr channel images of the visible light image in the channel dimension. After stitching, the image is converted to RGB format to obtain a color fusion image.

[0073] Fusion loss Including auxiliary strength loss Visible light gradient loss Infrared gradient loss Color consistency loss and the loss of exposure .

[0074] Fusion loss The calculation formula is shown in equation (8): (8); In equation (8), These are hyperparameters; in this embodiment, the hyperparameters are set as follows: .

[0075] In equation (8), the auxiliary strength loss The calculation formula is shown in equation (9): (9); In equation (9), and Represents the height and width of the Y-channel fused image. This represents the fused feature map of the Y channel. Infrared image; In equation (9), the visible light gradient loss The calculation formula is shown in equation (10): (10); In equation (10), and Represents the height and width of the Y-channel fused image. This represents the gradient value of the fused feature map of the Y channel. This represents the gradient value of the original visible light image. This represents the gradient value of the enhanced reflectance image. Represents the calculation of the first norm. This represents taking the absolute value. Represents the selection of the largest element; In equation (9), the infrared gradient loss The calculation formula is shown in equation (11): (11); In equation (11), and Represents the height and width of the Y-channel fused image. This represents the fused feature map of the Y channel. Infrared image, Represents the calculation of the first norm; In equation (9), color consistency loss The calculation formula is shown in equation (12): (12); In equation (12), and Represents the height and width of the Y-channel fused image. Represents the original visible light image. Represents a color-blended image. Indicates the number of image channels. This represents the pixel-level discrete cosine similarity calculation across the R, G, and B channels. The total number of pixels in the image. Represents the R, G, and B channels.

[0076] In equation (9), the overexposure suppression loss There are different calculation formulas for overexposed and non-overexposed areas. The calculation formula for the mask area is shown in equation (13): (13); The formula for calculating the non-masked region is shown in equation (14): (14); In equations (13) and (14), To reduce exposure loss in overexposed areas, For the exposure loss of non-masked areas, For the mask, -Mask (since the mask is a binary mask with only 0 and 1 values, the non-masked area can be obtained by subtracting the mask area from 1), γ1 is 0.1, which is the hyperparameter for balancing the loss weights, N is the number of iterations, and α1, α2, α3, and α4 are the hyperparameters for balancing the loss weights. For content loss, The structural loss is calculated using equations (15) and (16), where N represents the number of iterations and Mask represents the overexposed area. This represents a small constant that avoids division by zero.

[0077] (15); (16); In equations (15) and (16), The image is a fused image after overexposure suppression. Features extracted by the Sobel operator The hyperparameters for balancing the loss term are calculated as follows: the gradient is extracted using the Sobel operator, and then the average value of the gradient is calculated. ), Represents Sobel operations. This is the fused feature map of the Y channel. This is an infrared image.

[0078] In the mask area N is the number of training rounds; In non-masked areas =0.3, =0.8. This indicates the calculation of the average value.

[0079] The training process of the image fusion model proposed in this embodiment of the invention includes the following steps: Repeat steps S2-2 to S2-3 until the number of iterations (epochs) reaches a set threshold to obtain the trained image fusion model. Each iteration of steps S2-2 to S2-3 constitutes one iteration. In this embodiment, the set threshold for the number of iterations during image fusion model training is 2400. In this embodiment, the image fusion model is trained using an NVIDIA RTX4090 GPU chip, and the Adam optimizer is used to optimize the loss gradient and perform backpropagation. The batch size is set to 16, and the initial learning rate is set to 1×10⁻⁶. -4 .

[0080] After training, a trained image fusion model is obtained. The trained image fusion model can be used to fuse visible light images and infrared images. During fusion, only the reflectance feature map, visible light feature map and infrared feature map of the visible light image are extracted. After fusing these three feature maps, color space conversion is performed to obtain the final color fused image.

[0081] test: To verify the fusion effect of the method proposed in this embodiment, the following image fusion methods were first used: SeA (from "Image fusion in the loop of high-level vision tasks: Asemantic-aware real-time infrared and visible image fusion network"), PIA (from "PIAFusion: A progressive infrared and visible image fusion network based on illumination aware"), DIVFusion (from "DIVFusion: Darkness-free infrared and visible image fusion"), DDBF (from "Dispel darkness for better fusion: A controllable visual enhancer based on cross-modal conditional adversarial learning"), LENFusion (from "Lenfusion: a joint low-light enhancement and fusion network for nighttime infrared and visible image fusion"), SDSFusion (from "SDSFusion: ASemantic-Aware Infrared and Visible Image Fusion Network for DegradedScenes"), and DFVO (from "DFVO: Learning Darkness-free") image fusion method. The seven existing image fusion methods (Visible and Infrared Image Disentanglement and Fusion All at Once) generate color fused images based on infrared images and visible light images in low-light scenes from the test set. Then, the color fused images are used to generate color fused images using existing Matlab test modules (available at: https: / / github).Visual effects were tested at (com / Linfeng-Tang / Evaluation-for-Image-Fusion). Since SeA and PIA are not specifically designed for image fusion under low-light conditions, the SCI low-light enhancement method (from "Toward Fast, Flexible, and Robust Low-Light Image Enhancement") and the Zero-DCE low-light enhancement method (from "Zero-reference deep curve estimation for low-light image enhancement") were used to enhance the low-light visible light images before inputting them into the network for testing. The SCI method was used for low-light enhancement of PIAFSion, and the Zero-DCE method was used for low-light enhancement of SeAFusion and PIAFSion. The test results are shown in Tables 1 and 2. Figure 8 As shown, the generalization test results on the MSRS dataset are as follows: Figure 9 As shown.

[0082] Table 1

[0083] In Table 1, CC (correlation coefficient) measures the linear correlation between the fused image and the source image; a higher value indicates a stronger ability of the fused image to retain information from the source image. MSE (mean square error) reflects the pixel-level difference between the fused image and the source image; a lower value indicates less distortion. PSNR (peak signal-to-noise ratio) evaluates image quality based on pixel differences; a higher value indicates less image distortion and better quality. QAB / F measures the edge information transferred from the source image to the fused image; a higher value indicates better image fusion. NAB / F measures the degree of artifacts introduced during the fusion process; a lower value indicates fewer artifacts. SSIM (structural similarity) evaluates the structural similarity between the fused image and the source image; a higher value indicates higher image quality. Furthermore, the method proposed in this embodiment is represented in Table 1 using the Ours method.

[0084] As can be seen from Table 1: 1) The method proposed in this embodiment of the invention can achieve a higher CC value. The CC value obtained by the method proposed in this embodiment of the invention is about 2.89% higher than that of DFVO (0.693), which has the highest CC value among existing methods. CC (correlation coefficient) quantifies the degree of linear correlation between the fused image and the source image. The higher the value, the more fully the fused image retains the original information of the source image. 2) The method proposed in this embodiment of the invention can achieve a lower MSE value. The MSE value obtained by the method proposed in this embodiment of the invention is reduced by approximately 32.43% compared to DFVO (0.037), which has the lowest MSE value among existing methods. MSE is the mean squared value of pixel grayscale error, which directly reflects the pixel-level difference between the fused image and the source image. The significant reduction in MSE indicates that this method performs more accurate matching and fusion of pixel grayscale information of the source image during the fusion process. 3) The method proposed in this embodiment of the invention can achieve a higher PSNR value. The PSNR value obtained by the method proposed in this embodiment of the invention is approximately 3.04% higher than that of DFVO (62.588), which has the highest PSNR value among existing methods. PSNR reflects the overall degree of image distortion; a higher value indicates a smaller deviation between the fused image and the source image at the pixel level. The method proposed in this embodiment of the invention makes the fused image more visually natural, avoiding interference from distortion problems such as artifacts, overexposure, and grayscale banding, and significantly optimizing image quality. 4) The method proposed in this embodiment of the invention can achieve a high QAB / F value. The highest QAB / F value obtained by the method proposed in this embodiment of the invention is SDSFusion (0.399), which is improved by about 4.26%. This indicates that in the fusion process, this application allows the fused image to retain the salient outline of the infrared target while incorporating the delicate texture boundaries of the visible light scene. The resulting color fused image contains richer edge information and has a better fusion effect. 5) The method proposed in this embodiment of the invention can achieve a lower NAB / F value. The NAB / F value obtained by the method proposed in this embodiment of the invention is lower than that of the existing DIVFusion and Zero-DCE enhanced SeAFusion image fusion methods (which have the lowest NAB / F value among the aforementioned existing image fusion methods). This indicates that the method proposed in this embodiment of the invention produces fewer artifacts in the fused color image during the image fusion process of infrared images and visible light images in low-light scenes, and better reduces the interference caused by degradation problems such as low light and overexposure. 6) The method proposed in this embodiment of the invention can achieve a higher SSIM value. The SSIM value obtained by the method proposed in this embodiment of the invention is 1.45% higher than that of the DFVO image fusion method in the prior art (which has the highest SSIM value among the above-mentioned existing image fusion methods). This shows that in the process of image fusion of infrared images and visible light images in low light scenes, the fused image has better similarity to the source image in core dimensions such as brightness distribution, contrast change, and structural morphology.

[0085] In addition, from Figure 8 and Figure 9It can also be seen that the color fusion image obtained by the method proposed in the embodiments of the present invention has higher contrast and clearer texture details. Therefore, the color fusion image obtained by the method proposed in the embodiments of the present invention is more in line with human visual perception.

[0086] This invention proposes an image fusion method that combines low-light enhancement and overexposure suppression. The four convolutional modules of the visible light information extraction branch can specifically extract basic features such as texture and edges in the Y channel of the low-light visible light image. The illumination adaptive correction network connected to the visible light information extraction branch separates reflectance and degraded illumination features through dual-channel decomposition. The reflectance feature map and the enhanced reflectance image output by the reflectance feature extraction branch can retain the core information of the object, such as inherent texture and edges, which are not affected by illumination. The degraded illumination feature extraction branch locates and removes degraded components such as noise and weak signal in dark areas under low-light conditions, which can solve the low-light degradation problem from a physical perspective and avoid noise amplification and color shift caused by over-enhancement.

[0087] The feature learning network achieves effective alignment of low-light and normal-light feature distributions through the synergistic effect of 3×3 convolutions, 8×8 window attention, and MLP layers. The window attention mechanism can simultaneously capture the local texture correlation and global structural dependence of features, while residual connections ensure that shallow details are not lost. The final output refined visible light feature map is superior to existing feature maps without distribution alignment in terms of edge integrity and texture richness, providing higher quality visible light feature support for cross-modal fusion.

[0088] The overexposure suppression branch innovatively combines dark channel priors with language-guided masks. The language-guided binary mask, combined with difference loss, can improve the accuracy of overexposure area recognition. By combining the dark channel calculation submodule and the guided filtering optimization submodule, a collaborative overexposure judgment condition can be constructed. It can adaptively replace infrared features in overexposure areas to retain target saliency and retain visible light details in non-overexposure areas. This can effectively avoid incomplete overexposure suppression or loss of details in non-overexposure areas, especially solving the problem of local overexposure in fused images caused by headlight flare and strong street light at night.

[0089] The single-group convolutional module I of the infrared information extraction branch is designed specifically for the thermal radiation characteristics of infrared images, enabling efficient capture of target contour features and avoiding redundant computation. Its output features, along with visible light intermediate layer reflectivity features and refined feature maps, achieve cross-modal fusion through a lightweight, densely connected fusion network. The densely connected convolutional submodule integrates the complementarity of multimodal features, and shallow and deep injection ensures full interaction between visible light textures and infrared target features. The final color fused image is obtained by stitching the Y-channel fused image with the Cb and Cr channels, and optimizing using a color constraint function. Compared to existing methods that only output single-channel fused images or ignore color consistency, this effectively alleviates color cast issues. Test results show that the color fused image generated by this invention outperforms existing methods in terms of detail preservation in dark areas and overexposure suppression, while also possessing better contrast and overall brightness, better conforming to human visual perception. This effectively solves the core problems of existing low-light image fusion methods, such as severe detail loss, insufficient overexposure suppression, and poor visual quality of the fused image.

[0090] This invention also proposes an image fusion system that combines low-light enhancement and overexposure suppression, comprising: The image acquisition unit is used to acquire multiple paired visible light images and infrared images; The model training unit is used to train the image fusion model using paired visible light and infrared images. Upon completion of training, a trained image fusion model is obtained. During the training process, reflectance feature maps, visible light feature maps, and dark channel feature maps from the visible light image, and infrared feature maps from the infrared image, are extracted. The reflectance feature maps are decoded to obtain an enhanced reflectance image. The reflectance feature maps, visible light feature maps, and infrared feature maps are fused to obtain a fused feature map. The fused feature map undergoes color space conversion to obtain a color fused image. Overexposed and non-overexposed regions in the dark channel feature map are identified. Overexposed regions in the dark channel feature map are replaced with features from the infrared feature map, and non-overexposed regions are replaced with features from the enhanced reflectance image to obtain an overexposed-suppressed fused image. This overexposed-suppressed fused image is then used in the loss calculation of the image fusion model.

[0091] It should be noted that the image fusion system with low-light enhancement and overexposure suppression provided in the above embodiments is only illustrated by the division of the functional modules described above when fusing visible light and infrared images. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image fusion system with low-light enhancement and overexposure suppression provided in the above embodiments and the image fusion method embodiment with low-light enhancement and overexposure suppression belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0092] The present invention also discloses a computer device, the device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements an image fusion method that combines low-light enhancement and overexposure suppression as disclosed in the embodiments of the present invention.

[0093] The present invention also discloses a computer-readable storage medium storing a computer program adapted for loading and execution by a processor of an image fusion method that combines low-light enhancement and overexposure suppression as disclosed in the embodiments of the present invention.

[0094] The present invention also discloses a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements an image fusion method that combines low-light enhancement and overexposure suppression as disclosed in the embodiments of the present invention.

[0095] The method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0096] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0097] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An image fusion method that combines low-light enhancement and overexposure suppression, characterized in that, include: Acquire multiple paired visible light and infrared images in low-light scenes; The image fusion model is trained using the paired visible light and infrared images. Once training is complete, a trained image fusion model is obtained. During the training of the image fusion model, the visible light image in the low-light scene is converted into a YCbCr image. Then, the Y channel image, Cb channel image, and Cr channel image of the visible light image are separated to obtain the Y channel image, Cb channel image, and Cr channel image of the visible light image. The reflectance feature map, visible light feature map, and dark channel feature map of the Y channel image of the visible light image and the infrared feature map of the infrared image are extracted. The reflectance feature map is decoded to obtain the enhanced reflectance image. The reflectance feature map, visible light feature map, and infrared feature map are fused to obtain the Y-channel fused feature map; the fused feature map is then subjected to color space conversion to obtain a color fused image, which is used as the output of the image fusion model. Identify the overexposed and non-overexposed regions in the dark channel feature map; replace the overexposed regions in the dark channel feature map with features in the infrared feature map, and replace the non-overexposed regions with features in the enhanced reflectance image to obtain a fused image with overexposure suppression. The overexposure-suppressed fused image is included in the fusion loss calculation of the image fusion model; Fusion loss Including auxiliary strength loss Visible light gradient loss Infrared gradient loss Color consistency loss and the loss of exposure ; Color consistency loss The calculation formula is as follows: ； In the formula, and Represents the height and width of the color-blended image. Represents the original visible light image. Represents a color-blended image. Indicates the number of image channels. This represents the pixel-level discrete cosine similarity calculation across the R, G, and B channels. The total number of pixels in the image. Represents the R, G, and B channels; Overexposure to suppress loss Different calculation formulas apply to overexposed and non-overexposed areas. The calculation formula for the mask area is as follows: ； Formula for calculating non-masked regions: ； In the formula, To reduce exposure loss in overexposed areas, For the deexposure loss in non-overexposed areas, For the overexposed area, -Mask is the non-masked region obtained by subtracting the masked region from 1, γ1 is the hyperparameter for balancing the loss weights, N is the number of iterations, and α1, α2, α3, and α4 are the hyperparameters for balancing the loss weights. This represents a small constant that should be avoided when dividing by zero; For content loss, For structural loss, the calculation formula is: ； ； In the formula, The image is a fused image after overexposure suppression. Features extracted by the Sobel operator To balance the hyperparameters of the loss term, Represents Sobel operations. This is the fused feature map of the Y channel. This is an infrared image.

2. The image fusion method that combines low-light enhancement and overexposure suppression as described in claim 1, characterized in that, Calculate the global mean of the pixels in the dark channel feature map, and take a set multiple of this mean as the criterion for determining overexposed and non-overexposed areas; Regions in the dark channel feature map with pixels larger than the determination criteria are identified as overexposed regions. Regions in the dark channel feature map with pixels less than or equal to the determination criteria are identified as non-overexposed regions.

3. The image fusion method that combines low-light enhancement and overexposure suppression as described in claim 1, characterized in that, Obtain the mask of overexposed areas in a visible light image; The R, G, and B channel features of the visible light image are separated by channel splitting operation, and the minimum value between the three channel features is calculated sequentially by pixel-by-pixel minimization operation to obtain the initial feature map of the dark channel. Noise suppression and dark region enhancement are performed on the initial feature map of the dark channel to output a coarse feature map of the dark channel. Using the fused feature map as a guide image, the coarse feature map of the dark channel is filtered to obtain the final dark channel feature map.

4. The image fusion method that combines low-light enhancement and overexposure suppression as described in claim 1, characterized in that, The process of extracting reflectance feature maps from visible light images includes: The visible light image is convolved multiple times to obtain the initial visible light features; Extract initial reflectance features that are unaffected by changes in illumination from the initial visible light features; The reflectance feature map is obtained by multiplying the initial reflectance feature with the initial visible light feature.

5. The image fusion method that combines low-light enhancement and overexposure suppression as described in claim 1, characterized in that, During the training process of the image fusion model, the illuminance feature map of the visible light image is also extracted and decoded to obtain the enhanced illuminance image. The enhanced illuminance image and the enhanced reflectance image are then used in the loss calculation of the image fusion model.

6. The image fusion method that combines low-light enhancement and overexposure suppression as described in claim 1, characterized in that, The process of fusing reflectance feature maps, visible light feature maps, and infrared feature maps to obtain a fused feature map includes: The reflectance feature map and the infrared feature map are multiplied and then subjected to dense convolution to obtain the densely concatenated reflectance feature map. The densely concatenated reflectance feature map is multiplied with the visible light feature map to obtain the fused feature map after the first injection. The fused feature map after the first injection is concatenated with the visible light feature map through an attention mechanism to obtain the fused feature map after the second injection. The fused feature map after the second injection is decoded to obtain the fused feature map.

7. An image fusion system that combines low-light enhancement and overexposure suppression, characterized in that, include: The image acquisition unit is used to acquire multiple paired visible light images and infrared images in low-light scenes; The model training unit is used to train the image fusion model using the paired visible light image and infrared image. After training is completed, a trained image fusion model is obtained. The image fusion model takes visible light and infrared images as inputs and the predicted color fused image as output. It includes an infrared information extraction branch, a reflectance and illuminance feature extraction branch, a feature learning network, an overexposure suppression branch, and a lightweight dense connection fusion network. During the training of the image fusion model, the visible light image in a low-light scene is converted into a YCbCr image. Then, the Y channel image, Cb channel image, and Cr channel image of the visible light image are separated to obtain the Y channel image, Cb channel image, and Cr channel image of the visible light image, respectively. The reflectance feature map, visible light feature map, and dark channel feature map of the Y channel image of the visible light image, and the infrared feature map of the infrared image are extracted. The reflectance feature map is decoded to obtain an enhanced reflectance image. The reflectance feature map, visible light feature map, and infrared feature map are fused to obtain the fused feature map of the Y channel. The fused feature map is then color space converted to obtain a color fused image, which is used as the output of the image fusion model. Identify the overexposed and non-overexposed areas in the dark channel feature map; The overexposed areas in the dark channel feature map are replaced with features in the infrared feature map, and the non-overexposed areas are replaced with features in the enhanced reflectance image to obtain an overexposed fused image; the overexposed fused image is then used in the fusion loss calculation of the image fusion model. Fusion loss Including auxiliary strength loss Visible light gradient loss Infrared gradient loss Color consistency loss and the loss of exposure ; Color consistency loss The calculation formula is as follows: ； In the formula, and Represents the height and width of the color-blended image. Represents the original visible light image. Represents a color-blended image. Indicates the number of image channels. This represents the pixel-level discrete cosine similarity calculation across the R, G, and B channels. The total number of pixels in the image. Represents the R, G, and B channels; Overexposure to suppress loss Different calculation formulas apply to overexposed and non-overexposed areas. The calculation formula for the mask area is as follows: ； Formula for calculating non-masked regions: ； In the formula, To reduce exposure loss in overexposed areas, For the deexposure loss in non-overexposed areas, For the overexposed area, -Mask is the non-masked region obtained by subtracting the masked region from 1, γ1 is the hyperparameter for balancing the loss weights, N is the number of iterations, and α1, α2, α3, and α4 are the hyperparameters for balancing the loss weights. This represents a small constant that should be avoided when dividing by zero; For content loss, For structural loss, the calculation formula is: ； ； In the formula, The image is a fused image after overexposure suppression. Features extracted by the Sobel operator To balance the hyperparameters of the loss term, Represents Sobel operations. This is the fused feature map of the Y channel. This is an infrared image.

8. An electronic device, characterized in that, The device includes: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements an image fusion method according to any one of claims 1-6 that combines low-light enhancement and overexposure suppression.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed by an image fusion method according to any one of claims 1-6 that combines low-light enhancement and overexposure suppression.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements an image fusion method that combines low-light enhancement and overexposure suppression as described in any one of claims 1-6.

Images