Illumination perception infrared and visible light image fusion method based on angular point attention

Abstract

The invention relates to the technical field of image processing, and provides a corner attention-based illumination perception infrared and visible light image fusion method, which comprises the following steps of: inputting an infrared image and a visible light image into respective corresponding feature extractors to carry out step-by-step iterative feature extraction; calculating a difference infrared image corner feature and a difference visible light image corner feature of a preset feature extraction level, and fusing the difference infrared image corner feature into a level infrared image feature extracted by the corresponding feature extraction level; fusing the difference visible light image corner point features into hierarchical visible light image features extracted by corresponding feature extraction hierarchies; and splicing the obtained visible light image features and infrared image features to obtain spliced image features, and inputting the spliced image features into a preset image reconstruction device for image reconstruction to obtain a fused image. According to the method, the expression of texture details, edges and structures in the fusion process can be more meticulously concerned.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for fusing illumination-aware infrared and visible light images based on corner attention. Background Art

[0002] As an advanced image processing method, multimodal image fusion technology is committed to integrating image data from different sensing technologies and imaging modes into an output image with large information content and rich content. Its core purpose is to combine the advantages of each image source, overcome the inherent limitations of a single image, and provide higher quality and more information-intensive image results. The application range of this technology is extremely wide, covering many important fields such as medical imaging, remote sensing, and computer vision.

[0003] At present, the fusion of infrared and visible light images is a typical task in multimodal image fusion research. This task aims to combine the characteristics of two different band images to create images with richer information and more comprehensive perspectives. Infrared images are valued for their night vision capabilities and ability to penetrate smoke, while visible light images are known for providing high-resolution details. However, the key challenge to achieving efficient fusion lies in how to effectively maintain and enhance the tiny texture structures in the original image while deeply fusing multimodal image information, while reducing noise interference and avoiding unnecessary artifacts. Solving these scientific problems is crucial to promoting progress in this field. Specifically, the core of the task is to achieve the best of both worlds by fusing infrared and visible light images - retaining the thermal radiation characteristics of infrared images while retaining the detail clarity of visible light images. In addition, it is necessary to further emphasize key features in the source image, such as tree shade under low illumination conditions, fine structures on building surfaces, and zebra crossing textures on the road surface, to improve the overall visual information content and recognizability of the image.

[0004] With the continuous advancement of deep learning technology, deep learning-based methods have become important tools in the field of multimodal image fusion, including convolutional neural networks (CNN), autoencoders (AE), generative adversarial networks (GAN), Transformers, diffusion models (DM), and Mamba models. In particular, GAN has shown significant advantages in generating high-quality image fusion results due to its unique adversarial training mechanism. Through the interaction between the generator and the discriminator, GAN can capture and reproduce more delicate and realistic image features. More importantly, GAN demonstrates powerful unsupervised learning capabilities without explicitly setting the loss function. Despite this, GAN-based methods also face some challenges, such as low computational efficiency when processing large-scale images and poor performance in spatial structure-sensitive tasks that require precise location information, which all affect the final fusion effect.

[0005] Currently, fusion models face key challenges. First, although the illumination-aware subnetwork can construct the illumination-aware loss of the model, the network's reasoning performance can be further improved. In addition, a single attention mechanism usually only focuses on one aspect of the source image. Therefore, it is unable to capture the complex relationship between global and local features when processing multimodal image information. Finally, due to the lack of visual consistency, the fused image will have unnatural artifacts, distortions, or other visual defects. Summary of the invention

[0006] The present invention proposes a method for fusion of infrared and visible light images based on illumination perception and corner attention, so as to solve or partially solve the above problems.

[0007] The present invention provides a method for fusion of infrared and visible light images based on illumination perception and corner attention, the method comprising:

[0008] Acquire the infrared image and visible light image to be fused;

[0009] Input the infrared image and the visible light image into the corresponding feature extractors including multiple feature extraction levels, respectively, perform iterative feature extraction on the infrared image and the visible light image through the corresponding feature extractors, and calculate the differential image corner point features of the hierarchical features extracted by the preset feature extraction level in the feature extractor during the iterative process, obtain the differential infrared image corner point features and the differential visible light image corner point features of the corresponding feature extraction level, fuse the differential infrared image corner point features into the hierarchical infrared image features extracted by the corresponding feature extraction level, fuse the differential visible light image corner point features into the hierarchical visible light image features extracted by the corresponding feature extraction level, and input the fused features into the next feature extraction level of the current feature extraction level of the corresponding feature extractors for iteration, so as to obtain the infrared image features and the visible light image features;

[0010] The visible light image feature and the infrared image feature are spliced ​​to obtain a spliced ​​image feature, and the spliced ​​image feature is input into a preset image reconstructor for image reconstruction to obtain a fused image.

[0011] Furthermore, the calculation of the differential image corner features of the hierarchical features extracted by the preset feature extraction level in the feature extractor to obtain the differential infrared image corner features and the differential visible light image corner features of the corresponding feature extraction level includes:

[0012] Hierarchical infrared image features extracted based on target features and hierarchical visible light image features Compute intermediate infrared image features at the target feature extraction level and intermediate visible light image features The calculation is as follows:

[0013]

[0014] in, represents the element-wise sum operation, represents channel multiplication operation, P(·) represents pReLU function, and PAP(·) represents power average pooling;

[0015] Calculate the infrared image corner weights W at the target feature extraction level respectively COA_ir and visible light image corner weight W COA_vi , calculated as follows:

[0016]

[0017]

[0018] Where b is the batch size of the sample, c is the index of the channel, i and j are the row and column indices of the corner, rows and cols are the row and column positions of the corner, respectively. Indicates the corner point information of the intermediate infrared image features of the target feature extraction level, Indicates the corner point information of the intermediate visible light image features extracted at the target feature extraction level;

[0019] Calculate the infrared image corner feature F at the target feature extraction level COA_ir and visible light image corner feature F COA_vi , calculated as follows:

[0020]

[0021] The intermediate infrared image features at the target feature extraction level are fused with the infrared image corner features to obtain the differential infrared image corner features at the target feature extraction level. The intermediate visible light image features at the target feature extraction level are fused with the visible light image corner features to obtain the differential visible light image corner features at the target feature extraction level. The calculation is as follows:

[0022]

[0023] Further, the step of fusing the differential infrared image corner point features into the infrared image features of the corresponding feature extraction level, and fusing the differential visible light image corner point features into the visible light image features of the corresponding feature extraction level, comprises:

[0024] After performing an element-wise sum operation on the infrared image features and the differential infrared image corner point features obtained in the current feature extraction level, they are input into the next feature extraction level of the infrared image; after performing an element-wise sum operation on the visible light image features and the differential visible light image corner point features obtained in the current feature extraction level, they are input into the next feature extraction level of the visible light image.

[0025] Furthermore, after obtaining the fused image, the method further includes:

[0026] The overall loss L is calculated based on the following loss function formula:

[0027] L=λ1·L illum +λ2·L aux +λ3·L texture +λ4·L vif +λ5·L Qabf

[0028] Among them, L illum Represents the illumination perception loss, L aux Indicates auxiliary strength loss, L texture represents texture loss, L vif represents the loss of visual information fidelity, L Qabf represents the boundary fidelity quality loss, λ1, λ2, λ3, λ4 and λ5 represent the weights of the loss function respectively;

[0029] Model parameters of the feature extractor and the image reconstructor are adjusted based on the overall loss L.

[0030] Furthermore, before calculating the overall loss L, the method further includes:

[0031] Inputting the visible light image into a preset illumination perception subnetwork to obtain a contribution value of the infrared image in a feature fusion process and a contribution value of the visible light image in a feature fusion process;

[0032] Based on the contribution value W of the infrared image in the feature fusion process ir and the contribution value W of the visible light image in the feature fusion process vi Calculate the illumination perception loss:

[0033]

[0034] in, represents the intensity loss of the infrared image, Represents the intensity loss of the visible light image.

[0035] Furthermore, the illumination perception subnetwork is used to perform the following image operations:

[0036] Based on the preset illumination perception backbone network, the probability value P of the visible light image being in daytime is obtained. day and the probability value P of being at night night ; Wherein, the illumination perception backbone network is a network model based on the DenseNet-201 architecture obtained by pre-training, including a plurality of cascaded dense blocks, a transition block connected between each dense block, and a classification layer connected after the last dense block. The dense blocks of each level receive the feature image of the previous level, and generate a new feature image after feature processing of the feature image of the previous level. The new feature image obtains the feature image of the current level after reducing the image features through the transition block. The classification layer is used to predict the illumination probability according to the obtained feature image;

[0037] The contribution value W of the infrared image in the image fusion process is calculated based on the probability value of the visible light image being in the daytime and the probability value of being in the nighttime. ir And the contribution value W of the visible light image vi :

[0038]

[0039] Further,

[0040] The auxiliary strength loss is expressed as:

[0041]

[0042] Among them, I ir Indicates infrared image, I vi represents the visible light image, I f represents the fused image, H and W represent the height and width respectively;

[0043] The texture loss is expressed as:

[0044]

[0045] The visual information fidelity loss is expressed as:

[0046] L vif =1-δ f,ir -δ f,vi

[0047]

[0048] Among them, subbands represents a set of specific subbands, and They represent the ideal information extracted from specific sub-bands of infrared image, visible light image and fused image respectively; Represents the infrared image and the visible light image C i The Sth element ofi represents the random field of the i-th subband in the infrared image and the visible light image; and They are from infrared images ir i , visible light image vi i and the fused image f i The Sth element of They represent the output signals of the preset human visual system model with the ith sub-band of the infrared image and the visible light image as input, represents the output signal of the human visual system model with the ith subband of the fused image as input; R S,i represents the model parameters of the i-th subband;

[0049] The boundary fidelity quality loss is expressed as:

[0050]

[0051] Here, n represents the number of source images.

[0052] Furthermore, the feature extractor is composed of N cascaded convolutional layers, wherein the convolution kernel of the convolution layer in the first feature extraction level is 1×1 to minimize the differences in visual features and data distribution between images, and the convolution kernels of the convolution layers in the second to Nth feature extraction levels are 3×3 to extract deep features of infrared images and visible light images, respectively; wherein the convolution layers in the second to N-1th feature extraction levels all introduce a multimodal differential perception fusion module based on corner attention, N≥3.

[0053] Furthermore, the image reconstructor is composed of N cascaded convolutional layers, wherein the convolution kernels of the 1st to N-1th convolutional layers are 3×3, and the convolution kernel of the Nth convolutional layer is 1×1.

[0054] The present application provides a method for fusion of infrared and visible light images based on illumination perception of corner attention. In the process of performing step-by-step iterative feature extraction on the infrared image and the visible light image respectively by using a feature extractor, the differential infrared image corner features and the differential visible light image corner features of the preset feature extraction level are calculated, and the differential infrared image corner features are fused into the hierarchical infrared image features of the corresponding feature extraction level, and the differential visible light image corner features are fused into the hierarchical visible light image features of the corresponding feature extraction level, and the image is reconstructed by a feature reconstructor to finally obtain a fused image. The corner attention mechanism proposed in the present invention optimizes the boundaries and corners in the image, thereby enhancing the capture of edge and structural details, which is crucial to maintaining the clarity and definition of the image. Through this attention, more detailed attention can be paid to the performance of texture details, edges and structures in the fusion process. This not only helps to retain the key information in the original image, but also enhances the visual effect of the fused image, so that it can show richer details and layering while maintaining high resolution.

[0055] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Various other advantages and benefits will become apparent to those skilled in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. In the accompanying drawings:

[0057] Figure 1 It is a flow chart of a method for fusion of infrared and visible light images based on illumination perception and corner attention according to an embodiment of the present invention;

[0058] Figure 2 A network structure diagram of the illumination-aware infrared and visible light image fusion method based on corner attention according to an embodiment of the present invention;

[0059] Figure 3 It is a corner attention network model of an embodiment of the present invention;

[0060] Figure 4 The figure shows the comparison of the fusion results of the existing method and the present invention on the MSRS dataset. DETAILED DESCRIPTION

[0061] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0062] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0063] refer to Figure 1 , Figure 2 The illumination-aware infrared and visible light image fusion method based on corner attention proposed in an embodiment of the present invention comprises the following steps:

[0064] S1, obtaining the infrared image and the visible light image to be fused;

[0065] S2, inputting the infrared image and the visible light image into the corresponding feature extractors including multiple feature extraction levels, respectively, performing iterative feature extraction on the infrared image and the visible light image through the corresponding feature extractors, and calculating the differential image corner point features of the hierarchical features extracted by the preset feature extraction level in the feature extractor during the iterative process, obtaining the differential infrared image corner point features and the differential visible light image corner point features of the corresponding feature extraction level, and fusing the differential infrared image corner point features into the hierarchical infrared image features extracted by the corresponding feature extraction level, fusing the differential visible light image corner point features into the hierarchical visible light image features extracted by the corresponding feature extraction level, and inputting the fused features into the next feature extraction level of the current feature extraction level of the corresponding feature extractors for iteration, so as to obtain the infrared image features and the visible light image features;

[0066] S3. Splicing the visible light image feature and the infrared image feature to obtain a spliced ​​image feature, and inputting the spliced ​​image feature into a preset image reconstructor to reconstruct the image, so as to obtain a fused image.

[0067] like Figure 2 As shown, the present invention uses CNN as the backbone network, and the backbone network includes Figure 2 The encoder shown in (a) and Figure 2The decoder shown in (b), wherein the encoder is used as a feature extractor to extract features of infrared images and features of visible light images. In a specific embodiment of the present invention, the feature extractor is composed of 5 convolutional layers, the purpose of which is to fully extract common features and complementary features in infrared images and visible light images. Initially, a 1×1 convolutional layer is used to minimize the differences in visual features and data distribution between images. Subsequently, four convolutional layers sharing the same weights are applied to process the two images to extract their deep features. Except for the first convolutional layer, the convolution kernel size of all convolutional layers is set to 3×3. The number of output channels of each layer is 16, 16, 32, 64 and 128, respectively. In addition, the PReLU function is selected as the activation function of each layer of the feature extractor in the present invention.

[0068] It should be noted that the number of convolutional layers in the feature extractor of the present invention does not need to be limited to 5, that is, the feature extractor can be composed of N cascaded convolutional layers for processing infrared images and N cascaded convolutional layers for processing visible light images in parallel, wherein the convolution kernel of the convolutional layer in the first feature extraction level is 1×1 to minimize the differences in visual features and data distribution between images, and the convolution kernels of the convolutional layers in the second to Nth feature extraction levels are 3×3 to extract deep features of infrared images and visible light images, respectively; wherein the convolutional layers in the second to N-1th feature extraction levels all introduce a multimodal differential perception fusion module based on corner attention, and N≥3.

[0069] Among them, the infrared image is represented by I ir , the visible light image is represented as I vi , the infrared image feature obtained by the feature extractor is represented as F ir , the visible light image feature is represented by F vi After obtaining the infrared image features and the visible light image features, they are spliced ​​to obtain the spliced ​​image features F fusion , the splicing strategy is as follows:

[0070] F fusion =Concat(F ir ,F vi ) (1)

[0071] Among them, Concat(·) represents the connection on the channel dimension.

[0072] Furthermore, in a specific embodiment of the present invention, the decoder acts as an image reconstructor and uses the spliced ​​image feature F fusionAs input, it consists of five convolutional layers, which aims to merge image information and produce more comprehensive and valuable visual data. The convolution kernels of the first four layers are 3×3, and the fifth layer is 1×1. The number of channels of each layer of the image reconstructor is 128, 64, 32, 16 and 1 respectively. Specifically, it can be expressed as:

[0073] Ι fusion =Rec I (F fusion ) (2)

[0074] Among them, RecI represents the image reconstructor, I fusion Represents the fusion feature F from the image fusion The restored fused image.

[0075] It can be understood that in an optional embodiment of the present invention, the image reconstructor can also be composed of N cascaded convolutional layers, wherein the convolution kernels of the 1st to N-1th convolutional layers are 3×3, and the convolution kernel of the Nth convolutional layer is 1×1.

[0076] Furthermore, in step S2, the embodiment of the present invention uses a multimodal differential perception fusion module based on corner attention to extract corner information of the infrared image and the visible light image. Figure 2 (d) and Figure 3 The multimodal differential perception fusion module based on corner attention provided in an embodiment of the present invention is introduced in detail.

[0077] It should be noted that the working principle of the multimodal differential perception fusion module based on corner attention in the embodiment of the present invention can be summarized as follows: (1) the source image features are input into the differential amplifier as positive and negative complementary components respectively. (2) The differential amplifier amplifies the difference between the two signals.

[0078] The present invention adopts represents the hierarchical infrared image features of the target feature extraction level, that is, the hierarchical features extracted from the infrared image in the i-th convolutional layer, Represents the hierarchical visible light image features of the target feature extraction level, that is, the hierarchical features extracted from the visible light image in the i-th convolutional layer. The image features of the two modalities can also be expressed as:

[0079]

[0080] in, For shared features, and is a complementary component. This module can identify and amplify the difference features in the image, thereby helping to retain more details and information in the fused image. In addition, it also supports further integration of complementary feature information and channel weights. Therefore, the present invention adopts a multimodal difference perception fusion module based on corner attention to further extract the detail difference information in the image.

[0081] Specifically, in the embodiment of the present invention, the differential image corner point features of the hierarchical features extracted by the preset feature extraction level in the feature extractor are calculated to obtain the differential infrared image corner point features and the differential visible light image corner point features of the corresponding feature extraction level, including:

[0082] S21, hierarchical infrared image features extracted according to target features and hierarchical visible light image features Compute intermediate infrared image features at the target feature extraction level and intermediate visible light image features The calculation is as follows:

[0083]

[0084] in, represents the element-wise sum operation, represents channel multiplication operation, P(·) represents pReLU function, PAP(·) represents power average pooling, and They represent the intermediate infrared image features and the intermediate visible light image features at the target feature extraction level respectively;

[0085] Furthermore, the power average pooling PAP(·) first performs global average pooling and then raises the difference signal to the power of the pooling result. It is specifically expressed as:

[0086]

[0087] in, represents the differential signal of the two modes, b represents the batch size, c represents the number of channels, H and W represent the height and width respectively, and p represents the power operation. Compared with global average pooling, power average pooling is adaptive and can control the sensitivity to features by adjusting the power exponent. Moreover, power average pooling can better preserve the local details and edge information in the image. Therefore, power average pooling has higher adaptability, robustness and local feature retention.

[0088] S22, respectively calculate the infrared image corner weights W of the target feature extraction level COA_ir and visible light image corner weight W COA_vi , calculated as follows:

[0089]

[0090] Where b is the batch size of the sample, c is the index of the channel, i and j are the row and column indices of the corner, rows and cols are the row and column positions of the corner, respectively. Represents the information of the corner points of the intermediate infrared image features extracted at the target feature extraction level, Indicates the corner point information of the intermediate visible light image features extracted at the target feature extraction level;

[0091] Specifically, in the embodiment of the present invention, represents the corner response value of the pixel located at the bth batch, cth channel, row index rows[j] and column index cols[i] extracted from the intermediate infrared image of the target feature extraction level by the Harris corner detection algorithm, Represents the corner response value of the pixel located at the bth batch, cth channel, row index rows[j] and column index cols[i] extracted from the intermediate visible light image of the target feature extraction level by the Harris corner detection algorithm.

[0092] In the embodiment of the present invention, after obtaining the differential signal, the corner point information is obtained by the corner point attention module, specifically as follows: Figure 3 As shown, Figure 3 (a) Corner attention module for infrared images, Figure 3 (b) is the visible light image corner attention module, and the structures of the two are basically the same. Input the corner attention module of the infrared image, and input the intermediate visible light image features into the corner attention module of the visible light image to complete the operations in the above and subsequent steps.

[0093] S23, calculating the infrared image corner feature F at the target feature extraction level COA_ir and visible light image corner feature F COA_vi , calculated as follows:

[0094]

[0095] where F COA_ir Infrared image corner features representing the target feature extraction level, F COA_vi Visible light image corner features representing the target feature extraction level.

[0096] In the embodiment of the present invention, a channel multiplication operation is performed on the intermediate infrared image features and the infrared image corner point weights of the target feature extraction level to obtain the infrared image corner point features of the target feature extraction level; a channel multiplication operation is performed on the intermediate visible light image features and the visible light image corner point weights of the target feature extraction level to obtain the visible light image corner point features of the target feature extraction level; which is specifically expressed as:

[0097] S24, performing feature fusion on the intermediate infrared image features at the target feature extraction level and the infrared image corner point features to obtain differential infrared image corner point features at the target feature extraction level The intermediate visible light image features at the target feature extraction level are fused with the visible light image corner features to obtain the differential visible light image corner features at the target feature extraction level. The calculation is as follows:

[0098]

[0099] In an embodiment of the present invention, an element addition operation is performed on the infrared image corner point features at the target feature extraction level and the intermediate infrared image features to obtain the differential infrared image corner point features at the target feature extraction level; an element addition operation is performed on the visible light image corner point features at the target feature extraction level and the intermediate visible light image features to obtain the differential visible light image corner point features at the target feature extraction level.

[0100] Further, refer to Figure 2 It can be seen that in step S2, the differential infrared image corner point features at the target feature extraction level are obtained and differential visible light image corner features Afterwards, the hierarchical infrared image features obtained in the target feature extraction level are and differential infrared image corner features After performing the element summation operation, the next feature extraction level of the infrared image is input; the level visible light image features obtained in the current feature extraction level are and differential visible light image corner features After performing the element-wise summation operation, the visible light image is input to the next feature extraction level.

[0101] Furthermore, the loss function is crucial in the infrared and visible light image fusion model. It measures the difference between the model output and the real target, and guides the model to optimize the parameters by minimizing this difference, thereby effectively fusing the image features and improving the quality of the output image. Therefore, the embodiment of the present invention also designs a corresponding loss function to adjust the model parameters in the backbone network.

[0102] Specifically, after obtaining the fused image, the method further includes:

[0103] S4. Calculate the overall loss L based on the following loss function formula:

[0104] L=λ1·L illum +λ2·L aux +λ3·L texture +λ4·L vif +λ5·L Qabf (12)

[0105] Among them, L illum Represents the illumination perception loss, L aux Indicates auxiliary strength loss, L texture represents texture loss, L vif represents the loss of visual information fidelity, L Qabf represents the boundary fidelity quality loss, λ1, λ2, λ3, λ4 and λ5 represent the weights of the loss function respectively;

[0106] Among them, the illumination perception loss is expressed as:

[0107]

[0108] Among them, W vi Represents the contribution value of the visible light image, W ir represents the contribution value of the infrared image, represents the intensity loss of the infrared image, Represents the intensity loss of visible light images;

[0109] In the embodiment of the present invention, intensity loss refers to the difference or similarity in pixel intensity between the fused image and the source image. The intensity loss of an image is defined as follows:

[0110]

[0111] Here, H and W represent the height and width of the input image, respectively, and ||·||1 represents the l1 norm.

[0112] The auxiliary strength loss is expressed as:

[0113]

[0114] Among them, I ir Indicates infrared image, I vi represents the visible light image, I f represents the fused image, H and W represent the height and width respectively;

[0115] The texture loss is expressed as:

[0116]

[0117] The visual information fidelity loss is expressed as:

[0118]

[0119] Among them, subbands represents a set of specific subbands, and They represent the ideal information extracted from specific sub-bands of infrared image, visible light image and fused image respectively; Represents the infrared image and the visible light image C i The Sth element of i represents the random field of the i-th subband in the infrared image and the visible light image; and They are from infrared images ir i , visible light image vi i and the fused image f i The Sth element of They represent the output signals of the preset human visual system model with the ith sub-band of the infrared image and the visible light image as input, represents the output signal of the human visual system model with the ith subband of the fused image as input; R S,i represents the model parameters of the i-th subband;

[0120] Among them, the set of specific subbands is the grouping of different frequency ranges or spatial frequency components obtained in multi-resolution analysis and multi-band analysis (such as wavelet transform or discrete cosine transform). Subbands are the representation of different frequency regions after an image or signal has undergone a certain transformation (such as wavelet decomposition). Each subband represents a part of the features of infrared and visible light images, such as edge, texture and other detail information. The ideal information extracted from a specific subband represents the most helpful information that best represents the image feature task in infrared and visible light images. The ideal information should meet the following criteria: fidelity, complementarity, saliency, human visual system model, etc. The human visual system model is a mathematical and computational representation of the human eye and its related neural processing mechanisms, which is used to imitate or understand how humans perceive, process and interpret visual information.

[0121] The boundary fidelity quality loss is expressed as:

[0122]

[0123] Wherein, n represents the number of source images. In the present invention, the source images are input infrared images and visible light images, and thus n=2.

[0124] S5. Adjust the model parameters of the feature extractor and the image reconstructor based on the overall loss L.

[0125] Furthermore, the present invention calculates the contribution value of the visible light image and the contribution value of the infrared image in the image fusion process based on the illumination perception subnetwork. The network aims to improve the robustness and reasoning performance of image processing tasks by learning and understanding the light information in the image. During the day, visible light images capture a large amount of texture details and color information, while infrared images at night contain basic data such as thermal radiation. Therefore, the brightness probability serves as an indirect indicator of the information richness within the source image. Using this brightness probability, the brightness perception weights are calculated to represent the contribution of the source image through a brightness allocation mechanism. These brightness perception weights play a key role as parameters in the brightness perception loss function, guiding the training process of the framework. Refer to Figure 2 (c) Before calculating the overall loss L, the method of the embodiment of the present invention further includes:

[0126] S01. Inputting the visible light image into a preset illumination perception subnetwork model to obtain a contribution value of the infrared image in a feature fusion process and a contribution value of the visible light image in a feature fusion process;

[0127] S02, based on the contribution value W of the infrared image in the feature fusion process ir and the contribution value W of the visible light image in the feature fusion process vi The illumination perception loss is calculated.

[0128] Furthermore, the illumination perception subnetwork is used to complete image operations:

[0129] S011. Based on a preset illumination perception backbone network, obtain a probability value P that the visible light image is in daytime day and the probability value P of being at night night ; Wherein, the illumination perception backbone network is a network model based on the DenseNet-201 architecture obtained by pre-training, including a plurality of cascaded dense blocks, a transition block connected between each dense block, and a classification layer connected after the last dense block. The dense blocks of each level receive the feature image of the previous level, and generate a new feature image after feature processing of the feature image of the previous level. The new feature image obtains the feature image of the current level after reducing the image features through the transition block. The classification layer is used to predict the illumination probability according to the obtained feature image:

[0130] {P day ,P night}=SN(I vi ) (20)

[0131] Among them, the probability value P of the visible light image in the daytime day and the probability value P of being at night nightis the illumination probability, both are non-negative scalars, and SN(·) represents the illumination-aware subnetwork of DenseNet-201. The network is trained on a large-scale dataset to learn the illumination variation pattern in the image and attempt to perform illumination correction or illumination compensation to mitigate the impact of illumination variation on image processing tasks. Therefore, the illumination-aware weight of the image contribution is represented by the illumination probability calculated through the illumination allocation mechanism.

[0132] S012. Calculate the contribution value W of the infrared image in the image fusion process based on the probability value of the visible light image being in the daytime and the probability value of the visible light image being in the nighttime. ir And the contribution value W of the visible light image vi :

[0133]

[0134] The present invention creatively adopts the DenseNet-201 network architecture for illumination perception, wherein the illumination perception sub-network framework based on DenseNet-201 is as follows: Figure 2 As shown in (c), it mainly consists of 4 dense blocks, 3 transition blocks and 1 classification layer. In each dense block, each layer receives the feature maps of all previous layers as input and passes the generated feature maps to all subsequent layers. Dense blocks are key components in the DenseNet architecture. By allowing each layer to receive the feature maps of all previous layers as input and pass the generated feature maps to all subsequent layers, it enhances feature propagation and reuse, reduces the learning of redundant features, alleviates the gradient vanishing problem, and improves the parameter efficiency and feature diversity of the model. This approach makes deep network training more efficient and stable. This design helps solve the gradient vanishing problem in deep neural networks while also promoting feature reuse.

[0135] The input of the illumination perception subnetwork based on DenseNet-201 is the visible image, and the output is the probability of whether the visible image is daytime or nighttime. The transition block is located between two dense blocks and is mainly used to reduce the number of feature maps, thereby controlling the complexity of the model and preventing overfitting. The size of the convolutional layer kernel in each transition block is 1×1, and the stride in the 2×2 average pool is set to 2. The classification layer is the last part of the model and is responsible for converting the previously extracted features into the final prediction results. In this application, the classification layer needs to determine whether the input visible image is taken during the day or at night. The kernel size in the global average pool in the classification layer is 7×7, which is used to calculate the illumination probability based on the illumination information.

[0136] In addition, in order to verify the performance of the illumination-aware infrared and visible light image fusion method based on corner attention of the present invention, the MSRS dataset containing 1083 images is used to train the framework. The pixel size of these images is set to 64×64 as training samples. During the training process, 100 epochs are set and the batch size is set to 128. During the training process, the first 50 epochs maintain a fixed learning rate, while the next 50 epochs use exponential decay to adjust the learning rate. Initially, the learning rate is set to 10-3, the momentum is set to 0.9, and the weight decay is 10-4. It is implemented using PyTorch and is The algorithm is executed on 4090 GPU. In formula (16), λ1, λ2, λ3, λ4 and λ5 are set to 3, 7, 50, 1 and 3 respectively, which is based on preliminary parameter tuning experiments.

[0137] Figure 4 Table 1 and Table 2 systematically summarize and demonstrate the quantitative evaluation results of ten comparison algorithms on six common evaluation indicators. For each evaluation indicator, the average score was calculated based on 361 pairs of test images in the MSRS dataset, and the best result was marked in bold, while the second-best result was underlined to enhance the visual contrast effect. The analysis results show that the present invention has achieved the best level in all five evaluation indicators. This detailed data analysis not only demonstrates the excellent performance of the present invention in objective performance evaluation, but also provides strong evidence for the effectiveness and necessity of the design of the multimodal differential perception fusion module based on corner attention.

[0138] Table 1. Average index values ​​obtained by existing fusion methods and the present invention on the MSRS dataset

[0139]

[0140] This paper proposes an image fusion method with novel illumination perception, which combines source images through corner attention and further fuses texture information and thermal radiation information based on the perception of illumination conditions. An illumination perception subnetwork based on DenseNet-201 is introduced to enhance the probability estimation of training image pairs. A multimodal differential perception fusion module based on the corner attention module is proposed. The role of texture details, edges and structures in the visible image during the fusion process is emphasized.

[0141] Furthermore, the core purpose of the illumination perception subnetwork based on the DenseNet-201 architecture is to improve the probability estimation accuracy of image features under different illumination conditions by enhancing the ability to perceive illumination conditions in training images. Specifically, DenseNet-201, as a deep learning network, is known for its ability to effectively transmit deep feature information, which gives it a significant advantage in processing image data in complex scenes. The design of the illumination perception subnetwork fully considers the essential differences between infrared and visible light images, especially the impact of changes in illumination conditions on image quality. By incorporating the illumination perception mechanism into the framework of DenseNet-201, the subnetwork can more accurately capture and analyze the illumination information in the image, thereby providing more reliable input for the subsequent image fusion process. This improvement not only helps to improve the visual effect of the fused image, but also improves the effectiveness of the fused image for specific applications (such as target detection, environmental monitoring, etc.) to a certain extent.

[0142] Furthermore, the multimodal differential perception fusion module based on the corner attention module creatively combines the corner attention together, aiming to more effectively extract and fuse key information from infrared and visible light images. The corner attention mechanism optimizes the boundaries and corners in the image, strengthening the capture of edge and structural details, which is crucial to maintaining the clarity and definition of the image. Through this attention, more detailed attention can be paid to the performance of texture details, edges, and structures in the fusion process, especially when processing visible light images. This not only helps to retain the key information in the original image, but also enhances the visual effect of the fused image, making it show richer details and layering while maintaining high resolution.

[0143] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the claimed embodiments may be used in any combination.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fusion of infrared and visible light images based on illumination perception based on corner attention, characterized in that: The method comprises: Acquire the infrared image and visible light image to be fused; Input the infrared image and the visible light image into the corresponding feature extractors including multiple feature extraction levels, respectively, perform iterative feature extraction on the infrared image and the visible light image through the corresponding feature extractors, and calculate the differential image corner point features of the hierarchical features extracted by the preset feature extraction level in the feature extractor during the iterative process, obtain the differential infrared image corner point features and the differential visible light image corner point features of the corresponding feature extraction level, fuse the differential infrared image corner point features into the hierarchical infrared image features extracted by the corresponding feature extraction level, fuse the differential visible light image corner point features into the hierarchical visible light image features extracted by the corresponding feature extraction level, and input the fused features into the next feature extraction level of the current feature extraction level of the corresponding feature extractors for iteration, so as to obtain the infrared image features and the visible light image features; The visible light image feature and the infrared image feature are spliced ​​to obtain a spliced ​​image feature, and the spliced ​​image feature is input into a preset image reconstructor for image reconstruction to obtain a fused image.

2. The method according to claim 1, characterized in that The step of calculating the differential image corner point features of the hierarchical features extracted by the preset feature extraction level in the feature extractor to obtain the differential infrared image corner point features and the differential visible light image corner point features of the corresponding feature extraction level includes: Hierarchical infrared image features extracted based on target features and hierarchical visible light image features Compute intermediate infrared image features at the target feature extraction level and intermediate visible light image features The calculation is as follows: in, represents the element-wise sum operation, represents channel multiplication operation, P(·) represents pReLU function, and PAP(·) represents power average pooling; Calculate the infrared image corner weights W at the target feature extraction level respectively COA_ir and visible light image corner weight W COA_vi , calculated as follows: Where b is the batch size of the sample, c is the index of the channel, i and j are the row and column indices of the corner, rows and cols are the row and column positions of the corner, respectively. Indicates the corner point information of the intermediate infrared image features of the target feature extraction level, Indicates the corner point information of the intermediate visible light image features extracted at the target feature extraction level; Calculate the infrared image corner feature F at the target feature extraction level COA_ir and visible light image corner feature F COA_vi , calculated as follows: The intermediate infrared image features at the target feature extraction level are fused with the infrared image corner features to obtain the differential infrared image corner features at the target feature extraction level. The intermediate visible light image features at the target feature extraction level are fused with the visible light image corner features to obtain the differential visible light image corner features at the target feature extraction level. The calculation is as follows:

3. The method according to claim 1, characterized in that After obtaining the fused image, the method further includes: The overall loss L is calculated based on the following loss function formula: L=λ1·L illum +λ2·L aux +λ3·L texture +λ4·L vif +λ5·L Qabf Among them, L illum Represents the illumination perception loss, L aux Indicates auxiliary strength loss, L texture represents texture loss, L vif represents the loss of visual information fidelity, L Qabf represents the boundary fidelity quality loss, λ1, λ2, λ3, λ4 and λ5 represent the weights of the loss function respectively; Model parameters of the feature extractor and the image reconstructor are adjusted based on the overall loss L.

4. The method according to claim 3, characterized in that Before calculating the overall loss L, the method further includes: Inputting the visible light image into a preset illumination perception subnetwork to obtain a contribution value of the infrared image in a feature fusion process and a contribution value of the visible light image in a feature fusion process; Based on the contribution value W of the infrared image in the feature fusion process ir and the contribution value W of the visible light image in the feature fusion process vi Calculate the illumination perception loss: in, represents the intensity loss of the infrared image, Represents the intensity loss of the visible light image.

5. The method according to claim 4, characterized in that The illumination perception subnetwork is used to complete the following image operations: Based on the preset illumination perception backbone network, the probability value P of the visible light image being in daytime is obtained. day and the probability value P of being at night night ; Wherein, the illumination perception backbone network is a network model based on the DenseNet-201 architecture obtained by pre-training, including a plurality of cascaded dense blocks, a transition block connected between each dense block, and a classification layer connected after the last dense block. The dense blocks of each level receive the feature image of the previous level, and generate a new feature image after feature processing of the feature image of the previous level. The new feature image obtains the feature image of the current level after reducing the image features through the transition block. The classification layer is used to predict the illumination probability according to the obtained feature image; The contribution value W of the infrared image in the image fusion process is calculated based on the probability value of the visible light image being in the daytime and the probability value of being in the nighttime. ir And the contribution value W of the visible light image vi :

6. The method according to claim 3, characterized in that The auxiliary strength loss is expressed as: Among them, I ir Indicates infrared image, I vi represents the visible light image, I f represents the fused image, H and W represent the height and width respectively; The texture loss is expressed as: The visual information fidelity loss is expressed as: L vif =1-d f,ir -d f,vi Among them, subbands represents a set of specific subbands, and They represent the ideal information extracted from specific sub-bands of infrared image, visible light image and fused image respectively; Represents the infrared image and the visible light image C i The Sth element of i represents the random field of the i-th subband in the infrared image and the visible light image; and They are from infrared images ir i , visible light image vi i and the fused image f i The Sth element of They represent the output signals of the preset human visual system model with the ith sub-band of the infrared image and the visible light image as input, represents the output signal of the human visual system model with the ith subband of the fused image as input; R S,i represents the model parameters of the i-th subband; The boundary fidelity quality loss is expressed as: Here, n represents the number of source images.

7. The method according to claim 1, characterized in that The feature extractor consists of N cascaded convolutional layers, wherein the convolution kernel of the convolution layer in the first feature extraction level is 1×1 to minimize the differences in visual features and data distribution between images, and the convolution kernels of the convolution layers in the second to Nth feature extraction levels are 3×3 to extract deep features of infrared images and visible light images respectively; wherein the convolution layers in the second to N-1th feature extraction levels all introduce a multimodal differential perception fusion module based on corner attention, N≥3.

8. The method according to claim 7, characterized in that The image reconstructor is composed of N cascaded convolutional layers, wherein the convolution kernels of the 1st to N-1th convolutional layers are 3×3, and the convolution kernel of the Nth convolutional layer is 1×1.

Images