Infrared and visible light image fusion method and device based on primary and secondary coding interactive compensation

Abstract

The present invention discloses a method and device for fusion of infrared and visible light images based on interactive compensation of primary and secondary codes. The method comprises: extracting primary features from visible light images and infrared images to obtain primary visible light features and primary infrared features; extracting secondary features from the visible light images and infrared images to obtain secondary visible light features and secondary infrared features; performing cascade interaction on the primary visible light features and secondary visible light features to obtain target visible light features; performing cascade interaction on the primary infrared features and secondary infrared features to obtain target infrared features; generating an initial fused image based on the target visible light features and target infrared features; constructing a target loss function based on the visible light images and infrared images; and adjusting the initial fused image based on the target loss function to obtain a target fused image. The present invention can improve the quality of image fusion and can be widely applied in the field of image processing technology.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for fusing infrared and visible light images based on primary and secondary coding interactive compensation. Background Art

[0002] In recent years, with advances in sensor imaging devices and image processing technology, infrared and visible light image fusion technology has experienced explosive growth and is widely used in target tracking, person re-identification, remote sensing, and autonomous driving. Infrared images highlight thermal targets but lack clear scene texture, while visible light images offer high-resolution detail but are sensitive to weather conditions. Therefore, to address the differences in imaging and spectral response between these two modalities, deep learning-based infrared and visible light image fusion methods have emerged. These methods typically employ a primary encoder to extract features from the source image, but this inevitably results in the loss of important information during feature transmission, especially as the number of network layers increases. Current mainstream deep learning-based fusion methods employ a variety of loss functions to improve fusion performance, constraining the similarity between the source and fused images at multiple levels. However, the weights between these loss functions are often artificially set, which significantly limits the fusion performance. Furthermore, model parameter tuning complicates the method, compromising the quality of image fusion. Summary of the Invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a method and device for fusion of infrared and visible light images based on interactive compensation of primary and secondary codes, in order to solve at least one of the problems of the existing technology. The present invention can improve the quality of image fusion.

[0004] To achieve the above objectives, an embodiment of the present invention provides a method for fusion of infrared and visible light images based on interactive compensation of primary and secondary codes, the method comprising:

[0005] Collect visible light images and infrared light images;

[0006] Extracting main features of the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features;

[0007] performing auxiliary feature extraction on the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features;

[0008] Performing cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature;

[0009] Performing cascade interaction on the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain target infrared light features;

[0010] generating an initial fused image according to the target visible light characteristics and the target infrared light characteristics;

[0011] constructing a target loss function according to the visible light image and the infrared light image;

[0012] The initial fused image is adjusted according to the target loss function to obtain a target fused image.

[0013] In some embodiments, extracting main features from the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features includes the following steps:

[0014] Extracting features from the visible light image using a visible light main encoder to obtain multi-level visible light main features;

[0015] The infrared light image is subjected to feature extraction by an infrared light main encoder to obtain multi-level infrared light main features.

[0016] In some embodiments, extracting auxiliary features from the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features includes the following steps:

[0017] Extracting features from the visible light image using a visible light auxiliary encoder to obtain multi-level visible light auxiliary features;

[0018] The infrared light image is subjected to feature extraction by an infrared light-assisted encoder to obtain multi-level infrared light-assisted features.

[0019] In some embodiments, performing cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature includes the following steps:

[0020] The target visible light feature is obtained by performing a cascade interaction on the visible light main feature of the current layer, the visible light main features of the previous layers, and the visible light auxiliary features of the corresponding layers.

[0021] In some embodiments, the cascading interaction of the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain the target infrared light features includes the following steps:

[0022] The infrared light main feature of the current layer is cascaded and interacted with the infrared light main features of the previous layers and the infrared light auxiliary features of the corresponding layers to obtain the target infrared light feature.

[0023] In some embodiments, generating an initial fused image based on the target visible light characteristics and the target infrared light characteristics includes the following steps:

[0024] Performing enhanced fusion on the target visible light feature and the target infrared light feature to obtain a fusion feature;

[0025] The initial fused image is generated according to the fusion features.

[0026] In some embodiments, constructing a target loss function based on the visible light image and the infrared light image includes the following steps:

[0027] Obtaining an intensity weight coefficient of the infrared light image according to the visible light image and the infrared light image;

[0028] Acquiring first gradient information of the visible light image;

[0029] Acquiring second gradient information of the infrared light image;

[0030] Obtaining a gradient weight coefficient of the visible light image according to the first gradient information and the second gradient information;

[0031] The target loss function is constructed according to the intensity weight coefficient and the gradient weight coefficient.

[0032] In some embodiments, constructing the target loss function according to the intensity weight coefficient and the gradient weight coefficient comprises the following steps:

[0033] constructing an infrared intensity loss function according to the intensity weight coefficient, the visible light image, and the infrared light image;

[0034] Constructing a visible light loss function according to the gradient weight coefficient, the first gradient information, and the second gradient information;

[0035] The target loss function is obtained according to the infrared intensity loss function and the visible light loss function.

[0036] To achieve the above-mentioned purpose, another aspect of the present invention provides an infrared and visible light image fusion device based on primary and secondary coding interactive compensation, the device comprising:

[0037] The first module is used to collect visible light images and infrared light images;

[0038] A second module is configured to extract main features from the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features;

[0039] a third module, configured to extract auxiliary features from the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features;

[0040] a fourth module, configured to perform cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature;

[0041] A fifth module is configured to perform cascade interaction on the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain a target infrared light feature;

[0042] A sixth module is configured to generate an initial fused image based on the target visible light characteristics and the target infrared light characteristics;

[0043] A seventh module is configured to construct a target loss function based on the visible light image and the infrared light image;

[0044] An eighth module is configured to adjust the initial fused image according to the target loss function to obtain a target fused image.

[0045] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned infrared and visible light image fusion method based on interactive compensation of primary and auxiliary codes.

[0046] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the aforementioned infrared and visible light image fusion method based on interactive compensation of primary and auxiliary codes.

[0047] To achieve the above objectives, another aspect of an embodiment of the present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned infrared and visible light image fusion method based on interactive compensation of primary and secondary codes.

[0048] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a method and device for fusion of infrared and visible light images based on interactive compensation of primary and secondary codes, the scheme collects visible light images and infrared images; extracts main features of the visible light images and infrared images to obtain multi-level visible light main features and multi-level infrared main features; extracts auxiliary features of the visible light images and infrared images to obtain multi-level visible light auxiliary features and multi-level infrared auxiliary features; cascades the multi-level visible light main features and the multi-level visible light auxiliary features to obtain target visible light features; cascades the multi-level infrared main features and the multi-level infrared auxiliary features to obtain target infrared features; generates an initial fused image based on the target visible light features and the target infrared features; constructs a target loss function based on the visible light image and the infrared image; adjusts the initial fused image based on the target loss function to obtain a target fused image, which can improve the quality of image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 This is a flow chart of a method for fusion of infrared and visible light images based on interactive compensation of primary and secondary codes provided by an embodiment of the present invention;

[0051] Figure 2 Schematic diagram of an infrared and visible light image fusion network architecture based on primary and secondary coding interactive compensation provided by an embodiment of the present invention;

[0052] Figure 3 Schematic diagram of the workflow of an infrared and visible light image fusion network based on primary and secondary coding interactive compensation provided by an embodiment of the present invention;

[0053] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0055] It should be noted that although the functional modules are divided in the system schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100" and "second / S200" in the specification and claims and the above-mentioned figures may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to a determination".

[0056] The terms "at least one", "plurality", "each", "any", etc. used in the present invention include "at least one", "two" or "more than two", "plurality" or "more than two", "each" or "more than two", "each" or "more than two", and "any" or "more than one" in the present invention.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0058] Existing deep learning-based infrared and visible light image fusion methods typically use a main encoder to extract features from the source image during feature encoding. To expand the receptive field, a deeper network layer is usually required. However, as the number of network layers increases, some specific information of the source image is inevitably lost. Therefore, to improve the fusion performance of the model, relevant scholars have designed a large number of loss functions to constrain the model from multiple angles and multiple layers to obtain more comprehensive source image features. However, these loss functions are often correlated and artificially designed, which inevitably leads to the coupling of certain features and artificial limitations of the fusion model. At the same time, the more loss functions are designed, the more complex the balance of weight parameters becomes, resulting in low image fusion quality.

[0059] In view of this, if Figure 1 As shown, an embodiment of the present invention provides a method for fusion of infrared and visible light images based on interactive compensation of primary and secondary codes. The method may include but is not limited to steps S100 to S800:

[0060] Step S100, collecting visible light images and infrared light images;

[0061] Step S200, extracting main features from the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features;

[0062] Step S300, performing auxiliary feature extraction on the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features;

[0063] Step S400, performing cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature;

[0064] Step S500, performing cascade interaction on the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain a target infrared light feature;

[0065] Step S600, generating an initial fused image according to the target visible light characteristics and the target infrared light characteristics;

[0066] Step S700, constructing a target loss function according to the visible light image and the infrared light image;

[0067] Step S800: adjusting the initial fused image according to the target loss function to obtain a target fused image.

[0068] In some embodiments, infrared and visible light image fusion, based on interactive compensation between primary and secondary encoding, incorporates auxiliary features during the image feature encoding process. This can compensate for feature loss during the primary encoding process while enhancing the transfer of important features from the source image. A well-designed loss function can guide the fusion model to learn more accurate feature information.

[0069] In some embodiments, reference Figure 2 The infrared and visible light image fusion method based on the interaction compensation of primary and secondary codes is an end-to-end model. Its network architecture is as follows: Figure 2As shown in the figure, the network mainly consists of three components: an encoder, a feature integrator, and a decoder. The encoder includes: a visible light main encoder (Encoder_visp), a visible light auxiliary encoder (Encoder_visa), an infrared light main encoder (Encoder_irp), and an infrared light auxiliary encoder (Encoder_ira). The purpose of image fusion is to extract and fuse the most important information from multiple input source images. The most important information of infrared and visible light images can be defined as the intensity of infrared light and the detailed texture of visible light. For example, through the infrared and visible light image fusion network based on the interactive compensation of main and auxiliary encoding, the input infrared and visible light image pairs are feature extracted, and through the continuous learning of the encoder-feature integrator-decoder, a fused image that retains significant infrared light contrast and rich texture details is reconstructed.

[0070] In step S100 of some embodiments, in visible light image acquisition, visible light images generally use standard RGB sensors, which can capture red, green, and blue colors of light, so that visible light images can provide rich color information, which is crucial for color restoration and detail capture. Visible light images are suitable for scenes under daily lighting conditions, such as photography, monitoring, traffic monitoring, etc. In infrared light image acquisition, infrared images require the use of infrared sensors, which can sense radiation within the infrared spectrum and are generally used in night vision equipment, thermal imaging cameras, etc. Infrared images can provide temperature information of objects, which is very helpful for detecting thermal anomalies, electrical faults, etc. Infrared images are suitable for low-light or no-light environments, such as night monitoring, thermal imaging detection, medical imaging, etc.

[0071] In some embodiments, as Figure 2 As shown, the main encoder consists of a visible light main encoder (Encoder_visp) and an infrared light main encoder (Encoder_irp). Each main encoder contains four layers: a convolutional layer, a batch normalization (BN) layer, and a pre-resolved linear unit (PReLU) layer. The channel dimension of all three convolutional layers is set to 32, with a 3×3 kernel and a stride of 1. Padding is used during the convolution process to ensure consistent resolution of the convolutional features of the image and prevent information loss. All convolutional layers in the encoder are connected using a DenseNet structure to complement and reuse feature information.

[0072] In some embodiments, step S200 may include but is not limited to steps S210 to S220:

[0073] Step S210, extracting features from the visible light image using a visible light main encoder to obtain multi-level visible light main features;

[0074] Step S220 , performing feature extraction on the infrared light image through an infrared light main encoder to obtain multi-level infrared light main features.

[0075] In step S210 of some embodiments, the collected visible light image is read and input into the visible light main encoder, and features of the visible light image can be extracted to obtain multi-scale and multi-level visible light main features. Figure 2 As shown in the figure, the features of the visible light image input to the visible light main encoder are extracted to obtain the first layer of visible light main features, the second layer of visible light main features, the third layer of visible light main features, and the fourth layer of visible light main features. The feature extraction of each layer can be regarded as a coding of the visible light image information. As the network layer deepens, more abstract and advanced visible light main feature representations can be extracted, which is very beneficial for subsequent image analysis tasks.

[0076] In step S220 of some embodiments, the captured infrared image is read and input into the infrared main encoder, and features of the infrared image can be extracted to obtain multi-scale and multi-level infrared main features. Figure 2 As shown in the figure, the features of the infrared image input to the infrared main encoder are extracted to obtain the first, second, third, and fourth layers of infrared main features. The feature extraction at each layer can be regarded as a coding of the infrared image information. As the network layer deepens, more abstract and advanced infrared main feature representations can be extracted, which is very beneficial for subsequent image analysis tasks.

[0077] A suitable feature extraction encoder is a key element in image fusion. Its quality significantly impacts the subsequent image fusion quality. Existing deep learning methods typically increase network depth during the encoding process to extract multi-scale and deep features from the source image. While these methods expand the feature receptive field and extract deep features, the increased network size leads to a significant increase in computational complexity and can also cause vanishing or exploding gradients. In particular, this can cause information loss during feature transfer and lead to unexpected effects. Therefore, increasing network depth does not necessarily improve fusion quality.

[0078] In view of this, in some embodiments of the present invention, a pre-trained image auxiliary encoder is used to extract more complete source image features to assist and compensate for the information loss of the image main encoder in feature transmission. Figure 2As shown in the figure, the auxiliary encoder consists of a visible light auxiliary encoder (Encoder_visa) and an infrared light auxiliary encoder (Encoder_ira). By using the VGG-16 network as the auxiliary encoder, the advantages of the VGG feature extractor are integrated. It uses continuous small-scale convolution kernels (typically 3×3), and each convolution layer is followed by a nonlinear activation function (ReLU). Features are extracted through a large number of convolutional layers, which gives the VGG network a deeper structure and better feature extraction capabilities.

[0079] In some embodiments, step S300 may include but is not limited to steps S310 to S320:

[0080] Step S310, performing feature extraction on the visible light image by using a visible light auxiliary encoder to obtain multi-level visible light auxiliary features;

[0081] Step S320 , performing feature extraction on the infrared light image through an infrared light-assisted encoder to obtain multi-level infrared light-assisted features.

[0082] In step S310 of some embodiments, due to the multimodal diversity of visible light images, the visible light auxiliary encoder uses shallow features from the first, third, and fifth layers of a pre-trained VGG-16 network to extract rich texture information from the visible light image. This rich texture information is referred to as the visible light auxiliary features. The visible light auxiliary features extracted by the visible light auxiliary encoder can be used to assist the visible light main encoder, compensating for feature loss in the image main encoder while enhancing the output of important visible light features.

[0083] In step S320 of some embodiments, due to the multimodal diversity of infrared images, the infrared auxiliary encoder uses deep features from the second, fourth, and sixth layers of a pre-trained VGG-16 network to extract salient information from the infrared image after deep abstraction. This salient information is referred to as the infrared auxiliary features. Extracting the infrared auxiliary features through the infrared auxiliary encoder can assist the infrared main encoder, compensating for feature loss in the main image encoder while enhancing the output of important infrared features.

[0084] In some embodiments, in steps S400 to S500, in order to extract comprehensive features of the source image and prevent information loss during the deep feature transfer process, as shown in FIG. Figure 2 As shown in the figure, a feature transfer module based on interactive compensation between primary and auxiliary encoders is designed in the encoder. The primary feature extraction channel of each layer interacts with the primary feature channels of the previous layers and the auxiliary feature extraction channels of the current layer in a cascaded manner to achieve information compensation and interaction between the multi-scale deep features extracted by the primary and auxiliary encoders of the image.

[0085] In step S400 of some embodiments, the visible light main feature of the current layer is cascaded with the visible light main features of the previous layers and the visible light auxiliary features of the corresponding layers to obtain the target visible light feature. Figure 2 As shown, the second-layer visible light main features are cascaded with the first-layer visible light main features and the first-layer visible light auxiliary features. The first-layer visible light auxiliary features are extracted from the shallow features of the first layer of the VGG-16 network, which contains rich texture information of the visible light image. The third-layer visible light main features are cascaded with the first-layer visible light main features, the second-layer visible light main features, and the second-layer visible light auxiliary features. The second-layer visible light auxiliary features are extracted from the shallow features of the third layer of the VGG-16 network, which contains rich texture information of the visible light image. The fourth-layer visible light main features are cascaded with the first-layer visible light main features, the second-layer visible light main features, the third-layer visible light main features, and the third-layer visible light auxiliary features. The third-layer visible light auxiliary features are extracted from the shallow features of the fifth layer of the VGG-16 network, which contains rich texture information of the visible light image. The features obtained from these three cascade interactions are spliced ​​to obtain the target visible light features. This achieves information compensation and interaction between the multi-scale deep features extracted by the main and auxiliary image encoders.

[0086] In some embodiments, in step S500, the infrared light main feature of the current layer is cascaded with the infrared light main features of the previous layers and the infrared light auxiliary features of the corresponding layers to obtain the target infrared light feature. Figure 2 As shown, the second-layer infrared main features are cascaded with the first-layer infrared main features and the first-layer infrared auxiliary features. The first-layer infrared auxiliary features are derived from the deep features of the second layer of the VGG-16 network, representing the saliency information of the infrared image after deep abstraction. The third-layer infrared main features are cascaded with the first-layer infrared main features, the second-layer infrared main features, and the second-layer infrared auxiliary features. The second-layer infrared auxiliary features are derived from the deep features of the fourth layer of the VGG-16 network, representing the saliency information of the infrared image after deep abstraction. The fourth-layer infrared main features are cascaded with the first-layer infrared main features, the second-layer infrared main features, the third-layer infrared main features, and the third-layer infrared auxiliary features. The third-layer infrared auxiliary features are derived from the deep features of the sixth layer of the VGG-16 network, representing the saliency information of the infrared image after deep abstraction. The features obtained from these three cascade interactions are concatenated to obtain the target infrared features. This achieves information compensation and interaction between the multi-scale deep features extracted by the main and auxiliary image encoders.

[0087] In some embodiments, step S600 may include but is not limited to steps S610 to S620:

[0088] Step S610, performing enhanced fusion on the target visible light feature and the target infrared light feature to obtain a fused feature;

[0089] Step S620: generating the initial fused image according to the fusion features.

[0090] In step S610 of some embodiments, the target visible light feature and the target infrared light feature are enhanced and fused by the feature integrator to obtain a fused feature. The feature integrator can further enhance the deep features of the fused infrared and visible light main and auxiliary coding feature channels and improve the accuracy of feature expression. Figure 2 As shown in Figure 1, the feature integrator can include: convolution layer, BN layer and PReLU. The convolution kernel is set to 3×3, the channel dimension is 256, and the stride is 1.

[0091] In step S620 of some embodiments, the output of the feature integrator is passed to the decoder, that is, the fused features are input to the decoder to generate an initial fused image. Through the decoder, the multi-scale deep features extracted from the infrared image and the visible light image can be fused to generate an image with both infrared light intensity and visible light texture and rich information. Optionally, as Figure 2 As shown in the figure, the decoder contains a total of four convolutional blocks. The first convolutional block consists of a convolutional layer and a Prelude (PRelu). The second and third convolutional blocks consist of a convolutional layer, a batch normalization layer, and a Prelude (PRelu). The fourth convolutional block consists of a convolutional layer and a tanh activation function. The channel dimensions of the first to fourth convolutional blocks are set to 128, 64, 32, and 1, respectively. The convolution kernel is set to 3×3, with a stride of 1 and a padding of 1.

[0092] The loss function is another key factor affecting the performance of image fusion models. To improve the fusion performance of the model, relevant scholars have designed a large number of loss functions to constrain the model from multiple angles and layers to obtain more comprehensive source image features. However, this inevitably leads to the coupling of certain fusion features and artificial limitations of the fusion model.

[0093] In view of this, some embodiments of the present invention design an adaptive intensity and gradient loss function that can guide the fusion model to adaptively adjust the fusion weights to better obtain the most important information in the source image. In the task of infrared and visible light image fusion, embodiments of the present invention capture contrast and texture details, where infrared contrast information is represented by intensity and visible light texture details are characterized by gradient. The infrared and visible light fused image should have both significant contrast and rich texture information, that is, satisfy the requirements of significant infrared intensity information and large visible light gradient information.

[0094] In some embodiments, step S700 may include but is not limited to steps S710 to S750:

[0095] Step S710, obtaining an intensity weight coefficient of the infrared light image according to the visible light image and the infrared light image;

[0096] Step S720, obtaining first gradient information of the visible light image;

[0097] Step S730, obtaining second gradient information of the infrared light image;

[0098] Step S740: Obtaining a gradient weight coefficient of the visible light image according to the first gradient information and the second gradient information;

[0099] Step S750: construct the target loss function according to the intensity weight coefficient and the gradient weight coefficient.

[0100] In step S710 of some embodiments, the greater the difference between corresponding regions of the infrared and visible light images, the more important the specific information between them. Therefore, the importance of each modality's information is expressed by the difference distance between the infrared and visible light images, and intensity weights are assigned. The formula for calculating the intensity weight coefficient for the infrared image is as follows:

[0101]

[0102] Where μ represents the intensity weight coefficient of the infrared image; ir represents the infrared image; vis represents the visible light image; Rp(·) represents the repeated pooling operation; GMP(·) represents the maximum pooling operation.

[0103] In some embodiments, in steps S720 to S740, the greater the difference between corresponding regions of the infrared and visible light gradient images, the more important the specific information between the infrared and visible light images. Therefore, the importance of each modality's information is expressed based on the difference in gradient distance between the infrared and visible light images, and gradient weights are assigned. The formula for calculating the gradient weight coefficient for the visible light image is as follows:

[0104]

[0105] Where η represents the gradient weight coefficient of the visible light image; is the Sobel algorithm; Representing first gradient information of a visible light image; Represents the second gradient information of the infrared image. The Sobel algorithm can better express the gradient information of the source image.

[0106] In some embodiments, step S750 may include but is not limited to steps S751 to S753:

[0107] Step S751, constructing an infrared intensity loss function according to the intensity weight coefficient, the visible light image, and the infrared light image;

[0108] Step S752: constructing a visible light loss function according to the gradient weight coefficient, the first gradient information, and the second gradient information;

[0109] Step S753: Obtain the target loss function according to the infrared intensity loss function and the visible light loss function.

[0110] In step S751 of some embodiments, an infrared intensity loss function may be constructed based on the intensity weight coefficient μ, the visible light image vis, and the infrared light image ir. Then we have the following expression:

[0111]

[0112] Where, represents the infrared intensity loss function; max(·) represents the maximum value operation; ‖·‖1 represents the first normal form operation.

[0113] In step S752 of some embodiments, according to the gradient weight coefficient η, the first gradient information And the second gradient information Visible light loss function can be constructed Then we have the following expression:

[0114]

[0115] Where, Represents the visible light loss function.

[0116] In step S753 of some embodiments, a target loss function may be constructed based on the infrared intensity loss function and the visible light loss function. The target loss function is expressed as follows:

[0117]

[0118] Where, L total represents the target loss function; α is the balance coefficient.

[0119] The infrared intensity and visible light texture details retained in the fused image are adaptively adjusted using μ and η. When μ or η is positive, the intensity or gradient information of the source image needs to be strengthened; when μ or η is negative, the intensity or gradient information of the source image needs to be weakened; when μ or η is zero, the intensity or gradient information of the image remains unchanged.

[0120] In step S800 of some embodiments, the initial fused image is adjusted according to the target loss function of adaptive strength and gradient, which can guide the fusion model to adaptively adjust the fusion weights, so as to better obtain the most important information in the source image and obtain the target fused image.

[0121] For example, Figure 3 As shown, the steps of an infrared and visible light image fusion method based on primary and secondary coding interactive compensation are as follows:

[0122] Step 1: Read the acquired infrared image and visible light image.

[0123] Step 2: Input the infrared image and the visible light image into the infrared and visible light image fusion model with primary and secondary coding interactive compensation, wherein the infrared and visible light image fusion model with primary and secondary coding interactive compensation includes a primary and secondary interactive encoder, a feature integrator and a decoder.

[0124] Step 3: Use the main and auxiliary interactive encoders to extract the main features and auxiliary features of the visible light image and the infrared light image. In the feature extraction channel, the main feature extraction channel of each layer performs cascade interaction operations with the main feature channels of the previous layers and the auxiliary feature extraction channels of the current layer to obtain the target visible light features and target infrared light features.

[0125] Step 4: Input the target visible light features and the target infrared light features into the feature integrator, and perform enhanced fusion of the target visible light features and the target infrared light features to obtain the fusion features;

[0126] Step 5: Input the fused features into the decoder to generate an initial fused image with rich information, which has both infrared light intensity and visible light texture.

[0127] Step 6: Adjust the initial fused image according to the target loss function to obtain the target fused image. The target loss function is an adaptive intensity and gradient loss function designed based on visible light images and infrared light images.

[0128] An embodiment of the present invention further provides an infrared and visible light image fusion device based on interactive compensation of primary and secondary codes, which can implement the above-mentioned infrared and visible light image fusion method based on interactive compensation of primary and secondary codes. The device includes:

[0129] The first module is used to collect visible light images and infrared light images;

[0130] A second module is configured to extract main features from the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features;

[0131] a third module, configured to extract auxiliary features from the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features;

[0132] a fourth module, configured to perform cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature;

[0133] A fifth module is configured to perform cascade interaction on the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain a target infrared light feature;

[0134] A sixth module is configured to generate an initial fused image based on the target visible light characteristics and the target infrared light characteristics;

[0135] A seventh module is configured to construct a target loss function based on the visible light image and the infrared light image;

[0136] An eighth module is configured to adjust the initial fused image according to the target loss function to obtain a target fused image.

[0137] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] An embodiment of the present invention further provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned infrared and visible light image fusion method based on interactive compensation of primary and secondary codes. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0139] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] refer to Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0141] The processor 901 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0142] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the infrared and visible light image fusion method based on primary and secondary code interactive compensation according to the embodiment of the present invention.

[0143] Input / output interface 903, used to implement information input and output;

[0144] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0145] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0146] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0147] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned infrared and visible light image fusion method based on interactive compensation of primary and secondary codes.

[0148] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0149] An embodiment of the present invention further provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned infrared and visible light image fusion method based on interactive compensation of primary and secondary codes.

[0150] In summary, the infrared and visible light image fusion method and apparatus based on primary and secondary code interactive compensation according to the embodiments of the present invention have the following advantages:

[0151] 1. Current deep learning-based infrared and visible light image fusion methods typically use a single encoder to extract features during feature encoding. During feature transfer, increasing network depth inevitably leads to feature loss. This embodiment of the present invention introduces a pre-trained image reconstruction auxiliary encoder at the feature encoding end to assist the image fusion main encoder. This allows for better information transmission of important features and prevents feature loss, thereby improving the quality of image fusion.

[0152] 2. To improve the fusion performance of models, many researchers have designed a large number of loss functions to constrain the fused image from multiple perspectives. However, there is often a certain degree of correlation between the loss functions, and the weight relationships are usually artificially set, which inevitably greatly limits the fusion performance of the model. An adaptive intensity and gradient loss function designed in this embodiment can guide the fusion model to adaptively adjust the fusion weights, thereby better obtaining the most important information in the source image.

[0153] 3. By improving the performance of infrared and visible light image fusion, the embodiments of the present invention have great research value for subsequent advanced computer vision tasks, such as visual tracking, video surveillance, target detection, pedestrian re-identification, face recognition, and autonomous driving. Infrared and visible light image fusion plays an important role in sensitive target tracking and identification, perimeter protection, accident identification and evidence collection, and can serve security management, prevention and control, and other businesses. In addition, this technology can also be applied to multi-functional telescopes, night vision goggles, and other equipment, and can be installed in related equipment to achieve accurate detection of hidden targets in harsh environments. It has significant significance in security monitoring and object detection.

[0154] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0155] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0156] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0157] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0158] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0159] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0160] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0161] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0162] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for fusion of infrared and visible light images based on interactive compensation of primary and secondary codes, characterized in that: The following steps are involved: Collect visible light images and infrared light images; Extracting main features of the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features; performing auxiliary feature extraction on the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features; Performing cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature; Performing cascade interaction on the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain target infrared light features; generating an initial fused image according to the target visible light characteristics and the target infrared light characteristics; constructing a target loss function according to the visible light image and the infrared light image; The initial fused image is adjusted according to the target loss function to obtain a target fused image.

2. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 1 is characterized in that: The main feature extraction of the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features includes the following steps: Extracting features from the visible light image using a visible light main encoder to obtain multi-level visible light main features; The infrared light image is subjected to feature extraction by an infrared light main encoder to obtain multi-level infrared light main features.

3. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 1 is characterized in that: The auxiliary feature extraction is performed on the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features, including the following steps: Extracting features from the visible light image using a visible light auxiliary encoder to obtain multi-level visible light auxiliary features; The infrared light image is subjected to feature extraction by an infrared light-assisted encoder to obtain multi-level infrared light-assisted features.

4. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 1 is characterized in that: The step of performing cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature comprises the following steps: The target visible light feature is obtained by performing a cascade interaction on the visible light main feature of the current layer, the visible light main features of the previous layers, and the visible light auxiliary features of the corresponding layers.

5. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 1 is characterized in that: The cascading interaction of the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain the target infrared light features includes the following steps: The infrared light main feature of the current layer is cascaded and interacted with the infrared light main features of the previous layers and the infrared light auxiliary features of the corresponding layers to obtain the target infrared light feature.

6. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 1 is characterized in that: Generating an initial fused image according to the target visible light feature and the target infrared light feature comprises the following steps: Performing enhanced fusion on the target visible light feature and the target infrared light feature to obtain a fusion feature; The initial fused image is generated according to the fusion features.

7. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 1 is characterized in that: Constructing a target loss function according to the visible light image and the infrared light image includes the following steps: Obtaining an intensity weight coefficient of the infrared light image according to the visible light image and the infrared light image; Acquiring first gradient information of the visible light image; Acquiring second gradient information of the infrared light image; Obtaining a gradient weight coefficient of the visible light image according to the first gradient information and the second gradient information; The target loss function is constructed according to the intensity weight coefficient and the gradient weight coefficient.

8. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 7 is characterized in that: The target loss function is constructed according to the intensity weight coefficient and the gradient weight coefficient, comprising the following steps: constructing an infrared intensity loss function according to the intensity weight coefficient, the visible light image, and the infrared light image; constructing a visible light loss function according to the gradient weight coefficient, the first gradient information, and the second gradient information; The target loss function is obtained according to the infrared intensity loss function and the visible light loss function.

9. An infrared and visible light image fusion device based on primary and secondary coding interactive compensation, characterized in that: include: The first module is used to collect visible light images and infrared light images; A second module is configured to extract main features from the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features; a third module, configured to extract auxiliary features from the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features; a fourth module, configured to perform cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature; A fifth module is configured to perform cascade interaction on the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain a target infrared light feature; A sixth module is configured to generate an initial fused image based on the target visible light characteristics and the target infrared light characteristics; A seventh module is configured to construct a target loss function based on the visible light image and the infrared light image; An eighth module is configured to adjust the initial fused image according to the target loss function to obtain a target fused image.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 8.

Images