Infrared and visible light image fusion method and device based on main and auxiliary coding interaction compensation

Abstract

The invention discloses an infrared and visible light image fusion method and device based on main and auxiliary coding interaction compensation, and the method comprises the steps: carrying out the main feature extraction of a visible light image and an infrared light image, and obtaining a visible light main feature and an infrared light main feature; performing auxiliary feature extraction on the visible light image and the infrared light image to obtain visible light auxiliary features and infrared light auxiliary features; performing cascade interaction on the visible light main feature and the visible light auxiliary feature to obtain a target visible light feature; performing cascade interaction on the infrared light main feature and the infrared light auxiliary feature to obtain a target infrared light feature; generating an initial fusion image according to the target visible light feature and the target infrared light feature; constructing a target loss function according to the visible light image and the infrared light image; and adjusting the initial fusion image according to the target loss function to obtain a target fusion image. The image fusion quality can be improved, and the method can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and a device for fusion of infrared and visible light images based on primary and secondary coding interactive compensation. Background Art

[0002] In recent years, with the advancement of sensor imaging equipment and image processing technology, infrared and visible light image fusion technology has also ushered in a blowout development and has been widely used in target tracking, pedestrian re-identification, remote sensing detection and autonomous driving. Infrared images have prominent thermal targets but lack clear scene textures, while visible light images have high-resolution detailed scenes but are more sensitive to weather conditions. Therefore, in the face of the imaging and spectral response differences between these two modal images, there are infrared and visible light image fusion methods based on deep learning. Usually, a main encoder is used to extract features from the source image, but some important information will inevitably be lost during the feature transmission process, especially as the number of network layers increases. In order to improve the fusion performance of the model, the current mainstream deep learning-based fusion method designs a large number of loss functions to constrain the similarity between the source image and the fused image from multiple levels, but the weight relationship between the loss functions is usually set manually, and this artificiality will inevitably greatly limit the fusion performance of the model and the model parameter adjustment also increases the complexity of the method, thereby affecting the quality of image fusion. Summary of the invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide an infrared and visible light image fusion method and device based on primary and auxiliary coding interactive compensation, in order to solve at least one of the problems of the prior art. The present invention can improve the quality of image fusion.

[0004] To achieve the above object, an embodiment of the present invention provides an infrared and visible light image fusion method based on primary and secondary coding interactive compensation, the method comprising:

[0005] Collect visible light images and infrared light images;

[0006] Extracting main features of the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features;

[0007] Extracting auxiliary features from the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features;

[0008] Performing cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature;

[0009] Performing cascade interaction on the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain target infrared light features;

[0010] generating an initial fused image according to the target visible light feature and the target infrared light feature;

[0011] Constructing a target loss function according to the visible light image and the infrared light image;

[0012] According to the target loss function, the initial fused image is adjusted to obtain a target fused image.

[0013] In some embodiments, extracting main features from the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features includes the following steps:

[0014] Extracting features of the visible light image by using a visible light main encoder to obtain multi-level visible light main features;

[0015] The infrared light image is subjected to feature extraction through an infrared light main encoder to obtain multi-level infrared light main features.

[0016] In some embodiments, the extracting auxiliary features from the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features includes the following steps:

[0017] Extracting features from the visible light image using a visible light auxiliary encoder to obtain multi-level visible light auxiliary features;

[0018] The infrared light image is subjected to feature extraction through an infrared light-assisted encoder to obtain multi-level infrared light-assisted features.

[0019] In some embodiments, the cascading interaction of the multi-level visible light main features and the multi-level visible light auxiliary features to obtain the target visible light features includes the following steps:

[0020] The visible light main feature of the current layer is cascaded and interacted with the visible light main features of the previous layers and the visible light auxiliary features of the corresponding layers to obtain the target visible light feature.

[0021] In some embodiments, the cascading interaction of the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain the target infrared light features includes the following steps:

[0022] The infrared light main feature of the current layer is cascaded and interacted with the infrared light main features of the previous layers and the infrared light auxiliary features of the corresponding layers to obtain the target infrared light feature.

[0023] In some embodiments, generating an initial fused image according to the target visible light feature and the target infrared light feature comprises the following steps:

[0024] Performing enhanced fusion on the target visible light feature and the target infrared light feature to obtain a fusion feature;

[0025] The initial fused image is generated according to the fused features.

[0026] In some embodiments, constructing a target loss function according to the visible light image and the infrared light image comprises the following steps:

[0027] Acquire an intensity weight coefficient of the infrared light image according to the visible light image and the infrared light image;

[0028] Acquiring first gradient information of the visible light image;

[0029] Acquiring second gradient information of the infrared light image;

[0030] Acquire a gradient weight coefficient of the visible light image according to the first gradient information and the second gradient information;

[0031] The target loss function is constructed according to the intensity weight coefficient and the gradient weight coefficient.

[0032] In some embodiments, constructing the target loss function according to the intensity weight coefficient and the gradient weight coefficient comprises the following steps:

[0033] constructing an infrared intensity loss function according to the intensity weight coefficient, the visible light image and the infrared light image;

[0034] Constructing a visible light loss function according to the gradient weight coefficient, the first gradient information, and the second gradient information;

[0035] The target loss function is obtained according to the infrared intensity loss function and the visible light loss function.

[0036] To achieve the above object, another aspect of an embodiment of the present invention provides an infrared and visible light image fusion device based on primary and secondary coding interactive compensation, the device comprising:

[0037] The first module is used to collect visible light images and infrared light images;

[0038] The second module is used to extract main features of the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features;

[0039] A third module is used to extract auxiliary features from the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features;

[0040] A fourth module is used to perform cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature;

[0041] A fifth module is used to perform cascade interaction on the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain target infrared light features;

[0042] A sixth module is used to generate an initial fused image according to the target visible light feature and the target infrared light feature;

[0043] A seventh module is used to construct a target loss function according to the visible light image and the infrared light image;

[0044] The eighth module is used to adjust the initial fused image according to the target loss function to obtain a target fused image.

[0045] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned infrared and visible light image fusion method based on interactive compensation of primary and auxiliary codes.

[0046] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned infrared and visible light image fusion method based on primary and auxiliary coding interactive compensation.

[0047] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the aforementioned infrared and visible light image fusion method based on primary and secondary coding interactive compensation.

[0048] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a method and device for fusion of infrared and visible light images based on interactive compensation of primary and secondary codes, the scheme collects visible light images and infrared light images; extracts main features of the visible light images and infrared light images to obtain multi-level visible light main features and multi-level infrared light main features; extracts auxiliary features of the visible light images and infrared light images to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features; cascades the multi-level visible light main features with the multi-level visible light auxiliary features to obtain target visible light features; cascades the multi-level infrared light main features with the multi-level infrared light auxiliary features to obtain target infrared light features; generates an initial fused image based on the target visible light features and the target infrared light features; constructs a target loss function based on the visible light image and the infrared light image; adjusts the initial fused image based on the target loss function to obtain a target fused image, which can improve the quality of image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 It is a flow chart of a method for fusion of infrared and visible light images based on interactive compensation of primary and secondary codes provided by an embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of an infrared and visible light image fusion network architecture based on primary and secondary coding interactive compensation provided by an embodiment of the present invention;

[0052] Figure 3 It is a schematic diagram of the workflow of an infrared and visible light image fusion network based on primary and secondary coding interactive compensation provided by an embodiment of the present invention;

[0053] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention, and they are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0055] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the flow chart. The terms "first / S100" and "second / S200" in the specification and claims and the above-mentioned drawings may be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiment of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0056] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0058] Existing deep learning-based infrared and visible light image fusion methods usually use a main encoder to extract features from the source image during feature encoding. In order to expand the receptive field, it is usually necessary to set a deeper network layer. However, as the number of network layers increases, some specific information of the source image will inevitably be lost. Therefore, in order to improve the fusion performance of the model, relevant scholars have designed a large number of loss functions to constrain the model from multiple angles and multiple layers to obtain more comprehensive source image features. However, these loss functions are usually correlated and these loss functions are usually artificially designed, which will inevitably cause the coupling of obtaining certain features and the artificial limitations of the fusion model. At the same time, the more loss functions are designed, the more complex the balance of weight parameters is, resulting in low quality of image fusion.

[0059] In view of this, if Figure 1 As shown, an embodiment of the present invention provides an infrared and visible light image fusion method based on primary and secondary coding interactive compensation, which may include but is not limited to steps S100 to S800:

[0060] Step S100, collecting visible light images and infrared light images;

[0061] Step S200, extracting main features from the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features;

[0062] Step S300, performing auxiliary feature extraction on the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features;

[0063] Step S400, performing cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature;

[0064] Step S500, cascading and interacting the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain a target infrared light feature;

[0065] Step S600, generating an initial fused image according to the target visible light feature and the target infrared light feature;

[0066] Step S700, constructing a target loss function according to the visible light image and the infrared light image;

[0067] Step S800: adjusting the initial fused image according to the target loss function to obtain a target fused image.

[0068] In some embodiments, infrared and visible light image fusion based on interactive compensation of primary and secondary coding combines auxiliary features in the image feature coding process, which can make up for the feature loss of the image in the primary coding process and enhance the information transmission of important features of the source image. By designing a well-designed loss function, the fusion model can be guided to learn more accurate feature information.

[0069] In some embodiments, reference Figure 2 The infrared and visible light image fusion method based on the interaction compensation of primary and secondary codes is an end-to-end model. Its network architecture is as follows Figure 2As shown in the figure, the network mainly consists of three components: an encoder, a feature integrator and a decoder. The encoder includes: a visible light main encoder (Encoder_visp), a visible light auxiliary encoder (Encoder_visa), an infrared light main encoder (Encoder_irp) and an infrared light auxiliary encoder (Encoder_ira). The purpose of image fusion is to extract and fuse the most important information of multiple input source images. The most important information of infrared and visible light images can be defined as the intensity of infrared light and the detailed texture of visible light. Exemplarily, through a network of infrared and visible light image fusion based on the interaction compensation of main and auxiliary coding, the input infrared and visible light image pairs are feature extracted, and through the continuous learning of encoder-feature integrator-decoder, a fused image that retains significant infrared light contrast and rich texture details is reconstructed.

[0070] In step S100 of some embodiments, in visible light image acquisition, visible light images usually use standard RGB sensors, which can capture red, green and blue colors of light, so that visible light images can provide rich color information, which is crucial for color restoration and detail capture. Visible light images are suitable for scenes under daily lighting conditions, such as photography, monitoring, traffic monitoring, etc. In infrared light image acquisition, infrared images require the use of infrared sensors, which can sense radiation within the infrared spectrum and are usually used in night vision equipment, thermal imaging cameras, etc. Infrared images can provide temperature information of objects, which is very helpful for detecting thermal anomalies, electrical faults, etc. Infrared images are suitable for low-light or no-light environments, such as night monitoring, thermal imaging detection, medical imaging, etc.

[0071] In some embodiments, Figure 2 As shown in the figure, the main encoder consists of a visible light main encoder (Encoder_visp) and an infrared light main encoder (Encoder_irp). Each main encoder contains four layers: convolution layer, Batch Normalization (BN) layer, and PRelu layer. The channel dimensions of the three convolution layers are all set to 32, the convolution kernel is 3×3, and the stride is 1. Padding operation is used in the convolution process to keep the convolution feature resolution of the image consistent before and after to avoid information loss. All convolution layers in the encoder are connected using a Densenet structure to achieve the supplement and reuse of feature information.

[0072] In some embodiments, step S200 may include but is not limited to steps S210 to S220:

[0073] Step S210, extracting features from the visible light image through a visible light main encoder to obtain multi-level visible light main features;

[0074] Step S220, extracting features of the infrared light image through an infrared light main encoder to obtain multi-level infrared light main features.

[0075] In step S210 of some embodiments, the collected visible light image is read and input into the visible light main encoder, and features of the visible light image can be extracted to obtain multi-scale and multi-level visible light main features. Figure 2 As shown in the figure, the features of the visible light image input into the visible light main encoder are extracted to obtain the first layer of visible light main features, the second layer of visible light main features, the third layer of visible light main features and the fourth layer of visible light main features. The feature extraction of each layer can be regarded as an encoding of the visible light image information. As the network layer deepens, more abstract and advanced visible light main feature representations can be extracted, which is very beneficial for subsequent image analysis tasks.

[0076] In step S220 of some embodiments, the collected infrared light image is read and input into the infrared light main encoder, and features of the infrared light image can be extracted to obtain multi-scale and multi-level infrared light main features. Figure 2 As shown in the figure, the features of the infrared image input into the infrared main encoder are extracted to obtain the first layer of infrared main features, the second layer of infrared main features, the third layer of infrared main features and the fourth layer of infrared main features. The feature extraction of each layer can be regarded as an encoding of the infrared image information. As the network level deepens, more abstract and advanced infrared main feature representations can be extracted, which is very beneficial for subsequent image analysis tasks.

[0077] A reasonable feature extraction encoder is a key factor in image fusion. Its quality has a huge impact on the subsequent image fusion quality. In the existing deep learning methods, in the encoding process, in order to extract the multi-scale and deep features of the source image, the network depth is usually increased. These methods expand the feature receptive field and extract deep features, but the increase in the network will bring a sudden increase in the amount of calculation, and will also bring about the problem of gradient disappearance or explosion. In particular, in the process of feature transfer, information loss will occur and unknown effects will appear. It can be seen that the increase in network depth does not necessarily lead to an improvement in fusion quality.

[0078] In view of this, in some embodiments of the present invention, a pre-trained image auxiliary encoder is used to extract more complete source image features to assist and compensate for the information loss of the image main encoder in feature transmission. Figure 2As shown in the figure, the auxiliary encoder consists of a visible light auxiliary encoder (Encoder_visa) and an infrared light auxiliary encoder (Encoder_ira). By using the VGG-16 network as an auxiliary encoder, the advantages of the VGG feature extractor are integrated, which uses continuous small-scale convolution kernels (usually 3×3), and each layer of convolution is followed by a nonlinear activation function (ReLU), and features are extracted through a large number of convolution layers, which makes the VGG network have a deeper structure and better feature extraction capabilities.

[0079] In some embodiments, step S300 may include but is not limited to steps S310 to S320:

[0080] Step S310, extracting features from the visible light image using a visible light auxiliary encoder to obtain multi-level visible light auxiliary features;

[0081] Step S320: extracting features from the infrared light image using an infrared light-assisted encoder to obtain multi-level infrared light-assisted features.

[0082] In step S310 of some embodiments, due to the multimodal differences of visible light images, for the visible light auxiliary encoder, the shallow features of the first, third and fifth layers of the pre-trained VGG-16 network are used to extract rich texture information of the visible light image, and the rich texture information is the visible light auxiliary feature. The visible light auxiliary feature extracted by the visible light auxiliary encoder can be used to assist the visible light main encoder, and while compensating for the loss of the image main encoder features, the information output of important visible light features is enhanced.

[0083] In step S320 of some embodiments, due to the multimodal differences of infrared images, for the infrared auxiliary encoder, the deep features of the second, fourth and sixth layers of the pre-trained VGG-16 network are used to extract the significant information of the infrared image after deep abstraction, and the significant information is the infrared auxiliary feature. The infrared auxiliary feature extracted by the infrared auxiliary encoder can be used to assist the infrared main encoder, and while compensating for the loss of the image main encoder features, the information output of important infrared features is enhanced.

[0084] In some embodiments, in step S400 to step S500, in order to extract comprehensive features of the source image and prevent information loss during the deep feature transfer process, as shown in FIG. Figure 2 As shown in the figure, a feature transfer module based on the interaction compensation of main and auxiliary encoders is designed in the encoder. The main feature extraction channel of each layer is cascaded and interacted with the main feature channels of the previous layers and the auxiliary feature extraction channels of this layer to achieve information compensation and interaction of multi-scale deep features extracted by the main and auxiliary encoders of the image.

[0085] In step S400 of some embodiments, the visible light main feature of the current layer is cascaded with the visible light main features of the previous layers and the visible light auxiliary features of the corresponding layers to obtain the target visible light feature. Figure 2 As shown, the second layer of visible light main features are cascaded with the first layer of visible light main features and the first layer of visible light auxiliary features. The first layer of visible light auxiliary features are the rich texture information of the visible light image extracted by the shallow features of the first layer of the VGG-16 network. The third layer of visible light main features are cascaded with the first layer of visible light main features, the second layer of visible light main features and the second layer of visible light auxiliary features. The second layer of visible light auxiliary features are the rich texture information of the visible light image extracted by the shallow features of the third layer of the VGG-16 network. The fourth layer of visible light main features are cascaded with the first layer of visible light main features, the second layer of visible light main features, the third layer of visible light main features and the third layer of visible light auxiliary features. The third layer of visible light auxiliary features are the rich texture information of the visible light image extracted by the shallow features of the fifth layer of the VGG-16 network. The features obtained by the three cascade interactions are spliced ​​to obtain the target visible light features. The information compensation and interaction of the multi-scale deep features extracted by the main and auxiliary encoders of the image are realized.

[0086] In some embodiments, in step S500, the infrared light main feature of the current layer is cascaded with the infrared light main features of the previous layers and the infrared light auxiliary features of the corresponding layers to obtain the target infrared light feature. Figure 2 As shown, the second layer of infrared main features are cascaded with the first layer of infrared main features and the first layer of infrared auxiliary features. The first layer of infrared auxiliary features are extracted through the deep features of the second layer of the VGG-16 network to obtain the saliency information of the infrared image after deep abstraction. The third layer of infrared main features are cascaded with the first layer of infrared main features, the second layer of infrared main features and the second layer of infrared auxiliary features. The second layer of infrared auxiliary features are extracted through the deep features of the fourth layer of the VGG-16 network to obtain the saliency information of the infrared image after deep abstraction. The fourth layer of infrared main features are cascaded with the first layer of infrared main features, the second layer of infrared main features, the third layer of infrared main features and the third layer of infrared auxiliary features. The third layer of infrared auxiliary features are extracted through the deep features of the sixth layer of the VGG-16 network to obtain the saliency information of the infrared image after deep abstraction. The features obtained by the three cascade interactions are spliced ​​to obtain the target infrared features. The information compensation and interaction of the multi-scale deep features extracted by the main and auxiliary encoders of the image are realized.

[0087] In some embodiments, step S600 may include but is not limited to steps S610 to S620:

[0088] Step S610, performing enhanced fusion on the target visible light feature and the target infrared light feature to obtain a fusion feature;

[0089] Step S620: generating the initial fused image according to the fusion feature.

[0090] In step S610 of some embodiments, the target visible light feature and the target infrared light feature are enhanced and fused by a feature integrator to obtain a fused feature. The feature integrator can further enhance the deep features of the fused infrared and visible light main and auxiliary coding feature channels and improve the accuracy of feature expression. Figure 2 As shown, the feature integrator may include: convolution layer, BN layer and PRelu. The convolution kernel is set to 3×3, the channel dimension is 256, and the stride is 1.

[0091] In step S620 of some embodiments, the output of the feature integrator is passed to the decoder, that is, the fused features are input to the decoder to generate an initial fused image. Through the decoder, the multi-scale deep features extracted from the infrared image and the visible light image can be fused to generate an image with both infrared light intensity and visible light texture and rich information. Optionally, as Figure 2 As shown in the figure, the decoder contains 4 convolution blocks. The first convolution block consists of: convolution layer and PRelu. The second and third convolution blocks consist of: convolution layer, BN layer and PRelu. The fourth convolution block consists of: convolution layer and tanh activation function. The channel dimensions of the first to fourth convolution blocks are set to: 128, 64, 32 and 1 respectively. The convolution kernel is set to 3×3, the stride is 1, and the padding is set to 1.

[0092] The loss function is another key factor that affects the performance of the image fusion model. In order to improve the fusion performance of the model, relevant scholars have designed a large number of loss functions to constrain the model from multiple angles and multiple layers to obtain more comprehensive source image features, which will inevitably cause the coupling of certain fusion features and artificial limitations of the fusion model.

[0093] In view of this, in some embodiments of the present invention, an adaptive intensity and gradient loss function is designed to guide the fusion model to adaptively adjust the fusion weights to better obtain the most important information in the source image. In the infrared and visible light image fusion task, the embodiment of the present invention captures the contrast and texture details, where the infrared contrast information is represented by intensity and the visible light texture details are characterized by gradient. The infrared and visible light fused image should have both significant contrast and rich texture information, that is, satisfy the significant infrared intensity information and large visible light gradient information.

[0094] In some embodiments, step S700 may include but is not limited to steps S710 to S750:

[0095] Step S710, acquiring an intensity weight coefficient of the infrared light image according to the visible light image and the infrared light image;

[0096] Step S720, obtaining first gradient information of the visible light image;

[0097] Step S730, obtaining second gradient information of the infrared light image;

[0098] Step S740, acquiring a gradient weight coefficient of the visible light image according to the first gradient information and the second gradient information;

[0099] Step S750, constructing the target loss function according to the intensity weight coefficient and the gradient weight coefficient.

[0100] In step S710 of some embodiments, the greater the difference between the corresponding areas of the infrared and visible light images, the more important the specific information between the infrared and visible light images. Therefore, the different importance of each modality information is expressed by the difference distance between each infrared and visible light image, and the intensity weight is assigned. The calculation formula of the intensity weight coefficient of the infrared light image is as follows:

[0101]

[0102] Where μ represents the intensity weight coefficient of the infrared image; ir represents the infrared image; vis represents the visible light image; Rp(·) represents the repeated pooling operation; GMP(·) represents the maximum pooling operation.

[0103] In steps S720 to S740 of some embodiments, the greater the difference between the corresponding areas of the infrared and visible light gradient images, the more important the specific information between the infrared and visible light images. Therefore, the different importance of each modality information is expressed according to the difference distance between the gradients of each infrared and visible light image, and the gradient weight is assigned. The calculation formula of the gradient weight coefficient of the visible light image is as follows:

[0104]

[0105] Where η represents the gradient weight coefficient of the visible light image; It is the Sobel algorithm; Representing first gradient information of a visible light image; Represents the second gradient information of the infrared image. The Sobel algorithm can better express the gradient information of the source image.

[0106] In some embodiments, step S750 may include but is not limited to steps S751 to S753:

[0107] Step S751, constructing an infrared intensity loss function according to the intensity weight coefficient, the visible light image and the infrared light image;

[0108] Step S752, constructing a visible light loss function according to the gradient weight coefficient, the first gradient information and the second gradient information;

[0109] Step S753, obtaining the target loss function according to the infrared intensity loss function and the visible light loss function.

[0110] In step S751 of some embodiments, an infrared intensity loss function may be constructed according to the intensity weight coefficient μ, the visible light image vis, and the infrared light image ir. Then there are the following expressions:

[0111]

[0112] In the formula, represents the infrared intensity loss function; max(·) represents the maximum value operation; ‖·‖1 represents the first normal form operation.

[0113] In step S752 of some embodiments, according to the gradient weight coefficient η, the first gradient information And the second gradient information The visible light loss function can be constructed Then there are the following expressions:

[0114]

[0115] In the formula, Represents the visible light loss function.

[0116] In step S753 of some embodiments, a target loss function may be constructed according to the infrared intensity loss function and the visible light loss function. The expression of the target loss function is as follows:

[0117]

[0118] Where, L total represents the target loss function; α is the balance coefficient.

[0119] The infrared significant intensity and visible light texture details retained in the fusion image are adaptively adjusted through μ and η. When μ or η is a positive value, it means that the intensity or gradient information corresponding to the source image needs to be strengthened; when μ or η is a negative value, it means that the intensity or gradient information corresponding to the source image needs to be weakened; when μ or η is zero, it means that the intensity or gradient information corresponding to the image remains unchanged.

[0120] In step S800 of some embodiments, the initial fused image is adjusted according to the target loss function of adaptive strength and gradient, which can guide the fusion model to adaptively adjust the fusion weights, so as to better obtain the most important information in the source image and obtain the target fused image.

[0121] For example, Figure 3 As shown, the steps of an infrared and visible light image fusion method based on primary and secondary coding interactive compensation are as follows:

[0122] Step 1: Read the acquired infrared image and visible light image.

[0123] Step 2: Input the infrared image and the visible light image into the infrared and visible light image fusion model with primary and secondary coding interactive compensation, wherein the infrared and visible light image fusion model with primary and secondary coding interactive compensation includes a primary and secondary interactive encoder, a feature integrator and a decoder.

[0124] Step 3: Through the main-auxiliary interactive encoder, main feature extraction and auxiliary feature extraction are performed on the visible light image and the infrared light image. In the feature extraction channel, the main feature extraction channel of each layer is cascaded and interactively operated with the main feature channels of the previous layers and the auxiliary feature extraction channels of this layer to obtain the target visible light features and target infrared light features.

[0125] Step 4: Input the target visible light feature and the target infrared light feature into the feature integrator, and perform enhanced fusion on the target visible light feature and the target infrared light feature to obtain the fusion feature;

[0126] Step 5: Input the fused features into the decoder to generate an initial fused image with both infrared light intensity and visible light texture and rich information;

[0127] Step 6: According to the target loss function, the initial fused image is adjusted to obtain the target fused image. The target loss function is an adaptive intensity and gradient loss function designed based on the visible light image and the infrared light image.

[0128] The embodiment of the present invention further provides an infrared and visible light image fusion device based on primary and secondary code interactive compensation, which can implement the above-mentioned infrared and visible light image fusion method based on primary and secondary code interactive compensation. The device includes:

[0129] The first module is used to collect visible light images and infrared light images;

[0130] The second module is used to extract main features of the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features;

[0131] A third module is used to extract auxiliary features from the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features;

[0132] A fourth module is used to perform cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature;

[0133] A fifth module is used to perform cascade interaction on the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain target infrared light features;

[0134] A sixth module is used to generate an initial fused image according to the target visible light feature and the target infrared light feature;

[0135] A seventh module is used to construct a target loss function according to the visible light image and the infrared light image;

[0136] The eighth module is used to adjust the initial fused image according to the target loss function to obtain a target fused image.

[0137] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] The embodiment of the present invention further provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the infrared and visible light image fusion method based on primary and secondary coding interactive compensation is implemented. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0139] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] refer to Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0141] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0142] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes an infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to an embodiment of the present invention;

[0143] Input / output interface 903, used to implement information input and output;

[0144] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0145] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0146] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0147] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the infrared and visible light image fusion method based on primary and secondary coding interactive compensation is implemented.

[0148] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0149] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned infrared and visible light image fusion method based on primary and secondary coding interactive compensation.

[0150] In summary, the infrared and visible light image fusion method and device based on primary and secondary coding interactive compensation according to the embodiment of the present invention has the following advantages:

[0151] 1. For the current infrared and visible light image fusion method based on deep learning, an encoder is usually used to extract features during the feature encoding process. In the process of feature transmission, feature loss is inevitable as the network depth increases. The embodiment of the present invention introduces a pre-trained image reconstruction auxiliary encoder at the feature encoding end to assist the image fusion main encoder, which can better realize the information transmission of important features and prevent feature loss, thereby improving the quality of image fusion.

[0152] 2. Currently, many scholars have designed a large number of loss functions to constrain the fusion image from multiple directions and angles in order to improve the fusion performance of the model. However, there is usually a certain correlation between the loss functions and the weight relationship is usually set artificially, which will inevitably greatly limit the fusion performance of the model. An adaptive intensity and gradient loss function designed in an embodiment of the present invention can guide the fusion model to adaptively adjust the fusion weights, so as to better obtain the most important information in the source image.

[0153] 3. The embodiments of the present invention have great research value for subsequent computer advanced vision tasks, such as visual tracking, video surveillance, target detection, pedestrian re-identification, face recognition and autonomous driving, by improving the performance of infrared and visible light image fusion. Infrared and visible light image fusion plays an important role in sensitive target tracking and identification, perimeter protection, accident identification and evidence collection, and can serve security management, prevention and control and other businesses. In addition, this technology can also be applied to multi-functional telescopes, night vision goggles and other equipment and can be set in related equipment to achieve accurate detection of hidden targets in harsh environments. It is of great significance in security monitoring and object detection.

[0154] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0155] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0156] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0157] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0158] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0159] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0160] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0161] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0162] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for fusion of infrared and visible light images based on interactive compensation of primary and secondary codes, characterized in that: The following steps are involved: Collect visible light images and infrared light images; Extracting main features of the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features; Extracting auxiliary features from the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features; Performing cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature; Performing cascade interaction on the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain target infrared light features; generating an initial fused image according to the target visible light feature and the target infrared light feature; Constructing a target loss function according to the visible light image and the infrared light image; According to the target loss function, the initial fused image is adjusted to obtain a target fused image.

2. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 1 is characterized in that: The extracting main features of the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features includes the following steps: Extracting features of the visible light image by using a visible light main encoder to obtain multi-level visible light main features; The infrared light image is subjected to feature extraction through an infrared light main encoder to obtain multi-level infrared light main features.

3. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 1 is characterized in that: The extracting auxiliary features from the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features includes the following steps: Extracting features from the visible light image using a visible light auxiliary encoder to obtain multi-level visible light auxiliary features; The infrared light image is subjected to feature extraction through an infrared light-assisted encoder to obtain multi-level infrared light-assisted features.

4. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 1 is characterized in that: The step of cascading and interacting the multi-level visible light main features with the multi-level visible light auxiliary features to obtain a target visible light feature comprises the following steps: The visible light main feature of the current layer is cascaded and interacted with the visible light main features of the previous layers and the visible light auxiliary features of the corresponding layers to obtain the target visible light feature.

5. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 1 is characterized in that: The step of cascading and interacting the multi-level infrared light main features with the multi-level infrared light auxiliary features to obtain target infrared light features comprises the following steps: The infrared light main feature of the current layer is cascaded and interacted with the infrared light main features of the previous layers and the infrared light auxiliary features of the corresponding layers to obtain the target infrared light feature.

6. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 1 is characterized in that: The step of generating an initial fused image according to the target visible light feature and the target infrared light feature comprises the following steps: Performing enhanced fusion on the target visible light feature and the target infrared light feature to obtain a fusion feature; The initial fused image is generated according to the fused features.

7. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 1 is characterized in that: The constructing of a target loss function according to the visible light image and the infrared light image comprises the following steps: Acquire an intensity weight coefficient of the infrared light image according to the visible light image and the infrared light image; Acquiring first gradient information of the visible light image; Acquiring second gradient information of the infrared light image; Acquire a gradient weight coefficient of the visible light image according to the first gradient information and the second gradient information; The target loss function is constructed according to the intensity weight coefficient and the gradient weight coefficient.

8. The infrared and visible light image fusion method based on primary and secondary coding interactive compensation according to claim 7 is characterized in that: The constructing of the target loss function according to the intensity weight coefficient and the gradient weight coefficient comprises the following steps: constructing an infrared intensity loss function according to the intensity weight coefficient, the visible light image and the infrared light image; Constructing a visible light loss function according to the gradient weight coefficient, the first gradient information, and the second gradient information; The target loss function is obtained according to the infrared intensity loss function and the visible light loss function.

9. An infrared and visible light image fusion device based on primary and secondary coding interactive compensation, characterized in that: include: The first module is used to collect visible light images and infrared light images; The second module is used to extract main features of the visible light image and the infrared light image to obtain multi-level visible light main features and multi-level infrared light main features; A third module is used to extract auxiliary features from the visible light image and the infrared light image to obtain multi-level visible light auxiliary features and multi-level infrared light auxiliary features; A fourth module is used to perform cascade interaction on the multi-level visible light main features and the multi-level visible light auxiliary features to obtain a target visible light feature; A fifth module is used to perform cascade interaction on the multi-level infrared light main features and the multi-level infrared light auxiliary features to obtain target infrared light features; A sixth module is used to generate an initial fused image according to the target visible light feature and the target infrared light feature; A seventh module is used to construct a target loss function according to the visible light image and the infrared light image; The eighth module is used to adjust the initial fused image according to the target loss function to obtain a target fused image.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 8.

Images