Infrared image and visible light image fusion method, system, computing device and medium

Abstract

The present invention relates to a method, system, computing device and medium for fusing infrared images and visible light images. The method comprises: converting a first infrared image into a first pseudo visible light image with color information, and converting a first visible light image registered with the first infrared image into a first pseudo infrared image with visible light detail texture information; extracting a first Y channel of the first infrared image; extracting a second Y channel of the first pseudo infrared image; extracting a CrCb channel of the first pseudo visible light image; fusing the first Y channel and the second Y channel to obtain a target Y channel, recombining the target Y channel and the CrCb channel to obtain a target YCrCb color model, and mapping the target YCrCb color model to a corresponding RGB color model to obtain a fused target image. The method realizes the fusion of detail texture information of the first infrared image and the first visible light image, and has good imaging quality.

Description

Background Art

[0002] The fusion of infrared images and visible light images is an important image processing technology in computer vision and has been widely used in many scenarios, including pedestrian re-identification, target tracking and video surveillance. The purpose of image fusion is to extract more meaningful complementary feature representation information from different modal images and aggregate them to generate a fused image with more practical application value.

[0003] A large number of image fusion technical solutions are disclosed in the prior art, including frameworks based on autoencoders, convolutional neural networks, generative adversarial networks, and transformers. However, when these deep learning-based image fusion algorithms fuse visible light images and infrared images, the overexposed background of the visible light image blocks the detailed texture information from the infrared image, resulting in poor imaging quality of the target image obtained after fusion. Summary of the invention

[0004] In order to overcome the problem that when a traditional deep learning-based image fusion algorithm fuses visible light images and infrared images, the overexposed background of the visible light image obscures the detailed texture information from the infrared image, resulting in poor imaging quality of the target image obtained after the fusion, the present invention provides a method, system, computing device and medium for fusing infrared images and visible light images.

[0005] In a first aspect, in order to solve the above technical problems, the present invention provides a method for fusing infrared images and visible light images, comprising:

[0006] Converting the first infrared image into a first pseudo visible light image having color information, and converting the first visible light image registered with the first infrared image into a first pseudo infrared image having visible light detail texture information;

[0007] Extracting a first Y channel of the first infrared image in a corresponding YCrCb color model based on the thermal radiation information of the first infrared image;

[0008] Extracting a second Y channel of the first pseudo infrared image in a corresponding YCrCb color model based on the visible light detail texture information and thermal radiation information of the first pseudo infrared image;

[0009] Extracting a CrCb channel of the first pseudo visible light image in a corresponding YCrCb color model based on color information of the first pseudo visible light image;

[0010] The first Y channel and the second Y channel are fused to obtain a target Y channel, the target Y channel and the CrCb channel are recombined to obtain a target YCrCb color model, and the target YCrCb color model is mapped to a corresponding RGB color model to obtain a fused target image.

[0011] In a second aspect, in order to solve the above technical problems, the present invention provides an infrared image and visible light image fusion system, comprising:

[0012] a conversion module, configured to convert the first infrared image into a first pseudo visible light image having color information, and to convert the first visible light image registered with the first infrared image into a first pseudo infrared image having visible light detail texture information;

[0013] A first extraction module, configured to extract a first Y channel of the first infrared image in a corresponding YCrCb color model based on the thermal radiation information of the first infrared image;

[0014] A second extraction module, configured to extract a second Y channel of the first pseudo infrared image in a corresponding YCrCb color model based on the visible light detail texture information and thermal radiation information of the first pseudo infrared image;

[0015] a third extraction module, configured to extract a CrCb channel of the first pseudo visible light image in a corresponding YCrCb color model based on color information of the first pseudo visible light image;

[0016] An image fusion module is used to fuse the first Y channel and the second Y channel to obtain a target Y channel, recombine the target Y channel and the CrCb channel to obtain a target YCrCb color model, and map the target YCrCb color model to a corresponding RGB color model to obtain a fused target image.

[0017] In the third aspect, in order to solve the above-mentioned technical problem, the present invention provides a computing device, including a memory, a processor, and a program stored in the memory and running on the processor, and when the processor executes the program, the steps of the infrared image and visible light image fusion method as described above are implemented.

[0018] In a fourth aspect, in order to solve the above-mentioned technical problem, the present invention provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the steps of the infrared image and visible light image fusion method as described above.

[0019] The beneficial effect of the present invention is that by converting the first infrared image into the first pseudo visible light image across modes and converting the first visible light image into the first pseudo infrared image across modes, the first pseudo visible light image obtained has richer color information than the first infrared image, and the first pseudo infrared image obtained retains both the detail texture information of the first visible light image and the thermal radiation information of the infrared image. That is, when the first visible light image is converted into the first pseudo infrared image, the brightness of the overexposed area is suppressed by imitating the brightness characteristics of the infrared image, and the foreground target in the image is predicted to enhance the brightness of the foreground target, so that the first pseudo infrared image retains the detail texture information of the first visible light image and also has the thermal radiation information of the infrared image.

[0020] By extracting the first Y channel and the second Y channel of the first infrared image and the first pseudo infrared image in the YCrCb color model respectively, the Y channel represents the brightness of the image and carries all the information of the image except the color information. Therefore, the first Y channel has the thermal radiation information of the first infrared image, and the second Y channel has the thermal radiation information of the first pseudo infrared image and the detail texture information of the original visible light image. Then extract the CrCb channel of the first pseudo visible light image in the YCrCb color model to obtain the color information of the first pseudo visible light image.

[0021] In this way, by fusing the first Y channel and the second Y channel to obtain the target Y channel, since the first pseudo infrared image suppresses the brightness of the overexposed area and predicts the foreground target in the image to improve the brightness of the foreground target, the thermal radiation information of the first pseudo infrared image and the detail texture information of the original visible light image carried by the second Y channel will not block the detail texture information from the infrared image, so that the fused target image has the thermal radiation information of the first infrared image, the thermal radiation information of the first pseudo infrared image and the detail texture information of the original visible light image. That is, the fusion of the detail texture information of the first infrared image and the first visible light image is preliminarily realized. By recombining the target Y channel and the CrCb channel to obtain the target YCrCb color model, and then mapping the target YCrCb color model to the corresponding RGB color model, the fused target image can be obtained, which has the detail texture information of the first infrared image and the first visible light image and the color information of the first pseudo visible light image. The target image will not block the detail texture information from the original first infrared image due to the overexposed background of the original first visible light image, and has good imaging quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a method for fusing infrared images and visible light images provided by the present invention;

[0023] Figure 2An embodiment of image fusion provided by the present invention;

[0024] Figure 3 An embodiment of the joint training of a first target generator and a second target generator provided by the present invention;

[0025] Figure 4 Another embodiment of image fusion provided by the present invention;

[0026] Figure 5 A schematic diagram of the structure of a deep residual network provided by the present invention;

[0027] Figure 6 An embodiment of image cross-modal conversion provided by the present invention;

[0028] Figure 7 A schematic diagram of the structure of a double-loop cross-adversarial cross-modal conversion framework provided by the present invention;

[0029] Figure 8 A structural schematic diagram of a global structure tensor holding module provided by the present invention;

[0030] Fig. 9 A schematic diagram of the structure of an infrared image and visible light image fusion system provided by the present invention. DETAILED DESCRIPTION

[0031] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0032] Infrared and visible light image fusion is an important image processing technology in computer vision and has been widely used in various scenarios, including pedestrian re-identification, target tracking and video surveillance. Due to the different imaging characteristics of infrared sensors and visible light sensors, there are significant modal differences between infrared images and visible light images. Generally speaking, infrared images have more prominent thermal targets and global contrast, while visible light images have richer texture and structural information, but the contrast between targets and backgrounds is often poor. The purpose of image fusion is to extract more meaningful complementary feature representation information from different modal images and aggregate them to generate fused images with more practical application value.

[0033] Due to the practical value of infrared and visible light image fusion, it has attracted extensive attention and research, and a large number of image fusion technology solutions have been proposed in the field of image fusion. They can be summarized as traditional methods and methods based on deep learning. Classic traditional methods mainly include methods based on multi-scale transformation, methods based on sparse representation, methods based on subspace, methods based on saliency, and methods based on variational models. These methods usually rely on manually designed feature extractors to capture information in the image.

[0034] However, these feature extractors may not be able to adapt to complex scenes or image changes, resulting in limited quality of the fusion results. Deep learning-based methods mainly include frameworks based on autoencoders, frameworks based on convolutional neural networks, frameworks based on generative adversarial networks, and frameworks based on Transformers. Although the performance of recent image fusion algorithms based on deep learning has been greatly improved compared to traditional methods, these methods cannot effectively remove the interfering exposed background in visible light images, resulting in the overexposed background of visible light images blocking the detailed texture information from the infrared image, resulting in poor imaging quality of the fused target image. For example, the overexposed background information from the visible light image will block the structural texture information and infrared thermal targets from the infrared image in the fused image. Secondly, the introduction of an overly bright visible light image background is not conducive to the highlighting of infrared thermal targets.

[0035] The following describes an infrared image and visible light image fusion method, system, computing device and medium according to an embodiment of the present invention in conjunction with the accompanying drawings.

[0036] like Figure 1 As shown, the embodiment of the present disclosure provides a method for fusing an infrared image and a visible light image, comprising:

[0037] Step S11 : converting the first infrared image into a first pseudo visible light image with color information, and converting the first visible light image registered with the first infrared image into a first pseudo infrared image with visible light detail texture information.

[0038] It is understandable that the registration between infrared images and visible light images is a relatively mature existing technology, so it will not be described in detail here.

[0039] In addition, the visible light detail texture information is detail texture information of the visible light image.

[0040] Step S12: extracting a first Y channel of the first infrared image in a corresponding YCrCb color model based on the thermal radiation information of the first infrared image.

[0041] It can be understood that an infrared image is an image that distinguishes a target from its background based on radiation differences. Therefore, the thermal radiation information of an infrared image is the brightness characteristic of the infrared image. It can also be understood as the detailed texture information of the infrared image.

[0042] Specifically, the first infrared image is converted from an RGB color model to a YCrCb color model, and the first Y channel (also called the first Y intensity channel) of the YCrCb color model is extracted.

[0043] Step S13: extracting the second Y channel of the first pseudo infrared image in the corresponding YCrCb color model based on the visible light detail texture information and the thermal radiation information of the first pseudo infrared image.

[0044] Specifically, the first pseudo infrared image is converted from the RGB color model to the YCrCb color model, and the second Y channel (also called the second Y intensity channel) of the YCrCb color model is extracted.

[0045] Step S14: extracting the CrCb channel of the first pseudo visible light image in the corresponding YCrCb color model based on the color information of the first pseudo visible light image.

[0046] Specifically, the first pseudo visible light image is converted from an RGB color model to a YCrCb color model, and a CrCb channel of the YCrCb color model is extracted.

[0047] Step S15, the first Y channel and the second Y channel are merged to obtain a target Y channel, the target Y channel and the CrCb channel are recombined to obtain a target YCrCb color model, and the target YCrCb color model is mapped to a corresponding RGB color model to obtain a fused target image.

[0048] Specifically, the corresponding points of the pixel matrices of the first Y channel and the second Y channel are multiplied one by one, so as to achieve the purpose of correcting the thermal target area of ​​the infrared image and the rich and scarce infrared texture, and at the same time generate a fused Y channel pixel matrix, that is, the target Y channel of the final fused image (target image). Thus, the fusion of the detail texture information of the first infrared image and the first visible light image is preliminarily realized, and a grayscale fused image is obtained.

[0049] A method for fusing an infrared image and a visible light image provided by an embodiment of the present disclosure is adopted, by converting a first infrared image into a first pseudo visible light image across modes and converting the first visible light image into a first pseudo infrared image across modes, so that the obtained first pseudo visible light image has richer color information than the first infrared image, and the obtained first pseudo infrared image retains both the detail texture information of the first visible light image and the thermal radiation information of the infrared image. That is, when the first visible light image is converted into the first pseudo infrared image, the brightness of the overexposed area is suppressed by imitating the brightness characteristics of the infrared image, and the foreground target in the image is predicted to enhance the brightness of the foreground target, so that the first pseudo infrared image retains the detail texture information of the first visible light image and also has the thermal radiation information of the infrared image. That is, when the first visible light image is converted into the first pseudo infrared image, the brightness of the overexposed area is suppressed by imitating the brightness characteristics of the infrared image, and the foreground target in the image is predicted to enhance the brightness of the foreground target, so that the first pseudo infrared image retains the detail texture information of the first visible light image and also has the thermal radiation information of the infrared image.

[0050] By extracting the first Y channel and the second Y channel of the first infrared image and the first pseudo infrared image in the YCrCb color model respectively, the Y channel represents the brightness of the image and carries all the information of the image except the color information. Therefore, the first Y channel has the thermal radiation information of the first infrared image, and the second Y channel has the thermal radiation information of the first pseudo infrared image and the detail texture information of the original visible light image. Then extract the CrCb channel of the first pseudo visible light image in the YCrCb color model to obtain the color information of the first pseudo visible light image.

[0051] In this way, by fusing the first Y channel and the second Y channel to obtain the target Y channel, since the first pseudo infrared image suppresses the brightness of the overexposed area and predicts the foreground target in the image to improve the brightness of the foreground target, the thermal radiation information of the first pseudo infrared image and the detail texture information of the original visible light image carried by the second Y channel will not block the detail texture information from the infrared image, so that the fused target image has the thermal radiation information of the first infrared image, the thermal radiation information of the first pseudo infrared image and the detail texture information of the original visible light image. That is, the fusion of the detail texture information of the first infrared image and the first visible light image is preliminarily realized. By recombining the target Y channel and the CrCb channel to obtain the target YCrCb color model, and then mapping the target YCrCb color model to the corresponding RGB color model, the fused target image can be obtained, which has the detail texture information of the first infrared image and the first visible light image and the color information of the first pseudo visible light image. The target image will not block the detail texture information from the original first infrared image due to the overexposed background of the original first visible light image, and has good imaging quality.

[0052] For ease of understanding, combined Figure 2 As shown, Figure 2 This is an embodiment of image fusion. Figure 2 The method includes two groups of images: one is a1 first visible light image, b1 first infrared image, and c1 target image which is obtained by fusion of a1 first visible light image and b1 first infrared image; the other is a2 first visible light image, b2 first infrared image, and c2 target image which is obtained by fusion of a2 first visible light image and b2 first infrared image.

[0053] Preferably, the first infrared image is converted into a first pseudo visible light image with color information by a first target generator, and the first visible light image registered with the first infrared image is converted into a first pseudo infrared image with visible light detail texture information by a second target generator. The first target generator and the second target generator are jointly trained. Figure 3 As shown, Figure 3An embodiment of joint training of a first target generator and a second target generator is provided. The joint training process is:

[0054] S21, acquiring multiple pairs of registered second infrared images and second visible light images as training sets.

[0055] It can be understood that the type of the selected second infrared image includes: a conventional infrared image or an infrared image with a specific thermal target. In this way, by screening the type of the second infrared image, when the second visible light image imitates the brightness characteristics of the second infrared image to generate the second pseudo infrared image, the second pseudo infrared image has the thermal radiation information of the second infrared image, that is, the second pseudo infrared image has the brightness characteristics of the second infrared image. So that when there are requirements for infrared thermal targets, the fused target image can also reflect the infrared thermal target with specific requirements, thereby further improving the imaging quality of the target image. So that the target image can be applied in different high-order visual tasks.

[0056] S22, selecting a pair of registered second infrared image and second visible light image from the training set.

[0057] S23, input the second infrared image to the first initial generator to determine a second pseudo visible light image, input the second visible light image to the second initial generator to determine a second pseudo infrared image, and determine a first identification loss of the first discriminator based on the second pseudo visible light image and the second visible light image, and determine a second identification loss of the second discriminator based on the second pseudo infrared image and the second infrared image.

[0058] It can be understood that both the first initial generator and the second initial generator are cross-modal generators, wherein the first initial generator is a generator that converts the infrared image modality into the visible light image modality, and the second initial generator is a generator that converts the visible light image modality into the infrared image modality.

[0059] S24, input the second pseudo visible light image to the second initial generator to determine the reconstructed infrared image, input the second pseudo infrared image to the first initial generator to determine the reconstructed visible light image, and determine the total loss of the first initial generator and the second initial generator based on the second visible light image, the second infrared image, the second pseudo visible light image, the second pseudo infrared image, the reconstructed visible light image and the reconstructed infrared image.

[0060] S25, adjust the first discriminator based on the first discrimination loss, adjust the second discriminator based on the second discrimination loss, and adjust the first initial generator and the second initial generator based on the total loss, and repeat steps S22 to S25 until a preset number of iterations, and / or until the first discrimination loss, the second discrimination loss and the total loss converge to obtain a trained first target generator and a second target generator.

[0061] Specifically, at the end of each cycle, the network parameters of the first discriminator and the second discriminator are first unfrozen, and the network parameter values ​​of the first discriminator and the second discriminator are updated respectively by back-propagating the discriminator loss. Secondly, the network parameters of the first discriminator and the second discriminator are frozen, and the network parameters of the first initial generator and the second initial generator are unfrozen. The total loss of the first initial generator and the second initial generator is back-propagated to update the network parameters of the first initial generator and the second initial generator. This process is performed repeatedly until a preset number of cycles is reached, and / or a predetermined loss function threshold (also known as total loss convergence) is reached. Among them, one cycle period is 1 round of iteration.

[0062] In some embodiments, the preset number of rounds is 200 rounds.

[0063] In some embodiments, the present invention iterates through preset secondary rounds to obtain the trained first target generator and the second target generator. In other embodiments, the present invention determines the training comprehensive loss based on the first identification loss, the second identification loss and the total loss, and when the comprehensive loss converges, the trained first target generator and the second target generator are obtained. In other embodiments, the present invention iterates through preset secondary rounds to determine the comprehensive loss at each iteration. The first initial generator and the second initial generator corresponding to the secondary round when the comprehensive loss converges are respectively determined as the first target generator and the second target generator.

[0064] In this way, when the second infrared image is converted across modes by the first target generator, the color information of the second visible light image can be learned, so that the obtained second pseudo visible light image has richer color information than the original second infrared image to improve the imaging quality of the target image. In addition, when the second visible light image is converted across modes by the second target generator, the brightness characteristics of the second infrared image can be learned, the brightness of the overexposed area in the visible light image can be suppressed, and the foreground target area can be predicted to obtain brightness improvement, so that the obtained second pseudo infrared image has the detail texture information of the original second visible light image and the thermal radiation information of the infrared image. Therefore, when performing image fusion, the second pseudo infrared image will not block the detail texture information and infrared thermal target of the second infrared image, so that the target image has the detail texture information of the original second visible light image and the thermal radiation information after the conversion mode, as well as the thermal radiation information of the original second infrared image, and has higher imaging quality.

[0065] In addition, by jointly training the first initial generator and the second initial generator, the obtained first target generator and the second target generator can respectively learn the nonlinear mapping relationship between the foreground and the background of two different modalities, so as to achieve the brightness suppression of the highlighted background and the brightness enhancement of the foreground target in the visible light image to be fused. As a result, after the image fusion, the overexposed area in the visible light image to be fused will not block the detailed texture information and infrared target in the infrared image to be fused. Thus, better effects are achieved in correcting and highlighting the infrared target, retaining the rich texture information of the visible light scene and suppressing the intensity of its bright background information.

[0066] For ease of understanding, combined Figure 4 As shown, Figure 4This is another embodiment of image fusion. First, the first infrared image is input into the first target generator to obtain the first pseudo visible light image, and the first visible light image is input into the second target generator to obtain the first pseudo infrared image. Secondly, the RGB color model of the first infrared image is converted into the YCrCb color model by using the color conversion module 1 and the Y channel (i.e., the first Y channel) is extracted; the RGB color model of the first pseudo infrared image is converted into the CrCb color model by using the color conversion module 1 and the Y channel (i.e., the second Y channel) is extracted; the RGB color model of the first pseudo visible light image is converted into the YCrCb color model by using the color conversion module 2 and the CrCb channel is extracted. Thirdly, the Y channel of the first infrared image and the Y channel of the first pseudo infrared image are fused to obtain the fused Y channel (i.e., the target Y channel); the fused Y channel is fused with the CrCb channel of the first pseudo visible light image to obtain the YCrCb channel (i.e., the target YCrCb color model). Finally, the color conversion module 3 is used to convert the YCrCb into the RGB channel to obtain the fused target image.

[0067] The first target generator and the second target generator are both a deep residual network structure. Figure 5 As shown, Figure 5 It is a deep residual network structure.

[0068] For ease of understanding, combined Figure 4 , Figure 5 and Figure 6 As shown, Figure 6It is an embodiment of image cross-modal conversion. In this embodiment, the input layer (input image: the first visible light image or the first infrared image) is first edge-filled to maintain the size and boundary information of the tensor. It is followed by two convolutional layers. Convolutional layer 1 is formed by three convolution operations with a step size of 1 and a convolution kernel of 3×3 and a LeakyReLU activation function in series, the purpose of which is to increase the depth of the network, so that the network has more opportunities to learn more complex features, while also reducing the number of parameters and reducing the risk of overfitting. Convolutional layer 2 is formed by a convolution operation with a step size of 2 and a convolution kernel of 3×3 and a LeakyReLU activation function in series, which is used to reduce the size of the input image to one-fourth of the original size and extract more important features in the image. The LeakyReLU activation function is used here to prevent the gradient from disappearing. The middle part of the generator contains 18 residual blocks, each of which consists of two 3×3 convolution operations, two Instance Normalizations and a jump connection. The role of the residual block is to promote the flow of information so that the deeper features of the network can be better propagated, thereby helping the network to better learn the details and texture information of the image and improve the quality of the generated image. In addition, the jump connection can effectively avoid the problem of gradient disappearance. After the dense residual block, the image is upsampled to the original size through two convolution layers 3 and 4 with a stride of 2 and a convolution kernel of 3×3, and a convolution layer 5 with a stride of 1 and a convolution kernel of 7×7. Finally, the output layer uses the Tanh activation function to limit the output range to [-1,1].

[0069] It can be understood that the two initial generators have powerful feature expression and nonlinear mapping capabilities, respectively, with a set of densely connected residual blocks. First, the network learns the nonlinear mapping relationship between the foreground and background of visible light images to infrared images, and builds a probabilistic prediction model to suppress the highlight background and enhance the foreground target in the visible light image. Secondly, the network extracts important features of visible light images and infrared images, and learns the characteristic intensity distribution characteristics between the two modal images to achieve the preservation of the richness of the source image structure texture and the improvement of its intensity during the modal conversion process.

[0070] In addition, the first target generator and the second target generator are completed through joint training. The joint training process does not involve the fusion process of infrared images and visible light images. Due to the independence and functional completeness of the training network, it can be efficiently integrated into other image fusion systems to significantly improve their performance.

[0071] Preferably, determining the first identification loss of the first discriminator according to the second pseudo visible light image and the second visible light image includes: obtaining a first loss of the first discriminator identifying the second pseudo visible light image as a corresponding pseudo image. Obtaining a second loss of the first discriminator identifying the second visible light image as a corresponding real image. Determining the first identification loss based on the first loss and the second loss.

[0072] Since the discriminator can only identify the authenticity of one mode, the first discriminator is used to identify whether the second pseudo visible light image is the corresponding visible light image mode, and whether the second visible light image is the corresponding visible light image mode, and the first identification loss is determined to determine the identification accuracy of the first discriminator, that is, to determine the loss value of the first discriminator identifying the true image as the corresponding true image, and the pseudo image as the corresponding pseudo image. The network parameters of the first discriminator are adjusted so that the first discriminator can identify more accurately and improve the performance of the first discriminator. This prompts the first initial generator to generate a second pseudo visible light image that is difficult for the first discriminator to identify as true or false, so that the second infrared image can better imitate the color information of the second visible light image.

[0073] Specifically, obtaining the first loss of the first discriminator identifying the second pseudo visible light image as the corresponding pseudo image includes: adding a corresponding pseudo label to the second pseudo visible light image. Based on the second pseudo visible light image and the pseudo label, determining the first loss of the first discriminator identifying the second pseudo visible light image as the pseudo corresponding pseudo image is calculated as follows: L S1 =0.5×||D1(G vis )-F label1 ||2. Among them, L S1 is the first loss, D1 is the first discriminator, G vis is the second pseudo visible light image, F label1 is the corresponding pseudo label, and ||·||2 is the L2 norm.

[0074] It can be understood that the first loss is the loss value of the first discriminator discriminating the second pseudo visible light image as the corresponding pseudo label. label1 is a pseudo label of the same dimension and size as the second pseudo visible light image. In some embodiments, the pseudo label is a tensor data with all values ​​being 0.0.

[0075] Specifically, obtaining the second loss of the first discriminator identifying the second visible light image as the corresponding real image includes: adding the corresponding real label to the second visible light image. Based on the second visible light image and the real label, determining the second loss of the first discriminator identifying the second visible light image as the corresponding real image is calculated as follows: L S2 =0.5×||D1(I vis )-R label1 ||2. Among them, LS2 is the second loss, D1 is the first discriminator, I vis is the second visible light image, R label1 is the corresponding true label, and ||·||2 is the L2 norm.

[0076] It can be understood that the second loss is the loss value of the first discriminator discriminating the second visible light image as the corresponding true label. label1 is a true label of the same dimension and size as the second visible light image. In some embodiments, the true label is a tensor data with all values ​​1.0.

[0077] Specifically, determining the first identification loss based on the first loss and the second loss includes: a calculation formula is as follows: L D1 =L S1 +L S2 Among them, L D1 is the first identification loss, L S1 is the first loss, L S2 is the second loss. In other words, L D1 =0.5×(||D1(G vis )-F label1 ||2+||D1(I vis )-R label1 ||2).

[0078] In addition, after each training round, the first identification loss is obtained, and the network parameters of the first discriminator are adjusted through the back propagation of the first identification loss to improve the performance of the first discriminator.

[0079] Preferably, determining the second identification loss of the second discriminator according to the second pseudo infrared image and the second infrared image includes: obtaining a third loss of the second discriminator identifying the second pseudo infrared image as a corresponding pseudo image, obtaining a fourth loss of the second discriminator identifying the second infrared image as a corresponding real image, and determining the second identification loss based on the third loss and the fourth loss.

[0080] Since the discriminator can only identify the authenticity of one mode, the second discriminator is used to identify whether the second pseudo infrared image is the corresponding infrared image mode, and whether the second infrared image is the corresponding infrared image mode, and the second identification loss is determined to determine the identification accuracy of the second discriminator, that is, to determine the loss value of the second discriminator identifying the true image as the corresponding true image, and the pseudo image as the corresponding pseudo image. Thus, the network parameters of the second discriminator are adjusted so that the second discriminator can identify more accurately and improve the performance of the second discriminator. Thus, the second initial generator is prompted to generate a second pseudo infrared image that is difficult to be identified as true or false by the second discriminator, so that the second visible light image can better imitate the brightness characteristics of the second infrared image, so that the second pseudo infrared image has the detail texture information of the original second visible light image and the thermal radiation information of the second infrared image.

[0081] Specifically, obtaining the second loss of the second discriminator identifying the second pseudo infrared image as the corresponding pseudo image includes: adding a corresponding pseudo label to the second pseudo infrared image. Based on the second pseudo infrared image and the pseudo label, determining the third loss of the second discriminator identifying the second pseudo infrared image as the pseudo corresponding pseudo image is calculated as follows: L S3 =0.5×||D2(G ir )-F label2 ||2. Among them, L S3 is the third loss, D2 is the second discriminator, G ir is the second pseudo infrared image, F label2 is the corresponding pseudo label, and ||·||2 is the L2 norm.

[0082] It can be understood that the third loss is the loss value of the second discriminator discriminating the second pseudo infrared image as the corresponding pseudo label. label2 is a pseudo label of the same dimension and size as the second pseudo infrared image. In some embodiments, the pseudo label is a tensor data with all values ​​being 0.0.

[0083] Specifically, obtaining the fourth loss of the second discriminator identifying the second infrared image as the corresponding real image includes: adding the corresponding real label to the second infrared image. Based on the second infrared image and the real label, the fourth loss of the second discriminator identifying the second infrared image as the corresponding real image is determined, and the calculation formula is as follows: L S4 =0.5×||D2(I ir )-R label2 ||2. Among them, L S4 is the fourth loss, D2 is the second discriminator, I ir is the second infrared image, R label2 is the corresponding true label, and ||·||2 is the L2 norm.

[0084] It can be understood that the fourth loss is the loss value of the second discriminator discriminating the second infrared image as the corresponding true label. label2 is a true label of the same dimension and size as the second infrared image. In some embodiments, the true label is a tensor data with all values ​​being 1.0.

[0085] Specifically, the second identification loss is determined based on the third loss and the fourth loss, including: the calculation formula is as follows: L D2 =L S3 +L S4 Among them, L D2 is the second identification loss, L S3 For the third loss, L S4 is the fourth loss. In other words, L D2 =0.5×(||D2(G ir )-F label2 ||2+||D2(I ir )-R label2 ||2).

[0086] In addition, after each training round, the second identification loss is obtained, and the network parameters of the second discriminator are adjusted through the back propagation of the second identification loss to improve the performance of the second discriminator.

[0087] Preferably, determining the total loss of the first initial generator and the second initial generator according to the second visible light image, the second infrared image, the second pseudo visible light image, the second pseudo infrared image, the reconstructed visible light image, and the reconstructed infrared image includes: determining the adversarial loss based on the second pseudo visible light image and the second pseudo infrared image. Determining the perceptual style cycle consistency loss based on the second visible light image, the reconstructed visible light image, the second infrared image, and the reconstructed infrared image. Determining the global structure tensor perceptual loss based on the second visible light image, the reconstructed visible light image, the second infrared image, and the reconstructed infrared image. Determining the total loss of the first generator and the second generator based on the adversarial loss, the perceptual style cycle consistency loss, and the global structure tensor perceptual loss.

[0088] Adversarial loss (also known as generative adversarial loss) is the probability that the fake image generated by the generator is judged as a real image by the discriminator. The lower the value, the more realistic the fake image generated by the generator. By minimizing the generative adversarial loss, the quality of the fake image generated by the generator can be improved, so as to improve the imaging quality of the target image.

[0089] The perceptual style cycle consistency loss is used to calculate the perceptual style loss of the input image after two cross-modal conversions (for example, the second visible light image is cross-modally converted to a pseudo infrared image modality, which is cross-modally converted to a reconstructed visible light image modality; the second infrared image is cross-modally converted to a pseudo visible light image modality, which is cross-modally converted to a reconstructed infrared image modality.). In other words, by determining the modal style difference and structural texture difference between the input image and the reconstructed image obtained after two cross-modal conversions, the perceptual style cycle consistency loss is determined, thereby constraining the generator to generate a more realistic pseudo image. In order to further improve the imaging quality of the target image.

[0090] The global structure tensor-aware loss is used to determine the difference in perceptual structural information between images to guide the generator to generate clearer target modal structure texture information, while suppressing the generation of undesirable artifacts to further improve the imaging quality of the target image.

[0091] Preferably, determining the adversarial loss based on the second pseudo visible light image and the second pseudo infrared image includes: obtaining a first recognition loss when the first discriminator recognizes the second pseudo visible light image as a corresponding real image. Obtaining a second recognition loss when the second discriminator recognizes the second pseudo infrared image as a corresponding real image. Determining the adversarial loss based on the first recognition loss and the second recognition loss.

[0092] In this way, by determining the first recognition loss, the probability that the second pseudo visible light image generated by the first initial generator is judged as a real visible light image is determined. By determining the second recognition loss, the probability that the second pseudo infrared image generated by the second initial generator is judged as a real infrared image is determined. Based on the adversarial loss, the network parameters of the first initial generator and the second initial generator are adjusted to obtain the first target generator and the second target generator. Thus, the first target generator can learn the color information of the second visible light image to generate a pseudo visible light image with rich color information. And, the second target generator can learn the brightness characteristics of the second infrared image to generate a pseudo infrared image with visible light detail texture information and thermal radiation information.

[0093] Specifically, obtaining the first recognition loss of the first discriminator identifying the second pseudo visible light image as the corresponding real image includes: adding a corresponding real label to the second pseudo visible light image. Based on the second pseudo visible light image and the corresponding real label, determining the first recognition loss of the first discriminator identifying the second pseudo visible light image as the corresponding real image. The calculation formula is as follows: L G1 =||D1(G vis )-D1(R label3 )||2. Among them, L G1 is the first recognition loss. D1 is the first discriminator, Gvis is the second pseudo visible light image, R label3 is the true label corresponding to the second pseudo visible light image, and ||·||2 is the L2 norm.

[0094] It can be understood that the true label is a true label of the same dimension and size as the second pseudo visible light image. In some embodiments, the true label is a tensor data with all values ​​1.0. D1 is used to identify the authenticity of the image modality style and generate predicted tensor data with the same dimension and size as the input image and values ​​of 0.0 to 1.0. According to L G1 From the calculation formula, we can see that by solving the L2 norm through the tensor data of the true label and the predicted tensor data, we can get the first recognition loss.

[0095] Specifically, obtaining the second recognition loss of the second discriminator identifying the pseudo second infrared image as the corresponding real image includes: adding a corresponding real label to the second pseudo infrared image. Based on the second pseudo infrared image and the corresponding real label, determining the second recognition loss of the second discriminator identifying the pseudo second infrared image as the corresponding real image. The calculation formula is as follows: L G2 =||D2(G ir )-D2(R label4 )||2. Among them, L G2 is the second recognition loss. D2 is the second discriminator, G ir is the second pseudo infrared image, R label4 is the true label corresponding to the second pseudo infrared image, and ||·||2 is the L2 norm.

[0096] It can be understood that the true label is a true label of the same dimension and size as the second pseudo infrared image. In some embodiments, the true label is a tensor data with all values ​​1.0. D2 is used to identify the authenticity of the image modality style and generate predicted tensor data with the same dimension and size as the input image and values ​​of 0.0 to 1.0. According to L G2 From the calculation formula, we can see that by solving the L2 norm through the tensor data of the true label and the predicted tensor data, we can get the first recognition loss.

[0097] Specifically, based on the first recognition loss and the second recognition loss, the adversarial loss is determined, including: the calculation formula is as follows: L GAN =L G1 +L G2 Among them, L GAN To combat the loss, L G1 is the first recognition loss, L G2 is the second recognition loss. In other words, L GAN =||D1(G vis )-D1(R label3 )||2+||D2(G ir)-D2(R label4 )||2.

[0098] In addition, by adding corresponding true labels to the second pseudo visible light image and the second pseudo infrared image, the first discriminator and the second discriminator are deceived. Then the loss value of the first discriminator identifying the second pseudo visible light image as the corresponding true image and the loss value of the second discriminator identifying the second pseudo infrared image as the corresponding true image are calculated respectively. Then the function formula of these two loss values ​​is used:

[0099] L GAN =||D1(G vis )-D1(R label3 )||2+||D2(G ir )-D2(R label4 )||2, which constitutes the adversarial loss between the first generator and the second generator.

[0100] Preferably, determining the perceptual style cycle consistency loss based on the second visible light image, the reconstructed visible light image, the second infrared image, and the reconstructed infrared image includes: obtaining the perceptual difference and the style difference between the second visible light image and the reconstructed visible light image, and obtaining the first perceptual loss and the first style loss respectively. Obtaining the perceptual difference and the style difference between the second infrared image and the reconstructed infrared image, and obtaining the second perceptual loss and the second style loss respectively. Determining the perceptual style cycle consistency loss based on the first perceptual loss, the first style loss, the second perceptual loss, and the second style loss.

[0101] In this way, by determining the perceptual differences and style differences between the input image and the corresponding reconstructed image, the modal style differences and structural texture differences between the input image and the corresponding reconstructed image can be determined, and then based on the losses corresponding to these differences, the parameters of the networks of the first initial generator and the second initial generator are adjusted so that the first initial generator and the second initial generator generate more realistic pseudo images.

[0102] Specifically, the perception difference and style difference between the second visible light image and the reconstructed visible light image are obtained, and the first perception loss and the first style loss are obtained respectively, including: the calculation formula is as follows:

[0103] Among them, L C1 is the first perceptual loss, L style1 is the first style loss, I vis is the second visible light image, R visTo reconstruct the visible light image. ψ represents the VGG-19 model, which is a classic deep convolutional neural network model with 19 layers. It can be used to extract the feature representation of the input image at each network depth level. The previous layers are mainly responsible for capturing low-level features, such as edges, corners, etc. As the network deepens, the later layers gradually capture higher-level features, such as texture, shape, object parts, etc. The corresponding ψ i Represents the features of the i-th layer in the VGG-19 model. Υ represents the Gram matrix, which is used to suppress checkerboard artifacts. ω i is the weight corresponding to the i-th layer feature extracted by the VGG-19 model. The weight increases with the depth of the feature level. ||·||1 is the L1 norm, ||·||2 is the L2 norm, and ∑ is the summation operation.

[0104] In some embodiments, ω i According to experience, it is set

[0105] Specifically, the perception difference and style difference between the second infrared image and the reconstructed infrared image are obtained, and the second perception loss and the second style loss are obtained respectively, including: the calculation formula is as follows: L C2 =||I ir -R ir ||1+||ψ3(I ir )-ψ3(R ir )||1, Among them, L C2 is the second perceptual loss, L style2 is the second style loss, I ir is the second infrared image, R ir To reconstruct infrared images.

[0106] Specifically, determining the perceptual style cycle consistency loss based on the first perceptual loss, the first style loss, the second perceptual loss, and the second style loss includes: determining the total perceptual loss based on the first perceptual loss and the second perceptual loss, and determining the total style loss based on the first style loss and the second style loss. Determining the perceptual style cycle consistency loss based on the total perceptual loss and the total style loss. The calculation formula is as follows: L C =L C1 +L C2 , L style =L style1 +L style2 , L cycle =L C +λL style Among them, L C is the total perceptual loss, L style is the total style loss, L cycle is the perceptual style cycle consistency loss. λ is used to balance LC and L style In L cycle The influence of is set to 100.0.

[0107] Preferably, determining the global structure tensor perception loss based on the second visible light image, the reconstructed visible light image, the second infrared image, and the reconstructed infrared image includes: respectively obtaining the structure tensor information of the second visible light image, the reconstructed visible light image, the second infrared image, and the reconstructed infrared image. Fusing the structure tensor information of the second visible light image and the reconstructed infrared image to obtain the first fused structure tensor information. Fusing the structure tensor information of the second visible light image and the second infrared image to obtain the second fused structure tensor information. Fusing the structure tensor information of the reconstructed visible light image and the second infrared image to obtain the third fused structure tensor information. Based on the first fused structure tensor information, the second fused structure tensor information, and the third fused structure tensor information, determining the global structure tensor perception loss.

[0108] In this way, the first fusion structure tensor information, the second fusion structure tensor information and the third fusion structure tensor information are obtained by determining the structure tensor information of the three images. Then, based on the first fusion structure tensor information, the second fusion structure tensor information and the third fusion structure tensor information, the global structure tensor perceptual loss is determined, so that the difference in perceptual structure information between each pair of images can be determined to guide the pseudo image generated by the first initial generator and the second initial generator to have clearer target modal structure texture information. In addition, the global structure tensor information on the two paths (path 1: the second infrared image is converted into the second pseudo visible light image through the first initial generator, and the second pseudo visible light image is converted into the reconstructed infrared image through the second initial generator. Path 2: the second visible light image is converted into the second pseudo infrared image through the second initial generator, and the second pseudo infrared image is converted into the reconstructed visible light image through the first initial generator) is integrated, and the total amount of perceptual structure is maintained unchanged, so that the style consistency of the generated pseudo image and the real modality can be perceived at the level of image fusion, which can effectively overcome the dependence of the pseudo image on the need for true value label supervised learning.

[0109] Specifically, the structure tensor fusion function ξ is used to fuse the structure tensor information of the second visible light image and the reconstructed infrared image to obtain the first pair of fused structure tensor information ξ(R vis ,I ir ), fuse the structure tensor information of the second visible light image and the second infrared image to obtain a second pair of fused structure tensor information ξ(I vis ,I ir ), and fusing the reconstructed visible light image and the second infrared image to obtain the third fused structure tensor information ξ(R vis ,I ir ).

[0110] It can be understood that the structure tensor blending function ξ is a prior art, so it will not be described in detail here.

[0111] Specifically, based on the first fused structure tensor information, the second fused structure tensor information, and the third fused structure tensor information, the global structure tensor perception loss is determined, including: determining the first global structure tensor loss between the second pair of fused structure tensor information and the first pair of fused structure tensor information. Determine the second global structure tensor loss between the second fused structure tensor information and the third fused structure tensor information. Determine the global structure perception tensor loss based on the first global structure tensor loss and the second global structure tensor loss. The calculation formula is as follows:

[0112]

[0113] Among them, L STP is the global structure tensor-aware loss, is the first global structure tensor loss, is the second global structure tensor loss. λ1 and λ2 are preset coefficients. In some embodiments, based on experience and training results, λ1 and λ2 are set to 5×10 6 .

[0114] Due to the fact that the pseudo image lacks true value constraints, a large number of undesirable artifacts and serious structural texture degradation appear in the modal conversion process. The present invention calculates the global structure tensor through the perceptual loss to maintain the total perceptual structural information difference between modules, thereby perceiving the style consistency between the generated pseudo image and the real modal image at the image fusion level, so as to achieve the purpose of overcoming the dependence of the pseudo image on the true value label.

[0115] For ease of understanding, combined Figure 7 As shown, Figure 7 A dual-loop cross-adversarial cross-modal conversion framework is provided. The framework includes two cross-modal loop conversion paths and three global structure tensor retention modules. The first cross-modal loop conversion path includes a first initial generator, a second initial generator, and a first discriminator. The second cross-modal conversion path includes not only the first initial generator and the second initial generator of the first cross-modal loop conversion path, but also a second discriminator.

[0116] Specifically, the first cross-modal cyclic conversion path: the second infrared image is converted into a second pseudo visible light image via the first initial generator, and the second pseudo visible light image is converted into a reconstructed infrared image via the second initial generator. The first discriminator is used to identify the visible light image modality. The second cross-modal cyclic conversion path: the second visible light image is converted into a second pseudo infrared image via the second initial generator, and the second pseudo infrared image is converted into a reconstructed visible light image via the first initial generator. The second discriminator is used to identify the infrared image modality. The first perceptual style cycle consistency loss L between the second infrared image and the reconstructed infrared image is calculated respectively. cycle1 , and calculating the second perceptual style cycle consistency loss L between the second visible light image and the reconstructed visible light image cycle2 , thereby determining the total perceptual style cycle consistency loss L of the first initial generator and the second initial generator based on the two perceptual style cycle consistency losses cycle .

[0117] Three global structure tensor holding modules span across these two parallel cross-modal cyclic conversion paths. The three global structure tensor holding modules take three pairs of preset opposite modal images on two cross-modal cyclic conversion paths as input, and extract and synthesize the structure tensor information of each pair of images through a deep convolutional network. It can be understood that the pair of images input to the global structure tensor holding module is located on different cross-modal cyclic conversion paths.

[0118] In some embodiments, the global structure tensor holding module 1 is used to extract the structure tensor information of the second visible light image and the reconstructed infrared image as a pair of images (i.e., the images input to the global structure tensor holding module 1), and the global structure tensor holding module 2 is used to extract the structure tensor information of the second visible light image and the second infrared image as a pair of images (i.e., the images input to the global structure tensor holding module 2). The global structure tensor holding module 3 is used to extract the structure tensor information of the second infrared image and the reconstructed visible light image as a pair of images (i.e., the images input to the global structure tensor holding module 3). Then determine the first global structure tensor loss L between the global tensor holding module 1 and the global tensor holding module 2. STP1 , and the second global structure tensor loss L between the global tensor holding module 2 and the global tensor holding module 3 STP1 , and these two global structure tensor losses determine the global structure tensor perception loss L STP The global structure tensor-aware loss calculates the total perceived structural information difference between two groups of global structure tensor preserving modules (one group is global tensor preserving module 1 and global tensor preserving module 2, and the other group is global tensor preserving module 2 and global tensor preserving module 3) to guide the network to generate clearer target modality structure texture information while suppressing the generation of undesirable artifacts.

[0119] In addition, for ease of understanding, combined with Figure 8 As shown, Figure 8 A global structure tensor preservation module is provided. The global structure tensor preservation module includes two parallel structure tensor extractors and a serial structure tensor reconstructor. The two structure tensor extractors are used to extract structure tensor feature maps of input images of two cross-modal cyclic transformation paths (path 1 and path 2) respectively. Each structure tensor extractor contains a convolution operation, an activation operation and two residual blocks. The convolution kernel of the convolution operation is 3×3, and the activation function is LeakyReLU, which is used to extract shallow features. Each residual block consists of two 3×3 convolution layers, and the first layer is followed by a batch normalization and a leakyReLU unit layer. The output of the second layer is directly passed to the next level together with the input of the first layer, thereby alleviating the gradient vanishing problem during training. The reconstructor uses CNN to parameterize the fusion of the structure tensors of the two paths. The specific processing process is as follows: First, the structure tensor feature maps extracted in parallel on the two paths are fully connected and subjected to a 5×5 convolution operation. Secondly, the fused structure tensor feature maps are reconstructed through three serially connected residual blocks. Finally, a 3×3 convolutional layer is passed to generate an N-channel global structure tensor feature map. It can be understood that the structure tensor information of the finally generated global structure tensor feature map is the structure tensor information of the global structure tensor holding module.

[0120] Preferably, based on the adversarial loss, the perceptual style cycle consistency loss and the global structure tensor perceptual loss, the total loss of the first generator and the second generator is determined, including: the calculation formula is as follows: L G =L cycle +L GAN +L STP Among them, L G is the total loss of the first generator and the second generator, L cycle is the perceptual style cycle consistency loss, L GAN To generate the adversarial loss, L STF is a global structure tensor-aware loss.

[0121] Combination Fig. 9As shown, the embodiment of the present disclosure provides an infrared image and visible light image fusion system 100. The system includes: a conversion module 101, a first extraction module 102, a second extraction module 103, a third extraction module 104 and an image fusion module 105. Among them, the conversion module 101 is used to convert the first infrared image into a first pseudo visible light image with color information, and convert the first visible light image registered with the first infrared image into a first pseudo infrared image with visible light detail texture information. The first extraction module 102 is used to extract the first Y channel of the first infrared image in the corresponding YCrCb color model based on the thermal radiation information of the first infrared image. The second extraction module 103 is used to extract the second Y channel of the first pseudo infrared image in the corresponding YCrCb color model based on the visible light detail texture information and thermal radiation information of the first pseudo infrared image. The third extraction module 104 is used to extract the CrCb channel of the first pseudo visible light image in the corresponding YCrCb color model based on the color information of the first pseudo visible light image. The image fusion module 105 is used to fuse the first Y channel and the second Y channel to obtain a target Y channel, recombining the target Y channel and the CrCb channel to obtain a target YCrCb color model, and mapping the target YCrCb color model to a corresponding RGB color model to obtain a fused target image.

[0122] A system for fusion of infrared images and visible light images using an embodiment of the present disclosure converts a first infrared image into a first pseudo visible light image across modalities and converts the first visible light image into a first pseudo infrared image across modalities, so that the obtained first pseudo visible light image has richer color information than the first infrared image, and the obtained first pseudo infrared image retains both the detail texture information of the first visible light image and the thermal radiation information of the infrared image. That is, when the first visible light image is converted into the first pseudo infrared image, the brightness of the overexposed area is suppressed by imitating the brightness characteristics of the infrared image, and the foreground target in the image is predicted to enhance the brightness of the foreground target, so that the first pseudo infrared image retains the detail texture information of the first visible light image and also has the thermal radiation information of the infrared image. That is, when the first visible light image is converted into the first pseudo infrared image, the brightness of the overexposed area is suppressed by imitating the brightness characteristics of the infrared image, and the foreground target in the image is predicted to enhance the brightness of the foreground target, so that the first pseudo infrared image retains the detail texture information of the first visible light image and also has the thermal radiation information of the infrared image.

[0123] By extracting the first Y channel and the second Y channel of the first infrared image and the first pseudo infrared image in the YCrCb color model respectively, the Y channel represents the brightness of the image and carries all the information of the image except the color information. Therefore, the first Y channel has the thermal radiation information of the first infrared image, and the second Y channel has the thermal radiation information of the first pseudo infrared image and the detail texture information of the original visible light image. Then extract the CrCb channel of the first pseudo visible light image in the YCrCb color model to obtain the color information of the first pseudo visible light image.

[0124] In this way, by fusing the first Y channel and the second Y channel to obtain the target Y channel, since the first pseudo infrared image suppresses the brightness of the overexposed area and predicts the foreground target in the image to improve the brightness of the foreground target, the thermal radiation information of the first pseudo infrared image and the detail texture information of the original visible light image carried by the second Y channel will not block the detail texture information from the infrared image, so that the fused target image has the thermal radiation information of the first infrared image, the thermal radiation information of the first pseudo infrared image and the detail texture information of the original visible light image. That is, the fusion of the detail texture information of the first infrared image and the first visible light image is preliminarily realized. By recombining the target Y channel and the CrCb channel to obtain the target YCrCb color model, and then mapping the target YCrCb color model to the corresponding RGB color model, the fused target image can be obtained, which has the detail texture information of the first infrared image and the first visible light image and the color information of the first pseudo visible light image. The target image will not block the detail texture information from the original first infrared image due to the overexposed background of the original first visible light image, and has good imaging quality.

[0125] Preferably, the first infrared image is converted into a first pseudo visible light image with color information by a first target generator, and the first visible light image registered with the first infrared image is converted into a first pseudo infrared image with visible light detail texture information by a second target generator, and the first target generator and the second target generator are completed by joint training. The joint training process jointly trains the first target generator and the second target generator through a training set acquisition unit, a selection unit, a discriminator loss unit, a generator loss unit, and a parameter adjustment unit. Among them, the training set acquisition unit is used to acquire multiple pairs of registered second infrared images and second visible light images as training sets. The selection unit is used to select a pair of registered second infrared images and second visible light images from the training set. The discriminator loss unit is used to input the second infrared image into the first initial generator to determine the second pseudo visible light image, input the second visible light image into the second initial generator to determine the second pseudo infrared image, and determine the first discrimination loss of the first discriminator according to the second pseudo visible light image and the second visible light image, and determine the second discrimination loss of the second discriminator according to the second pseudo infrared image and the second infrared image. A generator loss unit is used to input the second pseudo visible light image into the second initial generator to determine the reconstructed infrared image, input the second pseudo infrared image into the first initial generator to determine the reconstructed visible light image, and determine the total loss of the first initial generator and the second initial generator according to the second visible light image, the second infrared image, the second pseudo visible light image, the second pseudo infrared image, the reconstructed visible light image, and the reconstructed infrared image. A parameter adjustment unit is used to adjust the first discriminator based on the first discrimination loss, adjust the second discriminator based on the second discrimination loss, and adjust the first initial generator and the second initial generator based on the total loss, and repeatedly execute the steps in the step selection unit, the discriminator loss unit, the generator loss unit, and the parameter adjustment unit until the preset rounds of iterations are performed, and / or until the first discrimination loss, the second discrimination loss, and the total loss converge to obtain the trained first target generator and the second target generator.

[0126] Preferably, the discriminator loss unit includes a discriminator loss subunit. The discriminator loss subunit is used to obtain a first loss when the first discriminator identifies the second pseudo visible light image as a corresponding pseudo image. Obtain a second loss when the first discriminator identifies the second visible light image as a corresponding real image. Determine the first discriminator loss based on the first loss and the second loss.

[0127] Preferably, the total loss unit of the generator includes an adversarial loss subunit, a perceptual style cycle consistency loss subunit, a global structure tensor perceptual loss subunit and a total loss subunit. Among them, the adversarial loss subunit is used to determine the adversarial loss based on the second pseudo visible light image and the second pseudo infrared image. The perceptual style cycle consistency loss subunit is used to determine the perceptual style cycle consistency loss based on the second visible light image, the reconstructed visible light image, the second infrared image and the reconstructed infrared image. The global structure tensor perceptual loss subunit is used to determine the global structure tensor perceptual loss based on the second visible light image, the reconstructed visible light image, the second infrared image and the reconstructed infrared image. The total loss subunit is used to determine the total loss of the first generator and the second generator based on the adversarial loss, the perceptual style cycle consistency loss and the global structure tensor perceptual loss.

[0128] Preferably, the adversarial loss subunit is specifically used to obtain a first recognition loss when the first discriminator recognizes the second pseudo visible light image as the corresponding real image, obtain a second recognition loss when the second discriminator recognizes the second pseudo infrared image as the corresponding real image, and determine the adversarial loss based on the first recognition loss and the second recognition loss.

[0129] Preferably, the perceptual style cycle consistency loss subunit is specifically used to obtain the perceptual difference and style difference between the second visible light image and the reconstructed visible light image, and obtain the first perceptual loss and the first style loss respectively. The perceptual difference and style difference between the second infrared image and the reconstructed infrared image are obtained, and the second perceptual loss and the second style loss are obtained respectively. The perceptual style cycle consistency loss is determined based on the first perceptual loss, the first style loss, the second perceptual loss, and the second style loss.

[0130] Preferably, the global structure tensor perception loss subunit is specifically used to obtain the structure tensor information of the second visible light image, the reconstructed visible light image, the second infrared image, and the reconstructed infrared image, respectively. The structure tensor information of the second visible light image and the reconstructed infrared image is fused to obtain the first fused structure tensor information. The structure tensor information of the second visible light image and the second infrared image is fused to obtain the second fused structure tensor information. The structure tensor information of the reconstructed visible light image and the second infrared image is fused to obtain the third fused structure tensor information. Based on the first fused structure tensor information, the second fused structure tensor information, and the third fused structure tensor information, the global structure tensor perception loss is determined.

[0131] A computing device according to an embodiment of the present invention includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, some or all steps of the above-mentioned infrared image and visible light image fusion method are implemented.

[0132] Among them, the computing device can be a computer, and correspondingly, its program is computer software. The above-mentioned parameters and steps in a computing device of the present invention can refer to the parameters and steps in the embodiment of the infrared image and visible light image fusion method mentioned above, and will not be repeated here.

[0133] In an embodiment of the present invention, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed, the steps of the above-mentioned infrared image and visible light image fusion method are executed.

[0134] The computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0135] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method of the embodiment of the present disclosure. The aforementioned computer-readable storage medium may be a non-transitory computer-readable storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes, or a transient computer-readable storage medium.

[0136] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code. Computer-readable storage media can be, for example, but not limited to - electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above.

[0137] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0138] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for fusing infrared images and visible light images, characterized in that: include: Converting the first infrared image into a first pseudo visible light image having color information, and converting the first visible light image registered with the first infrared image into a first pseudo infrared image having visible light detail texture information; Extracting a first Y channel of the first infrared image in a corresponding YCrCb color model based on the thermal radiation information of the first infrared image; Extracting a second Y channel of the first pseudo infrared image in a corresponding YCrCb color model based on the visible light detail texture information and thermal radiation information of the first pseudo infrared image; Extracting a CrCb channel of the first pseudo visible light image in a corresponding YCrCb color model based on color information of the first pseudo visible light image; Fusing the first Y channel and the second Y channel to obtain a target Y channel, recombining the target Y channel and the CrCb channel to obtain a target YCrCb color model, and mapping the target YCrCb color model to a corresponding RGB color model to obtain a fused target image; The first infrared image is converted into a first pseudo visible light image with color information by a first target generator, and the first visible light image registered with the first infrared image is converted into a first pseudo infrared image with visible light detail texture information by a second target generator. The first target generator and the second target generator are completed by joint training, and the joint training process is: S21, acquiring multiple pairs of registered second infrared images and second visible light images as training sets; S22, selecting a pair of registered second infrared image and second visible light image from the training set; S23, inputting the second infrared image into the first initial generator to determine a second pseudo visible light image, inputting the second visible light image into the second initial generator to determine a second pseudo infrared image, and determining a first identification loss of a first discriminator according to the second pseudo visible light image and the second visible light image, and determining a second identification loss of a second discriminator according to the second pseudo infrared image and the second infrared image; S24, inputting the second pseudo visible light image to the second initial generator to determine a reconstructed infrared image, inputting the second pseudo infrared image to the first initial generator to determine a reconstructed visible light image, and determining a total loss of the first initial generator and the second initial generator according to the second visible light image, the second infrared image, the second pseudo visible light image, the second pseudo infrared image, the reconstructed visible light image, and the reconstructed infrared image; S25. Adjust the first discriminator based on the first identification loss, adjust the second discriminator based on the second identification loss, and adjust the first initial generator and the second initial generator based on the total loss, and repeat steps S22 to S25 until a preset number of iterations, and / or until the first identification loss, the second identification loss and the total loss converge to obtain the trained first target generator and the second target generator.

2. The method according to claim 1, characterized in that The determining a first identification loss of a first discriminator according to the second pseudo visible light image and the second visible light image comprises: Obtaining a first loss of the first discriminator identifying the second pseudo visible light image as a corresponding pseudo image; Obtaining a second loss of the first discriminator identifying the second visible light image as a corresponding real image; The first identification loss is determined based on the first loss and the second loss.

3. The method according to claim 1 or 2, characterized in that: The determining the total loss of the first initial generator and the second initial generator according to the second visible light image, the second infrared image, the second pseudo visible light image, the second pseudo infrared image, the reconstructed visible light image, and the reconstructed infrared image comprises: determining a countermeasure loss based on the second pseudo visible light image and the second pseudo infrared image; determining a perceptual style cycle consistency loss based on the second visible light image, the reconstructed visible light image, the second infrared image, and the reconstructed infrared image; determining a global structure tensor-aware loss based on the second visible light image, the reconstructed visible light image, the second infrared image, and the reconstructed infrared image; Based on the adversarial loss, the perceptual style cycle consistency loss, and the global structure tensor perceptual loss, a total loss of the first initial generator and the second initial generator is determined.

4. The method according to claim 3, characterized in that The determining of the adversarial loss based on the second pseudo visible light image and the second pseudo infrared image includes: Obtaining a first recognition loss of the first discriminator for recognizing the second pseudo visible light image as a corresponding real image; Obtaining a second recognition loss for the second discriminator to recognize the second pseudo infrared image as a corresponding real image; The adversarial loss is determined based on the first recognition loss and the second recognition loss.

5. The method according to claim 3, characterized in that: The determining of the perceptual style cycle consistency loss based on the second visible light image, the reconstructed visible light image, the second infrared image, and the reconstructed infrared image comprises: Obtaining a perceptual difference and a style difference between the second visible light image and the reconstructed visible light image, and obtaining a first perceptual loss and a first style loss respectively; Obtaining a perceptual difference and a style difference between the second infrared image and the reconstructed infrared image, and obtaining a second perceptual loss and a second style loss respectively; The perceptual-style cycle consistency loss is determined based on the first perceptual loss, the first style loss, the second perceptual loss, and the second style loss.

6. The method according to claim 3, characterized in that The determining of a global structure tensor perception loss based on the second visible light image, the reconstructed visible light image, the second infrared image, and the reconstructed infrared image comprises: Respectively acquiring structural tensor information of the second visible light image, the reconstructed visible light image, the second infrared image, and the reconstructed infrared image; fusing the structural tensor information of the second visible light image and the reconstructed infrared image to obtain first fused structural tensor information; fusing the structure tensor information of the second visible light image and the second infrared image to obtain second fused structure tensor information; fusing the structure tensor information of the reconstructed visible light image and the second infrared image to obtain third fused structure tensor information; The global structure tensor perceived loss is determined based on the first fused structure tensor information, the second fused structure tensor information, and the third fused structure tensor information.

7. Infrared image and visible light image fusion system, characterized in that: include: a conversion module, configured to convert the first infrared image into a first pseudo visible light image having color information, and to convert the first visible light image registered with the first infrared image into a first pseudo infrared image having visible light detail texture information; A first extraction module, configured to extract a first Y channel of the first infrared image in a corresponding YCrCb color model based on the thermal radiation information of the first infrared image; A second extraction module, configured to extract a second Y channel of the first pseudo infrared image in a corresponding YCrCb color model based on the visible light detail texture information and thermal radiation information of the first pseudo infrared image; a third extraction module, configured to extract a CrCb channel of the first pseudo visible light image in a corresponding YCrCb color model based on color information of the first pseudo visible light image; An image fusion module, used for fusing the first Y channel and the second Y channel to obtain a target Y channel, recombining the target Y channel and the CrCb channel to obtain a target YCrCb color model, and mapping the target YCrCb color model to a corresponding RGB color model to obtain a fused target image; The conversion module is specifically used to convert the first infrared image into a first pseudo visible light image with color information through a first target generator, and convert the first visible light image registered with the first infrared image into a first pseudo infrared image with visible light detail texture information through a second target generator; wherein the first target generator and the second target generator are completed through joint training; the joint training process jointly trains the first target generator and the second target generator through a training set acquisition unit, a selection unit, a discriminator loss unit, a generator loss unit, and a parameter adjustment unit; A training set acquisition unit, used for acquiring a plurality of pairs of registered second infrared images and second visible light images as training sets; A selection unit, used for selecting a pair of registered second infrared images and second visible light images from a training set; a discriminator loss unit, configured to input the second infrared image into the first initial generator to determine a second pseudo visible light image, input the second visible light image into the second initial generator to determine a second pseudo infrared image, and determine a first discriminator loss of the first discriminator according to the second pseudo visible light image and the second visible light image, and determine a second discriminator loss of the second discriminator according to the second pseudo infrared image and the second infrared image; a generator loss unit, configured to input the second pseudo visible light image to the second initial generator to determine a reconstructed infrared image, input the second pseudo infrared image to the first initial generator to determine a reconstructed visible light image, and determine a total loss of the first initial generator and the second initial generator according to the second visible light image, the second infrared image, the second pseudo visible light image, the second pseudo infrared image, the reconstructed visible light image, and the reconstructed infrared image; A parameter adjustment unit is used to adjust the first discriminator based on the first discrimination loss, adjust the second discriminator based on the second discrimination loss, and adjust the first initial generator and the second initial generator based on the total loss, and repeatedly execute the steps in the selection unit, the discriminator loss unit, the generator loss unit and the parameter adjustment unit until a preset number of iterations and / or until the first discrimination loss, the second discrimination loss and the total loss converge to obtain the trained first target generator and the second target generator.

8. A computing device comprising a memory, a processor, and a program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of the infrared image and visible light image fusion method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the steps of the infrared image and visible light image fusion method according to any one of claims 1 to 6.

Images