Method and device for converting visible light image into infrared image based on domain difference information

Abstract

The invention relates to a method and equipment for converting a visible light image into an infrared image based on domain difference information, and belongs to the technical field of image processing and computer vision. The method comprises the following steps: acquiring a target visible light image and target noise; the target noise is random noise; enabling the target visible light image to pass through a pre-training sensing network to obtain target domain difference information; and enabling the target visible light image to pass through an encoder network of a trained diffusion model, and enabling the target domain difference information to pass through a decoder network of the trained diffusion model to obtain a target infrared image. According to the method, firstly, the domain difference characteristics between the infrared image and the visible light image are learned, and the difference information is fused to the image generation process, so that the conversion quality of the infrared image and the visible light image is higher.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and computer vision, and relates to a method and device for converting a visible light image into an infrared image based on domain difference information. Background Art

[0002] The image generation problem is a basic research problem in image processing. Due to the high cost of infrared cameras, the number of infrared and visible light paired images is very limited, which to a certain extent restricts the development of infrared and visible light dual-modal technology. Image generation technology can use image processing technology to convert images of one modality into another modality, and the generated images are very realistic. Infrared and visible light image pairs collected from real scenes inevitably have the problem that infrared and visible light images cannot be fully registered, and the existing image generation has poor generation quality on infrared and visible light paired images.

[0003] At present, most infrared and visible light dual-modal image generation algorithms are based on GANs networks. However, GANs networks are prone to mode collapse and training instability during training, and the quality of infrared images generated by them is often limited. In contrast, the diffusion model gradually generates high-quality real images from noisy images by learning the noise distribution, without the need to train the discriminator and generator separately like GANs. The diffusion model divides the generation process into two stages: the diffusion stage is used to gradually add noise, and the generation stage restores the image by gradually denoising, which has significant advantages in generation quality and stability. For example, Chinese patent CN118279130A proposes a multimodal infrared-to-visible light image conversion method. The method constructs an infrared-to-visible light image conversion model composed of an edge detector, a compressed sensing model, a text encoder, an image encoder, and a conditional latent diffusion model. The text embedding representation generated by the text encoder is used to guide the image generation process of the conditional latent diffusion model, so that the semantic information of the generated image conforms to the text description. At the same time, the image embedding representation generated by the image encoder is used as the conditional input of the conditional latent diffusion model to guide the conditional latent diffusion model to consider additional edge constraint information when generating images, so that the edge contour of the generated image is as similar as possible to the edge contour of the input image.

[0004] However, there are significant differences between infrared images and visible light images in imaging mechanisms and image detail presentation. Although the current multimodal image conversion technology can realize the conversion between infrared images and visible light images to a certain extent, it is still unable to accurately capture the domain difference characteristics between the two, resulting in the generated infrared images being of insufficient quality and low realism. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a method and device for converting a visible light image into an infrared image based on domain difference information. The shallow domain difference information between the infrared image and the visible light image can be learned through the pre-trained perception network. The diffusion model can control the generation of the infrared image according to the input conditional information and the output of the pre-trained perception network. The diffusion model continuously removes noise during the inference stage to generate a clean target infrared image. The diffusion model is more stable than the generative adversarial network training, and the fusion of the pre-trained model and the diffusion model can make the image quality generated by the conversion between the infrared modality and the visible light modality higher.

[0006] In a first aspect of the present invention, the present invention provides a method for converting a visible light image into an infrared image based on domain difference information, comprising the following steps:

[0007] Acquire a target visible light image and target noise; the target noise is random noise;

[0008] Passing the target visible light image through a pre-trained perception network to obtain target domain difference information;

[0009] The target visible light image is passed through the encoder network of the trained diffusion model, and the target domain difference information is passed through the decoder network of the trained diffusion model to obtain a target infrared image.

[0010] In a second aspect of the present invention, the present invention further provides a device for converting a visible light image into an infrared image based on domain difference information, the device comprising:

[0011] A first acquisition unit is used to acquire a target visible light image and target noise; the target noise is random noise;

[0012] A first perception unit, configured to pass the target visible light image through a pre-trained perception network to obtain target domain difference information;

[0013] The first conversion unit is used to pass the target visible light image through the encoder network of the trained diffusion model and pass the target domain difference information through the decoder network of the trained diffusion model to obtain the target infrared image.

[0014] Beneficial effects of the present invention: The embodiments of the present invention can learn shallow domain difference information between infrared images and visible light images through pre-trained perception networks. The diffusion model can control the generation of infrared images based on input conditional information and the output of the pre-trained perception network. The diffusion model continuously removes noise during the inference stage, thereby generating a clean target infrared image. The diffusion model is more stable than the training of the generative adversarial network, and the fusion of the pre-trained model and the diffusion model can make the image quality generated by the conversion between the infrared modality and the visible light modality higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0016] Figure 1 It is a flowchart of a method for converting a visible light image into an infrared image based on domain difference information according to an embodiment of the present invention;

[0017] Figure 2 is a schematic diagram of a perception network according to an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of a perception network according to a preferred embodiment of the present invention;

[0019] Figure 4 A schematic diagram of the training process of the diffusion model according to an embodiment of the present invention;

[0020] Figure 5 Schematic diagram of the reasoning process of the diffusion model of an embodiment of the present invention;

[0021] Figure 6 Schematic diagram of the structure of a device for converting a visible light image into an infrared image based on domain difference information according to an embodiment of the present invention;

[0022] Figure 7 A schematic diagram of a perceptual network pre-training device according to an embodiment of the present invention;

[0023] Figure 8 Schematic diagram of a diffusion model training device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than limiting the present application. The singular forms of "a", "said" and "the" used in the present application and claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0025] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" used may be interpreted as "at..." or "when..." or "in response to determination".

[0026] In an embodiment of the present application, a method for converting a visible light image into an infrared image based on domain difference information is proposed. The visible light image and the infrared image are images of the same target scene. The visible light image acquired by a front-end device such as a network camera, an analog camera, etc. can be converted into an infrared image. The visible light image stored by a back-end device such as a server, a management device, a storage device, etc. can also be converted into an infrared image.

[0027] The method and device for converting a visible light image into an infrared image based on domain difference information of the present invention will be further described below in conjunction with the accompanying drawings of the specification.

[0028] See also Figure 1 FIG. 1 is a flow chart of a method for converting a visible light image into an infrared image based on domain difference information. The method may include:

[0029] Step 101, obtaining a target visible light image and target noise; the target noise is random noise;

[0030] Exemplarily, the target visible light image can be obtained through front-end devices such as optical imaging devices such as network cameras and analog cameras, or through back-end devices such as servers, management devices, storage devices, etc. Exemplarily, the target noise is random noise that obeys some specific distributions, such as random noise that obeys Gaussian distribution (normal distribution); the embodiment of the present invention replaces the position of the noisy infrared image in the diffusion model with random noise, and gradually removes the target noise to obtain the target infrared image.

[0031] Step 102: Pass the target visible light image through a pre-trained perception network to obtain target domain difference information;

[0032] In the embodiment of the present invention, the perception network uses cleaning, formatting, normalization and other operations through data preprocessing, feature extraction, difference identification and other processes to ensure the consistency and comparability of data. Filtering, denoising, smoothing and other technologies can also be used to improve data quality. The process of extracting useful information from raw data by using operations such as dimensionality reduction, conversion and representation of data. By identifying the differences between different fields or data sources, the domain difference information of visible light images and infrared images can be extracted.

[0033] For example, Figure 2As shown in the figure, on the one hand, the perception network divides the visible light image into multiple token image blocks by performing image segmentation on the visible light image, transforms each image block token respectively, and obtains the feature information of each image block through multi-layer convolution; on the other hand, the perception network transforms and pools the infrared image to obtain the feature information of the infrared image; by calculating the loss of the feature information of the visible light image and the feature information of the infrared image, the model can learn the difference information between the infrared and visible light images to optimize the perception network, so that the perception network can perceive the domain difference information between the visible light image and the infrared image. Provide support for the generation of infrared images.

[0034] For example, Figure 3 As shown, the perception network includes a first pooling layer, a first reconstruction layer, a second pooling layer, a batch normalization layer, an activation function layer, a third pooling layer, and a second reconstruction layer. Taking the perception network structure as an example, the pre-training process of the perception network includes the following steps:

[0035] 111. Acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are a registered image pair of the same scene;

[0036] In the embodiment of the present invention, the front-end device may be a dual-sensor front-end device, where the dual sensors refer to a visible light sensor and an infrared light sensor. The visible light sensor is a sensor for obtaining the visible light intensity of an object, and the image collected by the visible light sensor is called a visible light image. The infrared light sensor is a sensor for obtaining the infrared light intensity of an object, and the image collected by the infrared light sensor is called an infrared light image. Based on this, the first visible light image and the first infrared image at the same collection time can be obtained through the dual sensors of the front-end device.

[0037] In an embodiment of the present invention, the first infrared image and the first visible light image are first registered, and these data sets are divided into a training set and a test set, and the training set and the test set contain image pairs of various scenes and lighting; then the infrared image and the visible light image are sent to a pre-trained network to learn the domain difference between the infrared image and the visible light image; the data set used in this embodiment can include a variety of scenes such as campuses, streets, woods and other complex scenes, and the resolution sizes of these images include various resolution sizes such as 512×512, 1024×1024 and so on.

[0038] The first infrared image and the first visible light image are first registered. After the registration is completed, the image pair is cropped to a resolution of 256×256, the first visible light image and the first infrared image are annotated, and then the data set is randomly divided into a training set and a test set. The first infrared and first visible light image pairs after registration are input into the pre-trained perception network model to learn the domain difference features between the two modalities. The learned domain difference information will be integrated with the feature map extracted during the image generation process through feature fusion, thereby improving the cross-modal conversion quality and accuracy of the generated image.

[0039] In some preferred embodiments, taking into account that the information levels contained in images in complex scenes and simple scenes are different, in simple scenes, simple scene images usually contain fewer objects, the background is relatively simple or does not interfere with the main object, has good lighting conditions, and a relatively fixed viewing angle, which makes their information volume relatively small and the complexity is low; while in complex scenes, complex scene images may contain a large number of objects, complex backgrounds such as campuses, streets, woods, etc., changeable lighting conditions and viewing angles, and mutual occlusion, which makes their information volume large and the complexity high.

[0040] Therefore, this embodiment uses a pre-classification network to determine whether the input first visible light image and the first infrared image are images of simple scenes or complex scenes. The pre-classification network can be a convolutional neural network (CNN) such as VGG, ResNet, etc. These networks can be directly used for classification tasks. If it is a simple scene, a lower domain weight is assigned to it, and if it is a complex scene, a higher domain weight is assigned to it. When pre-training the perception network, by determining whether the input first visible light image and the first infrared image are complex scenes or simple scenes, the corresponding domain weights are combined with the loss values ​​generated by the corresponding training samples, and the domain weights are used for back propagation, so that the network can dynamically adapt to scenes of different complexities.

[0041] 112. Pass the first infrared image and the first visible light image through a perception network to obtain a one-dimensional first infrared feature vector and a one-dimensional first visible light feature vector;

[0042] The first infrared image and the first visible light image are respectively passed through a perception network to obtain a one-dimensional first infrared feature vector and a one-dimensional first visible light feature vector, including:

[0043] Passing the first infrared image through a first pooling layer to obtain a one-dimensional first infrared feature vector;

[0044] dividing the first visible light image into a plurality of first visible light image blocks;

[0045] Each first visible light image block is respectively passed through a first reconstruction layer, a second pooling layer, a batch normalization layer, an activation function layer, a third pooling layer, and a second reconstruction layer to obtain a one-dimensional first visible light feature vector.

[0046] 113. Calculate a first loss by calculating a difference between the first infrared feature vector and the first visible light feature vector;

[0047] 114. Pre-train the perception network according to the first loss.

[0048] For example, before training the diffusion model, the pre-trained perception network is first trained to obtain a one-dimensional feature vector v by pooling and transforming the first infrared image. inf , pass the first visible light image through the perception network model to obtain a feature vector v of the same size vis , the L2 loss function is used to constrain the two feature vectors. The formula is: Loss = L2 (v vis ,v inf ).

[0049] The embodiment of the present invention enables the perception network to extract domain difference information between visible light images and infrared images by pre-training the perception network. By extracting the domain difference information of the target visible light image, the model can better generate the target infrared image.

[0050] 103. Pass the target visible light image through the trained encoder network of the diffusion model, and pass the target domain difference information through the trained decoder network of the diffusion model to obtain a target infrared image.

[0051] In the embodiment of the present invention, during the training process, the infrared image can be diffused and denoised; then the infrared image and the visible light image after denoising are sent to the diffusion model to learn the distribution of noise, and the pre-trained model participates in the learning of the network as a branch; finally, the visible light image is sent to the diffusion model for image generation, and the generated infrared image is close to the real situation by continuously removing noise. The present invention uses the diffusion model to realize the conversion between the infrared image and the visible light image, so that the generated result is more realistic.

[0052] For example, Figure 4 As shown in Figure 1, the diffusion model is based on an iterative process, which gradually adds noise to the data and then learns how to restore the original data from the noisy data. The diffusion model adopts the Unet network structure, including an encoder network and a decoder network.

[0053] The encoder network includes an encoder responsible for extracting high-level features from the input image. It consists of multiple convolutional layers and pooling layers, and its function is to gradually compress the spatial size of the image while extracting deep features of the image. In this process, as the number of network layers increases, the receptive field gradually increases, and a wider range of image information can be captured, such as the overall structure and contour of the image. With each convolutional layer, the number of channels of the image increases, so that richer features can be represented. Each stage typically includes two 3×3 convolutional layers followed by a 2×2 maximum pooling layer for downsampling. The embodiment of the present invention also introduces an embedding vector of time step t and incorporates it into the feature calculation of each layer so that the model can adjust its behavior according to different noise levels.

[0054] The decoder network reconstructs the high-level features extracted by the encoder back to the resolution of the original image. Corresponding to the encoder, it gradually restores the original size and detail information of the image through upsampling layers and convolution layers. The upsampling operation usually uses methods such as interpolation to expand the size of the feature map to match the size of the feature map of the corresponding level in the encoder, and then further refines and adjusts the features through the convolution layer to generate more accurate image details. It also consists of a series of convolutional layers, but transposed convolution (or deconvolution) is used after each stage to expand the spatial size of the feature map. Each step is connected to the feature map from the corresponding level of the encoder. This jump connection helps to retain fine-grained spatial information, making the generated image more accurate and clear.

[0055] In the embodiment of the present invention, in addition to using standard skip connections, this embodiment also uses a cross-attention module to enhance the modeling capability of global dependencies; fusing multimodal features through these methods can further improve the generation effect.

[0056] Taking the diffusion model as an example, the training process of the diffusion model includes the following steps:

[0057] 121. Acquire a second visible light image and a second infrared image; the second visible light image and the second infrared image are a registered image pair of the same scene;

[0058] In the embodiment of the present invention, the front-end device may be a dual-sensor front-end device, where the dual sensors refer to a visible light sensor and an infrared light sensor. The visible light sensor is a sensor for obtaining the visible light intensity of an object, and the image collected by the visible light sensor is called a visible light image. The infrared light sensor is a sensor for obtaining the infrared light intensity of an object, and the image collected by the infrared light sensor is called an infrared light image. Based on this, a second visible light image and a second infrared image at the same collection time can be obtained through the dual sensors of the front-end device.

[0059] In an embodiment of the present invention, the second infrared image is first registered with the second visible light image, and these data sets are divided into a training set and a test set, the training set and the test set contain image pairs of various scenes and lighting; then the infrared image and the visible light image are sent to a diffusion model to extract the domain difference between the infrared image and the visible light image; the data set used in this embodiment can include a variety of scenes such as campuses, streets, woods and other complex scenes, and the resolution sizes of these images include various resolution sizes such as 512×512, 1024×1024 and so on.

[0060] Among them, the second infrared image and the second visible light image are first registered. After the registration is completed, the image pair is cropped to a resolution of 256×256, the second visible light image and the second infrared image are annotated, and then the data set is randomly divided into a training set and a test set. On the one hand, the second infrared and second visible light image pairs after registration are input into the pre-trained perception network model to extract the domain difference information between the second visible light image and the second infrared image, and the domain difference information is input into the decoder network of the diffusion model. On the other hand, the second infrared and second visible light image pairs after registration are input into the encoder network of the diffusion model to realize the training of the diffusion model.

[0061] 122. Pass the second visible light image through a pre-trained perception network to obtain second domain difference information;

[0062] In this embodiment, after the second visible light image enters the perception network, the second domain difference information can be obtained. This domain difference information can reflect the domain difference information between the second visible light image and the second infrared image, so as to help the diffusion model to better generate the second infrared image.

[0063] 123. Add the first noise to the second infrared image according to the number of noise adding steps to obtain a second noisy infrared image;

[0064] In the embodiment of the present invention, a pair of the second infrared image and the second visible light image is randomly sampled from the training data, a time step is randomly sampled, and the time step is positively correlated with the noise intensity. Then, the obtained noise is fused with the infrared image to obtain an infrared image with noise. It can be expressed as:

[0065] Among them, I t represents the second noisy infrared image with the t-th step noise, I represents the randomly selected second infrared image, and ε is the initial noise.

[0066] 124. Pass the second noisy infrared image at each time step and the second visible light image as conditional information through an encoder network of a diffusion model, pass the second domain difference information as conditional information through a decoder network of a diffusion model, and predict the second noise at each time step;

[0067] In the embodiment of the present invention, the second noisy infrared image after adding noise and the second visible light image as a condition are sent to the diffusion model to learn the distribution of noise, and the pre-trained model is used as an input branch to fuse the feature map obtained by the infrared visible light image through the network. The training process is expressed by the formula: Among them, ε θ is the predicted noise, V is the second visible light image as a condition, t is the randomly sampled time step, and F is the second domain difference information output by the pre-trained network.

[0068] The process of predicting the second noise at each time step includes:

[0069] 131. Splice the second noisy infrared image and the second visible light image at each time step to obtain a second fused image;

[0070] 132. Input the second fused image and the number of noise addition steps into an encoder network of a diffusion model to extract a second encoding feature;

[0071] 133. Input the second domain difference information into a decoder network of a diffusion model, fuse it with the second encoding feature, and output the second noise at each time step.

[0072] Exemplarily, a noisy second noisy infrared image and its paired second visible light image are sent to a diffusion network for training, the second noisy infrared image and the second visible light image are spliced ​​into a six-channel image, and then fused with the feature vector after encoding at time step t, and then encoded and decoded, wherein the pre-trained model is also fused with the feature map in the model as a conditional branch, and the final output result is a predicted noise image, which is the second noise.

[0073] 125. Calculate the gradient loss according to the gradient difference between the first noise and the second noise at the same time step;

[0074] In the embodiment of the present invention, according to the predicted second noise ε θ The L2 loss is computed with the first noise ε generated at time step t.

[0075] For example, a loss function is used to evaluate the prediction noise ε θ The difference between and the actual noise ε. The most commonly used is the mean square error (MSE) loss:

[0076] Loss = L2(ε θ ,ε)

[0077] 126. Train the diffusion model based on minimizing the gradient loss.

[0078] In an embodiment of the present invention, stochastic gradient descent (SGD) or its variants (such as Adam optimizer) are used to minimize the above loss function. For each batch of data, the gradient of the loss relative to the model parameter θ is calculated, and the parameter is updated to reduce the loss. By minimizing the gradient loss, the diffusion model can gradually learn how to accurately denoise during the training process, thereby generating high-quality infrared images during testing. In this way, the model can learn the distribution of noise and can be used for noise removal in the inference stage.

[0079] In the embodiment of the present invention, during the test, the target visible light image is passed through the encoder network of the trained diffusion model, and the target noise is substituted for the x in the training process. T The target domain difference information is passed through the decoder network of the trained diffusion model, and a high-quality target infrared image is generated by a step-by-step denoising process, so that the generated result is close to the target infrared image of the real infrared image. It can be expressed as:

[0080]

[0081] Among them, z represents the target noise added in the denoising process, which can make the generated image diversified. It should be noted that noise is added only when t>1, otherwise z=0, I t-1 Represents an image with primary noise removed.

[0082] For example, Figure 5 As shown in the figure, the pre-trained perception network is used to extract domain difference information from the target visible light image, and the trained diffusion model is used to transform the target visible light image sent to the model. A target visible light image to be converted is sent in, and a target noise is randomly sampled. This target noise replaces x T The target visible light image, target domain difference information and target noise are sent to the diffusion model together. The model will continuously predict the noise, then remove the predicted noise from the randomly sampled noise, and then send it to the diffusion model again for noise removal. This operation is repeated until the image becomes a clean target infrared image x0.

[0083] In an embodiment of the present invention, the present invention further provides a device for converting a visible light image into an infrared image based on domain difference information, such as Figure 6 As shown, the device includes:

[0084] A first acquisition unit 201 is used to acquire a target visible light image and target noise; the target noise is random noise;

[0085] The first perception unit 202 is used to pass the target visible light image through a pre-trained perception network to obtain target domain difference information;

[0086] The first conversion unit 203 is used to pass the target visible light image through the encoder network of the trained diffusion model and pass the target domain difference information through the decoder network of the trained diffusion model to obtain the target infrared image.

[0087] The embodiment of the present invention obtains a target visible light image through a first acquisition unit 201; passes the target visible light image through a pre-trained perception network through a first perception unit 202 to obtain target domain difference information; passes the target visible light image through an encoder network of a trained diffusion model through a first conversion unit 203, and passes the target domain difference information through a decoder network of a trained diffusion model to obtain a target infrared image. The shallow domain difference information between the infrared image and the visible light image can be learned through the pre-trained perception network, and the diffusion model can control the generation of the infrared image based on the input conditional information and the output of the pre-trained perception network. The diffusion model continuously removes noise in the inference stage, thereby generating a clean target infrared image.

[0088] In the embodiment of the present invention, Figure 7 As shown, the device includes:

[0089] The second acquisition unit 211 is used to acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are a registered image pair;

[0090] The second sensing unit 212 is used to pass the first infrared image and the first visible light image through a sensing network to obtain a one-dimensional first infrared feature vector and a one-dimensional first visible light feature vector;

[0091] The first pre-training unit 213 is used to calculate a first loss by calculating the difference between the first infrared feature vector and the first visible light feature vector; and pre-train the perception network according to the first loss.

[0092] In the embodiment of the present invention, the first visible light image and the first infrared image are acquired by the second acquisition unit 211; the first visible light image and the first infrared image are a registered image pair; the first infrared image and the first visible light image are respectively passed through the perception network by the second perception unit 212 to obtain a one-dimensional first infrared feature vector and a one-dimensional first visible light feature vector; the difference between the first infrared feature vector and the first visible light feature vector is calculated by the first pre-training unit 213 to obtain a first loss; the perception network is pre-trained according to the first loss. This embodiment can realize the pre-training of the perception network so as to learn the shallow domain difference information between the infrared image and the visible light image.

[0093] In the embodiment of the present invention, Figure 8 As shown, the device includes:

[0094] A third acquisition unit 221 is used to acquire a second visible light image and a second infrared image; the second visible light image and the second infrared image are a registered image pair;

[0095] The third perception unit 222 is used to pass the second visible light image through a pre-trained perception network to obtain second domain difference information;

[0096] The second training unit 223 is used to gradually add the first noise to the second infrared image according to the number of noise adding steps to obtain a second noisy infrared image; pass the second noisy infrared image at each time step and the second visible light image as conditional information through the encoder network of the diffusion model, and pass the second domain difference information as conditional information through the decoder network of the diffusion model to predict the second noise at each time step; calculate the gradient loss according to the gradient difference between the first noise and the second noise at the same time step; and train the diffusion model by minimizing the gradient loss.

[0097] In the embodiment of the present invention, the second visible light image and the second infrared image are acquired by the third acquisition unit 221; the second visible light image and the second infrared image are a registered image pair; the second visible light image is passed through the pre-trained perception network by the third perception unit 222 to obtain the second domain difference information; the first noise is gradually added to the second infrared image according to the number of noise addition steps by the second training unit 223 to obtain the second noisy infrared image; the second noisy infrared image at each time step and the second visible light image as conditional information are passed through the encoder network of the diffusion model, and the second domain difference information as conditional information is passed through the decoder network of the diffusion model to predict the second noise at each time step; the gradient loss is calculated according to the gradient difference between the first noise and the second noise at the same time step; and the diffusion model is trained according to minimizing the gradient loss. This embodiment can realize the pre-training of the perception network so as to learn the shallow domain difference information between the infrared image and the visible light image. This embodiment can control the generation of the infrared image according to the input conditional information and the output of the pre-trained perception network, and the diffusion model continuously removes noise in the inference stage, thereby generating a clean target infrared image. The diffusion model is more stable than the generative adversarial network training, and the fusion of the pre-trained model and the diffusion model can make the image quality generated by the conversion between the infrared modality and the visible light modality higher.

[0098] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for converting a visible light image into an infrared image based on domain difference information, characterized in that: The following steps are involved: Acquire a target visible light image and target noise; the target noise is random noise; Passing the target visible light image through a pre-trained perception network to obtain target domain difference information; The target visible light image is passed through the encoder network of the trained diffusion model, and the target domain difference information is passed through the decoder network of the trained diffusion model to obtain a target infrared image.

2. The method for converting a visible light image into an infrared image based on domain difference information according to claim 1, characterized in that: The pre-training process of the perception network includes the following steps: Acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are a registered image pair of the same scene; Passing the first infrared image and the first visible light image through a perception network respectively to obtain a one-dimensional first infrared feature vector and a one-dimensional first visible light feature vector; By calculating the difference between the first infrared feature vector and the first visible light feature vector, a first loss is calculated; The perceptual network is pre-trained according to the first loss.

3. The method for converting a visible light image into an infrared image based on domain difference information according to claim 2, characterized in that: Passing the first infrared image and the first visible light image through a perception network respectively to obtain a one-dimensional first infrared feature vector and a one-dimensional first visible light feature vector comprises: Passing the first infrared image through a first pooling layer to obtain a one-dimensional first infrared feature vector; dividing the first visible light image into a plurality of first visible light image blocks; Each first visible light image block is respectively passed through a first reconstruction layer, a second pooling layer, a batch normalization layer, an activation function layer, a third pooling layer, and a second reconstruction layer to obtain a one-dimensional first visible light feature vector.

4. The method for converting a visible light image into an infrared image based on domain difference information according to claim 1, characterized in that: The training process of the diffusion model includes: Acquire a second visible light image and a second infrared image; the second visible light image and the second infrared image are a registered image pair of the same scene; Passing the second visible light image through a pre-trained perception network to obtain second domain difference information; Adding the first noise to the second infrared image according to the number of noise adding steps to obtain a second noisy infrared image; Passing the second noisy infrared image at each time step and the second visible light image as conditional information through an encoder network of a diffusion model, passing the second domain difference information as conditional information through a decoder network of a diffusion model, and predicting the second noise at each time step; The gradient loss is calculated based on the gradient difference between the first noise and the second noise at the same time step; The diffusion model is trained based on minimizing the gradient loss.

5. The method for converting a visible light image into an infrared image based on domain difference information according to claim 4, characterized in that: The process of predicting the second noise at each time step includes: splicing the second noisy infrared image and the second visible light image at each time step to obtain a second fused image; Inputting the second fused image and the number of noise addition steps into an encoder network of a diffusion model to extract a second encoding feature; The second domain difference information is input into the decoder network of the diffusion model, and after being fused with the second encoding feature, the second noise of each time step is output.

6. A device for converting a visible light image into an infrared image based on domain difference information, characterized in that: The device comprises: A first acquisition unit is used to acquire a target visible light image and target noise; the target noise is random noise; A first perception unit, configured to pass the target visible light image through a pre-trained perception network to obtain target domain difference information; The first conversion unit is used to pass the target visible light image through the encoder network of the trained diffusion model and pass the target domain difference information through the decoder network of the trained diffusion model to obtain the target infrared image.

7. The device for converting a visible light image into an infrared image based on domain difference information according to claim 6, characterized in that: Also includes: A second acquisition unit is used to acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are a registered image pair; A second perception unit is used to pass the first infrared image and the first visible light image through a perception network to obtain a one-dimensional first infrared feature vector and a one-dimensional first visible light feature vector; The first pre-training unit is used to calculate a first loss by calculating the difference between the first infrared feature vector and the first visible light feature vector; and pre-train the perception network according to the first loss.

8. The device for converting a visible light image into an infrared image based on domain difference information according to claim 6, characterized in that: Also includes: A third acquisition unit is used to acquire a second visible light image and a second infrared image; the second visible light image and the second infrared image are a registered image pair; a third perception unit, configured to pass the second visible light image through a pre-trained perception network to obtain second domain difference information; The second training unit is used to gradually add the first noise to the second infrared image according to the number of noise adding steps to obtain a second noisy infrared image; pass the second noisy infrared image at each time step and the second visible light image as conditional information through the encoder network of the diffusion model, and pass the second domain difference information as conditional information through the decoder network of the diffusion model to predict the second noise at each time step; calculate the gradient loss according to the gradient difference between the first noise and the second noise at the same time step; and train the diffusion model by minimizing the gradient loss.

Images