Method and apparatus for converting visible light images to infrared images based on feature fusion

Abstract

This invention relates to a method and apparatus for converting visible light images into infrared images based on feature fusion, belonging to the fields of image processing and computer vision technology. The method includes acquiring a target visible light image and a target noisy image; passing the target visible light image through a visible light precoding network to obtain a target visible light encoded image; passing the target visible light image through a pre-trained text generation model to obtain text description information of the target visible light image after preset constraints; passing the text description information of the target infrared image through a pre-trained text encoding model to obtain text encoding features of the target infrared image; fusing the target visible light encoded image, the text encoding features of the target infrared image, and the target noisy image through a trained diffusion model to obtain target infrared encoded features; and passing the target infrared encoded features through an infrared pre-decoding network to obtain the target infrared image. This invention can improve the image generation quality while avoiding the introduction of huge computational costs.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology, and relates to a method and device for converting visible light images into infrared images based on feature fusion. Background Technology

[0002] Image generation is a fundamental research problem in image processing. However, the high cost of acquiring infrared and visible light image pairs limits the widespread application of dual-modal infrared and visible light techniques. Conditional image generation methods can transform input images into new images with specific content by incorporating cue information. However, current image generation techniques still fall short in dual-modal infrared and visible light tasks, with generation quality and generalization ability needing improvement. Furthermore, as a low-visual-range task, computational complexity is a critical issue in image generation. Most existing methods directly transform the original image space, resulting in high computational complexity and difficulty in effectively reducing computational costs. Therefore, exploring more efficient generation methods has become an important research direction in this field.

[0003] Currently, most algorithms for generating dual-modal infrared and visible light images are based on GANs networks. However, GANs are often unstable during training, and the generated target images have low diversity, making it difficult to meet practical needs. In contrast, diffusion models learn the distribution of noise and gradually remove noise during the inference phase to generate target images, achieving higher generation quality. However, diffusion models require multiple iterations of denoising during inference, resulting in high time costs for inferring a single image. Especially with continuously increasing image resolution, the limitations of traditional CNNs in long-distance dependency modeling make it difficult to further improve image generation quality. Furthermore, to better control the generated target images, cue information needs to be involved in model training. Currently, most image generation tasks rely on text cueing, but text cueing has limited expressive power and cannot fully describe all the details of an image. While images can be used as cueing, the computational cost increases significantly with increasing image resolution. Therefore, how to improve image generation quality while avoiding introducing huge computational costs has become a key problem that urgently needs to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method and apparatus for converting visible light images to infrared images based on feature fusion. This invention utilizes a precoding network, which reduces computational costs; and by incorporating a multi-branch feature fusion model into the diffusion model, it improves the conversion quality of the image.

[0005] In a first aspect, the present invention provides a method for converting a visible light image into an infrared image based on feature fusion, the method comprising:

[0006] Acquire the visible light image and the noise image of the target;

[0007] The target visible light image is passed through a visible light precoding network to obtain a target visible light encoded image;

[0008] The target visible light image is processed by a pre-trained text generation model to obtain text description information of the target visible light image after pre-defined constraints; the text description information of the target visible light image after pre-defined constraints is used to indicate the text description information of the target infrared image;

[0009] The text description information of the target infrared image is used to obtain the text encoding features of the target infrared image through a pre-trained text encoding model;

[0010] The text encoding features of the target visible light encoded image, the target infrared image, and the target noisy image are fused using a trained diffusion model to obtain the target infrared encoded features;

[0011] The infrared encoded features of the target are passed through an infrared pre-decoding network to obtain the infrared image of the target.

[0012] In a second aspect, the present invention also provides an apparatus for converting a visible light image into an infrared image based on feature fusion, the apparatus comprising:

[0013] The first acquisition unit is used to acquire a visible light image of the target and a noise image of the target;

[0014] The first precoding unit is used to pass the target visible light image through a visible light precoding network to obtain a target visible light encoded image;

[0015] The first text extraction unit is used to extract text description information of the target visible light image after passing it through a pre-trained text generation model; the text description information of the target visible light image after passing it through the preset constraints is used to indicate the text description information of the target infrared image.

[0016] The first text conversion unit is used to convert the text description information of the target infrared image into text encoding features of the target infrared image through a pre-trained text encoding model.

[0017] The first encoding unit is used to fuse the text encoding features of the target visible light encoded image, the target infrared image, and the target noise image through a trained diffusion model to obtain the target infrared encoded features.

[0018] The first pre-decoding unit is used to pass the infrared encoded features of the target through an infrared pre-decoding network to obtain the infrared image of the target.

[0019] Furthermore, the device for converting visible light images to infrared images based on feature fusion also includes:

[0020] The second acquisition unit is used to acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are image pairs acquired at the same time and in the same scene;

[0021] The first precoding unit is further configured to generate a first visible light coded image by passing the first visible light image through a pre-trained visible light coding network;

[0022] The second precoding unit is used to generate a first infrared coded image by passing the first infrared image through a pre-trained infrared coding network.

[0023] The second text extraction unit is used to extract text description information of the first visible light image after passing it through a pre-trained text generation model; the text description information of the first visible light image after passing it through the preset constraints is used to indicate the text description information of the first infrared image.

[0024] The second text conversion unit is used to convert the text description information of the first infrared image into the first infrared text encoding feature through a pre-trained text encoding model.

[0025] The second encoding unit is used to generate a noisy infrared image by passing the first infrared encoded image and the first noise generated at the time step through a diffusion model; and to perform feature fusion of the noisy infrared image, the first infrared text encoded features, and the first visible light encoded image through a diffusion model to predict the second noise.

[0026] The first training unit is used to calculate the gradient loss function of the diffusion model based on the gradient of the first noise generated at the time step and the gradient of the model predicting the second noise; and to update the diffusion model based on the gradient loss function.

[0027] The beneficial effects of this invention are as follows: Before training the diffusion model, this embodiment first requires pre-training an encoder-decoder network. This network can map images to the latent space and map the target image in the latent space back to the original space. After training the encoder-decoder network, the visible light image and the infrared image are respectively fed into the encoder network for encoding. After encoding, the infrared image is denoised, and the text prompt is encoded by a text encoder. The generated feature vector, together with the visible light prompt and the denoised infrared image, participates in training through feature fusion to learn the noise distribution. The latent space representation of the target image is obtained through repeated iterations. The latent space representation of the target image is decoded by the decoder and mapped back to the original space to obtain the final generated target image. This method can reduce computational costs and significantly improve the image conversion quality. Attached Figure Description

[0028] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following figures are provided for illustration:

[0029] Figure 1 This is a schematic flowchart of a method for converting a visible light image into an infrared image based on feature fusion, according to an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of the diffusion model structure according to an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the feature extraction module according to an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the neighborhood self-attention module according to an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of a device for converting visible light images into infrared images based on feature fusion, according to an embodiment of the present invention.

[0034] Figure 6 This is a schematic diagram of a device for converting a visible light image into an infrared image based on feature fusion, which is a preferred embodiment of the present invention. Detailed Implementation

[0035] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0036] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0037] In this application embodiment, a method and device for converting visible light images into infrared images based on feature fusion are proposed. The visible light images and infrared images are images of the same target scene. Visible light images acquired by front-end devices such as network cameras and analog cameras can be converted into infrared images, and visible light images stored by back-end devices such as servers, management devices, and storage devices can also be converted into infrared images.

[0038] The method and apparatus for converting visible light images to infrared images based on feature fusion according to the present invention will be further described below with reference to the accompanying drawings.

[0039] See Figure 1 The diagram shown is a flowchart illustrating a method for converting a visible light image into an infrared image based on feature fusion. The method may include:

[0040] 101. Acquire the visible light image and the noise image of the target;

[0041] In this embodiment of the invention, the target visible light image can be acquired by front-end devices such as network cameras, analog cameras, and other optical imaging devices, or by back-end devices such as servers, management devices, and storage devices. For example, the target noise image is random noise following a specific distribution, such as a random noise image following a Gaussian distribution (normal distribution). In this embodiment of the invention, the position of the noisy infrared image in the diffusion model is replaced by the random noise image, and the target infrared image is obtained by gradually removing the target noise image.

[0042] 102. Pass the target visible light image through a visible light precoding network to obtain a target visible light encoded image;

[0043] Currently, for low-vision tasks, computational cost is typically proportional to the square of the image resolution. With technological advancements, image resolution continues to increase, leading to an exponential increase in computational cost for low-vision tasks. Higher resolutions not only increase computational cost but also render traditional convolutional neural networks (CNNs) increasingly inadequate for long-range dependent perception. To address this challenge, more powerful feature extraction networks are needed, capable of efficiently extracting effective features from large images while maintaining model performance and reducing computational costs. Furthermore, as resolution increases, reducing computational cost while ensuring generation quality becomes a key issue in improving the efficiency of low-vision tasks.

[0044] Based on this, embodiments of the present invention employ pre-coding and pre-decoding networks. These encoding / decoding networks map infrared or visible light images to the latent space and decode them back to the original space. For example, a target visible light image is input into a visible light pre-coding network. The network first encodes the target visible light image into a 64×64 resolution feature representation, and then inputs the encoded feature representation into a visible light decoding network for decoding, ultimately obtaining an image with a resolution of 256×256. By calculating the loss between the original image and the decoded image, the encoding / decoding network is optimized and pre-trained to ensure it can efficiently recover the original details of the image. Similarly, infrared images also require training an encoding / decoding network with the same structure to ensure that the infrared image can be efficiently mapped between the latent space and the original space.

[0045] The formulas involved in the pre-training process can be expressed as follows:

[0046] z = ε(x)

[0047]

[0048] Where, x∈R H×W×C H and W are the height and width of the image, and C is the number of channels. The encoder ε maps the input image x to the latent space z∈R. h×w×c h and w are the resolutions of the encoded image, and c is the number of channels in the latent space. The decoder D maps the latent space features z back to the original space to obtain the decoded image. During training, the goal is to minimize the loss between the original image and the decoded image, using mean squared error as the loss function, where N is the total number of pixels in the image, and x... i , These are the i-th pixel values ​​of the original image and the decoded image, respectively. During network training, the parameters of the encoder and decoder are updated by minimizing the loss function.

[0049] It should be noted that, in this embodiment of the invention, since it is only necessary to convert the visible light image into an infrared image, the visible light pre-decoding network is omitted.

[0050] 103. The target visible light image is processed through a pre-trained text generation model to obtain text description information of the target visible light image after pre-defined constraints; the text description information of the target visible light image after pre-defined constraints is used to indicate the text description information of the target infrared image;

[0051] In this embodiment of the invention, considering the significant differences in the imaging principles of visible light and infrared images, and that infrared images cannot present visual information such as color, the content of the generated text is strictly constrained, prohibiting the inclusion of information unrelated to infrared images, such as descriptions of color and illumination, to ensure that the generated text only describes the core content and structural information of the image.

[0052] In this embodiment of the invention, step 103 may specifically include:

[0053] 131. The target visible light image is processed through a pre-trained BLIP (Bootstrapping Language-Image Pre-training) text generation model to obtain text description information of the target visible light image; the text description information of the target visible light image includes color information, time information, object information, shape information, position information, and descriptions of other physical attributes;

[0054] 132. By adding a text constraint prefix and cleaning the generated text data, the text description information of the target visible light image is filtered out, and the color information and the time information are filtered out to obtain the text description information of the target visible light image after preset constraints.

[0055] For example, taking an indoor scene as an example, suppose the text description information of the target visible light image shows an indoor scene. In the center of the image, there is a rectangular table with some items on it. To the left of the table is a white ceramic cup, next to which is an open laptop with some text and numbers displayed on the screen. In order to make the text description information of the target visible light image indicative of the text description information of the target infrared image, this embodiment uses preset constraints to remove information about color, time, etc. from the text description information of the target visible light image. That is, the color of the table, the color of the cup, etc., are removed. After removing this information, the text description information of the target visible light image can focus on the image content itself, thus more accurately reflecting the text description information of the target infrared image.

[0056] 104. The text description information of the target infrared image is processed through a pre-trained text encoding model to obtain the text encoding features of the target infrared image;

[0057] In this embodiment of the invention, the text description information of the target infrared image is input into a pre-trained text encoder for encoding to generate text-encoded features describing the image content. These text-encoded features can be input into the diffusion model along with the latent space features extracted by the pre-encoding network. The generation of the image is constrained by feature fusion and attention encoding, which avoids introducing a large amount of computational overhead while improving the quality of image generation.

[0058] 105. The text encoding features of the target visible light encoded image, the target infrared image, and the target noise image are fused using a trained diffusion model to obtain the target infrared encoded features;

[0059] In this embodiment of the invention, for the diffusion model, such as Figure 2 As shown, on the one hand, an infrared image is input, and a pre-trained infrared encoder is used to map the state of the original infrared image and its various noise stages into a low-dimensional latent space. Similarly, a visible light image is input, and a pre-trained visible light encoder is used to map the state of the original visible light image into a low-dimensional latent space. This approach reduces computational cost because the diffusion model directly operates on lower-dimensional data. On the other hand, the visible light image is processed through a text encoding model and a text encoder to generate text-encoded features. These features are then used for back-reasoning to obtain a denoised latent representation, thereby generating the image.

[0060] Based on the above diffusion model, in this embodiment of the invention, the target visible light encoded image is used as the image condition, the text encoding features of the target infrared image are used as the text condition, and the target noisy image is used as the starting point of the generation process. The target infrared image that meets specific requirements is generated by guiding the generation of the above multimodal condition information. In this process, the diffusion model performs denoising operation on the input target noisy image to generate a potential representation that meets the multimodal constraints, which is the target infrared encoded feature. The target infrared encoded feature contains the semantic and structural information of the target infrared image.

[0061] In this embodiment of the invention, the training process of the diffusion model includes the following steps:

[0062] 151. Acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are image pairs acquired at the same time and in the same scene;

[0063] In this embodiment of the invention, for example, the front-end device can be a dual-sensor device, where the dual sensors refer to a visible light sensor and an infrared light sensor. The visible light sensor acquires the intensity of visible light from an object, and the image acquired by the visible light sensor is called a visible light image. The infrared light sensor acquires the intensity of infrared light from an object, and the image acquired by the infrared light sensor is called an infrared light image. Based on this, a first visible light image and a first infrared image can be acquired at the same acquisition time using the dual sensors of the front-end device.

[0064] In this embodiment of the invention, the first infrared image and the first visible light image are first registered, and these datasets are divided into training set and test set, which contain image pairs of various scenes and lighting conditions; then the infrared image and the visible light image are fed into a pre-trained network to learn the domain differences between the infrared image and the visible light image; the dataset used in this embodiment can contain various scenes such as campus, street, forest and other complex scenes, and the resolution of these images includes various resolutions such as 512×512, 1024×1024 and so on.

[0065] First, the first infrared image and the first visible light image are registered. After registration, the image pairs are cropped to a resolution of 256×256. The first visible light image and the first infrared image are labeled. Then, the dataset is randomly divided into a training set and a test set.

[0066] 152. The first visible light image and the first infrared image are respectively processed through their respective visible light precoding networks and infrared precoding networks to generate a first visible light coded image and a first infrared coded image;

[0067] In this embodiment of the invention, the registered first visible light image and the first infrared image can be mapped through their respective precoding networks. The visible light precoding network maps the first visible light image to a coding space, which is a low-dimensional, compact latent space representation. Similarly, the infrared precoding network maps the first infrared image to a coding space, which is also a low-dimensional, compact latent space representation.

[0068] For example, suppose the first visible light image and the first infrared image are 256×256 resolution images. After passing through their respective precoding networks, a 64×64 resolution first visible light coded image and first infrared coded image can be obtained. This compresses the size of the original image and reduces the computational cost of subsequent processes.

[0069] 153. The first visible light image is processed by a pre-trained text generation model to generate text description information of the first visible light image after pre-defined constraints; the text description information of the first visible light image after pre-defined constraints is used to indicate the text description information of the first infrared image.

[0070] In this embodiment of the invention, similar to steps 131-132, the embodiment of the invention can generate text description information of the first visible light image. The text description information of the first visible light image can serve as prompt information for the first infrared image, so as to generate the first infrared image in subsequent processes.

[0071] 154. The text description information of the first infrared image is passed through the corresponding pre-trained text coding network to generate the first infrared text coding feature;

[0072] 155. The first infrared encoded image and the first noise generated at the time step are used to generate a noisy infrared image through a diffusion model;

[0073] In this embodiment of the invention, a first infrared and a first visible light image pair are randomly sampled from the training data. A time step is randomly sampled, and the time step is positively correlated with the noise intensity. Then, the obtained first noise is fused with the first infrared image to obtain a noisy infrared image. This can be expressed by the formula:

[0074]

[0075] Among them, I t Let represent a noisy infrared image with noise at step t, where I represents the first randomly selected infrared image, and ε is the initial noise.

[0076] In this process, the infrared image can be diffused and denoised; then the denoised infrared image and the visible light image are fed into the diffusion model to learn the distribution of noise.

[0077] 156. The noisy infrared image, the first infrared text-coded feature, and the first visible light-coded image are fused using a diffusion model to predict the second noise.

[0078] In this embodiment of the invention, the process of predicting the second noise may include:

[0079] 251. The noisy infrared image generated according to the time step is stitched together with the first visible light coded image to obtain a first fused image;

[0080] 252. Input the first fused image and the number of noise addition steps into the four-branch encoder network of the diffusion model to extract the original features, local features, regional features and global features;

[0081] In this embodiment of the invention, in order to extract multi-scale information of the image, such as... Figure 3 As shown in the figure, this embodiment of the invention proposes a feature extraction module that can extract image features at three granularities: local, regional, and global, to obtain richer information. The four-branch encoder network includes an original information connection layer, a local information encoding layer, a regional information encoding layer, and a global information encoding layer. The original information connection layer uses skip connections to preserve original feature information and prevent gradient explosion or gradient vanishing. The local information encoding layer consists of two convolutional layers and a channel attention module to extract local information. The regional information encoding layer includes a window self-attention module to extract regional information. The global information encoder includes a neighborhood self-attention module to extract global information.

[0082] Specifically, compared to the ResNet Block used in traditional UNet, the feature extraction module of this embodiment performs better in long-range perception, overcoming the shortcomings of residual blocks. The feature extraction module comprises four branches: the first branch consists of two convolutional layers and a channel attention module, used to extract local information; the second branch uses skip connections to preserve original feature information and prevent gradient explosion or vanishing; the third branch extracts region information through a window self-attention mechanism; and the fourth branch extracts global information through a neighborhood self-attention mechanism. Finally, the feature information from the four branches is added together to integrate rich multi-scale features, thereby significantly improving the ability to generate high-quality images.

[0083] In embodiments of the present invention, such as Figure 4 As shown, the neighborhood self-attention module includes:

[0084] The first fused image and the feature vector after the noise-adding step encoding are input into the global information encoder of the diffusion model to extract the feature information of the target image block and adjacent image blocks.

[0085] The feature information of the target image block is transformed by Q linear transformation to obtain the first Q vector;

[0086] The feature information of the target image block is transformed by K linear transformation to obtain the first K vector;

[0087] The feature information of the adjacent image blocks is transformed by V linear transformation to obtain the first V vector;

[0088] The first Q vector and the first K vector are multiplied by a dot product to obtain the first QK vector;

[0089] The first QK vector and position encoding are processed to obtain the second QK vector;

[0090] The second QK vector and the first V vector are weighted to obtain the global information of the target image patch.

[0091] In this way, the model assigns different importance or preferences to elements at different coding positions in the sequence when processing sequential data. This preference enhances the model's ability to perceive positional information in the sequence data. This allows for the effective fusion of features within the neighborhood, thereby increasing the model's ability to perceive long distances and improving the quality of image generation in subsequent processes.

[0092] 253. The original features, the local features, the regional features, and the global features are fused to obtain the first encoded features;

[0093] In this embodiment of the invention, the feature information of the four branches is added together to integrate the first coding feature of multiple scales, thereby significantly improving the ability to generate high-quality images.

[0094] 254. Input the first infrared text encoding feature into the decoder network of the diffusion model, fuse it with the first encoding feature, and output the predicted second noise.

[0095] According to steps 251-254 of the present invention, the noise output by the model can be effectively predicted, and the diffusion model can be optimized by utilizing the difference between the input noise and the output noise.

[0096] 157. Based on the gradient of the first noise generated at the time step and the gradient of the second noise predicted by the model, the gradient loss function of the diffusion model is calculated.

[0097] In this embodiment of the invention, the training process is expressed by the following formula: in For the predicted noise, V ε F represents the features of the visible light image after encoding, which serves as a condition, where t is the time step of random sampling, and F... text The feature vector generated after the text prompt is processed by a pre-trained encoder.

[0098] 158. Update the diffusion model according to the gradient loss function.

[0099] It should be noted that through steps 151-158 of the embodiments of the present invention, the diffusion model can be trained so that the trained diffusion model can accurately generate the infrared image of the target based on the visible light image and the noise image of the target.

[0100] 106. The infrared encoded features of the target are passed through an infrared pre-decoding network to obtain an infrared image of the target.

[0101] In this embodiment of the invention, the target infrared coded features are converted into the final target infrared image through an infrared pre-decoding network, so that the target infrared coded features can be returned from the latent space to the image space.

[0102] Understandably, in this embodiment of the invention, through steps 101-106, a target visible light image is randomly acquired and input into a pre-trained visible light precoding network for encoding, extracting the feature representation in the latent space, which is the target visible light encoded image. Simultaneously, Gaussian noise is randomly sampled as the target noise image, and this target noise image and the target visible light encoded image are used as the initial input to the diffusion model. Furthermore, the textual description information of the target infrared image is encoded using a pre-trained text encoding model to generate a feature vector, which is then used as textual conditional information and fused with the target visible light encoded image, which serves as image conditional information. The fusion process is implemented through a cross-attention mechanism to ensure that these two conditional prompts effectively guide the generation process. The output of the diffusion model is the predicted noise distribution at the current step. Next, the predicted noise is removed from the initial input noise to obtain a denoised intermediate result. This intermediate result is then input back into the diffusion model to continue the denoising step. Through multiple iterations of denoising, image features are gradually restored until a clean latent space representation of the target infrared image, i.e., the target infrared encoded features, is finally recovered. This feature is then passed through an infrared precoding network to obtain the target infrared image.

[0103] In this embodiment of the invention, the invention also provides a device for converting visible light images into infrared images based on feature fusion, such as... Figure 5 As shown, the device includes:

[0104] The first acquisition unit 201 is used to acquire a visible light image of the target and a noise image of the target;

[0105] The first precoding unit 202 is used to pass the target visible light image through a visible light precoding network to obtain a target visible light encoded image;

[0106] The first text extraction unit 203 is used to extract text description information of the target visible light image after passing it through a pre-trained text generation model; the text description information of the target visible light image after passing it through the preset constraints is used to indicate the text description information of the target infrared image.

[0107] The first text conversion unit 204 is used to convert the text description information of the target infrared image into text encoding features of the target infrared image through a pre-trained text encoding model.

[0108] The first encoding unit 205 is used to fuse the text encoding features of the target visible light encoded image, the target infrared image, and the target noise image through a trained diffusion model to obtain the target infrared encoded features.

[0109] The first pre-decoding unit 206 is used to pass the infrared encoded features of the target through an infrared pre-decoding network to obtain an infrared image of the target.

[0110] In this embodiment of the invention, a first acquisition unit 201 acquires a target visible light image and a target noise image; a first precoding unit 202 passes the target visible light image through a visible light precoding network to obtain a target visible light encoded image; a first text extraction unit 203 passes the target visible light image through a pre-trained text generation model to obtain text description information of the target visible light image after preset constraints; the text description information of the target visible light image after preset constraints is used to indicate the text description information of the target infrared image; a first text conversion unit 204 passes the text description information of the target infrared image through a pre-trained text encoding model to obtain the text encoding features of the target infrared image; a first encoding unit 205 fuses the target visible light encoded image, the text encoding features of the target infrared image, and the target noise image through a trained diffusion model to obtain the target infrared encoded features; and a first pre-decoding unit 206 passes the target infrared encoded features through an infrared pre-decoding network to obtain the target infrared image. This reduces computational costs and significantly improves the conversion quality of visible light to infrared images.

[0111] In embodiments of the present invention, such as Figure 6 As shown, the device may further include:

[0112] The second acquisition unit 211 is used to acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are image pairs acquired at the same time and in the same scene;

[0113] The first precoding unit 202 is further configured to generate a first visible light coded image by passing the first visible light image through a pre-trained visible light coding network;

[0114] The second precoding unit 212 is used to generate a first infrared coded image by passing the first infrared image through a pre-trained infrared coding network.

[0115] The second text extraction unit 213 is used to extract the first visible light image through a pre-trained text generation model to obtain text description information of the first visible light image after preset constraints; the text description information of the first visible light image after preset constraints is used to indicate the text description information of the first infrared image.

[0116] The second text conversion unit 214 is used to convert the text description information of the first infrared image into the first infrared text encoding feature through a pre-trained text encoding model.

[0117] The second encoding unit 215 is used to generate a noisy infrared image by passing the first infrared encoded image and the first noise generated at the time step through a diffusion model; and to perform feature fusion of the noisy infrared image, the first infrared text encoded features, and the first visible light encoded image through a diffusion model to predict the second noise.

[0118] The first training unit 216 is used to calculate the gradient loss function of the diffusion model based on the gradient of the first noise generated at the time step and the gradient of the model predicting the second noise; and to update the diffusion model based on the gradient loss function.

[0119] In this embodiment of the invention, a first visible light image and a first infrared image are acquired by a second acquisition unit 211; the first visible light image and the first infrared image are image pairs acquired at the same time and in the same scene; the first precoding unit 202 is further configured to generate a first visible light coded image by passing the first visible light image through a pre-trained visible light coding network; the second precoding unit 212 passes the first infrared image through a pre-trained infrared coding network to generate a first infrared coded image; the second text extraction unit 213 passes the first visible light image through a pre-trained text generation model to obtain text description information of the first visible light image after preset constraints; the text description information of the first visible light image after preset constraints... The system uses textual description information to indicate the first infrared image. A second text conversion unit 214 processes the textual description information of the first infrared image through a pre-trained text encoding model to obtain first infrared textual encoding features. A second encoding unit 215 processes the first infrared encoded image and the first noise generated at the time step through a diffusion model to generate a noisy infrared image. The noisy infrared image, the first infrared textual encoding features, and the first visible light encoded image are fused using the diffusion model to predict the second noise. A first training unit 216 calculates the gradient loss function of the diffusion model based on the gradient of the first noise generated at the time step and the model's predicted second noise. The diffusion model is then updated based on the gradient loss function. This embodiment enables effective training of the diffusion model, allowing the trained diffusion model to generate the target infrared image.

[0120] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0121] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for converting visible light images into infrared images based on feature fusion, characterized in that, Includes the following steps: Acquire a target visible light image and a target noise image, wherein the target noise image includes a noise image with added random noise; pass the target visible light image through a visible light precoding network to obtain a target visible light encoded image; The target visible light image is processed by a pre-trained text generation model to obtain text description information of the target visible light image after pre-defined constraints; the text description information of the target visible light image after pre-defined constraints is used to indicate the text description information of the target infrared image; The target visible light image is processed through a pre-trained text generation model to obtain textual description information of the target visible light image after pre-defined constraints, including: The target visible light image is processed through a pre-trained BLIP text generation model to obtain textual description information of the target visible light image; the textual description information of the target visible light image includes color information, time information, object information, shape information, position information, and descriptions of other physical attributes; By adding a text constraint prefix and cleaning the generated text data, the text description information of the target visible light image is filtered out, removing the color information and the time information, to obtain the text description information of the target visible light image after preset constraints. The text description information of the target infrared image is used to obtain the text encoding features of the target infrared image through a pre-trained text encoding model; The text encoding features of the target visible light encoded image, the target infrared image, and the target noisy image are fused using a trained diffusion model to obtain the target infrared encoded features; The infrared encoded features of the target are passed through an infrared pre-decoding network to obtain the infrared image of the target.

2. The method for converting visible light images to infrared images based on feature fusion according to claim 1, characterized in that, The training process of the diffusion model includes the following steps: Acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are image pairs acquired at the same time and in the same scene; The first visible light image and the first infrared image are respectively processed through their respective visible light precoding networks and infrared precoding networks to generate a first visible light coded image and a first infrared coded image; The first visible light image is processed by a pre-trained text generation model to generate text description information of the first visible light image after pre-defined constraints; the text description information of the first visible light image after pre-defined constraints is used to indicate the text description information of the first infrared image. The text description information of the first infrared image is used to generate the first infrared text encoding features through a corresponding pre-trained text encoding network; The first infrared encoded image and the first noise generated at the time step are used to generate a noisy infrared image through a diffusion model; The noisy infrared image, the first infrared text-coded feature, and the first visible light-coded image are fused using a diffusion model to predict the second noise. The gradient loss function of the diffusion model is calculated based on the gradient of the first noise generated at the time step and the gradient of the second noise predicted by the model. The diffusion model is updated based on the gradient loss function.

3. The method for converting visible light images to infrared images based on feature fusion according to claim 2, characterized in that, The step of fusing the noisy infrared image, the first infrared text-coded feature, and the first visible light-coded image using a diffusion model to predict the second noise includes the following steps: The noisy infrared image generated according to the time step is stitched together with the first visible light coded image to obtain the first fused image; The first fused image and the number of noise-adding steps are input into the four-branch encoder network of the diffusion model to extract the original features, local features, regional features and global features; The original features, the local features, the regional features, and the global features are fused to obtain the first encoded features; The first infrared text encoding feature is input into the decoder network of the diffusion model, and after being fused with the first encoding feature, the predicted second noise is output.

4. The method for converting visible light images to infrared images based on feature fusion according to claim 3, characterized in that, The four-branch encoder network includes a raw information connection layer, a local information encoding layer, a region information encoding layer, and a global information encoding layer. The raw information connection layer uses skip connections to preserve raw feature information and prevent gradient explosion or gradient vanishing. The local information encoding layer consists of two convolutional layers and a channel attention module to extract local information. The region information encoding layer includes a window self-attention module to extract region information. The global information encoder includes a neighborhood self-attention module to extract global information.

5. The method for converting visible light images to infrared images based on feature fusion according to claim 4, characterized in that, The neighborhood self-attention module includes: The first fused image and the feature vector after the noise-adding step encoding are input into the global information encoder of the diffusion model to extract the feature information of the target image block and adjacent image blocks. The feature information of the target image block is transformed by Q linear transformation to obtain the first Q vector; The feature information of the target image block is transformed by K linear transformation to obtain the first K vector; The feature information of the adjacent image blocks is transformed by V linear transformation to obtain the first V vector; The first Q vector and the first K vector are multiplied by a dot product to obtain the first QK vector; The first QK vector and position encoding are processed to obtain the second QK vector; The second QK vector and the first V vector are weighted to obtain the global information of the target image patch.

6. A device for converting a visible light image into an infrared image based on feature fusion, characterized in that, The device includes: The first acquisition unit is used to acquire a visible light image of the target and a noise image of the target, wherein the noise image of the target includes a noise image with random noise added; The first precoding unit is used to pass the target visible light image through a visible light precoding network to obtain a target visible light encoded image; The first text extraction unit is used to extract text description information of the target visible light image after passing it through a pre-trained text generation model; the text description information of the target visible light image after passing it through the preset constraints is used to indicate the text description information of the target infrared image. The target visible light image is processed through a pre-trained text generation model to obtain textual description information of the target visible light image after pre-defined constraints, including: The target visible light image is processed through a pre-trained BLIP text generation model to obtain textual description information of the target visible light image; the textual description information of the target visible light image includes color information, time information, object information, shape information, position information, and descriptions of other physical attributes; By adding a text constraint prefix and cleaning the generated text data, the text description information of the target visible light image is filtered out, removing the color information and the time information, to obtain the text description information of the target visible light image after preset constraints. The first text conversion unit is used to convert the text description information of the target infrared image into text encoding features of the target infrared image through a pre-trained text encoding model. The first encoding unit is used to fuse the text encoding features of the target visible light encoded image, the target infrared image, and the target noise image through a trained diffusion model to obtain the target infrared encoded features. The first pre-decoding unit is used to pass the infrared encoded features of the target through an infrared pre-decoding network to obtain the infrared image of the target.

7. The device for converting visible light images to infrared images based on feature fusion according to claim 6, characterized in that, Also includes: The second acquisition unit is used to acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are image pairs acquired at the same time and in the same scene; The first precoding unit is further configured to generate a first visible light coded image by passing the first visible light image through a pre-trained visible light coding network; The second precoding unit is used to generate a first infrared coded image by passing the first infrared image through a pre-trained infrared coding network. The second text extraction unit is used to extract text description information of the first visible light image after passing it through a pre-trained text generation model; the text description information of the first visible light image after passing it through the preset constraints is used to indicate the text description information of the first infrared image. The second text conversion unit is used to convert the text description information of the first infrared image into the first infrared text encoding feature through a pre-trained text encoding model. The second encoding unit is used to generate a noisy infrared image by passing the first infrared encoded image and the first noise generated at the time step through a diffusion model; and to perform feature fusion of the noisy infrared image, the first infrared text encoded features, and the first visible light encoded image through a diffusion model to predict the second noise. The first training unit is used to calculate the gradient loss function of the diffusion model based on the gradient of the first noise generated at the time step and the gradient of the model predicting the second noise; and to update the diffusion model based on the gradient loss function.

Images