Method and device for converting visible light image into infrared image based on feature fusion

Abstract

The invention relates to a method and equipment for converting a visible light image into an infrared image based on feature fusion, and belongs to the technical field of image processing and computer vision. The method comprises the following steps: acquiring a target visible light image and a target noise image; enabling the target visible light image to pass through a visible light pre-coding network to obtain a target visible light coding image; the target visible light image passes through a pre-trained text generation model, and text description information of the target visible light image after preset constraint is obtained; processing the text description information of the target infrared image through a pre-trained text coding model to obtain text coding features of the target infrared image; performing feature fusion on the target visible light coding image, the text coding features of the target infrared image and the target noise image through a trained diffusion model to obtain a target infrared coding feature; and enabling the target infrared coding feature to pass through an infrared pre-decoding network to obtain a target infrared image. According to the method, the image generation quality can be improved, and meanwhile, the introduction of huge calculation cost is avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and computer vision, and relates to a method and device for converting a visible light image into an infrared image based on feature fusion. Background Art

[0002] The image generation problem is a basic research problem in image processing. However, due to the high cost of acquiring infrared and visible light image pairs, this has limited the widespread application of infrared and visible light dual-modal technology to a certain extent. Conditional image generation methods can convert input images into new images with specific content by combining prompt information. However, the current image generation technology still performs poorly in infrared and visible light dual-modal tasks, and the generation quality and generalization ability need to be improved. At the same time, as a low-level visual task, image generation is a key issue that cannot be ignored in terms of computational complexity. Most existing methods directly transform in the original image space, which has high computational complexity and is difficult to effectively reduce the computational cost. Therefore, exploring more efficient generation methods has become an important research direction in this field.

[0003] At present, most algorithms for infrared and visible light dual-modal image generation are based on GANs networks, but GANs are often unstable during training, and the diversity of generated target images is low, which is difficult to meet actual needs. In contrast, the diffusion model has higher generation quality by learning the distribution of noise and gradually removing noise in the inference stage to generate target images. However, the diffusion model inference process requires multiple iterations of denoising, and the time cost of inferring an image is high. Especially when the image resolution continues to increase, the limitations of traditional CNN in long-distance dependency modeling make it difficult to further improve the image generation quality. In addition, in order to better control the generated target image, prompt information needs to participate in model training. At present, most image generation tasks rely on text prompts, but the expressive power of text prompts is limited and it is difficult to fully describe all the detailed information of the image. Although images can be used as prompts, the amount of calculation will increase significantly as the resolution of the prompt image increases. Therefore, how to improve the image generation quality while avoiding the introduction of huge computational costs has become a key issue to be solved in this field. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method and device for converting visible light images into infrared images based on feature fusion. The present invention uses a precoding network to reduce the computational cost; by adding a multi-branch feature fusion model to the diffusion model, the image conversion quality can be improved.

[0005] In a first aspect of the present invention, the present invention proposes a method for converting a visible light image into an infrared image based on feature fusion, the method comprising:

[0006] Acquire a target visible light image and a target noise image;

[0007] Passing the target visible light image through a visible light precoding network to obtain a target visible light coded image;

[0008] The target visible light image is passed through a pre-trained text generation model to obtain text description information of the target visible light image after a preset constraint; the text description information of the target visible light image after a preset constraint is used to indicate the text description information of the target infrared image;

[0009] Passing the text description information of the target infrared image through a pre-trained text encoding model to obtain text encoding features of the target infrared image;

[0010] The target visible light coded image, the text coded features of the target infrared image and the target noise image are subjected to feature fusion through the trained diffusion model to obtain the target infrared coded features;

[0011] The infrared coding features of the target are passed through an infrared pre-decoding network to obtain an infrared image of the target.

[0012] In a second aspect of the present invention, the present invention further proposes a device for converting a visible light image into an infrared image based on feature fusion, the device comprising:

[0013] A first acquisition unit, used for acquiring a target visible light image and a target noise image;

[0014] A first precoding unit, configured to pass the target visible light image through a visible light precoding network to obtain a target visible light coded image;

[0015] A first text extraction unit is used to pass the target visible light image through a pre-trained text generation model to obtain text description information of the target visible light image after a preset constraint; the text description information of the target visible light image after the preset constraint is used to indicate the text description information of the target infrared image;

[0016] A first text conversion unit, configured to obtain text encoding features of the target infrared image by passing the text description information of the target infrared image through a pre-trained text encoding model;

[0017] A first encoding unit is used to perform feature fusion on the target visible light coded image, the text coded features of the target infrared image, and the target noise image through a trained diffusion model to obtain a target infrared coded feature;

[0018] The first pre-decoding unit is used to obtain the target infrared image by passing the target infrared coding feature through an infrared pre-decoding network.

[0019] Furthermore, the device for converting a visible light image into an infrared image based on feature fusion further comprises:

[0020] A second acquisition unit is used to acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are an image pair of the same scene acquired at the same time;

[0021] The first pre-coding unit is further used to generate a first visible light coded image by passing the first visible light image through a pre-trained visible light coding network;

[0022] A second pre-coding unit, configured to pass the first infrared image through a pre-trained infrared coding network to generate a first infrared coded image;

[0023] A second text extraction unit is used to pass the first visible light image through a pre-trained text generation model to obtain text description information of the first visible light image after a preset constraint; the text description information of the first visible light image after the preset constraint is used to indicate the text description information of the first infrared image;

[0024] A second text conversion unit, configured to obtain a first infrared text coding feature by passing the text description information of the first infrared image through a pre-trained text coding model;

[0025] The second encoding unit is used to generate a noisy infrared image by passing the first infrared coded image and the first noise generated by the time step through a diffusion model; and perform feature fusion on the noisy infrared image, the first infrared text coding feature, and the first visible light coded image through a diffusion model to predict a second noise;

[0026] The first training unit is used to calculate the gradient loss function of the diffusion model according to the first noise generated at the time step and the gradient of the second noise predicted by the model; and update the diffusion model according to the gradient loss function.

[0027] Beneficial effects of the present invention: Before training the diffusion model, the embodiment of the present invention first needs to pre-train a codec network, which can map the image to the latent space and map the target image in the latent space back to the original space. After the codec network is trained, the visible light image and the infrared image are respectively sent to the encoder network for encoding. After the encoding is completed, the infrared image is denoised. At the same time, the text prompt is encoded by the text encoder, and the generated feature vector is trained together with the visible light prompt and the noisy infrared image through feature fusion to learn the distribution of noise. The latent space representation of the target image is obtained through repeated iterations. The latent space representation of the target image is decoded by the decoder and mapped back to the original space to obtain the final generated target image. This method can reduce the computational cost and significantly improve the image conversion quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0029] Figure 1 It is a flowchart of a method for converting a visible light image into an infrared image based on feature fusion according to an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of the diffusion model structure of an embodiment of the present invention;

[0031] Figure 3 is a structural schematic diagram of a feature extraction module according to an embodiment of the present invention;

[0032] Figure 4 Schematic diagram of the structure of the neighborhood self-attention module of an embodiment of the present invention;

[0033] Figure 5 A schematic diagram of a device for converting a visible light image into an infrared image based on feature fusion according to an embodiment of the present invention;

[0034] Figure 6 The figure is a schematic diagram of a device for converting visible light images into infrared images based on feature fusion, which is a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0035] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than limiting the present application. The singular forms of "a", "said" and "the" used in the present application and claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0036] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" used may be interpreted as "at..." or "when..." or "in response to determination".

[0037] In an embodiment of the present application, a method and device for converting visible light images into infrared images based on feature fusion are proposed. The visible light image and the infrared image are images of the same target scene. The visible light image acquired by a front-end device such as a network camera, an analog camera, etc. can be converted into an infrared image. The visible light image stored by a back-end device such as a server, a management device, a storage device, etc. can also be converted into an infrared image.

[0038] The method and device for converting visible light images into infrared images based on feature fusion of the present invention will be further described below in conjunction with the drawings in the specification.

[0039] See also Figure 1 FIG. 1 is a flow chart of a method for converting a visible light image into an infrared image based on feature fusion, and the method may include:

[0040] 101. Acquire a target visible light image and a target noise image;

[0041] In the embodiment of the present invention, the target visible light image can be obtained through front-end devices such as optical imaging devices such as network cameras and analog cameras, and can also be obtained through back-end devices such as servers, management devices, storage devices, etc. Exemplarily, the target noise image is random noise that obeys some specific distributions, such as a random noise image that obeys Gaussian distribution (normal distribution); the embodiment of the present invention replaces the position of the noisy infrared image in the diffusion model with a random noise image, and gradually removes the target noise image to obtain the target infrared image.

[0042] 102. Pass the target visible light image through a visible light precoding network to obtain a target visible light coded image;

[0043] Currently, for low-level vision tasks, the amount of computation is generally proportional to the square of the image resolution. With the advancement of technology, image resolution continues to increase, which leads to an exponential increase in the amount of computation for low-level vision tasks. Higher resolution not only increases the amount of computation, but also makes the performance of traditional convolutional neural networks (CNNs) in long-distance dependency perception gradually insufficient. In order to meet this challenge, it is necessary to introduce more powerful feature extraction networks that can efficiently extract effective features from large-size images, maintain model performance while reducing computational costs. In addition, as the resolution increases, how to reduce the amount of computation while ensuring the quality of generation becomes a key issue in improving the efficiency of low-level vision tasks.

[0044] Based on this, the embodiments of the present invention adopt a precoding network and a predecoding network. The function of these coding and decoding networks is to map infrared images or visible light images to latent space and decode them from latent space back to original space. For example, the target visible light image is input into the visible light precoding network. The visible light precoding network first encodes the target visible light image into a feature representation with a resolution of 64×64, and then inputs the encoded feature representation into the visible light decoding network for decoding. The final image resolution is 256×256. By calculating the loss between the original image and the decoded image, the coding and decoding network is optimized and pretrained to ensure that it can efficiently restore the original details of the image. Similarly, infrared images also need to train a coding and decoding network with the same structure to ensure that the infrared image can be efficiently mapped between the latent space and the original space.

[0045] Among them, the formula involved in the pre-training process can be expressed as:

[0046] z=ε(x)

[0047]

[0048] Where x∈R H×W×C , H and W are the height and width of the image, and C is the number of channels. The encoder ε maps the input image x to the latent space z∈R h×w×c , h and w are the encoded image resolutions, and c is the number of channels in the latent space. The decoder D maps the latent space feature z back to the original space to obtain the decoded image During the training process, the goal is to minimize the loss between the original image and the decoded image, using the mean square error as the loss function, where N is the total number of pixels in the image, x i , are the i-th pixel values ​​of the original image and the decoded image, respectively. When training the network, the parameters of the encoder and decoder are updated by minimizing the loss function.

[0049] It should be noted that, in the embodiment of the present invention, since only the visible light image needs to be converted into an infrared image, the embodiment of the present invention cancels the visible light pre-decoding network.

[0050] 103. The target visible light image is passed through a pre-trained text generation model to obtain text description information of the target visible light image after a preset constraint; the text description information of the target visible light image after the preset constraint is used to indicate the text description information of the target infrared image;

[0051] In an embodiment of the present invention, considering the significant differences between the imaging principles of visible light and infrared images, infrared images cannot present visual information such as color. Therefore, strict constraints are imposed on the content when generating text, and information irrelevant to infrared images such as descriptions of color and lighting is prohibited, ensuring that the generated text only describes the core content and structural information in the image.

[0052] In the embodiment of the present invention, the step 103 may specifically include:

[0053] 131. The target visible light image is subjected to a pre-trained BLIP (Bootstrapping Language-Image Pre-training, BLIP) text generation model to obtain text description information of the target visible light image; the text description information of the target visible light image includes color information, time information, object information, shape information, position information and description of other physical attributes;

[0054] 132. Filter the text description information of the target visible light image by adding a text constraint prefix and cleaning the generated text data, filter out the color information and the time information, and obtain the text description information of the target visible light image after the preset constraints.

[0055] Exemplarily, taking a certain indoor scene as an example, assuming that the text description information of the target visible light image shows an indoor scene. In the center of the picture, there is a rectangular table with some items on it. On the left side of the table is a white ceramic cup, and next to it is an open laptop with some text and numbers displayed on the screen. In order to allow the text description information of the target visible light image to indicate the text description information of the target infrared image, this embodiment uses preset constraints to remove the text description information of the target visible light image, deleting information about color, information about time, etc. That is, the color of the table, the color of the cup, etc. are deleted. After removing this information, the text description information of the target visible light image can focus on the image content itself, thereby more accurately reflecting the text description information of the target infrared image.

[0056] 104. Passing the text description information of the target infrared image through a pre-trained text encoding model to obtain text encoding features of the target infrared image;

[0057] In an embodiment of the present invention, the text description information of the target infrared image is input into a pre-trained text encoder for encoding to generate text encoding features describing the image content. These text encoding features can be input into the diffusion model together with the latent space features extracted by the pre-encoding network. The image generation is constrained by feature fusion and attention encoding, which avoids the introduction of a large amount of computational overhead and improves the image generation quality.

[0058] 105. Perform feature fusion on the target visible light coded image, the text coded features of the target infrared image, and the target noise image through a trained diffusion model to obtain target infrared coded features;

[0059] In the embodiment of the present invention, for the diffusion model, Figure 2 As shown in the figure, on the one hand, an infrared image is input, and the state of the original infrared image and its various noise stages is mapped to a low-dimensional latent space using a pre-trained infrared encoder. A visible light image is input, and the state of the original visible light image is mapped to a low-dimensional latent space using a pre-trained visible light encoder. The benefit of this is that the computational cost can be reduced because the diffusion model directly acts on lower-dimensional data. On the other hand, the visible light image is passed through a text encoding model and a text encoder to generate text encoding features; this information is used through reverse reasoning to obtain the denoised latent representation to generate an image.

[0060] Based on the above diffusion model, in an embodiment of the present invention, the target visible light coded image is used as an image condition, the text coding feature of the target infrared image is used as a text condition, and the target noise image is used as the starting point of the generation process. The above multimodal condition information is used to guide the generation of a target infrared image that meets specific requirements. In this process, the diffusion model performs a denoising operation on the input target noise image to generate a potential representation that meets the multimodal constraints, namely, the target infrared coding feature, which contains the semantic and structural information of the target infrared image.

[0061] In an embodiment of the present invention, the training process of the diffusion model includes the following steps:

[0062] 151. Acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are an image pair of the same scene acquired at the same time;

[0063] In the embodiment of the present invention, the front-end device may be a dual-sensor front-end device, where the dual sensors refer to a visible light sensor and an infrared light sensor. The visible light sensor is a sensor for obtaining the visible light intensity of an object, and the image collected by the visible light sensor is called a visible light image. The infrared light sensor is a sensor for obtaining the infrared light intensity of an object, and the image collected by the infrared light sensor is called an infrared light image. Based on this, the first visible light image and the first infrared image at the same collection time can be obtained through the dual sensors of the front-end device.

[0064] In an embodiment of the present invention, the first infrared image and the first visible light image are first registered, and these data sets are divided into a training set and a test set, and the training set and the test set contain image pairs of various scenes and lighting; then the infrared image and the visible light image are sent to a pre-trained network to learn the domain difference between the infrared image and the visible light image; the data set used in this embodiment can include a variety of scenes such as campuses, streets, woods and other complex scenes, and the resolution sizes of these images include various resolution sizes such as 512×512, 1024×1024 and so on.

[0065] Among them, the first infrared image and the first visible light image are first registered. After the registration is completed, the image pair is cropped to a resolution of 256×256, the first visible light image and the first infrared image are annotated, and then the data set is randomly divided into a training set and a test set.

[0066] 152. Pass the first visible light image and the first infrared image through their respective visible light precoding networks and infrared precoding networks to generate a first visible light coded image and a first infrared coded image;

[0067] In the embodiment of the present invention, the registered first visible light image and the first infrared image can pass through their respective precoding networks, and the visible light precoding network can map the first visible light image to a coding space, which is a low-dimensional, compact latent space representation. The infrared precoding network can map the first infrared image to a coding space, which is also a low-dimensional, compact latent space representation.

[0068] For example, assuming that the first visible light image and the first infrared image are images with a resolution of 256×256, after passing through their respective precoding networks, a first visible light coded image and a first infrared coded image with a resolution of 64×64 can be obtained, thereby compressing the size of the original image and reducing the computational cost of subsequent processes.

[0069] 153. The first visible light image is passed through a pre-trained text generation model to generate text description information of the first visible light image after a preset constraint; the text description information of the first visible light image after the preset constraint is used to indicate the text description information of the first infrared image;

[0070] In an embodiment of the present invention, similar to steps 131-132, the embodiment of the present invention can generate text description information of the first visible light image, and the text description information of the first visible light image can be used as prompt information of the first infrared image to generate the first infrared image in a subsequent process.

[0071] 154. Passing the text description information of the first infrared image through a corresponding pre-trained text encoding network to generate a first infrared text encoding feature;

[0072] 155. Pass the first infrared coded image and the first noise generated by the time step through a diffusion model to generate a noisy infrared image;

[0073] In the embodiment of the present invention, a pair of first infrared and first visible light images is randomly sampled from the training data, a time step is randomly sampled, and the time step is positively correlated with the noise intensity. Then, the obtained first noise is fused with the first infrared image to obtain an infrared image with noise. It can be expressed as:

[0074]

[0075] Among them, I t represents the noisy infrared image with the t-th step noise, I represents the first randomly selected infrared image, and ε is the initial noise.

[0076] In this process, the infrared image can be diffused and denoised; then the noisy infrared image and visible light image are sent to the diffusion model to learn the distribution of noise.

[0077] 156. Fusing the noisy infrared image, the first infrared text coding feature, and the first visible light coding image through a diffusion model to predict a second noise;

[0078] In the embodiment of the present invention, the process of predicting the second noise may include:

[0079] 251. Splicing the noisy infrared image generated according to the time step with the first visible light coded image to obtain a first fused image;

[0080] 252. Input the first fused image and the number of noise addition steps into a four-branch encoder network of a diffusion model to extract original features, local features, regional features, and global features;

[0081] In the embodiment of the present invention, in order to extract multi-scale information of an image, such as Figure 3 As shown, an embodiment of the present invention proposes a feature extraction module that can extract image features from three granularities: local, regional and global, to obtain richer information. The four-branch encoder network includes an original information connection layer, a local information encoding layer, a regional information encoding layer and a global information encoding layer. The original information connection layer adopts a jump connection to retain the original feature information and prevent gradient explosion or gradient disappearance; the local information encoding layer includes two convolutional layers and a channel attention module to extract local information; the regional information encoding layer includes a window self-attention module to extract regional information; the global information encoder includes a neighborhood self-attention module to extract global information.

[0082] Specifically, compared with the ResNet Block used in traditional UNet, the feature extraction module of the embodiment of the present invention performs better in long-distance perception and overcomes the shortcomings of the residual block. The feature extraction module includes four branches: the first branch consists of two convolutional layers and a channel attention module, which is used to extract local information; the second branch uses a skip connection to retain the original feature information and prevent gradient explosion or gradient disappearance; the third branch extracts regional information through a window self-attention mechanism, and the fourth branch extracts global information through a neighborhood self-attention mechanism. Finally, the feature information of the four branches is added together to integrate rich features at multiple scales, thereby significantly improving the ability to generate high-quality images.

[0083] In the embodiment of the present invention, Figure 4 As shown, the neighborhood self-attention module includes:

[0084] Inputting the first fused image and the feature vector after the noise step encoding into the global information encoder of the diffusion model to extract the feature information of the target image block and the adjacent image blocks;

[0085] Transform the feature information of the target image block through Q linear transformation to obtain a first Q vector;

[0086] Performing K linear transformations on the feature information of the target image block to obtain a first K vector;

[0087] Transform the feature information of the adjacent image blocks by V linear transformation to obtain a first V vector;

[0088] Perform dot product processing on the first Q vector and the first K vector to obtain a first QK vector;

[0089] Processing the first QK vector and the position code to obtain a second QK vector;

[0090] The second QK vector and the first V vector are weighted to obtain global information of the target image block.

[0091] In this way, when processing sequence data, the model assigns different importance or preferences to elements at different encoding positions in the sequence. This preference can enhance the model's ability to perceive position information in sequence data. This can effectively fuse features within the neighborhood, thereby increasing the model's ability to perceive over long distances, and improving the quality of image generation in subsequent processes.

[0092] 253. Fusing the original feature, the local feature, the regional feature, and the global feature to obtain a first coding feature;

[0093] In the embodiment of the present invention, the feature information of the four branches is added together to integrate the multi-scale first coding features, thereby significantly improving the ability to generate high-quality images.

[0094] 254. Input the first infrared text encoding feature into a decoder network of a diffusion model, fuse it with the first encoding feature, and output a predicted second noise.

[0095] According to steps 251 - 254 of the embodiment of the present invention, the noise output by the model can be effectively predicted, and the difference between the input noise and the output noise can be used to optimize the diffusion model.

[0096] 157. Calculate the gradient loss function of the diffusion model according to the first noise generated at the time step and the gradient of the second noise predicted by the model;

[0097] In the embodiment of the present invention, the training process is expressed by the formula: in is the predicted noise, V ε is the feature of the conditional visible light image after being encoded by the encoder, t is the time step of random sampling, and F text Feature vector generated after passing the text prompt through the pre-trained encoder.

[0098] 158. Update the diffusion model according to the gradient loss function.

[0099] It should be noted that, through steps 151-158 of the embodiment of the present invention, the diffusion model can be trained so that the trained diffusion model can accurately generate a target infrared image based on the target visible light image and the target noise image.

[0100] 106. Pass the target infrared coding features through an infrared pre-decoding network to obtain a target infrared image.

[0101] In an embodiment of the present invention, the target infrared coding features are converted into a final target infrared image through an infrared pre-decoding network, so that the target infrared coding features can be returned from the latent space to the image space.

[0102] It can be understood that in the embodiment of the present invention, through steps 101-106, a target visible light image is randomly obtained, and it is input into the pre-trained visible light precoding network for encoding, and the feature representation in the latent space is extracted, that is, the target visible light coded image. At the same time, a Gaussian noise is randomly sampled as the target noise image, and the target noise image and the target visible light coded image are used as the initial input of the diffusion model. In addition, the text description information of the target infrared image is encoded by the pre-trained text coding model to generate a feature vector, and as text condition information, it is fused with the target visible light coded image as the image condition information. The fusion process is realized by the cross attention mechanism to ensure that these two condition prompt information can effectively guide the generation process. The output of the diffusion model is the noise distribution predicted under the current step. Then, the predicted noise is removed from the initial input noise to obtain the denoised intermediate result. The intermediate result is input into the diffusion model again, and the denoising step is continued. Through multiple iterations of denoising, the image features are gradually restored until a clean latent space representation of the target infrared image, that is, the target infrared coded feature, is finally restored, and then it is passed through the infrared precoding network to obtain the target infrared image.

[0103] In an embodiment of the present invention, the present invention further provides a device for converting a visible light image into an infrared image based on feature fusion, such as Figure 5 As shown, the device includes:

[0104] A first acquisition unit 201 is used to acquire a target visible light image and a target noise image;

[0105] A first precoding unit 202 is used to pass the target visible light image through a visible light precoding network to obtain a target visible light coded image;

[0106] The first text extraction unit 203 is used to pass the target visible light image through the pre-trained text generation model to obtain text description information of the target visible light image after the preset constraints; the text description information of the target visible light image after the preset constraints is used to indicate the text description information of the target infrared image;

[0107] A first text conversion unit 204 is used to obtain text coding features of the target infrared image by passing the text description information of the target infrared image through a pre-trained text coding model;

[0108] The first encoding unit 205 is used to perform feature fusion on the target visible light coded image, the text coded features of the target infrared image and the target noise image through a trained diffusion model to obtain a target infrared coded feature;

[0109] The first pre-decoding unit 206 is used to pass the target infrared coding features through an infrared pre-decoding network to obtain a target infrared image.

[0110] In the embodiment of the present invention, a target visible light image and a target noise image are acquired by a first acquisition unit 201; a target visible light image is passed through a visible light precoding network by a first precoding unit 202 to obtain a target visible light coded image; a target visible light image is passed through a pre-trained text generation model by a first text extraction unit 203 to obtain text description information of the target visible light image after preset constraints; the text description information of the target visible light image after preset constraints is used to indicate the text description information of the target infrared image; a text conversion unit 204 passes the text description information of the target infrared image through a pre-trained text coding model to obtain text coding features of the target infrared image; a first coding unit 205 performs feature fusion on the target visible light coded image, the text coding features of the target infrared image, and the target noise image through a trained diffusion model to obtain target infrared coding features; a first pre-decoding unit 206 passes the target infrared coding features through an infrared pre-decoding network to obtain a target infrared image. This can reduce the computational cost and significantly improve the conversion quality of visible light to infrared images.

[0111] In the embodiment of the present invention, Figure 6 As shown, the device may also include:

[0112] The second acquisition unit 211 is used to acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are an image pair of the same scene acquired at the same time;

[0113] The first pre-coding unit 202 is further configured to generate a first visible light coded image by passing the first visible light image through a pre-trained visible light coding network;

[0114] A second pre-coding unit 212, configured to generate a first infrared coded image by passing the first infrared image through a pre-trained infrared coding network;

[0115] The second text extraction unit 213 is used to obtain text description information of the first visible light image after being subjected to preset constraints by passing the first visible light image through a pre-trained text generation model; the text description information of the first visible light image after being subjected to preset constraints is used to indicate text description information of the first infrared image;

[0116] A second text conversion unit 214 is used to obtain a first infrared text coding feature by passing the text description information of the first infrared image through a pre-trained text coding model;

[0117] The second encoding unit 215 is used to generate a noisy infrared image by passing the first infrared coded image and the first noise generated by the time step through a diffusion model; and perform feature fusion on the noisy infrared image, the first infrared text coding feature, and the first visible light coded image through a diffusion model to predict a second noise;

[0118] The first training unit 216 is used to calculate the gradient loss function of the diffusion model according to the first noise generated at the time step and the gradient of the second noise predicted by the model; and update the diffusion model according to the gradient loss function.

[0119] In the embodiment of the present invention, a first visible light image and a first infrared image are acquired by a second acquisition unit 211; the first visible light image and the first infrared image are an image pair of the same scene acquired at the same time; the first precoding unit 202 is further used to pass the first visible light image through a pretrained visible light coding network to generate a first visible light coded image; the first infrared image is passed through a pretrained infrared coding network through a second precoding unit 212 to generate a first infrared coded image; the first visible light image is passed through a pretrained text generation model through a second text extraction unit 213 to obtain text description information of the first visible light image after preset constraints; the text description information of the first visible light image after preset constraints The information is used to indicate the text description information of the first infrared image; the text description information of the first infrared image is passed through the pre-trained text encoding model by the second text conversion unit 214 to obtain the first infrared text encoding feature; the first infrared encoding image and the first noise generated by the time step are passed through the diffusion model by the second encoding unit 215 to generate a noisy infrared image; the noisy infrared image, the first infrared text encoding feature, and the first visible light encoding image are feature fused by the diffusion model to predict the second noise; the gradient of the second noise is predicted by the first training unit 216 according to the first noise generated by the time step and the model, and the gradient loss function of the diffusion model is calculated; according to the gradient loss function, the diffusion model is updated. This embodiment can realize the effective training of the diffusion model, so that the trained diffusion model can generate the target infrared image.

[0120] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0121] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for converting visible light images into infrared images based on feature fusion, characterized in that: The following steps are involved: Acquire a target visible light image and a target noise image; Passing the target visible light image through a visible light precoding network to obtain a target visible light coded image; The target visible light image is passed through a pre-trained text generation model to obtain text description information of the target visible light image after a preset constraint; the text description information of the target visible light image after a preset constraint is used to indicate the text description information of the target infrared image; Passing the text description information of the target infrared image through a pre-trained text encoding model to obtain text encoding features of the target infrared image; The target visible light coded image, the text coded features of the target infrared image and the target noise image are subjected to feature fusion through the trained diffusion model to obtain the target infrared coded features; The infrared coding features of the target are passed through an infrared pre-decoding network to obtain an infrared image of the target.

2. The method for converting visible light images into infrared images based on feature fusion according to claim 1, characterized in that: The target visible light image is passed through a pre-trained text generation model to obtain text description information of the target visible light image after preset constraints, including: The target visible light image is passed through the pre-trained BLIP text generation model to obtain text description information of the target visible light image; the text description information of the target visible light image includes color information, time information, object information, shape information, position information and description of other physical attributes; The text description information of the target visible light image is filtered by adding a text constraint prefix and cleaning the generated text data, the color information and the time information are filtered out, and the text description information of the target visible light image after the preset constraints is obtained.

3. The method for converting visible light images into infrared images based on feature fusion according to claim 1, characterized in that: The training process of the diffusion model includes the following steps: Acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are an image pair of the same scene acquired at the same time; Passing the first visible light image and the first infrared image through their respective visible light precoding networks and infrared precoding networks, respectively, to generate a first visible light coded image and a first infrared coded image; The first visible light image is passed through a pre-trained text generation model to generate text description information of the first visible light image after a preset constraint; the text description information of the first visible light image after the preset constraint is used to indicate the text description information of the first infrared image; Passing the text description information of the first infrared image through a corresponding pre-trained text encoding network to generate a first infrared text encoding feature; The first infrared coded image and the first noise generated by the time step are passed through a diffusion model to generate a noisy infrared image; The noisy infrared image, the first infrared text coding feature, and the first visible light coding image are feature-fused through a diffusion model to predict a second noise; Calculate the gradient loss function of the diffusion model according to the first noise generated at the time step and the gradient of the second noise predicted by the model; The diffusion model is updated according to the gradient loss function.

4. The method for converting visible light images into infrared images based on feature fusion according to claim 3 is characterized in that: The step of fusing the noisy infrared image, the first infrared text coding feature, and the first visible light coding image through a diffusion model to predict the second noise comprises the following steps: splicing the noisy infrared image generated according to the time step with the first visible light coded image to obtain a first fused image; Inputting the first fused image and the number of noise addition steps into a four-branch encoder network of a diffusion model to extract original features, local features, regional features, and global features; Fusing the original feature, the local feature, the regional feature and the global feature to obtain a first coding feature; The first infrared text coding feature is input into the decoder network of the diffusion model, and after being fused with the first coding feature, a predicted second noise is output.

5. The method for converting visible light images into infrared images based on feature fusion according to claim 4, characterized in that: The four-branch encoder network includes an original information connection layer, a local information encoding layer, a regional information encoding layer and a global information encoding layer. The original information connection layer adopts a jump connection to retain the original feature information and prevent gradient explosion or gradient disappearance; the local information encoding layer includes two convolutional layers and a channel attention module to extract local information; the regional information encoding layer includes a window self-attention module to extract regional information; the global information encoder includes a neighborhood self-attention module to extract global information.

6. The method for converting visible light images into infrared images based on feature fusion according to claim 5, characterized in that: The neighborhood self-attention module includes: Inputting the first fused image and the feature vector after the noise step encoding into the global information encoder of the diffusion model to extract the feature information of the target image block and the adjacent image blocks; Transform the feature information of the target image block through Q linear transformation to obtain a first Q vector; Performing K linear transformations on the feature information of the target image block to obtain a first K vector; Transform the feature information of the adjacent image blocks by V linear transformation to obtain a first V vector; Perform dot product processing on the first Q vector and the first K vector to obtain a first QK vector; Processing the first QK vector and the position code to obtain a second QK vector; The second QK vector and the first V vector are weighted to obtain global information of the target image block.

7. A device for converting visible light images into infrared images based on feature fusion, characterized in that: The device comprises: A first acquisition unit, used for acquiring a target visible light image and a target noise image; A first precoding unit, configured to pass the target visible light image through a visible light precoding network to obtain a target visible light coded image; A first text extraction unit is used to pass the target visible light image through a pre-trained text generation model to obtain text description information of the target visible light image after a preset constraint; the text description information of the target visible light image after the preset constraint is used to indicate the text description information of the target infrared image; A first text conversion unit, configured to obtain text encoding features of the target infrared image by passing the text description information of the target infrared image through a pre-trained text encoding model; A first encoding unit is used to perform feature fusion on the target visible light coded image, the text coded features of the target infrared image, and the target noise image through a trained diffusion model to obtain a target infrared coded feature; The first pre-decoding unit is used to obtain the target infrared image by passing the target infrared coding feature through an infrared pre-decoding network.

8. The device for converting visible light images into infrared images based on feature fusion according to claim 7, characterized in that: Also includes: A second acquisition unit is used to acquire a first visible light image and a first infrared image; the first visible light image and the first infrared image are an image pair of the same scene acquired at the same time; The first pre-coding unit is further used to generate a first visible light coded image by passing the first visible light image through a pre-trained visible light coding network; A second pre-coding unit, configured to pass the first infrared image through a pre-trained infrared coding network to generate a first infrared coded image; A second text extraction unit is used to pass the first visible light image through a pre-trained text generation model to obtain text description information of the first visible light image after a preset constraint; the text description information of the first visible light image after the preset constraint is used to indicate the text description information of the first infrared image; A second text conversion unit, configured to obtain a first infrared text coding feature by passing the text description information of the first infrared image through a pre-trained text coding model; The second encoding unit is used to generate a noisy infrared image by passing the first infrared coded image and the first noise generated by the time step through a diffusion model; and perform feature fusion on the noisy infrared image, the first infrared text coding feature, and the first visible light coded image through a diffusion model to predict a second noise; The first training unit is used to calculate the gradient loss function of the diffusion model according to the first noise generated at the time step and the gradient of the second noise predicted by the model; and update the diffusion model according to the gradient loss function.

Images