Visible and infrared light image conversion system and method based on diffusion model

Abstract

The invention relates to a visible and infrared light image conversion system and method based on a diffusion model, and the method comprises the steps: fusing a text description through a visual language understanding module to generate an image text description, and taking the image text description as a control flag bit of a diffusion model algorithm; a diffusion model algorithm executed by the diffusion model framework can accurately regulate and control the style characteristics of the infrared image according to the semantic instruction, the technical defect that traditional single-style image conversion is insufficient in the aspect of style control is finally overcome, and semantic information and controllable flag bits are fully utilized; and the semantic consistency of the infrared image key area generated by the whole system is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular, to a visible and infrared image conversion system and method based on a diffusion model. BACKGROUND

[0002] Converting a visible light image into an infrared image is a core task in the field of image analysis, aiming to convert objects such as targets and scenes in the visible light image into the infrared modality. This technology has important application value in the fields of video monitoring, moving target detection and tracking, and ecological environment assessment.

[0003] However, the process of converting a visible light image into an infrared image faces complex technical challenges. First, the imaging of an infrared image is greatly related to the semantic information of the surrounding environment and target objects. For example, different parts of the same building have similar color distributions in the visible light modality, but due to different heat absorption, the temperature radiation is different, resulting in significant differences in the infrared modality pixel values of different parts. Therefore, it is necessary to analyze the environmental information such as light, season, etc. shown in the visible light image to ensure the quality of the generated infrared image. Second, the differences in imaging conditions, sensor types, etc. between different infrared datasets result in significant differences in the styles of infrared images of different datasets. A model trained on one dataset is difficult to apply directly on another dataset. Finally, due to the difficulty in obtaining infrared images compared to visible light images, the scale of public infrared image datasets is small. These phenomena greatly limit the usability of the technology of converting a visible light image into an infrared image.

[0004] For the task of converting a visible light image into an infrared image, some existing methods directly define it as a simple image pixel-by-pixel translation problem, and use commonly used image generation models such as variational autoencoder, generative adversarial network and diffusion model to complete this task, but it is difficult to deal with the problem that the imaging of an infrared image is greatly affected by the semantic information of the surrounding environment.

[0005] In order to deal with the problem that the imaging of an infrared image is greatly affected by the semantic information of the surrounding environment, some existing methods incorporate low-level semantic information such as structure maps and geometric consistency into the generation model, but these low-level semantic information plays a limited role and does not fully utilize the large amount of semantic information exhibited in the visible light image. In order to solve the problem that the imaging of an infrared image is greatly affected by the imaging conditions and sensor types, some existing methods introduce physical information constraints in the generation model, but it is still difficult to compensate for the style differences between different datasets. SUMMARY

[0006] The technical problem to be solved by the present application is how to overcome the technical defects of the prior art, such as the lack of semantic information utilization and style control in the visible light image to infrared image conversion method.

[0007] The present application provides a visible and infrared light image conversion system based on a diffusion model, which comprises: A visual language understanding module is configured to perform image coding on the visible light image based on self-attention mapping to obtain image features, and then fuse a preset text description generation control flag on the image features and perform decoding to obtain an image text description; A diffusion model framework is in communication with the visual language understanding module and is configured to convert a preset noise image, the visible light image, a segmentation image of the visible light image, infrared single light data of the visible light image and the image text description into three styles of infrared images through a diffusion model algorithm; The segmentation image of the visible light image is a semantic segmentation image, an instance segmentation image or a panoramic segmentation image of the visible light image, and the infrared single light data of the visible light image is an infrared light image obtained by performing visible light-infrared light image conversion on the visible light image.

[0008] The visible and infrared light image conversion system based on a diffusion model disclosed in the present application fuses a text description to generate an image text description through a visual language understanding module, uses the image text description as a control flag of the diffusion model algorithm, and enables the diffusion model algorithm executed by the diffusion model framework to accurately regulate the style features of the infrared image according to semantic instructions, thereby overcoming the technical defects of the traditional single style conversion in terms of style control. Moreover, the diffusion model framework combines the visible light image, the segmentation image and the infrared single light data to fully utilize semantic information, significantly reduces the error accumulation of the traditional multi-stage conversion, and improves the image conversion speed and the structural similarity (SSIM) index under the same hardware conditions. Furthermore, the visual language understanding module also performs image coding on the visible light image based on self-attention mapping, and the image coding technology based on self-attention mapping deeply binds the text description and the visual features to fully utilize semantic information. The control flag greatly improves the semantic consistency of the generated infrared image in key areas (such as the human body contour and high-temperature objects), and avoids the problem of disconnection between the text and the image in the traditional method.

[0009] In one possible implementation, the visual language understanding module comprises: An image encoder is configured to perform image coding on the visible light image based on self-attention mapping to obtain image features; The text decoder, in communication with the image encoder and the diffusion model framework, is configured to first incorporate a preset text description generation control flag into the image feature, and then decode the image feature incorporating the text description generation control flag to obtain an image text description.

[0010] The visual language understanding module with the above structure and function can perform image coding on the visible light map based on self-attention mapping, incorporate a preset text description generation control flag, and further ensure that the diffusion model algorithm can accurately regulate the style features of the infrared image according to semantic instructions, thereby improving the conversion efficiency.

[0011] In a possible implementation, the image encoder includes a self-attention module, an encoding controller, and a feedforward module one connected in sequence. The encoding controller is also in communication with the text decoder. The encoding controller is configured to call the self-attention module to perform self-attention mapping on the visible light map to obtain self-attention features, and sum the self-attention features with the visible light map to obtain a summed image. Then, the encoding controller calls the feedforward module one to perform mapping of a feedforward network model on the summed image to obtain a feedforward feature map, and sums the feedforward feature map with the summed image to obtain the image feature. The image encoder with the above structure forms a collaborative strategy of a self-attention mechanism and a residual connection. Through the collaborative optimization of the self-attention mechanism and the residual connection, the problem of long-distance dependence being ignored, feature loss, and modal split in a traditional encoder can be solved.

[0012] In a possible implementation, the text decoder includes a causal attention module, a cross-attention module, a feedforward module two, and a decoding controller. The decoding controller is in communication with the causal attention module, the cross-attention module, the feedforward module two, and the diffusion model framework. The cross-attention module is also in communication with the encoding controller. The decoding controller is configured to call the causal attention module to perform causal attention mapping on the text description generation control flag to obtain causal attention features, and sum the causal attention features with the text description generation control flag to obtain a fusion feature one. Then, the decoding controller calls the cross-attention module to perform cross-attention mapping on the fusion feature one and the image feature to obtain a cross feature, and sums the cross feature with the fusion feature one to obtain a fusion feature two. Finally, the decoding controller calls the feedforward module two to perform mapping of a feedforward network model on the fusion feature two to obtain a feedforward text, and sums the feedforward text with the fusion feature two to obtain the image text description.

[0013] Through the decoding controller cooperates with the cascade processing mechanism of the multi-module, the generation quality and semantic accuracy of the image text description are significantly improved, which is specifically manifested as feature fusion optimization, cross-modal alignment enhancement and generation efficiency improvement.

[0014] In a possible implementation, the diffusion model framework comprises: A latent space encoding module configured to perform latent space encoding on the noise map, the visible light map, the segmentation map of the visible light map, and the infrared single-light data of the visible light map to obtain encoding one, encoding two, and encoding three; A denoising module in communication with the decoding controller and the latent space encoding module, and configured to perform denoising on the encoding one to obtain denoising result one, perform denoising on the encoding two to obtain denoising result two, and perform denoising on the fusion result of the image text description and the encoding three to obtain denoising result three in parallel; A latent space decoding module in communication with the denoising module, and configured to perform latent space decoding on the denoising result one, the denoising result two, and the denoising result three respectively to obtain three styles of infrared images.

[0015] The diffusion model framework with the above structure and function further reduces the error accumulation of the traditional multi-stage conversion under the assistance of the denoising module and the latent space decoding module, so that the image conversion speed and the structural similarity index are further improved.

[0016] In a possible implementation, the latent space encoding module is configured to operate as follows: The infrared single-light data of the visible light map is subjected to mapping operation of the latent space encoding algorithm to obtain the encoding one; The noise map and the mapping operation result of the latent space encoding algorithm performed on the visible light map are summed to obtain the encoding two; The noise map, the mapping operation result of the latent space encoding algorithm performed on the visible light map, and the mapping operation result of the latent space encoding algorithm performed on the segmentation map of the visible light map are summed to obtain the encoding three.

[0017] The latent space encoding module operating in the above manner can efficiently and accurately perform latent space encoding on the noise map, the visible light map, the segmentation map of the visible light map, and the infrared single-light data of the visible light map to obtain the encoding one, the encoding two, and the encoding three, thereby ensuring the orderly operation of the diffusion model algorithm.

[0018] In a possible implementation, the denoising module comprises: A text encoder in communication with the decoding controller, and configured to perform text encoding on the image text description to obtain text features; a cross-attention layer model, in communication with the text encoder, configured to perform cross-attention mapping on the text features to obtain cross-text features; a first-stage denoising network, in communication with the latent space encoding module, configured to denoise the first encoding to obtain the first denoising result; a second-stage denoising network, in communication with the latent space encoding module, configured to denoise the second encoding to obtain the second denoising result; a third-stage denoising network, in communication with both the cross-attention layer model and the latent space encoding module, configured to denoise the fusion result of the cross-text features and the third encoding to obtain the third denoising result.

[0019] The denoising module with the above structure and function extracts image description features through the text encoder, and fuses the image encoding after mapping by the cross-attention layer, so that the denoising process strictly follows the text semantic guidance, solves the detail distortion problem caused by semantic deviation in traditional denoising, and especially improves the detail restoration accuracy of complex scenes (such as medical images and artistic creation). Moreover, the first stage processes the basic noise (encoding one), the second stage refines the high-frequency details (encoding two), and the third stage fuses the text features to realize semantic calibration. This kind of phased progressive denoising reduces the computational load of a single calculation, realizes dynamic resource allocation, and enhances noise robustness.

[0020] In a possible implementation, the first-stage denoising network, the second-stage denoising network, and the third-stage denoising network each include a denoising encoder and a denoising decoder, the denoising encoder is a network structure formed by an input convolutional layer model, multiple cross-attention down-sampling modules, a down-sampling module, and a bottleneck module connected in sequence, and the denoising decoder is a network structure formed by an up-sampling module, multiple cross-attention up-sampling modules, and an output convolutional layer model connected in sequence; The input convolutional layer model in the first-stage denoising network and the second-stage denoising network is in communication with the latent space encoding module, the input convolutional layer model in the third-stage denoising network is in communication with both the cross-attention layer model and the latent space encoding module, the up-sampling module is in communication with the bottleneck module, and the output convolutional layer model is in communication with the latent space decoding module; The number of cross-attention down-sampling modules is the same as that of cross-attention up-sampling modules, and they are in one-to-one communication.

[0021] This modular design can support adaptation to different modal inputs, and can solve the universality problem of cross-modal data collaborative denoising while realizing denoising.

[0022] In a possible implementation, the latent space decoding module includes three latent space decoders in one-to-one communication with the output convolutional layer model to decode the denoising results obtained by the denoising module, further improving the infrared images of three different styles obtained.

[0023] Another technical solution of the present application is to provide a visible and infrared light image conversion method based on a diffusion model, comprising the following steps: S1: constructing an infrared single light data set, a visible light-infrared image pair data set, and a specific style visible light-infrared image pair data set; S2: constructing a diffusion model training framework based on progressive learning, and using the infrared single light data set, the visible light-infrared image pair data set, and the specific style visible light-infrared image pair data set constructed in step S1 to optimize the parameters of the diffusion model training framework to obtain a diffusion model framework; S3: performing image encoding based on self-attention mapping on the visible light image through a visual language understanding module to obtain image features, and decoding the image features by fusing a preset text description generation control flag to obtain an image text description; S4: converting a preset noise map, the visible light image, a segmentation map of the visible light image, infrared single light data of the visible light image, and the image text description into infrared images of three styles through the diffusion model framework.

[0024] The visible and infrared light image conversion method based on the diffusion model disclosed in the present application first constructs an infrared single light data set, a visible light-infrared image pair data set, and a specific style visible light-infrared image pair data set, and uses these data sets to optimize the parameters of the diffusion model training framework in three stages to obtain a diffusion model framework, thereby effectively combining the visual language information provided by the multi-modal visual language large model, realizing semantic-driven conversion of visible light images to infrared images. At the same time, this three-stage progressive learning framework can solve the problem of too large infrared imaging style difference between different data sets, and the collection of the infrared single light data set, the visible light-infrared image pair data set, and the specific style visible light-infrared image pair data set supports the entire training process, realizing high-quality conversion of visible light images to infrared images, and overcoming the technical defects of the existing visible light image conversion infrared image method in terms of semantic information utilization and style control. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The structure diagram of the visible and infrared light image conversion system based on the diffusion model disclosed in the embodiments of the present application is shown in the figure; Figure 2A running schematic diagram of the visual language understanding module disclosed in the embodiments of the present application; Figure 3 A running schematic diagram of the diffusion model framework disclosed in the embodiments of the present application; Figure 4 A structural schematic diagram of the first-stage denoising network, the second-stage denoising network, or the third-stage denoising network disclosed in the embodiments of the present application; Figure 5 A flowchart of the method disclosed in the embodiments of the present application. DETAILED DESCRIPTION

[0026] First, those skilled in the art should understand that these embodiments are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments as needed in order to adapt to specific application occasions.

[0027] In the description of the embodiments of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms "communication connection", "communication" should be understood in a broad sense, that is, to have a connection mode of transmitting information to each other, for example, can be through a conductive wire to realize circuit connection, or through a wireless signal channel (channel) to realize electrical connection, or a combination of the two.

[0028] In the embodiments of the present application, unless otherwise explicitly specified and limited, the first feature is "on" or "under" the second feature, which can be direct contact between the first and second features, or indirect contact between the first and second features through an intermediate medium. Moreover, the first feature "above", "over" and "on" the second feature can be directly above or obliquely above the first feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "under" and "under" the second feature can be directly below or obliquely below the first feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0029] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Referring to Figure 1 The embodiments of the present application disclose a visible and infrared image conversion system based on a diffusion model, Figure 1 A structural schematic diagram of the conversion system, the conversion system comprises a visual language understanding module and a diffusion model framework, wherein the diffusion model framework communicates with the visual language understanding module.

[0031] Referring to Figure 1 and Figure 2In the conversion system, the visual language understanding module is configured to perform self-attention mapping based image encoding on the visible light image to obtain image features, then fuse preset text description generation control flags on the image features, and perform decoding to obtain image text descriptions. In this embodiment, the visual language understanding module includes an image encoder and a text decoder, wherein the text decoder simultaneously communicates with the image encoder and the diffusion model framework.

[0032] Referring to Figure 2 In the visual language understanding module, the image encoder is configured to perform self-attention mapping based image encoding on the visible light image to obtain image features. In this embodiment, the image encoder includes a self-attention module, an encoding controller (the operation principle of which is shown in Figure 2 ), and a feedforward module one connected in series, and the encoding controller also communicates with the text decoder. As shown in Figure 2 , the encoding controller is configured to call the self-attention module to perform self-attention mapping processing on the visible light image to obtain self-attention features, and perform summing on the self-attention features and the visible light image to obtain a summed image; then call the feedforward module one to perform mapping processing of a feedforward network model on the summed image to obtain a feedforward feature map, and perform summing on the feedforward feature map and the summed image to obtain image features.

[0033] Referring to Figure 2 In the visual language understanding module, the text decoder is configured to perform decoding on the image features to obtain image text descriptions by fusing preset text description generation control flags (the preset text description generation control flags are referred to as text description generation controls in Figure 2 ) on the image features. In this embodiment, the text decoder is an image feature based text decoder, including a causal attention module, a cross-attention module, a feedforward module two, and a decoding controller (the operation principle of the decoding controller is shown in Figure 2 ), and the decoding controller simultaneously communicates with the causal attention module, the cross-attention module, the feedforward module two, and the diffusion model framework, and the cross-attention module also communicates with the encoding controller. In this embodiment, the feedforward module one and the feedforward module two are feedforward modules of the same specification. It should be noted that if it is really necessary to reduce resource consumption, the text decoder and the image encoder can share one feedforward module, but in this embodiment, in order to avoid information crosstalk, the text decoder and the image encoder are each configured with a feedforward module of the same specification, as shown in Figure 2 .

[0034] Please continue to refer to Figure 2In the text decoder, the decoding controller is configured to call the causal attention module to perform causal attention mapping on the text description generation control flag to obtain causal attention features, and sum the causal attention features with the text description generation control flag to obtain fusion feature one; then call the cross attention module to perform cross attention mapping on fusion feature one and image features to obtain cross features, and sum the cross features with fusion feature one to obtain fusion feature two; finally call the feedforward module two to perform the mapping processing of fusion feature two on the feedforward network model to obtain feedforward text, and sum the feedforward text with fusion feature two to obtain image text description.

[0035] See Figure 1 , Figure 2 , Figure 3 and Figure 4 In this conversion system, the diffusion model framework is set up to use a diffusion model algorithm to convert the preset noise map, visible light map, visible light map segmentation map, and infrared single-light data of the visible light map (…). Figure 3 The infrared single-light data of the visible light image (referred to as infrared image) and the image text description are converted into three styles of infrared images, namely... Figure 3 Examples of the first, second, and third stages of generation are provided. The segmentation map of the visible light image can be a semantic segmentation map, an instance segmentation map, or a panoramic segmentation map. Figure 2 The leftmost part of the diagram shows the origin of the visible light image segmentation map used in this embodiment, which is obtained by transforming the visible light image through a pre-trained segmentation model. The infrared single-light data of the visible light image is any infrared image obtained by performing visible-infrared image conversion on the visible light image. This can be the result of performing visible-infrared image conversion on the visible light image using existing variational autoencoders, generative adversarial networks, or diffusion models.

[0036] See Figure 1 and Figure 3 In this embodiment, the diffusion model framework includes a latent space encoding module, a denoising module, and a latent space decoding module. The denoising module communicates with both the decoding controller and the latent space encoding module, and the latent space decoding module communicates with the denoising module.

[0037] In the diffusion model framework, the latent space coding module is configured to perform latent space coding on the noise map, visible light map, visible light map segmentation map, and infrared single-light data of the visible light map to obtain Code 1, Code 2, and Code 3. Figure 3The diagram illustrates the operating principle of the latent space coding module within the diffusion model framework. In this embodiment, the latent space coding module is configured to operate as follows: First, the infrared single-light data of the visible light image is mapped using a latent space coding algorithm to obtain code one; second, the noise image is summed with the mapping result of the latent space coding algorithm performed on the visible light image to obtain code two; third, the noise image, the mapping result of the latent space coding algorithm performed on the visible light image, and the mapping result of the latent space coding algorithm performed on the segmented image of the visible light image are summed to obtain code three.

[0038] Specifically, the latent space coding module operates according to the following calculation formula: , , , In the formula, Infrared single-light data representing the visible light pattern; Represents a noise graph; A segmented image representing the visible light spectrum; Represents a visible light diagram; The mapping operation represents the latent space coding algorithm; This represents the summation operation; Represents code one; Representative code two; This represents code three.

[0039] See Figure 3 and Figure 4 In the diffusion model framework, the denoising module is configured to run in parallel: denoising code 1 to obtain denoised result 1; denoising code 2 to obtain denoised result 2; and denoising the fusion result of the image text description and code 3 to obtain denoised result 3. See also... Figure 3 In this embodiment, the denoising module includes a text encoder and a cross-attention layer model (…). Figure 3 The denoising network consists of three stages: a cross-attention layer (referred to as the cross-attention layer), a first-stage denoising network, a second-stage denoising network, and a third-stage denoising network. The text encoder communicates with the decoding controller, the cross-attention layer model communicates with the text encoder, the first-stage denoising network communicates with the latent space coding module, the second-stage denoising network communicates with the latent space coding module, and the third-stage denoising network communicates with both the cross-attention layer model and the latent space coding module.

[0040] In the denoising module, the text encoder is configured to encode the text description of the image to obtain text features. The cross-attention layer model is configured to perform cross-attention mapping on the text features to obtain cross-text features. The first-stage denoising network is configured to denoise the first encoding to obtain denoised result one. The second-stage denoising network is configured to denoise the second encoding to obtain denoised result two. The third-stage denoising network is configured to denoise the fusion result of the cross-text features and the third encoding to obtain denoised result three.

[0041] See Figure 4 In the denoising module, the first-stage denoising network, the second-stage denoising network, and the third-stage denoising network all contain a denoising encoder and a denoising decoder. The denoising encoder is a model consisting of an input convolutional layer (…). Figure 4 The network structure consists of an input convolutional layer (referred to as the input convolutional layer), multiple cross-attention downsampling modules, a downsampling module, and a bottleneck module, connected in series. The denoising decoder is a model consisting of an upsampling module, multiple cross-attention upsampling modules, and an output convolutional layer. Figure 4 The network structure consists of concatenated output convolutional layers (referred to as output convolutional layers in the original text). In the first and second stage denoising networks, the input convolutional layer models communicate with the latent space coding module. In the third stage denoising network, the input convolutional layer models communicate with both the cross-attention layer model and the latent space coding module. The upsampling module communicates with the bottleneck module, and all output convolutional layer models communicate with the latent space decoding module. The number of cross-attention downsampling modules is the same as the number of cross-attention upsampling modules, and they communicate one-to-one. In this embodiment, the bottleneck module is the two-dimensional cross-attention UNet bottleneck module. Figure 4 In this context, N represents the number of channels, and H and W are the height and width of the input image after passing through the latent space encoder.

[0042] See Figure 3 In the diffusion model framework, the latent space decoding module is configured to perform latent space decoding on denoised result one, denoised result two, and denoised result three respectively to obtain infrared images of three styles. Figure 3 The diagram illustrates the operating principle of the latent space decoding module. The mapping operation represents the latent space decoder model. In this embodiment, the latent space decoding module includes three latent space decoders, which communicate with the output convolutional layer model in a one-to-one manner.

[0043] See Figure 5 The conversion method corresponding to the visible and infrared light image conversion system based on the diffusion model in this embodiment will be further disclosed below. Figure 5 The overall flowchart of this conversion method is as follows: S1: Constructing infrared single light dataset, visible light-infrared image pair dataset and specific style visible light-infrared image pair dataset. In this embodiment, a large-scale infrared single light dataset, a large-scale visible light-infrared image pair dataset and a specific style visible light-infrared image pair dataset are constructed. Specifically, the infrared single light dataset contains 500,000 infrared single light pictures, the visible light-infrared image pair dataset contains 70,000 visible light-infrared pictures, and the specific style visible light-infrared image pair dataset contains 4,300 pictures. To ensure fairness, the specific style visible light-infrared image pair dataset is not included in the visible light-infrared image pair dataset.

[0044] S2: Constructing a diffusion model training framework based on progressive learning, and using the infrared single light dataset, the visible light-infrared image pair dataset and the specific style visible light-infrared image pair dataset constructed in step S1 to optimize the parameters of the diffusion model training framework to obtain the diffusion model framework. Figure 3 The dashed arrow in the diffusion model training framework represents the training order. In this embodiment, a diffusion model training framework based on progressive learning is constructed. The parameter optimization process includes training and testing processes. The training includes three stages of training to receive the visual language features of the visible light image and the style features of the specific infrared image and generate the corresponding high-quality infrared image.

[0045] The diffusion model training framework constructed as shown in Figure 3 includes a latent space encoding module, a latent space decoding module, a text encoder, a cross-attention layer model (referred to as a cross-attention layer in Figure 3 ), a first-stage denoising network, a second-stage denoising network and a third-stage denoising network. The three-stage denoising networks have the same overall architecture. In the training process, the latent space encoding module, the latent space decoding module, the text encoder and the cross-attention layer model do not participate in parameter updating, and only the three denoising networks are updated in the training process.

[0046] Specifically, step S2 includes the following steps: S21: First-stage training of the diffusion model training framework based on the large-scale infrared single light dataset. First, the objective function for first-stage training is constructed as follows: , wherein, represents the objective function for fine-tuning the diffusion model training framework in the first stage, represents the mean, represents the input image (generally refers to the infrared single light data of the visible light image, the noise image, the segmentation image of the visible light image or the visible light image), represents the true noise, represents the guided condition Down predicting the noise of the denoising network, denoising network, denoising network, noise map of the step, denoising steps, guiding condition, representing a standard normal distribution.

[0047] Subsequently, the random noise map and the keyword "infrared" are input into the diffusion model training framework, at this time the diffusion model training framework updates the guiding condition in the objective function to be the keyword "infrared", and the infrared image is taken as the training target. The diffusion model training framework is fine-tuned in the form of low-rank matrix decomposition, so that the diffusion model training framework has the ability to generate infrared images according to the keyword "infrared". The formula of low-rank matrix decomposition is as follows: , , wherein, denotes the model parameters of the first-stage denoising network in the diffusion model training framework after updating, denotes the model parameters of the original first-stage denoising network, denotes the updated parameters of the first-stage denoising network before and after updating, and denotes the matrix obtained by performing low-rank decomposition on the updated parameter matrix, , and denotes the dimension of the first-stage denoising network model parameters, denotes the size of the intrinsic rank, .

[0048] S22: Based on the large-scale visible-infrared image pair dataset, the diffusion model training framework is trained in the second stage. The noise map and the visible light image are combined in the form of channel addition at this time. The diffusion model training framework updates the objective function in step S21, but the guiding condition is the visible light image. The corresponding infrared image is taken as the training target. The diffusion model training framework extracts the universal mapping relationship between the visible light image and the corresponding infrared image, so that the diffusion model training framework has the ability to convert the visible light image into the infrared image.

[0049] S23: Construct a visual language understanding module to obtain an image text description of the visible light image and obtain a segmentation image using a pre-trained segmentation model. The segmentation image, the visible light image and the random noise image are combined in a channel addition manner, and the image text description is introduced in the form of cross attention. At this time, the diffusion model training framework updates the objective function as in step S21, but the guide condition is the visual language information of the image and the visible light image, and the infrared image is taken as the training target. The style information is obtained based on a specific style visible light-infrared image pair dataset, and the diffusion model training framework is used to fuse the visual language information and the style information to obtain an infrared image of a specific style. The introduction of multiple conditions adopts a classifier-free guidance strategy loss function, which is expressed as follows: , wherein, represents the classifier-free guidance strategy loss function, represents the prediction of the diffusion model training framework for the condition-guided diffusion, represents the noise image of step, represents the visible light image in the guide condition, represents the segmentation image in the guide condition, represents the image text description in the guide condition, represents the weight of the visible light image in the guide condition, represents the weight of the segmentation image in the guide condition, represents the weight of the image text description in the guide condition, represents that the guide condition is empty.

[0050] S24: Test the diffusion model training framework in the specific style visible light-infrared image pair dataset. Specifically, Frechet Inception Distance, Peak Signal-to-Noise Ratio and Structural Similarity are used as image generation quality evaluation indexes to evaluate the generation performance of the visible light image to infrared image based on the progressive learning and visual language understanding. The formula of Frechet Inception Distance is as follows: , , , wherein, represents the feature of the real infrared image, represents the output mean value of the Inception feature extractor, represents the real infrared image, represents the feature of the generated infrared image, represents the generated infrared image, represents the Frechet Inception Distance, a mean value representing the real infrared image feature, a mean value representing the generated infrared image feature, a trace of a matrix, a covariance representing the real infrared image feature, a covariance representing the generated infrared image feature.

[0051] The peak signal-to-noise ratio formula is: , , wherein, a mean square error between the real infrared image and the generated infrared image, a length of the infrared image, a width of the infrared image, a pixel of the real infrared image, a pixel of the generated infrared image, a maximum numerical value of the image color, a peak signal-to-noise ratio.

[0052] The structural similarity formula is: , wherein, the structural similarity, the real infrared image, the generated infrared image, a mean value of the real infrared image, a mean value of the generated infrared image, a variance of the real infrared image, a variance of the generated infrared image, a covariance between the real infrared image and the generated infrared image, , two constants.

[0053] S3: image encoding based on self-attention mapping of the visible light image through the visual language understanding module to obtain image features, and decoding the image features in a manner of generating a control flag through the image features and the preset text description to obtain an image text description.

[0054] S4: converting the preset noise map, the visible light image, the segmentation map of the visible light image, the infrared single light data of the visible light image, and the image text description into three styles of infrared images through the diffusion model framework.

[0055] The above method first constructs an infrared single light dataset, a visible light-infrared image pair dataset and a specific style visible light-infrared image pair dataset, and uses these datasets to perform three-stage parameter optimization on the diffusion model training framework to obtain the diffusion model framework, thereby effectively combining the visual language information provided by the multi-modal visual language large model, realizing semantic-driven conversion of a visible light image to an infrared image. Meanwhile, the three-stage progressive learning framework can solve the problem of too large infrared imaging style difference between different datasets, and the collection of a large-scale infrared single light dataset, a large-scale visible light-infrared image pair dataset and a specific style visible light-infrared image pair dataset supports the entire training process, realizing high-quality conversion of a visible light image to an infrared image, and overcoming the technical defects of the prior art in the aspects of semantic information utilization and style control.

[0056] The visible and infrared light image conversion system based on the diffusion model disclosed in the embodiment fuses the image text description generated by the visual language understanding module to generate the text description of the image, and uses the image text description as the control flag of the diffusion model algorithm, so that the diffusion model algorithm executed by the diffusion model framework can accurately regulate the style features of the infrared image according to the semantic instructions, solving the technical defects of the traditional single style conversion in the aspect of style control. Moreover, the diffusion model framework combines the visible light image, the segmentation image and the infrared single light data, significantly reduces the error accumulation of the traditional multi-stage conversion, and the image conversion speed and the structural similarity index can be improved under the same hardware conditions. Furthermore, the image encoding technology using the self-attention mapping deeply binds the text description and the visual features to fully utilize the semantic information, and the control flag greatly improves the semantic consistency of the key regions of the generated infrared image, avoiding the problem of disconnection between the text and the image in the traditional method.

[0057] In the description of the embodiments of the present application, it should be explained that, in the description of the present application, the terms indicating the direction or position relationship are based on the direction or position relationship shown in the drawings, which is only for the convenience of description, and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0058] In the description of the application, the description of the terms "one embodiment", "some embodiments", "in this embodiment", "specific example", or "some examples" and the like means that the specific features, mechanisms, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. In the description, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0059] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A system for converting visible and infrared light images based on a diffusion model, characterized in that, The application comprises: a visual language understanding module configured to perform self-attention mapping-based image coding on a visible light image to obtain image features, then fuse preset text description generation control flags on the image features, and perform decoding to obtain an image text description; a diffusion model framework in communication with the visual language understanding module and configured to convert a preset noise image, the visible light image, a segmentation image of the visible light image, infrared single-light data of the visible light image, and the image text description into three styles of infrared images through a diffusion model algorithm. The segmentation image of the visible light image is a semantic segmentation image, an instance segmentation image, or a panorama segmentation image of the visible light image, and the infrared single-light data of the visible light image is an infrared light image obtained by performing visible light-infrared light image conversion on the visible light image.

2. The diffuse model based visible and infrared light image conversion system of claim 1, wherein, The visual language understanding module comprises: an image encoder configured to perform self-attention mapping-based image coding on the visible light image to obtain image features; a text decoder in communication with the image encoder and the diffusion model framework and configured to first fuse preset text description generation control flags on the image features, and then perform decoding on the image features fused with the text description generation control flags to obtain an image text description.

3. The diffuse model based visible to infrared image conversion system of claim 2, wherein, The image encoder comprises a self-attention module, an encoding controller, and a feedforward module one connected in series, the encoding controller is also in communication with the text decoder, and the encoding controller is configured to call the self-attention module to perform self-attention mapping processing on the visible light image to obtain self-attention features, sum the self-attention features with the visible light image to obtain a summed image, then call the feedforward module one to perform mapping processing of a feedforward network model on the summed image to obtain a feedforward feature map, and sum the feedforward feature map with the summed image to obtain the image features.

4. The diffuse model based visible and infrared image conversion system of claim 3, wherein, The text decoder comprises a causal attention module, a cross-attention module, a feedforward module two, and a decoding controller, the decoding controller is in communication with the causal attention module, the cross-attention module, the feedforward module two, and the diffusion model framework, and the cross-attention module is also in communication with the encoding controller; the decoding controller is configured to call the causal attention module to perform causal attention mapping on the text description generation control flags to obtain causal attention features, sum the causal attention features with the text description generation control flags to obtain fusion features one, then call the cross-attention module to perform cross-attention mapping on the fusion features one and the image features to obtain cross-features, sum the cross-features with the fusion features one to obtain fusion features two, and finally call the feedforward module two to perform mapping processing of a feedforward network model on the fusion features two to obtain a feedforward text, and sum the feedforward text with the fusion features two to obtain the image text description.

5. The diffuse model based visible to infrared image conversion system of claim 4, wherein, The diffusion model framework comprises: a latent space encoding module configured to encode the noise map, the visible light map, a segmentation map of the visible light map, and infrared single-light data of the visible light map in a latent space to obtain encoding one, encoding two, and encoding three; a denoising module in communication with the decoding controller and the latent space encoding module, and configured to simultaneously run denoising on the encoding one to obtain denoising result one, denoising on the encoding two to obtain denoising result two, and denoising on a fusion result of the image text description and the encoding three to obtain denoising result three; a latent space decoding module in communication with the denoising module, and configured to respectively decode the denoising result one, the denoising result two, and the denoising result three in the latent space to obtain infrared images of three styles.

6. The diffuse model based visible and infrared image conversion system of claim 5, wherein, The latent space encoding module is configured to run as follows: performing a mapping operation of a latent space encoding algorithm on the infrared single-light data of the visible light map to obtain the encoding one; performing a sum operation on the noise map and a mapping operation result of the latent space encoding algorithm performed on the visible light map to obtain the encoding two; performing a sum operation on the noise map, a mapping operation result of the latent space encoding algorithm performed on the visible light map, and a mapping operation result of the latent space encoding algorithm performed on the segmentation map of the visible light map to obtain the encoding three.

7. Diffusion model based visible and infrared light image conversion system according to claim 5 or 6, characterized in that, The denoising module includes: a text encoder in communication with the decoding controller, and configured to encode the image text description to obtain text features; a cross-attention layer model in communication with the text encoder, and configured to perform cross-attention mapping on the text features to obtain cross-text features; a first-stage denoising network in communication with the latent space encoding module, and configured to denoise the encoding one to obtain the denoising result one; a second-stage denoising network in communication with the latent space encoding module, and configured to denoise the encoding two to obtain the denoising result two; a third-stage denoising network in communication with the cross-attention layer model and the latent space encoding module, and configured to denoise a fusion result of the cross-text features and the encoding three to obtain the denoising result three.

8. The diffuse model based visible and infrared light image conversion system of claim 7, wherein, The first-stage denoising network, the second-stage denoising network, and the third-stage denoising network each include a denoising encoder and a denoising decoder, the denoising encoder is a network structure formed by an input convolutional layer model, multiple cross-attention down-sampling modules, a down-sampling module, and a bottleneck module connected in series, and the denoising decoder is a network structure formed by an up-sampling module, multiple cross-attention up-sampling modules, and an output convolutional layer model connected in series; the input convolutional layer model in the first-stage denoising network and the second-stage denoising network is in communication with the latent space encoding module, the input convolutional layer model in the third-stage denoising network is in communication with the cross-attention layer model and the latent space encoding module at the same time, the up-sampling module is in communication with the bottleneck module, and the output convolutional layer model is in communication with the latent space decoding module; The cross-attention down-sampling modules and the cross-attention up-sampling modules are in one-to-one communication.

9. The diffuse model based visible and infrared image conversion system of claim 8, wherein, The latent space decoding module comprises three latent space decoders, which are in one-to-one communication with the output convolutional layer model.

10. A method for converting a visible and infrared light image based on a diffusion model, characterized in that, The diffusion model-based visible and infrared image conversion system according to any one of claims 1-9, comprising the following steps: S1: constructing an infrared single-light dataset, a visible-infrared image pair dataset, and a specific style visible-infrared image pair dataset; S2: constructing a diffusion model training framework based on progressive learning, and using the infrared single-light dataset, the visible-infrared image pair dataset, and the specific style visible-infrared image pair dataset constructed in step S1 to optimize the parameters of the diffusion model training framework to obtain a diffusion model framework; S3: performing image coding based on self-attention mapping on the visible light image through a visual language understanding module to obtain image features, and decoding the image features by fusing a preset text description generation control flag into the image features to obtain an image text description; S4: converting a preset noise map, the visible light image, a segmentation map of the visible light image, infrared single-light data of the visible light image, and the image text description into infrared images of three styles through the diffusion model framework.

Images