A Text-Guided Infrared Image Generation Method Based on a Diffusion Model

Abstract

This invention discloses a method for generating infrared images based on a diffusion model text-guided network, comprising: 1. acquiring training set images; 2. constructing a diffusion model text-guided network; 3. extracting features from the visible light infrared training set and image weather text descriptions through the diffusion model text-guided network; 4. constructing a total loss in the feature extraction process of the visible light infrared training set and image weather text descriptions through the diffusion model text-guided network; 5. training the diffusion model text-guided network based on the total loss; and 6. generating infrared images based on the trained diffusion model text-guided network. This invention features a simple and rationally designed method that utilizes weather text descriptions of weather conditions and couples multimodal information from visible light images and weather text descriptions into the diffusion model text-guided network for training, thereby enabling the generation of infrared images under different weather conditions and improving the quality of generated infrared images.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for generating infrared images based on a diffusion model and text guidance. Background Technology

[0002] Visible light images generally only reflect the color and texture of an object, while infrared radiation temperature is related not only to the material properties of the object but also to the weather conditions at the time. However, this information is difficult to obtain solely from visible light images. Infrared images are obtained by "measuring" the heat radiated outwards by an object, and they can better reflect the thermal characteristics of the target. Based on the collected infrared image data, it can be found that infrared images exhibit different characteristics under different weather conditions.

[0003] Infrared images can provide reliable and diverse basic data for tasks such as infrared imaging guidance, infrared target detection, and target tracking. However, high-fidelity infrared image generation technology can not only effectively save the cost of acquiring infrared images, but also provide rich data support for subsequent research. Infrared image generation is a complex process that also needs to consider weather and environmental factors. Currently, the quality of infrared images generated based on visible light images is low. Therefore, it is necessary to explore infrared image generation methods that couple multimodal information inputs.

[0004] Therefore, a well-designed infrared image generation method based on diffusion model text guidance is needed. This method utilizes weather text descriptions of weather conditions and couples multimodal information from visible light images and weather text descriptions into a diffusion model text guidance network for training. This enables the generation of infrared images under different weather conditions and improves the quality of generated infrared images. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an infrared image generation method based on diffusion model text guidance, which addresses the shortcomings of the prior art. The method has simple steps and reasonable design. It uses weather text to describe weather conditions and couples the multimodal information of visible light images and weather text descriptions into the diffusion model text guidance network for training, thereby realizing the generation of infrared images under different weather conditions and improving the generation quality of infrared images.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: an infrared image generation method based on diffusion model text guidance, the method comprising the following steps:

[0007] Step 1: Obtaining the training set images:

[0008] Visible light images and corresponding infrared images are acquired from the target area under different weather conditions, and the corresponding infrared images are recorded as infrared real images to obtain a visible light infrared training set; wherein, the visible light infrared training set includes visible light images and infrared real images;

[0009] Step 2: Construct a diffusion model text guidance network; wherein the diffusion model text guidance network includes a text encoder, a VAE encoder, a UNet and a VAE decoder, as well as a first image discriminator D, a second image discriminator Dgc and a text discriminator DT;

[0010] Step 3: Extract features from the visible light and infrared training set and the image weather text descriptions using a diffusion model text-guided network;

[0011] Step 4: Construct the total loss in the visible light infrared training set and image weather text descriptions after feature extraction by the diffusion model text-guided network:

[0012] according to The total loss was obtained. ;in, This represents the first loss coefficient. This represents the second loss coefficient. This represents the third loss coefficient. This represents the fourth loss coefficient. This represents the fifth loss coefficient. This represents the sixth loss coefficient. This represents the LSGANs loss of the first image discriminator D. This represents the LSGANs loss of the second image discriminator, Dgc. This represents the LSGANs loss of the text discriminator DT. Represents geometrically consistent loss. This represents the CLIP text-image association loss. This represents the loss from the identity mapping;

[0013] Step 5: Train the diffusion model text guidance network based on the total loss;

[0014] Step 6: Generate infrared images based on the trained diffusion model text-guided network.

[0015] The above-mentioned infrared image generation method based on a diffusion model and text guidance, further includes step three, which is as follows:

[0016] Step 301: Based on the visible light image x and the infrared real image y of the target domain, obtain the visible light image weather text description and the infrared image weather text description corresponding to the target domain; wherein, the visible light image weather text description and the infrared image weather text description are both English fields;

[0017] Step 302: Input any visible light image x into the VAE encoder using a computer to obtain the encoded output feature map; input the infrared image weather text description into the text encoder for processing using a computer to obtain the text output features;

[0018] The computer is used to input the encoded output feature map and the text output feature into UNet for processing, and the UNet output feature map is obtained.

[0019] Step 303: Use a computer to input the UNet output feature map into the VAE decoder for processing to obtain the first decoded output feature map, and denote the first decoded output feature map as the infrared generated image y';

[0020] Step 304: Use a computer to perform geometric transformation on the visible light image x to obtain the transformed visible light image fx. Then, input the transformed visible light image fx into the diffusion model text guidance network according to the methods in steps 302 to 303 to obtain the second decoding output feature map. The second decoding output feature map is denoted as the transformed infrared generated image fy.

[0021] Step 305: The computer is used to input the generated infrared image y' and the real infrared image y into the first image discriminator D for processing, to obtain the probability feature map D(y') output by the first discriminator D of the generated infrared image and the probability feature map D(y) output by the first discriminator D of the real infrared image.

[0022] Step 306: The transformed infrared generated image fy and the infrared real image y are input into the second image discriminator Dgc for processing by a computer, to obtain the probability feature map Dgc(fy) output by the transformed infrared generated image after passing through the second discriminator Dgc and the probability feature map Dgc(y) output by the infrared real image after passing through the second discriminator Dgc.

[0023] Step 307: Using a computer, the real text output features, fake text output features, intermediate feature maps output by the first discriminator D from the real infrared image, and intermediate feature maps output by the first discriminator D from the generated infrared image are combined and input into the text discriminator DT for processing to obtain the various probability feature maps output by the text discriminator.

[0024] Step 308: Perform inverse geometric transformation on the transformed infrared generated image fy to obtain the inverse geometric transformed infrared generated image z';

[0025] The infrared generated image y' is processed by geometric transformation to obtain the geometrically transformed infrared generated image z;

[0026] Step 309: Input the visible light image x and the corresponding visible light image weather text description into the diffusion model text guidance network according to the methods in steps 302 to 303 to obtain the visible light generated image xg;

[0027] The infrared real image y and the corresponding infrared image weather text description are input into the diffusion model text guidance network according to the methods in steps 302 to 303 to obtain the infrared output image yh; wherein, multiple pairs of infrared real images and infrared image weather text descriptions are input into the CLIP network model for training to obtain the trained CLIP network model.

[0028] The above-mentioned infrared image generation method based on diffusion model text guidance, further, step 307, is as follows:

[0029] Step 3071: Use a computer to record the infrared image weather text description corresponding to the target domain as the real infrared image weather text description, and record the infrared image weather text description after the weather conditions are changed as the fake infrared image weather text description. The real infrared image weather text description and the fake infrared image weather text description are processed by a text encoder to obtain the real text output features and the fake text output features.

[0030] Step 3072: Combine the real text output features and the intermediate feature map output by the first discriminator D of the real infrared image into a positive sample; combine the real text output features and the intermediate feature map output by the first discriminator D of the real infrared image into a first negative sample; and combine the fake text output features and the intermediate feature map output by the first discriminator D of the real infrared image into a second negative sample.

[0031] Step 3073: Input the positive sample, the first negative sample, and the second negative sample into the text discriminator DT for processing to obtain the probability feature map DT(z) output by the text discriminator DT for the positive sample, the probability feature map DT(f1) output by the text discriminator DT for the first negative sample, and the probability feature map DT(f2) output by the text discriminator DT for the second negative sample.

[0032] The above-mentioned infrared image generation method based on diffusion model text guidance further includes a text discriminator DT comprising a linear layer, a preprocessing layer, an L2 normalization layer, and a self-attention layer. The linear layer comprises an MPL layer and a text feature reshaping module. The preprocessing layer comprises a first convolutional layer, a second convolutional layer, a first activation function layer, a third convolutional layer, and a second activation function layer connected in sequence. The self-attention layer comprises a fourth convolutional layer, a fifth convolutional layer, and a sixth convolutional layer connected in parallel.

[0033] The output of the self-attention layer is also connected to a seventh convolutional layer and a head layer. The head layer includes an eighth convolutional layer, instance normalization, a third activation function layer, and a ninth convolutional layer.

[0034] The above-mentioned infrared image generation method based on diffusion model text guidance, further, step 3073, is as follows:

[0035] Step A01: Use a computer to enhance the real text output features through an MPL layer to obtain text enhancement features;

[0036] Step A02: The computer is used to process the text enhancement features through the text feature reshaping module and convert them into a text reshaping feature map that matches the intermediate feature map output by the first image discriminator.

[0037] Step A03: Using a computer, the text reconstructed feature map and the infrared real image are processed by tensor concatenation of the intermediate feature map output by the first discriminator D to obtain the concatenated feature map.

[0038] Step A04: Using a computer, the stitched feature map is input into the first convolutional layer, the second convolutional layer, the first activation function layer, the third convolutional layer, and the second activation function layer for preprocessing in sequence to obtain the preprocessed feature map;

[0039] Step A05: Use a computer to obtain the numerical values ​​corresponding to all channels at any pixel from the preprocessed feature map, and record them as the pixel channel dimension vector; then perform L2 normalization on the pixel channel dimension vector to obtain the pixel channel dimension normalized vector.

[0040] Step A06: Repeat step A05 multiple times to obtain the normalized vectors of all pixel channel dimensions. Then, combine the normalized vectors of all pixel channel dimensions according to the pixel order to obtain the combined feature map. Then, perform tensor transformation on the combined feature map to obtain a normalized feature map of the same size as the preprocessed feature map.

[0041] Step A07: The normalized feature map is processed by a computer through the fourth convolutional layer to obtain the first output feature matrix;

[0042] The normalized feature map is processed through the fifth convolutional layer to obtain the second output feature matrix;

[0043] The normalized feature map is processed through the sixth convolutional layer to obtain the third output feature matrix;

[0044] Step A08: Using a computer, the first output feature matrix, the second output feature matrix, and the third output feature matrix are all subjected to the first tensor transformation to obtain the query matrix Q, the key matrix, and the value matrix;

[0045] Step A09: Use a computer to add a row of key vectors to the row direction of the key matrix to obtain the added key matrix K;

[0046] A computer is used to add a row of value vectors to the rows of the value matrix, resulting in the added value matrix V;

[0047] Step A10: Using a computer, the query matrix Q, the added key matrix K, and the added value matrix V are... The output features of the self-attention layer are obtained. ;in, It is the column dimension of the key matrix;

[0048] Step A11: The output features of the self-attention layer are transformed by a computer through a second tensor transformation to obtain a tensor-transformed feature map, which is then processed by the seventh convolutional layer to obtain an intermediate transition feature map.

[0049] Step A12: Use a computer to pass the intermediate transition feature map through the Head layer to obtain the probability feature map DT(z) output by the text discriminator DT for positive samples.

[0050] Step A13: Following the methods of steps A01 to A12, obtain the probability feature map DT(f1) output by the text discriminator DT for the first negative sample and the probability feature map DT(f2) output by the text discriminator DT for the second negative sample.

[0051] The above-mentioned infrared image generation method based on diffusion model text guidance, further, step four, is as follows:

[0052] Step 401, according to The LSGANs loss of the first discriminator D is obtained. ;in, Let represent the value of the i-th pixel in the probability feature map D(y) output by the first discriminator D of the real infrared image. Let represent the i-th pixel value in the probability feature map D(y') output by the first discriminator D of the infrared generated image, where i and m are positive integers; 1≤i≤m, and m is the total number of pixels in the probability feature map;

[0053] Step 402, according to The LSGANs loss of the second discriminator Dgc is obtained. ;in, This represents the value of the i-th pixel in the probability feature map Dgc(y) output by the second discriminator Dgc of the real infrared image. This represents the value of the i-th pixel in the probability feature map Dgc(fy) output by the second discriminator Dgc after the transformed infrared generated image is processed.

[0054] Step 403, according to The LSGANs loss of the text discriminator DT is obtained. ;in, This represents the i-th pixel value in the probability feature map DT(z) output by the text discriminator DT after a positive sample passes through it. This represents the i-th pixel value in the probability feature map DT(f1) output by the text discriminator DT after the first negative sample has passed through it. This represents the i-th pixel value in the probability feature map DT(f2) output by the text discriminator DT after the second negative sample is processed;

[0055] Step 404: Based on the infrared image weather text description and the infrared generated image, the trained CLIP network model is used to obtain the CLIP loss. ; and according to The CLIP text-image association loss is obtained. ;

[0056] Step 405, according to The geometrically consistent loss is obtained. ;in, Let represent the 1-norm between the infrared generated image y' and the infrared generated image z' after geometric inverse transformation. This represents the 1-norm between the transformed infrared image fy and the geometrically transformed infrared image z. The LPIPS distance represents the distance between the infrared generated image y' and the infrared generated image z' after inverse geometric transformation. This represents the LPIPS distance between the transformed infrared image fy and the geometrically transformed infrared image z;

[0057] Step 406, according to The identity mapping loss is obtained. ;in, This represents the 1-norm between the visible light image x and the generated visible light image xg. The 1-norm represents the relationship between the true infrared image y and the output infrared image yh.

[0058] The aforementioned infrared image generation method based on diffusion model text guidance further includes, in a further embodiment, an input convolutional layer, a first DownEncoderBlock2D, a second DownEncoderBlock2D, a third DownEncoderBlock2D, and a fourth DownEncoderBlock2D.

[0059] The VAE decoder includes an output convolutional layer plus UNetMidBlock2D, a first UpDecoderBlock2D, a second UpDecoderBlock2D, a third UpDecoderBlock2D, and a fourth UpDecoderBlock2D;

[0060] A first cross-layer driven fusion module is set between the output of the first DownEncoderBlock2D and the input of the fourth UpDecoderBlock2D; a second cross-layer driven fusion module is set between the output of the second DownEncoderBlock2D and the input of the third UpDecoderBlock2D; a third cross-layer driven fusion module is set between the output of the third DownEncoderBlock2D and the input of the third UpDecoderBlock2D; a zero convolutional layer is set between the output of the fourth DownEncoderBlock2D and the input of the first UpDecoderBlock2D; the output of the zero convolutional layer is also input to the third cross-layer driven fusion module; the output of the third cross-layer driven fusion module is also input to the second cross-layer driven fusion module; and the output of the second cross-layer driven fusion module is also input to the first cross-layer driven fusion module.

[0061] The first, second, and third cross-layer driven fusion modules have the same structure and each includes an upsampling module, two tensor splitting modules, four LDC modules, and one LCBG module. Each LDC module consists of layer normalization and dilated convolution, and each LCBG module consists of layer normalization, two-dimensional convolution, batch normalization, and GELU activation function.

[0062] In the above-mentioned infrared image generation method based on diffusion model text guidance, further, in step 302, when the visible light image x is input into the VAE encoder to obtain the encoded output feature map, the visible light image x is first input into the input convolutional layer of the VAE encoder to obtain the first feature map. The first feature map is then passed through the first DownEncoderBlock2D, the second DownEncoderBlock2D, the third DownEncoderBlock2D and the fourth DownEncoderBlock2D in sequence to obtain the second feature map, the third feature map, the fourth feature map and the fifth feature map.

[0063] In step 303, when the computer inputs the UNet output feature map into the VAE decoder for processing, the specific steps are as follows:

[0064] The fifth feature map is passed through a zero convolutional layer to output the first transformed feature map. The fourth feature map and the first transformed feature map are passed through the third cross-layer driven fusion module to output the second transformed feature map.

[0065] The third feature map and the second transformed feature map are processed by the second cross-layer driving fusion module to output the third transformed feature map, and the second feature map and the third transformed feature map are processed by the first cross-layer driving fusion module to output the fourth transformed feature map;

[0066] The UNet output feature map, the first transformed feature map, the second transformed feature map, the third transformed feature map, and the fourth transformed feature map are processed by the VAE decoder to obtain the first decoded output feature map. The UNet output feature map is then processed by the output convolutional layer of the VAE decoder and then processed by UNetMidBlock2D to obtain the sixth feature map. The sixth feature map and the first transformed feature map are added together and then processed by the first UpDecoderBlock2D to obtain the seventh feature map. The seventh feature map and the second transformed feature map are added together and then processed by the second UpDecoderBlock2D to obtain the eighth feature map. The eighth feature map and the third transformed feature map are added together and then processed by the third UpDecoderBlock2D to obtain the ninth feature map. The ninth feature map and the fourth transformed feature map are added together and then processed by the fourth UpDecoderBlock2D to obtain the tenth feature map. The tenth feature map is then processed by subsequent modules of the VAE decoder to output the first decoded output feature map.

[0067] The aforementioned infrared image generation method based on a diffusion model and text guidance further involves passing the fourth feature map and the first transformed feature map through a third cross-layer driving fusion module to output a second transformed feature map. The specific process is as follows:

[0068] Step A: The fourth feature map is split into four fourth feature map segments with the same number of channels by the first tensor splitting module; the first transformed feature map is split into four first transformed upsampled feature maps by the upsampling module, and the first transformed upsampled feature map is split into four first transformed feature map segments with the same number of channels by the second tensor splitting module.

[0069] Step B: Concatenate the four fourth feature split images with the same number of channels and the four first transformation feature split images with the same number of channels respectively to obtain four pre-concatenated feature images with the same number of channels;

[0070] Step C: The first pre-stitched feature map is passed through the first LDC module to output the first extracted feature map; the second pre-stitched feature map is passed through the second LDC module to output the second extracted feature map; the third pre-stitched feature map is passed through the third LDC module to output the third extracted feature map; the fourth pre-stitched feature map is passed through the fourth LDC module to output the fourth extracted feature map.

[0071] Step D: The first, second, third, and fourth extracted feature maps are concatenated to obtain the concatenated feature map.

[0072] Step E: Pass the concatenated feature map through the LCBG module to obtain the second transformed feature map;

[0073] Following steps A to E, the third feature map and the second transformed feature map are processed by the second cross-layer driving fusion module to output the third transformed feature map.

[0074] Following steps A to E, the second feature map and the third transformed feature map are processed by the first cross-layer driving fusion module to output the fourth transformed feature map.

[0075] The above-mentioned infrared image generation method based on diffusion model text guidance further includes step six, which specifically involves using a computer to input the subsequent visible light image and the required infrared image weather text description into the trained diffusion model text guidance network's text encoder, VAE encoder, UNet, and VAE decoder for processing, thereby obtaining an infrared generated image corresponding to the required infrared image weather text description.

[0076] Compared with the prior art, the present invention has the following advantages:

[0077] 1. The method of the present invention has simple steps and reasonable design, and solves the problem of low quality in infrared image generation based on visible light images.

[0078] 2. This invention employs a diffusion model text-guided network comprising a text encoder, a VAE encoder, a UNet and a VAE decoder, as well as a first image discriminator D, a second image discriminator Dgc, and a text discriminator DT. By combining the discriminator with a one-step diffusion model, the complexity of adversarial training is avoided, the extraction process of the diffusion model is optimized, the number of iterations is reduced, and the generation efficiency is improved. In addition, the first image discriminator D and the second image discriminator Dgc are used to distinguish between the original image and the geometrically transformed image, guiding the generation of more realistic images and improving the robustness of the network. By setting up a text discriminator that accepts text features extracted by the text encoder and image features output from the intermediate layers of the image discriminator, effective input of cross-modal features is achieved, reducing the training difficulty of direct input and reducing training costs.

[0079] 3. The text discriminator of the present invention considers the positive samples of the combination of real text output features and the intermediate feature map output by the first discriminator D of the real infrared image, the first negative samples of the combination of real text output features and the intermediate feature map output by the first discriminator D of the real infrared image, and the second negative samples of the combination of fake text output features and the intermediate feature map output by the first discriminator D of the real infrared image. Through this asymmetric negative sample strategy, the discriminator is forced to learn more fine-grained modal association features.

[0080] 4. The total loss in this invention considers the LSGANs loss of the first image discriminator, the LSGANs loss of the second image discriminator, the LSGANs loss of the text discriminator, geometric consistency loss, CLIP text-image association loss, and identity mapping loss. The two image discriminators are responsible for judging the authenticity of the infrared generated image and the transformed infrared generated image, while the text discriminator and CLIP are responsible for judging the association between weather conditions and infrared generated images. The LSGANs loss of the text discriminator and the CLIP text-image association loss establish the strong and weak association between weather text descriptions and infrared images, which strengthens the control of grayscale features in infrared generated images. Furthermore, the use of geometric consistency loss can reduce the training burden and reduce equipment requirements, effectively control the number of model parameters, use less GPU memory during training, improve training efficiency, and accelerate network convergence. Identity mapping not only plays a role in stabilizing network training but also enhances the network's image generation ability under different conditions.

[0081] 5. This invention utilizes weather text descriptions of weather conditions and couples multimodal information from visible light images and weather text descriptions into a diffusion model text-guided network for training, thereby enabling the generation of infrared images under different weather conditions by altering the weather conditions.

[0082] In summary, the method of this invention is simple in steps and reasonable in design. It uses weather text to describe weather conditions and couples multimodal information from visible light images and weather text descriptions into a diffusion model text-guided network for training, thereby enabling the generation of infrared images under different weather conditions and improving the quality of infrared image generation.

[0083] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0084] Figure 1 This is a flowchart of the method of the present invention.

[0085] Figure 2 This is a schematic diagram of the structure of the diffusion model text guidance network of this invention.

[0086] Figure 3This is a schematic diagram of the text discriminator DT of the present invention.

[0087] Figure 4 This is a schematic diagram showing the positions of the zero convolutional layer, the first cross-layer driven fusion module, the second cross-layer driven fusion module, and the third cross-layer driven fusion module in the diffusion model text guidance network of the present invention.

[0088] Figure 5 This is a schematic diagram of the structure of the first cross-layer driving fusion module, the second cross-layer driving fusion module and the third cross-layer driving fusion module of the present invention. Detailed Implementation

[0089] like Figures 1 to 2 As shown, the infrared image generation method based on diffusion model text guidance of the present invention includes the following steps:

[0090] Step 1: Obtaining the training set images:

[0091] Visible light images and corresponding infrared images are acquired from the target area under different weather conditions, and the corresponding infrared images are recorded as infrared real images to obtain a visible light infrared training set; wherein, the visible light infrared training set includes visible light images and infrared real images;

[0092] Step 2: Construct a diffusion model text guidance network; wherein the diffusion model text guidance network includes a text encoder, a VAE encoder, a UNet and a VAE decoder, as well as a first image discriminator D, a second image discriminator Dgc and a text discriminator DT;

[0093] Step 3: Extract features from the visible light and infrared training set and the image weather text descriptions using a diffusion model text-guided network;

[0094] Step 4: Construct the total loss in the visible light infrared training set and image weather text descriptions after feature extraction by the diffusion model text-guided network:

[0095] according to The total loss was obtained. ;in, This represents the first loss coefficient. This represents the second loss coefficient. This represents the third loss coefficient. This represents the fourth loss coefficient. This represents the fifth loss coefficient. This represents the sixth loss coefficient. This represents the LSGANs loss of the first image discriminator D. This represents the LSGANs loss of the second image discriminator, Dgc. This represents the LSGANs loss of the text discriminator DT. Represents geometrically consistent loss. This represents the CLIP text-image association loss. This represents the loss from the identity mapping;

[0096] Step 5: Train the diffusion model text guidance network based on the total loss;

[0097] Step 6: Generate infrared images based on the trained diffusion model text-guided network.

[0098] In this embodiment, step three is as follows:

[0099] Step 301: Based on the visible light image x and the infrared real image y of the target domain, obtain the visible light image weather text description and the infrared image weather text description corresponding to the target domain; wherein, the visible light image weather text description and the infrared image weather text description are both English fields;

[0100] Step 302: Input any visible light image x into the VAE encoder using a computer to obtain the encoded output feature map; input the infrared image weather text description into the text encoder for processing using a computer to obtain the text output features;

[0101] The computer is used to input the encoded output feature map and the text output feature into UNet for processing, and the UNet output feature map is obtained.

[0102] Step 303: Use a computer to input the UNet output feature map into the VAE decoder for processing to obtain the first decoded output feature map, and denote the first decoded output feature map as the infrared generated image y';

[0103] Step 304: Use a computer to perform geometric transformation on the visible light image x to obtain the transformed visible light image fx. Then, input the transformed visible light image fx into the diffusion model text guidance network according to the methods in steps 302 to 303 to obtain the second decoding output feature map. The second decoding output feature map is denoted as the transformed infrared generated image fy.

[0104] Step 305: The computer is used to input the generated infrared image y' and the real infrared image y into the first image discriminator D for processing, to obtain the probability feature map D(y') output by the first discriminator D of the generated infrared image and the probability feature map D(y) output by the first discriminator D of the real infrared image.

[0105] Step 306: The transformed infrared generated image fy and the infrared real image y are input into the second image discriminator Dgc for processing by a computer, to obtain the probability feature map Dgc(fy) output by the transformed infrared generated image after passing through the second discriminator Dgc and the probability feature map Dgc(y) output by the infrared real image after passing through the second discriminator Dgc.

[0106] Step 307: Using a computer, the real text output features, fake text output features, intermediate feature maps output by the first discriminator D from the real infrared image, and intermediate feature maps output by the first discriminator D from the generated infrared image are combined and input into the text discriminator DT for processing to obtain the various probability feature maps output by the text discriminator.

[0107] Step 308: Perform inverse geometric transformation on the transformed infrared generated image fy to obtain the inverse geometric transformed infrared generated image z';

[0108] The infrared generated image y' is processed by geometric transformation to obtain the geometrically transformed infrared generated image z;

[0109] Step 309: Input the visible light image x and the corresponding visible light image weather text description into the diffusion model text guidance network according to the methods in steps 302 to 303 to obtain the visible light generated image xg;

[0110] The infrared real image y and the corresponding infrared image weather text description are input into the diffusion model text guidance network according to the methods in steps 302 to 303 to obtain the infrared output image yh; wherein, multiple pairs of infrared real images and infrared image weather text descriptions are input into the CLIP network model for training to obtain the trained CLIP network model.

[0111] In this embodiment, step 307 is specifically performed as follows:

[0112] Step 3071: Use a computer to record the infrared image weather text description corresponding to the target domain as the real infrared image weather text description, and record the infrared image weather text description after the weather conditions are changed as the fake infrared image weather text description. The real infrared image weather text description and the fake infrared image weather text description are processed by a text encoder to obtain the real text output features and the fake text output features.

[0113] Step 3072: Combine the real text output features and the intermediate feature map output by the first discriminator D of the real infrared image into a positive sample; combine the real text output features and the intermediate feature map output by the first discriminator D of the real infrared image into a first negative sample; and combine the fake text output features and the intermediate feature map output by the first discriminator D of the real infrared image into a second negative sample.

[0114] Step 3073: Input the positive sample, the first negative sample, and the second negative sample into the text discriminator DT for processing to obtain the probability feature map DT(z) output by the text discriminator DT for the positive sample, the probability feature map DT(f1) output by the text discriminator DT for the first negative sample, and the probability feature map DT(f2) output by the text discriminator DT for the second negative sample.

[0115] like Figure 3 As shown, in this embodiment, the text discriminator DT includes a linear layer, a preprocessing layer, an L2 normalization layer, and a self-attention layer. The linear layer includes an MPL layer and a text feature reshaping module. The preprocessing layer includes a first convolutional layer, a second convolutional layer, a first activation function layer, a third convolutional layer, and a second activation function layer connected in sequence. The self-attention layer includes a fourth convolutional layer, a fifth convolutional layer, and a sixth convolutional layer connected in parallel.

[0116] The output of the self-attention layer is also connected to a seventh convolutional layer and a head layer. The head layer includes an eighth convolutional layer, instance normalization, a third activation function layer, and a ninth convolutional layer.

[0117] In this embodiment, step 3073 is specifically performed as follows:

[0118] Step A01: Use a computer to enhance the real text output features through an MPL layer to obtain text enhancement features;

[0119] Step A02: The computer is used to process the text enhancement features through the text feature reshaping module and convert them into a text reshaping feature map that matches the intermediate feature map output by the first image discriminator.

[0120] Step A03: Using a computer, the text reconstructed feature map and the infrared real image are processed by tensor concatenation of the intermediate feature map output by the first discriminator D to obtain the concatenated feature map.

[0121] Step A04: Using a computer, the stitched feature map is input into the first convolutional layer, the second convolutional layer, the first activation function layer, the third convolutional layer, and the second activation function layer for preprocessing in sequence to obtain the preprocessed feature map;

[0122] Step A05: Use a computer to obtain the numerical values ​​corresponding to all channels at any pixel from the preprocessed feature map, and record them as the pixel channel dimension vector; then perform L2 normalization on the pixel channel dimension vector to obtain the pixel channel dimension normalized vector.

[0123] Step A06: Repeat step A05 multiple times to obtain the normalized vectors of all pixel channel dimensions. Then, combine the normalized vectors of all pixel channel dimensions according to the pixel order to obtain the combined feature map. Then, perform tensor transformation on the combined feature map to obtain a normalized feature map of the same size as the preprocessed feature map.

[0124] Step A07: The normalized feature map is processed by a computer through the fourth convolutional layer to obtain the first output feature matrix;

[0125] The normalized feature map is processed through the fifth convolutional layer to obtain the second output feature matrix;

[0126] The normalized feature map is processed through the sixth convolutional layer to obtain the third output feature matrix;

[0127] Step A08: Using a computer, the first output feature matrix, the second output feature matrix, and the third output feature matrix are all subjected to the first tensor transformation to obtain the query matrix Q, the key matrix, and the value matrix;

[0128] Step A09: Use a computer to add a row of key vectors to the row direction of the key matrix to obtain the added key matrix K;

[0129] A computer is used to add a row of value vectors to the rows of the value matrix, resulting in the added value matrix V;

[0130] Step A10: Using a computer, the query matrix Q, the added key matrix K, and the added value matrix V are... The output features of the self-attention layer are obtained. ;in, It is the column dimension of the key matrix;

[0131] Step A11: The output features of the self-attention layer are transformed by a computer through a second tensor transformation to obtain a tensor-transformed feature map, which is then processed by the seventh convolutional layer to obtain an intermediate transition feature map.

[0132] Step A12: Use a computer to pass the intermediate transition feature map through the Head layer to obtain the probability feature map DT(z) output by the text discriminator DT for positive samples.

[0133] Step A13: Following the methods of steps A01 to A12, obtain the probability feature map DT(f1) output by the text discriminator DT for the first negative sample and the probability feature map DT(f2) output by the text discriminator DT for the second negative sample.

[0134] In this embodiment, step four is as follows:

[0135] Step 401, according to The LSGANs loss of the first discriminator D is obtained. ;in, Let represent the value of the i-th pixel in the probability feature map D(y) output by the first discriminator D of the real infrared image. Let represent the i-th pixel value in the probability feature map D(y') output by the first discriminator D of the infrared generated image, where i and m are positive integers; 1≤i≤m, and m is the total number of pixels in the probability feature map;

[0136] Step 402, according to The LSGANs loss of the second discriminator Dgc is obtained. ;in, This represents the value of the i-th pixel in the probability feature map Dgc(y) output by the second discriminator Dgc of the real infrared image. This represents the value of the i-th pixel in the probability feature map Dgc(fy) output by the second discriminator Dgc after the transformed infrared generated image is processed.

[0137] Step 403, according to The LSGANs loss of the text discriminator DT is obtained. ;in, This represents the i-th pixel value in the probability feature map DT(z) output by the text discriminator DT after a positive sample passes through it. This represents the i-th pixel value in the probability feature map DT(f1) output by the text discriminator DT after the first negative sample has passed through it. This represents the i-th pixel value in the probability feature map DT(f2) output by the text discriminator DT after the second negative sample is processed;

[0138] Step 404: Based on the infrared image weather text description and the infrared generated image, the trained CLIP network model is used to obtain the CLIP loss. ; and according to The CLIP text-image association loss is obtained. ;

[0139] Step 405, according to The geometrically consistent loss is obtained. ;in, Let represent the 1-norm between the infrared generated image y' and the infrared generated image z' after geometric inverse transformation. This represents the 1-norm between the transformed infrared image fy and the geometrically transformed infrared image z. The LPIPS distance represents the distance between the infrared generated image y' and the infrared generated image z' after inverse geometric transformation. This represents the LPIPS distance between the transformed infrared image fy and the geometrically transformed infrared image z;

[0140] Step 406, according to The identity mapping loss is obtained. ;in, This represents the 1-norm between the visible light image x and the generated visible light image xg. The 1-norm represents the relationship between the true infrared image y and the output infrared image yh.

[0141] like Figure 4 and Figure 5 As shown, in this embodiment, the VAE encoder includes an input convolutional layer, a first DownEncoderBlock2D, a second DownEncoderBlock2D, a third DownEncoderBlock2D, and a fourth DownEncoderBlock2D;

[0142] The VAE decoder includes an output convolutional layer plus UNetMidBlock2D, a first UpDecoderBlock2D, a second UpDecoderBlock2D, a third UpDecoderBlock2D, and a fourth UpDecoderBlock2D;

[0143] A first cross-layer driven fusion module is set between the output of the first DownEncoderBlock2D and the input of the fourth UpDecoderBlock2D; a second cross-layer driven fusion module is set between the output of the second DownEncoderBlock2D and the input of the third UpDecoderBlock2D; a third cross-layer driven fusion module is set between the output of the third DownEncoderBlock2D and the input of the third UpDecoderBlock2D; a zero convolutional layer is set between the output of the fourth DownEncoderBlock2D and the input of the first UpDecoderBlock2D; the output of the zero convolutional layer is also input to the third cross-layer driven fusion module; the output of the third cross-layer driven fusion module is also input to the second cross-layer driven fusion module; and the output of the second cross-layer driven fusion module is also input to the first cross-layer driven fusion module.

[0144] The first, second, and third cross-layer driven fusion modules have the same structure and each includes an upsampling module, two tensor splitting modules, four LDC modules, and one LCBG module. Each LDC module consists of layer normalization and dilated convolution, and each LCBG module consists of layer normalization, two-dimensional convolution, batch normalization, and GELU activation function.

[0145] In this embodiment, when the visible light image x is input into the VAE encoder to obtain the encoded output feature map in step 302, the visible light image x is first input into the input convolutional layer of the VAE encoder to obtain the first feature map. The first feature map is sequentially passed through the first DownEncoderBlock2D, the second DownEncoderBlock2D, the third DownEncoderBlock2D and the fourth DownEncoderBlock2D to obtain the second feature map, the third feature map, the fourth feature map and the fifth feature map.

[0146] In step 303, when the computer inputs the UNet output feature map into the VAE decoder for processing, the specific steps are as follows:

[0147] The fifth feature map is passed through a zero convolutional layer to output the first transformed feature map. The fourth feature map and the first transformed feature map are passed through the third cross-layer driven fusion module to output the second transformed feature map.

[0148] The third feature map and the second transformed feature map are processed by the second cross-layer driving fusion module to output the third transformed feature map, and the second feature map and the third transformed feature map are processed by the first cross-layer driving fusion module to output the fourth transformed feature map;

[0149] The UNet output feature map, the first transformed feature map, the second transformed feature map, the third transformed feature map, and the fourth transformed feature map are processed by the VAE decoder to obtain the first decoded output feature map. The UNet output feature map is then processed by the output convolutional layer of the VAE decoder and then processed by UNetMidBlock2D to obtain the sixth feature map. The sixth feature map and the first transformed feature map are added together and then processed by the first UpDecoderBlock2D to obtain the seventh feature map. The seventh feature map and the second transformed feature map are added together and then processed by the second UpDecoderBlock2D to obtain the eighth feature map. The eighth feature map and the third transformed feature map are added together and then processed by the third UpDecoderBlock2D to obtain the ninth feature map. The ninth feature map and the fourth transformed feature map are added together and then processed by the fourth UpDecoderBlock2D to obtain the tenth feature map. The tenth feature map is then processed by subsequent modules of the VAE decoder to output the first decoded output feature map.

[0150] In this embodiment, the fourth feature map and the first transformed feature map are processed by the third cross-layer driving fusion module to output the second transformed feature map. The specific process is as follows:

[0151] Step A: The fourth feature map is split into four fourth feature map segments with the same number of channels by the first tensor splitting module; the first transformed feature map is split into four first transformed upsampled feature maps by the upsampling module, and the first transformed upsampled feature map is split into four first transformed feature map segments with the same number of channels by the second tensor splitting module.

[0152] Step B: Concatenate the four fourth feature split images with the same number of channels and the four first transformation feature split images with the same number of channels respectively to obtain four pre-concatenated feature images with the same number of channels;

[0153] Step C: The first pre-stitched feature map is passed through the first LDC module to output the first extracted feature map; the second pre-stitched feature map is passed through the second LDC module to output the second extracted feature map; the third pre-stitched feature map is passed through the third LDC module to output the third extracted feature map; the fourth pre-stitched feature map is passed through the fourth LDC module to output the fourth extracted feature map.

[0154] Step D: The first, second, third, and fourth extracted feature maps are concatenated to obtain the concatenated feature map.

[0155] Step E: Pass the concatenated feature map through the LCBG module to obtain the second transformed feature map;

[0156] Following steps A to E, the third feature map and the second transformed feature map are processed by the second cross-layer driving fusion module to output the third transformed feature map.

[0157] Following steps A to E, the second feature map and the third transformed feature map are processed by the first cross-layer driving fusion module to output the fourth transformed feature map.

[0158] In this embodiment, step six specifically involves using a computer to input the subsequent visible light image and the required infrared image weather text description into the text encoder, VAE encoder, UNet, and VAE decoder of the trained diffusion model text guidance network for processing, thereby obtaining an infrared generated image corresponding to the required infrared image weather text description.

[0159] In this embodiment, the specific implementation involves different weather conditions: sunny, rainy, snowy, foggy, and cloudy. There are 5978 pairs of images for sunny days, 756 pairs for rainy days, 32 pairs for snowy days, 416 pairs for foggy days, and 2070 pairs for cloudy days. Each weather condition has a visible light image and its corresponding infrared image as a pair.

[0160] In this embodiment, a DJI Matrice 350 RTK drone is selected as the platform, and a Zenmuse H20N gimbal camera is used to capture raw visible light images and corresponding raw infrared images. The Zenmuse H20N gimbal camera includes both a visible light camera and an infrared camera. The raw visible light image size is 3×1950×1080, and the raw infrared image size is 3×640×512.

[0161] In this embodiment, the specific implementation step one is as follows:

[0162] Step 101: Using a computer, the non-rigid registration method in the interactive registration estimation application of Matlab software is used to register the original visible light image and the original infrared image of any target domain to obtain a preliminary registered image sample; wherein, the registered visible light image and the infrared image in the preliminary registered image sample are the same size and are both 3×640×512;

[0163] Step 102: Using a computer and Matlab software, scale any registered visible light image and infrared image in the preliminary registered image sample to obtain a pair of scaled visible light images and infrared images.

[0164] Step 103: Repeat step 102 multiple times to denot multiple pairs of scaled visible light images and infrared images as visible light and infrared training sets; wherein, the scaled visible light images are denoted as visible light images, and the scaled infrared images are denoted as infrared real images, and the visible light images and infrared real images are the same size and are both 3×256×256.

[0165] In this embodiment, the weather text description of the visible light image corresponding to the target domain is "a sunny / rainy / cloudy / snowy / foggy visible image", that is, a visible light image of a sunny / rainy / cloudy / snowy / foggy day.

[0166] The infrared image weather text description is "a sunny / rainy / cloudy / snowy / foggy infraredimage", which means an infrared image of a sunny / rainy / cloudy / snowy / foggy day.

[0167] In this embodiment, CLIP stands for Contrastive Language-Image Pre-Training, and the CLIP network is a multimodal pre-trained neural network. The CLIP network model can refer to conventional network models in this field.

[0168] In this embodiment, the CLIP network model training and CLIP loss are specifically implemented. When acquiring data, the batch size is 8, and the batch size for the remaining networks is 1.

[0169] In this embodiment, when the infrared image weather text description corresponding to the real infrared image y is "as sunny infrared image", it is a real infrared image weather text description. Then, the infrared image weather text description after the weather conditions are changed, such as "a rainy visible image", "a cloudy visible image", "a snowy visible image" or "a foggy visible image", is a fake infrared image weather text description.

[0170] In this embodiment, the size of the feature map is represented by the number of channels × width × height.

[0171] In this embodiment, the fifth feature map is then output as an encoded output feature map by the subsequent part of the VAE encoder. The size of the encoded output feature map is 4×32×32, the size of the text output feature map is 1×77×1024, the size of the UNet output feature map is 4×32×32, and the size of the decoded output feature map is 3×256×256.

[0172] In this embodiment, the geometric transformations include translation, rotation, scaling, flipping, affine transformation, and perspective transformation. More specifically, the geometric transformation involves rotating the image 90° clockwise. The inverse geometric transformation involves rotating the image 90° counterclockwise.

[0173] In this embodiment, the text encoder, VAE encoder, UNet, and VAE decoder can refer to the conventional SD-Turbo model, which is a distillation model based on the Stable Diffusion 2.1 architecture.

[0174] In this embodiment, the first image discriminator and the second image discriminator have the same structure and both use the PatchGAN discriminator, i.e., the Markov discriminator. They are the first image discriminator D and the second image discriminator Dgc, respectively. Their purpose is to determine the authenticity of the infrared image generated by the model.

[0175] In this embodiment, the first image discriminator D includes a first CBL module, a second CBL module, a third CBL module, and two convolutional output layers. Each of the three CBL modules is composed of Conv+Bn+LeakyReLU activation functions. In the first CBL module, the convolutional kernel size is 4×4, the number of convolutional kernels is 64, the stride is 2, and the padding is 1. In the second CBL module, the convolutional kernel size is 4×4, the number of convolutional kernels is 128, the stride is 2, and the padding is 1. In the third CBL module, the convolutional kernel size is 4×4, the number of convolutional kernels is 256, the stride is 2, and the padding is 1. Each convolutional output layer has a convolutional kernel size of 4×4, a single convolutional kernel, a stride of 1, and padding of 1.

[0176] In this embodiment, when the generated infrared image y' or the real infrared image y' is input into the first image discriminator for processing D, the first CBL module outputs a feature map of 64×128×128. The feature map of 64×128×128 is then processed by the second CBL module to output a feature map of 128×64×64. The feature map of 128×64×64 is then processed by the third CBL module to output a feature map of 256×32×32. The feature map of 256×32×32 is then processed by two convolutional output layers to output probability features. Figure 1 ×30×30.

[0177] In this embodiment, step 306 can be implemented with reference to the method of step 305.

[0178] In this embodiment, the intermediate feature map output by the first discriminator D is the input feature map of the probability feature map. In specific implementation, the intermediate feature map output by the first discriminator D is the output feature map of the third CBL module.

[0179] In this embodiment, the MPL layer includes a first hidden layer and a second hidden layer. The actual text output features are output as feature 1×77×4096 through the first hidden layer, and feature 1×77×4096 is output as text enhancement feature 1×77×1024 through the second hidden layer.

[0180] In this embodiment, the text feature reshaping module can be implemented using the rearrange function. In the first convolutional layer, the kernel size is 1×1, the number of kernels is 333, the stride is 1, and the padding is 0. In the second convolutional layer, the kernel size is 3×3, the number of kernels is 256, the stride is 1, and the padding is 1. In the third convolutional layer, the kernel size is 3×3, the number of kernels is 256, the stride is 1, and the padding is 1. Both the first and second activation function layers use the LeakyReLU activation function.

[0181] The structures in the fourth to sixth convolutional layers are the same, and the kernel size is 1×1, the number of kernels is 512, the stride is 1, and the padding is 0.

[0182] In the seventh convolutional layer, the kernel size is 1×1, the number of kernels is 256, the stride is 1, and the padding is 0.

[0183] The eighth convolutional layer has a kernel size of 4×4, a kernel count of 256, a stride of 1, and padding of 1. The ninth convolutional layer has a kernel size of 4×4, a kernel count of 1, a stride of 1, and padding of 1.

[0184] In this embodiment, when adding a row of key vectors and a row of value vectors in step A09, the element values ​​of the vectors are zero.

[0185] In this embodiment, the size of the text reshaping feature map is 77×32×32, the size of the spliced ​​feature map is 333×32×32, the size of the preprocessed feature map is 256×32×32, the size of the combined feature map is 1×256×1024, the size of the normalized feature map is 256×32×32, and the size of the first output feature matrix to the third output feature matrix is ​​512×32×32.

[0186] The query matrix, key matrix, and value matrix are all 8×1024×64 in size, the key vector is 8×1×64 in size, and after adding, the key matrix K is 8×1025×64 in size, the value vector is 8×1×64 in size, and the value matrix V is 8×1025×64 in size.

[0187] The size of the output feature map of the self-attention layer is 8×1024×64, the size of the tensor transformation feature map is 512×32×32, the size of the intermediate transition feature map is 256×32×32, and the size of the probability feature map is 1×30×30.

[0188] In this embodiment, a zero-convolutional layer, a first cross-layer driven fusion module, a second cross-layer driven fusion module, and a third cross-layer driven fusion module are set up. The zero-convolutional layer preserves the original image structure information and provides it to the driven fusion layer. In the visible light image to infrared image conversion task, the cross-layer driven fusion module effectively fuses and transmits feature information from different levels, captures and understands features at different scales, and helps to preserve the details and structure of the original image. By fusing features from different levels, the network's ability to understand and express image content is improved. The quality of infrared images is significantly improved, thereby solving the problem of easy loss of structural information and detailed texture information during the infrared image generation process.

[0189] In this embodiment, the kernel size in the zero convolutional layer is 1×1, the stride is 1, the padding is 0, a zero initialization strategy is adopted, and the number of kernels is 512; the upsampling module reduces the number of channels to half of the original number and the width and height to twice the original number.

[0190] In this embodiment, specifically, in the dilated convolution of the third cross-layer driving fusion module, the size of the convolution kernel is 3×3, the sliding stride is 1, and the number of convolution kernels is 128; in the first LDC module, the dilation rate of the dilated convolution is 7, and the padding is 7; in the second LDC module, the dilation rate of the dilated convolution is 5, and the padding is 5; in the third LDC module, the dilation rate of the dilated convolution is 2, and the padding is 2; in the fourth LDC module, the dilation rate of the dilated convolution is 1, and the padding is 1.

[0191] The difference is that the number of convolution kernels in the dilated convolution of both the second cross-layer driven fusion module and the first cross-layer driven fusion module is 64.

[0192] In this embodiment, specifically, the two-dimensional convolution in the LCBG module of the third and second cross-layer driven fusion modules is as follows: the kernel size is 3×3, the stride is 1, the padding is 1, and the number of kernels is 512. The first cross-layer driven fusion module differs from the third and second cross-layer driven fusion modules only in the number of kernels, which is 256.

[0193] In this embodiment, specifically, the size of the first feature map is 64×256×256, the size of the second feature map is 128×256×256, the size of the fourth transformed feature map is 256×256×256; the size of the third feature map is 128×128×128, the size of the third transformed feature map is 512×128×128; the size of the fourth feature map is 256×64×64, the size of the second transformed feature map is 512×64×64; the size of the fifth feature map is 512×32×32, and the size of the first transformed feature map is 512×32×32.

[0194] The size of the fourth feature split map is 64×64×64; the size of the first transformed upsampled feature map is 256×64×64; the size of the first transformed feature split map is 64×64×64; the size of the first concatenated feature map is 128×64×64; the size of the first to fourth extracted feature maps is 128×64×64; the size of the last concatenated feature map is 512×64×64; and the size of the second transformed feature map is 512×64×64.

[0195] In this embodiment, the column dimension of the key matrix is ​​specifically implemented. It is 64.

[0196] In this embodiment, specifically, step five, which trains the diffusion model text guidance network based on the total loss, uses the Adam optimizer. During training, The values ​​were set to 0.5, 0.5, 0.5, 1.0, 1.0, and 1.0, respectively. The text encoder did not require training. Furthermore, Low-Rank Adaptation (LoRA) was employed in the training of the diffusion model's text-guided network; for details, please refer to the paper "LoRA: Low-Rank Adaptation of Large Language Models".

[0197] In this embodiment, the diffusion model text guidance network undergoes a total of 50,000 iterations to ensure model convergence. The text-image association loss uses a pre-trained CLIP network model to improve the ability to recognize infrared image features, thereby correctly associating weather text descriptions of various weather conditions with the correct infrared images.

[0198] In summary, the method of this invention is simple in steps and reasonable in design. It uses weather text to describe weather conditions and couples multimodal information from visible light images and weather text descriptions into a diffusion model text-guided network for training, thereby enabling the generation of infrared images under different weather conditions and improving the quality of infrared image generation.

[0199] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An infrared image generation method based on diffusion model text guidance, characterized in that, The method comprises the following steps: Step one, acquisition of training set images: Visible light images and corresponding infrared images are collected under different weather conditions, and the corresponding infrared images are recorded as infrared real images, to obtain a visible light-infrared training set; wherein the visible light-infrared training set comprises visible light images and infrared real images; Step two, constructing a diffusion model text guide network; wherein the diffusion model text guide network comprises a text encoder, a VAE encoder, a UNet, and a VAE decoder, as well as a first image discriminator D, a second image discriminator Dgc, and a text discriminator DT; Step three, feature extraction of the visible light-infrared training set and image weather text description through the diffusion model text guide network; Step four, constructing a total loss in the feature extraction of the visible light-infrared training set and image weather text description through the diffusion model text guide network: According to , the total loss is obtained; wherein, denotes a first loss coefficient, denotes a second loss coefficient, denotes a third loss coefficient, denotes a fourth loss coefficient, denotes a fifth loss coefficient, denotes a sixth loss coefficient, denotes the LSGANs loss of the first image discriminator D, denotes the LSGANs loss of the second image discriminator Dgc, denotes the LSGANs loss of the text discriminator DT, denotes the geometric consistency loss, denotes the CLIP text-image correlation loss, denotes the identity mapping loss; Step five, training the diffusion model text guide network based on the total loss; Step six, generating infrared images based on the trained diffusion model text guide network.

2. The method for generating an infrared image based on a diffusion model text guide according to claim 1, characterized in that: Step three, the specific process is as follows: Step 301, obtaining corresponding visible light image weather text description and infrared image weather text description according to the visible light image x and the infrared real image y of the target domain; wherein the visible light image weather text description and the infrared image weather text description are both English fields; Step 302, inputting any visible light image x into the VAE encoder using a computer to obtain an encoding output feature map; inputting the infrared image weather text description into the text encoder using a computer to obtain a text output feature; and inputting the encoding output feature map and the text output feature into the UNet using a computer to obtain a UNet output feature map; Step 303, inputting the UNet output feature map into the VAE decoder using a computer to obtain a first decoding output feature map, and recording the first decoding output feature map as an infrared generated image y'; Step 304, processing the visible light image x through geometric transformation using a computer to obtain a transformed visible light image fx, and inputting the transformed visible light image fx into the diffusion model text guide network according to the method of steps 302 to 303 to obtain a second decoding output feature map, and recording the second decoding output feature map as a transformed infrared generated image fy; Step 305, inputting the infrared generated image y' and the infrared real image y into the first image discriminator D using a computer to obtain the probability feature map D(y') output by the first discriminator D for the infrared generated image and the probability feature map D(y) output by the first discriminator D for the infrared real image; Step 306, inputting the transformed infrared generated image fy and the infrared real image y into the second image discriminator Dgc using a computer to obtain the probability feature map Dgc(fy) output by the second discriminator Dgc for the transformed infrared generated image and the probability feature map Dgc(y) output by the second discriminator Dgc for the infrared real image; Step 307, inputting the infrared generated image y' and the infrared real image y into the text discriminator DT using a computer to obtain the probability feature map DT(y') output by the text discriminator DT for the infrared generated image and the probability feature map DT(y) output by the text discriminator DT for the infrared real image; Step 307, the specific process is as follows: Step 3071, the computer is used to change the corresponding infrared image weather text description of the target domain into a real infrared image weather text description, and the infrared image weather text description after changing the weather condition is changed into a false infrared image weather text description, and the real infrared image weather text description and the false infrared image weather text description are processed by a text encoder respectively to obtain a real text output feature and a false text output feature; Step 3072, the real text output feature and the intermediate feature map output by the first discriminator D from the real infrared image are combined as a positive sample, the real text output feature and the intermediate feature map output by the first discriminator D from the generated infrared image are combined as a first negative sample, and the false text output feature and the intermediate feature map output by the first discriminator D from the real infrared image are combined as a second negative sample; Step 3073, the positive sample, the first negative sample and the second negative sample are respectively input into the text discriminator DT to obtain a probability feature map DT(z) output by the text discriminator DT from the positive sample, a probability feature map DT(f1) output by the text discriminator DT from the first negative sample and a probability feature map DT(f2) output by the text discriminator DT from the second negative sample. The text discriminator DT comprises a linear layer, a preprocessing layer, L2 normalization and a self-attention layer, the linear layer comprises an MPL layer and a text feature remodeling module, the preprocessing layer comprises a first convolutional layer, a second convolutional layer, a first activation function layer, a third convolutional layer and a second activation function layer connected in sequence, and the self-attention layer comprises a fourth convolutional layer, a fifth convolutional layer and a sixth convolutional layer connected in parallel; 3. The method for generating an infrared image based on a diffusion model text guide according to claim 2, characterized in that: The output of the self-attention layer is further connected with a seventh convolutional layer and a Head layer, and the Head layer comprises an eighth convolutional layer, instance normalization, a third activation function layer and a ninth convolutional layer. Step 3073, the specific process is as follows: ​ ​ 4. The method for generating an infrared image based on a diffusion model text guide according to claim 3, characterized in that: ​ ​ 5. The method for generating an infrared image based on a diffusion model text guide according to claim 4, characterized in that: ​ Step A01, use a computer to enhance the real text output features through the MPL layer to obtain text enhanced features; Step A02, use a computer to convert the text enhanced features into text remodeling feature maps matching the intermediate feature maps output by the first image discriminator through the text feature remodeling module; Step A03, use a computer to obtain a spliced feature map by splicing the text remodeling feature map and the intermediate feature map output by the first discriminator D through the infrared real image; Step A04, use a computer to input the spliced feature map into the first convolutional layer, the second convolutional layer, the first activation function layer, the third convolutional layer and the second activation function layer for preprocessing in turn to obtain a preprocessed feature map; Step A05, use a computer to obtain a pixel channel dimension vector corresponding to the value of all channel numbers at any pixel point from the preprocessed feature map; and perform L2 normalization on the pixel channel dimension vector to obtain a pixel channel dimension normalized vector; Step A06, repeat step A05 multiple times to obtain all pixel channel dimension normalized vectors, then combine the all pixel channel dimension normalized vectors in pixel order to obtain a combined feature map; and change the combined feature map through tensor to obtain a normalized feature map with the same size as the preprocessed feature map; Step A07, use a computer to process the normalized feature map through the fourth convolutional layer to obtain a first output feature matrix; Step A08, use a computer to process the normalized feature map through the fifth convolutional layer to obtain a second output feature matrix; Step A09, use a computer to process the normalized feature map through the sixth convolutional layer to obtain a third output feature matrix; Step A08, use a computer to perform the first tensor change on the first output feature matrix, the second output feature matrix and the third output feature matrix to obtain a query matrix Q, a key matrix and a value matrix; Step A09, use a computer to add a row of key vectors to the row direction of the key matrix to obtain an added key matrix K; use a computer to add a row of value vectors to the row direction of the value matrix to obtain an added value matrix V; Step A10, using a computer, combines the query matrix Q, the post-addition key matrix K, and the post-addition value matrix V according to to obtain output features from the self-attention layer ; wherein, is the column dimension of the key matrix; Step A11, use a computer to perform the second tensor change on the output feature of the self-attention layer to obtain a tensor conversion feature map, and process the tensor conversion feature map through the seventh convolutional layer to obtain an intermediate transition feature map; Step A12, use a computer to obtain a probability feature map DT(z) output by the text discriminator DT for the positive sample through the Head layer; Step A13, according to the method of steps A01 to A12, obtain a probability feature map DT(f1) output by the text discriminator DT for the first negative sample and a probability feature map DT(f2) output by the text discriminator DT for the second negative sample.

6. The method for generating an infrared image based on a diffusion model text guide according to claim 5, characterized in that: Step four, the specific process is as follows: Step 401, according to , obtaining the LSGANs loss of the first discriminator D ; wherein, indicates the i-th pixel value in the probability feature map D(y) output by the first discriminator D for the infrared real image, indicates the i-th pixel value in the probability feature map D(y') output by the first discriminator D for the infrared generated image, i and m are positive integers; 1≤i≤m, and m is the total number of pixels of the probability feature map. Step 402, according to , get the LSGANs loss of the second discriminator Dgc ; wherein, represents the i-th pixel value in the probability feature map Dgc(y) output by the second discriminator Dgc for the infrared real image, represents the i-th pixel value in the probability feature map Dgc(fy) output by the second discriminator Dgc for the transformed infrared generated image; Step 403, according to , obtaining the LSGANs loss of the text discriminator DT ; wherein, represents the i-th pixel value in the probability feature map DT(z) output by the text discriminator DT for a positive sample, represents the i-th pixel value in the probability feature map DT(f1) output by the text discriminator DT for a first negative sample, represents the i-th pixel value in the probability feature map DT(f2) output by the text discriminator DT for a second negative sample; In step 404, the CLIP loss is obtained based on the infrared image weather text description and the infrared generated image through the trained CLIP network model ; and according to , the CLIP text-image correlation loss is obtained ; Step 405, according to , get the geometric consistency loss ; wherein, represents the 1-norm between the infrared generated image y' and the geometric inverse transformed infrared generated image z', represents the 1-norm between the transformed infrared generated image fy and the geometric transformed infrared generated image z, represents the LPIPS distance between the infrared generated image y' and the geometric inverse transformed infrared generated image z', represents the LPIPS distance between the transformed infrared generated image fy and the geometric transformed infrared generated image z; Step 406, according to , get identity mapping loss ; wherein, represents the 1-norm between the visible light image x and the visible light generated image xg, represents the 1-norm between the infrared real image y and the infrared output image yh.

7. The method for generating an infrared image based on a diffusion model text guide according to claim 1, characterized in that: The VAE encoder includes an input convolutional layer, a first DownEncoderBlock2D, a second DownEncoderBlock2D, a third DownEncoderBlock2D and a fourth DownEncoderBlock2D; The VAE decoder comprises an output convolutional layer plus a UNetMidBlock2D, a first UpDecoderBlock2D, a second UpDecoderBlock2D, a third UpDecoderBlock2D and a fourth UpDecoderBlock2D; A first cross-layer driving fusion module is arranged between the output of the first DownEncoderBlock2D and the input of the fourth UpDecoderBlock2D, a second cross-layer driving fusion module is arranged between the output of the second DownEncoderBlock2D and the input of the third UpDecoderBlock2D, and a third cross-layer driving fusion module is arranged between the output of the third DownEncoderBlock2D and the input of the third UpDecoderBlock2D; a zero convolutional layer is arranged between the output of the fourth DownEncoderBlock2D and the input of the first UpDecoderBlock2D, the output of the zero convolutional layer is also input into the third cross-layer driving fusion module, the output of the third cross-layer driving fusion module is also input into the second cross-layer driving fusion module, and the output of the second cross-layer driving fusion module is also input into the first cross-layer driving fusion module; The first cross-layer driving fusion module, the second cross-layer driving fusion module and the third cross-layer driving fusion module have the same structure and each comprise an up-sampling module, two tensor splitting modules, four LDC modules and one LCBG module; each LDC module is composed of layer normalization and dilated convolution, and each LCBG module is composed of layer normalization, two-dimensional convolution, batch normalization and GELU activation function.

8. The method for generating an infrared image based on a diffusion model text guide according to claim 7, characterized in that: In step 302, when the visible light image x is input into the VAE encoder to obtain the encoded output feature map, the visible light image x is first input into the input convolutional layer of the VAE encoder to obtain a first feature map, and the first feature map is sequentially input into the first DownEncoderBlock2D, the second DownEncoderBlock2D, the third DownEncoderBlock2D and the fourth DownEncoderBlock2D to obtain a second feature map, a third feature map, a fourth feature map and a fifth feature map; In step 303, when the UNet output feature map is input into the VAE decoder for processing by using a computer, the following steps are performed: The fifth feature map is input into the zero convolutional layer to output a first converted feature map, and the fourth feature map and the first converted feature map are input into the third cross-layer driving fusion module to output a second converted feature map; The third feature map and the second converted feature map are input into the second cross-layer driving fusion module to output a third converted feature map, and the second feature map and the third converted feature map are input into the first cross-layer driving fusion module to output a fourth converted feature map; The fourth converted feature map is input into the output convolutional layer of the VAE decoder to output the visible light image x. The UNet output feature map, the first conversion feature map, the second conversion feature map, the third conversion feature map and the fourth conversion feature map pass through the VAE decoder to obtain a first decoding output feature map; wherein the UNet output feature map passes through an output convolutional layer of the VAE decoder plus UNetMidBlock2D to obtain a sixth feature map; the sixth feature map and the first conversion feature map are added and input into a first UpDecoderBlock2D for processing to obtain a seventh feature map; the seventh feature map and the second conversion feature map are added and input into a second UpDecoderBlock2D for processing to obtain an eighth feature map; the eighth feature map and the third conversion feature map are added and input into a third UpDecoderBlock2D for processing to obtain a ninth feature map; the ninth feature map and the fourth conversion feature map are added and input into a fourth UpDecoderBlock2D for processing to obtain a tenth feature map; the tenth feature map passes through a subsequent module of the VAE decoder to output the first decoding output feature map.

9. The method for generating an infrared image based on a diffusion model text guide according to claim 8, characterized in that: The fourth feature map and the first conversion feature map pass through a third cross-layer driving fusion module to output a second conversion feature map, and the specific process is as follows: Step A, the channel number of the fourth feature map is split through a first tensor splitting module to obtain four fourth feature split maps with the same channel number; the first conversion feature map passes through an upsampling module to obtain a first conversion upsampling feature map, and the first conversion upsampling feature map is split through a second tensor splitting module to obtain four first conversion feature split maps with the same channel number; Step B, the four fourth feature split maps with the same channel number and the four first conversion feature split maps with the same channel number are respectively spliced to obtain four front spliced feature maps with the same channel number; Step C, the first front spliced feature map passes through a first LDC module to output a first extracted feature map; the second front spliced feature map passes through a second LDC module to output a second extracted feature map; the third front spliced feature map passes through a third LDC module to output a third extracted feature map; and the fourth front spliced feature map passes through a fourth LDC module to output a fourth extracted feature map; Step D, the first extracted feature map, the second extracted feature map, the third extracted feature map and the fourth extracted feature map pass through splicing to obtain a rear spliced feature map; Step E, the rear spliced feature map passes through an LCBG module to obtain the second conversion feature map; According to the method of steps A to E, the third feature map and the second conversion feature map pass through a second cross-layer driving fusion module to output a third conversion feature map; According to the method of steps A to E, the second feature map and the third conversion feature map pass through a first cross-layer driving fusion module to output a fourth conversion feature map.

10. The method for generating an infrared image based on a diffusion model text guide according to claim 2, characterized in that: Step six is specifically: using a computer to input the subsequent visible light image and the required infrared image weather text description into the text encoder, the VAE encoder, the UNet and the VAE decoder of the trained diffusion model text guided network for processing to obtain an infrared generated image corresponding to the required infrared image weather text description.

Images