A cross-modal text-to-infrared image generation method based on thermal mask constraint

Abstract

The present application belongs to the technical field of infrared image, and relates to a cross-modal text-to-infrared image generation method based on thermal mask constraint, which comprises the following steps: constructing a text-infrared image pair dataset based on a large model; determining the gray threshold of the infrared image by using the mean method, and generating the thermal mask information of the infrared image as the thermal mask information constraint by using the gray threshold; training an autoencoder to reconstruct the infrared image; using the text-infrared image pair dataset and time coding as input, and training an infrared image generation model by using a latent diffusion model, the autoencoder and the thermal mask information constraint; and using the Gaussian noise, the text in the text-infrared image pair dataset and the time coding as input, and generating the infrared image by using the infrared image generation model. The method has the beneficial effect of effectively improving the thermal characteristic expression capability of the generated infrared image, and realizing the target of generating high-quality infrared images from text under the condition of no visible light input.

Description

Technical Field

[0001] This invention relates to the field of infrared imaging technology, and in particular to a method for generating cross-modal text-to-infrared images based on thermal mask constraints. Background Technology

[0002] Infrared imagery is widely used in military applications, security surveillance, and autonomous driving due to its ability to capture thermal radiation information, provide clear global target outlines, and remain robust in low light, smoke, and camouflage conditions. However, collecting infrared images requires specialized and expensive equipment, and acquiring images of specific scenes is time-consuming and labor-intensive. This leads to a scarcity of infrared image datasets, limiting the research and application of computer vision technology in the infrared field.

[0003] Early methods for obtaining infrared images mainly relied on simulation modeling and manual feature processing. Although these methods considered various physical and low-level features of different objects, thus providing a certain degree of physical reliability, the wide variety of objects in nature and the complex and unpredictable surface temperature patterns made manual analysis, segmentation, and modeling of each image very time-consuming and laborious, and prone to inaccuracies and inconsistencies.

[0004] In recent years, with the advancement of deep learning, researchers have attempted to use deep models to convert visible light images into infrared images. However, due to the influence of atmospheric radiation and the intrinsic radiation of the target, the temperature of infrared images varies significantly across different regions. Therefore, accurately matching and predicting the temperature of the corresponding region becomes extremely challenging when converting from visible light images to infrared images. Furthermore, under low-light or even no-light conditions, the quality of infrared images generated from visible light images degrades or even fails. Summary of the Invention

[0005] Technical problems to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a cross-modal text-to-infrared image generation method based on thermal mask constraints, which solves the technical problem of poor infrared image generation quality caused by over-reliance on visible light image input in the prior art, especially the technical defects of difficulty in extracting effective features from models and unstable generation results in low light or no light environment.

[0007] Technical solution

[0008] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0009] In a first aspect, the present invention provides a method for generating cross-modal text-to-infrared images based on thermal mask constraints, comprising:

[0010] A text-infrared image pair dataset was constructed based on a large model.

[0011] The grayscale threshold of the infrared image is determined by the mean value method, and the grayscale threshold is used to generate the thermal mask information of the infrared image as a constraint on the thermal mask information.

[0012] An autoencoder is trained to reconstruct an infrared image. The autoencoder consists of an encoder and a decoder. The encoder compresses the high-dimensional infrared image to obtain a low-dimensional vector, and the decoder restores the low-dimensional vector to the infrared image.

[0013] Using a text-infrared image dataset and time encoding as input, an infrared image generation model is trained by constraining the latent diffusion model, autoencoder, and thermal mask information.

[0014] Using Gaussian noise, text-infrared image data, and text and time encoding from the dataset as input, an infrared image generation model is used to generate infrared images.

[0015] Optionally, a text-infrared image pair dataset is constructed based on a large model, including:

[0016] A large model was used to perform a preliminary description of the visible light image, including the background, number of people, and vehicles.

[0017] Based on the infrared image corresponding to the description content, obtain the manually corrected description;

[0018] The locally deployed large model is used to optimize the syntax and expression of the manually corrected description, resulting in an optimized description.

[0019] An open-source text summarization model is used to summarize the optimized description, generating a concise description.

[0020] We will construct a text-infrared image pair dataset by combining optimized descriptions, concise descriptions, and corresponding infrared images.

[0021] Optionally, the grayscale threshold τ of the infrared image can be determined using the mean method, as shown in the following formula:

[0022] τ= ;

[0023] in, This represents the mean-standard deviation method. This represents the percentile method, where μ is the mean, σ is the standard deviation, k is the multiplier, and p is the percentage of radiation intensity fluctuation. , , Indicates the width of the image. Indicates the height of the image, Indicates position on the image The pixel value at that location.

[0024] Optionally, using a text-infrared image pair dataset and time encoding as input, an infrared image generation model is trained using latent diffusion models, autoencoders, and thermal mask information constraints, including:

[0025] The dataset and time-encoded text-infrared images were used as input.

[0026] The encoder compresses the high-dimensional infrared image to obtain a low-dimensional vector, and uses the low-dimensional vector, text, and time encoding as input to the latent diffusion model.

[0027] The decoder restores the output of the latent diffusion model to an infrared image;

[0028] Constructing the loss function of the potential diffusion model and thermal mask information constraint loss function ;

[0029] Loss function based on potential diffusion model and thermal mask information constraint loss function Obtain the total loss function ;

[0030] By minimizing the total loss function An infrared image generation model was obtained.

[0031] Optionally, the loss function of the latent diffusion model ,in, For the desired operation, t is the time step of the diffusion process. For infrared images in the text-infrared image dataset, The square of the L2 norm, This is real noise. This refers to the noise predicted by the model.

[0032] Optionally, the thermal mask information constrains the loss function. ,in, It is a real infrared image, and Z is the latent feature output by the diffusion model. It is a decoder for an autoencoder. It is a weight matrix inspired by thermophysics. , Indicates in The pixel value at that location is α = 0.5 and β = 5.

[0033] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed, implements the method for generating cross-modal text to infrared images based on thermal mask constraints as described in any of the first aspects above.

[0034] Thirdly, the present invention provides a storage device, including a storage medium and a processor, wherein the storage medium stores a computer program, and when the program is executed by the processor, it implements the method for generating cross-modal text to infrared images based on thermal mask constraints as described in any of the first aspects above.

[0035] Beneficial effects

[0036] The beneficial effects of this invention are as follows: This invention provides a cross-modal text-to-infrared image generation method based on thermal mask constraints. Using natural language descriptions as input, it utilizes a diffusion model to directly generate infrared images from text without visible light image assistance. This overcomes the limitation of traditional methods that rely on visible light images and solves the problem of poor infrared image generation quality in low-light or no-light environments. By introducing a thermal mask as prior information in thermal physics, and explicitly constraining the model to generate key thermal distribution regions in the image, the generated image better conforms to the physical characteristics of infrared imaging. Guiding the diffusion model to accurately model the thermal structure distribution in the infrared image under pure text input conditions effectively improves the thermal characteristic expression capability of the generated infrared image, achieving the goal of generating high-quality infrared images from text without visible light input. Attached Figure Description

[0037] Figure 1 A flowchart illustrating a cross-modal text-to-infrared image generation method based on thermal mask constraints provided in an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram illustrating the selection of infrared image thermal mask information thresholds provided in an embodiment of the present invention.

[0039] Figure 3 A cross-modal text-to-infrared image generation framework based on thermal mask constraints is provided in this embodiment of the invention.

[0040] Figure 4 Comparison of infrared image generation effects provided in embodiments of the present invention. Detailed Implementation

[0041] To better explain and facilitate understanding of the present invention, it will be described in detail below with reference to the accompanying drawings and specific embodiments. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a clearer and more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0042] Firstly, referring to Figure 1 This embodiment provides a method for generating cross-modal text-to-infrared images based on thermal mask constraints, including:

[0043] S1, a text-infrared image pair dataset is constructed based on a large model.

[0044] Optionally, a text-infrared image pair dataset is constructed based on a large model, including:

[0045] A large model was used to perform a preliminary description of the visible light image, including the background, number of people, and vehicles.

[0046] Based on the infrared image corresponding to the description content, obtain the manually corrected description;

[0047] The locally deployed large model is used to optimize the syntax and expression of the manually corrected description, resulting in an optimized description.

[0048] An open-source text summarization model is used to summarize the optimized description, generating a concise description.

[0049] We will construct a text-infrared image pair dataset by combining optimized descriptions, concise descriptions, and corresponding infrared images.

[0050] The problem of poor quality infrared image generation under low light (or even no light) conditions is addressed by constructing a text-infrared image pair dataset.

[0051] S2, the grayscale threshold of the infrared image is determined by the mean method, and the infrared image thermal mask information is generated using the grayscale threshold as a constraint on the thermal mask information.

[0052] Optionally, the grayscale threshold τ of the infrared image can be determined using the mean method, as shown in the following formula:

[0053] τ= ;

[0054] in, This represents the mean-standard deviation method. This represents the percentile method, where μ is the mean, σ is the standard deviation, k is the multiplier, and p is the percentage of radiation intensity fluctuation. , , Indicates the width of the image. Indicates the height of the image, Indicates position on the image The pixel value at that location.

[0055] To obtain more accurate infrared image thermal mask information, this embodiment compares various methods to obtain different infrared image grayscale thresholds, specifically including the Otsu method, the mean-standard deviation method, the percentage method, and the mean of the mean-standard deviation method and the percentage method (hereinafter referred to as the mean method).

[0056] By comparison Figure 2The results of the methods shown above revealed that the infrared image thermal mask information obtained by the mean method contains less interference. Therefore, this invention determines the mean method as the threshold selection method for constructing infrared image thermal mask information.

[0057] The mean value method is used to dynamically process each infrared image. Pixels with values ​​greater than the grayscale threshold are retained, while those with values ​​less than the grayscale threshold are set to 0, thus obtaining the thermal mask information of all infrared images.

[0058] S3 trains an autoencoder to reconstruct the infrared image. The autoencoder consists of an encoder and a decoder. The encoder compresses the high-dimensional infrared image to obtain a low-dimensional vector, and the decoder restores the low-dimensional vector to the infrared image.

[0059] Autoencoders are used to reconstruct infrared images. In latent diffusion models, autoencoders are used to compress high-dimensional images into a latent space, thereby reducing the computational complexity of the diffusion process while preserving key semantic features.

[0060] S4 uses a text-infrared image pair dataset and time encoding as input, and trains an infrared image generation model using latent diffusion model, autoencoder and thermal mask information constraints.

[0061] Optionally, using a text-infrared image pair dataset and time encoding as input, an infrared image generation model is trained using latent diffusion models, autoencoders, and thermal mask information constraints, including:

[0062] The dataset and time-encoded text-infrared images were used as input.

[0063] The encoder compresses the high-dimensional infrared image to obtain a low-dimensional vector, and uses the low-dimensional vector, text, and time encoding as input to the latent diffusion model.

[0064] The decoder restores the output of the latent diffusion model to an infrared image;

[0065] Constructing the loss function of the potential diffusion model and thermal mask information constraint loss function ;

[0066] Loss function based on potential diffusion model and thermal mask information constraint loss function Obtain the total loss function ;

[0067] By minimizing the total loss function An infrared image generation model was obtained.

[0068] In the diffusion model, the autoencoder acts as a compression and decompression process to reconstruct the image. The autoencoder uses an encoder to map a high-dimensional image into a low-dimensional vector, and then a decoder restores the original image from the low-dimensional vector. This reduces the computational cost of subsequent diffusion processes while preserving key information. The diffusion model uses the low-dimensional vector encoded by the encoder, thus reducing the computational burden of the diffusion model.

[0069] Loss function of potential diffusion model It is a measure Real noise and The difference between the noise predicted by the model is the core metric of the loss function. The thermal mask information constrains the loss function. Its core is spatially adaptive masking L1 loss, which aims to enhance the thermodynamic consistency of key regions (such as thermal boundaries and high-temperature points).

[0070] Optionally, the loss function of the latent diffusion model ,in, For the desired operation, t is the time step of the diffusion process. For infrared images in the text-infrared image dataset, The square of the L2 norm, This is real noise. This refers to the noise predicted by the model.

[0071] Optionally, the thermal mask information constrains the loss function. ,in, It is a real infrared image, and Z is the latent feature output by the diffusion model. It is a decoder for an autoencoder. It is a weight matrix inspired by thermophysics. , Indicates in In the process of generating infrared images guided by thermal masking, it is necessary to ensure that the deviation penalty for high-radiation areas (such as vehicle engines and human bodies) is higher than that for background areas. Based on experience, the coefficients α and β are set to 0.5 and 5, respectively.

[0072] S5 uses Gaussian noise, text-infrared image, and text and time encoding in the dataset as input, and generates infrared images using an infrared image generation model.

[0073] like Figure 3 The present invention presents a cross-modal text-to-infrared image generation framework based on thermal mask constraints. This invention, based on thermal mask constraints, abandons the dependence of traditional deep generation models on visible light images, which not only improves the generation quality of infrared images, but also solves the problem of poor generation quality of infrared images under low light or no light conditions.

[0074] To verify the effectiveness of this invention, a comparison was made using both quantitative indicators (Fréchet Inception Distance, an index for evaluating the similarity in feature distribution between generated and real images; a lower value indicates higher quality and greater realism of the generated image) and qualitative analysis. Figure 4 As shown. Compared with the prior art, the technical solution proposed in this invention uses cross-modal text to generate infrared images and uses thermal mask information to constrain thermophysical radiation, which not only improves the generation quality of infrared images, but also makes the thermal distribution information (geometric shape) of objects more reasonable and clear.

[0075] Specific quantitative indicators are the present invention (FID: 71.79), PID (FID: 72.48), and InfraGAN (FID: 86.50). In qualitative analysis, the present invention provides more reasonable and clearer information on the thermal distribution (geometry) of objects. Furthermore, because the present invention uses a text generation method, the generated results exhibit a certain degree of diversity, thus the object position differs from that generated using visible light images.

[0076] In a second aspect, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed, implements the method for generating cross-modal text to infrared images based on thermal mask constraints as described in any of the first aspects above.

[0077] Thirdly, embodiments of the present invention provide a storage device, including a storage medium and a processor, wherein the storage medium stores a computer program, and when the program is executed by the processor, it implements the method for generating cross-modal text to infrared images based on thermal mask constraints as described in any of the first aspects above.

[0078] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

[0080] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A cross-modal text-to-infrared image generation method based on a hot mask constraint, characterized by, The application relates to a cross-modal text-to-infrared image generation method based on a heat mask constraint. An infrared image-text pair dataset is constructed based on a large model, including: A visible light image is preliminarily described by using a large model, and the description content includes background, number of people and vehicle; Based on the infrared image corresponding to the description content, an artificial correction description is obtained; A local deployed large model is used to optimize the syntax and expression of the artificial correction description, and an optimized description is obtained; An open-source text summary model is used to process the optimized description to generate a concise description; The optimized description, the concise description and the corresponding infrared image are combined to construct the infrared image-text pair dataset; The mean value method is used to determine the gray threshold of the infrared image, and the gray threshold of the infrared image is used to generate infrared image heat mask information as heat mask information constraint; An autoencoder is trained to reconstruct the infrared image, the autoencoder includes an encoder and a decoder, the encoder is used to compress the high-dimensional infrared image to obtain a low-dimensional vector, and the decoder is used to restore the low-dimensional vector to the infrared image; The infrared image generation model is trained by using the infrared image-text pair dataset and time coding as input, using the latent diffusion model, the autoencoder and the heat mask information constraint, including: The infrared image-text pair dataset and the time coding are used as input; The encoder is used to compress the high-dimensional infrared image to obtain a low-dimensional vector, and the low-dimensional vector, the text and the time coding are used as input of the latent diffusion model; The decoder is used to restore the output of the latent diffusion model to the infrared image; Constructing potential diffusion model loss function and hot mask information constraint loss function ; loss function based on a latent diffusion model and a hotmask information constraint loss function resulting in a total loss function ; wherein the thermal mask information constrains the loss function wherein, is a real infrared image, Z is a latent feature output by the diffusion model, is a decoder of the autoencoder, is a thermophysical-inspired weight matrix, , denotes the pixel value at , and a is 0.5 and b is 5. by minimizing a total loss function obtain an infrared image generation model; The infrared image generation model is used to generate the infrared image by using the Gaussian noise, the text in the infrared image-text pair dataset and the time coding as input.

2. The method of claim 1, wherein, The mean value method is used to determine the gray threshold tau of the infrared image, and the formula is as follows: τ = 1 - e~τ ; wherein, denotes the mean-standard deviation method, denotes the percentile method, μ is the mean, σ is the standard deviation, k is the multiple factor, and p is the proportion of fluctuation of the radiation intensity, , , denotes the width of the image, denotes the height of the image, denotes the pixel value at the position on the image.

3. The method of claim 2, wherein, Potential diffusion model loss function wherein, is the desired operation, t is the time step of the diffusion process, is an infrared image in the text-infrared image dataset, is the L2 norm squared, is the true noise, is the model predicted noise.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the cross-modal text-to-infrared image generation method based on the heat mask constraint.

5. A storage device comprising a storage medium and a processor, the storage medium storing a computer program, characterized in that, The processor executes the computer program to realize the cross-modal text-to-infrared image generation method based on the heat mask constraint.

Images