A method and system for generating visible-to-infrared images based on a prototype-guided diffusion model.

Abstract

This invention provides a method and system for generating visible light to infrared images based on a prototype-guided diffusion model, relating to the field of computer vision technology. The method includes: collecting visible light images to be converted; loading a pre-constructed set of infrared prototype features and pre-trained model parameters; and inputting the image and feature set into a prototype-guided diffusion model network. This network generates the corresponding infrared image through the collaborative work of its internal prototype feature construction, local prototype attention, and noise prediction modules. This invention effectively solves the problems of poor physical consistency and detail distortion in generated images caused by insufficient modeling of infrared thermal features in existing methods by extracting thermal radiation prototypes from real infrared data and introducing them as strong semantic priors into the diffusion generation process. This significantly improves the thermal structure realism and detail fidelity of the synthesized infrared image.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for generating visible light to infrared images based on a prototype-guided diffusion model. Background Technology

[0002] In scenarios such as intelligent navigation and maritime surveillance, visible light imaging has become the mainstream visual perception method due to its low equipment cost and ease of acquisition. However, the image quality of visible light (RGB) images deteriorates significantly in complex environments such as nighttime, fog, or low illumination, severely impacting the reliability of the visual system. In contrast, infrared (IR) imaging relies on the object's own thermal radiation for imaging, making it less susceptible to ambient light conditions and offering all-weather, robust observation advantages, thus playing an irreplaceable role in scenarios such as nighttime surveillance and target tracking.

[0003] However, acquiring large-scale, high-quality real infrared images faces numerous challenges: infrared sensors are expensive, acquisition environments are limited, and infrared images from different scenarios vary significantly in thermal distribution and texture details, resulting in poor annotation consistency and limited dataset size. Therefore, synthesizing corresponding infrared images from readily available visible light images has become an important way to compensate for the scarcity of infrared data and support downstream vision tasks.

[0004] Currently, visible light to infrared image conversion methods based on deep generative models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) have made some progress. However, these methods are mostly statistical distribution matching and lack constraints on the physical laws of thermal radiation. As a result, although the generated images are visually feasible, they lack realism in radiation characteristics and are difficult to apply directly to visual tasks that require high image realism. Summary of the Invention

[0005] To address the problems in the existing generation methods mentioned above, such as insufficient modeling of the structural characteristics and semantic constraints of infrared images, leading to local structural distortion and poor consistency in synthesized images, this invention provides a visible light to infrared image generation method and system based on a prototype-guided diffusion model by introducing a prototype feature guidance mechanism and a multi-level attention fusion strategy.

[0006] To achieve the above objectives, a first aspect of the present invention provides a visible-to-infrared image generation method based on a prototype-guided diffusion model, comprising: Collect visible light images to be converted; Load the pre-built infrared prototype feature set and the parameters of the pre-trained prototype-guided diffusion model network; The visible light image to be converted and the infrared prototype feature set are input into the prototype-guided diffusion model network loaded with parameters to generate an infrared image corresponding to the visible light image to be converted. The prototype-guided diffusion model network includes: a prototype feature construction module, used to retrieve relevant prototype features from the infrared prototype feature set and generate a thermal semantic vector; a local prototype attention module, used to perform matching and fusion of local regions and infrared prototype features on the feature map of the diffusion model; and a diffusion noise prediction network, used to perform noise prediction in the diffusion denoising process by combining the thermal semantic vector and the conditional features extracted from the visible light image.

[0007] Furthermore, the infrared prototype feature set and the prototype-guided diffusion model network are obtained through a training phase, which includes: Acquire visible light images and their corresponding real infrared images as training image pairs; Construct an infrared prototype feature set; Data augmentation is performed on the training image pairs to obtain training samples; A prototype-guided diffusion model network is constructed, and the network is trained using the training samples and the infrared prototype feature set to obtain the trained model parameters.

[0008] Furthermore, the construction of the infrared prototype feature set specifically includes: The visible light image is segmented to obtain multiple region masks; wherein the region masks cover at least one structure among the ship body, the ship edge, the high reflectivity area of ​​the sea surface, and the engine heat source area; Based on the corresponding real infrared images, calculate the thermal response score of the region corresponding to each mask, and select high-heat regions according to the thermal response score; Extract the region sub-map of the high-heat region in the real infrared image, and perform latent space feature extraction on the region sub-map to obtain the region latent variable features; Cluster all the regional latent variable features to form the infrared prototype feature set.

[0009] Furthermore, the training of the prototype guided diffusion model network specifically includes: The visible light images used for training are input into the conditional encoder to extract visible light conditional features; The visible light conditional features and the infrared prototype feature set are input into the prototype feature construction module and the local prototype attention module to obtain a thermal semantic vector that integrates the infrared prototype features; Forward diffusion noise addition is performed on real infrared images in the latent space; In each step of diffusion denoising, the noisy latent variables, time step information, and the hot semantic vector are jointly input into the diffusion noise prediction network to predict the noise of the current step. The loss is calculated based on the predicted noise, and the parameters of the prototype guided diffusion model network are optimized.

[0010] Furthermore, during the training process, the diffusion noise prediction network calculates the predicted noise using the following formula: ; in, For network prediction noise, For the latent variables after adding noise, For time step information, For the aforementioned thermal semantic vector, This represents a conditional denoising network.

[0011] Furthermore, the specific steps for generating the infrared image include: Starting with random noise in the latent space, a back-diffusion denoising process is performed, wherein the noise prediction at each step is guided by a thermal semantic vector generated from the visible light image to be converted and the infrared prototype feature set. The latent variables obtained at the end of the backdiffusion process are restored to the image domain by the decoder to obtain the generated infrared image.

[0012] A second aspect of the present invention provides a visible-to-infrared image generation system based on a prototype-guided diffusion model, comprising: Image collection unit, used to collect visible light images to be converted; The resource loading unit is used to load the pre-built infrared prototype feature set and the parameters of the pre-trained prototype guided diffusion model network. An infrared image generation unit is used to input the visible light image to be converted and the infrared prototype feature set into the prototype guided diffusion model network loaded with parameters, and generate an infrared image corresponding to the visible light image to be converted.

[0013] A third aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps in the visible light to infrared image generation method based on a prototype guided diffusion model as described in the first aspect of the present invention.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the visible-to-infrared image generation method based on a prototype-guided diffusion model as described in the first aspect of the present invention.

[0015] A fifth aspect of the present invention provides a computer program product comprising software code, wherein the program in the software code performs the steps of the visible light to infrared image generation method based on a prototype-guided diffusion model as described in the first aspect of the present invention.

[0016] Compared with existing technologies, the visible light to infrared image generation method and system based on a prototype-guided diffusion model provided by this invention have the following advantages: (1) This invention extracts representative thermal pattern prototypes from real infrared data and uses them as strong semantic priors to guide the generation process, so that the diffusion model can explicitly align with the statistical law of thermal distribution in the infrared domain, effectively alleviating the thermal distribution prediction bias caused by "different colors and different spectra"; thereby significantly improving the physical consistency of the generated infrared image in terms of heat source morphology, thermal gradient continuity and other aspects.

[0017] (2) By performing fine matching and fusion of local regions and infrared prototypes on the multi-scale feature map of the diffusion model in the form of a sliding window, the present invention realizes the differentiated and refined guidance of infrared thermal features on different spatial locations of the image, thereby effectively improving the reconstruction quality and structural fidelity of local details such as small targets, edges and complex heat source areas.

[0018] (3) In this invention, prototype features are injected as stabilizing conditions into each step of the denoising prediction of the diffusion model. This guidance mechanism provides the model with a clear direction for infrared domain optimization, enhances the stability of the training process, and improves the model's understanding and alignment of cross-modal semantics, making the generation process more controllable and reliable. Attached Figure Description

[0019] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0020] Figure 1 This is a flowchart of a visible light to infrared image generation method based on a prototype-guided diffusion model, provided in Embodiment 1 of the present invention.

[0021] Figure 2 The overall flowchart of the visible light to infrared image generation method based on the prototype guided diffusion model provided in Embodiment 1 of the present invention can be divided into a training stage on the left and an inference / generation stage on the right.

[0022] Figure 3 This is a training structure diagram of the prototype guided diffusion model network provided in Embodiment 1 of the present invention.

[0023] Figure 4This is a diagram illustrating the inference structure of the prototype guided diffusion model network provided in Embodiment 1 of the present invention.

[0024] Figure 5 This is a network structure diagram of the prototype feature construction module provided in Embodiment 1 of the present invention.

[0025] Figure 6 This is a network structure diagram of a local prototype attention module provided in Embodiment 1 of the present invention.

[0026] Figure 7 This is an architecture diagram of a visible light to infrared image generation system based on a prototype-guided diffusion model, provided in Embodiment 2 of the present invention. Detailed Implementation

[0027] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0028] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0030] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0031] Example 1 like Figure 1 This embodiment provides a visible light to infrared image generation method based on a prototype-guided diffusion model, including: Collect visible light images to be converted; Load the pre-built infrared prototype feature set and the parameters of the pre-trained prototype-guided diffusion model network; The visible light image to be converted and the infrared prototype feature set are input into the prototype-guided diffusion model network loaded with parameters to generate an infrared image corresponding to the visible light image to be converted. The prototype-guided diffusion model network includes: a prototype feature construction module, used to retrieve relevant prototype features from the infrared prototype feature set and generate a thermal semantic vector; a local prototype attention module, used to perform matching and fusion of local regions and infrared prototype features on the feature map of the diffusion model; and a diffusion noise prediction network, used to perform noise prediction in the diffusion denoising process by combining the thermal semantic vector with the conditional features extracted from the visible light image.

[0032] This method uses a pre-generated set of infrared prototype features containing knowledge of infrared thermal patterns as a reference for the generation process, inputting it along with the visible light image to be processed into an optimized prototype-guided diffusion model network. The prototype feature construction module in this network is responsible for quickly retrieving the most relevant thermal features to the current scene from a massive set of prototypes, forming a condensed thermal semantic vector. The local prototype attention module performs fine-grained, position-aware prototype matching on feature maps at different levels within the network, ensuring that thermal features are accurately injected into the corresponding local regions of the image. The diffusion noise prediction network ultimately utilizes this guiding information to predict in a direction consistent with the true infrared thermal distribution at each step of the denoising process. Through this series of operations, the problem of blind and physically incompatible thermal distribution in existing methods for generating infrared images is solved, enabling stable and controllable generation of highly physically realistic infrared images based on the semantic content of the input visible light image.

[0033] Specifically, the infrared prototype feature set and the prototype-guided diffusion model network are obtained through a training phase, which includes: Acquire visible light images and their corresponding real infrared images as training image pairs; Construct an infrared prototype feature set; Data augmentation is performed on the training image pairs to obtain training samples; A prototype-guided diffusion model network is constructed, and the network is trained using the training samples and the infrared prototype feature set to obtain the trained model parameters.

[0034] This method decomposes the model's learning process into two stages: knowledge extraction (building a prototype library) and knowledge application (training the generative network). First, interpretable thermal pattern prototypes are extracted unsupervised from real infrared images to form a stable prior knowledge base. Then, the diffusion model is trained to learn how to utilize this prior knowledge for conditional generation. This two-stage paradigm solves the problems of training instability and easy mode collapse caused by end-to-end models simultaneously learning modality alignment and image generation. The model training process is more robust, and the generated infrared images closely match the thermal radiation patterns summarized from real data.

[0035] Specifically, the construction of the infrared prototype feature set includes: The visible light image is segmented to obtain multiple region masks; wherein the region masks cover at least one structure among the ship body, the ship edge, the high reflectivity area of ​​the sea surface, and the engine heat source area; Based on the corresponding real infrared images, calculate the thermal response score of the region corresponding to each mask, and select high-heat regions according to the thermal response score; Extract the region sub-map of the high-heat region in the real infrared image, and perform latent space feature extraction on the region sub-map to obtain the region latent variable features; Cluster all the regional latent variable features to form the infrared prototype feature set.

[0036] This method mimics how humans understand infrared images—focusing on salient heat source regions. By ranking thermal responses, the system automatically focuses on the thermal areas with the most information; latent variable features extracted by a variational autoencoder capture the essential thermal patterns of the regions; and finally, clustering summarizes these essential features into a finite number of prototypes. This process solves the problem of the difficulty and incompleteness of manually defining thermal features, automatically constructing a well-represented infrared prototype knowledge base covering typical thermal scenarios, providing reliable and rich thermal semantic priors for subsequent generation.

[0037] Specifically, training the prototype guided diffusion model network includes: The visible light images used for training are input into the conditional encoder to extract visible light conditional features; The visible light conditional features and the infrared prototype feature set are input into the prototype feature construction module and the local prototype attention module to obtain a thermal semantic vector that integrates the infrared prototype features; Forward diffusion noise addition is performed on real infrared images in the latent space; In each step of diffusion denoising, the noisy latent variables, time step information, and the hot semantic vector are jointly input into the diffusion noise prediction network to predict the noise of the current step. The loss is calculated based on the predicted noise, and the parameters of the prototype guided diffusion model network are optimized.

[0038] Specifically, during the training process, the diffuse noise prediction network calculates the predicted noise using the following formula: ; in, For network prediction noise, For the latent variables after adding noise, For time step information, For the aforementioned thermal semantic vector, This represents a conditional denoising network.

[0039] In the standard denoising framework of the diffusion model, a time-step-dependent conditional vector generated from prototype features is introduced. This conditional vector is used in each step of the denoising prediction. The middle part plays a guiding role, forcing the predicted noise to not only remove random noise, but also to The model is modified to conform to the thermal distribution space defined by the infrared prototype characteristics. This solves the problem that it is difficult to control specific properties of the output mode (such as thermal distribution) when the diffusion model is generated under unconditional or weak conditions. This enables the diffusion model to generate under cross-modal conditions, and the generated latent variables can be accurately mapped to the infrared image space with a reasonable thermal structure.

[0040] Specifically, the steps for generating the infrared image include: Starting with random noise in the latent space, a back-diffusion denoising process is performed, wherein the noise prediction at each step is guided by a thermal semantic vector generated from the visible light image to be converted and the infrared prototype feature set. The latent variables obtained at the end of the backdiffusion process are restored to the image domain by the decoder to obtain the generated infrared image.

[0041] This method performs a controlled iterative optimization process from pure noise to a target infrared image. In each iteration, the network predicts the most likely denoising direction based on the current noisy image and conditional guidance, gradually subtracting noise and injecting infrared feature information. This solves the problem of detail loss or structural disorder caused by traditional methods that may generate images in a single step, and can generate high-quality infrared images with rich details, continuous thermal structure, and conformity to physical laws.

[0042] In one specific embodiment, such as Figure 2 As shown, this visible-to-infrared image generation method based on a prototype-guided diffusion model includes: 1. Acquire visible light and infrared registered images and construct an infrared prototype library; First, visible light and infrared image pairs are acquired. For each image pair, the RGB image is used as the semantic structure reference, and the IR image is used as the thermal radiation reference input. Using the visible light image, a pre-trained segmentation model (SAM) is used to automatically generate structural masks covering the ship's main body, hull edges, high-reflectivity sea surface areas, and engine heat source areas. The corresponding infrared image is then used to calculate the thermal response map of each region. Based on the thermal response intensity, several thermally salient regions are selected from all regions and used as the basic units for subsequent prototype feature construction.

[0043] Subsequently, the selected region is cropped from the infrared image to obtain a sub-image, and the latent space thermal features of this region are extracted using a fixed infrared encoder. After summing the region features from all images, K-means clustering is used to extract K (K=100) infrared prototype centers, each representing a typical thermal structure pattern. The resulting infrared prototype library serves as the thermal semantic prior for the diffusion generation process. Finally, all registered images are proportionally divided into training and test sets. The training set is used to optimize the generation model parameters, while the test set is used to verify the quality and structural consistency of the generated infrared images.

[0044] 2. Data augmentation and unified preprocessing of input data; First, the input visible light and infrared images are uniformly scaled to a fixed resolution of 512×512; random image cropping and horizontal flipping are then performed. All enhancement operations must maintain consistency between the RGB image and its corresponding IR image; the enhanced image is then normalized to linearly map its pixel value range to […]. The interval is [1, 1].

[0045] 3. Build and train a prototype-guided diffusion generation network. A prototype-guided diffusion generation network was built, and the network parameters were optimized using data-enhanced visible light and infrared images. The L1 loss function was used to supervise the training of the network, and the optimal network parameters were saved. First, infrared images By fixed encoder Mapping to the latent space representation: ; in, The height of the original image. The width of the original image. The spatial resolution of the latent feature map. For potential feature dimensions.

[0046] Meanwhile, visible light images via condition encoder Feature extraction: ; Introducing a pre-built set of infrared prototypes Each of them Indicate category The infrared modal feature center. During training, conditional features... Cross-attention fusion is performed between the local prototype attention module and the infrared prototype: ; Conditional characteristics after fusion It carries infrared structural priors to guide the denoising prediction of the diffusion model.

[0047] A forward diffusion process is applied to the latent space to gradually add noise to the latent variables: ; in, , Indicates noise scheduling parameters, The noise is random Gaussian noise that follows a standard normal distribution. At each time step... The model is based on the conditions Predicted noise: ; in, This represents the conditional U-Net denoising network. Because the prototype features provide physical constraints in the infrared domain, the model can more effectively align with the distribution of infrared thermal features while learning the noise map. In each training iteration, the model recovers potentially clean samples from the predicted noise: ; Then through a fixed decoder Mapping back to image space: ; decoder During training, the model remains frozen, providing only a reversible mapping between the latent space and the pixel space. To enhance the model's infrared physical consistency, prototype features are used as high-weight conditional information in the noise prediction phase to guide multiple time steps, enabling the model to be guided by thermal features at different diffusion stages.

[0048] 4. For practical applications of infrared image generation, the first step is to acquire the RGB image of the ship to be generated; after the preprocessing module unifies the size, it is input into the model; the model performs a diffusion backsampling process in the latent space with random noise as the starting point; in each diffusion step, the prototype feature constraint module and the local attention module jointly guide the generation of thermal structures; when the diffusion step is completed, the final latent variables are restored by the decoder into an infrared image with a real thermal radiation pattern.

[0049] To test the effectiveness of this algorithm in generating infrared images, it was evaluated on the publicly available datasets KAIST and FLIR. The final test results showed PSNR of 24.32 and 17.81, SSIM of 0.9285 and 0.6817, FID of 50.44 and 80.49, and LPIPS of 0.126 and 0.301, surpassing most existing methods. Furthermore, a ship recognition model was trained and tested on the publicly available dataset VAIS. Using the infrared images generated by the method provided in this invention as training data, the detection performance was improved, with results approaching or even approaching the upper limit performance of training with real infrared data.

[0050] Example 2 like Figure 7 This embodiment provides a visible light to infrared image generation system based on a prototype-guided diffusion model, including: Image collection unit, used to collect visible light images to be converted; The resource loading unit is used to load the pre-built infrared prototype feature set and the parameters of the pre-trained prototype guided diffusion model network. An infrared image generation unit is used to input the visible light image to be converted and the infrared prototype feature set into the prototype guided diffusion model network loaded with parameters, and generate an infrared image corresponding to the visible light image to be converted.

[0051] Example 3 Embodiment 3 of the present invention provides an electronic device.

[0052] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, it implements the steps in the visible light to infrared image generation method based on a prototype guided diffusion model as described in Embodiment 1 of the present invention.

[0053] The detailed steps are the same as those of the visible light to infrared image generation method based on the prototype guided diffusion model provided in Example 1, and will not be repeated here.

[0054] Example 4 Embodiment 4 of the present invention provides a computer-readable storage medium.

[0055] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the visible light to infrared image generation method based on a prototype guided diffusion model as described in Embodiment 1 of the present invention.

[0056] The detailed steps are the same as those of the visible light to infrared image generation method based on the prototype guided diffusion model provided in Example 1, and will not be repeated here.

[0057] Example 5 Embodiment 5 of the present invention provides a computer program product.

[0058] A computer program product includes software code, wherein the program in the software code performs the steps of the visible light to infrared image generation method based on the prototype-guided diffusion model as described in Embodiment 1 of the present invention.

[0059] The detailed steps are the same as those of the visible light to infrared image generation method based on the prototype guided diffusion model provided in Example 1, and will not be repeated here.

[0060] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0061] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0062] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0063] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0064] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A method for visible to infrared image generation based on a prototype-guided diffusion model, characterized by, include: Collect visible light images to be converted; Load the pre-built infrared prototype feature set and the parameters of the pre-trained prototype-guided diffusion model network; The visible light image to be converted and the infrared prototype feature set are input into the prototype-guided diffusion model network loaded with parameters to generate an infrared image corresponding to the visible light image to be converted. The prototype-guided diffusion model network includes: a prototype feature construction module, used to retrieve relevant prototype features from the infrared prototype feature set and generate a thermal semantic vector; a local prototype attention module, used to perform matching and fusion of local regions and infrared prototype features on the feature map of the diffusion model; and a diffusion noise prediction network, used to perform noise prediction in the diffusion denoising process by combining the thermal semantic vector with the conditional features extracted from the visible light image. The infrared prototype feature set and the prototype guided diffusion model network are obtained through a training phase, which includes: acquiring visible light images and their corresponding real infrared images as training image pairs; constructing an infrared prototype feature set; performing data augmentation on the training image pairs to obtain training samples; building a prototype guided diffusion model network; and training the prototype guided diffusion model network using the training samples and the infrared prototype feature set to obtain trained model parameters. The training of the prototype-guided diffusion model network specifically includes: inputting a visible light image for training into a conditional encoder to extract visible light conditional features; inputting the visible light conditional features and the infrared prototype feature set into the prototype feature construction module and the local prototype attention module to obtain a thermal semantic vector fused with the infrared prototype features; performing forward diffusion noise addition on the real infrared image in the latent space; at each step of diffusion denoising, inputting the denoised latent variables, time step information, and the thermal semantic vector into the diffusion noise prediction network to predict the noise of the current step; calculating the loss based on the predicted noise and optimizing the parameters of the prototype-guided diffusion model network.

2. The method as described in claim 1, characterized in that, The construction of the infrared prototype feature set specifically includes: The visible light image is segmented to obtain multiple region masks; wherein the region masks cover at least one structure among the ship body, the ship edge, the high reflectivity area of ​​the sea surface, and the engine heat source area; Based on the corresponding real infrared images, calculate the thermal response score of the region corresponding to each mask, and select high-heat regions according to the thermal response score; Extract the region sub-map of the high-heat region in the real infrared image, and perform latent space feature extraction on the region sub-map to obtain the region latent variable features; Cluster all the regional latent variable features to form the infrared prototype feature set.

3. The method as described in claim 1, characterized in that, During training, the noise prediction network calculates the predicted noise using the following formula: ； in, For network prediction noise, For the latent variables after adding noise, For time step information, For the aforementioned thermal semantic vector, This represents a conditional denoising network.

4. The method as described in claim 1, characterized in that, The specific steps for generating the infrared image include: Starting with random noise in the latent space, a back-diffusion denoising process is performed, wherein the noise prediction at each step is guided by a thermal semantic vector generated from the visible light image to be converted and the infrared prototype feature set. The latent variables obtained at the end of the backdiffusion process are restored to the image domain by the decoder to obtain the generated infrared image.

5. A visible light to infrared image generation system based on a prototype-guided diffusion model, characterized in that, include: Image collection unit, used to collect visible light images to be converted; The resource loading unit is used to load the pre-built infrared prototype feature set and the parameters of the pre-trained prototype guided diffusion model network. An infrared image generation unit is used to input the visible light image to be converted and the infrared prototype feature set into the prototype-guided diffusion model network loaded with parameters to generate an infrared image corresponding to the visible light image to be converted. The prototype-guided diffusion model network includes: a prototype feature construction module, used to retrieve relevant prototype features from the infrared prototype feature set and generate a thermal semantic vector; a local prototype attention module, used to perform matching and fusion of local regions and infrared prototype features on the feature map of the diffusion model; and a diffusion noise prediction network, used to perform noise prediction by combining the thermal semantic vector and the conditional features extracted from the visible light image during the diffusion denoising process. The infrared prototype feature set and the prototype guided diffusion model network are obtained through a training phase, which includes: acquiring visible light images and their corresponding real infrared images as training image pairs; constructing an infrared prototype feature set; performing data augmentation on the training image pairs to obtain training samples; building a prototype guided diffusion model network; and training the prototype guided diffusion model network using the training samples and the infrared prototype feature set to obtain trained model parameters. The training of the prototype-guided diffusion model network specifically includes: inputting a visible light image for training into a conditional encoder to extract visible light conditional features; inputting the visible light conditional features and the infrared prototype feature set into the prototype feature construction module and the local prototype attention module to obtain a thermal semantic vector fused with the infrared prototype features; performing forward diffusion noise addition on the real infrared image in the latent space; at each step of diffusion denoising, inputting the denoised latent variables, time step information, and the thermal semantic vector into the diffusion noise prediction network to predict the noise of the current step; calculating the loss based on the predicted noise and optimizing the parameters of the prototype-guided diffusion model network.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of the visible light to infrared image generation method based on the prototype guided diffusion model as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the visible light to infrared image generation method based on the prototype guided diffusion model as described in any one of claims 1 to 4.

8. A computer program product, comprising software code, characterized in that, The program in the software code performs the steps of the visible light to infrared image generation method based on the prototype-guided diffusion model as described in any one of claims 1 to 4.

Images