Computer image generation method based on multi-modal image conversion

Abstract

The invention discloses a computer image generation method based on multi-modal image conversion. The computer image generation method comprises the following steps: step 1, constructing an improved CycleGAN network architecture; step 2, performing initialization setting; step 3, constructing a data set; step 4, model training; 5, generating a primary visible light color image; step 6, self-adaptive post-processing is carried out; according to the method, color generation is guided through temperature and scene information, the problem of color distortion of an existing CycleGAN is solved, and the quality of the generated image is ensured through quality index evaluation and post-processing optimization of the generated image; by adopting a U-Net + + network structure, the capturing capability of image multi-scale details is enhanced, and the loss of weak thermal signals is avoided; by introducing a self-attention module, the ability of the model to capture global semantic information is enhanced, training oscillation caused by local feature over-fitting is avoided, and training stability and model performance are improved through adaptive weight adjustment.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a computer image generation method based on multimodal image conversion. Background Art

[0002] Thermal infrared images capture the thermal radiation of objects and play a key role in scenarios such as nighttime surveillance, security inspection, and industrial fault diagnosis. However, their grayscale characteristics make it difficult to intuitively display scene details and lack natural color information, resulting in low target recognition efficiency. Visible light images, with their rich color and texture features, are easier for humans to observe and understand, but they are significantly limited by lighting conditions and struggle to capture effective information in low-light or dark environments. Therefore, the conversion technology between thermal infrared and visible light images has important research value and broad application prospects. Among existing methods for converting thermal infrared images to visible light images, unsupervised image conversion methods are generally used. CycleGAN, as a classic model for unsupervised image-to-image conversion, can achieve cross-domain conversion without strictly paired training data. However, it still has the following shortcomings: 1. CycleGAN focuses on the overall authenticity of the image during image conversion rather than precise color matching. Therefore, there is a problem of color distortion, and the generated visible light color image cannot completely and accurately restore the color of the real visible light image; 2. In complex conversion tasks, CycleGAN tends to ignore some specific patterns in the source domain, resulting in the loss of some details after conversion. For example, some weak thermal signal features in thermal infrared images are easily not well preserved when converted to visible light color images, which affects the analysis and understanding of image details; 3. Because a delicate balance needs to be achieved between the generator and the discriminator, during the confrontation process, the update speed and parameter adjustment of both parties will cause oscillations in the CycleGAN training process, which not only leads to a decline in the quality of generated images and reduced training efficiency, but also weakens the generalization ability and reliability of the model in practical applications. Summary of the Invention

[0003] The purpose of the present invention is to provide a computer image generation method based on multimodal image conversion to solve the problems raised in the above background technology.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a computer image generation method based on multimodal image conversion, comprising the following steps: step 1, constructing an improved CycleGAN network architecture; step 2, initializing settings; step 3, constructing a dataset; step 4, model training; step 5, generating a preliminary visible light color image; step 6, adaptive post-processing;

[0005] In the above step 1, two pairs of generators and discriminators of the improved CycleGAN are constructed;

[0006] In the above step 2, the weight parameters of all generators and discriminators are initialized using Gaussian distribution, the dynamic weight parameters of the loss function are initialized, and the Adam optimizer is configured;

[0007] In step 3 above, an unpaired image dataset of source domain A and target domain B is constructed, where source domain A is a thermal infrared image containing samples of different scenes and different temperature ranges, and target domain B is a visible light color image of the corresponding scene;

[0008] In step 4 above, the dataset constructed in step 3 is used for model training. In each iteration, the discriminator is updated first. For each discriminator, the loss of the real image and the generated image is calculated respectively. The discriminator parameters are back-propagated and updated. Then, the generator is updated. The adversarial loss, cycle consistency loss, temperature constraint loss, and color fidelity loss are calculated. The generator parameters are back-propagated and updated. The weight parameters are then adjusted according to the dynamic weight adjustment strategy. The model performance is regularly evaluated on the validation set to obtain the best model.

[0009] In the above step 5, the thermal infrared image, temperature information and scene label are input into the best model obtained in step 4, and the generator G A→B Generate preliminary visible light color images;

[0010] In the above step 6, the quality index of the image generated in step 5 is calculated. If the quality index of the generated image is lower than a preset threshold, the generated image is post-processed and optimized.

[0011] Preferably, in step 1, the two pairs of generators include generator G A→B and generator G B→A , the two pairs of discriminators include discriminator D B and the discriminator D A .

[0012] Preferably, the generator G A→B The input is a thermal infrared image Temperature information and scene tags Output is visible light color image Generator G A→B Using U-Net++ network, thermal infrared image Extract image features F through convolutional layers A =ConvNet(A), temperature information is embedded into feature vector E through the fully connected layer t =FC(t), the scene label is embedded into the feature vector E through the fully connected layer s =FC(s), temperature and scene features are concatenated with image features in the channel dimension to obtain the fusion feature F concat=Concat(F A ,E t ,E s ); Among them, the encoder of the U-Net++ network gradually extracts multi-scale features, and the decoder restores spatial resolution through skip connections. Specifically, the encoder gradually extracts multi-scale features through 5 downsampling blocks, each of which contains 2 convolutional layers and 1 maximum pooling layer, and introduces a self-attention module after the third sampling block; the decoder restores spatial resolution through skip connections and 5 upsampling blocks, each of which contains 2 convolutional layers and 1 transposed convolutional layer, and adopts an attention gating mechanism in the decoder to selectively focus on important features.

[0013] Preferably, the discriminator D B The input is a visible light color image The PatchGAN architecture is adopted, and a multi-layer convolutional network is used to gradually extract features and reduce spatial resolution. The last layer uses 1×1 convolution to output the discrimination result map. Among them, the multi-layer convolutional network uses 5 convolution blocks, each of which contains a convolution layer, a LeakyReLU activation function and spectral normalization. A self-attention module is inserted after the third convolution block.

[0014] Preferably, the generator G B→A The input is a visible light color image The output layer uses the tanh activation function to map the output value to the range of [-1,1], which corresponds to the temperature range of the thermal infrared image, and outputs the thermal infrared image. Generator G B→A Using U-Net network, visible light color image Extract features F through convolutional layers B =ConvNet(B); the U-Net encoder gradually extracts multi-scale features through four downsampling blocks, each of which contains a convolutional layer, instance normalization, and a LeakyReLU activation function; the decoder restores spatial resolution through four upsampling blocks, each of which contains a convolutional layer, instance normalization, and a LeakyReLU activation function.

[0015] Preferably, the discriminator D A The input is a thermal infrared image The PatchGAN architecture is adopted, and a multi-layer convolutional network is used to gradually extract features and reduce spatial resolution. The last layer uses 1×1 convolution to output the discrimination result map. The multi-layer convolutional network uses 4 convolution blocks, each of which contains a convolution layer, a LeakyReLU activation function and spectral normalization. A self-attention module is inserted after the third convolution block.

[0016] Preferably, in step 1, the loss function of the improved CycleGAN is specifically:

[0017] Fighting Loss:

[0018]

[0019] Cycle consistency loss:

[0020]

[0021] Temperature constraint loss:

[0022]

[0023] Where T() is the function of estimating temperature from the visible light image;

[0024] Loss of color fidelity:

[0025]

[0026] Where ΔE is the color difference, B * is the reference visible light image;

[0027] Total loss function:

[0028] L total =L adv (G A→B ,D B )+L adv (G B→A ,D A )+λ1·L cycle +λ2·L temp +λ3·L color

[0029] where λ1 is L cycle The dynamic weight parameter, the initial value is 10, λ2 is L temp The dynamic weight parameter, the initial value is 5, λ3 is L color The dynamic weight parameter of , the initial value is 3.

[0030] Preferably, in step 4, the dynamic weight adjustment strategy is specifically:

[0031] λ1(t)=λ 1,init ·exp(-α1·t)

[0032] λ2(t)=λ 2,init ·(1-exp(-α2·t))

[0033] λ3(t)=λ 3,init min(1,t / τ)

[0034] Where t is the number of training iterations, and α1, α2, and τ are hyperparameters.

[0035] Preferably, in step six, the quality indicators of the generated image include the color fidelity indicator CIEDE2000 color difference and the structural similarity SSIM.

[0036] Preferably, in step six, the post-processing optimization includes guided filtering and color correction. Guided filtering utilizes edge information of the thermal infrared image to optimize color boundaries and enhance details. Color correction adjusts the hue and saturation of the generated image based on a reference color distribution.

[0037] Compared with the existing technology, the beneficial effects of the present invention are as follows: the present invention uses temperature and scene information to guide color generation, solves the color distortion problem of the existing CycleGAN, and ensures the quality of the generated image by evaluating the quality indicators of the generated image and post-processing optimization; by adopting the U-Net++ network structure, the ability to capture multi-scale details of the image is enhanced, avoiding the loss of weak thermal signals; by introducing the self-attention module, the model's ability to capture global semantic information is enhanced, avoiding training oscillations caused by overfitting of local features, and improving training stability and model performance through adaptive weight adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] Please see the attached Figure 1 The present invention provides an embodiment of a computer image generation method based on multimodal image conversion, comprising the following steps: step 1, constructing an improved CycleGAN network architecture; step 2, initializing settings; step 3, constructing a data set; step 4, model training; step 5, generating a preliminary visible light color image; step 6, adaptive post-processing;

[0041] In the above step 1, two pairs of generators and discriminators of improved CycleGAN are constructed. The two pairs of generators include generator G A→B and generator G B→A , the two pairs of discriminators include discriminator D B and the discriminator DA ; Generator G A→B The input is a thermal infrared image Temperature information and scene tags Output is visible light color image Generator G A→B Using U-Net++ network, thermal infrared image Extract image features F through convolutional layers A =ConvNet(A), temperature information is embedded into feature vector E through the fully connected layer t =FC(t), the scene label is embedded into the feature vector E through the fully connected layer s =FC(s), temperature and scene features are concatenated with image features in the channel dimension to obtain the fusion feature F concat =Concat(F A ,E t ,E s ); Among them, the encoder of the U-Net++ network gradually extracts multi-scale features, and the decoder restores spatial resolution through skip connections. Specifically, the encoder gradually extracts multi-scale features through 5 downsampling blocks, each of which contains 2 convolutional layers and 1 maximum pooling layer, and introduces a self-attention module after the third sampling block; the decoder restores spatial resolution through skip connections and 5 upsampling blocks, each of which contains 2 convolutional layers and 1 transposed convolutional layer, and the attention gating mechanism is used in the decoder to selectively focus on important features; the discriminator D B The input is a visible light color image The PatchGAN architecture is used to gradually extract features and reduce spatial resolution using a multi-layer convolutional network. The last layer uses a 1×1 convolution to output the discrimination result map. The multi-layer convolutional network uses 5 convolution blocks, each of which contains a convolution layer, a LeakyReLU activation function, and spectral normalization. A self-attention module is inserted after the third convolution block. The generator G B→A The input is a visible light color image The output layer uses the tanh activation function to map the output value to the range of [-1,1], which corresponds to the temperature range of the thermal infrared image, and outputs the thermal infrared image. Generator G B→A Using U-Net network, visible light color image Extract features F through convolutional layers B=ConvNet(B); The encoder of the U-Net network gradually extracts multi-scale features through 4 downsampling modules, each of which contains a convolution layer, instance normalization, and a LeakyReLU activation function; the decoder restores the spatial resolution through 4 upsampling blocks, each of which contains a convolution layer, instance normalization, and a LeakyReLU activation function; the discriminator D A The input is a thermal infrared image The PatchGAN architecture is adopted, and a multi-layer convolutional network is used to gradually extract features and reduce spatial resolution. The last layer uses 1×1 convolution to output the discrimination result map. The multi-layer convolutional network uses 4 convolution blocks, each of which contains a convolution layer, a LeakyReLU activation function and spectral normalization. A self-attention module is inserted after the third convolution block. The loss function of the improved CycleGAN is specifically:

[0042] Fighting Loss:

[0043]

[0044] Cycle consistency loss:

[0045]

[0046] Temperature constraint loss:

[0047]

[0048] Where T() is the function of estimating temperature from the visible light image;

[0049] Loss of color fidelity:

[0050]

[0051] Where ΔE is the color difference, B * is the reference visible light image;

[0052] Total loss function:

[0053] L total =L adv (G A→B ,D B )+L adv (G B→A ,D A )+λ1·L cycle +λ2·L temp +λ3·L color

[0054] where λ1 is L cycleThe dynamic weight parameter, the initial value is 10, λ2 is L temp The dynamic weight parameter, the initial value is 5, λ3 is L color The dynamic weight parameter of , the initial value is 3;

[0055] In the above step 2, the weight parameters of all generators and discriminators are initialized using Gaussian distribution, the dynamic weight parameters of the loss function are initialized, and the Adam optimizer is configured;

[0056] In step 3 above, an unpaired image dataset of source domain A and target domain B is constructed, where source domain A is a thermal infrared image containing samples of different scenes and different temperature ranges, and target domain B is a visible light color image of the corresponding scene;

[0057] In step 4 above, the dataset constructed in step 3 is used for model training. In each iteration, the discriminator is updated first. For each discriminator, the loss of the real image and the generated image is calculated respectively. The discriminator parameters are back-propagated and updated. Then, the generator is updated. The adversarial loss, cycle consistency loss, temperature constraint loss, and color fidelity loss are calculated. The generator parameters are back-propagated and updated. The weight parameters are then adjusted according to the dynamic weight adjustment strategy. The model performance is regularly evaluated on the validation set to obtain the best model. The dynamic weight adjustment strategy is specifically as follows:

[0058] λ1(t)=λ 1,init ·exp(-α1·t)

[0059] λ2(t)=λ 2,init ·(1-exp(-α2·t))

[0060] λ3(t)=λ 3,init min(1,t / τ)

[0061] Where t is the number of training iterations, α1, α2, τ are hyperparameters;

[0062] In the above step 5, the thermal infrared image, temperature information and scene label are input into the best model obtained in step 4, and the generator G A→B Generate preliminary visible light color images;

[0063] In the above step six, the quality index of the image generated in step five is calculated. If the quality index of the generated image is lower than a preset threshold, the generated image is post-processed and optimized. The quality index of the generated image includes the color fidelity index CIEDE2000 color difference and the structural similarity SSIM. The post-processing optimization includes guided filtering and color correction. The guided filtering uses the edge information of the thermal infrared image to optimize the color boundary and enhance the details. The color correction adjusts the hue and saturation of the generated image based on the reference color distribution.

[0064] Experimental example:

[0065] The method provided in the above embodiment is used to convert the thermal infrared image into a visible light color image. The thermal infrared image is an image of a human body in an indoor environment. The human body temperature is about 37°C and the background temperature is about 25°C. The results are shown in the following table:

[0066]

[0067] Based on the above, the advantages of the present invention are that, when the present invention is used, by adding temperature information and scene labels to the input of the visible light color image generator, the model is guided to generate colors that are more in line with the real distribution according to different scenes, and by calculating the color fidelity and structural similarity of the generated image, post-processing optimization is automatically triggered when the quality index of the image is lower than the threshold, thereby improving the visual effect of the generated image; the visible light color image generator adopts the U-Net++ network to enhance the ability to capture multi-scale details of the image and avoid the smoothing of small-size thermal features during the conversion. By adding the temperature distribution constraint of the thermal infrared image to the loss function, the loss of key thermal signal features during the image conversion process is avoided; by introducing the self-attention module in the generator and the discriminator, the model's ability to capture global semantic information is enhanced, and training oscillations caused by overfitting of local features are avoided. By dynamically adjusting the loss function weight, the adversarial loss weight is emphasized in the early stage of training to help the model quickly learn basic conversion capabilities. In the middle of training, the cycle consistency loss weight is increased to ensure structure retention. In the later stage of training, the weight of temperature constraint and color fidelity loss is gradually increased to optimize details and color accuracy.

[0068] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A computer image generation method based on multimodal image conversion, comprising the following steps: Step 1: Build an improved CycleGAN network architecture; Step 2: Initialize settings; Step 3: Build a dataset; Step 4: Model training; Step 5: Generate a preliminary visible light color image; Step 6: Adaptive post-processing; It is characterized by: In the above step 1, two pairs of generators and discriminators of the improved CycleGAN are constructed; In the above step 2, the weight parameters of all generators and discriminators are initialized using Gaussian distribution, the dynamic weight parameters of the loss function are initialized, and the Adam optimizer is configured; In step 3 above, an unpaired image dataset of source domain A and target domain B is constructed, where source domain A is a thermal infrared image containing samples of different scenes and different temperature ranges, and target domain B is a visible light color image of the corresponding scene; In step 4 above, the dataset constructed in step 3 is used for model training. In each iteration, the discriminator is updated first. For each discriminator, the loss of the real image and the generated image is calculated respectively. The discriminator parameters are back-propagated and updated. Then, the generator is updated. The adversarial loss, cycle consistency loss, temperature constraint loss, and color fidelity loss are calculated. The generator parameters are back-propagated and updated. The weight parameters are then adjusted according to the dynamic weight adjustment strategy. The model performance is regularly evaluated on the validation set to obtain the best model. In the above step 5, the thermal infrared image, temperature information and scene label are input into the best model obtained in step 4, and the generator G A→B Generate preliminary visible light color images; In the above step 6, the quality index of the image generated in step 5 is calculated. If the quality index of the generated image is lower than a preset threshold, the generated image is post-processed and optimized.

2. The computer image generation method based on multimodal image conversion according to claim 1, characterized in that: In step 1, the two pairs of generators include generator G A→B and generator G B→A , the two pairs of discriminators include discriminator D B and the discriminator D A .

3. The computer image generation method based on multimodal image conversion according to claim 2, characterized in that: The generator G A→B The input is a thermal infrared image Temperature information and scene tags Output is visible light color image Generator G A→B Using U-Net++ network, thermal infrared image Extract image features F through convolutional layers A =ConvNet(A), temperature information is embedded into feature vector E through the fully connected layer t =FC(t), the scene label is embedded into the feature vector E through the fully connected layer s =FC(s), temperature and scene features are concatenated with image features in the channel dimension to obtain the fusion feature F concat =Concat(F A ,E t ,E s ); Among them, the encoder of the U-Net++ network gradually extracts multi-scale features, and the decoder restores spatial resolution through skip connections. Specifically, the encoder gradually extracts multi-scale features through 5 downsampling blocks, each of which contains 2 convolutional layers and 1 maximum pooling layer, and introduces a self-attention module after the third sampling block; the decoder restores spatial resolution through skip connections and 5 upsampling blocks, each of which contains 2 convolutional layers and 1 transposed convolutional layer, and adopts an attention gating mechanism in the decoder to selectively focus on important features.

4. The computer image generation method based on multimodal image conversion according to claim 2, characterized in that: The discriminator D B The input is a visible light color image The PatchGAN architecture is adopted, and a multi-layer convolutional network is used to gradually extract features and reduce spatial resolution. The last layer uses 1×1 convolution to output the discrimination result map. Among them, the multi-layer convolutional network uses 5 convolution blocks, each of which contains a convolution layer, a LeakyReLU activation function and spectral normalization. A self-attention module is inserted after the third convolution block.

5. The computer image generation method based on multimodal image conversion according to claim 2, characterized in that: The generator G B→A The input is a visible light color image The output layer uses the tanh activation function to map the output value to the range of [-1,1], which corresponds to the temperature range of the thermal infrared image, and outputs the thermal infrared image. Generator G B→A Using U-Net network, visible light color image Extract features F through convolutional layers B =ConvNet(B); the U-Net encoder gradually extracts multi-scale features through four downsampling blocks, each of which contains a convolutional layer, instance normalization, and a LeakyReLU activation function; the decoder restores spatial resolution through four upsampling blocks, each of which contains a convolutional layer, instance normalization, and a LeakyReLU activation function.

6. The computer image generation method based on multimodal image conversion according to claim 2, characterized in that: The discriminator D A The input is a thermal infrared image The PatchGAN architecture is adopted, and a multi-layer convolutional network is used to gradually extract features and reduce spatial resolution. The last layer uses 1×1 convolution to output the discrimination result map. The multi-layer convolutional network uses 4 convolution blocks, each of which contains a convolution layer, a LeakyReLU activation function and spectral normalization. A self-attention module is inserted after the third convolution block.

7. The computer image generation method based on multimodal image conversion according to claim 1, characterized in that: In step 1, the loss function of the improved CycleGAN is specifically: Fighting Loss: Cycle consistency loss: Temperature constraint loss: Where T() is the function of estimating temperature from the visible light image; Loss of color fidelity: Where ΔE is the color difference, B * is the reference visible light image; Total loss function: L total =L adv (G A→B ,D B )+L adv (G B→A ,D A )+λ1·L cycle +λ2·L temp +λ3·L color where λ1 is L cycle The dynamic weight parameter, the initial value is 10, λ2 is L temp The dynamic weight parameter, the initial value is 5, λ3 is L color The dynamic weight parameter of , the initial value is 3.

8. The computer image generation method based on multimodal image conversion according to claim 1, characterized in that: In step 4, the dynamic weight adjustment strategy is specifically as follows: λ1(t)=λ 1,init ·exp(-α1·t) λ2(t)=λ 2,init ·(1-exp(-α2·t)) λ3(t)=λ 3,init ·min(1,t / τ) Where t is the number of training iterations, and α1, α2, and τ are hyperparameters.

9. The computer image generation method based on multimodal image conversion according to claim 1, characterized in that: In the step 6, the quality indicators of the generated image include the color fidelity indicator CIEDE2000 color difference and the structural similarity SSIM.

10. The computer image generation method based on multimodal image conversion according to claim 1, characterized in that: In step six, the post-processing optimization includes guided filtering and color correction. The guided filtering utilizes the edge information of the thermal infrared image to optimize the color boundary and enhance the details. The color correction adjusts the hue and saturation of the generated image based on the reference color distribution.

Images