Data conversion methods and apparatus, model training methods, equipment and media

Abstract

This application provides a data conversion method and apparatus, a model training method, device, and medium. The method includes: acquiring an infrared image to be converted; using the infrared image to be converted as input to a trained image conversion model, and converting it through a multi-task generator to output a corresponding visible light image; the image conversion model is an adversarial generative network model, including a multi-task generator and a discriminator; the training tasks in one iteration of the multi-task generator include: a conversion task that uses an infrared training sample image as input to convert it into a corresponding visible light converted image; a super-resolution task that uses a blurred visible light training sample image as input to perform super-resolution processing to reconstruct a visible light image; and a super-resolution guidance constraint task that uses the visible light converted image as input; the discriminator alternately uses the visible light training sample image and the output of the multi-task generator as input, and satisfies the adversarial generative loss function as constraint.

Description

Technical Field

[0001] This application relates to the field of image technology, and in particular to a method and apparatus for infrared-to-visible light data conversion based on visual details, a training method for an image conversion model, a computer device, and a computer-readable storage medium. Background Technology

[0002] Image processing technology is widely applied in various fields, including scientific research, industry, and daily life. For example, surveillance and security, as an important measure in modern society, play a crucial role in maintaining public safety and protecting property. By monitoring and recording activities, it helps prevent crime and provides strong evidence for investigation and prosecution. In the field of surveillance and security, both infrared and visible light surveillance play important roles, directly related to people's daily lives and serving as vital safeguards for life and property. Currently, data generated by infrared and visible light surveillance are the most common types of surveillance data. Visible light data can image normally in clear daylight, but it is difficult to guarantee security at night or in inclement weather. While infrared data can image under various conditions, it contains less semantic information, making it less effective for timely detection of security risks.

[0003] Currently, to improve the observability of infrared images, infrared and visible light data fusion is commonly used for perception enhancement, compensating for the lack of perception capabilities at night or in inclement weather. For example, Chinese patent applications CN202310267771.6 ("Infrared-Visible Light Fusion Method, System, and Readable Storage Medium Based on Feature Enhancement"), CN201821327749.7 ("An Infrared-Visible Light Fusion Panoramic Imaging Device"), and CN202211030333.X ("An Infrared-Visible Light Fusion Target Detection Method") all employ infrared and visible light data fusion. However, these algorithms still have significant drawbacks—poor visual detail and amplified errors resulting from the fusion process. While infrared and visible light data fusion algorithms combine infrared and visible light data from the shooting scene, adding color to the gray-white infrared structural information to achieve a certain degree of perception enhancement, they still cannot meet the actual needs of surveillance and security. The data distributions of infrared and visible light differ significantly, making forced fusion insufficient for true visual enhancement. For example, infrared data lacks semantic information, and its grayscale tone prevents the fused result from possessing the rich detail of true visible light data. Furthermore, visible light data itself contains noise at night or in complex weather conditions; while correctly captured portions can enhance infrared imagery, erroneous portions can have a negative impact. Therefore, in practical security monitoring, more effective methods are needed to deliver richer visual effects, facilitating the timely detection of security vulnerabilities in monitored scenarios.

[0004] Another method for perceptual enhancement of infrared images is to directly convert infrared data into visible light data. This method uses stable infrared data as input and clear target visible light as the result, which can simultaneously solve the problems of insufficient semantic information in the imaging results of fusion methods and the noise of visible light data under complex weather conditions. For example, Chinese patent application number "CN202110814218.0" entitled "Method for converting infrared video to visible light video in autonomous driving". However, the conversion approach adopted by these methods still requires the use of visible light data paired with infrared data as an approximation target for infrared conversion during model training. This easily leads to the loss of the original detail and structural information advantages of infrared data, and the visible light visual information is prone to insufficient detail, resulting in the conversion result being constrained by the defects of the two data types. Summary of the Invention

[0005] To address the existing technical problems, this application provides a method and apparatus for infrared-to-visible light data conversion based on visual details, which can effectively preserve the structural information of infrared images and enhance the visual detail information of visible light, as well as a training method for image conversion models, computer equipment, and computer-readable storage medium.

[0006] A first aspect of this application provides a method for converting infrared to visible light data based on visual details, including:

[0007] Acquire the infrared image to be converted;

[0008] The infrared image to be converted is used as input to the trained image conversion model, and the corresponding visible light image is converted and output by the multi-task generator of the image conversion model.

[0009] The image conversion model is an adversarial generative network model, comprising a multi-task generator and a discriminator. The training set of the image conversion model includes infrared training sample images and visible light training sample images. During the training process, the training tasks in one iteration of the multi-task generator include: a conversion task that uses the infrared training sample images as input to convert them into corresponding visible light converted images; a super-resolution task that uses the blurred visible light training sample images as input to perform super-resolution processing to reconstruct the visible light images; and a super-resolution guidance constraint task that uses the visible light converted images obtained from the conversion tasks as input to perform super-resolution processing to reconstruct the visible light converted images. The discriminator alternately uses the visible light training sample images and the output of one iteration of the multi-task generator as input, and satisfies the adversarial generative loss function as a constraint.

[0010] Secondly, a training method for an image conversion model is also provided, including:

[0011] Obtain a training set; the training set includes an infrared image set consisting of infrared training sample images and a visible light image set consisting of visible light training sample images;

[0012] Construct an initial adversarial generative network model; wherein the initial adversarial generative network model includes a multi-task generator and a discriminator;

[0013] The initial adversarial generative network model is iteratively trained based on the training set. In each iteration, the multi-task generator performs three training tasks: a conversion task, a super-resolution task, and a super-resolution guidance and constraint task. The conversion task selects an infrared training sample image as input to obtain a corresponding visible light converted image. The super-resolution task selects a blurred visible light training sample image as input and performs super-resolution processing to obtain a reconstructed visible light image. The super-resolution guidance and constraint task takes the visible light converted image obtained from the conversion task as input and performs super-resolution processing to obtain a reconstructed visible light converted image.

[0014] The discriminator alternately takes the visible light training sample image and the output of one iteration of the multi-task generator as input, and is constrained by the adversarial generation loss function;

[0015] The model is trained iteratively until the preset conditions are met, and the model parameters are saved to obtain the image conversion model.

[0016] Thirdly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the infrared-to-visible light data conversion method based on visual details as described in any embodiment of this application, or implements the image conversion model training method as described in any embodiment of this application.

[0017] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the infrared-to-visible light data conversion method based on visual details as described in any embodiment of this application, or implements the image conversion model training method as described in any embodiment of this application.

[0018] In the above embodiments, the image conversion model adopts an adversarial generative network model as its basic architecture. The training set of the image conversion model includes infrared training sample images and visible light training sample images. During the training process, the multi-task generator includes, in one iteration, performing a conversion task with the infrared training sample images as input, performing a super-resolution task with the blurred visible light training sample images as input, and performing a super-resolution guidance constraint task with the visible light converted image output from the conversion task as input. In this way, the discriminator can extract visual information of real visible light data through the visible light training sample images for learning and discrimination. The learning and discrimination capabilities are then combined. This is equivalent to identifying the output of the multi-task generator in one iteration, such as identifying the visible light converted image, reconstructed visible light image, and reconstructed visible light converted image output by the three training tasks in one iteration. The training is carried out with the adversarial generation loss function as a constraint, and the model parameters of the multi-task generator are updated in reverse recursion by minimizing the loss value. After training for multiple iterations, the resulting image conversion model multi-task generator can retain the original structural information of the infrared image when converting the infrared image to the visible light image, effectively enhance the visual details of the visible light, and achieve a visual enhancement effect that approximates the real visible light image.

[0019] Furthermore, during the training process of the discriminator, the super-resolution task of the multi-task generator can enhance the visible light visual details based on the infrared-to-visible light data conversion performed by the multi-task generator. The introduction of visible light training sample images enables the discrimination ability to obtain real visible light data visual information. Therefore, it is no longer necessary to rely on paired infrared and visible light images to complete the training, reducing the difficulty of model training. It also avoids the problem that the conversion effect of the trained image conversion model from infrared to visible light is limited by the quality of the paired visible light images during model training. This achieves the optimization of the authenticity of the generated results while preserving the structural information in the infrared images.

[0020] In the above embodiments, the infrared-to-visible light data conversion device based on visual details, the training method for the image conversion model, the computer equipment, and the computer-readable storage medium are based on the same concept as the corresponding infrared-to-visible light data conversion method embodiments based on visual details, and thus have the same technical effects as the corresponding infrared-to-visible light data conversion method embodiments based on visual details, which will not be repeated here. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating an application scenario of an infrared-to-visible light data conversion method based on visual details in one embodiment;

[0022] Figure 2 This is a flowchart of an infrared-to-visible light data conversion method based on visual details in one embodiment;

[0023] Figure 3 This is a schematic diagram of the model network architecture of an image conversion model in one embodiment;

[0024] Figure 4 This is a schematic diagram illustrating the principle of the model network architecture of an image conversion model in one embodiment;

[0025] Figure 5 This is a flowchart illustrating the training process of an image conversion model in one embodiment;

[0026] Figure 6 A flowchart of an infrared-to-visible light data conversion method based on visual details, as shown in a specific example;

[0027] Figure 7 and Figure 8 This is a schematic diagram illustrating how, in one embodiment, the device collects raw infrared images under different weather conditions and scenarios as infrared images to be converted, and outputs the corresponding visible light images after conversion by the multi-task generator of the image conversion model.

[0028] Figure 9 This is a schematic diagram of an infrared-to-visible light data conversion device based on visual details in one embodiment.

[0029] Figure 10 This is a schematic diagram of the structure of a computer device in one embodiment.

[0030] Embodiments of the present invention

[0031] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] In the following description, the phrase "some embodiments" refers to a subset of all possible embodiments. It should be noted that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0034] In the following description, the terms "first, second, and third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0035] Please see Figure 1 This diagram illustrates an optional application scenario of the infrared-to-visible light data conversion method based on visual details provided in this application embodiment. The device that uses this method to convert acquired infrared images into visible light images can be a monitoring device 10 in the field of surveillance and security. The image conversion model can be trained on a more powerful intelligent device 20 and then used on the monitoring device 10. The monitoring device 10 uses the real-time acquired infrared images as the images to be converted, processes them through the image conversion model, and outputs a visible light image with enhanced visible light visual information. This improves the visibility of the infrared images acquired by the monitoring device 10 in night vision or special weather monitoring scenarios, and consequently, the visibility when displayed on the monitoring screen. The infrared-to-visible light data conversion method based on visual details provided in this application embodiment is also considered a cross-modal monitoring data conversion method based on global perception matching that can convert grayscale infrared data into color visible light data. The intelligent device 20 can refer to a personal computer, server, or other device with stronger data storage and computing capabilities.

[0036] However, it is understandable that the monitoring device 10 itself can also be an intelligent device with data storage and computing capabilities, and the training of the image conversion model can also be completed directly on the monitoring device 10 side.

[0037] Please see Figure 2 The infrared-to-visible light data conversion method based on visual details provided in this application includes the following steps:

[0038] S101, acquire the infrared image to be converted.

[0039] S103, the infrared image to be converted is used as the input of the trained image conversion model, and the corresponding visible light image is converted and output by the multi-task generator of the image conversion model.

[0040] The image conversion model is an adversarial generative network model, comprising a multi-task generator and a discriminator. The training set of the image conversion model includes infrared training sample images and visible light training sample images. During the training process, the training tasks in one iteration of the multi-task generator include: a conversion task that uses the infrared training sample images as input to convert them into corresponding visible light converted images; a super-resolution task that uses the blurred visible light training sample images as input to perform super-resolution processing to reconstruct the visible light images; and a super-resolution guidance constraint task that uses the visible light converted images obtained from the conversion tasks as input to perform super-resolution processing to reconstruct the visible light converted images. The discriminator alternately uses the visible light training sample images and the output of one iteration of the multi-task generator as input, and satisfies the adversarial generative loss function as a constraint.

[0041] The infrared image to be converted refers to an infrared image that is input into an image conversion model to perform infrared-to-visible light data conversion, thereby obtaining a visible light image with enhanced visual detail information. In this embodiment, taking the application of the infrared-to-visible light data conversion method based on visual detail to security monitoring equipment as an example, the infrared image to be converted refers to the original infrared image acquired in real time by the security monitoring equipment.

[0042] Please see Figure 3 and Figure 4The image conversion model employs a Generative Adversarial Network (GAN) framework. A multi-task generator G takes infrared images as input and performs infrared-to-visible light data conversion. During the training of the GAN model, the multi-task generator G jointly completes three training tasks in each iteration. These three tasks correspond to consistency constraint strategies, and each uses a specific loss function to constrain the execution of its respective training task. Training Task 1 involves converting the infrared training sample images into visible light data to obtain the corresponding visible light converted image. Training Task 2 involves using a blurred visible light training sample image as input and performing super-resolution processing to reconstruct the visible light image. Training Task 3 involves using the visible light converted image obtained from the conversion tasks as input and performing super-resolution processing to reconstruct the visible light converted image, a super-resolution-guided constraint task.

[0043] The discriminator D alternately takes real visible light training sample images as input and the output of the multi-task generator G in one iteration as input. The discriminator D's ability to extract and distinguish visible light visual information based on real visible light data is applied to the output of the multi-task generator G in one iteration for discrimination. For example, it distinguishes the visible light converted image, reconstructed visible light image, and reconstructed visible light converted image output by the three training tasks in one iteration, respectively. The model parameters are updated by back-recursion and optimization under the constraint of the adversarial generation loss function. Ultimately, the visible light image obtained by the multi-task generator G after converting the infrared image can retain the original structural information of the infrared image, effectively enhance the visible light visual details, and achieve a visual enhancement effect that approximates the real visible light image.

[0044] It should be noted that the infrared training sample images and visible light training sample images used as the training set are used by the discriminator, which alternately takes real visible light data from the visible light training sample images and the output of one iteration of the multi-task generator as input. The super-resolution task of the multi-task generator, which is based on the visible light training sample images as the learning target, is associated with the visible light converted image obtained from the initial infrared to visible light data conversion through super-resolution guidance and constraint tasks. This can enhance the multi-task generator's ability to learn visible light visual detail enhancement and improve the visible light visual detail of the final generated visible light image. Therefore, the training data of the image conversion model does not rely on paired data as supervision signals, which can avoid annotation costs, reduce training difficulty, and improve practicality. Furthermore, the visible light training sample images in the training set serve as approximation targets for the visible light visual enhancement effect after image conversion by the image conversion model. The infrared training sample images can be infrared images collected under different environments and weather conditions, while the visible light training sample images can be high-quality visible light images collected only under clear weather conditions. This ensures that the conversion effect of the image conversion model on visible light images can always approach the data of visible light under clear weather conditions, preserving the original structural information of the infrared image while retaining the visible light visual detail enhancement effect.

[0045] Additionally, it should be noted that the three training tasks in the multi-task generator are jointly trained. The discriminator's ability to distinguish between genuine and fake outputs from the multi-task generator can also focus solely on the reconstructed visible light conversion image obtained after the multi-task generator undergoes a super-resolution guided constraint task. In some optional embodiments, during the discriminator's training process, which alternates between using visible light training sample images and the output of one iteration of the multi-task generator as input, it can also retain only the reconstructed visible light conversion image obtained after the multi-task generator undergoes a super-resolution guided constraint task as input for calculating the adversarial generative loss.

[0046] After training is complete, the image conversion model is ported to a device that needs to convert infrared to visible light data. At this time, the multi-task generator in the image conversion model is used to convert the infrared image to be converted into the corresponding visible light image.

[0047] In the above embodiments, the image conversion model adopts an adversarial generative network model as its basic architecture. The training set of the image conversion model includes infrared training sample images and visible light training sample images. During the training process, the multi-task generator includes, in one iteration, performing a conversion task with the infrared training sample images as input, performing a super-resolution task with the blurred visible light training sample images as input, and performing a super-resolution guidance constraint task with the visible light converted image output from the conversion task as input. In this way, the discriminator can extract visual information of real visible light data through the visible light training sample images for learning and discrimination. The learning and discrimination capabilities are then combined. This is equivalent to identifying the output of the multi-task generator in one iteration, such as identifying the visible light converted image, reconstructed visible light image, and reconstructed visible light converted image output by the three training tasks in one iteration. The training is carried out with the adversarial generation loss function as a constraint, and the model parameters of the multi-task generator are updated in reverse recursion by minimizing the loss value. After training for multiple iterations, the resulting image conversion model multi-task generator can retain the original structural information of the infrared image when converting the infrared image to the visible light image, effectively enhance the visual details of the visible light, and achieve a visual enhancement effect that approximates the real visible light image.

[0048] Furthermore, during the training process of the discriminator, the super-resolution task of the multi-task generator can enhance the visible light visual details based on the infrared-to-visible light data conversion performed by the multi-task generator. The introduction of visible light training sample images enables the discrimination ability to obtain real visible light data visual information. Therefore, it is no longer necessary to rely on paired infrared and visible light images to complete the training, reducing the difficulty of model training. It also avoids the problem that the conversion effect of the trained image conversion model from infrared to visible light is limited by the quality of the paired visible light images during model training. This achieves the optimization of the authenticity of the generated results while preserving the structural information in the infrared images.

[0049] In some embodiments, the adversarial generation loss function is shown in Formula 1 below:

[0050]

[0051] in, The adversarial generation loss function is defined as follows: D represents the discriminator, G represents the multi-task generator, and D(y) = 0. j ) represents the discrimination result of the j-th visible light training sample image in the visible light image set, D(G(x) i )) represent the outputs of the three training tasks in one iteration, namely the identification results of the visible light converted image corresponding to the i-th infrared training sample image, the reconstructed visible light image corresponding to the j-th visible light training sample image, and the reconstructed visible light converted image corresponding to the visible light converted image of the i-th infrared training sample image.

[0052] in, The maximum likelihood estimation algorithm is represented by a generative adversarial network framework composed of a multi-task generator G and a discriminator D. Constrained by an adversarial generation loss function, the discriminator D learns to visually discriminate visible light data. Then, the discriminator D's visual discrimination ability is used to evaluate the effects of the visible light converted images, reconstructed visible light images, and reconstructed visible light converted images output by the multi-task generator G from multiple training tasks. Through generative adversarial operations, the generation effect of the multi-task generator G approximates the visual effect of real visible light in the training set, preserving the original structural information of the infrared data. Furthermore, through super-resolution reconstruction, the generated results can present the visual details of visible light.

[0053] In some embodiments, the infrared-to-visible light data conversion method based on visual details further includes obtaining the image conversion model through model training. The model training process includes:

[0054] Obtain a training set containing both infrared and visible light image sets;

[0055] An initial adversarial generative network (PGN) model is constructed, and iterative training is performed on the initial GPN model based on the training set. The initial GPN model includes a multi-task generator and a discriminator. In one iteration, the multi-task generator performs three training tasks: a conversion task, a super-resolution task, and a super-resolution guidance and constraint task. The conversion task involves selecting an infrared training sample image as input to obtain a corresponding visible light converted image. The super-resolution task involves selecting a blurred visible light training sample image as input and performing super-resolution processing to obtain a reconstructed visible light image. The super-resolution guidance and constraint task involves using the visible light converted image obtained from the conversion task as input and performing super-resolution processing to obtain a reconstructed visible light converted image. The discriminator alternately uses the visible light training sample image and the output of one iteration of the multi-task generator as input, and satisfies the adversarial generative loss function as a constraint.

[0056] The model is trained iteratively until the preset conditions are met, and the model parameters are saved to obtain the image conversion model.

[0057] The acquisition of the training set, which includes both infrared and visible light image sets, can be represented by formulas 2 and 3 below:

[0058]

[0059]

[0060] Among them, D s It refers to the infrared image set, D t It refers to the set of visible light images, xi This refers to infrared training sample images, y j This refers to visible light training sample images, n s It represents the total number of images in the infrared image set, where i represents the i-th image, and n... t It represents the total number of images in the visible light image set, where j represents the j-th image.

[0061] Define the set of infrared images used as input as D. s The target output is a set of clear visible light images under clear weather conditions, designated as D. t It should be noted that, to ensure conversion effectiveness under different environments, the infrared image set can collect infrared data under different weather and environmental conditions, such as nighttime infrared images, snowy infrared images, and foggy infrared images, as input. The ultimate goal of infrared data under different weather and environmental conditions is to convert it into clear, visually detailed visible light data. In this embodiment, infrared images under different weather and environmental conditions within the infrared image set can increase the richness of the input during model training, which is beneficial to improving the robustness of the trained image conversion model. For the visible light image set, only sufficiently high-quality visible light data collected under clear weather conditions needs to be considered as the training target; infrared and visible light data do not need to be paired, i.e., n s and n t The number of images can be unequal, and i and j can be any image selected from the infrared image set and the visible light image set, respectively. This eliminates the need for manual annotation and image registration, thus improving the practical value of the model.

[0062] The initial adversarial generative network model consists of a multi-task generator G and a discriminator D for discriminating visible light visual information. In the trained image transformation model, the multi-task generator G performs infrared-to-visible light data conversion on the input infrared image and outputs a visible light image. During model training, in each iteration, the multi-task generator G jointly executes the transformation task, the super-resolution task, and the super-resolution guidance constraint task. This allows the multi-task generator G to generate adversarial strategies and loss functions to convert the input infrared data into visible light results that match the style of the visible light training sample images in the training set.

[0063] In a specific example, both the converter and the discriminator use residual convolutional neural networks, such as ResNet, as the backbone network. The excellent properties of its residual connections and convolutions effectively extract key information from infrared and visible light data. The multi-task generator can be divided into an encoder and a decoder. The discriminator for visible light visual information discrimination consists of an encoder. In the conversion task, the encoder of the multi-task generator primarily extracts effective information (such as structural information) from the infrared data, while the decoder adds visible light style information (such as color) to the structural information to generate a corresponding visible light conversion result that conforms to the visible light visual effect. The discriminator's encoder is responsible for extracting the visual information of the real visible light data and the generated visible light conversion result. It learns and distinguishes between the two through generative adversarial strategies, guiding the conversion task and optimizing the generated result. In the super-resolution task, the encoder of the multi-task generator extracts feature information (such as morphology, structure, and texture) from the input blurred visible light data, while the decoder completes the visual detail completion process, i.e., performs super-resolution processing of the visible light data and outputs the reconstructed visible light result. Here, the input blurred visible light data is the visible light data obtained by randomly blurring and downsampling the visible light training sample image, followed by upsampling. The encoder of the discriminator is responsible for extracting the visual detail information of the real visible light data and the generated reconstructed visible light result. It learns and distinguishes through generative adversarial strategies, guiding the super-resolution task and optimizing the generation result. In the super-resolution guidance and constraint task, the multi-task generator takes the visible light converted image obtained from the conversion task as input. The encoder performs super-resolution processing based on the input visible light data, extracting feature information (such as morphology, structure, and texture) from the input visible light data, while the decoder also completes the visual detail completion process, i.e., performs super-resolution processing of the visible light data and outputs the reconstructed visible light result. Here, the input visible light data is the visible light data obtained by converting the infrared training sample image to visible light data.

[0064] It's important to note that the transformation task and the super-resolution task are style transfer processing performed by the multi-task generator using infrared data as input and image reconstruction processing performed using visible light data as input, respectively. In a single iteration, the transformation and super-resolution tasks are unrelated. However, the setting of the super-resolution guidance constraint task combines the transformation and super-resolution tasks of the multi-task generator. Specifically, due to the inherent imaging properties of infrared data, it is coarser and has less detailed texture compared to visible light images. Therefore, even with the constraint of preserving the original structural information of the infrared data, the resulting visible light image is unlikely to have good detail and texture, potentially leading to poor visual detail in the transformed visible light image. The setting of the super-resolution guidance constraint task guides the generation result of the infrared-to-visible light data conversion. Again, the super-resolution task is used. Therefore, by using the super-resolution-guided constraint task, the trained super-resolution task is integrated into the conversion task. After training, the multi-task generator G and the decoder can directly and consistently fit with the decoder of the super-resolution task in the conversion task, ultimately achieving one-step infrared-to-visible light data conversion and super-resolution reconstruction to enhance visible light visual details.

[0065] Iterative training reaches the preset conditions, which can be reaching the preset maximum number of iterations or the convergence of the model's loss function.

[0066] Thus, through the trained multi-task generator, not only is the basic conversion from infrared data to visible light data achieved, but the generation effect of the multi-task generator G also presents a visual effect close to that of real visible light as the training set. It can retain the original structural information of the infrared data, and through super-resolution reconstruction, the generation result can present the visual details of visible light.

[0067] In the above embodiments, the image conversion model employs a multi-task generator and a discriminator. On one hand, the discriminator and the multi-task generator form a generative adversarial framework, imposing basic constraints on the output of the multi-training tasks performed by the multi-task generator, so that the generation results of the multi-training tasks meet the requirements of visible light data. On the other hand, the training settings of the multi-task generator include a conversion task with infrared data as input and a super-resolution task with visible light data as input. The super-resolution task is integrated into the conversion task through super-resolution guidance and constraint tasks. By combining the adversarial generative network architecture of the multi-task generator and discriminator with multiple optimization strategies, infrared data can be quickly converted into visible light data that is more in line with visual intuition. Furthermore, by combining the structural information of infrared data with the enhancement of visible light visual details, it can better solve the problem of visual effect enhancement and detail preservation in the field of surveillance and security through infrared-to-visible light conversion.

[0068] It should be noted that in the super-resolution task, blurring the visible light training sample images can be accomplished using a traditional algorithm model established with a pre-defined blurring formula, i.e., a blurring model (such as...). Figure 3 As shown, the blurred visible light data is used to simulate the image quality of the converted visible light result in the conversion task, so as to provide a multi-task generator that can use the blurred visible light data to learn the super-resolution processing of the visible light data through the super-resolution task and output the detailed and clearer reconstructed visible light result.

[0069] In some embodiments, the conversion task is constrained by the infrared training sample image and the corresponding visible light converted image satisfying the structural consistency contrast loss function;

[0070] The structural consistency comparison loss function is shown in Formula 4 below:

[0071]

[0072] in, V represents the structural consistency contrastive loss function. q The extracted features f of the infrared training sample images i Selected anchor point features These represent the extracted features of the corresponding visible light converted images. Selected positive and negative sample features, v q and positive samples For the same location on the feature map, negative samples There is no positional correspondence between the anchor point feature and the anchor point feature.

[0073] The training of the multi-task generator for the conversion task incorporates a structural consistency contrastive loss function as a constraint to further ensure that the visible light conversion result retains the key structural information of the original infrared image. For a given infrared data x... i ∈D s A multi-task generator that converts infrared to visible light can obtain the result related to x. i Corresponding visible light data In the transformation task, the encoder of the multi-task generator extracts image feature information from the infrared training sample images, denoted as f. i , representing the image features of the i-th infrared training sample image. In the super-resolution guided constraint task, the generated visible light results will be... Similarly, the encoder of the multi-task generator extracts image feature information, denoted as... The image features represent the visible light result generated from the i-th infrared training sample image. Design the positive and negative sample pairs required for the contrastive loss, and apply the features f... i Let this be the query feature, from which anchor feature v is selected. q On the other hand, the characteristics of the generated visible light results These are denoted as key features, from which positive and negative sample features are selected. and Specifically, from f i 64 query feature points were randomly selected from the data. q Therefore, in Similarly, there will be 64 positive key sample feature points with the same corresponding position. The remaining non-corresponding positions are negative sample feature points. Calculate the structural consistency loss between positive and negative sample features as follows:

[0074] In the above embodiments, a multi-task generator is used to extract features from the input infrared training sample image and the corresponding generated visible light converted image. By constraining the features at the corresponding positions to be as similar as possible, the generated visible light converted image retains the original structural information of the infrared image. The structural consistency contrast loss function brings the positive sample distance of the anchor feature query and key closer and pushes away the distance from the negative sample, thereby achieving the constraint of content consistency.

[0075] In some embodiments, the super-resolution task is constrained by the reconstructed visible light image and the corresponding visible light training sample image satisfying the super-resolution training supervision loss function;

[0076] The super-resolution training supervision loss function is shown in Formulas 5 to 7 below:

[0077]

[0078]

[0079]

[0080] in, Let y represent the super-resolution training supervision loss function. j This represents the j-th visible light training sample image. This represents the blurred result of the j-th visible light training sample image. Downsample indicates random downsampling, and Upsample indicates upsampling. Let represent the reconstructed visible light image corresponding to the j-th visible light training sample image.

[0081] The multi-task generator achieves super-resolution reconstruction of details from blurred visible light data through a super-resolution training supervision strategy. In the super-resolution task, the multi-task generator combines the input visible light data y j ∈D t Supervised training is performed on the multi-task generator G. First, the visible light data is blurred using random blurring (such as direct downsampling or Gaussian blurring), and then downsampled to obtain a low-resolution and blurred visible light result, denoted as G. The blurred data was then resampled to the original image size and recorded as the blurred result. The blurred result is fed into a multi-task generator for detailed reconstruction. The constrained reconstruction result and the original image input visible light data y j Maintain consistency.

[0082] Specifically, Formula 7 is the input of real, clear visible light data y. j Convert to fuzzy input The downsampling equation randomly selects a method from bilinear interpolation or similar techniques to reduce resolution, resulting in a low-resolution, blurry image. The upsampling equation, on the other hand, uses bilinear interpolation to obtain the visible light data y from the original image. j A uniformly sized blurred visible light image. A multi-task generator takes the blurred visible light image as input and obtains a reconstructed visible light image y. r j ec The super-resolution training supervised loss function constrains the consistency between the reconstructed visible light image and the original visible light image, thereby training the multi-task generator to learn to reconstruct visual details from the input visible light data and improving the sensitivity of the multi-task generator's decoder to visible light visual details.

[0083] In some embodiments, the super-resolution guidance constraint task is constrained by the reconstructed visible light converted image and the corresponding visible light converted image satisfying the super-resolution guidance constraint loss function.

[0084] The super-resolution guidance constraint loss function is shown in Formula 8 below:

[0085]

[0086] in, Let G(x) represent the super-resolution guidance constraint loss function. i () represents the visible light converted image corresponding to the i-th infrared training sample image. G(x) represents i The reconstructed visible light converted image obtained through super-resolution processing, This is the gradient truncation symbol.

[0087] The super-resolution guidance constraint strategy combines the conversion task of the multi-task generator on infrared data input with the super-resolution task on visible light data input. In Equation 8, It serves as guidance information for exceeding the score, The generated results are then fed back into the multi-task generator G. Since the input is visible light data, G performs a super-resolution task to obtain more refined visual results. Gradient clipping (grad_clip) addresses the vanishing and exploding gradient problems that occur during backpropagation in neural network model training. These problems prevent model parameters from being updated for extended periods, hindering effective training. Gradient clipping solves these issues by setting a threshold; if the pre-updated gradient is less than the threshold, it is set to the threshold value. The purpose of gradient clipping is to control the super-resolution reconstruction process in the super-resolution-guided constraint task to obtain more refined visual results, rather than proceeding in reverse.

[0088] In the above embodiments, the super-resolution guidance constraint task, leveraging the processing capabilities of the super-resolution reconstructed image already trained by the super-resolution task, performs detail correction and completion on the infrared-to-visible light conversion results. Combined with... Figure 4 As shown in Equation 8, truncating the gradient makes the result of the infrared image after the super-resolution task become a pseudo-label, which guides and corrects the visible light generation result in the conversion task, allowing the model to obtain a visible light image with enhanced visual details in one go for the input infrared image.

[0089] Thus, the overall loss of the image conversion model is shown in Equation 9 below:

[0090]

[0091] in, This represents the adversarial generative loss function. This represents the structural consistency comparison loss function. The super-resolution training supervision loss function is... For the super-resolution guidance constraint loss function, λ1,

[0092] λ2 and λ3 are the balancing hyperparameters. During model training, gradient descent is used for backpropagation in each iteration to update the parameters of the entire model and minimize the loss function, so that the multi-task generator can eventually correctly convert infrared data into refined visible light data.

[0093] Please see Figure 5 In another aspect, this application provides a method for training an image conversion model, comprising:

[0094] S201, Obtain the training set; the training set includes an infrared image set composed of infrared training sample images and a visible light image set composed of visible light training sample images.

[0095] Infrared training sample images and visible light training sample images do not need to be paired. To ensure the conversion effect of different types of infrared images, infrared training sample images can be collected from various infrared images acquired under different weather conditions and environments, while visible light training sample images are acquired from sufficiently clear and high-quality visible light images under clear weather conditions. The clarity and high quality of the visible light training sample images are the desired effect after infrared image conversion.

[0096] S203, Construct an initial adversarial generative network model; wherein, the initial adversarial generative network model includes a multi-task generator and a discriminator.

[0097] The initial adversarial generative network model's main architecture can be constructed using a ResNet residual convolutional neural network, comprising a multi-task generator and a discriminator. Through generative adversarial interaction, the multi-task generator can directly transform input infrared data into visible light results with visible light style and enhanced visible light details. The multi-task generator G consists of an encoder and a decoder. The encoder extracts feature information from the input data in the corresponding training task, while the decoder completes the information according to the extracted feature information and the corresponding training task to obtain the corresponding generated result. The discriminator consists of an encoder responsible for extracting visible light visual information from real visible light data and the generated results output by the multi-task generator, for learning and discrimination.

[0098] S205, the initial adversarial generative network model is iteratively trained based on the training set. In one iteration, the multi-task generator performs three training tasks: a conversion task, a super-resolution task, and a super-resolution guidance and constraint task. The conversion task selects an infrared training sample image as input to obtain a corresponding visible light converted image. The super-resolution task selects a blurred visible light training sample image as input and performs super-resolution processing to obtain a reconstructed visible light image. The super-resolution guidance and constraint task takes the visible light converted image obtained from the conversion task as input and performs super-resolution processing to obtain a reconstructed visible light converted image. The discriminator alternately takes the visible light training sample image and the output of one iteration of the multi-task generator as input and satisfies the adversarial generative loss function as constraint.

[0099] The initial adversarial generative network model is iteratively trained based on the training set. In one iteration, the training tasks of the multi-task generator include: 1. First, selecting infrared training sample images from the infrared image set as input to obtain corresponding visible light converted images, and calculating a first loss value with the constraint that the infrared training sample images and the corresponding visible light converted images satisfy the structural consistency contrast loss function; 2. Then, selecting visible light training sample images from the visible light image set, using the blurred visible light training sample images as input, performing super-resolution processing to obtain reconstructed visible light images, and calculating a second loss value with the constraint that the reconstructed visible light images and the corresponding original visible light training sample images satisfy the super-resolution training supervision loss function; 3. Then, using the visible light converted images obtained from the infrared training sample images as input, performing super-resolution processing to obtain reconstructed visible light converted images, and calculating a third loss value with the constraint that the reconstructed visible light converted images and the corresponding visible light converted images satisfy the super-resolution supervision constraint loss function. The discriminator alternately uses the visible light training sample images from the visible light image set and the output of one iteration of the multi-task generator as input, and calculates a fourth loss value with the constraint that the adversarial generative loss function is satisfied. It should be noted that, for the three training tasks in one iteration of the multi-task generator, the discriminator uses the output of the multi-task generator as input to calculate three corresponding fourth loss values. That is, during training, for each of the three training tasks in one iteration of the multi-task generator, the initial adversarial generative network model performs backpropagation based on the total loss of the first, second, third, and fourth loss values ​​to update the model parameters. Alternatively, if the three training tasks of the multi-task generator are joint training tasks, the discriminator can use the output of the multi-task generator as input, or it can only use the reconstructed visible light conversion image output after the joint execution of the three training tasks as input to calculate the corresponding fourth loss value. The initial adversarial generative network model then performs backpropagation based on the total loss of the first, second, third, and fourth loss values ​​to update the model parameters.

[0100] Regarding the structural consistency comparison strategy: In the conversion task, the encoder extracts structural information from the infrared data, while the decoder needs to imbue the structural information with visible light visual information to achieve the basic conversion from infrared to visible light data. The multi-task generator extracts features from both the infrared training sample images and the corresponding visible light converted images to obtain f. i and By constraining feature points at corresponding positions in the extracted features to be as close as possible, the consistency of content (the original structural information of the infrared training sample images) is ensured. Contrastive loss brings positive samples of the query and key closer together and widens the distance to negative samples, thus achieving the constraint of content consistency (as shown in the appendix). Figure 4 (as shown in Formula 4).

[0101] Regarding the super-resolution training supervision strategy, a multi-task generator first learns super-resolution processing capabilities using blurred visible light training sample images. The super-resolution processing capabilities then convert the blurred visible light data into clear visible light data (as shown in the attached image). Figure 4 As shown in Formula 5, with a well-designed training team, the encoder and decoder of the multi-task generator learn the super-resolution processing capabilities to extract features from the input visible light data and complete the visible light visual details, thereby enhancing the multi-task generator's ability to ultimately convert infrared data into visible light data.

[0102] Regarding the super-resolution guidance and constraint strategy, the super-resolution knowledge already trained on the input of visible light data is utilized through multi-task generation to perform detailed correction and completion on the conversion result of infrared to visible light data (visible light converted image) (as shown in the appendix). Figure 4 (As shown in Formula 8). Truncation of the gradient allows the converted and super-resolution results of the infrared training sample images to become pseudo-labels, guiding and correcting the generation results of visible light data in the conversion task. This enables the model's multi-task generator to obtain visually enhanced visible light image outputs based on the input infrared image in one go, thus optimizing the generation effect of infrared to visible light conversion.

[0103] S207, iterate the training until the preset conditions are met, save the model parameters, and obtain the image conversion model.

[0104] The above embodiments provide a training process for the image conversion model to obtain an image conversion model based on visual details for infrared to visible light data conversion, which is no longer subject to the original defects of infrared and visible light images, and has sufficient constraints to preserve the original structural information in the original infrared image and enhance the visual detail information in the visible light image.

[0105] Please see Figure 6 To gain a more comprehensive understanding of the infrared-to-visible light data conversion method based on visual details provided in the embodiments of this application, the following explanation uses the conversion of infrared data to visible light data in the field of surveillance and security as an example:

[0106] S11, given the infrared dataset D s and visible light dataset D t Balance the hyperparameters λ1, λ2, and λ3, and determine the maximum number of iterations I.

[0107] S12 builds a multi-task generator and discriminator based on ResNet and initializes the model parameters.

[0108] S13, randomly sample an infrared data sample x i And a visible light data sample y jThe infrared data is input into the multi-task generator G to obtain the generated visible light result. Blurred visible light data samples The input multi-task generator G is used to obtain a reconstructed visible light image through super-resolution processing. The multi-task generator learns the super-resolution processing capability for visible light data input; then it processes the infrared data samples x... i The corresponding generated visible light result G(x) i Input the multi-task generator to obtain the reconstructed visible light result after super-resolution processing. That is, the multi-task generator targets infrared data sample x i Output visible light visuals with enhanced detail Visible light data sample data y j The result D(y) is obtained by feeding it into the discriminator D. j Similarly, the generated visible light results will be... The result is obtained by sending the sample to the discriminator D.

[0109] S14, Calculate the input infrared data sample x i Features f i and the generated visible light results Features Structural consistency comparison loss constraints.

[0110] S15, Super-resolution training, using visible light data samples y j Convert to blurred visible light image The image is fed into a multi-task generator G to obtain a reconstructed visible light image. Calculate the input visible light data sample y j and reconstructing visible light images Consistency constraints between them.

[0111] S16, Super-resolution guidance constraint, optimizes the infrared-to-visible light data conversion task through the trained super-resolution task, so that the two tasks are combined.

[0112] S17, calculate the total loss of the model (as shown in Equation 9), and use the gradient descent algorithm to backpropagate and update the model parameters.

[0113] S18, repeat S13 to S17 until the maximum number of iterations I is reached or the loss function converges.

[0114] S19, after training, the image conversion model is transferred to the monitoring equipment. The monitoring equipment directly sends the captured infrared data x into the trained multi-task generator G to directly obtain the corresponding visible light result G(x) that meets the requirements.

[0115] like Figure 7 and Figure 8 The diagram illustrates how, after the image conversion model of this application is ported to the device, the device uses original infrared images collected under different weather conditions and scenarios as the infrared images to be converted. These images are then converted by the multi-task generator of the image conversion model, outputting the corresponding visible light images. It can be seen that the image conversion model of this application, when converting infrared images into visible light images, possesses at least the following characteristics:

[0116] 1. Visible light visual information is highly intuitive. Compared to infrared and visible light fusion algorithms, the generated visible light results can directly provide rich visual details in the monitoring scene, which is conducive to security personnel to detect security risks in a timely manner. Moreover, in low-light scenarios such as night vision, the quality of the visible light images acquired by the infrared and visible light fusion method is poor due to the lighting conditions. In this case, even if infrared and visible light fusion is used, the fused image result cannot achieve the goal of improving the visible light visual effect. Furthermore, the infrared and visible light fusion method requires real-time synchronous acquisition of registrable infrared and visible light images. Therefore, the image acquisition equipment must include dual-light cameras for infrared and visible light acquisition, which not only complicates the algorithm but also requires additional hardware costs. In contrast, this invention utilizes a trained image conversion model, which only requires real-time acquired infrared images as input. Unlike the images obtained by infrared and visible light fusion, regardless of the imaging weather or environmental conditions, the model conversion can produce visible light images that approximate the visible light visual effect acquired in clear daytime weather. The imaging effect provides rich visual details, just like visible light images under clear weather conditions.

[0117] 2. The generated visible light results have better visual detail. By introducing a super-resolution task, the multi-task generator G is guided to learn both the transformation and super-resolution tasks simultaneously and combine them, ultimately resulting in visible light results with rich visual detail.

[0118] 3. Visual Error Correction. The multi-task generator combines transformation and super-resolution tasks, which can correct visual errors that occur during the transformation process and can be optimized through multiple iterations.

[0119] Please see Figure 9In another aspect, this application provides an infrared-to-visible light data conversion device based on visual details, comprising: an acquisition module 11 for acquiring an infrared image to be converted; and an image conversion model 12 for taking the infrared image to be converted as input and converting it through a multi-task generator to output a corresponding visible light image; wherein the image conversion model is an adversarial generative network model, including the multi-task generator and a discriminator, and the training set of the image conversion model includes infrared training sample images and visible light training sample images; during the training process of the image conversion model, the training tasks in one iteration of the multi-task generator include: a conversion task of converting the infrared training sample image as input to obtain a corresponding visible light converted image; a super-resolution task of using the blurred visible light training sample image as input and performing super-resolution processing to obtain a reconstructed visible light image; and a super-resolution guidance constraint task of using the visible light converted image obtained from the conversion task as input and performing super-resolution processing to obtain a reconstructed visible light converted image; the discriminator alternately takes the visible light training sample image and the output of the multi-task generator as input and satisfies the adversarial generative loss function as constraint.

[0120] Optionally, the adversarial generation loss function is as follows:

[0121]

[0122] in, The adversarial generation loss function is defined as follows: D represents the discriminator, G represents the multi-task generator, and D(y) = 0. j ) represents the discrimination result of the j-th visible light training sample image in the visible light image set, D(G(x) i )) represents the identification result of the visible light converted image, reconstructed visible light image, or reconstructed visible light converted image corresponding to the i-th infrared training sample image in the infrared image set.

[0123] Optionally, the infrared-to-visible light data conversion device based on visual details further includes a training module for acquiring a training set containing both infrared and visible light image sets; constructing an initial adversarial generative network (PGN) model; and iteratively training the initial GPN model based on the training set. The initial GPN model includes a multi-task generator and a discriminator. In one iteration, the multi-task generator performs three training tasks: a conversion task, a super-resolution task, and a super-resolution guidance and constraint task. The conversion task involves selecting an infrared training sample image as input to obtain a corresponding visible light converted image. The super-resolution task involves selecting a blurred visible light training sample image as input and performing super-resolution processing to obtain a reconstructed visible light image. The super-resolution guidance and constraint task involves using the visible light converted image obtained from the conversion task as input and performing super-resolution processing to obtain a reconstructed visible light converted image. The discriminator alternately uses the visible light training sample image and the output of one iteration of the multi-task generator as input, and satisfies the adversarial generative loss function as a constraint. Iterative training continues until a preset condition is met, and the model parameters are saved to obtain the image conversion model.

[0124] Optionally, the conversion task is constrained by the structural consistency contrast loss function satisfying the infrared training sample image and the corresponding visible light converted image;

[0125] The structural consistency comparison loss function is as follows:

[0126]

[0127] in, V represents the structural consistency contrastive loss function. q The extracted features f of the infrared training sample images i Selected anchor point features These represent the extracted features of the corresponding visible light converted images. Selected positive and negative sample features, v q and positive samples For the same location on the feature map, negative samples There is no positional correspondence between the anchor point feature and the anchor point feature.

[0128] Optionally, the super-resolution task is constrained by the reconstructed visible light image and the corresponding visible light training sample image satisfying the super-resolution training supervision loss function;

[0129] The super-resolution training supervision loss function is as follows:

[0130]

[0131]

[0132]

[0133] in, Let y represent the super-resolution training supervision loss function. j This represents the j-th visible light training sample image. This represents the blurred result of the j-th visible light training sample image. Downsample indicates random downsampling, and Upsample indicates upsampling. Let represent the reconstructed visible light image corresponding to the j-th visible light training sample image.

[0134] Optionally, the super-resolution guidance constraint task is constrained by the reconstructed visible light converted image and the corresponding visible light converted image satisfying the super-resolution guidance constraint loss function;

[0135] The super-resolution guidance constraint loss function is as follows:

[0136]

[0137] in, Let G(x) represent the super-resolution guidance constraint loss function. i () represents the visible light converted image corresponding to the i-th infrared training sample image. G(x) represents i The reconstructed visible light converted image obtained through super-resolution processing, This is the gradient truncation symbol.

[0138] The infrared-to-visible light data conversion device based on visual details provided in the above embodiments is only illustrated by the division of the above-described program modules during the image conversion process. In practical applications, the above processing can be assigned to different program modules as needed, thus dividing the internal structure of the device into different program modules to complete all or part of the method steps described above. Furthermore, the infrared-to-visible light data conversion device based on visual details provided in the above embodiments and the infrared-to-visible light data conversion method embodiments based on visual details belong to the same concept; their specific implementation process is detailed in the method embodiments and will not be repeated here.

[0139] Please see Figure 10 In another aspect of the embodiments of this application, a computer device is also provided, including a memory 211 and a processor 212. The memory 211 stores a computer program. When the computer program is executed by the processor, the processor 212 performs the steps of the infrared-to-visible light data conversion method based on visual details provided in any of the above embodiments of this application, or performs the steps of the image conversion model training method provided in any of the above embodiments of this application.

[0140] In another aspect, this application also provides a storage medium storing a computer program. When the computer program is executed by a processor, the processor performs the steps of the infrared-to-visible light data conversion method based on visual details provided in any of the above embodiments of this application, or the steps of the image conversion model training method provided in any of the above embodiments of this application, achieving the same technical effect. To avoid repetition, further details are omitted here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0141] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0142] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, infrared thermal imager, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0143] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for converting infrared to visible light data based on visual details, characterized in that, include: An image conversion model is obtained through model training. Model training includes: Obtain a training set containing both infrared and visible light image sets; An initial adversarial generative network (PGN) model is constructed, and iterative training is performed on the initial GPN model based on the training set. The initial GPN model includes a multi-task generator and a discriminator. In one iteration, the multi-task generator performs three training tasks: a conversion task, a super-resolution task, and a super-resolution guidance and constraint task. The conversion task involves selecting an infrared training sample image as input to obtain a corresponding visible light converted image. The super-resolution task involves selecting a blurred visible light training sample image as input and performing super-resolution processing to obtain a reconstructed visible light image. The super-resolution guidance and constraint task involves using the visible light converted image obtained from the conversion task as input and performing super-resolution processing to obtain a reconstructed visible light converted image. The discriminator alternately uses the visible light training sample image and the output of one iteration of the multi-task generator as input, and is constrained by satisfying the adversarial generative loss function. The image conversion model is obtained by iterative training until the preset conditions are met and the model parameters are saved. Acquire the infrared image to be converted; The infrared image to be converted is used as input to the trained image conversion model, and the corresponding visible light image is converted and output by the multi-task generator of the image conversion model. The image conversion model is an adversarial generative network model, comprising a multi-task generator and a discriminator. The training set of the image conversion model includes infrared training sample images and visible light training sample images. During the training process, the training tasks in one iteration of the multi-task generator include: a conversion task that uses the infrared training sample images as input to convert them into corresponding visible light converted images; a super-resolution task that uses the blurred visible light training sample images as input to perform super-resolution processing to reconstruct the visible light image; and a super-resolution guidance constraint task that uses the visible light converted images obtained from the conversion task as input to perform super-resolution processing to reconstruct the visible light converted image. The discriminator alternately uses the visible light training sample images and the output of one iteration of the multi-task generator as input, and satisfies the adversarial generative loss function as a constraint.

2. The infrared-to-visible light data conversion method based on visual details as described in claim 1, characterized in that, The adversarial generation loss function is as follows: ； in, Adversarial generative loss function Indicates the discriminator, Indicates a multi-task generator. This represents the identification result of the j-th visible light training sample image in the visible light image set. This represents the identification result of the visible light converted image, reconstructed visible light image, or reconstructed visible light converted image corresponding to the i-th infrared training sample image in the infrared image set, where, This represents the maximum likelihood estimation algorithm.

3. The infrared-to-visible light data conversion method based on visual details as described in claim 1, characterized in that, The training sets for the infrared image set and the visible light image set are as follows: ； ； in, It refers to an infrared image set. It refers to the set of visible light images. This refers to infrared training sample images. This refers to visible light training sample images. It represents the total number of images in the infrared image set. Indicates the first One picture, It is the total number of images in the visible light image set. Indicates the first [Number of pictures] 4. The infrared-to-visible light data conversion method based on visual details as described in claim 1, characterized in that, The conversion task is constrained by the structural consistency contrast loss function satisfying the infrared training sample image and the corresponding visible light converted image. The structural consistency comparison loss function is as follows: ； in, This represents the structural consistency comparison loss function. Extracted features of infrared training sample images Selected anchor point features These represent the extracted features of the corresponding visible light converted images. Selected positive and negative sample features and positive samples For the same location on the feature map, negative samples There is no positional correspondence between the anchor point feature and the anchor point feature.

5. The infrared-to-visible light data conversion method based on visual details as described in claim 1, characterized in that, The super-resolution task is constrained by the super-resolution training supervision loss function being satisfied by the reconstructed visible light image and the corresponding visible light training sample image. The super-resolution training supervision loss function is as follows: ； ； ； in, This represents the super-resolution training supervision loss function. This represents the j-th visible light training sample image. This represents the result of blurring the j-th visible light training sample image. Indicates random downsampling. Indicates upsampling, This represents the reconstructed visible light image obtained by reconstructing the j-th visible light training sample image through the multi-task generator G.

6. The infrared-to-visible light data conversion method based on visual details as described in claim 1, 4, or 5, characterized in that, The super-resolution guidance constraint task is constrained by the super-resolution guidance constraint loss function satisfying the reconstructed visible light converted image and the corresponding visible light converted image. The super-resolution guidance constraint loss function is as follows: ； in, This represents the super-resolution guidance constraint loss function. This represents the visible light converted image corresponding to the i-th infrared training sample image. express The reconstructed visible light converted image obtained through super-resolution processing, This is the gradient truncation symbol.

7. A training method for an image conversion model, characterized in that, include: Obtain the training set; The training set includes an infrared image set consisting of infrared training sample images and a visible light image set consisting of visible light training sample images; Construct an initial adversarial generative network model; wherein the initial adversarial generative network model includes a multi-task generator and a discriminator; The initial adversarial generative network model is iteratively trained based on the training set. In one iteration, the multi-task generator performs three training tasks: a conversion task, a super-resolution task, and a super-resolution guidance and constraint task. The conversion task selects an infrared training sample image as input to obtain a corresponding visible light converted image. The super-resolution task selects a blurred visible light training sample image as input and performs super-resolution processing to obtain a reconstructed visible light image. The super-resolution guidance and constraint task takes the visible light converted image obtained from the conversion task as input and performs super-resolution processing to obtain a reconstructed visible light converted image. The discriminator alternately takes the visible light training sample image and the output of one iteration of the multi-task generator as input, and satisfies the adversarial generative loss function as a constraint. The model is trained iteratively until the preset conditions are met, and the model parameters are saved to obtain the image conversion model.

8. An infrared-to-visible light data conversion device based on visual details, characterized in that, include: The acquisition module is used to acquire the infrared image to be converted; An image conversion model is used to take the infrared image to be converted as input, and convert it through the multi-task generator of the image conversion model to output the corresponding visible light image. The image conversion model is an adversarial generative network model, comprising a multi-task generator and a discriminator. The training set of the image conversion model includes infrared training sample images and visible light training sample images. During the training process, the training tasks in one iteration of the multi-task generator include: a conversion task that uses the infrared training sample images as input to convert them into corresponding visible light converted images; a super-resolution task that uses the blurred visible light training sample images as input to perform super-resolution processing to reconstruct the visible light images; and a super-resolution guidance constraint task that uses the visible light converted images obtained from the conversion tasks as input to perform super-resolution processing to reconstruct the visible light converted images. The discriminator alternately uses the visible light training sample images and the output of one iteration of the multi-task generator as input, and satisfies the adversarial generative loss function as a constraint. A training module is used to acquire a training set containing infrared and visible light image sets; construct an initial adversarial generative network (GGN) model; and iteratively train the initial GGN model based on the training set. The initial GGN model includes a multi-task generator and a discriminator. In one iteration, the multi-task generator performs three training tasks: a conversion task, a super-resolution task, and a super-resolution guidance and constraint task. The conversion task involves selecting infrared training sample images as input to obtain corresponding visible light converted images. The super-resolution task involves selecting blurred visible light training sample images as input and performing super-resolution processing to obtain reconstructed visible light images. The super-resolution guidance and constraint task involves using the visible light converted images obtained from the conversion task as input and performing super-resolution processing to obtain reconstructed visible light converted images. The discriminator alternately uses the visible light training sample images and the output of one iteration of the multi-task generator as input, and satisfies the adversarial generative loss function as a constraint. Iterative training continues until preset conditions are met, and the model parameters are saved to obtain the image conversion model.

9. A computer device, characterized in that, It includes a processor and a memory connected to the processor, the memory storing a computer program that can be executed by the processor; When the computer program is executed by the processor, it implements the infrared-to-visible light data conversion method based on visual details as described in any one of claims 1 to 6, or the training method for the image conversion model as described in claim 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the infrared-to-visible light data conversion method based on visual details as described in any one of claims 1 to 6, or the training method for the image conversion model as described in claim 7.

Images