A method, device, equipment and storage medium for infrared image conversion training

Abstract

The present invention discloses a conversion training method, device, equipment and computer-readable storage medium for infrared images. The method includes: obtaining a first image domain set and a second image domain set; based on inter-frame difference consistency and contrast learning, using the first image domain set and the second image domain set to iteratively train weight parameters of a preset generator and a preset discriminator to obtain a trained preset generator; according to the trained preset generator, obtaining a conversion generator; the present invention introduces the idea of contrast learning to avoid the strict bidirectional mapping method of the existing cycle consistency idea, and can be better applied to infrared image conversion with a time span, so that the converted daytime true color image can retain the original semantic structure information of the nighttime infrared image; based on inter-frame difference consistency, the idea of inter-frame difference is used to constrain the inter-frame difference between input and output, which can effectively prevent the problem of inter-frame flickering in the generated true color video.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an infrared image conversion training method, device, equipment and computer-readable storage medium. Background Art

[0002] Due to its unique thermal imaging principle, infrared imaging technology can capture images at night or in extreme environments that are invisible to the naked eye or various visible light detectors. This unique advantage has led to its widespread application in a wide range of fields, including nighttime surveillance, fire detection, and nighttime assisted driving. However, infrared imaging also has significant drawbacks. The resulting grayscale images do not conform to the human visual perception system. Furthermore, compared to true-color visible light images, infrared images lack much detailed texture information, making various back-end image algorithms difficult to implement.

[0003] In recent years, deep learning has achieved results far exceeding those of traditional algorithms in various image processing tasks. In the field of infrared color night vision, infrared true color conversion algorithms based on deep neural networks have also emerged. Compared to traditional fusion techniques, these algorithms can directly convert infrared images into visible true color images, demonstrating certain technical advantages. However, deep neural networks for infrared true color conversion have several drawbacks: supervised learning algorithms require strictly paired and registered samples, which are nearly impossible to obtain for cross-temporal conversion tasks such as infrared color night vision. Unsupervised learning algorithms are generally based on the principle of cycle consistency. While these algorithms do not require strictly paired samples, their training requirements are often too stringent. The bidirectional mapping between infrared and visible images requires paired generator and discriminator models, which requires high computing power. Cycle consistency is prone to mode collapse, or training failure, when the difference between infrared and visible images is too large, resulting in unstable generated images. Furthermore, existing algorithms lack constraints on temporal continuity. When the conversion task becomes a video of consecutive frames, style drift or inter-frame flickering can occur, significantly reducing the visual quality of the generated video.

[0004] Therefore, how to train a conversion model for infrared images that can generate realistic and detailed visible light true color images while avoiding the occurrence of inter-frame flicker problems and improving user experience is an urgent problem that needs to be solved today. Summary of the Invention

[0005] The purpose of the present invention is to provide an infrared image conversion training method, device, equipment and computer-readable storage medium, so that the conversion model of the trained infrared image can generate realistic and detailed visible light true color images while avoiding the occurrence of inter-frame flicker problems.

[0006] To solve the above technical problems, the present invention provides a conversion training method for infrared images, comprising:

[0007] Acquire a first image domain set and a second image domain set; wherein the first image domain set includes infrared images corresponding to infrared videos, and the second image domain set includes true color images corresponding to true color videos, and the infrared videos and the true color videos have the same scene;

[0008] Based on inter-frame difference consistency and contrast learning, the weight parameters of a preset generator and a preset discriminator are iteratively trained using the first image domain set and the second image domain set to obtain a trained preset generator; wherein the preset generator is used to convert the infrared image into a converted true color image, and the preset discriminator is used to discriminate between true and false results corresponding to the input true color image;

[0009] According to the trained preset generator, a conversion generator is obtained, so as to use the conversion generator to perform image conversion on the actual infrared video to obtain the target true color video.

[0010] Optionally, the iterative training of weight parameters of a preset generator and a preset discriminator using the first image domain set and the second image domain set based on inter-frame difference consistency and contrast learning includes:

[0011] Based on the idea of inter-frame difference consistency, the weight parameters of the preset generator and the preset discriminator are iteratively trained using the first image domain set and the second image domain set, so that the inter-frame difference between the corresponding consecutive frame images input and output of the preset generator is less than a preset value;

[0012] Based on the idea of contrastive learning of semantic structure, the weight parameters of the preset generator and the preset discriminator are iteratively trained using the first image domain set and the second image domain set to ensure that the semantic information of the corresponding images input and output of the preset generator are consistent.

[0013] Optionally, the contrastive learning idea is specifically implemented based on a semantic structure loss function, and the semantic structure loss function includes a multi-layer infrared image block contrast loss function and a multi-layer true color image block contrast loss function;

[0014] Among them, the multi-layer infrared image block contrast loss function is The multi-layer true color image block contrast loss function is: X is the first image domain set, Y is the second image domain set, l is the target convolution layer in the encoder of the preset generator, L is the number of target convolution layers in the encoder, s is the target position in each target convolution layer, S l is the number of target positions in the target convolutional layer, z lis the feature generated after passing through the encoder and the preset multilayer perceptron network, is a feature at the target position corresponding to the infrared image or the true color image and the converted true color image, is a feature at the target position in the infrared image or the true color image that does not correspond to the converted true color image, is a feature at the target position in the infrared image or the true color image corresponding to the converted true color image.

[0015] Optionally, the inter-frame difference consistency idea is specifically implemented based on an inter-frame difference consistency loss function, and the inter-frame difference consistency loss function is

[0016] Wherein, T is the total number of frames of the infrared video, I t is the input frame sequence of the preset generator, is the output frame sequence of the preset generator, φ(x t )=f m (x t+1 )-f m (x t ), x t is the gap between the t+1th frame and the tth frame, m is the target feature layer, f m (x t ) is the feature extracted by the convolutional layer of the preset convolutional neural network.

[0017] Optionally, the iterative training of weight parameters of a preset generator and a preset discriminator using the first image domain set and the second image domain set based on inter-frame difference consistency and contrast learning further includes:

[0018] Based on the idea of generative adversarial training, the weight parameters of the preset generator and the preset discriminator are iteratively trained simultaneously using the first image domain set and the second image domain set, so that the scores of the true and false results corresponding to the true color image output by the preset discriminator are equal to those of the true and false results corresponding to the converted true color image.

[0019] Optionally, the generative adversarial idea is specifically implemented based on a generative adversarial loss function, which is G(·) is the output of the preset generator, D(·) is the output of the preset discriminator, X is the infrared image, Y is the true color image, y k is the true color video frame image of the kth frame, x i is the infrared video frame image of the i-th frame.

[0020] Optionally, the method of iteratively training weight parameters of a preset generator and a preset discriminator using the first image domain set and the second image domain set based on inter-frame difference consistency and contrast learning to obtain a trained preset generator includes:

[0021] According to the preset loss function, the weight parameters of the preset generator and the preset discriminator are iteratively trained using the first image domain set and the second image domain set to obtain a trained preset generator; wherein the preset loss function is the sum of the products of the semantic structure loss function, the inter-frame difference consistency loss function and the generative adversarial loss function and the weight coefficients of their respective corresponding loss functions.

[0022] Optionally, after obtaining the conversion generator based on the trained preset generator, the method further includes:

[0023] Acquire a set of images to be converted; wherein the set of images to be converted includes the infrared images to be converted corresponding to the actual infrared video;

[0024] The conversion generator is used to perform image conversion on the infrared image to be converted to obtain the target true color video.

[0025] Optionally, obtaining the first image domain set and the second image domain set includes:

[0026] Acquire training video data; wherein the training video data includes the infrared video and the true color video;

[0027] Dividing the training video data into frames to obtain single-frame images;

[0028] Converting the single-frame image to obtain a target single-frame image with a preset image specification;

[0029] According to the video frame sequence, a preset number of continuous single-frame images of the target are spliced to obtain an infrared image corresponding to the infrared video and a true color image corresponding to the true color video.

[0030] The present invention also provides an infrared image conversion training device, comprising:

[0031] an acquisition module, configured to acquire a first image domain set and a second image domain set; wherein the first image domain set includes infrared images corresponding to infrared videos, and the second image domain set includes true color images corresponding to true color videos, and the infrared videos and the true color videos have the same scene;

[0032] a training module configured to iteratively train weight parameters of a preset generator and a preset discriminator using the first image domain set and the second image domain set based on inter-frame difference consistency and contrast learning, thereby obtaining a trained preset generator; wherein the preset generator is configured to convert the infrared image into a converted true color image, and the preset discriminator is configured to discriminate between true and false results corresponding to the input true color image;

[0033] The generation module is used to obtain a conversion generator based on the trained preset generator, so as to use the conversion generator to perform image conversion on the actual infrared video to obtain the target true color video.

[0034] The present invention also provides an infrared image conversion training device, comprising:

[0035] memory for storing computer programs;

[0036] The processor is configured to implement the steps of the infrared image conversion training method described above when executing the computer program.

[0037] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the infrared image conversion training method as described above are implemented.

[0038] The present invention provides a conversion training method for infrared images, comprising: obtaining a first image domain set and a second image domain set; wherein the first image domain set includes infrared images corresponding to infrared videos, and the second image domain set includes true color images corresponding to true color videos, and the infrared videos and true color videos have the same scene; based on inter-frame difference consistency and contrast learning, using the first image domain set and the second image domain set to iteratively train weight parameters of a preset generator and a preset discriminator to obtain a trained preset generator; wherein the preset generator is used to convert infrared images into converted true color images, and the preset discriminator is used to discriminate between true and false results corresponding to input true color images; based on the trained preset generator, a conversion generator is obtained, so as to use the conversion generator to perform image conversion on the actual infrared video to obtain a target true color video;

[0039] As can be seen, the present invention uses the first image domain set and the second image domain set to iteratively train the weight parameters of the preset generator and the preset discriminator based on inter-frame difference consistency and contrastive learning to obtain a trained preset generator. The introduction of contrastive learning avoids the strict bidirectional mapping method of the existing cycle consistency idea, making it more suitable for infrared image conversion with time spans, and allowing the converted daytime true-color image to retain the original semantic structure information of the nighttime infrared image, achieving the conversion and generation of realistic and detailed daytime true-color images. Based on inter-frame difference consistency, the inter-frame difference idea is used to constrain the inter-frame differences between input and output, which can effectively prevent the inter-frame flicker problem caused by the generated true-color video. In addition, the present invention also provides an infrared image conversion training device, equipment, and computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0041] Figure 1 A flowchart of an infrared image conversion training method provided by an embodiment of the present invention;

[0042] Figure 2 A schematic structural diagram of a residual module of an infrared image conversion training method provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the structure of a Markov discriminator of a conversion training method for infrared images provided by an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of a semantic structure loss function based on contrastive learning in a conversion training method for infrared images provided by an embodiment of the present invention;

[0045] Figure 5 A structural block diagram of an infrared image conversion training device provided by an embodiment of the present invention;

[0046] Figure 6 A schematic structural diagram of an infrared image conversion training device provided by an embodiment of the present invention;

[0047] Figure 7 This is a schematic diagram of the specific structure of an infrared image conversion training device provided in this embodiment. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0049] Please refer to Figure 1 , Figure 1 This is a flow chart of a conversion training method for infrared images provided by an embodiment of the present invention. The method may include:

[0050] Step 101: Obtain a first image domain set and a second image domain set; wherein the first image domain set includes infrared images corresponding to infrared videos, and the second image domain set includes true color images corresponding to true color videos, and the infrared videos and true color videos have the same scene.

[0051] It can be understood that the first image domain set in this step can be a set of infrared images corresponding to infrared video (such as night infrared video), and the second image domain set in this step can be a set of true color images corresponding to true color video (such as daytime true color video) of the same scene as the infrared video.

[0052] Specifically, the specific method for the processor to obtain the first image domain set and the second image domain set in this step can be set by the designer according to practical scenarios and user needs. For example, the processor can directly receive the first image domain set and the second image domain set. The processor can also pre-process the received infrared video and true color video to construct and generate the first image domain set and the second image domain set. For example, a vehicle-mounted binocular camera device can be used to collect dual-light video data of the same scene in two time periods, daytime and nighttime, namely, infrared video and visible light true color video during the daytime and infrared video and visible light true color video during the nighttime, so that the processor can pre-process the received infrared video at night and visible light true color video during the daytime to construct and generate the first image domain set and the second image domain set. The binocular camera can also perform registration and synchronization processing in the hardware part to ensure that the frame rate and scene of the infrared video at night and the visible light true color video during the daytime are the same.

[0053] Correspondingly, the specific method for the above-mentioned processor to preprocess the received infrared video and true color video and construct and generate the first image domain set and the second image domain set can be set by the designer, such as the processor obtaining training video data; dividing the training video data into frames to obtain single-frame images; converting the single-frame images to obtain target single-frame images of preset image specifications; splicing a preset number of continuous target single-frame images according to the video frame sequence to obtain infrared images corresponding to the infrared video and true color images corresponding to the true color video; wherein the training video data includes infrared video and true color video. For example, the processor can perform three pre-processing operations on the received nighttime infrared video and daytime true color video: framing, converting (resizing), and splicing (merging). The framing operation converts the nighttime infrared video and daytime true color video into multiple consecutive single-frame images; the conversion operation converts the single-frame images into a target single-frame image of a preset image size (e.g., 256*256); and the splicing operation splices a preset number (n) of consecutive target single-frame images to form a set of n consecutive frames as one image (i.e., infrared image or true color image). For example, the target single-frame images of n consecutive frames are spliced in the width or length direction to form a set of n consecutive frames as one image. In other words, whether it is the first image domain set corresponding to the infrared video (e.g., nighttime infrared video) or the second image domain set corresponding to the true color video (e.g., daytime true color video), each image in the set (i.e., sample set) can be spliced from the target single-frame images corresponding to n consecutive original frame sequence images (i.e., single-frame images), where n can be a positive integer between 2 and 5. The processor may only perform the two pre-processing operations of framing and splicing or the two pre-processing operations of framing and conversion on the received nighttime infrared video and daytime true color video. This embodiment does not impose any limitation on this.

[0054] Step 102: Based on inter-frame difference consistency and contrast learning, the weight parameters of the preset generator and the preset discriminator are iteratively trained using the first image domain set and the second image domain set to obtain a trained preset generator; wherein the preset generator is used to convert the infrared image into a converted true color image, and the preset discriminator is used to determine the true or false results corresponding to the input true color image.

[0055] It is understandable that this embodiment provides an infrared true-color night vision algorithm based on contrastive learning and inter-frame differential loss: I2V-NET (Infrared to Visible Network), which can not only obtain more detailed local style features than existing cycle consistency algorithms, but also effectively avoid the problem of inter-frame flicker. The overall training structure of I2V-NET is simple, consisting of a generator (i.e., a preset generator) and a discriminator (i.e., a preset discriminator), without the need for other complex auxiliary structures, and is fast and effective in training.

[0056] Specifically, in this embodiment, the preset generator can convert the infrared image corresponding to the infrared video into the corresponding true color image (i.e., converted true color image); the preset discriminator can discriminate the true or false result corresponding to the input true color image, that is, judge whether the input true color image is a true color image from the second image domain set (i.e., real true color image), or a converted true color image generated by the preset generator (i.e., converted true color image). The specific structure of the preset generator and the preset discriminator in this embodiment can be set by the designer according to the practical scenario and user needs. The preset generator may include an encoder (Encoder), a converter and a decoder; wherein the encoder may include three groups of "Conv+instance norm+relu", which are used to downsample the input frame sequence image into a feature map with a set number of channels, Conv can be a convolution layer, instance norm can be a normalization layer, and relu can be a nonlinear activation layer; the converter is composed of a preset number of modules (m) residual modules (Resnet_block), such as Figure 2 As shown in , its function is to reorganize and transform the feature map obtained after the encoder downsampling; the decoder restores the image size through deconvolution operation, and finally obtains the generated image (i.e., the converted true color image). The preset discriminator can be used as follows Figure 3 The Markov discriminator (PatchGAN) shown in the figure can be composed entirely of convolutional layers, and the final output is a k*k matrix. The mean of the true or false outputs of the output matrix is finally taken as the output of the true or false result; each output in the output matrix represents a receptive field in the input image (such as a converted true color image or a true color image), corresponding to a patch of the input image.

[0057] It should be noted that in this step, the weight parameters of the preset generator and preset discriminator of I2V-NET are iteratively trained based on inter-frame difference consistency and contrastive learning. After several iterations, the weight parameters of the appropriate preset generator and preset discriminator (such as deep neural network weights) can be trained to obtain the generator (i.e., conversion generator) for image conversion of actual infrared video using the trained preset generator.

[0058] Correspondingly, for the specific method of iteratively training the weight parameters of the preset generator and the preset discriminator using the first image domain set and the second image domain set based on the consistency of inter-frame differences and contrastive learning in this step, the designer can set it by himself according to the practical scenario and user needs. For example, in this step, the processor can iteratively train the weight parameters of the preset generator and the preset discriminator using the first image domain set and the second image domain set based on the idea of consistency of inter-frame differences, so that the inter-frame difference between the corresponding continuous frame images of the input and output preset generators is less than the preset value; the weight parameters of the preset generator and the preset discriminator can be iteratively trained using the first image domain set and the second image domain set based on the idea of contrastive learning of semantic structure, so that the semantic information of the corresponding images of the input and output preset generators is consistent; for example, the processor can use the preset loss function based on the idea of consistency of inter-frame differences and the idea of contrastive learning based on semantic structure, The weight parameters of the preset generator and the preset discriminator are iteratively trained using the first image domain set and the second image domain set to obtain a trained preset generator; wherein the preset loss function may include a semantic structure loss function based on contrastive learning, which is used to adjust the semantic structure information of the image generated by the preset generator, and the converted true color image generated by the preset generator is trained based on the idea of contrastive learning to retain the original semantic structure information of the infrared image, so that the semantic information of the corresponding image of the input and output preset generators is consistent, that is, the contrastive learning idea based on semantic structure can be specifically implemented based on the semantic structure loss function; the preset loss function may also include an inter-frame difference consistency loss function, which is used to adjust the inter-frame difference between the continuous frame images generated by the preset generator and the continuous frame images input, so that the inter-frame difference between the corresponding continuous frame images of the input and output preset generators is less than a preset value, so as to utilize the inter-frame difference consistency loss (temporal loss) function, under the premise of ensuring that the preset generator can correctly complete the generation task, the true color images of the continuous frames generated by it are consistent with the inter-frame transformation of the original input video, that is, they have the same temporal coherence, and improve the problem of inter-frame flickering in the generated video frame sequence, that is, the inter-frame difference consistency idea can be specifically implemented based on the inter-frame difference consistency loss function. The processor can also use a preset network model based on the inter-frame difference consistency idea and the contrast learning idea based on the semantic structure, and use the first image domain set and the second image domain set to iteratively train the weight parameters of the preset generator and the preset discriminator to obtain a trained preset generator. This embodiment does not impose any restrictions on this.

[0059] Correspondingly, in this step, based on the generative adversarial thinking, the weight parameters of the preset generator and the preset discriminator can be iteratively trained simultaneously using the first image domain set and the second image domain set, so that the true and false results corresponding to the true color image output by the preset discriminator are equal to the true and false results corresponding to the converted true color image (e.g., both are 0.5), so that the preset discriminator has a sufficiently strong ability to distinguish true from false, and the converted true color image generated by the preset generator is more like the actual true color image. For example, the above-mentioned preset loss function can also include a generative adversarial loss function, which is used to adjust the output of the preset discriminator for the true color image and the converted converted true color image and adjust the output of the preset generator, so that the preset discriminator has a sufficiently strong ability to distinguish true from false, and the converted true color image generated by the preset generator is more like the actual true color image, that is, the generative adversarial thinking can be specifically implemented based on the generative adversarial loss function; that is, in this step, the weight parameters of the preset generator and the preset discriminator can be iteratively trained using the first image domain set and the second image domain set according to the inter-frame difference consistency loss function, the semantic structure loss function and the generative adversarial loss function to obtain the trained preset generator.

[0060] Specifically, for the above-mentioned generative adversarial loss function, since the preset generator (G) and the preset discriminator (D) of I2V-NET are trained at the same time, the training process of the two is a binary game process. The purpose of training the preset discriminator is to make it have a strong enough ability to distinguish between true and false, that is, when the generated result fake_B of the preset generator passes through the preset discriminator, the preset discriminator can give a low score, and the original true color image real_B in the second image domain set gives a high score after passing through the preset discriminator; however, when training the preset generator, it is just the opposite. The training goal of the preset generator is to get a high score when the generated image fake_B of the preset generator passes through the preset discriminator, that is, fake_B is more in line with the distribution law of real_B, and can achieve the degree of being indistinguishable from the real in style, and "more like" the true color image in the second image domain set (that is, the target domain true color image) in appearance. In this way, the two form a maximum and minimum value game during training, and the loss function can be the generative adversarial loss (GAN_loss) function:

[0061]

[0062] in, can be a generative adversarial loss function, G(·) can be the output of a preset generator, D(·) can be the output of a preset discriminator, X can be an infrared image in the first image domain set (i.e., a sample image in the source domain), Y can be a true color image in the second image domain set (i.e., a sample image in the target domain), and y k Can be the true color video frame image of the kth frame, x i It can be the infrared video frame image of the i-th frame.

[0063] Correspondingly, for the above-mentioned semantic structure loss function based on contrastive learning, the traditional method based on CycleGAN (a cyclic generative adversarial network) is to maintain the consistency of the input and output semantic structures of the generator through the idea of bidirectional mapping constructed by cyclic consistency, which will result in the need to train two generators and two discriminators at the same time during training, which consumes a lot of computing power, and this strict bidirectional mapping based on the original image is prone to training failure, and the generated image lacks detailed information. The I2V-NET in this embodiment only needs a preset generator and a preset discriminator, and finds the corresponding mutual information by comparing the input and output image blocks (patch), and obtains an output with a strong content correspondence by maximizing the mutual information. The method of maximizing mutual information can be the idea of contrastive learning, by replacing the query signal v and the positive signal v + and N negative signals v - Mapped to a K-dimensional vector, where is a K-dimensional real number, is an N×K dimensional real number. To prevent mode collapse, these vectors are normalized to the unit sphere, thus establishing an N+1 classification problem. Finally, the probability of a positive sample being selected is expressed by calculating the cross entropy loss. The loss function can be the information network loss (infoNCE Loss) function:

[0064]

[0065] Where τ is the preset scaling factor.

[0066] In this embodiment, a multi-layer and paired contrastive learning method can be used. Under unsupervised learning conditions, the semantic information of the generated image can be constrained to be consistent with the input image at the same time in the image (image) and image patch (patch). Therefore, the image generated by the preset generator has common features in content of the entire image, and each corresponding patch of the input and output also has such a corresponding relationship, such as Figure 4 As shown in the figure, if a car patch is generated in the daytime true color image, then the nighttime infrared image before conversion should also contain the corresponding car patch, rather than patches from trees or other background parts. In this embodiment, the encoder part of the preset generator in I2V-NET can be G enc , by reusing G enc And then add the preset multi-layer perceptron (MLP) network H l (like Figure 4The MLP network in the middle) stacks the input and output features of the preset generator; for example, the patches of the real night infrared image real_A and the generated corresponding true color image fake_B are respectively sent to G enc and H l , generating a series of features Then, the corresponding PatchNCE loss function can be used to compare the differences between image patches, so as to match the image patches at corresponding positions of input and output. Taking the infrared image in the first image domain set (i.e., the source domain sample image) as an example, the semantic structure loss function can include a multi-layer infrared image patch contrast loss function L PatchNCE (G,H,X):

[0067]

[0068] Wherein, l can be the target convolution layer in the encoder of the preset generator, such as any one convolution layer, multiple convolution layers, or all convolution layers of interest in the encoder. The more layers of interest, the better the effect. L can be the number of target convolution layers in the encoder of the preset generator, that is, the number of convolution layers of interest in the encoder. X can be the first image domain set. s can be the target position in each target convolution layer, that is, the position of interest in each convolution layer. S l It can be the number of target positions in the target convolution layer, that is, the number of positions of interest in the target convolution layer; l is the feature generated after passing through the encoder and the preset multilayer perceptron network, is the feature at the target position corresponding to the input image (such as infrared image) and the converted true color image, is the feature at the target position in the input image that does not correspond to the converted true color image, It can be a feature at a target position in the input image corresponding to the converted true color image, and x~X can represent that x obeys the probability distribution of the first image domain set X.

[0069] In order to prevent the preset generator from making unnecessary changes, the corresponding image patch network loss function can also be used for the true color images in the second image domain set (i.e., the target domain sample images), that is, the semantic structure loss function can include a multi-layer true color image patch contrast loss function L PatchNCE (G,H,Y):

[0070]

[0071] Wherein, Y can be the second image domain set, is the feature at the target position corresponding to the input image (such as a true color image) and the converted true color image, is the feature at the target position that does not correspond to the input image and the converted true color image, It can be a feature at a target position in the input image corresponding to the converted true color image, and y~Y can represent that y obeys the probability distribution of the second image domain set Y.

[0072] Specifically, for the above-mentioned inter-frame difference consistency loss function, the current existing algorithms generate frame sequences stably by training a new auxiliary structure, thereby alleviating the inter-frame flicker problem of the generated video, such as using a dual-channel generator, etc., and some obtain stable video effects by calculating optical flow loss; but these algorithms require a large amount of computation, which is not convenient to use in practice. In this embodiment, there is no need to train redundant auxiliary structures, and the algorithm is more practical and superior; because the original infrared video (or content video) is time-coherent, and this coherence characteristic becomes an inter-frame difference after encoding, it is possible to obtain a video that is approximately stable as the input by requiring the converted video, that is, the generated true color video, to have similar inter-frame differences in the model. For example, in I2V-NET, the inter-frame difference consistency loss function L temp To improve the inter-frame flickering problem in the generated video frame sequence, the formula of the inter-frame difference consistency loss function can be:

[0073]

[0074] Where T is the total number of frames of infrared video, I t It can be a sequence of input frames for the preset generator. The output frame sequence of the preset generator can be t )=f m (x t+1 )-f m (x t ), x t f can be the difference between the t+1th frame and the tth frame, m can be the target feature layer; m (x t ) can be features extracted by the convolutional layer of a preset convolutional neural network, such as features obtained by extracting the outputs of each convolutional layer of a pre-trained vgg16 (a convolutional neural network, i.e., a preset convolutional neural network).

[0075] That is to say, the preset loss function in this step (i.e., the total loss function of I2V-NET) can be the sum of the products of each loss function and the weight coefficients of their corresponding loss functions. For example, when the preset loss function includes the above-mentioned semantic structure loss function, inter-frame difference consistency loss function and generative adversarial loss function, the preset loss function can be the sum of the products of the semantic structure loss function, inter-frame difference consistency loss function and generative adversarial loss function and the weight coefficients of their corresponding loss functions; for example, the preset loss function: L(G,H,D,X,Y)=L gan(G,D,X,Y)+λ X L PatchNCE (G,H,X)+λ Y L PatchNCE (G,H,Y)+λ T L temp , where λ X ,λ Y and λ T The weight coefficients of the corresponding loss functions can be preset respectively.

[0076] It can be understood that, in this step, the processor uses the first image domain set and the second image domain set to iteratively train the weight parameters of the preset generator and the preset discriminator according to the preset loss function to obtain the specific process of the trained preset generator, which can be set by the designer. For example, the processor can construct a preset generator and a preset discriminator; use the preset generator to perform image conversion on the infrared image to obtain a converted true color image; use the preset discriminator to discriminate the converted true color image based on the true color image to obtain the true and false results corresponding to the converted true color image; judge whether the preset number of iterations has been reached; if so, determine that the preset generator training is completed, and execute step 103; if not, the preset loss function can be used to adjust the weight parameters of the preset generator and the preset discriminator, and after the adjustment is completed, perform image conversion on the infrared image using the preset generator to obtain the converted true color image, and continue iterative training; for example, the weight parameters of the preset generator and the preset discriminator of I2V-NET in this embodiment can be used pytorch 1.7.0 (an open source Python machine learning library software) deep learning framework, using Xavier (a parameter initialization method) random parameter initialization and Adam (an optimizer) optimizer for training, with an initial learning rate of 0.0002.

[0077] Step 103: Obtain a conversion generator based on the trained preset generator, and use the conversion generator to perform image conversion on the actual infrared video to obtain the target true color video.

[0078] It is understandable that in this step, the processor can use the trained preset generator to obtain a generator model (ie, conversion generator) for performing true color conversion on the night infrared video that actually needs to be converted (ie, actual infrared video).

[0079] Specifically, for this step, the specific method of obtaining the conversion generator based on the trained preset generator can be set by the designer. For example, the processor can load the weight parameters of the trained preset generator into the newly constructed generator model and determine the loaded generator model as the conversion generator; the processor can also directly determine the trained preset generator as the conversion generator. This embodiment does not impose any restrictions on this.

[0080] Correspondingly, the method provided in this embodiment may also include using a conversion generator to convert the actual infrared video. For example, a processor may obtain a set of images to be converted; and use the conversion generator to convert the infrared images to obtain a target true-color video. The set of images to be converted includes the infrared images to be converted corresponding to the actual infrared video. For example, the processor may read a sequence of nighttime infrared video frames in real time and feed them into a pre-loaded conversion generator to generate a continuous and stable daytime true-color video, thus achieving the daytime infrared true-color conversion task.

[0081] It should be noted that the method provided in this embodiment may also include a testing process of a conversion generator or a trained preset generator, such as a processor may obtain a test image set; use the conversion generator to perform image conversion on the test infrared image in the test image set to obtain a test converted true color video; compare and display the frame sequence of the test converted true color video with the sequence of the test true color video; wherein the test image set may include a test infrared image corresponding to a test infrared video (such as a night infrared video), and the test true color video may be a true color video of the same scene as the test infrared video (such as a night true color video).

[0082] In this embodiment, the embodiment of the present invention is based on inter-frame difference consistency and contrastive learning, and uses the first image domain set and the second image domain set to iteratively train the weight parameters of the preset generator and the preset discriminator to obtain the trained preset generator. The introduction of the idea of contrastive learning avoids the strict two-way mapping method of the existing cycle consistency idea, and can be better applicable to the conversion of infrared images with time spans, and enables the converted daytime true color images to retain the original semantic structure information of the nighttime infrared images, thereby realizing the conversion and generation of realistic and detailed daytime true color images; based on inter-frame difference consistency, the idea of inter-frame difference is used to constrain the inter-frame difference between input and output, which can effectively prevent the inter-frame flicker problem in the generated true color video.

[0083] Corresponding to the above method embodiment, an embodiment of the present invention further provides an infrared image conversion training device. The infrared image conversion training device described below and the infrared image conversion training method described above can refer to each other.

[0084] Please refer to Figure 5 , Figure 5This is a structural block diagram of an infrared image conversion training device provided by an embodiment of the present invention. The device may include:

[0085] An acquisition module 10 is configured to acquire a first image domain set and a second image domain set; wherein the first image domain set includes infrared images corresponding to the infrared video, and the second image domain set includes true color images corresponding to the true color video, and the infrared video and the true color video have the same scene;

[0086] A training module 20 is configured to iteratively train weight parameters of a preset generator and a preset discriminator using the first image domain set and the second image domain set based on inter-frame difference consistency and contrast learning, thereby obtaining a trained preset generator; wherein the preset generator is configured to convert an infrared image into a converted true color image, and the preset discriminator is configured to determine whether an input true color image corresponds to a true or false result;

[0087] The generation module 30 is used to obtain a conversion generator based on the trained preset generator, so as to use the conversion generator to perform image conversion on the actual infrared video to obtain the target true color video.

[0088] Optionally, the training module 20 may include:

[0089] The inter-frame difference consistency training submodule is used to iteratively train the weight parameters of the preset generator and the preset discriminator using the first image domain set and the second image domain set based on the inter-frame difference consistency idea, so that the inter-frame difference between the corresponding consecutive frame images input and output of the preset generator is less than a preset value;

[0090] The contrastive learning training submodule is used to iteratively train the weight parameters of the preset generator and the preset discriminator using the first image domain set and the second image domain set based on the contrastive learning idea of semantic structure, so as to keep the semantic information of the corresponding images of the input and output preset generators consistent.

[0091] Optionally, the contrastive learning idea is specifically implemented based on a semantic structure loss function, which includes a multi-layer infrared image block contrast loss function and a multi-layer true color image block contrast loss function;

[0092] Among them, the contrast loss function of the multi-layer infrared image block is The multi-layer true color image block contrast loss function is X is the first image domain set, Y is the second image domain set, l is the target convolution layer in the encoder of the preset generator, L is the number of target convolution layers in the encoder, s is the target position in each target convolution layer, S l is the number of target locations in the target convolutional layer, z l is the feature generated after passing through the encoder and the preset multilayer perceptron network, is the feature at the target position corresponding to the infrared image or true color image and the converted true color image, is the feature at the target position in the infrared image or true color image that does not correspond to the converted true color image, is the feature at the target position in the infrared image or true color image corresponding to the converted true color image.

[0093] Optionally, the inter-frame difference consistency idea is specifically implemented based on the inter-frame difference consistency loss function, which is

[0094] Where T is the total number of frames of infrared video, I t is the input frame sequence of the preset generator, is the output frame sequence of the preset generator, φ(x t )=f m (x t+1 )-f m (x t ), x t is the gap between the t+1th frame and the tth frame, m is the target feature layer, f m (x t ) is the feature extracted by the convolutional layer of the preset convolutional neural network.

[0095] Optionally, the training module 20 may further include:

[0096] The generative adversarial training submodule is used to iteratively train the weight parameters of the preset generator and the preset discriminator simultaneously based on the generative adversarial idea using the first image domain set and the second image domain set, so that the true and false results corresponding to the true color image output by the preset discriminator are equal to the true and false results corresponding to the converted true color image.

[0097] Optionally, the generative adversarial idea is implemented based on the generative adversarial loss function, which is G(·) is the output of the preset generator, D(·) is the output of the preset discriminator, X is the infrared image, Y is the true color image, y k is the true color video frame image of the kth frame, x i is the infrared video frame image of the i-th frame.

[0098] Optionally, the training module 20 can be specifically used to iteratively train the weight parameters of the preset generator and the preset discriminator using the first image domain set and the second image domain set according to a preset loss function to obtain a trained preset generator; wherein the preset loss function is the sum of the products of the semantic structure loss function, the inter-frame difference consistency loss function and the generative adversarial loss function and the corresponding loss function weight coefficients.

[0099] Optionally, the device may further include:

[0100] The conversion acquisition module is used to acquire a set of images to be converted; wherein the set of images to be converted includes infrared images to be converted corresponding to the actual infrared video;

[0101] The conversion generation module is used to use the conversion generator to perform image conversion on the infrared image to be converted to obtain the target true color video.

[0102] Optionally, the acquisition module 10 may include:

[0103] The video acquisition submodule is used to acquire training video data; wherein the training video data includes infrared video and true color video;

[0104] The frame segmentation submodule is used to segment the training video data into frames to obtain single-frame images;

[0105] A conversion submodule, used to convert a single-frame image to obtain a target single-frame image with preset image specifications;

[0106] The splicing submodule is used to splice a preset number of continuous target single-frame images according to the video frame sequence to obtain an infrared image corresponding to the infrared video and a true color image corresponding to the true color video.

[0107] In this embodiment, the embodiment of the present invention uses the training module 20 to iteratively train the weight parameters of the preset generator and the preset discriminator based on the inter-frame difference consistency and contrastive learning, and obtains the trained preset generator. The introduction of the idea of contrastive learning avoids the strict two-way mapping method of the existing cycle consistency idea, can be better applicable to the conversion of infrared images with time spans, and enables the converted daytime true color images to retain the original semantic structure information of the nighttime infrared images, thereby realizing the conversion and generation of realistic and detailed daytime true color images; based on the inter-frame difference consistency, the idea of inter-frame difference is used to constrain the inter-frame difference between input and output, which can effectively prevent the inter-frame flicker problem in the generation of true color video.

[0108] Corresponding to the above method embodiment, an embodiment of the present invention further provides an infrared image conversion training device. The infrared image conversion training device described below and the infrared image conversion training method described above can refer to each other.

[0109] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of an infrared image conversion training device provided by an embodiment of the present invention. The electronic device may include:

[0110] Memory D1, for storing computer programs;

[0111] The processor D2 is configured to implement the steps of the infrared image conversion training method provided in the above method embodiment when executing a computer program.

[0112] For details, please refer to Figure 7 , Figure 7 This is a schematic diagram of the specific structure of an infrared image conversion training device provided in this embodiment. The conversion training device 310 can vary significantly depending on configuration or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors), memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) storing applications 342 or data 344. The memory 332 and storage medium 330 may be either transient or persistent storage. The program stored in the storage medium 330 may include one or more modules (not shown), each of which may include a series of instruction operations within the electronic device. Furthermore, the CPU 322 may be configured to communicate with the storage medium 330, executing the series of instruction operations stored in the storage medium 330 on the conversion training device 310.

[0113] The conversion training device 310 may further include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0114] The steps in the infrared image conversion training method described above can be implemented by the structure of the infrared image conversion training device.

[0115] Corresponding to the above method embodiment, an embodiment of the present invention further provides a computer-readable storage medium. The computer-readable storage medium described below and the infrared image conversion training method described above can refer to each other.

[0116] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the infrared image conversion training method provided in the above method embodiment.

[0117] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, which may store program codes.

[0118] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. References to the common and similar parts between the various embodiments are sufficient. The devices, apparatuses, and computer-readable storage media disclosed in the embodiments are described briefly because they correspond to the methods disclosed in the embodiments. For relevant details, refer to the description of the methods.

[0119] The above is a detailed introduction to the infrared image conversion training method, device, equipment, and computer-readable storage medium provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A conversion training method for infrared images, characterized in that: include: Acquire a first image domain set and a second image domain set; wherein the first image domain set includes infrared images corresponding to infrared videos, and the second image domain set includes true color images corresponding to true color videos, and the infrared videos and the true color videos have the same scene; Based on inter-frame difference consistency and contrast learning, the weight parameters of a preset generator and a preset discriminator are iteratively trained using the first image domain set and the second image domain set to obtain a trained preset generator; wherein the preset generator is used to convert the infrared image into a converted true color image, and the preset discriminator is used to discriminate between true and false results corresponding to the input true color image; According to the trained preset generator, a conversion generator is obtained, so as to use the conversion generator to perform image conversion on the actual infrared video to obtain the target true color video; The iterative training of weight parameters of a preset generator and a preset discriminator using the first image domain set and the second image domain set based on inter-frame difference consistency and contrast learning includes: Based on the inter-frame difference consistency concept, the weight parameters of the preset generator and the preset discriminator are iteratively trained using the first image domain set and the second image domain set, so that the inter-frame difference between corresponding consecutive frame images input and output of the preset generator is less than a preset value; wherein the inter-frame difference consistency concept is specifically implemented based on an inter-frame difference consistency loss function; Based on the idea of contrastive learning of semantic structure, the weight parameters of the preset generator and the preset discriminator are iteratively trained using the first image domain set and the second image domain set to ensure that the semantic information of the corresponding images input and output of the preset generator are consistent.

2. The infrared image conversion training method according to claim 1, characterized in that: The contrastive learning concept is specifically implemented based on a semantic structure loss function, which includes a multi-layer infrared image block contrast loss function and a multi-layer true color image block contrast loss function; Among them, the multi-layer infrared image block contrast loss function is The multi-layer true color image block contrast loss function is: X is the first image domain set, Y is the second image domain set, l is the target convolution layer in the encoder of the preset generator, L is the number of target convolution layers in the encoder, s is the target position in each target convolution layer, S l is the number of target positions in the target convolutional layer, is the feature at the target position generated after passing through the encoder and the preset multilayer perceptron network, is a feature at the target position in the infrared image or the true color image that does not correspond to the converted true color image, is a feature at the target position in the infrared image or the true color image corresponding to the converted true color image.

3. The infrared image conversion training method according to claim 1, characterized in that: The inter-frame difference consistency loss function is: Wherein, T is the total number of frames of the infrared video, I t is the input frame sequence of the preset generator, is the output frame sequence of the preset generator, φ(x t )=f m (x t+1 )-f m (x t ), x t is the gap between the t+1th frame and the tth frame, m is the target feature layer, f m (x t ) is the feature extracted by the convolutional layer of the preset convolutional neural network.

4. The infrared image conversion training method according to claim 1, characterized in that: The method of iteratively training the weight parameters of the preset generator and the preset discriminator using the first image domain set and the second image domain set based on inter-frame difference consistency and contrast learning further includes: Based on the idea of generative adversarial training, the weight parameters of the preset generator and the preset discriminator are iteratively trained simultaneously using the first image domain set and the second image domain set, so that the scores of the true and false results corresponding to the true color image output by the preset discriminator are equal to those of the true and false results corresponding to the converted true color image.

5. The infrared image conversion training method according to claim 4, characterized in that: The generative adversarial idea is specifically implemented based on the generative adversarial loss function, which is: G(·) is the output of the preset generator, D(·) is the output of the preset discriminator, X is the infrared image, Y is the true color image, y k is the true color video frame image of the kth frame, x i is the infrared video frame image of the i-th frame.

6. The infrared image conversion training method according to claim 4, characterized in that: The method of iteratively training weight parameters of a preset generator and a preset discriminator using the first image domain set and the second image domain set based on inter-frame difference consistency and contrast learning to obtain a trained preset generator includes: According to the preset loss function, the weight parameters of the preset generator and the preset discriminator are iteratively trained using the first image domain set and the second image domain set to obtain a trained preset generator; wherein the preset loss function is the sum of the products of the semantic structure loss function, the inter-frame difference consistency loss function and the generative adversarial loss function and the weight coefficients of their respective corresponding loss functions.

7. The infrared image conversion training method according to claim 1, characterized in that: After obtaining the conversion generator based on the trained preset generator, the following steps are further included: Acquire a set of images to be converted; wherein the set of images to be converted includes the infrared images to be converted corresponding to the actual infrared video; The conversion generator is used to perform image conversion on the infrared image to be converted to obtain the target true color video.

8. The infrared image conversion training method according to any one of claims 1 to 7, characterized in that: The obtaining of the first image domain set and the second image domain set comprises: Acquire training video data; wherein the training video data includes the infrared video and the true color video; Dividing the training video data into frames to obtain single-frame images; Converting the single-frame image to obtain a target single-frame image with a preset image specification; According to the video frame sequence, a preset number of continuous single-frame images of the target are spliced to obtain an infrared image corresponding to the infrared video and a true color image corresponding to the true color video.

9. A conversion training device for infrared images, characterized in that: include: an acquisition module, configured to acquire a first image domain set and a second image domain set; wherein the first image domain set includes infrared images corresponding to infrared videos, and the second image domain set includes true color images corresponding to true color videos, and the infrared videos and the true color videos have the same scene; a training module configured to iteratively train weight parameters of a preset generator and a preset discriminator using the first image domain set and the second image domain set based on inter-frame difference consistency and contrast learning, thereby obtaining a trained preset generator; wherein the preset generator is configured to convert the infrared image into a converted true color image, and the preset discriminator is configured to discriminate between true and false results corresponding to the input true color image; A generation module is used to obtain a conversion generator based on the trained preset generator, so as to use the conversion generator to perform image conversion on the actual infrared video to obtain the target true color video; Wherein, the training module includes: An inter-frame difference consistency training submodule is configured to iteratively train the weight parameters of the preset generator and the preset discriminator using the first image domain set and the second image domain set based on the inter-frame difference consistency concept, so that the inter-frame difference between the input and output of corresponding consecutive frame images of the preset generator is less than a preset value; wherein the inter-frame difference consistency concept is specifically implemented based on an inter-frame difference consistency loss function; The contrastive learning training submodule is used to iteratively train the weight parameters of the preset generator and the preset discriminator using the first image domain set and the second image domain set based on the contrastive learning idea of semantic structure, so as to keep the semantic information of the corresponding images input and output of the preset generator consistent.

10. A conversion training device for infrared images, characterized in that: include: memory for storing computer programs; A processor is configured to implement the steps of the infrared image conversion training method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the infrared image conversion training method according to any one of claims 1 to 8 are implemented.

Images