Training method of image restoration model guided by infrared image and image restoration method

Abstract

The invention provides a training method of an image restoration model guided by an infrared image and an image restoration method. The method comprises the following steps: acquiring a sample data set; constructing an image restoration model; inputting the sample visible light degraded image and the corresponding sample infrared image into a generator to obtain a sample visible light restoration image, and inputting the sample visible light restoration image and the sample visible light normal image into a discriminator; obtaining a first classification probability corresponding to the sample visible light restoration image and a second classification probability corresponding to the sample visible light normal image; and according to the sample visible light normal image, the sample visible light restoration image, the first classification probability and the second classification probability, adjusting model parameters of the image restoration model, continuing to train the adjusted image restoration model until training is finished to obtain a target image restoration model, and performing image restoration based on the target image restoration model. Structural information contained in the infrared image is fully utilized, and the image restoration accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a training method for an image restoration model guided by an infrared image and an image restoration method. Background Art

[0002] Under adverse weather conditions such as low light, haze, rain and snow, visible light images will degrade during imaging due to insufficient light intensity, scattering and occlusion of water vapor, smoke, and rain and snow, resulting in a decrease in imaging quality. Due to the lack of imaging details under these extreme conditions, traditional image restoration algorithms are difficult to effectively restore image details. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] According to a first aspect of the present application, a training method for an infrared image-guided image restoration model is provided, comprising obtaining a sample data set, wherein each sample data in the sample data set comprises a sample visible light normal image, a sample visible light degraded image and a corresponding sample infrared image; constructing an image restoration model to be trained for image restoration, wherein the image restoration model comprises a generator and a discriminator to be trained; inputting the sample visible light degraded image and the corresponding sample infrared image into the generator to obtain a sample visible light restoration image, and inputting the sample visible light restoration image and the sample visible light normal image into the discriminator to obtain a first classification probability corresponding to the sample visible light restoration image and a second classification probability corresponding to the sample visible light normal image; adjusting model parameters of the image restoration model according to the sample visible light normal image, the sample visible light restoration image, the first classification probability and the second classification probability, and continuing to train the adjusted image restoration model until the training is completed to obtain a target image restoration model, wherein the target image restoration model comprises a trained target generator and a target discriminator.

[0005] According to a second aspect of the present application, an image restoration method is provided, comprising: obtaining a visible light degraded image to be restored and a corresponding infrared image; inputting the visible light degraded image to be restored and the corresponding infrared image into a target image restoration model, and having a target generator in the target image restoration model process the visible light degraded image to be restored and the corresponding infrared image to obtain a target visible light restoration image, wherein the target image restoration model is a model trained using the training method described in the first aspect.

[0006] According to a third aspect of the present application, a training device for an infrared image-guided image restoration model is provided, comprising: a first acquisition module, used to acquire a sample data set, wherein each sample data in the sample data set comprises a sample visible light normal image, a sample visible light degraded image and a corresponding sample infrared image; a model construction module, used to construct an image restoration model to be trained for image restoration, wherein the image restoration model comprises a generator and a discriminator to be trained; a second acquisition module, used to input the sample visible light degraded image and the corresponding sample infrared image into the generator to obtain a sample visible light restoration image, and input the sample visible light restoration image and the sample visible light normal image into the discriminator to obtain a first classification probability corresponding to the sample visible light restoration image and a second classification probability corresponding to the sample visible light normal image; a training module, used to adjust the model parameters of the image restoration model according to the sample visible light normal image, the sample visible light restoration image, the first classification probability and the second classification probability, and continue to train the adjusted image restoration model until the training is completed to obtain a target image restoration model, wherein the target image restoration model comprises a trained target generator and a target discriminator.

[0007] According to a fourth aspect of the present application, an image restoration device is provided, comprising: an acquisition module, used to acquire a visible light degraded image to be restored and a corresponding infrared image; a restoration module, used to input the visible light degraded image to be restored and the corresponding infrared image into a target image restoration model, and a target generator in the target image restoration model processes the visible light degraded image to be restored and the corresponding infrared image to obtain a target visible light restoration image, wherein the target image restoration model is a model trained using the training method described in the first aspect.

[0008] In order to achieve the above-mentioned objectives, the fifth aspect of the present application proposes an electronic device, characterized in that it includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, it implements the training method of the infrared image-guided image restoration model as described in the first aspect or the image restoration method as described in the second aspect.

[0009] In order to achieve the above-mentioned objectives, the sixth aspect of the present application proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the training method of the infrared image-guided image restoration model described in the first aspect or the image restoration method described in the second aspect.

[0010] In order to achieve the above-mentioned objectives, the seventh aspect of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the training method of the infrared image-guided image restoration model according to the first aspect or the image restoration method according to the second aspect.

[0011] The technical solution provided by the embodiments of the present application includes at least the following beneficial effects: The present application provides a training method for an infrared image-guided image restoration model. By constructing an image restoration model and training the image restoration model based on a sample visible light degraded image and a corresponding sample infrared image, and performing image restoration through a target image restoration model, the structural information contained in the infrared image can be fully utilized, thereby improving the accuracy and reliability of image restoration.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application. Figure 1 A flowchart of a method for training an infrared image-guided image restoration model provided in an embodiment of the present application; Figure 2 A schematic flow chart of another infrared image-guided image restoration model training method provided in an embodiment of the present application; Figure 3 A schematic diagram of a process of an image restoration method provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of a training device for an infrared image-guided image restoration model provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of an image restoration device provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0015] The following example describes in detail the training method of the infrared image-guided image restoration model of the present application.

[0016] Figure 1 A schematic flowchart of a method for training an infrared image-guided image restoration model provided in an embodiment of the present application.

[0017] like Figure 1 As shown, the training method of the infrared image guided image restoration model proposed in this embodiment specifically includes the following steps: S101. Acquire a sample data set, wherein each sample data in the sample data set includes a sample visible light normal image, a sample visible light degraded image, and a corresponding sample infrared image.

[0018] In the embodiment of the present application, a normal visible light image of a sample can be obtained. And the corresponding sample infrared image , the sample visible light normal image is degraded to obtain the sample visible light degraded image , the sample visible light normal image, the sample visible light degraded image and the corresponding sample infrared image are associated to obtain a sample data set.

[0019] It should be noted that by performing degradation processing on the sample visible light normal image, the complex degradation phenomena in the real environment (such as low light, haze, rain and snow, etc.) can be simulated.

[0020] It should be noted that the present application does not limit the specific method of performing degradation processing on the sample visible light normal image, which can be selected according to actual conditions.

[0021] Optionally, the sample normal visible light image may be input into a pre-trained image degradation model, and the image degradation model may perform degradation processing on the sample normal visible light image to obtain a sample degraded visible light image.

[0022] Optionally, the sample visible light normal image may be degraded by using an image processing tool to obtain a sample visible light degraded image.

[0023] In an embodiment of the present application, after obtaining the sample visible light degraded image, in order to facilitate subsequent model training, the sample visible light degraded image and the sample infrared image can be spatially aligned based on an image registration algorithm to obtain the final sample visible light degraded image.

[0024] S102. Construct an image restoration model to be trained for image restoration, wherein the image restoration model includes a generator and a discriminator to be trained.

[0025] It should be noted that infrared images have the ability to penetrate interference such as smoke and haze, and can capture details that visible light images cannot obtain. Research based on the fusion of infrared images and visible light images has made certain progress. In related technologies, most of them rely on traditional image registration and simple fusion operations, and cannot make full use of the information in infrared images for image restoration. In addition, due to the lack of high-quality visible light image information, the existing fusion of infrared images and visible light images often uses unsupervised training methods, or in Euclidean space, makes the fused image close to the degraded visible light image and infrared image at the same time, that is: the optimization goal makes the distance between the fused image and the degraded visible light image as small as possible, and the distance between the fused image and the infrared image as small as possible. This conflicting optimization goal causes some details in the fused image to be missing. Since infrared images obtain the large structure of the scene and lose details such as texture, while visible light images obtain the color and texture information of the scene, the two are complementary. Most of the existing visible light image and infrared image fusions treat the two equally, and are not designed from the perspective of one modality assisting the restoration of another modality.

[0026] In an embodiment of the present application, after constructing the image restoration model, the sample visible light degraded images and the corresponding sample infrared images can be used as training data, and the image restoration model can be trained based on the sample visible light degraded images and the corresponding sample infrared images.

[0027] The goal of the generator is to generate a repaired image that is as close as possible to the sample's normal visible light image.

[0028] Among them, the goal of the discriminator is to distinguish the generated repaired image from the sample visible light normal image.

[0029] Optionally, the structure of the discriminator can be a convolutional neural network (CNN). S103. Input the sample visible light degraded image and the corresponding sample infrared image into the generator to obtain a sample visible light repaired image, and input the sample visible light repaired image and the sample visible light normal image into the discriminator to obtain a first classification probability corresponding to the sample visible light repaired image and a second classification probability corresponding to the sample visible light normal image.

[0030] In the embodiment of the present application, after obtaining the sample visible light degraded image and the corresponding sample infrared image, the sample visible light degraded image and the corresponding sample infrared image can be input into the generator, and the generator processes the sample visible light degraded image and the corresponding sample infrared image to obtain the sample visible light repaired image. .

[0031] It should be noted that in order to enhance the realism and detail fidelity of the generated sample visible light restoration images, a discriminator is introduced on the basis of the generator to form a generative adversarial network (GAN) framework. Through adversarial training, the restoration effect of the generator is improved.

[0032] In the embodiment of the present application, after obtaining the sample visible light repaired image, the sample visible light repaired image and the sample visible light normal image can be input into the discriminator to obtain the first classification probability corresponding to the sample visible light repaired image. The second classification probability corresponding to the sample visible light normal image .

[0033] Among them, the first classification probability is the output probability of the discriminator for the sample visible light repaired image, and the second classification probability is the output probability of the discriminator for the sample visible light normal image.

[0034] S104. Adjust the model parameters of the image restoration model according to the sample visible light normal image, the sample visible light restoration image, the first classification probability and the second classification probability, and continue to train the adjusted image restoration model until the training is completed to obtain a target image restoration model, wherein the target image restoration model includes a trained target generator and a target discriminator.

[0035] In an embodiment of the present application, the total loss function of the image restoration model can be determined according to the sample visible light normal image, the sample visible light restoration image, the first classification probability and the second classification probability, and based on the total loss function, the model parameters of the image restoration model are adjusted, and the adjusted image restoration model is continued to be trained until the training is completed to obtain the target image restoration model.

[0036] Optionally, according to the total loss function, the model parameters of the image restoration model are adjusted until the loss function meets the training end condition, and the image restoration model after the last adjustment of the model parameters is determined as the trained target image restoration model.

[0037] It should be noted that the present application does not limit the setting of the training end condition, and it can be selected according to the actual situation.

[0038] Optionally, the training end condition can be set as the total loss function value being less than a preset loss threshold; optionally, the training end condition can be set as the number of adjustments to the parameters of the image restoration model reaching a preset number threshold.

[0039] The training method of the infrared image guided image restoration model provided in the present application is to obtain a sample data set, wherein each sample data in the sample data set includes a sample visible light degraded image and a corresponding sample infrared image, to construct an image restoration model to be trained for image restoration, wherein the image restoration model includes a generator and a discriminator to be trained, the sample visible light degraded image and the corresponding sample infrared image are input into the generator to obtain a sample visible light restoration image, and the sample visible light restoration image and the sample visible light normal image are input into the discriminator to obtain a first classification probability corresponding to the sample visible light restoration image and a first classification probability corresponding to the sample visible light normal image. Binary classification probability, according to the sample visible light normal image, the sample visible light repaired image, the first classification probability and the second classification probability, the model parameters of the image repair model are adjusted, and the adjusted image repair model is continued to be trained until the training is completed to obtain the target image repair model, wherein the target image repair model includes a trained target generator and a target discriminator. Therefore, by constructing an image repair model and training the image repair model based on the sample visible light degraded image and the corresponding sample infrared image, and performing image repair through the target image repair model, the structural information contained in the infrared image can be fully utilized, thereby improving the accuracy and reliability of image repair.

[0040] Figure 2 A schematic flowchart of a method for training an infrared image-guided image restoration model provided in an embodiment of the present application.

[0041] like Figure 2 As shown, the training method of the infrared image guided image restoration model proposed in this embodiment specifically includes the following steps: S201 . Acquire a sample data set, wherein each sample data in the sample data set includes a sample visible light normal image, a sample visible light degraded image, and a corresponding sample infrared image.

[0042] S202: construct an image restoration model to be trained for image restoration, wherein the image restoration model includes a generator and a discriminator to be trained.

[0043] The steps S201-S202 may be implemented in any of the implementation methods in the embodiments of the present application and will not be described in detail here.

[0044] S203, inputting the sample visible light degraded image and the corresponding sample infrared image into the generator, and using the encoder in the generator to extract features from the sample visible light degraded image and the sample infrared image to obtain target image features.

[0045] The generator includes an encoder and a decoder, for example, an encoder-decoder structure in the form of a U-Net.

[0046] It should be noted that the present disclosure does not limit the specific method of extracting features from sample visible light degraded images and sample infrared images to obtain target image features, and the method can be selected according to actual conditions.

[0047] Optionally, feature extraction may be performed on the sample visible light degraded image and the sample infrared image respectively to obtain a first image feature corresponding to the sample visible light degraded image and a second image feature corresponding to the sample infrared image, and feature fusion may be performed on the first image feature and the second image feature to obtain a target image feature.

[0048] Optionally, the sample visible light degradation image and the sample infrared image may be combined, and feature extraction may be performed on the combined image to obtain target image features.

[0049] S204: The decoder in the generator restores the target image features to obtain a sample visible light repaired image.

[0050] S205 . Input the sample visible light restoration image and the sample visible light normal image into a discriminator to obtain a first classification probability corresponding to the sample visible light restoration image and a second classification probability corresponding to the sample visible light normal image.

[0051] S206. Determine a content loss function according to the sample visible light restoration image and the sample visible light normal image.

[0052] Among them, the content loss function is used to directly measure the sample visible light restoration image and a sample visible light normal image The difference between them, using pixel-level L1 loss, the expression of the content loss function can be:

[0053] in, is the content loss function, Repair the image for visible light of the sample, is the normal image of the sample in visible light.

[0054] S207, obtaining the similarity between the sample visible light restoration image and the sample visible light normal image, and determining the perceptual loss function according to the similarity.

[0055] It should be noted that the structural similarity index (SSIM) can be used to measure the similarity between two images. and a sample visible light normal image The structural similarity of the images is in the range of [0,1]. The closer to 1, the more similar the images are. By minimizing the perceptual loss function , the model will generate images that are more consistent with structural features, and the expression of the perceptual loss function can be:

[0056] in, is the perceptual loss function, Repair the image for visible light of the sample, is the normal image of the sample in visible light.

[0057] S208. Determine an adversarial loss function according to the first classification probability and the second classification probability.

[0058] In the embodiment of the present application, the expression of the adversarial loss function can be:

[0059] in, To combat the loss function, is the second classification probability, is the first classification probability.

[0060] S209. Based on the content loss function, the perceptual loss function and the adversarial loss function, determine the total loss function of the image restoration model, and based on the total loss function, adjust the model parameters of the image restoration model, and continue to train the adjusted image restoration model until the training is completed to obtain the target image restoration model.

[0061] It should be noted that in order to optimize the image restoration model, the first weight factor corresponding to the content loss function, the second weight factor corresponding to the perceptual loss function, and the third weight factor corresponding to the adversarial loss function can be obtained. According to the content loss function and the first weight factor, the perceptual loss function and the second weight factor, and the adversarial loss function and the third weight factor, the total loss function of the image restoration model is determined, where the weight factor is used to control the importance of the loss function.

[0062] In the embodiment of the present application, the expression of the total loss function of the image restoration model can be:

[0063] in, is the total loss function, is the content loss function, is the first weight factor, is the perceptual loss function, is the second weight factor, To combat the loss function, is the third weighting factor.

[0064] In an embodiment of the present application, after obtaining the total loss function of the image restoration model, the model parameters of the image restoration model can be continuously adjusted through the back propagation algorithm, and the adjusted image restoration model can be continuously trained until the training is completed to obtain the target image restoration model.

[0065] In summary, the training method of the infrared image guided image restoration model provided in the present application constructs an image restoration model and takes the infrared image as the input of the image restoration model, thereby inputting the information of the infrared image into the image restoration model as an auxiliary feature of the visible light image. While the information of the infrared image assists in guidance, the corresponding normal visible light image is used as supervisory information. By continuously adjusting the model parameters of the image restoration model, the target image restoration model is obtained, which improves the generalization ability and restoration accuracy of the target image restoration model and solves the problem that visible light degraded images are difficult to restore in complex environments.

[0066] It should be noted that after obtaining the trained target image restoration model, image restoration can be performed based on the target image restoration model.

[0067] The image restoration method of the present application is described in detail below using embodiments.

[0068] Figure 3 A schematic diagram of the process flow of the image restoration method provided in an embodiment of the present application.

[0069] like Figure 3 As shown, the image restoration method proposed in this embodiment specifically includes the following steps: S301: Obtain a visible light degraded image to be restored and a corresponding infrared image.

[0070] It should be noted that the present application does not limit the specific method of obtaining the visible light degraded image to be repaired and the corresponding infrared image, which can be selected according to actual conditions.

[0071] Optionally, in a harsh environment, image acquisition can be performed by a visible light camera to obtain a visible light degraded image to be repaired, and image acquisition can be performed synchronously by an infrared camera to obtain a corresponding infrared image.

[0072] S302: input the visible light degraded image to be repaired and the corresponding infrared image into the target image repair model, and the target generator in the target image repair model processes the visible light degraded image to be repaired and the corresponding infrared image to obtain a target visible light repaired image.

[0073] Among them, the target image restoration model is a model trained using the above training method.

[0074] The image restoration method provided by the present application obtains a visible light degraded image to be restored and a corresponding infrared image, inputs the visible light degraded image to be restored and the corresponding infrared image into a target image restoration model, and processes the visible light degraded image to be restored and the corresponding infrared image by a target generator in the target image restoration model to obtain a target visible light restoration image. Therefore, the present application makes full use of the structural information contained in the infrared image, can better restore the degraded part in the visible light degraded image, improves the restoration accuracy of the visible light degraded image, can be applied to image restoration in a variety of complex and harsh environments, and has wide applicability.

[0075] In order to implement the above embodiment, this embodiment provides a training device for an image restoration model guided by an infrared image. Figure 4 A schematic diagram of the structure of a training device for an infrared image-guided image restoration model provided in an embodiment of the present application.

[0076] like Figure 4 As shown, the infrared image-guided image restoration model training device 1000 includes: a first acquisition module 110, a model building module 120, a second acquisition module 130 and a training module 140.

[0077] A first acquisition module 110 is used to acquire a sample data set, wherein each sample data in the sample data set includes a sample visible light normal image, a sample visible light degraded image and a corresponding sample infrared image; A model building module 120, used to build an image restoration model to be trained for image restoration, wherein the image restoration model includes a generator and a discriminator to be trained; A second acquisition module 130 is used to input the sample visible light degraded image and the corresponding sample infrared image into the generator to obtain a sample visible light repaired image, and input the sample visible light repaired image and the sample visible light normal image into a discriminator to obtain a first classification probability corresponding to the sample visible light repaired image and a second classification probability corresponding to the sample visible light normal image; The training module 140 is used to adjust the model parameters of the image restoration model according to the sample visible light normal image, the sample visible light restoration image, the first classification probability and the second classification probability, and continue to train the adjusted image restoration model until the training is completed to obtain a target image restoration model, wherein the target image restoration model includes a trained target generator and a target discriminator.

[0078] According to one embodiment of the present application, the first acquisition module 110 is further used to: acquire a sample visible light normal image and a corresponding sample infrared image, perform degradation processing on the sample visible light normal image to obtain a sample visible light degraded image; and associate the sample visible light normal image, the sample visible light degraded image and the corresponding sample infrared image to obtain the sample data set.

[0079] According to one embodiment of the present application, the first acquisition module 110 is further used to: input the sample visible light normal image into a pre-trained image degradation model, and use the image degradation model to perform degradation processing on the sample visible light normal image to obtain the sample visible light degraded image.

[0080] According to an embodiment of the present application, the first acquisition module 110 is further used to: spatially align the sample visible light degraded image and the sample infrared image based on an image registration algorithm to obtain a final sample visible light degraded image.

[0081] According to one embodiment of the present application, the second acquisition module 130 is further used to: the encoder in the generator performs feature extraction on the sample visible light degraded image and the sample infrared image to obtain target image features; and the decoder in the generator performs restoration on the target image features to obtain the sample visible light repaired image.

[0082] According to one embodiment of the present application, the training module 140 is also used to: determine a content loss function based on the sample visible light restoration image and the sample visible light normal image; obtain the similarity between the sample visible light restoration image and the sample visible light normal image, and determine a perceptual loss function based on the similarity; determine an adversarial loss function based on the first classification probability and the second classification probability; determine a total loss function of the image restoration model based on the content loss function, the perceptual loss function and the adversarial loss function, and adjust the model parameters of the image restoration model based on the total loss function.

[0083] The training device of the infrared image guided image restoration model provided by the present application obtains a sample data set, wherein each sample data in the sample data set includes a sample visible light normal image, a sample visible light degraded image and a corresponding sample infrared image, and constructs an image restoration model to be trained for image restoration, wherein the image restoration model includes a generator and a discriminator to be trained, the sample visible light degraded image and the corresponding sample infrared image are input into the generator to obtain a sample visible light restoration image, and the sample visible light restoration image and the sample visible light normal image are input into the discriminator to obtain a first classification probability corresponding to the sample visible light restoration image and a first classification probability corresponding to the sample visible light normal image. According to the sample visible light normal image, the sample visible light repaired image, the first classification probability and the second classification probability, the model parameters of the image repair model are adjusted, and the adjusted image repair model is continued to be trained until the training is completed to obtain the target image repair model, wherein the target image repair model includes a trained target generator and a target discriminator. Therefore, by constructing an image repair model and training the image repair model based on the sample visible light degraded image and the corresponding sample infrared image, and performing image repair through the target image repair model, the structural information contained in the infrared image can be fully utilized, thereby improving the accuracy and reliability of image repair.

[0084] In order to implement the above embodiment, this embodiment provides an image restoration device. Figure 5 A schematic diagram of the structure of an image restoration device provided in an embodiment of the present application.

[0085] like Figure 5 As shown, the image restoration device 2000 includes: an acquisition module 210 and a restoration module 220.

[0086] An acquisition module 210 is used to acquire a visible light degraded image to be restored and a corresponding infrared image; The restoration module 220 is used to input the visible light degraded image to be restored and the corresponding infrared image into a target image restoration model, and the target generator in the target image restoration model processes the visible light degraded image to be restored and the corresponding infrared image to obtain a target visible light restoration image, wherein the target image restoration model is a model trained using the training method described in the first aspect.

[0087] The image restoration device provided by the present application obtains a visible light degraded image to be restored and a corresponding infrared image, inputs the visible light degraded image to be restored and the corresponding infrared image to a target image restoration model, and processes the visible light degraded image to be restored and the corresponding infrared image by a target generator in the target image restoration model to obtain a target visible light restoration image. Therefore, the present application makes full use of the structural information contained in the infrared image, can better restore the degraded part in the visible light degraded image, improves the restoration accuracy of the visible light degraded image, can be applied to image restoration in a variety of complex and harsh environments, and has wide applicability.

[0088] In order to implement the above embodiment, the present application also proposes an electronic device 3000, such as Figure 6 As shown, it includes: a memory 310, a processor 320, and a computer program stored in the memory 310 and executable on the processor 320. When the processor executes the program, the training method of the infrared image-guided image restoration model as described in the first aspect or the image restoration method as described in the second aspect is implemented.

[0089] In order to implement the above embodiments, the present application proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the training method of the infrared image-guided image restoration model described in the first aspect or the image restoration method described in the second aspect.

[0090] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which, when executed by a processor, implements the training method of the infrared image-guided image restoration model described in the first aspect or the image restoration method described in the second aspect.

[0091] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0092] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A training method for an infrared image-guided image restoration model, characterized in that: The method comprises: Acquire a sample data set, wherein each sample data in the sample data set includes a sample visible light normal image, a sample visible light degraded image, and a corresponding sample infrared image; Constructing an image restoration model to be trained for image restoration, wherein the image restoration model includes a generator and a discriminator to be trained; Inputting the sample visible light degraded image and the corresponding sample infrared image into the generator to obtain a sample visible light repaired image, and inputting the sample visible light repaired image and the sample visible light normal image into the discriminator to obtain a first classification probability corresponding to the sample visible light repaired image and a second classification probability corresponding to the sample visible light normal image; According to the sample visible light normal image, the sample visible light repaired image, the first classification probability and the second classification probability, the model parameters of the image repair model are adjusted, and the adjusted image repair model is continued to be trained until the training is completed to obtain a target image repair model, wherein the target image repair model includes a trained target generator and a target discriminator.

2. The method according to claim 1, characterized in that The obtaining of the sample data set comprises: Acquire a sample visible light normal image and a corresponding sample infrared image, and perform degradation processing on the sample visible light normal image to obtain a sample visible light degraded image; The sample visible light normal image, the sample visible light degraded image and the corresponding sample infrared image are associated to obtain the sample data set.

3. The method according to claim 2, characterized in that The step of performing degradation processing on the sample visible light normal image to obtain the sample visible light degraded image includes: The sample visible light normal image is input into a pre-trained image degradation model, and the image degradation model performs degradation processing on the sample visible light normal image to obtain the sample visible light degraded image.

4. The method according to claim 3, characterized in that The method further comprises: Based on an image registration algorithm, the sample visible light degraded image and the sample infrared image are spatially aligned to obtain a final sample visible light degraded image.

5. The method according to claim 1, characterized in that The step of inputting the sample visible light degraded image and the sample infrared image into the generator to obtain a sample visible light repaired image comprises: The encoder in the generator extracts features from the sample visible light degraded image and the sample infrared image to obtain target image features; The decoder in the generator restores the target image features to obtain the sample visible light repaired image.

6. The method according to claim 1, characterized in that The adjusting the model parameters of the image restoration model according to the sample visible light normal image, the sample visible light restoration image, the first classification probability and the second classification probability includes: Determining a content loss function according to the sample visible light restoration image and the sample visible light normal image; Obtaining the similarity between the sample visible light restoration image and the sample visible light normal image, and determining a perceptual loss function according to the similarity; Determining an adversarial loss function according to the first classification probability and the second classification probability; Based on the content loss function, the perceptual loss function and the adversarial loss function, a total loss function of the image restoration model is determined, and based on the total loss function, model parameters of the image restoration model are adjusted.

7. An image restoration method, characterized in that: The method comprises: Acquire a visible light degraded image to be restored and a corresponding infrared image; The visible light degraded image to be repaired and the corresponding infrared image are input into a target image repair model, and the target generator in the target image repair model processes the visible light degraded image to be repaired and the corresponding infrared image to obtain a target visible light repaired image, wherein the target image repair model is a model trained by the training method described in any one of claims 1 to 6.

8. A training device for an infrared image-guided image restoration model, characterized in that: The device comprises: A first acquisition module is used to acquire a sample data set, wherein each sample data in the sample data set includes a sample visible light normal image, a sample visible light degraded image and a corresponding sample infrared image; A model building module, used to build an image restoration model to be trained for image restoration, wherein the image restoration model includes a generator and a discriminator to be trained; a second acquisition module, configured to input the sample visible light degraded image and the corresponding sample infrared image into the generator to obtain a sample visible light repaired image, and input the sample visible light repaired image and the sample visible light normal image into a discriminator to obtain a first classification probability corresponding to the sample visible light repaired image and a second classification probability corresponding to the sample visible light normal image; A training module is used to adjust the model parameters of the image restoration model according to the sample visible light normal image, the sample visible light restoration image, the first classification probability and the second classification probability, and continue to train the adjusted image restoration model until the training is completed to obtain a target image restoration model, wherein the target image restoration model includes a trained target generator and a target discriminator.

9. An image restoration device, characterized in that: The device comprises: An acquisition module, used for acquiring a visible light degraded image to be restored and a corresponding infrared image; A restoration module is used to input the visible light degraded image to be restored and the corresponding infrared image into a target image restoration model, and a target generator in the target image restoration model processes the visible light degraded image to be restored and the corresponding infrared image to obtain a target visible light restoration image, wherein the target image restoration model is a model trained by the training method according to any one of claims 1 to 6.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 or claim 7 is implemented.

Images