Model training method and device, computer device, storage medium and program product

Abstract

The application relates to a model training method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: acquiring a sample low-resolution image, a sample visible light image and a sample high-resolution image; inputting the sample low-resolution image into a first adversarial generator of a first adversarial network in an image conversion model to generate a pseudo visible light image; inputting the pseudo visible light image into a first auxiliary generator to obtain a first pseudo low-resolution image; training the first adversarial network by using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image and the sample visible light image; and training a second adversarial network in the image conversion model by using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image. The image conversion model trained by the method can convert a low-resolution image into a clear high-resolution image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model training method and device, computer equipment, a storage medium and a program product. BACKGROUND

[0002] The power system occupies a crucial position in social production and daily life. Multi-perspective and multi-period inspection of the power system helps us monitor the operation state of the power equipment, helps us accurately understand the operation and maintenance of each link of the power system, and facilitates timely maintenance by the operation and maintenance personnel. In addition to daytime image acquisition by a camera on a drone to obtain images, the power system also needs to be inspected at night by an infrared imager through infrared imaging technology to obtain images of the power system.

[0003] The images obtained by the drone or the infrared imaging technology are low-resolution images with low resolution, which need to be processed. The high-resolution images obtained by using a traditional model to process the low-resolution images are not clear enough, and therefore need to be improved. SUMMARY

[0004] Therefore, it is necessary to provide a model training method and device, computer equipment, a storage medium and a program product to solve the above technical problems.

[0005] In a first aspect, the present application provides a model training method. The method comprises:

[0006] obtaining a sample low-resolution image, a sample visible light image and a sample high-resolution image;

[0007] inputting the sample low-resolution image into a first adversarial generator of a first adversarial network in an image conversion model to generate a pseudo visible light image;

[0008] inputting the pseudo visible light image into a first auxiliary generator to obtain a first pseudo low-resolution image;

[0009] training the first adversarial network using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image and the sample visible light image;

[0010] training a second adversarial network in the image conversion model using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image.

[0011] In one embodiment, the training of the first adversarial network using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image and the sample visible light image comprises:

[0012] The first adversarial discriminator of the first adversarial network is trained by using the pseudo visible light image and the sample visible light image.

[0013] The first adversarial generator of the first adversarial network is trained by using the first pseudo low-resolution image and the sample low-resolution image.

[0014] In one embodiment, the training of the first adversarial generator by using the first pseudo low-resolution image and the sample low-resolution image comprises:

[0015] The first adversarial generator is trained according to the sample low-resolution image based on a loss function.

[0016] The norm loss is determined according to the first pseudo low-resolution image and the sample low-resolution image.

[0017] The trained first adversarial generator is optimized by using the norm loss.

[0018] In one embodiment, the training of the second adversarial network in the image conversion model by using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image comprises:

[0019] The sample low-resolution image and the pseudo visible light image are fused to obtain a fused image.

[0020] The fused image is input into a second adversarial generator of the second adversarial network in the image conversion model to generate a pseudo high-resolution image.

[0021] The second adversarial network is trained by using the pseudo high-resolution image and the sample high-resolution image.

[0022] In one embodiment, the method further comprises:

[0023] The pseudo high-resolution image is input into a second auxiliary generator to obtain a second pseudo low-resolution image.

[0024] The rationality of the second adversarial generator of the trained second adversarial network is verified according to the second pseudo low-resolution image and the sample low-resolution image.

[0025] In one embodiment, the first adversarial generator of the first adversarial network is a variational autoencoder, and the second adversarial generator of the second adversarial network is a residual network.

[0026] In a second aspect, the application further provides an optimization device of a target detection model. The device comprises:

[0027] An image acquisition module is configured to acquire a sample low-resolution image, a sample visible light image, and a sample high-resolution image.

[0028] A first generation module is configured to input the sample low-resolution image into a first adversarial generator of a first adversarial network in an image conversion model to generate a pseudo visible light image.

[0029] A second generation module is configured to input the pseudo visible light image into a first auxiliary generator to obtain a first pseudo low-resolution image.

[0030] A first training module is configured to train the first adversarial network by using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image, and the sample visible light image.

[0031] A second training module is configured to train a second adversarial network in the image conversion model by using the sample low-resolution image, the pseudo visible light image, and the sample high-resolution image.

[0032] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor. The memory stores a computer program. The processor implements the following steps when executing the computer program:

[0033] acquiring a sample low-resolution image, a sample visible light image, and a sample high-resolution image;

[0034] inputting the sample low-resolution image into a first adversarial generator of a first adversarial network in an image conversion model to generate a pseudo visible light image;

[0035] inputting the pseudo visible light image into a first auxiliary generator to obtain a first pseudo low-resolution image;

[0036] training the first adversarial network by using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image, and the sample visible light image;

[0037] training a second adversarial network in the image conversion model by using the sample low-resolution image, the pseudo visible light image, and the sample high-resolution image.

[0038] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the following steps:

[0039] acquiring a sample low-resolution image, a sample visible light image, and a sample high-resolution image;

[0040] inputting the sample low-resolution image into a first adversarial generator of a first adversarial network in the image conversion model to generate a pseudo visible light image;

[0041] inputting the pseudo visible light image into a first auxiliary generator to obtain a first pseudo low-resolution image;

[0042] training the first adversarial network by using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image and the sample visible light image;

[0043] training a second adversarial network in the image conversion model by using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image.

[0044] In a fifth aspect, the present application further provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the following steps:

[0045] obtaining a sample low-resolution image, a sample visible light image and a sample high-resolution image;

[0046] inputting the sample low-resolution image into a first adversarial generator of a first adversarial network in the image conversion model to generate a pseudo visible light image;

[0047] inputting the pseudo visible light image into a first auxiliary generator to obtain a first pseudo low-resolution image;

[0048] training the first adversarial network by using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image and the sample visible light image;

[0049] training a second adversarial network in the image conversion model by using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image.

[0050] The above model training method, device, computer device, storage medium and program product, by inputting the sample low-resolution image into the first adversarial generator of the first adversarial network, generating the pseudo visible light image, and inputting the pseudo visible light image into the first auxiliary generator to obtain the first pseudo resolution image. The first adversarial network is trained by using the pseudo visible light image, the first pseudo resolution image, the sample low-resolution image and the sample visible light image; the second adversarial network in the image conversion model is trained by using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image. The image conversion model trained by this training can convert the low-resolution image into a clear high-resolution image. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1A flow chart of a model training method in one embodiment;

[0052] Figure 2 A flow chart of training the first GAN in one embodiment;

[0053] Figure 3 A flow chart of training the second GAN in one embodiment;

[0054] Figure 4 A flow chart of a model training method in another embodiment

[0055] Figure 5 A structural block diagram of a model training apparatus in one embodiment;

[0056] Figure 6 A structural block diagram of a model training apparatus in another embodiment;

[0057] Figure 7 A structural block diagram of a model training apparatus in yet another embodiment;

[0058] Figure 8 An internal structural diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0060] In one embodiment, as shown in FIG. 1, a model training method is provided, which can be applied to a terminal or a server. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers. Taking the case that the method is applied to the server for example, the method specifically includes the following steps: Figure 1

[0061] S101, obtaining a sample low-resolution image, a sample visible light image and a sample high-resolution image.

[0062] Specifically, the sample image set can be obtained by collecting images of samples through image collection devices on unmanned aerial vehicles or tower camera. Each sample image set can include multiple sample groups, each sample group including a sample low-resolution image, a sample visible light image and a sample high-resolution image; the three sample images included in each sample group correspond to the same scene.

[0063] ​S102, input the sample low-resolution image into a first adversarial generator of a first adversarial network in the image conversion model to generate a pseudo visible light image.

[0064] In the embodiment, the image conversion model is a model for converting a low-resolution image into a high-resolution model; optionally, the image conversion model comprises two adversarial networks; wherein the first adversarial network is used to convert a low-resolution image into a visible light image.

[0065] Specifically, the sample low-resolution image is input into the first adversarial generator of the first adversarial network, and the first adversarial generator generates a pseudo visible light image based on the pixel probability distribution of the sample low-resolution image. Optionally, each sample low-resolution image in each sample group corresponds to a pseudo visible light image.

[0066] S103, input the pseudo visible light image into the first auxiliary generator to obtain a first pseudo low-resolution image.

[0067] In the embodiment, the first auxiliary generator is used to assist in training the first adversarial network, and further used to assist in training the first adversarial generator in the first adversarial network.

[0068] Specifically, the first adversarial discriminator of the first adversarial network can be used to distinguish the pseudo visible light image corresponding to each sample group from the sample visible light image in each sample group, and obtain the pseudo visible light image distinguished by the first adversarial discriminator and having a large difference from the sample visible light image; the obtained pseudo visible light image is input into the first auxiliary generator, and the first auxiliary generator generates a first pseudo low-resolution image based on the pixel probability distribution of the pseudo visible light image.

[0069] S104, the first adversarial network is trained using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image and the sample visible light image.

[0070] Specifically, according to the pre-set loss function, the training loss is determined based on the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image and the sample visible light image, and the first adversarial network is trained using the training loss.

[0071] S105, the second adversarial network in the image conversion model is trained using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image.

[0072] Optionally, the second adversarial network in the image conversion model is configured to convert the visible light image into a high-resolution image. The second adversarial network comprises a second adversarial generator and a second adversarial discriminator. Further, the first adversarial generator and the second adversarial generator are different. Optionally, the first adversarial generator of the first adversarial network is a variational autoencoder. The second adversarial generator of the second adversarial network is a residual network. The variational autoencoder is a generative network structure based on variational Bayesian inference. Unlike the traditional autoencoder which describes the latent space in a numerical manner, it describes the observation of the latent space in a probabilistic manner. The residual network is easy to optimize and can improve accuracy by increasing the depth. The residual block inside uses a skip connection to alleviate the gradient vanishing problem caused by increasing the depth of the deep neural network.

[0073] Specifically, based on the pre-set loss function, the sample low-resolution image, the pseudo visible light image and the sample high-resolution image, the training loss is determined to train the second adversarial network in the image conversion model.

[0074] It can be understood that after the first adversarial network and the second adversarial network are trained, a trained image conversion model is obtained. Then, the real-time collected target low-resolution image can be input into the trained image conversion model to obtain a cleaned high-resolution image.

[0075] The above model training method generates a pseudo visible light image by inputting the sample low-resolution image into the first adversarial generator of the first adversarial network, and obtains a first pseudo-resolution image by inputting the pseudo visible light image into the first auxiliary generator. The first adversarial network is trained by using the pseudo visible light image, the first pseudo-resolution image, the sample low-resolution image and the sample visible light image. The second adversarial network in the image conversion model is trained by using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image. The image conversion model trained by this training method can convert a low-resolution image into a clear high-resolution image.

[0076] In one embodiment, step S104 is further refined and can specifically include the following steps:

[0077] S201, the first adversarial discriminator of the first adversarial network is trained by using the pseudo visible light image and the sample visible light image.

[0078] Specifically, based on the pre-set loss function, the training loss is determined according to the pseudo visible light image and the sample visible light image; and the first adversarial discriminator is trained by using the training loss.

[0079] The loss function can be:

[0080]

[0081] wherein, log1 is close to 0, x is a sample visible light image, and logD1(x) is close to 0; in represents a pseudo visible light image, and the true value is close to 1, because 1-1=0, infinitely approaches a negative number, that is, the first adversarial discriminator optimizes in the direction of loss.

[0082] S202, the first pseudo low-resolution image and the sample low-resolution image are used to train the first adversarial generator.

[0083] Optionally, the first adversarial generator can be trained based on a loss function according to the sample low-resolution image; the norm loss is determined according to the first pseudo low-resolution image and the sample low-resolution image; and the trained first adversarial generator is optimized using the norm loss. Wherein, the definition of the loss function is as follows:

[0084]

[0085] wherein, KL represents divergence, x represents a sample low-resolution image, z represents sampling, pn is a prior distribution of an image, q v is a parameterized distribution used to approximate the real situation, generally replaced by a normal distribution, and L represents the number of sampling points.

[0086] Specifically, the sample low-resolution image can be substituted into the loss function, and the first adversarial generator is optimized with the purpose of minimizing the loss function; then, based on the norm loss function (for example, the L1 norm loss function), the norm loss is determined according to the first pseudo low-resolution image and the sample low-resolution image; and the network parameters of the trained first adversarial generator are adjusted using the norm loss.

[0087] It can be understood that in the embodiment, the first auxiliary generator is introduced to optimize the first adversarial generator, so that the image output by the trained first adversarial generator is closer to the input.

[0088] In the embodiment, the first adversarial discriminator of the first adversarial network is trained by the pseudo visible light image and the sample visible light image, and the first adversarial generator is trained by the first pseudo low-resolution image and the sample low-resolution image, and a specific method for training the first adversarial network is given.

[0089] In one of the embodiments, step S105 is further refined, as shown in Figure 3 specifically, the following steps can be included:

[0090] S301, the sample low-resolution image and the pseudo visible light image are fused to obtain a fused image. ​

[0091] Specifically, when the sample low-resolution image and the pseudo visible light image are fused, the texture of the fused image can be adjusted based on the sample low-resolution image, so that the fused image contains more texture information. The obtained fused image has the same brightness as the sample low-resolution image and has the same gradient as the pseudo visible light image.

[0092] wherein the fused image is expressed as:

[0093]

[0094] wherein, represents the base layer image;

[0095] represents the detail layer image.

[0096] The texture information of the fused image is as follows:

[0097]

[0098] Specifically, I V represents the pseudo visible light image, b represents the label of the sample low-resolution image (1 for a real image and 0 for no annotation), D(I v ) represents the recognition result of I V .

[0099] wherein the content loss of the fused image is as follows:

[0100]

[0101] wherein, H and W represent the height and width of the input image, ||·||F represents the matrix norm, I f is the infrared image, I i is the visible light image, and Δ is the derivative symbol, which means taking the gradient of the image.

[0102] S302, input the fused image into the second adversarial generator of the second adversarial network of the image conversion model to generate a pseudo high-resolution image.

[0103] Optionally, the second adversarial generator is mainly used for reconstructing the resolution of the fused image, and then the given low-resolution fused image is input into the second adversarial generator, and the second adversarial generator restores the corresponding pseudo high-resolution image based on the related algorithm.

[0104] S303, using the pseudo high-resolution image and the sample high-resolution image, training the second adversarial network.

[0105] Specifically, the second adversarial discriminator in the second adversarial network can be trained by using the pseudo high-resolution image and the sample high-resolution image in combination with a loss function shown in the following formula (6).

[0106]

[0107] wherein, V, C represent the channel size and the number of feature spectrum respectively, and ||·||F represents the Frobenius norm. HR denotes the sample high-resolution image, I SR denotes the pseudo high-resolution image.

[0108] Further, the second adversarial generator in the second adversarial network of the embodiment can be a pre-trained residual network and can be directly used.

[0109] Optionally, the pseudo high-resolution image can be input to a second auxiliary generator to obtain a second pseudo low-resolution image, and the rationality of the trained second adversarial generator in the second adversarial network can be verified according to the second pseudo low-resolution image and the sample low-resolution image. The second auxiliary generator and the second adversarial generator are in an inverted network relationship, have the same network parameters, and only differ in the moment of gradient parameter update.

[0110] Specifically, the rationality of the second adversarial generator can be determined by comparing the similarity of the second pseudo low-resolution image and the sample low-resolution image. If the similarity of the second pseudo low-resolution image and the sample low-resolution image is greater than a threshold, it indicates that the second adversarial generator is not reasonable, and if the similarity of the second pseudo low-resolution image and the sample low-resolution image is less than the threshold, it indicates that the second adversarial generator is reasonable. Further, the second adversarial generator can be re-optimized in the case that the second adversarial generator is not reasonable.

[0111] In the embodiment, by fusing the sample low-resolution image and the pseudo visible light image to obtain a fusion image, and inputting the fusion image to the second adversarial generator in the second adversarial network of the image conversion model to generate a pseudo high-resolution image, and then using the pseudo high-resolution image and the sample high-resolution image to train the second adversarial network, a specific method for training the second adversarial network is given.

[0112] In one embodiment, as Figure 4 shown, a preferred example of a model training method is provided, and the specific implementation process can include:

[0113] S401, obtaining a sample low-resolution image, a sample visible light image and a sample high-resolution image.

[0114] S402, inputting the sample low-resolution image to the first adversarial generator in the first adversarial network of the image conversion model to generate a pseudo visible light image.

[0115] S403, input the pseudo visible light image to the first auxiliary generator to obtain a first pseudo low resolution image.

[0116] S404, training the first adversarial discriminator of the first adversarial network by using the pseudo visible light image and the sample visible light image.

[0117] S405, training the first adversarial generator by using the first pseudo low resolution image and the sample low resolution image.

[0118] S406, fusing the sample low resolution image and the pseudo visible light image to obtain a fusion image.

[0119] S407, inputting the fusion image to the second adversarial generator of the second adversarial network in the image conversion model to generate a pseudo high resolution image.

[0120] S408, training the second adversarial network by using the pseudo high resolution image and the sample high resolution image.

[0121] It should be understood that, although each step in the flowchart involved in the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiments of the present application also provide an optimization device of a target detection model for implementing the optimization method of the target detection model as described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more target detection model optimization device embodiments provided below can refer to the limitations of the target detection model optimization method in the above text, which will not be repeated here.

[0123] In one embodiment, as shown in the figure, a model training device 1 is provided, such as Figure 5 , comprising an image acquisition module 10, a first generation module 20, a second generation module 30, a first training module 40, and a second training module 50, wherein:

[0124] The image acquisition module 10 is configured to acquire a sample low-resolution image, a sample visible light image, and a sample high-resolution image.

[0125] The first generation module 20 is configured to input the sample low-resolution image into a first adversarial generator of a first adversarial network in the image conversion model to generate a pseudo visible light image.

[0126] The second generation module 30 is configured to input the pseudo visible light image into a first auxiliary generator to obtain a first pseudo low-resolution image.

[0127] The first training module 40 is configured to train the first adversarial network by using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image, and the sample visible light image.

[0128] The second training module 50 is configured to train a second adversarial network in the image conversion model by using the sample low-resolution image, the pseudo visible light image, and the sample high-resolution image.

[0129] In one of the embodiments, as shown in Figure 6 The first training module 40 further includes the following units:

[0130] The discriminator training unit 41 is configured to train a first adversarial discriminator of the first adversarial network by using the pseudo visible light image and the sample visible light image.

[0131] The generator training unit 42 is configured to train the first adversarial generator by using the first pseudo low-resolution image and the sample low-resolution image.

[0132] In one of the embodiments, the generator training unit 42 is further configured to:

[0133] train the first adversarial generator according to the sample low-resolution image based on a loss function, and determine a norm loss according to the first pseudo low-resolution image and the sample low-resolution image.

[0134] In one of the embodiments, the second training module 50 is refined based on Figure 5 or Figure 6 For example, the second training module 50 is refined based on Figure 5 As shown in Figure 7 The second training module 50 further includes the following units:

[0135] The image fusion unit 51 is configured to fuse the sample low-resolution image and the pseudo visible light image to obtain a fused image.

[0136] The high-resolution generation unit 52 is configured to input the fused image into a second adversarial generator of a second adversarial network in the image conversion model to generate a pseudo high-resolution image.

[0137] The second training unit 53 is configured to train the second generative adversarial network by using the pseudo high-resolution image and the sample high-resolution image.

[0138] In one of the embodiments, the model training apparatus 1 further comprises the following modules:

[0139] The third generation module is configured to input the pseudo high-resolution image into the second auxiliary generator to obtain a second pseudo low-resolution image.

[0140] The verification module is configured to verify the rationality of the second generative adversarial network of the trained second generative adversarial network according to the second pseudo low-resolution image and the sample low-resolution image.

[0141] In one of the embodiments, the first generative adversarial network is a variational autoencoder, and the second generative adversarial network is a residual network.

[0142] The modules in the model training apparatus can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of the processor of the computer device in hardware form, or stored in the memory of the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the modules.

[0143] In one of the embodiments, a computer device is provided, which can be a server. The internal structure diagram of the computer device can be as shown in FIG. 8. Figure 8 The computer device comprises a processor, a memory, and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store the initial height of the to-be-tested bellows, the height, temperature, and performance change rate of the standard bellows, and other related data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a model training method.

[0144] Those skilled in the art can understand that Figure 8 The structure shown in FIG. 8 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. Specifically, the computer device can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0145] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the following steps when executing the computer program:

[0146] obtaining a sample low-resolution image, a sample visible light image and a sample high-resolution image;

[0147] inputting the sample low-resolution image into a first adversarial generator of a first adversarial network in the image conversion model to generate a pseudo visible light image;

[0148] inputting the pseudo visible light image into a first auxiliary generator to obtain a first pseudo low-resolution image;

[0149] training the first adversarial network using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image and the sample visible light image;

[0150] training a second adversarial network in the image conversion model using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image.

[0151] In one embodiment, the processor, when executing the logic in the computer program for training the first adversarial network using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image and the sample visible light image, further implements the following steps:

[0152] training a first adversarial discriminator of the first adversarial network using the pseudo visible light image and the sample visible light image; and training the first adversarial generator using the first pseudo low-resolution image and the sample low-resolution image.

[0153] In one embodiment, the processor, when executing the logic in the computer program for training the first adversarial generator using the first pseudo low-resolution image and the sample low-resolution image, further implements the following steps:

[0154] training the first adversarial generator based on a loss function according to the sample low-resolution image; determining a norm loss according to the first pseudo low-resolution image and the sample low-resolution image; and optimizing the trained first adversarial generator using the norm loss.

[0155] In one embodiment, the computer program, when executing the logic for training the second adversarial network in the image conversion model using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image, further implements the following steps:

[0156] fusing the sample low-resolution image and the pseudo visible light image to obtain a fused image; inputting the fused image into a second adversarial generator of a second adversarial network in the image conversion model to generate a pseudo high-resolution image; and training the second adversarial network using the pseudo high-resolution image and the sample high-resolution image.

[0157] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0158] inputting the pseudo high-resolution image into the second auxiliary generator to obtain a second pseudo low-resolution image; and verifying rationality of the second adversarial generator of the second adversarial network according to the second pseudo low-resolution image and the sample low-resolution image.

[0159] In one embodiment, the first adversarial generator of the first adversarial network is a variational autoencoder, and the second adversarial generator of the second adversarial network is a residual network.

[0160] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. When the computer program is executed by the processor, the following steps are implemented:

[0161] obtaining a sample low-resolution image, a sample visible light image and a sample high-resolution image;

[0162] inputting the sample low-resolution image into the first adversarial generator of the first adversarial network in the image conversion model to generate a pseudo visible light image;

[0163] inputting the pseudo visible light image into the first auxiliary generator to obtain a first pseudo low-resolution image;

[0164] training the first adversarial network by using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image and the sample visible light image;

[0165] training the second adversarial network in the image conversion model by using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image.

[0166] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0167] training the first adversarial discriminator of the first adversarial network by using the pseudo visible light image and the sample visible light image; and training the first adversarial generator by using the first pseudo low-resolution image and the sample low-resolution image.

[0168] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0169] The first generative adversarial network is trained according to the sample low-resolution image based on a loss function; a norm loss is determined according to the first pseudo low-resolution image and the sample low-resolution image; and the trained first generative adversarial network is optimized by using the norm loss.

[0170] In one embodiment, the logic for training the second generative adversarial network in the image conversion model using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image, when executed by the processor, further implements the following steps:

[0171] The sample low-resolution image and the pseudo visible light image are fused to obtain a fused image; the fused image is input into a second generative adversarial network of the second generative adversarial network in the image conversion model to generate a pseudo high-resolution image; and the second generative adversarial network is trained by using the pseudo high-resolution image and the sample high-resolution image.

[0172] In one embodiment, the logic in the computer program, when executed by the processor, further implements the following steps:

[0173] The pseudo high-resolution image is input into the second auxiliary generator to obtain a second pseudo low-resolution image; and the trained second generative adversarial network of the second generative adversarial network is verified for rationality according to the second pseudo low-resolution image and the sample low-resolution image.

[0174] In one embodiment, the first generative adversarial network of the first generative adversarial network is a variational autoencoder; and the second generative adversarial network of the second generative adversarial network is a residual network.

[0175] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the following steps:

[0176] A sample low-resolution image, a sample visible light image and a sample high-resolution image are obtained;

[0177] The sample low-resolution image is input into a first generative adversarial network of the first generative adversarial network in the image conversion model to generate a pseudo visible light image;

[0178] The pseudo visible light image is input into a first auxiliary generator to obtain a first pseudo low-resolution image;

[0179] The first generative adversarial network is trained by using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image and the sample visible light image;

[0180] The second generative adversarial network in the image conversion model is trained by using the sample low-resolution image, the pseudo visible light image and the sample high-resolution image.

[0181] In one embodiment, the logic of training the first GAN in the computer program is further to cause the processor to perform the following steps when executed:

[0182] The first discriminator of the first GAN is trained using the fake visible image and the sample visible image; and the first generator of the first GAN is trained using the first fake low resolution image and the sample low resolution image.

[0183] In one embodiment, the logic of training the first generator in the computer program is further to cause the processor to perform the following steps when executed:

[0184] The first generator is trained based on the loss function and the sample low resolution image; a norm loss is determined based on the first fake low resolution image and the sample low resolution image; and the trained first generator is optimized using the norm loss.

[0185] In one embodiment, the logic of training the second GAN in the computer program is further to cause the processor to perform the following steps when executed:

[0186] The sample low resolution image and the fake visible image are fused to obtain a fused image; the fused image is input to the second generator of the second GAN of the image conversion model to generate a fake high resolution image; and the second GAN is trained using the fake high resolution image and the sample high resolution image.

[0187] In one embodiment, the logic of training the second GAN in the computer program is further to cause the processor to perform the following steps when executed:

[0188] The fake high resolution image is input to the second auxiliary generator to obtain a second fake low resolution image; and the trained second generator of the second GAN is verified based on the second fake low resolution image and the sample low resolution image.

[0189] In one embodiment, the first generator of the first GAN is a variational autoencoder; and the second generator of the second GAN is a residual network.

[0190] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0191] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0192] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A model training method, characterized in that, The method includes: Acquire low-resolution images, visible light images, and high-resolution images of the samples; The sample low-resolution image is input into the first adversarial generator of the first adversarial network in the image conversion model to generate a pseudo visible light image. The pseudo visible light image is input into the first auxiliary generator to obtain the first pseudo low-resolution image; The first adversarial network is trained using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image, and the sample visible light image. The low-resolution image of the sample and the pseudo-visible light image are fused to obtain a fused image; The fused image is input into the second adversarial generator of the second adversarial network in the image conversion model to generate a pseudo high-resolution image; The second adversarial network is trained using the pseudo high-resolution image and the sample high-resolution image.

2. The method according to claim 1, characterized in that, The step of training the first adversarial network using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image, and the sample visible light image includes: The first adversarial discriminator of the first adversarial network is trained using the pseudo visible light image and the sample visible light image. The first adversarial generator is trained using the first pseudo low-resolution image and the sample low-resolution image.

3. The method according to claim 2, characterized in that, The step of training the first adversarial generator using the first pseudo-low-resolution image and the sample low-resolution image includes: Based on the loss function, the first adversarial generator is trained using the sample low-resolution image; Determine the norm loss based on the first pseudo low-resolution image and the sample low-resolution image; The trained first adversarial generator is optimized using the norm loss.

4. The method according to claim 1, characterized in that, The method further includes: The pseudo high-resolution image is input into the second auxiliary generator to obtain the second pseudo low-resolution image; The rationality of the second adversarial generator of the trained second adversarial network is verified based on the second pseudo low-resolution image and the sample low-resolution image.

5. The method according to any one of claims 1-4, characterized in that, The first adversarial generator of the first adversarial network is a variational autoencoder; the second adversarial generator of the second adversarial network is a residual network.

6. The method according to claim 4, characterized in that, The second auxiliary generator and the second adversarial generator are in a network inversion relationship.

7. A model training device, characterized in that, The device includes the following modules: The image acquisition module is used to acquire low-resolution images, visible light images, and high-resolution images of the samples. The first generation module is used to input the sample low-resolution image into the first adversarial generator of the first adversarial network in the image conversion model to generate a pseudo visible light image. The second generation module is used to input the pseudo visible light image into the first auxiliary generator to obtain the first pseudo low-resolution image. The first training module is used to train the first adversarial network using the pseudo visible light image, the first pseudo low-resolution image, the sample low-resolution image, and the sample visible light image. The second training module is used to fuse the sample low-resolution image and the pseudo visible light image to obtain a fused image; input the fused image into the second adversarial generator of the second adversarial network in the image conversion model to generate a pseudo high-resolution image; and use the pseudo high-resolution image and the sample high-resolution image to train the second adversarial network.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images