Generative adversarial network training method, image brightness enhancement method, device and medium

Abstract

The present invention discloses a generative adversarial network training method, an image brightness enhancement method, a computer device and a storage medium. The generative adversarial network training method includes the steps of obtaining a training set and executing a training process. The image brightness enhancement method in this embodiment utilizes the characteristics of image color constancy, utilizes the property that the same object has the same reflection ability to light of different wavelengths, constrains the consistency of the reflection components of the same group of low-brightness images and the reflection components of high-brightness images, uses a discriminator to learn the mapping of incident components of different brightness, thereby mapping the incident components of the low-brightness image to the incident components of the high-brightness image, and then combining them with the low-brightness reflection components to obtain the final calibrated high-brightness image. The image brightness enhancement method in this embodiment uses very few prior assumptions, so it can effectively avoid image deviations caused by prior assumption estimation deviations. The present invention is widely used in the field of image processing technology.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a generative adversarial network training method, an image brightness enhancement method, a computer device and a storage medium. Background Art

[0002] When performing image processing, image brightness enhancement is usually involved to address problems such as low contrast and underexposure in the image. When using existing technologies to enhance image brightness, the noise in the image is amplified along with the brightness, resulting in obvious noise points in the image after brightness enhancement, which affects the image quality.

[0003] Terminology explanation:

[0004] Retinex theory: An image optical theory inspired by the human visual system. This theory holds that a captured image can be decomposed into two parts, namely the reflection component and the incident component (also known as the illumination component). Let S represent the image, then it can be decomposed into Where R represents the reflected component and I represents the incident component. The human eye can automatically remove the influence of different light sources and different reflected spectra under different light sources, and recognize the color R of the object itself, while the image S obtained by the camera will vary with the different illumination I. Summary of the invention

[0005] In response to at least one of the above technical problems, an object of the present invention is to provide a generative adversarial network training method, an image brightness enhancement method, a computer device and a storage medium.

[0006] On the one hand, an embodiment of the present invention includes a generative adversarial network training method, wherein the generative adversarial network includes a brightness enhancement network, a first discriminator and a second discriminator, and the generative adversarial network training method includes:

[0007] Acquire a training set; the training set includes at least one first image and at least one second image; the brightness of the second image is higher than the brightness of the first image; the first image can be decomposed into a first incident image layer and a first reflected image layer, and the second image can be decomposed into a second incident image layer and a second reflected image layer;

[0008] Execute a training process for the generative adversarial network; during the training process, use the first incident image layer as the input of the brightness enhancement network, use the second reflected image layer and the output of the brightness enhancement network as the input of the first discriminator, use the output of the brightness enhancement network as the expected output of the first discriminator, use the synthesis result of the output of the brightness enhancement network and the first reflected image layer and the second image as the input of the second discriminator, and use the synthesis result of the output of the brightness enhancement network and the first reflected image layer as the expected output of the second discriminator.

[0009] Furthermore, during the training process, parameters of the enhanced network, parameters of the first discriminator, and parameters of the second discriminator are updated asynchronously.

[0010] Further, the asynchronously updating the parameters of the first discriminator and the parameters of the second discriminator includes:

[0011] Alternatingly executing a first updating process and a second updating process;

[0012] In the first updating process, the parameters of the enhanced network are fixed, the parameters of the first discriminator are updated according to the input of the first discriminator and the expected output of the first discriminator, and the parameters of the second discriminator are updated according to the input of the second discriminator and the expected output of the second discriminator;

[0013] In the second updating process, the parameters of the first discriminator and the parameters of the second discriminator are fixed, and the parameters of the enhanced network are updated according to the input of the first discriminator and the expected output of the first discriminator and the input of the second discriminator and the expected output of the second discriminator.

[0014] Furthermore, the stopping condition of the training process is that the loss function of the generative adversarial network reaches a first threshold; the loss function includes a first term, which is a linear combination of the L1 norm of the first image and the L1 norm of the second image.

[0015] Furthermore, the loss function also includes a second term, which is an L1 norm determined by the first reflected image layer and the second reflected image layer.

[0016] Furthermore, the loss function also includes a third term, and the third term is determined by the incident image layer loss and the total loss.

[0017] Furthermore, the generative adversarial network also includes a first decomposition network and a second decomposition network, the first decomposition network is used to decompose the first image into the first incident image layer and the first reflected image layer, and the second decomposition network is used to decompose the second image into the second incident image layer and the second reflected image layer.

[0018] On the other hand, an embodiment of the present invention further includes a method for enhancing image brightness, comprising:

[0019] acquiring a third image;

[0020] Inputting the third image into a generative adversarial network; the generative adversarial network is trained by the training method in this embodiment;

[0021] Receive a fourth image output by the generative adversarial network.

[0022] On the other hand, an embodiment of the present invention further includes a computer device, including a memory and a processor, wherein the memory is used to store at least one program, and the processor is used to load the at least one program to execute the method described in the embodiment.

[0023] On the other hand, an embodiment of the present invention further includes a storage medium storing a program executable by a processor, wherein the program executable by the processor is used to execute the method described in the embodiment when executed by the processor.

[0024] The beneficial effects of the present invention are as follows: the image brightness enhancement method in the present embodiment uses a trained generative adversarial network to perform image brightness enhancement, utilizes the characteristics of image color constancy, utilizes the property that the same object has consistent reflection ability for light of different wavelengths, constrains the consistency of the reflection components of the same group of low-brightness images and the reflection components of high-brightness images, uses a discriminator to learn the mapping of incident components of different brightnesses, thereby mapping the incident components of the low-brightness image to the incident components of the high-brightness image, and then combining them with the low-brightness reflection components to obtain the final calibrated high-brightness image. The image brightness enhancement method in the present embodiment uses very few prior assumptions, and therefore can effectively avoid image deviations caused by prior assumption estimation biases. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a method for generating adversarial network training in an embodiment;

[0026] Figure 2 A diagram showing the structure and working principle of a generative adversarial network used in the embodiments;

[0027] Figure 3 FIG. 4 is a flow chart of the method for enhancing image brightness in an embodiment. DETAILED DESCRIPTION

[0028] In this embodiment, a generative adversarial network training method is provided. The generative adversarial network used includes a brightness enhancement network, a first discriminator and a second discriminator. Figure 1 , the training method for generating adversarial networks includes the following steps:

[0029] P1. Get the training set;

[0030] P2. Execute a training process for the generative adversarial network.

[0031] In this embodiment, when executing step P1, first prepare t1 first images and t2 second images Thus forming a collection of images We can set t1:t2 = 1:N, that is, the number of the second images is an integer multiple of the number of the first images. Each first image and second image in the image set is normalized and processed. 60% of the images are randomly selected to form the training set, and the remaining 40% of the images form the test set. The ratio of the number of first images to the second images in the training set and the test set is 1:N.

[0032] In this embodiment, the brightness of any second image is higher than the brightness of any first image, the brightness of each second image may be the same or different, and the brightness of each first image may also be the same or different.

[0033] In this embodiment, according to the Retinex theory, each first image is decomposed into a corresponding first incident image layer and a first reflected image layer, and each second image is decomposed into a corresponding second incident image layer and a second reflected image layer. Decompose into the first incident image layer and the first reflective image layer The second image Decompose into the second incident image layer and the second reflective image layer

[0034] The decomposition of the first image and the second image can be performed by obtaining an image set. When completed, or the acquired image collection Each of the first image and the second image in has itself been decomposed into a corresponding first incident image layer, a first reflected image layer, a second incident image layer, and a second reflected image layer.

[0035] The decomposition of the first image and the second image can also be completed by the generative adversarial network. To achieve the decomposition function of the first image and the second image, the generative adversarial network also includes a first decomposition network and a second decomposition network, wherein the first decomposition network is used to decompose the first image into a first incident image layer and a first reflected image layer, and the second decomposition network is used to decompose the second image into a second incident image layer and a second reflected image layer.

[0036] In this embodiment, the structure and principle of the generative adversarial network used are as follows: Figure 2 As shown. The training process of the generative adversarial network is achieved by asynchronously updating the parameters of the first discriminator and the parameters of the second discriminator. The training process of the generative adversarial network includes a first update process and a second update process that are executed alternately in multiple cycles, that is, the second update process is executed once after the first update process is executed once, and the first update process is executed once after the second update process is executed once. Therefore, the training process of the generative adversarial network is the first update process, the second update process, the first update process, the second update process..., or the second update process, the first update process, the second update process, the first update process...

[0037] The first update process is described by taking a first update process as an example. In the first update process, the parameters of the enhancement network are fixed. Specifically, the parameters of the enhancement network are determined by the second update process that was last executed before the first update process. In the first update process, a first image S obtained from the training set is L After being input into the generative adversarial network, it is decomposed into the first incident image layer I by the first decomposition network L and the first reflective image layer R L , the first incident image layer I L The enhanced network increases the brightness, that is, the output of the enhanced network With the first incident image layer I L There is a difference in brightness compared to the second image S obtained from the training set. N After being input into the generative adversarial network, it is decomposed into the second incident image layer I by the second decomposition network N and the second reflective image layer R N . Enhance the output of the network and the second incident image layer I N is input to the first discriminator, which enhances the output of the network and the second incident image layer I N The judgment is made based on which image layer is obtained from the original high-brightness image. N From the second image S with higher brightness N , so the second incident image layer IN Set as the expected output of the first discriminator, and adjust the parameters of the first discriminator by comparing the actual output of the first discriminator with the expected output. The direction and size of the adjustment can be determined by the gradient of the loss function. With the first reflective image layer R L The result is obtained by superposition synthesis Synthesis results and the second image S N is input to the second discriminator, which classifies the synthesis result and the second image S N The judgment criterion is which image is the original high-brightness image. Since the second image S N It has a larger size than the first image S L High brightness, and the synthetic result The incident component The first incident image layer I is processed by the enhancement network L The brightness is increased, so the second image S N Set as the expected output of the second discriminator, and adjust the parameters of the second discriminator by comparing the actual output of the second discriminator with the expected output. The direction and size of the adjustment can be determined by the gradient of the loss function. After completing the parameter adjustment of the first discriminator and the parameter adjustment of the second discriminator, the first update process is completed.

[0038] The second update process is described by taking a second update process as an example. In this second update process, the parameters of the first discriminator and the parameters of the second discriminator are fixed. Specifically, the parameters of the first discriminator and the parameters of the second discriminator are determined by the first update process that was last executed before the second update process was executed. In this second update process, the parameters of the enhanced network are adjusted according to the distance between the actual output of the first discriminator and the expected output and the distance between the actual output of the second discriminator and the expected output. The direction and size of the adjustment can be determined by the gradient of the loss function. After completing the parameter adjustment of the enhanced network, the second update process is completed.

[0039] In this embodiment, the training process of the generative adversarial network can be quantified by the loss function of the generative adversarial network, specifically, the value of the loss function is determined after completing a first update process or a second update process, and when the value of the loss function is less than or equal to a small first threshold, the training process of the generative adversarial network is stopped. The stopping condition of the training process can also be set to the number of executions of the first update process reaching a fixed value, the number of executions of the second update process reaching a fixed value, or the sum of the number of executions of the first update process and the second update process reaching a fixed value.

[0040] In this embodiment, the loss function L includes the first term L r , the second term L c and the third term L GAN , specifically L = L r +L c +L GAN .

[0041] In this embodiment, L r L is a fidelity term used to ensure that I and R decomposed from image S can be reconstructed back to S. r is defined as a linear combination of the L1 norm of the first image and the L1 norm of the second image, specifically:

[0042]

[0043] where λ L and λ N is the weight.

[0044] In this embodiment, a linear combination of L1 norms is used to determine the fidelity term L r , which can enhance the robustness of training.

[0045] In this embodiment, L c is the consistency loss function, specifically L c =‖R L -R N ‖1, that is, the first reflection image layer R L and the second reflective image layer R N According to the color constancy theory, the reflectivity of an object should remain consistent, and the reflective layers decomposed from an image under different illumination should remain consistent. Therefore, by observing the first reflective image layer R L and the second reflective image layer R N The size of the determined L1 norm is used to determine the consistency loss, which can better reflect the performance of the enhanced network.

[0046] In this embodiment, L GAN To combat the loss, specifically, L GAN =L GAN1 +L GAN2 .in:

[0047]

[0048]

[0049] L GAN1 is the incident image layer loss, L GAN2 is the total loss. represents discriminator 1, Represents discriminator 2. Since the first discriminator is used to determine whether its input is the second image S N The separated second incident image layer I N , or by the first image S L The separated first incident image layer I L The second discriminator is used to determine whether its input is a real second image S N Or by the first incident image layer I L After enhanced processing Therefore, the adversarial loss determined by the incident image layer loss and the total loss can better reflect the performance of the first discriminator and the second discriminator.

[0050] In summary, including the first item L r , the second term L c and the third term L GAN The loss function L can better reflect the performance of the enhanced network, the first discriminator and the second discriminator in the generative adversarial network, thereby providing a reference for the parameter adjustment direction of the enhanced network, the first discriminator and the second discriminator, obtaining a better training speed and making the training more robust.

[0051] The generative adversarial network trained by the training method of this embodiment learns the mapping features between the incident component of the first image with lower brightness to the incident component of the second image with higher brightness, and has the ability to enhance the brightness information of the low-brightness image while keeping the reflected component unchanged, becoming an end-to-end deep learning model, that is, by inputting a low-brightness image into the trained generative adversarial network, the generative adversarial network can output a corresponding image with increased brightness.

[0052] In this embodiment, a generative adversarial network trained by the training method of this embodiment is used to perform an image brightness enhancement method. Figure 3 , the image brightness enhancement method comprises the following steps:

[0053] S1. Acquire a third image;

[0054] S2. inputting the third image into the generative adversarial network;

[0055] S3. Receive a fourth image output by the generative adversarial network.

[0056] In this embodiment, after executing step S2, the generative adversarial network decomposes the third image into corresponding incident image layers and reflected image layers through the first decomposition network, wherein the incident image layer is brightness-increased by the enhancement network and then superimposed with the reflected image layer to obtain a fourth image for output, and the brightness of the output fourth image is higher than that of the third image, thereby achieving image brightness enhancement.

[0057] The image brightness enhancement method in this embodiment uses a trained generative adversarial network to perform image brightness enhancement. Specifically, the incident image layer of the input image is brightness enhanced and then superimposed with the reflected image layer of the input image to obtain an output image. The image brightness enhancement method in this embodiment utilizes the characteristics of image color constancy and the property that the same object has the same reflection ability for light of different wavelengths. It constrains the consistency of the reflection components of the same group of low-brightness images and the reflection components of high-brightness images, and uses a discriminator to learn the mapping of incident components of different brightness, thereby mapping the incident component of the low-brightness image to the incident component of the high-brightness image, and then combining it with the low-brightness reflection component to obtain the final calibrated high-brightness image. The image brightness enhancement method in this embodiment uses very few prior assumptions, so it can effectively avoid image deviations caused by prior assumption estimation biases.

[0058] In this embodiment, a computer device includes a memory and a processor, the memory is used to store at least one program, and the processor is used to load the at least one program to execute the generative adversarial network training method or image brightness enhancement method in the embodiment to achieve the same technical effect as described in the embodiment.

[0059] In this embodiment, a storage medium stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to execute the generative adversarial network training method or image brightness enhancement method in the embodiment to achieve the same technical effect as described in the embodiment.

[0060] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it may be directly fixed or connected to another feature, or it may be indirectly fixed or connected to another feature. In addition, the descriptions of up, down, left, right, etc. used in this disclosure are only relative to the relative positional relationship of the components of the present disclosure in the accompanying drawings. The singular forms of "a", "said" and "the" used in this disclosure are also intended to include the plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by technicians in this technical field. The terms used in the specification of this embodiment are only for describing specific embodiments, not for limiting the present invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.

[0061] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0062] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0063] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed jointly on one or more processors, by hardware or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0064] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or a portion thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described in this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0065] The computer program can be applied to input data to perform the functions described in the present embodiment, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.

[0066] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and / or implementation method may have various modifications and changes.

Claims

1. A generative adversarial network training method, wherein the generative adversarial network comprises a brightness enhancement network, a first discriminator and a second discriminator, characterized in that: The generative adversarial network training method comprises: Acquire a training set; the training set includes at least one first image and at least one second image; the brightness of the second image is higher than the brightness of the first image; the first image can be decomposed into a first incident image layer and a first reflected image layer, and the second image can be decomposed into a second incident image layer and a second reflected image layer; Execute a training process for the generative adversarial network; during the training process, use the first incident image layer as the input of the brightness enhancement network, use the second incident image layer and the output of the brightness enhancement network as the input of the first discriminator, use the second incident image layer as the expected output of the first discriminator, use the output of the brightness enhancement network and the synthesis result of the first reflected image layer and the second image as the input of the second discriminator, and use the second image as the expected output of the second discriminator.

2. The generative adversarial network training method according to claim 1, characterized in that: During the training process, parameters of the brightness enhancement network, parameters of the first discriminator, and parameters of the second discriminator are asynchronously updated.

3. The generative adversarial network training method according to claim 2, characterized in that: The asynchronously updating the parameters of the brightness enhancement network, the parameters of the first discriminator, and the parameters of the second discriminator includes: Alternatingly executing a first updating process and a second updating process; In the first updating process, the parameters of the brightness enhancement network are fixed, the parameters of the first discriminator are updated according to the input of the first discriminator and the expected output of the first discriminator, and the parameters of the second discriminator are updated according to the input of the second discriminator and the expected output of the second discriminator; In the second updating process, the parameters of the first discriminator and the parameters of the second discriminator are fixed, and the parameters of the brightness enhancement network are updated according to the input of the first discriminator and the expected output of the first discriminator and the input of the second discriminator and the expected output of the second discriminator.

4. The generative adversarial network training method according to claim 1, characterized in that: The stopping condition of the training process is that the loss function of the generative adversarial network reaches a first threshold; the loss function includes a first term, which is the loss of the first image. norm and the second image Linear combination of norms.

5. The generative adversarial network training method according to claim 4, characterized in that: The loss function also includes a second term, which is determined by the first reflected image layer and the second reflected image layer. Norm.

6. The generative adversarial network training method according to claim 4 or 5, characterized in that: The loss function further includes a third term, which is determined by an incident image layer loss and a total loss, wherein the incident image layer loss is determined based on the first incident image layer and the second incident image layer.

7. The generative adversarial network training method according to any one of claims 1 to 5, characterized in that: The generative adversarial network also includes a first decomposition network and a second decomposition network, the first decomposition network is used to decompose the first image into the first incident image layer and the first reflected image layer, and the second decomposition network is used to decompose the second image into the second incident image layer and the second reflected image layer.

8. A method for enhancing image brightness, characterized in that: include: acquiring a third image; Inputting the third image into a generative adversarial network; The generative adversarial network is trained by the method according to any one of claims 1 to 7; Receive a fourth image output by the generative adversarial network.

9. A computer device, characterized in that The invention comprises a memory and a processor, wherein the memory is used to store at least one program, and the processor is used to load the at least one program to execute the method according to any one of claims 1 to 8.

10. A storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 8 when executed by the processor.

Images