Training method of image processing model, and image processing method and device

Abstract

The invention provides a training method of an image processing model and an image processing method and device.The training method comprises the steps that different simulated dazzle light images and low-illumination images in a low-illumination image library are combined, and complex illumination images corresponding to the low-illumination images are synthesized; each low-illumination image corresponds to a pseudo normal illumination image after illumination enhancement; jointly processing the complex illumination image through a dazzle removal sub-model and an illumination enhancement sub-model of an image processing model to obtain a first illumination enhancement image with image illumination enhancement and dazzle removal; and the image processing model determines different types of losses of the image processing model based on an output result of the dazzle-eliminating sub-model and a low-illumination image and an output result of the illumination enhancement sub-model and a pseudo normal illumination image, and jointly optimizes the image processing model. And the trained image processing model is obtained to carry out image illumination enhancement and dazzle light degradation restoration on the image at the same time, so that the overall quality of image illumination enhancement and dazzle light restoration is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing model training method, an image processing method and a device. Background Art

[0002] Image enhancement under complex lighting conditions has gradually become a research direction that has attracted much attention. Among them, uneven lighting and glare are two prominent problems faced in the practical application of visual algorithms, which limit the processing effect of existing technologies. First, uneven lighting causes objects in the image to have significantly different brightness and contrast, making the edges and details of the objects difficult to identify. Secondly, glare is a visual artifact caused by reflection and scattering inside the camera due to strong light sources. It may block the image content and cause loss of image details, while affecting the perception and credibility of image analysis.

[0003] Existing technologies cannot cope with the problem of image quality degradation caused by complex lighting environments. For example, image illumination enhancement algorithms will also enhance glare areas, while glare removal algorithms cannot enhance low-illuminance areas. Directly combining the two methods will result in visual artifacts, which may further lead to image degradation in severe cases. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide an image processing model training method, an image processing method and a device, which can simultaneously perform lighting enhancement and glare repair on an image, thereby improving the overall image quality after complex lighting image repair.

[0005] An embodiment of the present application provides a training method for an image processing model, the training method comprising:

[0006] Combining different simulated glare images with low-illumination images in a low-illumination image library to synthesize a complex illumination image corresponding to the low-illumination image; wherein each low-illumination image corresponds to a pseudo-normal illumination image after illumination enhancement;

[0007] Inputting the synthesized complex illumination image into an image processing model, and jointly processing the complex illumination image through a glare removal sub-model and an illumination enhancement sub-model of the image processing model to obtain a first illumination enhanced image with enhanced image illumination and glare removal;

[0008] The image processing model determines different types of losses of the image processing model based on the output result of the de-glare sub-model and the low-illuminance image, the output result of the illumination enhancement sub-model and the pseudo-normal illumination image;

[0009] The image processing model is jointly optimized based on different types of losses of the image processing model until the image processing model meets the first training completion condition, thereby obtaining a trained image processing model; the trained image processing model is used to simultaneously perform image illumination enhancement and glare degradation repair on the image.

[0010] In some embodiments, the image processing model training method further includes:

[0011] The low illumination images in the low illumination image library are processed based on the trained illumination enhancement mentor model to generate pseudo normal illumination images corresponding to the low illumination images.

[0012] In some embodiments, in the training method of the image processing model, the trained illumination enhancement mentor model is obtained by training a generative adversarial network based on unpaired low illumination images and normal illumination images.

[0013] In some embodiments, in the training method of the image processing model, the trained illumination enhancement tutor model is obtained by training based on the following steps:

[0014] Randomly sample low-illumination images and normal-illumination images from a low-illumination image library and a normal-illumination image library;

[0015] The generator of the illumination enhancement mentor model processes the low illumination image to obtain a second illumination enhancement image, and sends the second illumination enhancement image and the normal illumination image to the discriminator of the illumination enhancement mentor model to distinguish true from false and calculate the discrimination loss;

[0016] The illumination enhancement mentor model calculates a first perceptual loss between the low illumination image and a second illumination enhanced image;

[0017] The illumination enhancement tutor model is optimized based on the first perceptual loss and the discriminative loss until the illumination enhancement tutor model meets a second training completion condition.

[0018] In some embodiments, in the training method of the image processing model, the glare removal sub-model and illumination enhancement sub-model of the image processing model are cascaded front and back, and the glare removal sub-model and illumination enhancement sub-model have the same structure and share encoder weights.

[0019] In some embodiments, in the training method of the image processing model, the synthesized complex illumination image is input into the image processing model, and the complex illumination image is jointly processed by the deglare sub-model and the illumination enhancement sub-model of the image processing model to obtain a first illumination enhanced image with enhanced image illumination and deglare, including:

[0020] Inputting the synthesized complex illumination image into the image processing model, and processing the complex illumination image through the deglare sub-model of the image processing model to obtain a first deglare image;

[0021] The first de-glare image is input into the illumination enhancement sub-model of the image processing model, and the first de-glare image is processed by the illumination enhancement sub-model to obtain a first illumination enhanced image.

[0022] In some embodiments, in the training method of the image processing model, combining different simulated glare images with low-illumination images in a low-illumination image library to synthesize a complex illumination image corresponding to the low-illumination image includes:

[0023] Randomly sample a low-light image from a low-light image library;

[0024] Randomly combine simulated glare images in the glare component library;

[0025] The simulated glare image, the low illumination image and the simulated noise are combined to synthesize a complex illumination image corresponding to the low illumination image.

[0026] In some embodiments, in the training method of the image processing model, the image processing model determines different types of losses of the image processing model based on the output result of the deglare sub-model and the low-illuminance image, the output result of the illumination enhancement sub-model and the pseudo-normal illumination image; including:

[0027] Determining a reconstruction loss of an image processing model based on a first comparison result between a first deglare image output by the deglare sub-model and a low illumination image, and a first comparison result between a first illumination enhanced image output by the illumination enhancement sub-model and a pseudo normal illumination image;

[0028] Based on the second comparison results of the first de-glare image features output by the de-glare sub-model at different neural layers and the low-illuminance image, and the second comparison results of the first illumination enhancement image features output by the illumination enhancement sub-model at different neural layers and the pseudo-normal illumination image, the second perceptual loss of the image processing model is determined.

[0029] In some embodiments, an image processing method is further provided, the method comprising:

[0030] Acquire an image to be processed for glare repair and image illumination enhancement;

[0031] The image to be processed is input into a trained image processing model, and the image to be processed is jointly processed by the deglare sub-model and the illumination enhancement sub-model of the trained image processing model to obtain a processed target image.

[0032] In some embodiments, in the image processing method, the step of inputting the image to be processed into a trained image processing model, and jointly processing the image to be processed by the deglare sub-model and the illumination enhancement sub-model of the trained image processing model to obtain a processed target image includes:

[0033] Inputting the image to be processed into a trained image processing model, processing the image to be processed by the deglare sub-model of the trained image processing model to obtain a second deglare image, and processing the second deglare image by the illumination enhancement sub-model of the trained image processing model to obtain a third illumination enhanced image;

[0034] The trained image processing model outputs the third illumination enhanced image as a processed target image.

[0035] In some embodiments, in the image processing method, the trained image processing model includes a glare removal sub-model and an illumination enhancement sub-model, the glare removal sub-model and the illumination enhancement sub-model are cascaded front and back, the glare removal sub-model and the illumination enhancement sub-model have the same structure and share encoder weights.

[0036] In some embodiments, a training device for an image processing model is also provided, the training device comprising:

[0037] A synthesis module, used for combining different simulated glare images with low illumination images in a low illumination image library to synthesize a complex illumination image corresponding to the low illumination image; wherein each low illumination image corresponds to a pseudo normal illumination image;

[0038] A processing module, used for inputting the synthesized complex illumination image into the image processing model, and jointly processing the complex illumination image through the deglare sub-model and the illumination enhancement sub-model of the image processing model to obtain a first illumination enhanced image with enhanced image illumination and deglare;

[0039] A determination module, used for the image processing model to determine different types of losses of the image processing model based on the output result of the de-glare sub-model and the low-illuminance image, the output result of the illumination enhancement sub-model and the pseudo-normal illumination image;

[0040] An optimization module is used to jointly optimize the image processing model based on different types of losses of the image processing model until the image processing model meets the first training completion condition, thereby obtaining a trained image processing model; the trained image processing model is used to simultaneously perform image illumination enhancement and glare degradation repair on the image.

[0041] In some embodiments, an image processing device is further provided, the device comprising:

[0042] An acquisition module, used for acquiring an image to be processed for glare repair and image illumination enhancement;

[0043] The joint processing module is used to input the image to be processed into the trained image processing model, and jointly process the image to be processed through the deglare sub-model and illumination enhancement sub-model of the trained image processing model to obtain a processed target image.

[0044] In some embodiments, an electronic device is also provided, including: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the training method of the image processing model are executed, or the steps of the image processing method are executed.

[0045] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the training method of the image processing model or the steps of the image processing method are executed.

[0046] In an embodiment of the present application, a training method for an image processing model, an image processing method and an apparatus are provided. The training method for the image processing model combines different simulated glare images and low-illuminance images in a low-illuminance image library to synthesize a complex illumination image corresponding to the low-illuminance image; wherein each low-illuminance image corresponds to a pseudo-normal illumination image; the synthesized complex illumination image is input into the image processing model, and the complex illumination image is jointly processed by a de-glare sub-model and an illumination enhancement sub-model of the image processing model to obtain a first illumination-enhanced image with enhanced image illumination and de-glare; the image processing model determines different types of losses of the image processing model based on the output result of the de-glare sub-model and the low-illuminance image, the output result of the illumination enhancement sub-model and the pseudo-normal illumination image; the image processing model is jointly optimized based on the different types of losses of the image processing model until the image processing model meets the first training completion condition to obtain a trained image processing model; the trained image An image processing model is used to simultaneously perform image illumination enhancement and glare degradation repair on an image; the training method described in the embodiment of the present application regards illumination enhancement and de-glare as related tasks, and the de-glare sub-model and the illumination enhancement sub-model of the image processing model jointly process complex illumination images, and the de-glare sub-model and the illumination enhancement sub-model interact with the model domain knowledge, and the de-glare sub-model cooperates with the illumination enhancement sub-model to process the glare degradation area to be consistent with the texture structure of other low-illuminance areas to be enhanced, and the illumination enhancement sub-model will not re-enhance the area after glare repair, effectively restore the image details of the dark or overexposed area, avoid the problem of the illumination enhancement method strengthening the glare effect, improve the overall visual effect of the image, and make the processed image clearer and more natural; further, during training, unpaired low-illuminance images are processed to obtain corresponding paired pseudo-normal illumination images and complex illumination images, which solves the problem that it is difficult to collect a large number of paired complex illumination images-normal illumination images in reality, and greatly reduces the model training cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flowchart of a method for training an image processing model according to an embodiment of the present application is shown;

[0049] Figure 2 A flow chart of a method for training an illumination enhancement tutor model according to an embodiment of the present application is shown;

[0050] Figure 3 Another method flow chart of training the illumination enhancement tutor model according to an embodiment of the present application is shown;

[0051] Figure 4 A flow chart of a method for synthesizing a complex illumination image corresponding to a low illumination image according to an embodiment of the present application is shown;

[0052] Figure 5 A schematic diagram of generating a simulated glare image according to an embodiment of the present application is shown;

[0053] Figure 6 A flow chart of a method for jointly processing the complex illumination image by using the deglare sub-model and the illumination enhancement sub-model of the image processing model according to an embodiment of the present application is shown;

[0054] Figure 7 A schematic diagram of the image processing model described in the embodiment of the present application is shown;

[0055] Figure 8 A flow chart of a method for determining different types of losses of an image processing model according to an embodiment of the present application is shown;

[0056] Fig. 9 A schematic diagram showing a training method for an image processing model according to an embodiment of the present application is shown;

[0057] Fig.10 A flowchart of the image processing method according to an embodiment of the present application is shown;

[0058] Fig.11 A schematic diagram of the structure of a training device for an image processing model according to an embodiment of the present application is shown;

[0059] Fig.12 A schematic diagram of the structure of the image processing device according to an embodiment of the present application is shown;

[0060] Fig.13 A schematic structural diagram of an electronic device described in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0061] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0062] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0063] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0064] In recent years, with the rapid development of computer vision and image processing technology, image enhancement under complex lighting conditions has gradually become a research direction that has attracted much attention. In practical applications, lighting conditions are unpredictable, which can easily lead to a decline in image quality and even the loss of key texture information. Improving image quality in complex lighting environments is of great significance to security fields such as railway all-weather security and automobile autonomous driving. Uneven lighting and glare are two prominent problems faced in the practical application of visual algorithms, which limit the processing effect of existing technologies. First, uneven lighting causes objects in the image to have significantly different brightness and contrast, making the edges and details of the objects difficult to identify. Secondly, glare is a visual artifact caused by internal reflection and scattering of the camera due to strong light sources, which may block the image content and cause the loss of image details, while affecting the perception and credibility of image analysis.

[0065] Existing technologies cannot cope with the problem of image quality degradation caused by complex lighting environments. For example, image illumination enhancement algorithms will also enhance glare areas, while glare removal algorithms cannot enhance low-illuminance areas. Directly combining the two methods will result in visual artifacts, which may further lead to image degradation in severe cases.

[0066] Based on this, in an embodiment of the present application, a training method, an image processing method and an apparatus for an image processing model are provided. The training method of the image processing model combines different simulated glare images and low-illuminance images in a low-illuminance image library to synthesize a complex illumination image corresponding to the low-illuminance image; wherein each low-illuminance image corresponds to a pseudo-normal illumination image; the synthesized complex illumination image is input into the image processing model, and the complex illumination image is jointly processed by the de-glare sub-model and the illumination enhancement sub-model of the image processing model to obtain a first illumination-enhanced image with enhanced image illumination and de-glare; the image processing model determines different types of losses of the image processing model based on the output result of the de-glare sub-model and the low-illuminance image, the output result of the illumination enhancement sub-model and the pseudo-normal illumination image; the image processing model is jointly optimized based on the different types of losses of the image processing model until the image processing model meets the first training completion condition to obtain a trained image processing model; the trained image processing model is An image processing model is provided for simultaneously performing image illumination enhancement and glare degradation repair on an image; the training method described in the embodiment of the present application regards illumination enhancement and de-glare as related tasks, the de-glare sub-model and the illumination enhancement sub-model of the image processing model jointly process complex illumination images, the de-glare sub-model and the illumination enhancement sub-model interact with the model domain knowledge, the de-glare sub-model cooperates with the illumination enhancement sub-model to process the glare degradation area to be consistent with the texture structure of other low illumination areas to be enhanced, and the illumination enhancement sub-model will not re-enhance the area after glare repair, effectively restore the image details of the dark or overexposed area, avoid the problem of the illumination enhancement method strengthening the glare effect, improve the overall visual effect of the image, and make the processed image clearer and more natural; further, during training, unpaired low illumination images are processed to obtain corresponding paired pseudo-normal illumination images and complex illumination images, which solves the problem that it is difficult to collect a large number of paired complex illumination images-normal illumination images in reality, and greatly reduces the model training cost.

[0067] Please refer to Figure 1 , Figure 1 A flow chart of the training method of the image processing model described in the embodiment of the present application is shown; Figure 1 As shown, the training method includes the following steps S101-S104:

[0068] S101, combining different simulated glare images with low illumination images in a low illumination image library to synthesize a complex illumination image corresponding to the low illumination image; wherein each low illumination image corresponds to a pseudo normal illumination image;

[0069] S102, inputting the synthesized complex illumination image into an image processing model, and jointly processing the complex illumination image through a glare removal sub-model and an illumination enhancement sub-model of the image processing model to obtain a first illumination enhanced image with enhanced image illumination and glare removal;

[0070] S103, the image processing model determines different types of losses of the image processing model based on the output result of the glare removal sub-model and the low illumination image, the output result of the illumination enhancement sub-model and the pseudo normal illumination image;

[0071] S104. Jointly optimize the image processing model based on different types of losses of the image processing model until the image processing model meets a first training completion condition, thereby obtaining a trained image processing model; the trained image processing model is used to simultaneously perform image illumination enhancement and glare degradation repair on the image.

[0072] In step S101, different simulated glare images are combined with low-illumination images in a low-illumination image library to synthesize a complex illumination image corresponding to the low-illumination image; wherein each low-illumination image corresponds to a pseudo normal illumination image after illumination enhancement.

[0073] Here, the pseudo normal illumination image is obtained after the illumination of the low illumination image is enhanced.

[0074] That is, the low illumination image is subjected to glare simulation processing and illumination enhancement processing respectively, so as to obtain a pair of complex illumination images and a pseudo normal illumination image, wherein the pair of complex illumination images and the pseudo normal illumination image are based on the low illumination image.

[0075] Specifically, the training method further includes: performing illumination enhancement processing on the low illumination image to obtain a pseudo normal illumination image.

[0076] Specifically, the low-illuminance image is subjected to illumination enhancement processing to obtain a pseudo-normal illumination image, including: processing the low-illuminance image in the low-illuminance image library based on a trained illumination enhancement mentor model to generate a pseudo-normal illumination image corresponding to the low-illuminance image.

[0077] The illumination enhancement tutor model utilizes deep learning techniques, particularly convolutional neural networks (CNNs) or their variants, such as generative adversarial networks (GANs), to learn and simulate the mapping relationship from low illumination environments to normal illumination environments.

[0078] In the process of the trained illumination enhancement mentor model processing the low illumination image in the low illumination image library, the illumination enhancement mentor model receives the low illumination image as input, and generates a pseudo normal illumination image corresponding to the input image through its internal network structure and parameters; the pseudo normal illumination image is visually brighter and has richer details, and retains the natural color and texture information of the original image as much as possible, thereby effectively improving the visibility and quality of the image.

[0079] In an embodiment of the present application, the trained illumination enhancement mentor model is obtained by training a generative adversarial network based on unpaired low illumination images and normal illumination images.

[0080] That is to say, before training the image processing model described in the embodiment of the present application, a large number of unpaired low-light image libraries X are first used. l and normal illumination image library X n For two image domains, an illumination enhancement tutor model T is pre-trained based on generative adversarial training.

[0081] Collecting paired trainable data (complex illumination images and normal illumination images) is very difficult and costly, so in the embodiment of the present application, an unsupervised method based on generative adversarial training is used to pre-train an illumination enhancement mentor model to obtain pseudo-normal illumination image labels for low-light images.

[0082] For details, please refer to Figure 2 , Figure 2 The flowchart of the method for training the illumination enhancement tutor model in the embodiment of the present application is shown; the trained illumination enhancement tutor model is obtained by training based on the following steps S201-S204:

[0083] S201, randomly sampling low illumination images and normal illumination images from a low illumination image library and a normal illumination image library;

[0084] S202, the generator of the illumination enhancement tutor model processes the low illumination image to obtain a second illumination enhancement image, and sends the second illumination enhancement image and the normal illumination image to the discriminator of the illumination enhancement tutor model to distinguish true from false and calculate the discrimination loss;

[0085] S203, the illumination enhancement mentor model calculates a first perceptual loss between the low illumination image and the second illumination enhanced image;

[0086] S204. Optimizing the illumination enhancement tutor model based on the first perception loss and the discrimination loss until the illumination enhancement tutor model meets a second training completion condition.

[0087] Please refer to Figure 3 , Figure 3A flow chart of another method for training the illumination enhancement tutor model described in an embodiment of the present application is shown.

[0088] like Figure 2 and Figure 3 As shown, in step S201, from the low-light image library X l and normal illumination image library x n Randomly sample low-light images x l and the normal illumination image x n , at this time, the sampled low-illumination image and normal-illumination image are completely different scenes.

[0089] Then, the low-light image x l Input to the illumination enhancement tutor model, the generator G of the illumination enhancement tutor model processes the low illumination image x l Get the illumination enhanced image G(x l ). The illumination enhanced image G(x l ) and the normal illumination image x n The information is sent to the discriminator D of the illumination enhancement tutor model to distinguish true from false and calculate the discrimination loss. In the embodiment of the present application, the discrimination loss is also called the least squares generative adversarial network loss (LSGAN), which is specifically the following formula (1):

[0090]

[0091] in, is the probability distribution of the real data, is the probability distribution of false data, is the expected value; D Ra It is a relative discriminator structure, and the specific function is as follows:

[0092]

[0093] Among them, σ represents the sigmoid activation function. Through this adversarial training, the illumination-enhanced image output by the generator is made as realistic as possible, that is, the illumination is as similar as possible to the normal illumination image.

[0094]

[0095] Represents a more realistic normal illumination image x n and the illumination enhanced image G(x l ) in the relative discriminator D Ra The relative difference in true and false probabilities is mapped to the interval (0,1) through the sigmoid function σ;

[0096]

[0097] represents the opposite comparison, that is, comparing the illumination enhanced image G(x l ) and the true normal illumination image x n In the relative discriminator D Ra The relative difference in true and false probabilities in the relative discriminator D Ra It is used to estimate the probability that real data is more real than fake data, and guide the generator to synthesize fake images that are more real than real data;

[0098] Characterize the discriminator loss function in the least squares generative adversarial network loss, where D Ra (x n ,G(x l )) represents the illumination enhanced image G(x) generated by the generator l ) and the true normal illumination image x n The relative difference between true and false probabilities in the discriminator D;

[0099] Denotes the normal illumination image x sampled from the true distribution n , the discriminator should distinguish x n Relative to G(x l ) is more true; that is, D Ra (x n ,G(x l )) should be close to 1;

[0100] Represents the image illumination enhanced image G(x l ), the discriminator should distinguish G(x l ) relative to x n More false, that is, D Ra (x n ,G(x l )) should be close to 0;

[0101] Characterize the generator loss function, where

[0102] The goal of the generator is to generate an image that is as realistic as possible, so that the discriminator can generate the generated illumination enhanced image G(x l ) is mistakenly considered to be a normal illumination image x n More realistic, that is, D Ra (G(x l ), x n ) should be close to 1;

[0103] Indicates that the discriminator will sample the normal illumination image x from the true distribution n , which is identified as the generated illumination enhanced image G(xl ) than x n More false, that is, D Ra (x n ,G(x l )) should be close to 0.

[0104] LSGAN solves some problems in GAN training by replacing the binary cross entropy loss function in GAN with a square loss function, making model training more stable and avoiding problems such as gradient disappearance and mode collapse.

[0105] Finally, calculate the low-light image x l and the illumination enhanced image G(x l ) so that the content features remain unchanged before and after enhancement and ensure the perceptual consistency. After the generator G is trained, it becomes the required illumination enhancement mentor model T.

[0106] The illumination enhancement tutor model is optimized based on the first perceptual loss and the discriminant loss until the illumination enhancement tutor model meets the second training completion condition. Specifically, when the first perceptual loss and the discriminant loss no longer decrease significantly in a plurality of consecutive iterations, it can be considered that the model has converged and meets the second training completion condition; or, when the first preset number of training times is reached, it is determined that the second training completion condition is met.

[0107] Aiming at the difficulty of collecting training data for pairs of complex illumination images and normal illumination images, complex illumination images with glare are simulated and generated based on real glare image reference images.

[0108] Please refer to Figure 4 , Figure 4 A flow chart of a method for synthesizing a complex illumination image corresponding to a low illumination image according to an embodiment of the present application is shown; the method of combining different simulated glare images with low illumination images in a low illumination image library to synthesize a complex illumination image corresponding to the low illumination image includes the following steps S401-S403:

[0109] S401, randomly sampling a low illumination image from a low illumination image library;

[0110] S402, randomly combining a simulated glare image in a glare component library;

[0111] S403: Combine the simulated glare image, the low illumination image and the simulated noise to synthesize a complex illumination image corresponding to the low illumination image.

[0112] Here, the simulated noise is random noise simulated by normal distribution. The noise generated by many natural phenomena and physical processes tends to Gaussian distribution. It can be regarded as a reasonable approximation to regard the noise as Gaussian noise. Of course, other noise models can also be selected.

[0113] That is to say, firstly, a simulated glare image x is randomly combined in the glare component library. f , and compare the simulated glare image with the low-light image library X l Randomly sampled low-light image x l Combine to obtain a synthetic complex illumination image Where N is a normal distribution with a mean of 0 and a variance of σ. Since a real image disturbed by light cannot be a simple superposition of a clean image and a damaged image, N is used to simulate the noise randomly caused in real image acquisition, thereby obtaining a more realistic complex light image.

[0114] The glare image library refers to a database containing a collection of images with glare effects; specifically, in the embodiment of the present application, the glare image library includes a collection of various glare components. The glare components include light source images, scattered glare images, and reflected glare images. Scattered glare includes flash images, streak light images, and glare images, etc.; the number of each component is multiple.

[0115] Please refer to Figure 5 , Figure 5 FIG. 4 shows a schematic diagram of generating a simulated glare image according to an embodiment of the present application; Figure 5 As shown in the figure, when the light source is directly irradiated into the camera, the light is reflected and scattered in the camera system, causing glare. That is, the glare in the image is composed of three parts: light source, scattered glare and reflected glare; in scattered glare, flash is a pattern of multiple radial stripes caused by aperture shape, linear impurities and lens defects; streak light is a linear flare that is significantly longer and brighter than flash; reflected glare (also known as ghosting) is caused by reflections from multiple air-glass lens surfaces. First, obtain a reference image of a real glare image, analyze and disassemble the various components of the glare to obtain three parts: light source, scattered glare and reflected glare, and further disassemble the scattered glare to obtain flash, streak light and glare; draw the images of each component through software, and perform random transformations such as angle rotation, stretching transformation, color transformation, contrast adjustment, etc. to form a glare image library; finally, randomly sample and combine from the glare image library when in use to generate a simulated glare image.

[0116] The above two technical solutions are used to generate pseudo normal illumination images and glare images, which are used to generate low illumination image-normal illumination image-complex illumination image pairs in the same scene, avoiding the high cost of collecting large amounts of data in the existing technology and improving the practicality and economy of the technology.

[0117] In the step S102, the synthesized complex illumination image is input into an image processing model, and the complex illumination image is jointly processed by a deglare sub-model and an illumination enhancement sub-model of the image processing model to obtain a first illumination enhanced image with enhanced image illumination and deglare.

[0118] Previous methods treated illumination enhancement and glare removal as two independent tasks, resulting in the inability to effectively handle the complex situation in which both problems exist in practical applications. Simply combining illumination enhancement and glare removal methods may lead to more serious degradation of the results. Without a reasonable framework design, the model cannot accurately learn which areas in the image need illumination enhancement and glare removal.

[0119] Therefore, in an embodiment of the present application, two parts, namely a de-glare sub-model and an illumination enhancement sub-model, are set in the image processing model, and the de-glare sub-model and the illumination enhancement sub-model jointly process the complex illumination image, and the knowledge of the two models interact with each other, thereby effectively processing illumination enhancement and de-glare.

[0120] In terms of the framework structure, the glare removal sub-model and the illumination enhancement sub-model of the image processing model are cascaded front and back, and the glare removal sub-model and the illumination enhancement sub-model have the same structure and share encoder weights.

[0121] Based on this, please refer to Figure 6 The synthesized complex illumination image is input into the image processing model, and the complex illumination image is jointly processed by the deglare sub-model and the illumination enhancement sub-model of the image processing model to obtain a first illumination enhanced image with enhanced image illumination and deglare, including the following steps S601-S602:

[0122] S601, inputting the synthesized complex illumination image into the image processing model, and processing the complex illumination image through the deglare sub-model of the image processing model to obtain a first deglare image;

[0123] S602: Input the first de-glare image to the illumination enhancement sub-model of the image processing model, and process the first de-glare image through the illumination enhancement sub-model to obtain a first illumination enhanced image.

[0124] For details, please refer to Figure 7 , Figure 7 A schematic diagram of the image processing model described in an embodiment of the present application is shown; in the image processing model, the de-glare sub-model (first-stage model) and the illumination enhancement sub-model (second-stage model) each include an encoder and a decoder; the encoder structures of the de-glare sub-model and the illumination enhancement sub-model are the same and share weights; the decoder structures of the de-glare sub-model and the illumination enhancement sub-model are the same.

[0125] In some embodiments, the deglare sub-model and the illumination enhancement sub-model use a shared encoder ε, but use separate decoders D1 and D2.

[0126] In this way, from the perspective of the framework structure, the image processing model can also be called a two-stage encoding-decoding model; the de-glare sub-model and the illumination enhancement sub-model are cascaded front and back. Specifically, in the embodiment of the present application, the first stage realizes the de-glare of the image, and the second stage realizes the illumination enhancement of the image; the de-glare sub-model is the first stage model, and the illumination enhancement sub-model is the second stage model; that is, the output of the de-glare sub-model is the input of the illumination enhancement sub-model.

[0127] Specifically, the synthesized complex illumination image is processed by a two-stage encoding-decoding model; the first stage deglare sub-model processes the low illumination image to obtain the first deglare image, and the second stage illumination enhancement sub-model processes the first deglare image to obtain an illumination enhanced image; that is, in the processing process, the first stage realizes image deglare, and the second stage realizes image illumination enhancement. Figure 7 As shown in the figure, the structure of each stage includes an encoder and a decoder, where the encoder is used to extract multi-level features of the image, and the decoder is used to generate images from the features; although the two-stage models have different purposes, they both need to understand the global structure of the image and the local texture of specific areas (low-illuminance areas and glare-degraded areas) through the encoder, and extract multi-level semantic information to provide to the decoder. At the same time, the de-glare sub-model needs to cooperate with the illumination enhancement sub-model to process the glare-degraded area to be consistent with the texture structure of other low-illuminance areas to be enhanced, while the illumination enhancement model cannot enhance the area after glare repair. In summary, the two stages achieve joint processing of low-illuminance images and avoid repair degradation by interacting with model domain knowledge.

[0128] Furthermore, the two stages share the encoder ε and use separate decoders D1 and D2, which can achieve better joint learning and reduce the number of model parameters.

[0129] First, the shared encoder can extract common features of the input data, which are useful for both decoders D1 and D2. Since the encoder ε is shared, it receives gradient information from decoders D1 and D2 during training, which helps the encoder ε learn more comprehensive features, capture the correlation between the deglare task and the illumination enhancement task, and share useful information between the two tasks, so that the glare-degraded area can be processed to have a consistent texture structure with other low-illuminance areas to be enhanced. The illumination enhancement model cannot re-enhance the area after glare repair, thereby achieving better linkage learning.

[0130] In order to reduce the number of model parameters, since the encoder E is shared, there is no need to train independent encoders for different tasks, thus reducing the total number of model parameters.

[0131] In the step S103, the image processing model determines different types of losses of the image processing model based on the output result of the de-glare sub-model and the low-illuminance image, and the output result of the illumination enhancement sub-model and the pseudo-normal-illuminance image.

[0132] The different types of losses include reconstruction loss and second perception loss, wherein each loss is determined by fusing the output result of the de-glare sub-model with the low-illuminance image, the output result of the illumination enhancement sub-model with the pseudo normal-illuminance image.

[0133] The reconstruction loss is used to measure the difference between the reconstructed image and the original image. The perceptual loss is used to capture the high-level semantic differences between images, rather than just the differences at the pixel level.

[0134] Different types of loss functions can provide different supervisory signals during the training process, guiding the model to converge in a better direction. By integrating these loss functions, a more comprehensive and effective optimization goal can be formed to accelerate the model training process; it also enables the model to focus on multiple optimization goals at the same time during the training process, maintain stable performance in complex environments, reduce overfitting, and improve the generalization ability of the model.

[0135] Please refer to Figure 8 The image processing model determines different types of losses of the image processing model based on the output result of the deglare sub-model and the low-illuminance image, the output result of the illumination enhancement sub-model and the pseudo normal-illuminance image; comprising the following steps S801-S802:

[0136] S801, determining a reconstruction loss of an image processing model based on a first comparison result between a first de-glare image output by the de-glare sub-model and a low-illuminance image, and a first comparison result between a first illumination-enhanced image output by the illumination-enhanced sub-model and a pseudo-normal-illuminance image;

[0137] S802. Determine a second perceptual loss of the image processing model based on a second comparison result between a first de-glare image feature output by the de-glare sub-model at different neural layers and a low-illuminance image, and a second comparison result between a first illumination-enhanced image feature output by the illumination enhancement sub-model at different neural layers and a pseudo-normal-illuminance image.

[0138] In this way, the reconstruction loss simultaneously measures the difference between the first deglare image output by the deglare sub-model and the low-illuminance image, and the difference between the first illumination-enhanced image output by the illumination enhancement sub-model and the pseudo-normal illumination image. The second perceptual loss simultaneously measures the high-level semantic difference between the difference between the first deglare image output by the deglare sub-model and the low-illuminance image, and the difference between the first illumination-enhanced image output by the illumination enhancement sub-model and the pseudo-normal illumination image, thereby enhancing the joint performance of the deglare sub-model and the illumination enhancement sub-model.

[0139] Specifically, the output of the first stage deglare photon model With low light image x l , the output result of the illumination enhancement sub-model in the second stage With the pseudo normal illumination image T(x l ) jointly calculate the reconstruction loss and perceived loss Specifically, it is the following formula (3):

[0140]

[0141] Among them, ||·||1 and ||·||2 are L1 norm and L2 norm respectively; based on the reconstruction loss and the second perceptual loss The performance of the first and second stages of the generator is optimized from two different perspectives: image texture reconstruction and perception, so as to generate higher quality and more desired images.

[0142] Reconstruction losses middle, The calculation is the first deglare image generated by the first stage deglare sub-model With low light image x l The purpose is to ensure that the first-stage deglare sub-model can restore the basic structure and content of the image degraded by glare interference as much as possible; || I E -T(x l )||1 is the first illumination enhanced image output by the illumination enhancement sub-model in the second stage With the pseudo normal illumination image T(x l ), ensuring that the generated illumination enhanced image is as similar as possible to the desired, pseudo normal illumination image; in this way, the image after repairing the glare degradation area is as similar as possible to the original low illumination image, and the illumination enhanced image is as similar as possible to the desired, pseudo normal illumination image, and a function is used to consider the loss of the generator in the two stages.

[0143] Perceived loss middle, The calculation is the glare repair image generated by the first stage deglare sub-model With low light image x l The feature L2 distance at the output of different neural layers of the pre-trained network model, Characterize different neural layers pre-trained on large-scale image datasets; The value range is the neural layer of the pre-trained network model; the overall calculation is the sum of the feature differences of different layers, so as to ensure that the generated image is similar to the original low-light image in visual perception based on the feature extraction ability of the visual neural network. The illumination enhanced image generated by the second stage illumination enhancement sub-model is calculated With the pseudo normal illumination image T(x l ) at different neural layers, ensuring that the generated enhanced image is similar to the pseudo-normal illumination image at the high-level semantic level; in this way, at the semantic perception level, the image after repairing the glare degradation area is made as similar as possible to the original low-illumination image, and the image after illumination enhancement is made as similar as possible to the expected, pseudo-normal illumination image, and the perceptual loss of the two generators is considered with one function.

[0144] As an example only, the pre-trained network model is a VGG-19 model; The value range is ReLU1_2, ReLU2_2, ReLU3_2, ReLU4_3, ReLU5_3 neural layers.

[0145] The loss function of the image processing model described in the embodiment of the present application is designed jointly for the purpose of model implementation and the structure of the two stages of the model; for the glare repair problem, it is necessary to reconstruct the degraded area, and for the illumination enhancement problem, it is necessary to reconstruct the low-illuminance area; at the same time, glare repair and illumination enhancement are both achieved without changing the original image texture, and the generated image needs to be preserved unchanged from the original image in high-level semantics or visual perception; therefore, for the two sub-models of the two stages, although the purposes to be achieved are different, a unified reconstruction loss and perception loss are designed for training; such a unified loss function design can ensure that the gradient difference of the two-stage models is not large during training, making the model more stable during training, and the capabilities of the two-stage models grow synchronously, and there will be no problem of a unilateral model dominating the training process, so it can ensure that the final trained model has higher accuracy.

[0146] In the step S104, the image processing model is jointly optimized based on different types of losses of the image processing model until the image processing model meets the first training completion condition, thereby obtaining a trained image processing model; the trained image processing model is used to simultaneously perform image illumination enhancement and glare degradation repair on the image.

[0147] The first training completion condition may be that different types of losses of the image processing model meet preset training completion conditions, or may be that a preset second training number of times is reached.

[0148] The trained image processing model is used to simultaneously perform image illumination enhancement and glare degradation repair on an image. Specifically, an image to be processed for glare repair and image illumination enhancement is obtained;

[0149] The image to be processed is input into a trained image processing model, and the image to be processed is jointly processed by the deglare sub-model and the illumination enhancement sub-model of the trained image processing model to obtain a processed target image.

[0150] The following is a comprehensive description of the overall technical principles of the training method of the image processing model described in the embodiment of the present application; please refer to Fig. 9 , Fig. 9 The schematic diagram of the training method of the image processing model described in the embodiment of the present application is shown; Fig. 9 As shown, before training, a large number of unpaired low-light image libraries X are first used. l and normal illumination image library x n Two image domains, based on generative adversarial training, a illumination enhancement tutor model T is pre-trained; during training, simulated glare images x are first randomly combined in the glare component library f , and compare the simulated glare image with the image from X l Randomly sampled low-light image x l Combine to obtain a synthetic complex illumination image Where N is a normal distribution with a mean of 0 and a variance of σ. The synthesized complex illumination image is then processed by an image processing model (two-stage encoding-decoding model); the image processing model includes a first-stage model (de-glare sub-model) and a second-stage model (illuminance enhancement sub-model); the first-stage model processes the low-illuminance image to obtain a de-glare image, and the second-stage generator processes the de-glare image to obtain an illumination enhancement image; at the same time, the low-illuminance image x is processed by the pre-trained mentor model T l Get the pseudo normal illumination image T(x l ); Finally, the output of the first stage model With low light image x l , the output of the second stage generator With the pseudo normal illumination image T(x l ) jointly calculate the reconstruction loss and perceived loss Based on the reconstruction loss and perceived loss The image processing model is optimized.

[0151] The training method of the image processing model described in the embodiment of the present application combines the illumination enhancement and glare removal tasks together to form a comprehensive solution by designing a two-stage deep learning model and a reasonable training framework; during training, the knowledge interaction of the two task models of glare removal and illumination enhancement is used to intelligently identify the areas in the image that need to enhance the illumination and remove the glare, and a two-stage progressive processing is adopted, which can not only effectively repair the glare artifacts and improve the overall visual quality of the image, but also make the image more uniform in brightness in a complex lighting environment, while retaining more details and reducing image distortion; by generating a pseudo-normal illumination image and a glare image, they are used for the generation of low-illumination image-normal illumination image-complex illumination image pairs in the same scene, thereby avoiding the problem of high cost of large-scale data collection in the prior art and improving the practicality and economy of the technology.

[0152] Based on the same inventive concept, an image processing method corresponding to the training method of the image processing model is also provided in the embodiment of the present application; since the image processing method in the embodiment of the present application is based on the image processing model trained by the training method of the image processing model to process complex illumination images, the principle of solving the problem is similar to the training method of the image processing model in the embodiment of the present application, so the implementation of the image processing method can refer to the implementation of the training method of the image processing model, and the repeated parts will not be repeated.

[0153] Please refer to Fig.10 , Fig.10 The flowchart of the image processing method according to the embodiment of the present application is shown; the image processing method comprises the following steps S1001-S1002:

[0154] S1001, obtaining an image to be processed for glare repair and image illumination enhancement;

[0155] S1002, inputting the image to be processed into a trained image processing model, and jointly processing the image to be processed by the deglare sub-model and the illumination enhancement sub-model of the trained image processing model to obtain a processed target image.

[0156] In some embodiments, in the image processing method, the step of inputting the image to be processed into a trained image processing model, and jointly processing the image to be processed by the deglare sub-model and the illumination enhancement sub-model of the trained image processing model to obtain a processed target image includes:

[0157] Inputting the image to be processed into a trained image processing model, processing the image to be processed by the deglare sub-model of the trained image processing model to obtain a second deglare image, and processing the second deglare image by the illumination enhancement sub-model of the trained image processing model to obtain a third illumination enhanced image;

[0158] The trained image processing model outputs the third illumination enhanced image as a processed target image.

[0159] In some embodiments, the image processing method, the trained image processing model includes a glare removal sub-model and an illumination enhancement sub-model, the glare removal sub-model and the illumination enhancement sub-model are cascaded front and back, the glare removal sub-model and the illumination enhancement sub-model have the same structure and share encoder weights.

[0160] Based on the same inventive concept, the embodiments of the present application also provide a training device for an image processing model corresponding to the training method for the image processing model; since the principle of solving the problem by the training device in the embodiments of the present application is similar to the training method for the image processing model in the embodiments of the present application, the implementation of the training device can refer to the implementation of the training method for the image processing model, and the repeated parts will not be repeated.

[0161] Please refer to Fig.11 , Fig.11 A structural schematic diagram of a training device for an image processing model according to an embodiment of the present application is shown; the training device for an image processing model comprises:

[0162] The synthesis module 1101 is used to combine different simulated glare images with low illumination images in the low illumination image library to synthesize a complex illumination image corresponding to the low illumination image; wherein each low illumination image corresponds to a pseudo normal illumination image;

[0163] A processing module 1102 is used to input the synthesized complex illumination image into the image processing model, and jointly process the complex illumination image through the deglare sub-model and the illumination enhancement sub-model of the image processing model to obtain a first illumination enhanced image with enhanced image illumination and deglare;

[0164] A determination module 1103 is used for the image processing model to determine different types of losses of the image processing model based on the output result of the deglare sub-model and the low-illuminance image, the output result of the illumination enhancement sub-model and the pseudo normal-illuminance image;

[0165] The optimization module 1104 is used to jointly optimize the image processing model based on different types of losses of the image processing model until the image processing model meets the first training completion condition, thereby obtaining a trained image processing model; the trained image processing model is used to simultaneously perform image illumination enhancement and glare degradation repair on the image.

[0166] In some embodiments, in the training device for the image processing model, the training device further comprises:

[0167] A generation module is used to process the low-illuminance image in the low-illuminance image library based on the trained illumination enhancement mentor model to generate a pseudo normal illumination image corresponding to the low-illuminance image.

[0168] In some embodiments, in the training device for the image processing model, the trained illumination enhancement mentor model is obtained by training a generative adversarial network based on unpaired low-illumination images and normal-illumination images.

[0169] In some embodiments, the training device of the image processing model further includes a training module, wherein the training module is used to randomly sample low-illumination images and normal-illumination images from a low-illumination image library and a normal-illumination image library;

[0170] The generator of the illumination enhancement mentor model processes the low illumination image to obtain a second illumination enhancement image, and sends the second illumination enhancement image and the normal illumination image to the discriminator of the illumination enhancement mentor model to distinguish true from false and calculate the discrimination loss;

[0171] The illumination enhancement mentor model calculates a first perceptual loss between the low illumination image and a second illumination enhanced image;

[0172] The illumination enhancement tutor model is optimized based on the first perceptual loss and the discriminative loss until the illumination enhancement tutor model meets a second training completion condition.

[0173] In some embodiments, in the training device of the image processing model, the glare removal sub-model and illumination enhancement sub-model of the image processing model are cascaded front and back, and the glare removal sub-model and illumination enhancement sub-model have the same structure and share encoder weights.

[0174] In some embodiments, in the training device for the image processing model, the processing module, after inputting the synthesized complex illumination image into the image processing model, jointly processes the complex illumination image through the deglare sub-model and the illumination enhancement sub-model of the image processing model to obtain a first illumination-enhanced image with enhanced image illumination and deglare, is specifically used to:

[0175] Inputting the synthesized complex illumination image into the image processing model, and processing the complex illumination image through the deglare sub-model of the image processing model to obtain a first deglare image;

[0176] The first de-glare image is input into the illumination enhancement sub-model of the image processing model, and the first de-glare image is processed by the illumination enhancement sub-model to obtain a first illumination enhanced image.

[0177] In some embodiments, in the training device for the image processing model, the synthesis module, when combining different simulated glare images with low-illumination images in the low-illumination image library to synthesize a complex illumination image corresponding to the low-illumination image, is specifically used to:

[0178] Randomly sample a low-light image from a low-light image library;

[0179] Randomly combine simulated glare images in the glare component library;

[0180] The simulated glare image, the low illumination image and the simulated noise are combined to synthesize a complex illumination image corresponding to the low illumination image.

[0181] In some embodiments, in the training device of the image processing model, the determination module, when the image processing model determines different types of losses of the image processing model based on the output result of the deglare sub-model and the low-illuminance image, the output result of the illumination enhancement sub-model and the pseudo-normal illumination image, is specifically used to:

[0182] Determining a reconstruction loss of an image processing model based on a first comparison result between a first deglare image output by the deglare sub-model and a low illumination image, and a first comparison result between a first illumination enhanced image output by the illumination enhancement sub-model and a pseudo normal illumination image;

[0183] Based on the second comparison results of the first de-glare image features output by the de-glare sub-model at different neural layers and the low-illuminance image, and the second comparison results of the first illumination enhancement image features output by the illumination enhancement sub-model at different neural layers and the pseudo-normal illumination image, the second perceptual loss of the image processing model is determined.

[0184] Based on the same inventive concept, an image processing device corresponding to the image processing method is also provided in the embodiment of the present application; since the principle of solving the problem by the image processing device in the embodiment of the present application is similar to the above-mentioned image processing method in the embodiment of the present application, the implementation of the image processing device can refer to the implementation of the image processing method, and the repeated parts will not be repeated.

[0185] Please refer to Fig.12 , Fig.12 The schematic diagram of the structure of the image processing device according to the embodiment of the present application is shown; the image processing device comprises:

[0186] An acquisition module 1201 is used to acquire an image to be processed for glare repair and image illumination enhancement;

[0187] The joint processing module 1202 is used to input the image to be processed into the trained image processing model, and jointly process the image to be processed through the deglare sub-model and illumination enhancement sub-model of the trained image processing model to obtain a processed target image.

[0188] In some embodiments, the joint processing module in the image processing device, when inputting the image to be processed into the trained image processing model, jointly processing the image to be processed by the deglare sub-model and the illumination enhancement sub-model of the trained image processing model to obtain a processed target image, is specifically used to:

[0189] Inputting the image to be processed into a trained image processing model, processing the image to be processed by the deglare sub-model of the trained image processing model to obtain a second deglare image, and processing the second deglare image by the illumination enhancement sub-model of the trained image processing model to obtain a third illumination enhanced image;

[0190] The trained image processing model outputs the third illumination enhanced image as a processed target image.

[0191] In some embodiments, in the image processing device, the trained image processing model includes a de-glare sub-model and an illumination enhancement sub-model, the de-glare sub-model and the illumination enhancement sub-model are cascaded front and back, the de-glare sub-model and the illumination enhancement sub-model have the same structure and share encoder weights.

[0192] In some embodiments, an electronic device is also provided. Fig.13 , Fig.13 A schematic diagram of the structure of the electronic device described in an embodiment of the present application is shown; the electronic device 1300 includes: a processor 1302, a memory 1301 and a bus, the memory 1301 stores machine-readable instructions executable by the processor 1302, when the electronic device 1300 is running, the processor 1302 communicates with the memory 1301 through the bus, and when the machine-readable instructions are executed by the processor 1302, the steps of the training method of the image processing model are executed, or the steps of the image processing method are executed.

[0193] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the training method of the image processing model or the steps of the image processing method are executed.

[0194] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0195] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0196] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0197] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a platform server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0198] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A training method for an image processing model, characterized in that: The training method comprises: Combining different simulated glare images with low-illumination images in a low-illumination image library to synthesize a complex illumination image corresponding to the low-illumination image; wherein each low-illumination image corresponds to a pseudo-normal illumination image after illumination enhancement; Inputting the synthesized complex illumination image into an image processing model, and jointly processing the complex illumination image through a glare removal sub-model and an illumination enhancement sub-model of the image processing model to obtain a first illumination enhanced image with enhanced image illumination and glare removal; The image processing model determines different types of losses of the image processing model based on the output result of the de-glare sub-model and the low-illuminance image, the output result of the illumination enhancement sub-model and the pseudo-normal illumination image; The image processing model is jointly optimized based on different types of losses of the image processing model until the image processing model meets the first training completion condition, thereby obtaining a trained image processing model; the trained image processing model is used to simultaneously perform image illumination enhancement and glare degradation repair on the image.

2. The training method of the image processing model according to claim 1, characterized in that: The training method further comprises: The low illumination images in the low illumination image library are processed based on the trained illumination enhancement mentor model to generate pseudo normal illumination images corresponding to the low illumination images.

3. The training method of the image processing model according to claim 1 or 2, characterized in that: The trained illumination enhancement mentor model is obtained by training a generative adversarial network based on unpaired low illumination images and normal illumination images.

4. The training method of the image processing model according to claim 3, characterized in that: The trained illumination enhancement tutor model is obtained by training based on the following steps: Randomly sample low-illumination images and normal-illumination images from a low-illumination image library and a normal-illumination image library; The generator of the illumination enhancement mentor model processes the low illumination image to obtain a second illumination enhancement image, and sends the second illumination enhancement image and the normal illumination image to the discriminator of the illumination enhancement mentor model to distinguish true from false and calculate the discrimination loss; The illumination enhancement mentor model calculates a first perceptual loss between the low illumination image and a second illumination enhanced image; The illumination enhancement tutor model is optimized based on the first perceptual loss and the discriminative loss until the illumination enhancement tutor model meets a second training completion condition.

5. The training method of the image processing model according to claim 1, characterized in that: The glare removal sub-model and the illumination enhancement sub-model of the image processing model are cascaded front and back, and the glare removal sub-model and the illumination enhancement sub-model have the same structure and share encoder weights.

6. The training method of the image processing model according to claim 1 or 5, characterized in that: The synthesized complex illumination image is input into the image processing model, and the complex illumination image is jointly processed by the deglare sub-model and the illumination enhancement sub-model of the image processing model to obtain a first illumination enhanced image with enhanced image illumination and deglare, including: Inputting the synthesized complex illumination image into the image processing model, and processing the complex illumination image through the deglare sub-model of the image processing model to obtain a first deglare image; The first de-glare image is input into the illumination enhancement sub-model of the image processing model, and the first de-glare image is processed by the illumination enhancement sub-model to obtain a first illumination enhanced image.

7. The training method of the image processing model according to claim 1, characterized in that: The combining of different simulated glare images and low-illumination images in a low-illumination image library to synthesize a complex illumination image corresponding to the low-illumination image includes: Randomly sample a low-light image from a low-light image library; Randomly combine simulated glare images in the glare component library; The simulated glare image, the low illumination image and the simulated noise are combined to synthesize a complex illumination image corresponding to the low illumination image.

8. The method for training an image processing model according to claim 1, characterized in that: The image processing model determines different types of losses of the image processing model based on the output result of the deglare sub-model and the low illumination image, the output result of the illumination enhancement sub-model and the pseudo normal illumination image; including: Determining a reconstruction loss of an image processing model based on a first comparison result between a first deglare image output by the deglare sub-model and a low illumination image, and a first comparison result between a first illumination enhanced image output by the illumination enhancement sub-model and a pseudo normal illumination image; Based on the second comparison results of the first de-glare image features output by the de-glare sub-model at different neural layers and the low-illuminance image, and the second comparison results of the first illumination enhancement image features output by the illumination enhancement sub-model at different neural layers and the pseudo-normal illumination image, the second perceptual loss of the image processing model is determined.

9. An image processing method, characterized in that: The method comprises: Acquire an image to be processed for glare repair and image illumination enhancement; The image to be processed is input into a trained image processing model, and the image to be processed is jointly processed by the deglare sub-model and the illumination enhancement sub-model of the trained image processing model to obtain a processed target image.

10. The image processing method according to claim 9, characterized in that: The step of inputting the image to be processed into a trained image processing model, and jointly processing the image to be processed by a glare removal sub-model and an illumination enhancement sub-model of the trained image processing model to obtain a processed target image comprises: Inputting the image to be processed into a trained image processing model, processing the image to be processed by the deglare sub-model of the trained image processing model to obtain a second deglare image, and processing the second deglare image by the illumination enhancement sub-model of the trained image processing model to obtain a third illumination enhanced image; The trained image processing model outputs the third illumination enhanced image as a processed target image.

11. The image processing method according to claim 9 or 10, characterized in that: The trained image processing model includes a glare removal sub-model and an illumination enhancement sub-model, the glare removal sub-model and the illumination enhancement sub-model are cascaded front and back, the glare removal sub-model and the illumination enhancement sub-model have the same structure and share encoder weights.

12. A training device for an image processing model, characterized in that: The training device comprises: A synthesis module, used for combining different simulated glare images with low illumination images in a low illumination image library to synthesize a complex illumination image corresponding to the low illumination image; wherein each low illumination image corresponds to a pseudo normal illumination image; A processing module, used for inputting the synthesized complex illumination image into the image processing model, and jointly processing the complex illumination image through the deglare sub-model and the illumination enhancement sub-model of the image processing model to obtain a first illumination enhanced image with enhanced image illumination and deglare; A determination module, used for the image processing model to determine different types of losses of the image processing model based on the output result of the de-glare sub-model and the low-illuminance image, the output result of the illumination enhancement sub-model and the pseudo-normal illumination image; An optimization module is used to jointly optimize the image processing model based on different types of losses of the image processing model until the image processing model meets the first training completion condition, thereby obtaining a trained image processing model; the trained image processing model is used to simultaneously perform image illumination enhancement and glare degradation repair on the image.

13. An image processing device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed for glare repair and image illumination enhancement; The joint processing module is used to input the image to be processed into the trained image processing model, and jointly process the image to be processed through the deglare sub-model and illumination enhancement sub-model of the trained image processing model to obtain a processed target image.

14. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the training method of the image processing model as described in claims 1 to 9 are executed, or the steps of the image processing method as described in any one of claims 10 to 11 are executed.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the training method of the image processing model as described in claims 1 to 9, or executes the steps of the image processing method as described in any one of claims 10 to 11.

Images