Semantic-guided extreme overexposure image restoration method

Abstract

The invention discloses a semantic-guided extreme overexposure image restoration method. The method comprises the following steps: 1, obtaining a data set, and dividing the data set into a training set and a test set; 2, extracting brightness information of the overexposure image, and generating a brightness map; 3, constructing an overexposure image brightness correction network model; 4, training an overexposure image brightness correction network model; 5, correcting the overexposure image by using the trained model, and generating a corrected image and a mask of a brightness saturation region; 6, constructing a pre-trained brightness saturation completion network model, and performing information completion by using the corrected image, semantic graph and brightness cue word; and 7, fusing the complemented brightness saturation region with the unsaturated region to obtain a final output image. According to the method, semantic information is applied to an image restoration process, restoration is effectively guided and constrained, and a visual pleasure and semantic coherence result is generated.

Description

Technical Field

[0001] The present invention relates to the field of extreme overexposed image restoration, and mainly to a semantically guided extreme overexposed image restoration method. Background Art

[0002] Overexposure is a common challenge in digital photography, usually caused by an overly bright environment or incorrect camera settings such as exposure time and aperture. This problem can result in saturated areas of brightness in the image, especially in the sky, loss of details, and the image appearing too bright or pure white. This overexposure problem not only reduces the visual appeal of the image, but also complicates post-processing.

[0003] Restoration of extremely overexposed images faces two major challenges. First, these images are usually affected by an overall brightness imbalance, which requires correcting the brightness of the image to a more normal range. Second, due to the excessive brightness, there are brightness saturated areas in the image where detail information is lost. These areas usually appear white and lack surface information, making their restoration difficult.

[0004] Some recent new methods attempt to solve the overexposure problem through exposure correction and high dynamic range (HDR) reconstruction. Although these methods improve visual quality by adjusting contrast and brightness, they often introduce artifacts or fail to recover lost details in brightness saturated areas in extreme overexposure cases. On the other hand, single-frame HDR has difficulty handling extreme overexposure problems due to insufficient information in the original image, while multi-frame HDR requires multiple exposures, which also places requirements on the input image.

[0005] The motivation of this paper is to address the problem of global brightness imbalance in extremely overexposed images and the challenge of recovering details in brightness-saturated regions where surface texture information is lost but semantic information is still present. A novel approach is proposed that employs an overexposure correction network and dynamic convolution to effectively correct image brightness while minimizing noise and artifacts. For brightness-saturated regions where surface texture is lost, a generative model such as a diffusion model is used to fully recover the lost information using the learned data distribution. Recognizing that these regions often retain meaningful semantic information, semantic information is introduced to guide the restoration process of the diffusion model. The proposed method avoids artifacts at the edges of brightness-saturated regions and uses semantic information to guide the restoration, producing visually superior results. Summary of the invention

[0006] Purpose of the invention: In view of the problems existing in the above-mentioned background technology, the present invention provides a semantically guided diffusion model for restoration of extremely overexposed images, and the method mainly has the following two contributions: (i) The present invention designs a novel framework specifically for restoring overexposed images. The framework can correct overexposed images pixel by pixel and solve the brightness saturation areas caused by excessive lighting. In addition, the robust prior knowledge of the diffusion model is used to restore the brightness saturation areas, which improves the fidelity of the restoration process. (ii) The present invention designs a unique combination of semantic information to provide effective guidance and constraints for image restoration. By guiding stable diffusion with semantic information, it is ensured that the restored brightness saturation areas maintain the original semantic structure and consistency of the image. This method effectively prevents the occurrence of blurred or unrealistic restoration and produces visually attractive and semantically coherent results. The present invention demonstrates that the proposed method outperforms the current state-of-the-art methods through extensive experimental comparisons, including visual quality assessment, no-reference / full-reference image quality assessment, and human subjective surveys.

[0007] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0008] A semantically guided extreme overexposed image restoration method, comprising:

[0009] Obtain a data set, the data set including an overexposed image I h , dividing the data set into a training set and a test set; extracting the input overexposed image I pixel by pixel h The brightness information is used to obtain the brightness map B; an overexposed image brightness correction network model is constructed, and the overexposed image I in the training set is h and the overexposed image I h The corresponding brightness map B is input into the overexposed image brightness correction network model; it is judged whether the overexposed image I is completed. h Whether all training rounds are completed, and a trained over-exposed image brightness correction network model is obtained; the over-exposed image I h Send it into the trained overexposed image brightness correction network model to obtain the corrected image I c , for the corrected image I c After semantic segmentation and brightness extraction operations, the semantic map S and the mask M of the brightness saturation area are obtained; a pre-trained brightness saturation completion network model is built and the corrected image I c As input, the semantic map S is introduced to use semantic information to complete the mask M of the brightness saturated area. At the same time, the brightness prompt word P is introduced to control the brightness of the information-completed part to avoid overexposure or low brightness. The brightness saturated area after information completion and the unsaturated area after brightness correction are fused according to the mask M of the brightness saturated area to obtain the final output Io.

[0010] Preferably, the overexposed image I in the data set h Converted to a grayscale image. Since each pixel value in the grayscale image represents its corresponding brightness value, the grayscale image is directly used as the brightness map B.

[0011] Preferably, the overexposed image brightness correction network model includes 7 convolutional layers connected in series, wherein the first and last layers use ordinary standard convolutional layers, and the 2nd to 6th layers use brightness-aware variable convolutional layers. For the activation function, a ReLU activation function is connected in series after each convolutional layer in the first 6 layers, and a Tanh activation function is connected in series after the 7th convolutional layer;

[0012] Using the input image I h The extracted brightness map B and standard convolution X, for the standard convolution X, the kernel generator generates two basic convolution kernels k1 and k2; the brightness map B is input into the brightness-aware variable convolution layer to predict the guided feature, which is used to determine the pixel weights w1 and w2; under the guidance of the weights w1 and w2, the basic kernels k1 and k2 are assigned to brightness saturated or unsaturated areas.

[0013] Preferably, the overexposed image I h Input into the overexposed image brightness correction network model to obtain the corrected image I c ;

[0014] For the corrected image I c , using the exposure control loss L exp , color constancy loss L col , local smoothness loss L tv and spatial consistency loss L spa , train the overexposed image brightness correction network model in an end-to-end manner:

[0015]

[0016] Where N is the number of non-overlapping local regions of size 16×16, k is a parameter, and Y k is the average intensity value of the local area in the enhanced image, E refers to the gray level in the RGB color space, which is set to 0.6;

[0017]

[0018] Where R is the number of iterations, r is the parameter, and Represents horizontal and vertical gradient operations respectively, represents the curve parameter graph, ξ represents the color channel;

[0019]

[0020] Among them J p represents the average intensity value of p channels in the enhanced image, J q represents the average intensity value of q channels in the enhanced image, and μ represents a pair of channels (p, q);

[0021]

[0022] Where K is the number of local regions, i is the i-th local region, Ω(i) is the four adjacent regions above, below, left, and right of the local region i as the center; Y i represents the average intensity value of the i-th local area in the enhanced image, Y j Represents the average intensity value of the local area in the enhanced image, I i Represents the average intensity value of the i-th local area in the input image, I j Represents the average intensity value of the th local area in the input image.

[0023] Preferably, the overexposed image is input into a trained overexposed image brightness correction network model to obtain a corrected image I c : to I c After semantic segmentation and brightness extraction operations, the corresponding semantic map S and the mask M of the brightness saturated area are obtained; the semantic map S is obtained by a general semantic segmentation model Oneformer; the mask M of the brightness saturated area is extracted by first calculating the brightness value of each pixel. When the brightness value is greater than the threshold H, the pixel is considered to belong to the brightness saturated area.

[0024] Preferably, the brightness saturation completion network model includes a variational autoencoder VAE, a denoising U-Net, a semantic control network and a CLIP text encoder: The variational autoencoder VAE processes the corrected image I c to generate latent features, which are then subjected to random noise processing to generate noisy latent features z t ; To ensure that the brightness oversaturated part of the restored image is semantically consistent with the original features, the semantic map S is input into the semantic control network to extract the semantic condition c, which guides the pre-trained noise prediction network ∈ θ The noise added to the latent features is predicted; in addition, the CLIP text encoder extracts the text guidance condition y, which links the prompt word to the brightness of the generated image; the guided noise prediction is expressed as:

[0025]

[0026] where k is a factor that adjusts the strength of the guidance, and t is the number of time steps. Indicates that the boot condition is empty;

[0027] After noise prediction, the DDIM sampler is used to speed up the sampling process and gradually denoise to obtain the denoised feature z0:

[0028]

[0029] where η is usually set to 0, α and σ are predefined hyperparameters, and ∈ represents random Gaussian noise.

[0030] Preferably, in order to complete the restoration, the predicted z0 is passed through the decoder D to obtain the image generated by the brightness saturation completion network model Then the final image I is synthesized using the following formula o , the corrected image I is divided into regions under the guidance of the mask M of the brightness saturation region c Image generated by the brightness saturation completion network model To perform the fusion:

[0031]

[0032] A terminal device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the restoration method for extremely overexposed images using a semantically guided diffusion model as described in any one of claims 1 to 7.

[0033] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor executes the steps of the method for repairing an extremely overexposed image using a semantically guided diffusion model as described in any one of claims 1 to 7.

[0034] Beneficial effects:

[0035] (1) We design a novel framework specifically for restoring overexposed images. This framework can correct overexposed images pixel by pixel and address brightness saturation regions caused by over-illumination. In addition, we use the robust prior knowledge inherent in the diffusion model to restore brightness saturation regions, improving the fidelity of the restoration process.

[0036] (2) The present invention designs a unique method that combines semantic information to provide effective guidance and constraints for image restoration. By guiding the stable diffusion with semantic information, it ensures that the restored brightness saturated areas maintain the original semantic structure and consistency of the image. This method effectively prevents the occurrence of blurry or unrealistic restoration and produces visually attractive and semantically coherent results.

[0037] (3) We demonstrate that the proposed method outperforms the current state-of-the-art methods through extensive experimental comparisons, including visual quality assessment, no-reference / full-reference image quality assessment, and human subjective survey. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of a restoration method for an extremely overexposed image using a semantically guided diffusion model provided by the present invention;

[0039] Figure 2 It is an algorithm framework diagram of a restoration method for extremely overexposed images using a semantically guided diffusion model provided by the present invention;

[0040] Figure 3 It is a structural diagram of an overexposed image brightness correction network model in a restoration method for extremely overexposed images using a semantically guided diffusion model provided by the present invention.

[0041] Figure 4 It is a structural diagram of a brightness saturation completion network model in a restoration method for extremely overexposed images using a semantically guided diffusion model provided by the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The present invention provides a semantically guided extreme overexposed image restoration method, comprising the following steps:

[0044] A semantically guided extreme overexposed image restoration method comprises the following steps:

[0045] Step S1: Obtain a data set, wherein the data set includes an overexposed image I h , dividing the data set into a training set and a test set;

[0046] In order to effectively train the model, a dataset containing a large number of extremely overexposed images is required. The training set is derived from the MIT-Adobe FiveK dataset, which includes 5000 original RGB images and 5 professionally rendered RGB images for each original image. Adobe Camera Raw SDK is used in Photoshop to simulate overexposed images with a relative EV of +1.5, which are adjusted to extremely overexposed images as input for the training set. It is worth noting that since the overexposure correction network is trained in an unsupervised manner, the training dataset consists only of these simulated overexposed images without any real images. For testing, 124 overexposed images are randomly selected from the SICE dataset as the test set, which includes low-contrast images at different exposure levels and their corresponding high-quality counterparts.

[0047] Step S2: Extract the input overexposed image I pixel by pixel h The brightness information of the image is obtained to obtain the corresponding brightness map B;

[0048] Specifically, the input RGB image I h Convert to grayscale image. In grayscale image, the value of each pixel is its corresponding brightness value. Since each pixel value in grayscale image represents its corresponding brightness value, the grayscale image is directly used as brightness map B.

[0049] Step S3: construct an overexposed image brightness correction network model, and use the overexposed image I in the training set h The corresponding brightness map B is input into the overexposed image brightness correction network model to perform image brightness correction network to obtain the corrected image I c ;

[0050] Specifically, refer to Figure 3 The overexposed image brightness correction network model consists of 7 convolutional layers connected in series, of which the first and last layers use ordinary standard convolutional layers, and the 2nd to 6th layers use brightness-aware variable convolutional layers. For the activation function, a ReLU activation function is connected in series after each convolutional layer in the first 6 layers, and a Tanh activation function is connected in series after the 7th convolutional layer;

[0051] The brightness-aware variable convolutional layer is designed to adaptively adjust the correction strength according to the changing overexposure conditions. hThe brightness map B extracted from the image and the standard convolution X are used to generate brightness-aware variable convolution kernels by predicting a set of basic convolution kernels and guided features. For the standard convolution X, the kernel generator generates two basic convolution kernels, k1 and k2. The brightness map B is input into the convolution layer to predict the guided features, which are used to determine the pixel weights w1 and w2. Under the guidance of these weights, the basic kernels k1 and k2 are assigned to brightness saturated or unsaturated regions, enabling the brightness-aware variable convolution kernel to perform targeted feature extraction.

[0052] Step S4: Determine whether the overexposed image I is completed. h Whether all training rounds have been completed and a trained over-exposed image brightness correction network model has been obtained;

[0053] For the corrected image I c , using the exposure control loss L exp , color constancy loss L col , local smoothness loss Ltv and spatial consistency loss L spa , train the overexposed image brightness correction network model in an end-to-end manner;

[0054] For the overexposed image brightness correction network model, the exposure control loss L is used exp To control the exposure of an image, it measures the deviation between the average brightness of a local area and the brightness required for correct exposure:

[0055]

[0056] Where N represents the number of non-overlapping local regions of size 16×16, Y is the average intensity value of the local region in the enhanced image, and E refers to the grayscale level in the RGB color space, which is set to 0.6.

[0057] In addition, to help the network accurately estimate the illumination and parameter maps, a local smoothness loss is applied during the brightness correction process, denoted as L tv :

[0058]

[0059] Where R is the number of iterations, and They represent horizontal and vertical gradient operations respectively, A represents the curve parameter map, and ξ represents the color channel.

[0060] In order to minimize the shift and ensure that the distribution of RGB channels remains balanced, a color constancy loss is adopted, denoted as L col :

[0061]

[0062] Among them J p represents the average intensity value of p channels in the enhanced image, J q represents the average intensity value of q channels in the enhanced image, and μ represents a pair of channels (p, q);

[0063] In addition, using L spa To measure the change in pixel value differences between adjacent regions after correction, aiming to maintain spatial consistency:

[0064]

[0065] Where K is the number of local regions, Ω(i) is the four neighboring regions (upper, lower, left, and right) centered on region i. Let Y and denote the average intensity values ​​of the local regions in the enhanced version and the input image, respectively.

[0066] Step S5: Overexposed image I h Send it into the trained overexposed image brightness correction network model to obtain the corrected image I c , to I c After semantic segmentation and brightness extraction operations, the corresponding semantic map S and the mask M of the brightness saturated area are obtained;

[0067] to I c After semantic segmentation and brightness extraction operations, the corresponding semantic map S and brightness saturation area mask M are obtained. The semantic map S is obtained by a general semantic segmentation model Oneformer; the brightness saturation area mask M is extracted by first calculating the brightness value of each pixel. When the brightness value is greater than the threshold H, the pixel is considered to belong to the brightness saturation area.

[0068] Step S6: construct a pre-trained brightness saturation completion network model and transform the corrected image I c As input, the semantic graph S is introduced to use semantic information to complete the information of the brightness saturated area M, and the brightness prompt word S is introduced to control the brightness of the completed part to avoid overexposure or low brightness;

[0069] Specifically, refer to Figure 4 , the brightness saturation completion network model includes variational autoencoder VAE, denoising U-Net, semantic control network and CLIP text encoder: variational autoencoder VAE processes the corrected image I c to generate latent features, which are then subjected to random noise processing to generate noisy latent features z t ; To ensure that the brightness oversaturated part of the restored image is semantically consistent with the original features, the semantic map S is input into the semantic control network to extract the semantic condition c, which guides the pre-trained noise prediction network ∈ θThe noise added to the latent features is predicted; in addition, the CLIP text encoder extracts the text guidance condition y, which links the prompt word to the brightness of the generated image; the guided noise prediction is expressed as:

[0070]

[0071] where k is a factor that adjusts the strength of the guidance, and t is the number of time steps. Indicates that the boot condition is empty;

[0072] After noise prediction, the DDIM sampler is used to speed up the sampling process and gradually denoise to obtain the denoised feature z0:

[0073]

[0074] where η is usually set to 0, and α and σ are predefined hyperparameters.

[0075] Step S7: The brightness saturated area after information completion and the non-saturated area after brightness correction are merged according to the mask M of the brightness saturated area to obtain the final output I o ; In order to complete the recovery, the predicted z0 is passed through the decoder D to generate the image The final image is then synthesized using the following formula: the corrected image I is converted region by region under the guidance of the mask M of the brightness saturated area c Image generated by the brightness saturation completion network model To perform the fusion:

[0076]

[0077] This approach ensures that brightness-saturated areas are repaired while keeping unsaturated areas intact, resulting in a coherent and visually pleasing final image.

[0078] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A semantically guided extreme overexposed image restoration method, characterized in that: include: Obtain a data set, the data set including an overexposed image I h , dividing the data set into a training set and a test set; Extract the input overexposed image I pixel by pixel h The brightness information is used to obtain the brightness map B; an overexposed image brightness correction network model is constructed, and the overexposed image I in the training set is h and the overexposed image I h The corresponding brightness map B is input into the overexposed image brightness correction network model; it is judged whether the overexposed image I is completed. h Whether all training rounds are completed, and a trained over-exposed image brightness correction network model is obtained; the over-exposed image I h Send it into the trained overexposed image brightness correction network model to obtain the corrected image I c , for the corrected image I c After semantic segmentation and brightness extraction operations, the semantic map S and the mask M of the brightness saturation area are obtained; a pre-trained brightness saturation completion network model is built and the corrected image I c As input, the semantic graph S is introduced to use semantic information to complete the mask M of the brightness saturated area, and the brightness prompt word P is introduced to control the brightness of the information-completed part to avoid overexposure or low brightness; the brightness saturated area after information completion and the non-saturated area after brightness correction are fused according to the mask M of the brightness saturated area to obtain the final output I o .

2. The semantically guided extreme overexposed image restoration method according to claim 1, characterized in that: The overexposed image I in the dataset h Converted to a grayscale image. Since each pixel value in the grayscale image represents its corresponding brightness value, the grayscale image is directly used as the brightness map B.

3. The semantically guided extreme overexposed image restoration method according to claim 1, characterized in that: The overexposed image brightness correction network model includes 7 convolutional layers connected in series, of which the first and last layers use ordinary standard convolutional layers, and the 2nd to 6th layers use brightness-aware variable convolutional layers. For the activation function, a ReLU activation function is connected in series after each convolutional layer in the first 6 layers, and a Tanh activation function is connected in series after the 7th convolutional layer; Using the input image I h The extracted brightness map B and the standard convolution X, for the standard convolution X, the kernel generator generates two basic convolution kernels k1 and k2; The brightness map B is input into a brightness-aware variable convolutional layer to predict guided features, which are used to determine pixel weights w1 and w2; under the guidance of weights w1 and w2, basic kernels k1 and k2 are assigned to brightness saturated or non-saturated regions.

4. The semantically guided extreme overexposed image restoration method according to claim 1, characterized in that: Overexposed image I h Input into the overexposed image brightness correction network model to obtain the corrected image I c ; For the corrected image I c , using the exposure control loss L exp , color constancy loss L col , local smoothness loss L tv and spatial consistency loss L spa , train the overexposed image brightness correction network model in an end-to-end manner: Where N is the number of non-overlapping local regions of size 16×16, k is a parameter, and Y k is the average intensity value of the local area in the enhanced image, E refers to the gray level in the RGB color space, which is set to 0.6; Where R is the number of iterations, r is the parameter, ▽ x and y Represents horizontal and vertical gradient operations respectively, represents the curve parameter graph, ξ represents the color channel; Among them J p represents the average intensity value of p channels in the enhanced image, J q represents the average intensity value of q channels in the enhanced image, and μ represents a pair of channels (p, q); Where K is the number of local regions, i is the i-th local region, Ω(i) is the four adjacent regions above, below, left, and right of the local region i as the center; Y i represents the average intensity value of the i-th local area in the enhanced image, Y j Represents the average intensity value of the jth local area in the enhanced image, I i Represents the average intensity value of the i-th local area in the input image, I j Represents the average intensity value of the jth local area in the input image.

5. The semantically guided diffusion model restoration method for extremely overexposed images according to claim 1, characterized in that: The overexposed image is input into the trained overexposed image brightness correction network model to obtain the corrected image I c : to I c After semantic segmentation and brightness extraction operations, the corresponding semantic map S and the mask M of the brightness saturated area are obtained; The semantic map S is obtained by a general semantic segmentation model Oneformer; the mask M of the brightness saturated area is extracted by first calculating the brightness value of each pixel. When the brightness value is greater than the threshold H, the pixel is considered to belong to the brightness saturated area.

6. The semantically guided diffusion model restoration method for extremely overexposed images according to claim 1, characterized in that: The brightness saturation completion network model includes variational autoencoder VAE, denoising U-Net, semantic control network and CLIP text encoder: Variational autoencoder VAE processes the corrected image I c to generate latent features, which are then subjected to random noise processing to generate noisy latent features z t ; To ensure that the brightness oversaturated part of the restored image is semantically consistent with the original features, the semantic map S is input into the semantic control network to extract the semantic condition c, which guides the pre-trained noise prediction network ∈ θ The noise added to the latent features is predicted; in addition, the CLIP text encoder extracts the text guidance condition y, which links the prompt word to the brightness of the generated image; the guided noise prediction is expressed as: where k is a factor that adjusts the strength of the guidance, and t is the number of time steps. Indicates that the boot condition is empty; After noise prediction, the DDIM sampler is used to speed up the sampling process and gradually denoise to obtain the denoised feature z0: where η is usually set to 0, α and σ are predefined hyperparameters, and ∈ represents random Gaussian noise.

7. The semantically guided diffusion model restoration method for extremely overexposed images according to claim 6, characterized in that: In order to complete the restoration, the predicted z0 is passed through the decoder D to obtain the image generated by the brightness saturation completion network model Then the final image I is synthesized using the following formula o , the corrected image I is divided into regions under the guidance of the mask M of the brightness saturation region c Image generated by the brightness saturation completion network model To perform the fusion:

8. A terminal device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the restoration method for extremely overexposed images using a semantically guided diffusion model as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the method for repairing an extremely overexposed image using a semantically guided diffusion model as described in any one of claims 1 to 7.

Images