Method and apparatus for hdr lighting enhancement based on lighting scene

Abstract

The application relates to a method and device for HDR light enhancement based on a light scene, which comprises the following steps: classifying a light scene of an original image by using a pre-constructed MLP network classification model to obtain a light scene type; the light scene type comprises a difficult scene, a medium scene, a simple scene, a normal scene and a backlight scene; performing exposure enhancement processing on the original image according to the light scene type to obtain a group of exposure enhancement images; synthesizing the original image and the exposure enhancement images to obtain an HDR result image; and performing histogram equalization processing on the HDR result image to generate an HDR exposure enhancement image. The application classifies the light scene of an input image, completes HDR light enhancement on the classified image, obtains a light enhancement result, classifies the light scene, can perform more refined light enhancement on different light scenes, thereby obtaining a better output result, and has good universality.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a method and device for HDR light enhancement based on a light scene. BACKGROUND

[0002] One difficulty in light enhancement of an image is how to make the algorithm applicable to a variety of light scenes. The image content is obviously different between different scenes, and the light conditions are not the same. Therefore, in order to match as many light scenes as possible, the light enhancement algorithm is required to have sufficient intelligence; on the other hand, the algorithm is mainly applied to mobile phones, tablets and other devices, which puts higher requirements on the efficiency of the algorithm.

[0003] In related technologies, the development of image light enhancement algorithms at the present stage mainly focuses on the field of deep neural networks. However, the algorithms in this field that meet the above intelligence requirements are usually bound to a network model with a large number of parameters, which has high time and memory costs during running, greatly limiting the possibility of transplanting the algorithm to mobile phones, tablets and other devices. Secondly, traditional algorithms commonly used for light enhancement, such as high dynamic range imaging (HDR), are more suitable for mobile phones, tablets and other devices in terms of efficiency, but they are only applicable to a few light scenes under a specific algorithm configuration. Therefore, if the algorithm is to successfully process images of other light scenes, the algorithm configuration must be adjusted accordingly, which is a time-consuming task that greatly increases the time cost of the editor.

[0004] In addition, traditional HDR light enhancement algorithms often require a set of highly consistent images, except for a certain difference in exposure, which limits the HDR light enhancement of a single image and has poor universality. SUMMARY

[0005] Therefore, the purpose of the present application is to overcome the shortcomings of the prior art and provide a method and device for HDR light enhancement based on a light scene to solve the problem of poor universality of HDR light enhancement of a single image in the prior art.

[0006] To achieve the above purpose, the present application adopts the following technical solution: a method for HDR light enhancement based on a light scene, comprising:

[0007] using a pre-constructed MLP network classification model to classify the light scene of the original image to obtain a light scene type; the light scene type includes a difficult scene, an intermediate scene, a simple scene, a normal scene and a backlit scene;

[0008] According to the light scene type, different degrees of exposure enhancement processing are performed on the original image to obtain a set of exposure enhancement images;

[0009] The original image and the set of exposure enhancement images are synthesized to obtain an HDR result image;

[0010] The HDR result image is subjected to histogram equalization processing to generate an HDR exposure enhancement image.

[0011] Further, the light scene classification of the original image using the pre-constructed MLP network classification model comprises:

[0012] The original image is converted into a gray image;

[0013] According to the gray value of each pixel in the gray image, the mean and standard deviation of all pixel gray values are calculated;

[0014] The mean, standard deviation, and gray histogram of the gray image are input into the MLP network classification model to output the light scene type of the original image.

[0015] Further, the exposure enhancement processing of the original image according to the light scene type to obtain a set of exposure enhancement images comprises:

[0016] The HDR light enhancement parameters are determined according to the light scene type;

[0017] A set of exposure images are generated by combining the HDR light enhancement parameters and the original image, and the ideal exposure is obtained by aligning the set of exposure images;

[0018] The linear estimation parameters of the ideal exposure and the exposure of the exposure image are determined;

[0019] Based on the linear estimation parameters, the set of exposure images and the original image are synthesized to obtain a set of exposure enhancement images.

[0020] Further, the synthesis of the set of exposure images and the original image based on the linear estimation parameters to obtain a set of exposure enhancement images comprises:

[0021] The average values of the RGB three color components of the original image are calculated by traversing the pixels of the set of exposure images and the original image;

[0022] Based on the average values and the preset color median, the weight values are calculated;

[0023] The updated pixel values of the original image are calculated using the linear estimation parameters;

[0024] Based on the updated pixel values and the weight values, new pixel values are calculated to obtain a set of exposure enhancement images.

[0025] Further, further comprising:

[0026] Correcting color components in the updated pixel values that exceed a preset value.

[0027] Further, further comprising:

[0028] Using a tone mapping algorithm to tone map the new pixel values to obtain a set of image color values of the exposure enhanced images.

[0029] Further, the HDR light enhancement parameters include:

[0030] Exposure intensity distribution rules, the number of exposure surrounding images, noise resistance factors, local contrast enhancement factors, HDR enhancement intensities, and tone mapping curves.

[0031] The embodiment of the application provides a device for HDR light enhancement based on a light scene, which comprises:

[0032] A classification module is configured to classify a light scene of an original image by using a pre-constructed MLP network classification model to obtain a light scene type; the light scene type includes a difficult scene, a medium scene, a simple scene, a normal scene and a backlight scene.

[0033] An enhancement module is configured to perform exposure enhancement processing on the original image in different degrees according to the light scene type to obtain a set of exposure enhanced images.

[0034] A synthesis module is configured to synthesize the original image and the set of exposure enhanced images to obtain an HDR result image.

[0035] A processing module is configured to perform histogram equalization processing on the HDR result image to generate an HDR exposure enhanced image.

[0036] The technical scheme of the application can achieve the beneficial effects including:

[0037] The application provides a method and device for HDR light enhancement based on a light scene, which classifies a light scene of an input image, performs HDR light enhancement on the classified image, and obtains a result after light enhancement. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only only some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative work based on these drawings also belong to the protection scope of the present application.

[0039] Figure 1 The steps of the method for performing HDR lighting enhancement based on a lighting scene according to the present application are shown in the flowchart.

[0040] Figure 2 The flowchart of the method for performing HDR lighting enhancement based on a lighting scene according to the present application is shown in the flowchart.

[0041] Figure 3 The flowchart of the image lighting scene classification provided by the present application is shown in the flowchart.

[0042] Figure 4 The flowchart of the HDR lighting enhancement provided by the present application is shown in the flowchart.

[0043] Figure 5 The structure diagram of the device for performing HDR lighting enhancement based on a lighting scene according to the present application is shown in the structure diagram. DETAILED DESCRIPTION

[0044] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only only some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative work based on these drawings also belong to the protection scope of the present application.

[0045] The specific method and device for performing HDR lighting enhancement based on a lighting scene provided in the embodiments of the present application will be introduced below in combination with the accompanying drawings.

[0046] As shown in the method for performing HDR lighting enhancement based on a lighting scene provided in the embodiments of the present application, the method comprises: Figure 1

[0047] S101, performing lighting scene classification on the original image by using a pre-constructed MLP network classification model to obtain a lighting scene type; the lighting scene type comprises a difficult scene, a medium scene, a simple scene, a normal scene and a backlit scene;

[0048] ​It can be understood that five categories of illumination scenes are provided in the present application, which are difficult, medium, simple, normal and backlight. Among them, difficult is a relatively dark and black image that is difficult to see light, for example, an image taken at night; medium is an image that can see a small part of light, for example, an image taken in cloudy weather, with some light in the dark sky; simple type is an image containing a small part of light, only a small part of the image needs to be enhanced by light; normal type is an image with normal light, for example, an image taken on a sunny day; and backlight type is an image with objects behind the light, for example, an image taken by the photographer facing the light.

[0049] S102, performing different degree of exposure enhancement processing on the original image according to the illumination scene type to obtain a set of exposure enhancement images;

[0050] It can be understood that, according to the different illumination scene types, the present application selects the corresponding HDR light enhancement processing, so as to obtain exposure enhancement images with different processing.

[0051] S103, synthesizing the original image and the set of exposure enhancement images to obtain an HDR result image;

[0052] Specifically, the pixels of the original image and the exposure enhancement image are synthesized to obtain the HDR result image.

[0053] S104, performing histogram equalization processing on the HDR result image to generate an HDR exposure enhancement image.

[0054] Specifically, the HDR result image of the present application uses CLAHE for histogram equalization processing, enhances image details, suppresses noise signals, makes the image clearer, and generates the final HDR exposure enhancement image.

[0055] The working principle of the method for HDR light enhancement based on illumination scene is as follows: Figure 2 As shown in the figure, the present application first classifies the image according to the illumination scene, so as to give the algorithm a certain degree of intelligence at a very low cost, and then according to the classification result, the image is subjected to light enhancement processing according to the pre-set HDR enhancement algorithm configuration that fits the image illumination scene.

[0056] In some embodiments, the illumination scene classification of the original image by using the pre-constructed MLP network classification model comprises:

[0057] Converting the original image into a gray image;

[0058] According to the gray value of each pixel in the gray image, the mean value and the standard deviation of all pixel gray values are calculated;

[0059] The mean value, standard deviation and gray scale histogram of the gray scale image are input into the MLP network classification model, and the light scene type of the original image is output.

[0060] The present application can easily observe the statistical characteristics of image pixels under different classifications, such as mean value and standard deviation, according to the sample images of each classification. In terms of mean value, the relationship of difficult<medium<simple<normal; in terms of standard deviation, the back light classification is obviously higher than other classifications. However, the statistical characteristics are overall, and under a certain classification, it is inevitable that part of the individuals do not match the statistical characteristics of the whole, so it is not appropriate to use only mean value and standard deviation as the judgment basis for a single individual in the present application.

[0061] As shown in Figure 3 As a specific embodiment, the present application also introduces a histogram statistical algorithm to count the frequency of pixel gray scale values in the image. Since the gray scale can be divided into 256 levels, the algorithm returns a one-dimensional array with a length of 256, which is also called a gray scale histogram. The kth element of the gray scale histogram has an index value of k-1, and the element records the number of pixels with a gray scale value equal to k-1 in the image.

[0062] The MLP network classification model constructed by the present application is a simple MLP (Multi-Layer Perceptron) network for processing light scene classification tasks. The network input is an N*1*258 tensor, where N represents the batch size of the network, i.e. the number of images, and 1*258 represents that a single image corresponds to a one-dimensional array with a length of 258. The first 256 bits come from the gray scale histogram of the image and have been normalized, as shown in equation (1); the last two bits are equal to the mean value and standard deviation of all pixel gray scale values in the image, as shown in equations (2) and (3).

[0063]

[0064] In equation (1), input represents the input array corresponding to the image, h represents the gray scale histogram of the image, and [i] represents the ith+1 element of the array. [j] is the same.

[0065]

[0066]

[0067] In equations (2) and (3), input represents the input array corresponding to the image,

[256] and

[257] represent the 257th and 258th elements of the array, respectively, gray represents the gray scale image, which records the gray scale values of each pixel in the original image, and the width and height are the same as the original image. Figure 1where gray[i][j] represents the gray value of the i-th row and j-th column of the gray image, and w and h represent the width and height of the image, respectively.

[0068] The network output is an N*1*5 tensor, where N represents the batch size, 1*5 represents that the output of a single image is a one-dimensional array with a length of 5, and the five elements of the array correspond to the five categories of the illumination scene, respectively. The value of each element represents the probability that the image belongs to the classification. The greater the value, the more likely the image belongs to this category. Therefore, the category corresponding to the element with the largest value is the final classification to which the image belongs.

[0069] In some embodiments, the different degrees of exposure enhancement processing of the original image according to the illumination scene type to obtain a set of exposure enhanced images comprises:

[0070] determining an HDR illumination enhancement parameter according to the illumination scene type;

[0071] generating a set of exposure images in combination with the HDR illumination enhancement parameter and the original image, and performing alignment processing on the set of exposure images to obtain an ideal exposure;

[0072] determining a linear estimation parameter of the ideal exposure and the exposure of the exposure image;

[0073] synthesizing the set of exposure images and the original image based on the linear estimation parameter to obtain a set of exposure enhanced images.

[0074] As an embodiment, the synthesizing the set of exposure images and the original image based on the linear estimation parameter to obtain a set of exposure enhanced images comprises:

[0075] traversing the pixels of the set of exposure images and the original image, and calculating the average value of the RGB three color components of the original image;

[0076] calculating a weight value based on the average value and a preset color median value;

[0077] calculating the updated pixel value of the original image using the linear estimation parameter;

[0078] calculating a new pixel value based on the updated pixel value and the weight value to obtain a set of exposure enhanced images.

[0079] In some embodiments, the method further comprises:

[0080] correcting the color component in the updated pixel value that exceeds the preset value.

[0081] In some embodiments, the method further comprises:

[0082] The new pixel value is subjected to tone mapping by using a tone mapping algorithm to obtain image color values of a set of exposure enhanced images.

[0083] Specifically, as shown in Figure 4 , first, appropriate HDR light enhancement parameters are determined according to the obtained image light scene result, wherein the HDR light enhancement parameters include exposure intensity distribution rules, the number of exposure surrounding images, an anti-noise factor, a local contrast enhancement factor, an HDR enhancement intensity, a tone mapping curve, etc.

[0084] Secondly, a corresponding set of exposure images is generated according to the determined exposure intensity distribution rules, and then the set of exposure images is subjected to alignment processing to obtain an aligned “ideal” exposure. For images of different exposure degrees, parameters A and B in formula (4) need to be determined so that the pixels in the image can be fitted to the ideal exposure parameters as much as possible:

[0085] Y i =AX i +B (4)

[0086] To solve the parameters, input the pixel sampling list X[n] of the exposure image, the pixel sampling list Y[n] of the “ideal” exposure image, and the weight list W[n] for preventing overexposure. Then the following equation group of formula (5) is obtained:

[0087]

[0088] Set the minimum estimated error value to solve the equation group to obtain the relationship between the bright and dark parts in different exposure images and the “ideal” exposure image. The same processing is performed on all exposure images to obtain a set of pixel value estimation linear relationship parameters A[n] and B[n].

[0089] Then, the different exposure degree images and the original image are synthesized, the linear relationship obtained above is used, and the corresponding weight value is set to traverse and adjust the value of the result pixel. The calculation for each pixel in the image is as follows:

[0090] The pixels of the generated different exposure degree images and the original image are traversed, and the average value AVG i of the RGB three components of the image P i is calculated, as shown in the following formula 6:

[0091]

[0092] Then, the color component average value AVG i of the original image is compared with the set color median value M, and the weight value W i is calculated by using formulas (7) and (8), wherein S is a color protection range with certain generalization that is adjusted according to experience:

[0093] Diff i = abs(AVG i -M) (7)

[0094]

[0095] A new pixel value P NEW is calculated using the linear relationship parameters A[n] and B[n], as shown in equation (9):

[0096] P NEW i = A i *P i +B i (9)

[0097] If the color component in P NEW exceeds 255, the following equation (10) and equation (11) need to be corrected:

[0098]

[0099]

[0100] Then the P NEW of the image set can be used to calculate the pixel value H of the new image using equation 12:

[0101]

[0102] Since the final synthesized H is a high dynamic range (HDR) image, in order to adapt it to a low dynamic range (LDR) display, a tone mapping algorithm needs to be used to map it to the standard range of the LDR image. We can choose an algorithm suitable for the image lighting scene from classic tone mapping algorithms (such as Reinhard, Filmic, ACES, etc.) to perform tone mapping to obtain the final image color value. The above calculation process can be calculated using a device GPU.

[0103] As Figure 5 shown, an embodiment of the present application provides a device for HDR lighting enhancement based on a lighting scene, comprising:

[0104] A classification module 201 is configured to classify the original image using a pre-constructed MLP network classification model to obtain a lighting scene type; the lighting scene type includes a difficult scene, a medium scene, a simple scene, a normal scene, and a backlight scene.

[0105] An enhancement module 202 is configured to perform exposure enhancement processing on the original image according to the lighting scene type to obtain a set of exposure enhancement images.

[0106] The synthesis module 203 is configured to synthesize the original image and the set of exposure enhanced images to obtain an HDR result image.

[0107] The processing module 204 is configured to perform histogram equalization processing on the HDR result image to generate an HDR exposure enhanced image.

[0108] The working principle of the device for performing HDR light enhancement based on a light scene provided by the embodiment of the present application is that the classification module 201 performs light scene classification on an original image by using a pre-constructed MLP network classification model to obtain a light scene type; the light scene type includes a difficult scene, a medium scene, a simple scene, a normal scene, and a backlight scene; the enhancement module 202 performs exposure enhancement processing on the original image to different degrees according to the light scene type to obtain a set of exposure enhanced images; the synthesis module 203 synthesizes the original image and the set of exposure enhanced images to obtain an HDR result image; and the processing module 204 performs histogram equalization processing on the HDR result image to generate an HDR exposure enhanced image.

[0109] In summary, the present application provides a method and a device for performing HDR light enhancement based on a light scene, the method including performing light scene classification on an original image by using a pre-constructed MLP network classification model to obtain a light scene type; the light scene type includes a difficult scene, a medium scene, a simple scene, a normal scene, and a backlight scene; performing exposure enhancement processing on the original image to different degrees according to the light scene type to obtain a set of exposure enhanced images; synthesizing the original image and the set of exposure enhanced images to obtain an HDR result image; and performing histogram equalization processing on the HDR result image to generate an HDR exposure enhanced image. The present application classifies the light scene of an input image, performs HDR light enhancement on the classified image, obtains a light enhanced result, classifies the light scene, and performs more refined light enhancement on different light scenes to obtain a better output result, thereby having good universality.

[0110] It can be understood that the method embodiments provided above correspond to the device embodiments described above, and the corresponding specific contents can be mutually referred to, which will not be described herein again.

[0111] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product in the form of one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program codes.

[0112] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0113] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0114] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0115] The above description is only specific embodiments of the application, but the protection scope of the application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered by the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A method for HDR lighting enhancement based on lighting scenes, characterized in that, include: The original image is classified into lighting scenes using a pre-built MLP network classification model to obtain the lighting scene type; The lighting scene types include difficult scenes, medium scenes, simple scenes, normal scenes, and backlit scenes; Determine HDR lighting enhancement parameters based on the type of lighting scene; A set of exposure images is generated by combining the HDR illumination enhancement parameters and the original image, and the set of exposure images is aligned to obtain the ideal exposure. Determine the linear estimation parameters between the ideal exposure and the exposure of the exposed image; Iterate through the pixels of a set of exposed images and the original image, and calculate the average value of the three color components of the RGB of the original image; Calculate the weight value based on the average value and the preset color median; The updated pixel values ​​of the original image are calculated using the linear estimation parameters; New pixel values ​​are calculated based on the updated pixel values ​​and weight values, resulting in a set of exposure-enhanced images; The original image and the set of exposure-enhanced images are combined to obtain the HDR result image; The HDR result image is subjected to histogram equalization to generate an HDR exposure-enhanced image.

2. The method according to claim 1, characterized in that, The method of classifying the lighting scene of the original image using a pre-built MLP network classification model includes: Convert the original image to a grayscale image; Based on the grayscale values ​​of each pixel in the grayscale image, calculate the mean and standard deviation of the grayscale values ​​of all pixels; The mean, standard deviation, and grayscale histogram of the grayscale image are input into the MLP network classification model, which outputs the lighting scene type of the original image.

3. The method according to claim 1, characterized in that, Also includes: Correct color components in the updated pixel values ​​that exceed preset values.

4. The method according to claim 3, characterized in that, Also includes: A tone mapping algorithm is used to tone map the new pixel values ​​to obtain a set of image color values ​​for the exposure-enhanced image.

5. The method according to claim 1, characterized in that, The HDR illumination enhancement parameters include: Exposure intensity distribution rules, number of exposure-bound images, noise reduction factor, local contrast enhancement factor, HDR enhancement intensity, and tone mapping curve.

6. A device for HDR lighting enhancement based on lighting scenes, characterized in that, include: The classification module is used to classify the lighting scene of the original image using a pre-built MLP network classification model to obtain the lighting scene type; The lighting scene types include difficult scenes, medium scenes, simple scenes, normal scenes, and backlit scenes; The enhancement module is used to determine HDR lighting enhancement parameters based on the type of lighting scene; A set of exposure images is generated by combining the HDR illumination enhancement parameters and the original image, and the set of exposure images is aligned to obtain the ideal exposure. Determine the linear estimation parameters between the ideal exposure and the exposure of the exposed image; Iterate through the pixels of a set of exposed images and the original image, and calculate the average value of the three color components of the RGB of the original image; Calculate the weight value based on the average value and the preset color median; The updated pixel values ​​of the original image are calculated using the linear estimation parameters; New pixel values ​​are calculated based on the updated pixel values ​​and weight values, resulting in a set of exposure-enhanced images; The compositing module is used to synthesize the original image and the set of exposure-enhanced images to obtain an HDR result image; The processing module is used to perform histogram equalization on the HDR result image to generate an HDR exposure-enhanced image.

Images