A method, device and storage medium for enhancing specular highlight images in real scenes

Abstract

The present invention discloses a method, apparatus and storage medium for enhancing the specular highlight image in a real scene. The method includes: obtaining the depth information of an image through color attenuation prior, and obtaining an initial unrefined transmission rate based on the depth information; constructing first and second adaptive transmission rate constraints for the initial transmission rate, and using the optimized transmission rate as the input of an L0 gradient minimum filter; using the L0 gradient minimum filter to optimize the transmission rate, obtaining an image after removing the specular highlight component, and performing illumination compensation on it; constructing a new illumination compensation function by combining the inverse trigonometric function and gamma correction to perform illumination compensation in the YUV space, enhancing the detail information of the image, expanding the dynamic range of the image, and increasing the saturation of the image. The apparatus includes: a processor and a memory. The present invention effectively solves the problem of loss of texture information in the area occluded by specular highlights in the image, and at the same time expands the dynamic range of the image and increases the saturation of the image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method, device and storage medium for enhancing a specular highlight image of a real scene. Background Art

[0002] In real-world scenes, when light is projected onto the surface of an object, it can be observed that most objects usually exhibit two types of reflections, namely diffuse reflection that reflects the surface information of the object and specular reflection that reflects the light source information.

[0003] Due to certain shooting angles and factors such as the object's surface being too smooth, objects usually produce relatively strong mirror reflections locally or even globally under lighting, resulting in the loss of image edge, texture and other detail information, which can cause errors in image segmentation, target detection and matching, and affect the accuracy and robustness of the algorithm.

[0004] Therefore, the study of highlight image enhancement methods is essential for the successful execution of advanced vision tasks. Summary of the invention

[0005] In view of the problem of texture information loss in specular highlight images in real scenes, the present invention proposes a specular highlight image enhancement method, device and storage medium. The specular highlight image processed by the present invention has significantly enhanced edge contrast and can retain more detail features than the original image; the present invention effectively solves the problem of texture information loss in the area blocked by the highlight in the image, while expanding the dynamic range of the image and increasing the saturation of the image, as described below:

[0006] In a first aspect, a method for enhancing a specular highlight image of a real scene is provided, the method comprising the following steps:

[0007] The depth information of the image is obtained through the color attenuation prior, and the initial unrefined transmission rate is obtained based on the depth information;

[0008] Constructing the first and second adaptive transmission rate constraints for initial transmission rates, and using the optimized transmission rates as inputs to the L0 gradient minimum filter;

[0009] The transmission rate is optimized using the L0 gradient minimization filter, the image after eliminating the highlight component is obtained, and the illumination compensation is performed;

[0010] A new illumination compensation function is constructed by combining inverse trigonometric functions and gamma correction to perform illumination compensation in YUV space, enhance image detail information, expand image dynamic range, and increase image saturation.

[0011] Wherein, the first adaptive transmission rate is:

[0012]

[0013] Among them, t1(x,y) is the optimized transmission rate, R is the red channel, G is the green channel, B is the blue channel, and I c (x, y) is the original highlight image, C is any channel among the three channels of R, G, and B, and A is the atmospheric light value;

[0014] The second adaptive transmission rate is:

[0015] t2(x,y)=t(x,y) z(x,y)

[0016] Among them, t(x,y) is the original transmission rate obtained from the depth map, and t2(x,y) is the adjusted transmission rate. z is the adjustment factor. Combining the two adaptive constraints, we get the transmission rate t3 after the constraint:

[0017] max(t1(x,y),t2(x,y))≤t3(x,y)≤1.

[0018] The method of combining the inverse trigonometric function and the gamma correction to construct a new illumination compensation function for illumination compensation in the YUV space is as follows:

[0019] Combining the inverse trigonometric function and gamma correction to construct a new lighting compensation function, for the point (x, y), the gamma correction value is:

[0020]

[0021] in, is the result of normalization of Y0, the value of b is 0 to 0.5, c is the operator for adjusting the gamma correction amplitude, and a is the stretching operator.

[0022]

[0023] Among them, mean is the mean operation;

[0024] Apply the gamma correction function to Y to obtain the enhanced illumination component Y1:

[0025]

[0026] Among them, Y(x,y) is the Y space after the initial image is transformed.

[0027] In a second aspect, a device for enhancing a specular highlight image in a real scene is provided, the device comprising:

[0028] An acquisition module, used for acquiring depth information of the image through color attenuation prior, and acquiring an initial unrefined transmission rate based on the depth information;

[0029] A construction module, used to construct the first and second adaptive transmission rate constrained initial transmission rates, and use the optimized transmission rates as inputs of the L0 gradient minimum filter;

[0030] The module for eliminating highlight components and initial illumination compensation is used to optimize the transmission rate using the L0 gradient minimization filter, obtain the image after eliminating highlight components, and perform initial illumination compensation on it;

[0031] Secondly, the illumination compensation module combines the inverse trigonometric function and gamma correction to construct a new illumination compensation function to perform illumination compensation in the YUV space, enhance the image detail information, expand the image dynamic range, and increase the image saturation.

[0032] In a third aspect, a device for enhancing specular highlight images of real scenes is provided, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute the method steps described in the first aspect.

[0033] In a fourth aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method steps described in the first aspect.

[0034] The beneficial effects of the technical solution provided by the present invention are:

[0035] 1. For images containing highlight areas, the text information covered by the highlight areas is often very blurry. After being processed by the present invention, the text information covered by the highlight areas is clearly distinguishable, thereby laying a good foundation for subsequent computer vision applications (text recognition or scene reconstruction, etc.);

[0036] 2. For images whose image quality is reduced due to the influence of highlights, after being processed by the present invention, the saturation of the image is increased and the image quality is improved;

[0037] 3. Images in real scene pictures often contain highlight areas and non-highlight areas. After being processed by the present invention, not only the text information in the highlight area is enhanced, but also the non-highlight area, especially the low-illuminance area, is improved, which greatly expands the dynamic range of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of a method for enhancing specular highlights in real scenes;

[0039] Figure 2 is a schematic diagram of a specular highlight image;

[0040] Figure 3 For Figure 2 Schematic diagram of the processed target image;

[0041] Figure 4 It is a structural schematic diagram of a device for enhancing specular highlights in real scenes;

[0042] Figure 5 Another structural schematic diagram of a device for enhancing specular highlight images in real scenes. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0044] Example 1

[0045] The embodiment of the present invention provides a method for enhancing a specular highlight image in a real scene, see Figure 1 , the method comprises the following steps:

[0046] Step 101: Use color attenuation prior to quickly obtain depth information of the image, and obtain an initial unrefined transmission rate based on the depth information;

[0047] Step 102: The specular highlight phenomenon in real scenes often occurs under low illumination conditions. Two adaptive adjustment factors are proposed to constrain and improve the initial transmission rate to prevent the image from darkening and losing details. The optimized transmission rate is used as the input of the L0 gradient minimum filter in step 103;

[0048] Step 103: To prevent halo artifacts caused by clustering of highlight pixels, an L0 gradient minimization filter is used to optimize the transmission rate to obtain an image after eliminating highlight components. Due to uneven lighting conditions, illumination compensation is finally performed on the image after eliminating highlights.

[0049] Step 104: Local specular highlight images mostly occur under normal illumination and low illumination conditions. A new illumination compensation function is constructed by combining inverse trigonometric functions and gamma correction to perform illumination compensation in the YUV space, thereby enhancing the image detail information, expanding the image dynamic range, and increasing the image saturation.

[0050] In summary, the embodiment of the present invention effectively solves the problem of texture information loss in the area blocked by highlights in the image through the above steps 101 to 104, while expanding the dynamic range of the image and increasing the saturation of the image.

[0051] Example 2

[0052] The scheme in Example 1 will be further introduced below in combination with specific calculation formulas and examples, as described below for details:

[0053] Step 201: The color attenuation prior is a linear model describing the correlation between scene depth and image saturation and brightness difference:

[0054] d(x,y)=θ0+θ1v(x,y)+θ2s(x,y)+ε(x,y) (1)

[0055] Among them, v and s are the brightness and saturation of the image respectively; θ0, θ1, θ2 are constant parameters of the model; ε represents the random variable of the model error, x, y are the image coordinates, and d is the image scene depth.

[0056] The present invention implements the method of quickly obtaining scene depth information through color attenuation prior and substituting it into the atmospheric reflection model:

[0057] I(x)=J(x)t(x)+A(1-t(x)) (2)

[0058] t(x)=e -βd(x) (3)

[0059] This results in the initial unrefined transmission rate.

[0060] Among them, I(x) is the original image, J(x) is the image after removing highlights, A is the atmospheric light value, β is the medium scattering constant, and d(x) is the scene depth.

[0061] Step 202: The local highlight image is not a degradation model that degrades uniformly like the fog image. It contains large highlight areas. The color attenuation prior uses a fixed β value to restore the image pixel value, which will lead to an underestimation of the transmission rate in the highlight area and an overestimation of the transmission rate in the normal illumination area. Two adaptive adjustment factors are defined to constrain the perfect transmission rate.

[0062] First, the first constraint is obtained from the atmospheric scattering model:

[0063]

[0064] Among them, t1(x,y) is the optimized transmission rate, R is the red channel, G is the green channel, B is the blue channel, and I c (x, y) is the original highlight image, and C is any one of the three channels R, G, and B.

[0065] The second constraint comes from the definition of the transmission rate. The smaller the transmission rate, the more highlight components will be removed. Therefore, a low transmission rate needs to be designed for the highlight area and a higher transmission rate needs to be designed for the area with normal illumination. Therefore, the embodiment of the present invention designs a nonlinear function to re-modify the initial transmission rate. First, the adjustment factor z is defined, which is:

[0066]

[0067] Among them, t(x,y) is the original transmission rate obtained from the depth map, and the final transmission rate after adjustment by the nonlinear factor is:

[0068] t2(x,y)=t(x,y) z(x,y) (6)

[0069] Among them, t is the original transmission rate, and t2 is the adjusted transmission rate. The transmission rate of the highlight part is very low. After modification, the transmission rate becomes lower, which better removes the highlight component; the transmission rate of the normal illumination and low illumination area becomes higher, which avoids the darkening of pixels. Combining the two adaptive constraints, the transmission rate t3 after the constraint is obtained:

[0070] max(t1(x,y),t2(x,y))≤t3(x,y)≤1 (7)

[0071] Step 203: For a specular highlight image with a large area of ​​highlight pixels, using only guided filtering based on local calculation cannot eliminate the clustering phenomenon of pixels, and the processed image will still have a halo effect. Therefore, the present invention implements the transmission rate after adaptive constraint using the L0 gradient minimization filter;

[0072] The adaptively constrained transmission rate t3 is used as the input q of the L0 gradient minimization filter. Considering the output gradient value according to the input image t3, the transmission rate t3 is first assigned to s, and then the auxiliary variables hp and vp are calculated.

[0073]

[0074] Among them, λ is the smoothing parameter, η is the automatic adjustment parameter, the initial value of η is 2, and iterates a certain number of times until η=η max , S p For each pixel, the color difference between adjacent pixels along the X and Y directions.

[0075] The final output s is obtained by formula (9):

[0076]

[0077] Here, F refers to the fast Fourier transform operator, F* refers to the complex conjugate operator, F(1) refers to the Fourier transform of the function, and F(q) refers to the Fourier transform of the input image. This is repeated a certain number of times until η equals η max In the embodiment of the present invention, λ=0.005, η=2, η max =10 5 .

[0078] After refinement, the output of the L0 gradient minimization filter is taken as the final transfer rate t4, and the scene brightness J is restored as follows:

[0079]

[0080] Among them, I(x,y) is the initial highlight image.

[0081] Step 204: using non-uniform illumination compensation in the YUV space;

[0082] The general gamma correction refines the input image through a power exponent γ. For a grayscale image I, the gamma correction output is O(x,y) for each pixel I(x,y);

[0083]

[0084] Among them, YUV is a brightness-chrominance color space.

[0085] However, local highlight images often occur under normal illumination and low illumination conditions. The saturation and brightness of the local area of ​​the image are too high. Ordinary gamma correction will reduce or increase the brightness of the image as a whole, causing the image to lose details. Therefore, the embodiment of the present invention considers local gamma correction. The exponent of the local gamma correction will not be a constant, but an equation that depends on the point (x, y) and is related to its neighboring pixel point N (x, y). Then equation (11) becomes:

[0086]

[0087] The embodiment of the present invention first converts the RGB image into the YUV space to obtain the brightness component Y of the image; then, in order to reduce the halo phenomenon, a guided filter is applied to the Y component to obtain the final brightness component Y0.

[0088] For local highlight images, in order to display the highlight area information without losing the area information of normal illumination, for normal illumination and low illumination areas, γ should be 1 or a constant less than 1; for highlight areas, γ should be a large constant. Therefore, the embodiment of the present invention combines inverse trigonometric functions and gamma correction to construct a new illumination compensation function. For point (x, y), the gamma correction value obtained by the embodiment of the present invention is:

[0089]

[0090] in, It is the result after Y0 is normalized. The value of b is usually 0-0.5. For local highlight images with lower ambient illumination, the embodiment of the present invention sets a higher b value to enhance the ambient brightness. c is an operator for adjusting the gamma correction amplitude, which is usually set to 0.1.

[0091] a is a stretching operator, which is used to adjust the stretching amplitude of the highlight image. The calculation method of a is as follows:

[0092]

[0093] Among them, mean is the mean operation. It can be obtained that the larger the mean of the highlight image, the smaller the a value is, and the corresponding highlight area is stretched more, which can show more detail information.

[0094] The gamma correction function of the embodiment of the present invention is applied to Y to obtain the enhanced illumination component Y1:

[0095]

[0096] Among them, Y(x,y) is the Y space after the initial image is transformed.

[0097] When converting the image from YUV space back to RGB space, in order to compensate for the change in saturation, the following formula is applied to the R, G, B components to obtain the final components R1, B1, G1:

[0098]

[0099] This results in the final restored image.

[0100] Example 3

[0101] The experimental object used in the embodiment of the present invention is a specular highlight image in a real scene. Figures 2 to 3 , the feasibility of the schemes in Examples 1 and 2 is verified, as described below:

[0102] Figure 2 An object whose surface is partially covered by highlights in a low-light environment. Due to the presence of highlights, the text information on the surface of the object is obscured, and the text is blurred and difficult to distinguish. At the same time, due to the low-light environment, the captured image has problems such as blurred content, severe noise, and loss of details. After the image is processed by the present invention, Figure 3 As shown, the text information on the surface of the object covered by the highlight becomes clearly discernible, indicating that the present invention can well enhance the area affected by the highlight. At the same time, due to the illumination compensation used in the embodiment of the present invention, the details around the object are displayed, the saturation of the image is enhanced, the clarity of the image is significantly improved, and the quality is comprehensively improved.

[0103] Example 4

[0104] A device for enhancing specular highlights in real scenes, see Figure 4 , the device comprises:

[0105] An acquisition module 1 is used to acquire depth information of an image through color attenuation priori, and acquire an initial unrefined transmission rate based on the depth information;

[0106] A construction module 2 is used to construct the first and second adaptive transmission rate constrained initial transmission rates, and use the optimized transmission rate as an input of the L0 gradient minimum filter;

[0107] The highlight component elimination and initial illumination compensation module 3 is used to optimize the transmission rate using the L0 gradient minimization filter, obtain the image after the highlight component is eliminated, and perform initial illumination compensation on it;

[0108] Again, the illumination compensation module 4 combines the inverse trigonometric function and the gamma correction to construct a new illumination compensation function to perform illumination compensation in the YUV space, enhance the image detail information, expand the image dynamic range, and increase the image saturation.

[0109] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0110] The execution subjects of the above modules can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and they are selected according to the needs of actual applications.

[0111] Example 5

[0112] A device for enhancing specular highlights in real scenes, see Figure 5 The device includes: a processor 5 and a memory 6, wherein the memory 6 stores program instructions, and the processor 5 calls the program instructions stored in the memory 6 to enable the device to execute the following method steps in Example 1:

[0113] The depth information of the image is obtained through the color attenuation prior, and the initial unrefined transmission rate is obtained based on the depth information;

[0114] Constructing the first and second adaptive transmission rate constraints for initial transmission rates, and using the optimized transmission rates as inputs to the L0 gradient minimum filter;

[0115] The transmission rate is optimized using the L0 gradient minimization filter, the image after eliminating the highlight component is obtained, and the illumination compensation is performed;

[0116] A new illumination compensation function is constructed by combining inverse trigonometric functions and gamma correction to perform illumination compensation in YUV space, enhance image detail information, expand image dynamic range, and increase image saturation.

[0117] The first adaptive transmission rate is specifically:

[0118] First adaptive transmission rate:

[0119]

[0120] Among them, t1(x,y) is the optimized transmission rate, R is the red channel, G is the green channel, B is the blue channel, and I c (x, y) is the original highlight image, C is any channel among the three channels of R, G, and B, and A is the atmospheric light value;

[0121] Second adaptive transmission rate:

[0122] t2(x,y)=t(x,y) z(x,y)

[0123] Among them, t(x,y) is the original transmission rate obtained from the depth map, and t2(x,y) is the adjusted transmission rate. z is the adjustment factor. Combining the two adaptive constraints, we get the transmission rate t3 after the constraint:

[0124] max(t1(x,y),t2(x,y))≤t3(x,y)≤1.

[0125] Among them, the new illumination compensation function is constructed by combining the inverse trigonometric function and gamma correction to perform illumination compensation in the YUV space as follows:

[0126] Combining the inverse trigonometric function and gamma correction to construct a new lighting compensation function, for the point (x, y), the gamma correction value is:

[0127]

[0128] in, is the result of normalization of Y0, the value of b is 0 to 0.5, c is the operator for adjusting the gamma correction amplitude, and a is the stretching operator.

[0129]

[0130] Among them, mean is the mean operation;

[0131] Apply the gamma correction function to Y to obtain the enhanced illumination component Y1:

[0132]

[0133] Among them, Y(x,y) is the Y space after the initial image is transformed.

[0134] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0135] The execution subjects of the above-mentioned processor 5 and memory 6 can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiment of the present invention does not limit the execution subject and it is selected according to the needs of actual applications.

[0136] The data signal is transmitted between the memory 6 and the processor 5 via the bus 7, which will not be described in detail in the embodiment of the present invention.

[0137] Example 6

[0138] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0139] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0140] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0141] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0142] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0143] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for enhancing specular highlight images in real scenes, characterized in that: The method comprises the following steps: The depth information of the image is obtained through the color attenuation prior, and the initial unrefined transmission rate is obtained based on the depth information; Constructing the first and second adaptive transmission rates to optimize the initial transmission rate, and using the optimized transmission rate as the input of the L0 gradient minimum filter; The transmission rate is optimized using the L0 gradient minimization filter, the image after eliminating the highlight component is obtained, and the illumination compensation is performed; Combining inverse trigonometric functions and gamma correction to construct a new illumination compensation function to perform illumination compensation in YUV space, enhance image detail information, expand image dynamic range, and increase image saturation; Wherein, the first adaptive transmission rate is: Among them, t1(x,y) is the optimized transmission rate, R is the red channel, G is the green channel, and B is the blue channel. I c (x, y) is the original highlight image, C is any channel among the three channels of R, G, and B, and A is the atmospheric light value; The second adaptive transmission rate is: t2(x,y)=t(x,y) z(x,y) Among them, t(x,y) is the original transmission rate obtained from the depth map, and t2(x,y) is the adjusted transmission rate. z is the adjustment factor. Combining the two adaptive optimizations, we get the optimized transmission rate t3: max(t1(x,y),t2(x,y))≤t3(x,y)≤1.

2. A method for enhancing a real scene specular highlight image according to claim 1, characterized in that: The new illumination compensation function constructed by combining the inverse trigonometric function and the gamma correction to perform illumination compensation in the YUV space is specifically: Combining the inverse trigonometric function and gamma correction to construct a new lighting compensation function, for the point (x, y), the gamma correction value is: in, is the result of normalization of Y0, the value of b is 0 to 0.5, c is the operator for adjusting the gamma correction amplitude, and a is the stretching operator. Among them, mean is the mean operation; Apply the gamma correction function to Y to obtain the enhanced illumination component Y1: Among them, Y(x,y) is the Y space after the initial image is transformed.

3. A device for enhancing specular highlights in real scenes, characterized in that: The device comprises: An acquisition module, used for acquiring depth information of the image through color attenuation prior, and acquiring an initial unrefined transmission rate based on the depth information; A construction module is used to construct a first and a second adaptive transmission rate to optimize the initial transmission rate, and use the optimized transmission rate as an input of the L0 gradient minimum filter; The module for eliminating highlight components and initial illumination compensation is used to optimize the transmission rate using the L0 gradient minimization filter, obtain the image after eliminating highlight components, and perform initial illumination compensation on it; Secondly, the illumination compensation module combines the inverse trigonometric function and gamma correction to construct a new illumination compensation function to perform illumination compensation in the YUV space, enhance the image details, expand the image dynamic range, and increase the image saturation; Wherein, the first adaptive transmission rate is: Among them, t1(x,y) is the optimized transmission rate, R is the red channel, G is the green channel, B is the blue channel, and I c (x, y) is the original highlight image, C is any channel among the three channels of R, G, and B, and A is the atmospheric light value; The second adaptive transmission rate is: t2(x,y)=t(x,y) z(x,y) Among them, t(x,y) is the original transmission rate obtained from the depth map, and t2(x,y) is the adjusted transmission rate. z is the adjustment factor. Combining the two adaptive optimizations, we get the optimized transmission rate t3: max(t1(x,y),t2(x,y))≤t3(x,y)≤1.

4. A device for enhancing specular highlights in real scenes, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method according to any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 2.

Images