A portrait enhancement processing method, apparatus, device, storage medium and product

Abstract

The embodiment of the present application provides a portrait enhancement processing method, device, equipment, storage medium and product. The technical scheme provided by the embodiment of the present application determines the background light color value and the foreground light color value according to the face region and the sclera region in the to-be-processed image, determines the reflection color value according to the difference between the original color value of the to-be-fused pixel point and the background light color value, determines the fusion color value according to the material point color value of the material pixel point, the reflection color value and the foreground light color value, and adds the background light color value to each fusion color value, so as to determine the target color value of each to-be-fused pixel point. The fusion of the target color value of each to-be-fused pixel point considers the influence of the foreground light color and the background light color on the portrait enhancement, the portrait enhancement effect can be adaptively adjusted following the change of the environment, the user does not need to manually adjust the portrait enhancement intensity, the flexible adjustment of the portrait enhancement effect is realized, and the portrait enhancement effect is effectively improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, and in particular to a portrait enhancement processing method and device, equipment, storage medium and product. BACKGROUND

[0002] With the rise of live streaming and short video services and the improvement of mobile device performance, the application of portrait enhancement technology based on computer vision and computer graphics is becoming more and more widespread. For example, in the video live streaming scenario, the application of portrait enhancement technology that fuses virtual makeup, beautification, stickers, special effects, etc. with the portrait is becoming more and more common.

[0003] Currently, the portrait enhancement processing scheme is generally based on the enhancement intensity configured by the user, that is, the portrait enhancement is performed according to the enhancement intensity set by the user. However, after the user completes the setting of the enhancement intensity, the portrait enhancement effect is fixed, and when the environment changes, the portrait enhancement effect remains in the set state, and the portrait enhancement effect is poor. SUMMARY

[0004] Embodiments of the present application provide a portrait enhancement processing method, device, equipment, storage medium and product to solve the technical problem of fixed portrait enhancement intensity and poor portrait enhancement effect in related technologies, flexibly adjust the portrait enhancement effect, and effectively improve the portrait enhancement effect.

[0005] In a first aspect, embodiments of the present application provide a portrait enhancement processing method, comprising:

[0006] determining a background light color value and a foreground light color value according to a face region and a sclera region in a to-be-processed image;

[0007] determining a reflection color value of each to-be-fused pixel point according to a difference between an original color value of the to-be-fused pixel point corresponding to an enhancement material in the to-be-processed image and the background light color value;

[0008] determining a fusion color value corresponding to each to-be-fused pixel point according to a material point color value of each material pixel point in the enhancement material, the reflection color value and the foreground light color value;

[0009] adding the background light color value to the fusion color value corresponding to each to-be-fused pixel point to obtain a target color value of each to-be-fused pixel point.

[0010] In a second aspect, embodiments of the present application provide a portrait enhancement processing device, comprising an ambient light determination module, a reflection light determination module, a color fusion module and a color determination module, wherein:

[0011] The ambient light determination module is configured to determine a background light color value and a foreground light color value according to a face region and a sclera region in the to-be-processed image.

[0012] The reflected light determination module is configured to determine a reflected color value of each to-be-fused pixel point according to a difference between an original color value of the to-be-fused pixel point corresponding to the enhancement material in the to-be-processed image and the background light color value.

[0013] The color fusion module is configured to determine a fusion color value corresponding to each to-be-fused pixel point according to a material point color value of each material pixel point in the enhancement material, the reflected color value, and the foreground light color value.

[0014] The color determination module is configured to add the background light color value to the fusion color value corresponding to each to-be-fused pixel point to obtain a target color value of each to-be-fused pixel point.

[0015] In a third aspect, an embodiment of the present application provides a portrait enhancement processing device, including a memory and one or more processors.

[0016] The memory is configured to store one or more programs.

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the portrait enhancement processing method in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a nonvolatile storage medium storing computer-executable instructions for performing the portrait enhancement processing method in the first aspect when executed by a computer processor.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product including a computer program stored in a computer-readable storage medium, and at least one processor of a device reads and executes the computer program from the computer-readable storage medium, so that the device performs the portrait enhancement processing method in the first aspect.

[0020] The embodiment of the present application determines the background light color value and the foreground light color value according to the face region and the sclera region in the to-be-processed image, determines the reflection color value according to the difference between the original color value of the to-be-fused pixel point and the background light color value, determines the fusion color value according to the material point color value of the material pixel point, the reflection color value and the foreground light color value, and adds the background light color value to each fusion color value, so as to determine the target color value of each to-be-fused pixel point. The fusion of the target color value of each to-be-fused pixel point considers the influence of the foreground light color and the background light color on the portrait enhancement, the portrait enhancement effect can be adaptively adjusted following the change of the environment, the user does not need to manually adjust the portrait enhancement intensity, the flexible adjustment of the portrait enhancement effect is realized, and the portrait enhancement effect is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a flowchart of a portrait enhancement processing method provided by the embodiment of the present application;

[0022] Figure 2 is a flowchart of another portrait enhancement processing method provided by the embodiment of the present application;

[0023] Figure 3 is a background light color value determination flowchart provided by the embodiment of the present application;

[0024] Figure 4 is a foreground light color value determination flowchart provided by the embodiment of the present application;

[0025] Figure 5 is a structural schematic diagram of a portrait enhancement processing device provided by the embodiment of the present application;

[0026] Figure 6 is a structural schematic diagram of a portrait enhancement processing device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purposes, technical solutions and advantages of the present application clearer, the following further describes specific embodiments of the present application with reference to the drawings. It can be understood that the specific embodiments described herein are merely used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only parts related to the present application are shown in the drawings, but not all. Before discussing the example embodiments in more detail, it should be mentioned that some example embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The above processes can be terminated when the operations are completed, but can also have additional steps not included in the drawings. The above processes can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0028] The portrait enhancement processing method provided by the present application can be applied to the beauty processing scene of the image, for example, collecting images in the live process or image shooting process, and adding makeup effects to the face in the collected images, realizing portrait enhancement processing of the image and synchronous change of the portrait enhancement effect following the environment light, improving the portrait enhancement fusion effect, aiming to add the foreground light color and the background light color reflected in the to-be-processed image into the generation process of the target color value of each to-be-fused pixel point, and the portrait enhancement effect can be adaptively adjusted following the change of the environment, without the need for the user to manually adjust the portrait enhancement intensity, realizing flexible adjustment of the portrait enhancement effect, and effectively improving the portrait enhancement effect. For the traditional portrait enhancement processing scheme, a slide bar for controlling the portrait enhancement intensity is generally used for configuration, and the user manually selects a portrait enhancement effect most consistent with his / her aesthetic sense, but after the user completes the setting of the portrait enhancement effect, the portrait enhancement effect is fixed, and when the environment changes, the portrait enhancement effect remains in the set state, the portrait enhancement effect is poor, for example, the virtual makeup is easily suspended in the background, the makeup and the real picture are split, and the makeup fusion effect is poor. Based on this, the portrait enhancement processing method provided by the present application is provided to solve the technical problem of poor portrait enhancement effect caused by the fixed portrait enhancement intensity in the existing portrait enhancement processing scheme.

[0029] Figure 1 A flowchart of the portrait enhancement processing method provided by the present application is given, and the portrait enhancement processing method provided by the present application can be executed by a portrait enhancement processing device, which can be realized by hardware and / or software, and integrated in a portrait enhancement processing apparatus.

[0030] The following describes the portrait enhancement processing method executed by the portrait enhancement processing device. Referring to Figure 1The human image enhancement processing method comprises the following steps:

[0031] S110: determining a background light color value and a foreground light color value according to a face region and a sclera region in the to-be-processed image.

[0032] The to-be-processed image provided by the scheme is an original image that needs to be enhanced (for example, adding a makeup effect to a face), and the to-be-processed image is a single image frame or multiple image frames (for example, a video stream). The to-be-processed image can be obtained through a video stream collected in real time, for example, in a live streaming scenario, a host end captures a video stream in real time through a camera module (for example, a camera), and each video frame in the video stream is taken as a to-be-processed image.

[0033] Among them, the ambient light is the light scattered in the shooting scene, which is related to the position and direction. The background light and the foreground light have greater influence on the imaging and makeup fusion effect of the to-be-processed image in the ambient light. The background light can be understood as the light entering the camera aperture from the opposite direction of the camera. These lights will directly enter the camera and make the camera sensor receive the background light signal. The foreground light can be understood as the light illuminating the face from the opposite direction of the face, so that the face can be illuminated as a sub-light source.

[0034] For example, after obtaining the to-be-processed image, the face region and the sclera region in the to-be-processed image are determined, and the background light color value corresponding to the background light in the shooting environment is calculated according to the color information in the face region, and the foreground light color value corresponding to the foreground light in the shooting environment is calculated according to the color information in the sclera region. The face region is the region corresponding to the face recognized in the to-be-processed image, and the sclera region is the region corresponding to the sclera recognized in the to-be-processed image, wherein the sclera region can be understood as the region corresponding to the white of the eye in the face.

[0035] Optionally, the determination of the face region and the sclera region in the to-be-processed image can be determined according to the face key points obtained by face recognition on the to-be-processed image, or the face region and the sclera region in the to-be-processed image can be obtained by analyzing and processing the to-be-processed image through a trained face recognition model and output by the face recognition model.

[0036] In one embodiment, the background light color value provided by the scheme can be determined based on the color value of each pixel point in the face region, for example, the background light color is determined based on the color value of one or more pixel points with the darkest color in the face region, and the foreground light color value can be determined based on the color value of each pixel point in the sclera region.

[0037] S120: determining a reflection color value of each to-be-fused pixel point according to a difference between an original color value of the to-be-fused pixel point corresponding to the enhancement material in the to-be-processed image and the background light color value.

[0038] The enhancement material provided by the scheme can be understood as a material for enhancing the portrait in the to-be-processed image, for example, a makeup material (such as virtual makeup, beautifying, stickers, special effects, etc.) for fusing makeup in the face in the to-be-processed image. After the enhancement material is fused into the to-be-processed image, the portrait in the to-be-processed image displays the enhancement result of the enhancement material and the to-be-processed image at the corresponding position, for example, the corresponding makeup material is pasted on the face. The enhancement material includes a plurality of material pixel points, and each material pixel point is configured with a corresponding material point color value.

[0039] Optionally, a plurality of enhancement materials can be configured in the portrait enhancement processing device, and the enhancement material can be selected by default or by the user as the enhancement material for portrait enhancement processing with the to-be-processed image. The enhancement material used for portrait enhancement of the to-be-processed image can be one or more, for example, a plurality of makeup materials are added to the face in the to-be-processed image at the same time.

[0040] For example, after the background light color value and the foreground light color value corresponding to the to-be-processed image are determined, the to-be-fused pixel points corresponding to the enhancement material in the to-be-processed image are determined, and the original color value corresponding to each to-be-fused pixel point is determined. The to-be-fused pixel points corresponding to the enhancement material in the to-be-processed image can be determined based on the graphics rendering pipeline configured in the portrait enhancement processing device (the graphics rendering pipeline can also manage the scaling, rotation, positioning, and occlusion relationship of the enhancement material), for example, the graphics rendering pipeline converts the coordinates of each point of the enhancement material from the world coordinate system to the screen coordinate system through vertex transformation, determines the position of the to-be-processed image relative to the screen, and according to the correspondence between the position of the pixel point in the screen relative to the screen coordinate and the position of the point in the enhancement material relative to the enhancement material, determines the image pixel point corresponding to each material pixel point in the enhancement material in the to-be-processed image. These image pixel points are the to-be-fused pixel points.

[0041] Further, the difference between the original color value and the background light color value of each to-be-fused pixel point in the to-be-processed image is calculated, and the difference corresponding to each to-be-fused pixel point is determined as the reflection color value of the to-be-fused pixel point. It needs to be explained that the existence of the background light is equivalent to covering the face area in the to-be-processed image with a layer of uniform background light color, and the background light should not act on the enhancement material. Before the enhancement material is fused into the to-be-processed image, the foreground light color value in the to-be-fused pixel point is temporarily deducted, which can effectively reduce the influence of the foreground light on the fusion of the enhancement material and the to-be-processed image, improve the natural degree of the fusion of the enhancement material into the to-be-processed image, and improve the portrait enhancement processing effect.

[0042] S130: Determine the blending color value corresponding to each pixel to be blended based on the pixel color value, reflection color value, and foreground light color value of each pixel in the enhanced material.

[0043] For example, after determining the reflection color value of the pixel to be fused, the fusion color value corresponding to each pixel to be fused can be determined based on the material point color value, reflection color value, and foreground light color value of each material pixel in the enhanced material, thereby achieving the effect of adding enhanced material to the image to be processed.

[0044] Optionally, the blending color value corresponding to each pixel to be blended can be determined by a pixel shader (or fragment shader). The pixel shader can be used to calculate the color or other attributes of each pixel within the controlled area. For example, by sending the source point color value, reflection color value, and foreground light color value to the pixel shader, the pixel shader can determine the blending color value of the corresponding pixel to be blended based on these values. In other words, the pixel shader implements the process of overlaying the enhanced material onto the image to be processed. Optionally, different overlay methods can be configured based on different enhanced materials.

[0045] The pixel shader is responsible for actually adding enhancement materials to the face. Traditional enhancement materials are typically blended with the face using methods such as premultiplication, multiply, and overlay, with the overlay mode selected based on the physical properties of different cosmetics (i.e., configuring different overlay methods). However, in traditional portrait enhancement methods, the enhancement materials are directly mixed with the original color values ​​output by the camera. This direct pixel color fusion method, for some enhancement materials using modes such as multiply (e.g., highlight eyeshadow, glitter), should have an actual brightness proportional to the foreground light intensity. However, the original portrait enhancement process crudely adds the brightness of the enhancement materials to the background brightness, resulting in overly bright enhancement areas in low-light scenes. For some enhancement materials using premultiplication (e.g., lipstick, mascara), the enhancement effect should only act on the reflected light from the face, meaning the enhancement material should only absorb a portion of the energy emitted from the face. However, the original portrait enhancement process applies the enhancement effect to the pixel brightness, effectively blending the enhancement material with the background light as well, resulting in overly dark enhancement areas in bright background scenes.

[0046] In traditional image enhancement processing, different enhancement materials can be superimposed onto the original image (i.e., the image to be processed) using different blending modes. The process of superimposing enhancement materials onto the original image can be abstracted as a function F. This function F should actually accept at least three parameters: the color value of the material points, the reflected color value after removing the background light, and the foreground light color value. In traditional image enhancement processing, the reflected color value is replaced by the original color value c, while the foreground light color value is set to a fixed value. The foreground light color value actually plays no role; for example, setting the foreground light color value to 1 means that the traditional material point color value, the reflected color value after removing the background light, and the foreground light color value are input as b, c, and 1, respectively. Therefore, the output of a traditional beauty model can be abstracted as: c ′ = (b, c, 1).

[0047] This method calculates the background light color value c within the face area. b It is no longer necessary to use the original color value c as an estimate of the reflected color value; instead, the background light color value c is separated from the original color value c. b Get the true reflected color value cc b Additionally, the foreground light color value c f It can also be determined based on the scleral region. Therefore, in the process of overlaying the enhancement material onto the original image, the processing method for each pixel to be fused can be expressed as: c′=F(b,cc) b c T c′ is the blended color value.

[0048] In one possible embodiment, when determining the fusion color value corresponding to each pixel to be fused based on the material point color value, reflection color value, and foreground light color value of each material pixel in the enhanced material, the portrait enhancement processing method provided by this solution may determine the fusion color value corresponding to each pixel to be fused based on the material point color value, reflection color value, foreground light color value, and a set brightness correction coefficient of each material pixel in the enhanced material. The brightness correction coefficient is used to correct the foreground light color value; that is, the product of the foreground light color value and the brightness correction coefficient is used as the corrected foreground light color value.

[0049] It needs to be explained that foreground light can be understood as light that directly illuminates the face. Within the visible light range (where the face's self-emission is negligible), if the camera, face position, and the face's optical properties remain unchanged, the brightness of the foreground light is proportional to the brightness of the light reflected from the face. However, the color value c of the foreground light... f The foreground color value c is proportional to, but not equal to, the actual foreground light brightness; therefore, when using it, the foreground light color value c... fA brightness correction factor k needs to be multiplied to obtain the true brightness factor. This factor k reflects the ratio of the reflectivity of the enhancement material to the reflectivity of the sclera. Optionally, the correction factor for different enhancement materials can be determined within the range of 2 to 8. Based on this, the processing method for each pixel to be fused during the process of superimposing the enhancement material onto the original image can be expressed as: c′=F(b,cc b , k·c f )

[0050] This solution corrects the foreground light color value using a brightness correction coefficient, enhancing the fusion of the source material and the image to be processed to better reflect changes in actual ambient light and effectively improve the portrait enhancement effect.

[0051] S140: Add the background light color value to the fusion color value corresponding to each pixel to be fused, and obtain the target color value of each pixel to be fused.

[0052] It should be explained that in this solution, the background light does not participate in the color fusion operation in portrait enhancement, but it does not mean that it will not affect the final output of the portrait enhancement process. The influence of the background light needs to be added to the fusion result to obtain the final portrait enhancement output of each pixel to be fused.

[0053] For example, after determining the fusion color value corresponding to each pixel to be fused, the background light color value is added to the fusion color value corresponding to each pixel to be fused, thus obtaining the target color value of each pixel to be fused. That is, the final target color value output for each pixel to be fused is output = c' + c. b , or output = F(b, cc) b , k·c f )+c b .

[0054] The above describes a process where background and foreground light color values ​​are determined based on the face and sclera regions in the image to be processed. The reflection color value is determined by the difference between the original color value of the pixel to be fused and the background light color value. A fusion color value is then determined based on the source pixel's color value, reflection color value, and foreground light color value. The background light color value is added to each fusion color value, thus determining the target color value for each pixel to be fused. The fusion of the target color value for each pixel considers the influence of foreground and background light colors on portrait enhancement. The portrait enhancement effect can adaptively adjust to changes in the environment, eliminating the need for manual adjustment of the enhancement intensity by the user, thus achieving flexible adjustment of the portrait enhancement effect and effectively improving the overall portrait enhancement result.

[0055] Based on the above embodiments, Figure 2A flowchart of another portrait enhancement processing method provided in an embodiment of this application is given, which is a concretization of the above-described portrait enhancement processing method. (Reference) Figure 2 The portrait enhancement processing method includes:

[0056] S210: Obtain facial key points in the image to be processed, and determine the face region and sclera region in the image to be processed based on the facial key points.

[0057] S220: Determine the background light color value based on the face region in the image to be processed, and determine the foreground light color value based on the sclera region in the image to be processed.

[0058] For example, an image to be processed is acquired, and facial key points in the image are determined using a pre-defined facial key point recognition algorithm or model. The coordinates of the facial key points are then transformed to the same coordinate system as the image to be processed, facilitating polygon scanning transformation (i.e., determining the polygons corresponding to the face and sclera regions) during face and sclera region recognition. Further, the face and sclera regions in the image to be processed are determined based on the individual facial key points. The face region can be determined based on key points around the face, and the sclera region can be determined based on key points corresponding to the eyes or the sclera itself.

[0059] After identifying the face and sclera regions, the background light color value is determined based on the color values ​​of each pixel in the face region of the image to be processed, and the foreground light color value is determined based on the color values ​​of each pixel in the sclera region of the image to be processed. This solution accurately identifies the face and sclera regions in the image to be processed through facial key points, more accurately estimates the background and foreground light color values ​​in the actual shooting environment, and enhances the fusion processing of the source material and the image to be processed to better match the changes in actual ambient light, effectively improving the portrait enhancement processing effect.

[0060] In one possible embodiment, such as Figure 3 The provided flowchart illustrates a process for determining background light color values. The portrait enhancement method provided in this solution, when determining background light color values ​​based on the face region in the image to be processed, includes:

[0061] S221: Determine the channel color values ​​of each image pixel in the face region of the image to be processed in multiple color channels.

[0062] S222: Determine the background light color value based on the minimum channel color value of each color channel.

[0063] For example, the channel color values ​​of each image pixel in the face region of the image to be processed are determined in multiple color channels. For each color channel, the minimum channel color value is determined, and the minimum channel color value corresponding to each color channel is used as the background light color value in the corresponding color channel.

[0064] For example, determine the channel color value of each image pixel in the three color channels of RGB (red, green, blue), determine the corresponding minimum channel color value in the R channel, G channel and B channel respectively, and determine the background light color value based on the three minimum channel color values.

[0065] It needs to be explained that the background light is the light that enters the camera directly. After being refracted by the lens, the background light acts directly on the camera's CMOS (Complementary Metal-Oxide-Semiconductor) sensor, converting light energy into electrical energy. This allows the camera to receive a certain intensity of light even when photographing black objects. Specifically, for the face area in the image to be processed, the background light color value c... b The values ​​remain largely consistent, and within the face area, at least one point can be found whose reflectivity to foreground light is close to 0. For any color channel at any point within the face area, its brightness is equal to the sum of the intensity of the face-reflected light and the background light intensity of that color channel. Therefore, c = c r +c b This applies to any image pixel within the face region. Furthermore, pupils, nostrils, the area between slightly parted lips, black beards, and black-rimmed glasses are typically candidate locations for determining the background light color value within the face region. Even when strong foreground light is projected onto the face, these areas often appear as dark blacks. Therefore, at any given moment, some of these locations within the face region F are exposed, and the brightness of these locations will equal the brightness of the background light. Thus, the brightness of the background light in a given color channel can be determined by the lowest brightness point in each color channel. The background light color value can be expressed as:

[0066] c b ={min x∈F (c x .red), min x∈F (c x .green), min x∈F (c x .blue)}

[0067] Where, min x∈F (c x .red) represents the minimum channel color value in the R channel of the face region, min x∈F (c x.green) represents the minimum channel color value in the G channel for the face region, min x∈F (c x .blue represents the minimum channel color value in the B channel of the face region, and x represents the image pixel in the image to be processed. This solution accurately determines the background light color value based on the minimum channel color value of each color channel in the face region. In the material fusion process, it can more accurately reduce the impact of background light on portrait enhancement processing, making the fusion processing of enhanced materials and the image to be processed more consistent with the actual changes in ambient light, and effectively improving the portrait enhancement processing effect.

[0068] In one possible embodiment, when determining the channel color values ​​of each image pixel in the face region of the image to be processed, this solution may determine the scanning area in the image to be processed based on the facial key points corresponding to the eyebrow and lip areas, and then determine the channel color values ​​of each image pixel in the scanning area in the multiple color channels.

[0069] For example, when determining the background light color value based on the face region, this solution can determine the scanning area in the image to be processed based on the facial key points corresponding to the eyebrows and lips. Then, based on the channel color values ​​of each image pixel in the scanning area across multiple color channels, the background light color value is determined based on the minimum channel color value. The eyebrows can be the upper part of the eyebrows and the outer edges of the left and right eyebrows; the lips can be the sides of the lips and the lower lip. That is, determining the scanning area based on the facial key points corresponding to the eyebrows and lips can be done by using the polygon outlined by the upper part of the eyebrows, the outer edges of the left and right eyebrows, the sides of the lips, and the lower lip as the scanning area. Since the scanning area is a region on the face where black is more likely to appear, this solution effectively reduces the amount of data processing on image pixels and improves the efficiency of determining the background light color value while ensuring the quality of the background light color value determination.

[0070] In one possible embodiment, such as Figure 4 The provided flowchart illustrates a process for determining foreground light color values. The portrait enhancement method provided in this solution, when determining foreground light color values ​​based on the scleral region in the image to be processed, includes:

[0071] S223: Determine the median color value corresponding to multiple color channels in the scleral region of the image to be processed.

[0072] S224: Determine the foreground light color value based on the median color value of each color channel.

[0073] For example, for each color channel in the scleral region, the channel color value of each image pixel in each color channel is determined, and the median color value corresponding to each color channel is calculated based on these channel color values. The foreground light color value is then determined based on the median color value of each color channel.

[0074] For example, determine the channel color value of each image pixel in the scleral region in the three RGB color channels, calculate the median color value in each of the three RGB color channels, and determine the color value of the foreground light corresponding to the three color channels based on the difference between the three color channels.

[0075] In one possible embodiment, when determining the foreground light color value based on the median color value of each color channel, this solution may determine the foreground light color value based on the difference between the median color value of each color channel and the background light color value.

[0076] For example, after determining the median color value for each color channel, the foreground light color value is determined based on the difference between the median color value of each color channel and the background light color value. For instance, after determining the median color values ​​for the three RGB color channels, the difference between the median color value of each of the three color channels and the determined background light color value is calculated, and the difference for each of the three color channels is determined as the foreground light color value corresponding to the three color channels. This solution accurately determines the foreground light color value based on the difference between the median color value of each color channel and the background light color value. In material blending, this more accurately reflects the impact of foreground light on portrait enhancement processing, effectively improving the portrait enhancement effect.

[0077] It needs to be explained that, for each image pixel in the face region of the image to be processed, the foreground light color value c f The color of the sclera remains largely consistent across all humans, regardless of skin color or race. Furthermore, the sclera color is white across all races. Therefore, the color of the sclera can be used to estimate foreground brightness. However, considering the occlusion of the sclera by eyelids, nictitating membranes, blood vessels, and eyelashes, as well as the inherent instability of facial landmark detection, which may result in some areas being too dark, and the relatively smooth sclera potentially causing high-brightness reflections, the median brightness within the bilateral scleral regions S is used as the foreground light estimate for each color channel. The scleral region can be delineated using facial landmarks on the upper and lower eyelids, or more precisely, determined by combining facial landmarks and iris landmarks. Additionally, every pixel within the facial region contains energy provided by the background light; therefore, the influence of background light on foreground light estimation can be subtracted when estimating foreground light. Based on this, the formula for estimating foreground light color can be expressed as:

[0078] c f ={med x∈S (c x .red), med x∈S (c x .green), med x∈S (c x .blue)}-c b

[0079] Among them, med x∈S (c x .red) represents the median color value in the R channel for the scleral region, med x∈S (c x .green) represents the median color value of the sclera region in the G channel. x∈S (c x (.blue) represents the median color value of the sclera region in the B channel, and x represents the image pixel in the image to be processed. This method accurately determines the foreground light color value based on the median color value of each color channel in the sclera region. In material fusion, it can more accurately reflect the influence of foreground light on portrait enhancement processing, and the fusion processing of enhanced materials and the image to be processed is more in line with the actual changes in ambient light, effectively improving the portrait enhancement processing effect.

[0080] S230: Determine the reflection color value of each pixel to be fused based on the difference between the original color value and the background light color value of the pixel to be fused corresponding to the enhanced material in the image to be processed.

[0081] S240: Determine the blending color value corresponding to each pixel to be blended based on the pixel color value, reflection color value, and foreground light color value of each pixel in the enhanced material.

[0082] S250: Add the background light color value to the corresponding blending color value of each pixel to be blended to obtain the target color value of each pixel to be blended.

[0083] The above describes a process where background and foreground light color values ​​are determined based on the face and sclera regions in the image to be processed. The reflection color value is determined by the difference between the original color value of the pixel to be fused and the background light color value. A fusion color value is then determined based on the source pixel's color value, reflection color value, and foreground light color value. The background light color value is added to each fusion color value, thus determining the target color value for each pixel to be fused. The fusion of the target color value for each pixel considers the influence of foreground and background light colors on portrait enhancement. The portrait enhancement effect can adaptively adjust to changes in the environment, eliminating the need for manual adjustment of the enhancement intensity by the user, thus achieving flexible adjustment of the portrait enhancement effect and effectively improving the overall portrait enhancement result. Meanwhile, by accurately determining the background light color value based on the minimum channel color value of each color channel in the face region, the influence of background light on portrait enhancement processing can be reduced more accurately during material fusion. Furthermore, by accurately determining the foreground light color value based on the median color value of each color channel in the sclera region, the influence of foreground light on portrait enhancement processing can be reflected more accurately during material fusion. This enhances the fusion processing of the enhanced material and the image to be processed, making it more consistent with the actual changes in ambient light and effectively improving the portrait enhancement processing effect. This paper applies image enhancement processing methods to makeup integration in live streaming scenarios. Under various live streaming environments such as strong backlighting, colored backlighting, and low light, the host adds virtual makeup and starts broadcasting. The image enhancement processing device reads each frame of the video stream in real time as the image to be processed, calculates the current ambient light information, uses the minimum value of each color channel in the face area as the background light estimate, and uses the median value of each color channel in the sclera area as the foreground light estimate. Finally, the foreground light and background light are applied to the pixel shader to dynamically change the intensity of the virtual makeup, thereby reducing the virtual makeup floating and the low degree of integration between the virtual makeup and the background, improving the makeup integration effect, improving the integration of makeup materials and background light, effectively reducing the sense of disharmony in makeup integration caused by changes in ambient light, and enhancing the user experience for both the host and the audience.

[0084] Figure 5 This is a schematic diagram of the structure of a portrait enhancement processing device provided in an embodiment of this application. (Reference) Figure 5 The portrait enhancement processing device includes an ambient light determination module 51, a reflected light determination module 52, a color fusion module 53, and a color determination module 54.

[0085] The ambient light determination module 51 is configured to determine the background light color value and the foreground light color value based on the face region and sclera region in the image to be processed; the reflected light determination module 52 is configured to determine the reflected color value of each pixel to be fused based on the difference between the original color value and the background light color value of the pixel to be fused corresponding to the enhanced material in the image to be processed; the color fusion module 53 is configured to determine the fusion color value corresponding to each pixel to be fused based on the material point color value, reflected color value, and foreground light color value of each material pixel in the enhanced material; and the color determination module 54 is configured to add the background light color value to the fusion color value corresponding to each pixel to be fused to obtain the target color value of each pixel to be fused.

[0086] The above describes a process where background and foreground light color values ​​are determined based on the face and sclera regions in the image to be processed. The reflection color value is determined by the difference between the original color value of the pixel to be fused and the background light color value. A fusion color value is then determined based on the source pixel's color value, reflection color value, and foreground light color value. The background light color value is added to each fusion color value, thus determining the target color value for each pixel to be fused. The fusion of the target color value for each pixel considers the influence of foreground and background light colors on portrait enhancement. The portrait enhancement effect can adaptively adjust to changes in the environment, eliminating the need for manual adjustment of the enhancement intensity by the user, thus achieving flexible adjustment of the portrait enhancement effect and effectively improving the overall portrait enhancement result.

[0087] In one possible embodiment, when the ambient light determination module 51 determines the background light color value and the foreground light color value based on the face region and the sclera region in the image to be processed, it is configured as follows:

[0088] Obtain facial key points in the image to be processed, and determine the face region and sclera region in the image to be processed based on the facial key points;

[0089] Background light color values ​​are determined based on the face region in the image to be processed, and foreground light color values ​​are determined based on the sclera region in the image to be processed.

[0090] In one possible embodiment, when determining the background light color value based on the face region in the image to be processed, the ambient light determination module 51 is configured as follows:

[0091] Determine the channel color values ​​of each image pixel in the face region of the image to be processed in multiple color channels;

[0092] The background light color value is determined based on the minimum channel color value of each color channel.

[0093] In one possible embodiment, when determining the channel color values ​​of each image pixel in the face region of the image to be processed, the ambient light determination module 51 is configured as follows:

[0094] Based on the facial key points corresponding to the eyebrows and lips, the scanning area in the image to be processed is determined.

[0095] Based on the channel color values ​​of each image pixel in the scanned area across multiple color channels.

[0096] In one possible embodiment, when determining the foreground light color value based on the scleral region in the image to be processed, the ambient light determination module 51 is configured as follows:

[0097] Determine the median color value corresponding to multiple color channels in the scleral region of the image to be processed;

[0098] The foreground light color value is determined based on the median color value of each color channel.

[0099] In one possible embodiment, when determining the foreground light color value based on the median color value of each color channel, the ambient light determination module 51 is configured as follows:

[0100] The foreground light color value is determined based on the difference between the median color value of each color channel and the background light color value.

[0101] In one possible embodiment, when the color fusion module 53 determines the fusion color value corresponding to each pixel to be fused based on the material pixel color value, reflection color value, and foreground light color value of each material pixel in the enhanced material, it is configured as follows:

[0102] Based on the material pixel color value, reflection color value, foreground light color value, and the set brightness correction coefficient of each material pixel in the enhanced material, determine the corresponding blending color value for each pixel to be blended.

[0103] It is worth noting that in the above embodiments of the portrait enhancement processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0104] This application also provides a portrait enhancement processing device, which can integrate the portrait enhancement processing apparatus provided in this application. Figure 6 This is a schematic diagram of the structure of a portrait enhancement processing device provided in an embodiment of this application. (Reference) Figure 6The portrait enhancement processing device includes: an input device 63, an output device 64, a memory 62, and one or more processors 61; the memory 62 is used to store one or more programs; when one or more programs are executed by one or more processors 61, the one or more processors 61 implement the portrait enhancement processing method provided in the above embodiments. The portrait enhancement processing device, apparatus, and computer provided above can be used to execute the portrait enhancement processing method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0105] This application also provides a non-volatile storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to perform the portrait enhancement processing method provided in the above embodiments. Of course, the computer-executable instructions provided in this application are not limited to the portrait enhancement processing method provided above; they can also perform related operations in the portrait enhancement processing method provided in any embodiment of this application. The portrait enhancement processing apparatus, device, and storage medium provided in the above embodiments can execute the portrait enhancement processing method provided in any embodiment of this application. Technical details not described in detail in the above embodiments can be found in the portrait enhancement processing method provided in any embodiment of this application.

[0106] Based on the above embodiments, this application also provides a computer program product. The technical solution of this application, in essence or in other words, the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes several instructions to cause a computer device, mobile terminal, or processor therein to execute all or part of the steps of the portrait enhancement processing method provided in the various embodiments of this application.

Claims

1. A method for enhancing human images, characterized in that, include: Based on the face region and sclera region in the image to be processed, the background light color value and the foreground light color value are determined, wherein the background light color value corresponds to the background light of the face region in the shooting environment, and the foreground light color value corresponds to the foreground light of the sclera region in the shooting environment. The reflection color value of each pixel to be fused is determined based on the difference between the original color value of the pixel to be fused corresponding to the enhanced material in the image to be processed and the background light color value. Based on the material point color value, the reflected color value, and the foreground light color value of each material pixel in the enhanced material, determine the fusion color value corresponding to each pixel to be fused; The background light color value is added to the fusion color value corresponding to each pixel to be fused, to obtain the target color value of each pixel to be fused.

2. The portrait enhancement processing method according to claim 1, characterized in that, The step of determining the background light color value and the foreground light color value based on the face region and sclera region in the image to be processed includes: Obtain facial key points in the image to be processed, and determine the face region and sclera region in the image to be processed based on the facial key points; The background light color value is determined based on the face region in the image to be processed, and the foreground light color value is determined based on the sclera region in the image to be processed.

3. The portrait enhancement processing method according to claim 2, characterized in that, Determining the background light color value based on the face region in the image to be processed includes: Determine the channel color values ​​of each image pixel in the face region of the image to be processed in multiple color channels; The background light color value is determined based on the minimum channel color value of each of the aforementioned color channels.

4. The portrait enhancement processing method according to claim 3, characterized in that, Determining the channel color values ​​of each image pixel in the face region of the image to be processed in multiple color channels includes: Based on the facial key points corresponding to the eyebrows and lips, the scanning area in the image to be processed is determined. Based on the channel color values ​​of each image pixel in the scanned area across multiple color channels.

5. The portrait enhancement processing method according to claim 2, characterized in that, Determining the foreground light color value based on the scleral region in the image to be processed includes: Determine the median color value corresponding to multiple color channels in the scleral region of the image to be processed; The foreground light color value is determined based on the median color value of each of the aforementioned color channels.

6. The portrait enhancement processing method according to claim 5, characterized in that, Determining the foreground light color value based on the median color value of each of the color channels includes: The foreground light color value is determined based on the difference between the median color value of each of the color channels and the background light color value.

7. The portrait enhancement processing method according to claim 1, characterized in that, The step of determining the blending color value corresponding to each pixel to be blended based on the pixel color value, the reflected color value, and the foreground light color value of each pixel in the enhanced material includes: Based on the material point color value, the reflected color value, the foreground light color value, and the set brightness correction coefficient of each material pixel in the enhanced material, the fusion color value corresponding to each pixel to be fused is determined.

8. A human image enhancement processing device, characterized in that, It includes an ambient light determination module, a reflected light determination module, a color blending module, and a color determination module, among which: The ambient light determination module is configured to determine background light color values ​​and foreground light color values ​​based on the face region and sclera region in the image to be processed, wherein the background light color value corresponds to the background light of the face region in the shooting environment, and the foreground light color value corresponds to the foreground light of the sclera region in the shooting environment. The reflected light determination module is configured to determine the reflected color value of each pixel to be fused based on the difference between the original color value of the pixel to be fused corresponding to the enhanced material in the image to be processed and the background light color value. The color fusion module is configured to determine the fusion color value corresponding to each pixel to be fused based on the pixel color value, the reflected color value, and the foreground light color value of each pixel in the enhanced material. The color determination module is configured to add the background light color value to the fusion color value corresponding to each pixel to be fused, so as to obtain the target color value of each pixel to be fused.

9. A human image enhancement processing device, characterized in that, include: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the portrait enhancement processing method as described in any one of claims 1-7.

10. A non-volatile storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the portrait enhancement processing method as described in any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the portrait enhancement processing method according to any one of claims 1-7.

Images