Eye light image feature processing method, electronic equipment and storage medium
The method corrects unnatural eye light enhancements in portraits by separating iris and highlight regions, determining enhancement parameters, and aligning eye light features with the background, achieving realistic eye light in low-light and telephoto shooting scenarios.
Patent Information
- Application Number
- CN202310644125.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-05-31
AI Technical Summary
Existing face enhancement deep models often mistakenly modify or add eye light features in portrait images, resulting in unnatural enhancements that degrade user experience.
A method involving a highlight separation algorithm to separate iris and highlight regions, determine catch light contrast enhancement parameters based on attribute parameters, and adaptively enhance eye light features to correct unnatural enhancements.
Enhances eye light features naturally, aligning them with the background to produce high-quality portraits with realistic eye light, meeting user demands for authenticity in low-light and telephoto shooting scenarios.
Smart Images

Figure CN119090730A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method for processing eye light image features, an electronic device, and a storage medium. Background Art
[0002] At present, compared with shooting scenes such as good lighting and close-ups, the portrait imaging effect taken by mobile phones under edge conditions such as dark light and long distance will be significantly reduced. In order to improve the quality of portrait images taken under such edge conditions, the mainstream method in the industry is usually to apply a face enhancement depth model to the original image taken after the user triggers the photo. The clarity and facial details of the resulting photos will be enhanced, and the quality of facial images will be improved.
[0003] However, when enhancing faces through the face enhancement deep model, the eye light features are often modified, smeared or added by mistake. The enhanced eye light looks extremely unnatural, does not meet the user's basic needs for the authenticity of the camera function, and affects the user experience. Summary of the invention
[0004] The present application provides a method for processing eye light image features and an electronic device, which solves the problem that after a photo taken by a user is processed by a face enhancement depth model, the eye light features of the portrait are mistakenly enhanced, resulting in distortion of the eye light of the portrait.
[0005] In a first aspect, the present application provides a method for processing eye light image features, the method comprising:
[0006] When the preset conditions are met, a first image and a second image containing facial features are obtained, where the second image is an image obtained by enhancing the first image using a face enhancement deep model;
[0007] Using a highlight separation algorithm, separating a first iris background region and a first highlight region from the iris region of the first image, and separating a second iris background region and a second highlight region from the iris region of the second image;
[0008] Determine a catch light contrast enhancement parameter according to the attribute parameters of the first highlight area; the attribute parameters include size, shape, average brightness and / or brightest point brightness; the catch light contrast enhancement parameter includes a gamma value, and / or a partition block size and a contrast limit of a contrast-limited adaptive histogram equalization (CLAHE);
[0009] Using the catch light contrast enhancement parameter, the image feature of the first highlight area is enhanced to obtain the enhanced catch light feature;
[0010] The enhanced catch light feature is background-fused with the image feature of the second iris background area to obtain a third image.
[0011] Through the processing method of the eye light image feature provided by the embodiment of the present application, for the scene where the eye light of a low-quality portrait is unnatural after being enhanced by the face enhancement depth model, the highlight area is first separated from the eye area of the low-quality portrait (first image), and then the eye light contrast enhancement parameter is determined according to the attribute parameters of the highlight area, and then the eye light is controllably enhanced according to the eye light contrast enhancement parameter. Then, the "abnormal" eye light feature in the high-quality portrait (second image) is replaced with the enhanced eye light feature, so as to correct the eye light feature, thereby obtaining a high-quality portrait with "normal" eye light after enhancement, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0012] It should be noted that the algorithm scenarios involved in this application are mainly scenarios where mobile phones take low-quality images, including but not limited to low-light and telephoto scenes, and may also be any other scenarios that may take low-quality images.
[0013] This application focuses on the user's demand for the fidelity of the face area in scenes such as low-light shooting and telephoto shooting, and can controllably enhance the human eye's catch light to compensate for the unnatural defect of the eye light caused by full-face enhancement through the face enhancement depth model.
[0014] In some other embodiments, the method further includes: when there is no catch light feature in the iris region, obtaining a fourth image according to the image features of the second iris background region. In the case where the low-quality portrait has no catch light feature, the erroneously added catch light feature can be directly removed from the corresponding high-quality portrait, which can also correct the defect of unnatural, i.e., erroneously added, catch light caused by full face enhancement using the face enhancement depth model.
[0015] When the electronic device has turned on the automatic correction function of the catch light, when the user triggers the photo, the electronic device will simultaneously obtain the original photo (low-quality portrait, first image) and the high-quality portrait (second image) obtained through the face enhancement depth model, and the catch light of the high-quality portrait is abnormal. The electronic device performs iris cutout processing and catch light / background separation processing on the low-quality portrait to obtain the eye background and catch light of the low-quality portrait. At the same time, the electronic device performs iris cutout processing and catch light / background separation processing on the high-quality portrait to obtain the eye background and abnormal catch light of the high-quality portrait.
[0016] In this case, the eye background of the low-quality portrait is discarded, and the abnormal eye light of the high-quality portrait is discarded. The eye light of the low-quality portrait is adaptively enhanced to obtain enhanced eye light. Then, the enhanced eye light after processing is fused with the eye background of the high-quality portrait to obtain a high-quality portrait with normal eye light.
[0017] In this implementation, the portrait in the photo obtained through image processing has eye light, and the eye light is normal, thereby meeting the user's demand for the fidelity of the face area in shooting scenarios such as low-light shooting and telephoto shooting that require algorithm-enhanced low-quality portrait photos.
[0018] It should be noted that the method for processing eye light image features provided in the embodiment of the present application can be used not only to correct the eye light of portraits in photos, but also to modify the eye light of animals in photos.
[0019] In some possible implementations, separating the first iris background region and the first highlight region from the iris region of the first image, and separating the second iris background region and the second highlight region from the iris region of the second image by using a highlight separation algorithm, includes:
[0020] The iris background area and highlight area are determined by the following equations:
[0021] V s ' pec =IV d ' iff
[0022] V d ' iff =I-kβ s
[0023] β s =I min -τ(p)
[0024] Among them, V′ spec Represents the image component of the highlight area; V′ diff represents the image component of the iris background area, I represents the image of the iris area, and I min It represents a single-channel image component obtained by taking the minimum value of the RGB three channels of each pixel point of the image I, τ(p) represents the offset related to the image pixel mean and variance, and k is the proportional coefficient.
[0025] The iris background area is a first iris background area, and the highlight area is a first highlight area; or, the iris background area is a second iris background area, and the highlight area is a second highlight area.
[0026] By using the highlight separation algorithm, the highlight part can be detected in the iris region of the human eye, for example, a mask of the highlight part can be generated by image threshold segmentation, wherein the connected components of the mask can be solved by the flood filling method.
[0027] In the embodiment of the present application, the catch light area mask is obtained by an automatic highlight removal algorithm, which can support the separation of discontinuous catch light areas of arbitrary shapes.
[0028] In some possible implementations, separating the first iris background area and the first highlight area from the iris area of the first image includes: normalizing pixel values of the iris area of the first image to the interval [0,1]; taking areas in the iris area of the first image whose pixel values are greater than a threshold α as possible highlight areas, where α is a value in the interval [0,1]; and extracting the first M largest connected component areas from the possible highlight areas as the first highlight areas by a flood filling method, where M is a positive integer.
[0029] In some possible implementations, the method further includes: obtaining a k value using a least squares method according to the diffuse reflection color intensity of the first M largest connected component regions and their edge regions.
[0030] It should be noted that the diffuse reflection color intensity of the highlight connected component area and its edge area should be consistent, that is, the mean value of the diffuse reflection component is equal. Under the smooth color transition criterion, the k value is obtained by the least square method.
[0031] It should also be noted that when the present application uses the de-highlighting algorithm to perform eye light / background separation, given that the eye light area in the actual scene may not be continuous and there may be multiple connected components, the first M largest connected components that "may be eye lights" are determined as highlight areas that need to be separated.
[0032] In the embodiment of the present application, the catch light area mask is obtained by a highlight separation algorithm (automatic highlight removal algorithm), which can support the separation of discontinuous catch light areas of arbitrary shapes.
[0033] In an embodiment of the present application, after the eye highlight area and the eye background area are separated by the eye light / background separation algorithm, taking into account the fact that part of the eyelid may remain in the human eye iris area obtained by the human eye iris cutout processing, and taking into account the possible errors in the de-highlighting algorithm, it may happen that there is a highlight area in the human eye iris area but the highlight feature in the highlight area is not actually the eye light. In other words, the original image taken by the user does not necessarily contain the eye light, and it is necessary to identify the eye highlight area in the original image, that is, to judge whether the low-quality portrait has the eye light feature.
[0034] Considering that not all images taken by users contain eye light, this application is based on the principle of high-fidelity enhancement. If a low-quality portrait does not have eye light features, the mistakenly added eye light features can be directly removed from the corresponding high-quality portrait. Since the eye light / background separation algorithm does not change the iris / sclera background of the human eye, the detailed information of the iris / sclera area of the high-quality portrait can be retained.
[0035] In the embodiment of the present application, the area of the eye highlight area separated from the human eye iris area can be used to determine whether there is catch light in the original image. For example, if the area of the highlight area is 0, it can be directly determined that there is no catch light in the highlight area. However, due to factors such as the fact that the human eye iris portion obtained by cutting out the image may have some eyelids remaining, and the de-highlighting algorithm may have errors, even if the area of the highlight area is not 0, the highlight area may still be without catch light.
[0036] In order to improve the accuracy of eye light recognition, in an embodiment of the present application, a joint judgment can be made based on multiple highlight attributes of the eye highlight area separated from the iris area of the human eye, so that it can be more accurately determined whether there is eye light in the original image.
[0037] In some possible implementations, before acquiring the attribute parameters of the first highlight region, the method further includes: judging whether the catch light feature exists in the iris region according to the attribute parameter set of the first highlight region; wherein the attribute parameter set includes at least two items of color moment, contrast, average brightness, saturation, and area. Acquiring the attribute parameters of the first highlight region includes: acquiring the attribute parameters of the first highlight region when the catch light feature exists in the iris region.
[0038] In some possible implementations, the attribute parameter includes the brightness of the brightest point; and the catch light contrast enhancement parameter includes the gamma value.
[0039] Determining the eye light contrast enhancement parameter according to the attribute parameter of the first highlight area includes: determining the ratio between the brightness of the brightest point in the first highlight area and 255 as the gamma value.
[0040] The method of using the catch light contrast enhancement parameter to enhance the image features of the first highlight area to obtain enhanced catch light features includes: using the gamma value to perform gamma correction on the image features of the first highlight area to stretch the image contrast of the first highlight area.
[0041] In some possible implementations, the attribute parameters include the length and width of the minimum circumscribed rectangle of the first highlight area; the eye light contrast enhancement parameters include the partition block size and contrast limit of the CLAHE; and the first highlight area includes one or more highlight areas.
[0042] The block size is min(L / 2,3), and the contrast limit is max(D / 2,2); L represents the shortest side of the first highlight area, D represents the maximum value of the histogram in the circumscribed rectangle of the highlight area with the largest area in the first highlight area, min() represents the minimum value, and max() represents the maximum value;
[0043] The method of using the catch light contrast enhancement parameter to enhance the image features of the first highlight area to obtain enhanced catch light features includes: using the block size and the contrast limit to redistribute pixels and reconstruct the grayscale values of pixels in the first highlight area to enhance the contrast of the first highlight area.
[0044] It should be noted that when the CLAHE method is used to enhance the eye light, each channel of the eye light image needs to be processed.
[0045] Through the above scheme, each eye light area can be adaptively enhanced in a targeted manner. Specifically, the enhancement parameters corresponding to a certain eye light area can be calculated according to the size, shape, brightness and other characteristics of the eye light area, and the eye light area can be enhanced in a targeted manner using the enhancement parameters corresponding to the eye light area. Therefore, the embodiments of the present application can adaptively enhance different eye light areas.
[0046] In some possible implementations, the preset condition is: in response to a first operation of a shutter button by a user, the first image is captured, the first image includes facial features, and an image quality of the first image is lower than or equal to a preset image quality.
[0047] In some possible implementations, the preset condition is: receiving a second operation of the user on the second image in the gallery, the second image including facial features and eye light features, and the second operation is used to adjust the eye light features.
[0048] In some possible implementations, separating the first iris background area and the first highlight area from the iris area of the first image by using a highlight separation algorithm includes: upsampling the eye area of the first image to obtain an enlarged eye area; separating the first iris background area and the first highlight area from the enlarged iris area by using the highlight separation algorithm; and downsampling the first iris background area and the first highlight area, respectively; wherein the reduction ratio of the downsampling is the inverse of the enlargement ratio of the upsampling.
[0049] In order to make the subsequent catch light / background separation more detailed, the iris / sclera area of the human eye is first upsampled to achieve image magnification, and then the upsampled iris / sclera area of the human eye is subjected to catch light / background separation processing.
[0050] It should be noted that if the image is directly downsampled, for example, using nearest linear interpolation, it is found that the downsampled image will have a lot of jagged edges. In some embodiments, the present application can use anti-aliasing downsampling to scale the size to the original image size to avoid jagged edges on the iris edge and the eye light edge.
[0051] In some possible implementations, upsampling the eye region of the first image to obtain an enlarged eye region includes: performing anti-aliasing upsampling and filtering denoising on the eye region of the first image; when the size of the enlarged eye region of the first image is smaller than a preset size threshold, performing anti-aliasing upsampling and filtering denoising on the eye region of the first image again; and when the size of the enlarged eye region of the first image is greater than or equal to the preset size threshold, obtaining the enlarged eye region.
[0052] In order to reduce the noise amplification caused by upsampling and the jagged edges of the iris and the eye light, the present application uses filtering and anti-aliasing upsampling alternately to achieve image magnification.
[0053] Through the solution of the present application, after multiple (for example, 2 to 3 times) cycles of anti-aliasing upsampling and filtering denoising processing, the human eye iris image is gradually enlarged (for example, 10 times), so that the subsequent eye light / background separation is more detailed, thereby improving the accuracy of image processing.
[0054] In some possible implementations, determining whether the catch light feature exists in the iris area based on the attribute parameter set of the first highlight area includes: obtaining the attribute parameter set of the first highlight area; inputting the attribute parameter set of the first highlight area into a preset binary classification model to obtain a judgment result, wherein the judgment result is that the catch light feature exists in the iris area or the catch light feature does not exist in the iris area.
[0055] In some possible implementations, the method further includes: using the attribute parameter set as a hyperparameter to offline train the preset binary classification model.
[0056] Through the present application scheme, it is comprehensively judged whether there is eye light in the original image based on multiple highlight attributes of the eye highlight area, which can improve the accuracy of eye light recognition.
[0057] In some possible implementations, the method of performing background fusion on the enhanced eye light feature and the image feature of the second iris background area to obtain a third image includes: determining an affine transformation matrix based on edge key points of the first iris background area and edge key points of the second iris background area; performing affine transformation on the enhanced eye light feature according to the affine transformation matrix to obtain an enhanced eye light feature aligned with the position of the second iris background area; and performing background fusion on the aligned enhanced eye light feature and the image feature of the second iris background area to obtain the third image.
[0058] Through affine transformation processing, the enhanced eye light feature is aligned with the eye background area of the high-quality portrait in position, so as to ensure that the positions of the enhanced eye light feature and the eye background area are accurately aligned during image fusion. Since the unchanged position of the eye light feature relative to the iris is an important factor in the unchanged eye expression, after the eye light feature is aligned with the iris, a natural and realistic eye light effect can be presented.
[0059] In some possible implementations, after obtaining the enhanced eye light feature aligned with the position of the second iris background area, the method further includes: performing adaptive feathering processing on the edge of the aligned enhanced eye light feature to obtain the feathered enhanced eye light feature.
[0060] By feathering the eye light, the inner and outer connecting parts of the eye light edge are blurred to play a gradient role, thereby achieving a natural connection effect and improving the image display effect.
[0061] Among them, the step of performing background fusion on the aligned enhanced eye light features and the image features of the second iris background area to obtain the third image includes: performing background fusion on the feathered enhanced eye light features and the image features of the second iris background area to obtain the third image.
[0062] In this way, the enhanced eye light feature can be naturally integrated with the human eye background area, so that the enhanced eye light feature can be aligned with the human eye background area in position, and the edge of the enhanced eye light feature is smooth, which can present a natural and realistic eye light effect.
[0063] Through the above scheme, the eye light of the low-quality portrait is adaptively enhanced to obtain enhanced eye light, and then the enhanced eye light is affine transformed and feathered, thereby realizing controllable enhancement of the eye light. Then, the processed enhanced eye light is fused with the eye background of the high-quality portrait to obtain a high-quality portrait with normal eye light.
[0064] It should be noted that the present application utilizes affine transformation and feathering processing to naturally blend the enhanced eye light into the eye area of the high-quality abnormal image, rather than simply copying and pasting the eye light.
[0065] In some possible implementations, before separating the first iris background area and the first highlight area from the iris area of the first image and separating the second iris background area and the second highlight area from the iris area of the second image by the highlight separation algorithm, the method further includes: performing iris cutout processing on the first image to extract the iris area from the first image; performing iris cutout processing on the second image to extract the iris area from the second image.
[0066] The method for processing eye light image features provided in the embodiments of the present application can be applied to user photo-taking scenarios, in which the electronic device automatically processes the eye light of a portrait. The method for processing eye light image features can also be applied to gallery photo editing scenarios, for example, it can be applied to portrait eye light editing in a mobile phone gallery, designated image eye light replacement, designated eye light template replacement, eye light enhancement / weakening / removal, etc.
[0067] For example, in the photo editing interface of the mobile phone gallery, one or more eye light templates can be provided for eye light replacement, or two pictures can be directly used for replacement; eye light enhancement / weakening controls, eye light replacement controls, and eye light removal controls can also be provided. Users can select one or more photos in the gallery according to their personal needs, and click different function controls to trigger the phone to edit the eye light of the portrait in the photo, such as enhancing / weakening / replacing / removing the eye light, by clicking on different function controls.
[0068] In a second aspect, the present application provides a device for processing eye light image features, the device comprising a unit for executing the method in the first aspect. The device may correspond to executing the method described in the first aspect, and the relevant description of the unit in the device can refer to the description of the first aspect, which will not be repeated here for the sake of brevity.
[0069] The method described in the first aspect above can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.
[0070] In a third aspect, the present application provides an electronic device, comprising a processor, a computer program or instructions stored in the processor and a memory, wherein the processor is configured to execute the computer program or instructions so that the method in the first aspect is executed.
[0071] In a fourth aspect, the present application provides a computer-readable storage medium on which a computer program (also referred to as instructions or codes) for implementing the method in the first aspect is stored. For example, when the computer program is executed by a computer, the computer can perform the method in the first aspect.
[0072] In a fifth aspect, the present application provides a chip, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof. Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.
[0073] In a sixth aspect, the present application provides a chip system, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof. Optionally, the chip system also includes a memory, and the memory is connected to the processor via a circuit or wire.
[0074] In a seventh aspect, the present application provides a computer program product, which includes a computer program (also referred to as instructions or codes). When the computer program is executed by an electronic device, the electronic device implements the method in the first aspect.
[0075] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0076] Figure 1 A schematic diagram of the sclera and iris in the eye region involved in an embodiment of the present application;
[0077] Figure 2 An illustration of the distorted eye light of a portrait in an image taken by a user in the related art after being enhanced by a face enhancement depth model Figure 1 ;
[0078] Figure 3 An illustration of the distorted eye light of a portrait in an image taken by a user in the related art after being enhanced by a face enhancement depth model Figure 2 ;
[0079] Figure 4 Schematic diagram of the process of processing the eye light image feature provided in the embodiment of the present application Figure 1 ;
[0080] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0081] Figure 6 A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;
[0082] Figure 7 Schematic diagram of the process of processing the eye light image feature provided in the embodiment of the present application Figure 2 ;
[0083] Figure 8 Schematic diagram of the process of processing the eye light image feature provided in the embodiment of the present application Figure 3 ;
[0084] Fig. 9 A schematic diagram of a human eye iris cutout in a method for processing eye light image features provided in an embodiment of the present application;
[0085] Fig.10 A schematic diagram of eye light / background separation in the method for processing eye light image features provided in an embodiment of the present application;
[0086] Fig.11 A schematic diagram of image upsampling in the method for processing eye light image features provided in an embodiment of the present application;
[0087] Fig.12 A schematic diagram of extracting eye highlight areas in a method for processing eye light image features provided in an embodiment of the present application;
[0088] Fig.13 A schematic diagram of upsampling, catch light / background separation, and downsampling in the method for processing catch light image features provided in an embodiment of the present application;
[0089] Fig.14 A schematic diagram of the fusion of the eye light and the background in the method for processing the eye light image feature provided in an embodiment of the present application;
[0090] Fig.15 A schematic diagram of three forms of affine transformation, namely, scaling, rotation and translation, in the method for processing eye light image features provided in an embodiment of the present application;
[0091] Fig.16 A schematic diagram of a method for processing eye light image features provided in an embodiment of the present application using affine transformation to enhance eye light features;
[0092] Fig.17 A schematic diagram of feathering processing in the method for processing eye light image features provided in an embodiment of the present application;
[0093] Fig.18 Schematic diagram of the implementation process of the method for processing eye light image features provided in the embodiment of the present application Figure 1 ;
[0094] Fig.19 Schematic diagram of the implementation process of the method for processing eye light image features provided in the embodiment of the present application Figure 2 ;
[0095] Fig. 20 A schematic diagram of the iris cutout process in the method for processing the eye light image feature provided in an embodiment of the present application;
[0096] Fig.21 A schematic diagram of a mask of an eye area, a key point of an eye center, and an iris area in an iris cutout process of a method for processing eye light image features provided in an embodiment of the present application;
[0097] Fig. 22 Schematic diagram of the process of extracting feature points of the iris area in the method for processing eye light image features provided in an embodiment of the present application Figure 1 ;
[0098] Fig.23 Schematic diagram of the process of extracting feature points of the iris area in the method for processing eye light image features provided in an embodiment of the present application Figure 2 ;
[0099] Fig.24 A schematic diagram of a process for selecting feature points of an iris region in a method for processing catch light image features provided in an embodiment of the present application;
[0100] Fig.25 A schematic diagram of the process of ellipse fitting in the method for processing eye light image features provided in an embodiment of the present application;
[0101] Fig.26 A schematic diagram of obtaining a mask of an iris region in the method for processing eye light image features provided in an embodiment of the present application;
[0102] Fig. 27A schematic diagram of the structure of a device for processing eye light image features provided in an embodiment of the present application. DETAILED DESCRIPTION
[0103] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0104] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.
[0105] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of objects. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two, for example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.
[0106] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0107] The catch light is used to create a bright spot of light in the eyes in portrait photography. The catch light usually creates a small round spot of light in the black pupil, which can make the eyes look lively. Figure 1 As shown, eye features may include sclera and iris, and the iris area is the black pupil. Among them, the catch light is usually displayed in the iris area.
[0108] The current portrait imaging effect of mobile phones in low light and long distance conditions is significantly lower than that in good light and close-up conditions. In order to improve the quality of portrait images taken in low light and long distance conditions, the mainstream method in the industry is usually to use an artificial intelligence (AI) depth model to enhance the face image on the original image after the user triggers the photo. The enhanced photo is clearer and has richer face details. For ease of explanation, the AI depth model for enhancing face images is referred to as the face enhancement depth model.
[0109] However, due to the low interpretability and controllability of the deep model for face image enhancement, the eye light features are often misprocessed after enhancement, and the processed eye light does not look natural enough, which does not meet the basic needs of users for the authenticity of the camera function. Eyes and eye light are one of the most prominent features of the face area. Changes in eye light are easily noticed by users. If the eye light enhancement effect is not realistic, it will greatly affect the overall effect of the face.
[0110] For example, if there is eye light in the original image, the face enhancement depth model will enhance the eye light, but the enhanced eye light does not look natural enough. Figure 2 (a) in the figure shows a low-quality portrait (original image) taken by a mobile phone under conditions such as low light and long distance. The original image has poor imaging effect, and there is catch light in the eye area. After the low-quality portrait is input into the face enhancement deep model for enhancement, the face enhancement deep model performs face enhancement on the low-quality portrait and outputs a high-quality portrait. Figure 2 (b) in the figure shows a high-quality portrait with face enhancement completed by a face enhancement deep model, in which the eye light in the eye area is enhanced, but the shape, size, intensity, etc. of the eye light are modified, so the enhanced eye light is unnatural and does not meet the user's basic demand for the authenticity of the photo-taking function.
[0111] For another example, when the original image does not have eye light, the face enhancement depth model will add eye light in the eye area, but the added eye light does not look natural enough. Figure 3 (a) in the figure shows a low-quality portrait (original image) taken by a mobile phone under conditions of low light and long distance, in which there is no catch light in the eye area. After the low-quality portrait is input into the face enhancement deep model for enhancement, the face enhancement deep model performs face enhancement on the low-quality portrait and outputs a high-quality portrait. Figure 3 (b) in the figure shows a high-quality portrait with face enhancement completed by the face enhancement deep model. Compared with the original image, eye light is added to the eye area. However, since there was no eye light originally, adding eye light will modify the original expression of the face. Therefore, adding eye light will appear unnatural and fail to meet the user's basic demand for the authenticity of the camera function.
[0112] How to controllably achieve eye light enhancement of low-quality images is a difficult problem that needs to be solved urgently.
[0113] In view of the above technical problems, the embodiments of the present application provide a method for processing eye light image features and an electronic device. For scenes where the eye light of a low-quality portrait is unnatural after being enhanced by a face enhancement depth model, the highlight area is first separated from the eye area of the low-quality portrait, and then the eye light contrast enhancement parameters are determined according to the attribute parameters of the highlight area. Then, the eye light is controllably enhanced according to the eye light contrast enhancement parameters. The enhanced eye light features are then used to replace the "abnormal" eye light features in the eye area of the high-quality portrait, thereby correcting the eye light features, thereby obtaining a high-quality portrait with "normal" eye light after enhancement, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0114] The embodiment of the present application can be implemented based on a computer vision algorithm, and the eye and eye light information in a low-quality portrait can be used to controllably enhance the natural enhancement of the eye light. In the embodiment of the present application, through improvements in the underlying system of the mobile phone, the eye light of the portrait in a low-quality image can be controllably enhanced without any additional hardware, so as to improve the user experience.
[0115] In order to better understand the embodiments of the present application, Figure 4 The embodiments of the present application are briefly described as follows:
[0116] Figure 4 The simplified algorithm flow of the present application is shown. Figure 4 As shown, in the embodiment of the present application, the eye light is controllably enhanced by performing image processing on low-quality portraits and high-quality portraits respectively.
[0117] The low-quality portrait may be a photo containing facial features taken by a user after pressing the shutter button of the camera of an electronic device under conditions such as dim light and long distance, or may be a photo containing facial features stored in an album or gallery of the electronic device. After the electronic device performs face enhancement on the low-quality portrait using a preset face enhancement depth model, a high-quality portrait is obtained. In a high-quality portrait, since the eye light after enhancement is unnatural, the image may be referred to as a high-quality portrait with "abnormal" eye light after enhancement.
[0118] In order to controllably enhance the eye light, first, based on the low-quality eye light features in the low-quality portrait, the embodiment of the present application adopts the eye light enhancement algorithm provided by the present application to achieve eye light enhancement and obtain enhanced eye light features. Then, the enhanced eye light features are used to replace the "abnormal" eye light features in the high-quality portrait, and the eye light features are corrected, so that a high-quality portrait with "normal" eye light after enhancement can be obtained. This satisfies the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0119] It should be noted that in the embodiments of the present application, "normal" eye light refers to the distortion effect of "abnormal" eye light. For example, if the eye light in a portrait is real and natural, it can be considered as "normal" eye light; or if there is no "abnormal" eye light in a portrait, it can also be considered as "normal" eye light.
[0120] The embodiments of the present application focus on the user's demand for the fidelity of the facial area in scenarios such as low-light shooting and telephoto shooting, and can controllably enhance the human eye's catch light to compensate for the unnatural defect of the eye light caused by full-face enhancement through the AI depth model.
[0121] Combination of the above Figure 4 The present invention describes a scheme for correcting "abnormal" eye light in a high-quality portrait when the original image has eye light: based on the low-quality portrait, it is determined that the original image has eye light, and then the eye light features are controllably enhanced based on the low-quality portrait. The enhanced eye light features are then used to replace the "abnormal" eye light features in the high-quality portrait, thereby correcting the "abnormal" eye light in the high-quality portrait to "normal" eye light, thereby satisfying the user's demand for the fidelity of the face area in shooting scenarios such as low-light shooting and telephoto shooting where low-quality portraits need to be enhanced by algorithms.
[0122] In addition, the embodiment of the present application also provides a solution for correcting the "abnormal" eye light in the high-quality portrait when the original image has no eye light: based on the low-quality portrait, it is judged that the original image has no eye light, and based on the high-quality portrait, the high-quality eye light feature and the high-quality human eye background are separated. Since the high-quality eye light feature appears as "abnormal" eye light, the high-quality eye light feature can be deleted and the high-quality human eye background can be retained, thereby correcting the "abnormal" eye light in the high-quality portrait, so that the high-quality portrait does not contain "abnormal" eye light, thereby meeting the user's demand for the fidelity of the face area in shooting scenarios such as low-light shooting and telephoto shooting where low-quality portraits need to be enhanced by algorithms.
[0123] The method for processing the eye light image feature provided in the embodiment of the present application can be applied to electronic devices with a shooting function. Figure 5, is a structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, an ambient light sensor 180L, and the like.
[0124] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0125] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. For example, the processor 110 is used to execute the processing method of the eye light image feature in the embodiment of the present application.
[0126] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0127] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0128] The external memory 120 generally refers to an external memory. In the embodiment of the present application, the external memory refers to a memory other than the memory of the electronic device and the cache of the processor, and the memory is generally a non-volatile memory.
[0129] The internal memory 121, which may also be referred to as "memory", may be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), and the like.
[0130] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel may be an organic light-emitting diode (OLED). In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0131] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[0132] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0133] The electronic device 100 also includes various sensors that can convert various physical signals into electrical signals.
[0134] The above is a specific description of the embodiment of the present application using the electronic device 100 as an example. It should be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.
[0135] The electronic device provided in the embodiments of the present application may be a user equipment (UE), for example, a mobile terminal (such as a user's mobile phone), a tablet computer, a desktop computer, a laptop computer, a handheld computer, a netbook, a personal digital assistant (PDA), and the like.
[0136] In addition, an operating system is running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on the operating system.
[0137] The operating system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.
[0138] Figure 6 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.
[0139] The layered architecture divides the system into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: application layer (applications), application framework layer (application framework), hardware abstraction layer (HAL), and kernel layer (kernel).
[0140] The application layer may include a series of application packages. In the embodiment of the present application, the application package may include a camera, a gallery, etc.
[0141] The application framework layer provides an application programming interface (API) and a programming framework for the application programs of the application layer. The application framework layer includes some predefined functions. In an embodiment of the present application, the application framework layer may include a camera access interface, wherein the camera access interface may include a human eye iris cutout service, a catch light / background separation service, a catch light enhancement service, and a catch light fusion service. The camera access interface is used to provide an application programming interface and a programming framework for the camera application.
[0142] The hardware abstraction layer is an interface layer located between the application framework layer and the kernel layer, and provides a virtual hardware platform for the operating system. In the embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.
[0143] The camera hardware abstraction layer can provide virtual hardware (referred to as camera HAL) for camera devices such as the front camera and the rear camera. The camera algorithm library can include a human eye iris cutout algorithm, a catch light / background separation algorithm, a catch light enhancement algorithm, and a catch light fusion algorithm. In other words, the camera algorithm library contains the operating code and data for implementing the processing method of the catch light image feature provided in the embodiment of the present application.
[0144] The kernel layer is a layer between hardware and software. The kernel layer includes drivers for various hardware. The kernel layer may include camera device drivers, digital signal processor drivers, and image processor drivers. Among them, the camera device driver is used to drive the camera sensor to capture images and drive the image signal processor to pre-process the image. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.
[0145] In addition, for the sake of convenience, Figure 6 The hardware layers that interact with the software layers are also shown. For example, the hardware layer includes the camera sensor, image signal processor, data signal processor, and image processor.
[0146] In combination with the above hardware structure and system structure, the following is an overview of the processing method of the eye light image feature in the embodiment of the present application:
[0147] (1) Image acquisition stage:
[0148] like Figure 7 As shown, the electronic device captures a first image (original image), and uses a face enhancement depth model to perform face enhancement processing on the first image to obtain a second image.
[0149] First, in response to the user's operation on the camera application icon (such as a click operation), the camera application calls the camera access interface of the application framework layer to start the camera application, and then sends an instruction to start the camera application by calling the camera device (such as the rear camera) in the camera hardware abstraction layer. The camera hardware abstraction layer sends the instruction to the camera device driver of the kernel layer. The camera device driver can start the sensor of the rear camera, that is, the electronic device starts the shooting function.
[0150] Then, in response to the user pressing the camera shutter button, the first image signal is collected by the rear camera, and the first image light signal is transmitted to the image signal processor for preprocessing to obtain a first image, which includes facial features. Then, the first image is transmitted to the camera hardware abstraction layer through the camera device driver. Then, the camera hardware abstraction layer can send the first image to the camera algorithm library.
[0151] Based on the support of digital signal processors and image processors, the camera algorithm library can first input the first image into a face enhancement depth model, perform face enhancement processing on the first image, and obtain a second image whose image quality is better than that of the first image.
[0152] For the sake of convenience, the first image is referred to as a low-quality portrait, and the second image is referred to as a high-quality portrait.
[0153] (2) Image processing stage:
[0154] like Figure 7 As shown, in the image processing stage, the image processing provided by the present application includes: ① human eye iris cutout; ② eye light / background separation; ③ eye light enhancement; ④ image fusion.
[0155] On the one hand, from the perspective of processing high-quality portraits with abnormal eye light after enhancement, the camera algorithm library first uses the human iris cutout algorithm provided in the present application to cut out the high-quality portrait, and extracts the human iris area image from the high-quality portrait; then the camera algorithm library uses the eye light / background separation algorithm provided in the present application to separate the eye highlight area and the eye background area of the high-quality portrait from the human iris area image, wherein the highlight feature in the eye highlight area of the high-quality portrait can be an "abnormal" eye light feature.
[0156] On the other hand, from the perspective of processing low-quality portraits, the camera algorithm library first uses the human eye iris cutout algorithm provided in this application to cut out the low-quality portrait, and extracts the human eye iris area image from the low-quality portrait; then the camera algorithm library uses the eye light / background separation algorithm provided in this application to separate the eye light features from the human eye iris area image, and obtains the eye highlight area and eye background area of the low-quality portrait.
[0157] In the embodiment of the present application, the catch light / background separation algorithm may include: first upsampling the human eye iris region image, then using a highlight removal algorithm to separate the eye highlight region (which may include the catch light feature) and the eye background region from the human eye iris region image, and then downsampling the separated eye highlight region and eye background region. The specific implementation process of the catch light / background separation algorithm will be described in detail below and will not be repeated here.
[0158] It should be noted that after downsampling the eye highlight area and the eye background area, taking into account the fact that part of the eyelid may remain in the iris area obtained by the iris cutout processing, and considering that the de-highlighting algorithm may have errors, it may appear that there is a highlight area in the iris area, but the highlight feature in the highlight area is not actually the catch light. In other words, the images taken by the user do not necessarily contain catch light, and the eye highlight area in the original image needs to be identified.
[0159] This application is based on the principle of high-fidelity enhancement, and determines whether there is eye light in the original image according to the parameters of the eye highlight area separated from the iris area of the human eye. The parameters of the highlight area may include the color moment, contrast, average brightness, saturation and / or area of the highlight area. Then, according to whether there is eye light in the original image, the corresponding eye light processing strategy is selected.
[0160] For example, in some scenarios, such as Figure 7 As shown, if there is eye light in the original image, the eye light enhancement algorithm provided by this application is used to perform "eye light enhancement" based on the low-quality eye light features. Then, the eye light fusion algorithm provided by this application is used to perform "image fusion" on the enhanced eye light features and the eye background area of the high-quality portrait.
[0161] For example, in some other scenarios, if there is no eye light in the original image, eye light enhancement and image fusion will not be performed, and the eye highlight area and eye background area of the low-quality portrait will be discarded, as well as the "abnormal" eye light features of the high-quality portrait. Only the eye background area of the high-quality portrait will be retained to meet the user's demand for the fidelity of the facial area in shooting scenarios such as low-light shooting and telephoto shooting where low-quality portraits need to be enhanced by algorithms.
[0162] It should be noted that after obtaining the eye highlight area and eye background area of the low-quality portrait and the eye highlight area and eye background area of the high-quality portrait, the eye background area of the low-quality portrait and the eye highlight area of the high-quality portrait are discarded.
[0163] (3) Image generation stage:
[0164] On the one hand, when there is eye light in the original image, the electronic device will perform "image fusion" on the enhanced eye light features with the eye background area of the high-quality portrait to obtain a high-quality portrait with controllable enhancement of the eye light.
[0165] That is to say, in order to controllably enhance the eye light, first, the embodiment of the present application uses the eye light enhancement algorithm provided by the present application based on the low-quality eye light features in the low-quality portrait to achieve eye light enhancement and obtain enhanced eye light features. Then, the enhanced eye light features are used to replace the "abnormal" eye light features in the high-quality portrait, and the eye light features are corrected, so that a high-quality portrait with "normal" eye light after enhancement can be obtained. In this case, the portrait in the photo obtained by image processing has eye light, and the eye light is normal, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0166] On the other hand, when there is no eye light in the original image, the electronic device discards the "abnormal" eye light features and only retains the eye background area of the high-quality portrait. In this way, there is no "abnormal" eye light in the high-quality portrait output at the end, thereby correcting the eye light. In this case, the portrait in the photo obtained by image processing does not contain "abnormal" eye light, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portrait photos.
[0167] After obtaining the photo through image processing, the camera algorithm library can send the photo obtained through image processing to the camera hardware abstraction layer. Then, the camera hardware abstraction layer can send the photo for display.
[0168] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.
[0169] The execution subject of the method for processing the eye light image feature provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the method for processing the eye light image feature, and the present application scheme can be implemented by hardware and / or software, which can be determined according to actual use requirements and is not limited by the embodiment of the present application. The following is an exemplary description of the method for processing the eye light image feature provided in the embodiment of the present application, taking an electronic device as an example.
[0170] Combine the following Figure 8 Describe the implementation process of the overall solution of this application. Figure 8 As shown, the method for processing eye light image features provided in the embodiment of the present application includes the following steps S201-S208.
[0171] First of all, it should be noted that in the embodiment of the present application, when the face enhancement function of the electronic device is turned on, the electronic device will identify whether the photographed photo contains facial features during the process of taking the photo. When the electronic device identifies that the photographed photo contains facial features and the image quality does not meet the preset image quality requirements, the electronic device uses the face enhancement depth model to process the photo to achieve full face enhancement. For the sake of convenience, the face enhancement function is turned on by default in the embodiment of the present application.
[0172] The embodiments of the present application focus on the user's demand for the fidelity of the face area in scenes such as low-light shooting and telephoto shooting, and can controllably enhance the eye light of the human eye to compensate for the defect of unnatural eye light caused by full-face enhancement through a face enhancement depth model.
[0173] S201A, obtain low-quality portrait.
[0174] S201B, obtain high-quality portraits.
[0175] The high-quality portrait is an image obtained by processing the low-quality portrait with the face enhancement deep model. After the low-quality portrait is processed with the face enhancement deep model, the eye light feature is often mistakenly enhanced, and the enhanced eye light looks very unnatural.
[0176] Among them, the low-quality portrait can also be called a low-quality facial image or a low-quality portrait original image or a low-quality original image, and the high-quality portrait can also be called a high-quality facial image or a high-quality portrait abnormal image or a high-quality abnormal image.
[0177] In the embodiment of the present application, in order to compensate for the unnatural defect of eye light caused by full face enhancement through the face enhancement depth model, the electronic device provides an automatic eye light correction function, which can implement the processing method of eye light image features provided in the embodiment of the present application. The electronic device can turn on or off the automatic eye light correction function in response to user operation.
[0178] In the case where the electronic device does not have the automatic correction function for eye light turned on, when the user triggers the photo shooting, the electronic device will input the original photo taken (called a low-quality portrait) into the preset face enhancement depth model to obtain a high-quality portrait with face enhancement. At this time, in the high-quality portrait, while the face is enhanced, the eye light is also enhanced, and the enhanced eye light may be unnatural. It should be noted that in the high-quality portrait, since the enhanced eye light is unnatural, the image can be called a high-quality portrait with "abnormal" enhanced eye light.
[0179] When the automatic eye light correction function of the electronic device is turned on, when the user triggers the photo taking function, the electronic device will simultaneously obtain the original photo (low-quality portrait) and the high-quality portrait obtained through the face enhancement depth model.
[0180] In some embodiments, a low-quality portrait is obtained by: under conditions of dim light, long distance, etc., after a user presses a camera shutter button of the electronic device, the electronic device takes a photo containing facial features.
[0181] S202A-S203A, perform iris cutout processing on the low-quality portrait to obtain the iris region of the low-quality portrait. Perform catch light / background separation processing on the iris region of the low-quality portrait to obtain the eye highlight region and eye background region of the low-quality portrait.
[0182] S202B-S203B, perform iris cutout processing on the high-quality portrait to obtain the iris / sclera region of the high-quality portrait. Perform catch light / background separation processing on the iris / sclera region of the high-quality portrait to obtain the eye highlight region and eye background region of the high-quality portrait.
[0183] Among them, S202A and S203A are for image processing of low-quality portraits. S202B and S203B are for image processing of high-quality portraits. The image processing of low-quality portraits is similar to the image processing of high-quality portraits. The processing method of the eye light image feature of the portrait provided by the present application is described in detail below in conjunction with the accompanying drawings.
[0184] For example, Fig. 9 As shown, the eye region in the portrait is first detected, such as the part circled by the black dotted line; then the eye region is subjected to eye cutout processing to obtain the eye image. Next, the iris region is detected in the eye image, such as the part circled by the white dotted line; then the iris cutout processing is performed on the iris region, and the iris image can be extracted from the portrait. The specific implementation process of the iris cutout algorithm will be described in detail below, which will not be repeated here.
[0185] In addition, the human eye image can also be processed by cutting out the human eye sclera. Fig. 9 As shown, the human eye sclera area is detected in the human eye image, and the annular part circled by the white dotted line is the human eye sclera area; then after performing human eye sclera cutout processing on the human eye sclera area, the human eye sclera image can be extracted from the portrait.
[0186] In some embodiments, if the iris region of a human eye is extracted from a human portrait, the portion of the iris region of the human eye excluding the highlight region may be referred to as the eye background region.
[0187] In other embodiments, if the sclera region of the human eye is extracted from the portrait, the portion of the sclera region of the human eye except the highlight region may also be referred to as the eye background region.
[0188] In some further embodiments, if the iris region and the sclera region are extracted from the portrait, the portions of the iris region and the sclera region except the highlight region may be referred to as the eye background region.
[0189] In the embodiment of the present application, the “eye background area” may also be referred to as the human eye iris / sclera background area, which covers the following three situations:
[0190] (1) “Eye background area” refers to the part of the iris area of the human eye excluding the highlight area.
[0191] (2) “Eye background area” refers to the part of the sclera area of the human eye excluding the highlight area.
[0192] (3) “Eye background area” refers to the iris area and the sclera area of the human eye excluding the highlight area.
[0193] It should be noted that the embodiment of the present application supports eye light / background separation processing for the iris area of the human eye, and can also perform eye light / background separation processing for the sclera area of the human eye.
[0194] For ease of explanation, each embodiment of the present application is exemplified by taking the eye light / background separation processing of the iris area of the human eye as an example.
[0195] For example, if Fig.10 As shown, first, the human eye iris cutout algorithm is used to process the portrait, and the human eye iris area is determined in the portrait. Then, the eye light / background separation algorithm is used to process the human eye iris area image to obtain the eye highlight area and the eye background area. In addition, the application will also perform eye light recognition based on whether the eye highlight area contains eye light, and the specific implementation process will be described in detail below.
[0196] In an embodiment of the present application, the catch light / background separation algorithm includes upsampling the human eye iris image, separating the eye highlight area and the eye background area from the human eye iris image based on a highlight removal algorithm, and downsampling the eye highlight area and the eye background area.
[0197] (1) Upsample the human iris image.
[0198] Here, upsampling refers to enlarging the image.
[0199] In the embodiment of the present application, upsampling can be achieved by using an interpolation method or a deconvolution (also called a transposed convolution) method. For example, the resize() function in the open source computer vision library OpenCV can be called to adjust the image size by an interpolation algorithm (such as a bilinear interpolation algorithm).
[0200] In order to make the subsequent catch light / background separation more detailed, the iris / sclera area of the human eye is first upsampled to achieve image magnification, and then the upsampled iris / sclera area of the human eye is subjected to catch light / background separation processing.
[0201] In order to reduce the noise amplification caused by upsampling and the jagged edges of the iris and the eye light, the present application uses filtering and anti-aliasing upsampling alternately to achieve image magnification.
[0202] Fig.11 Schematic diagram of the upsampling process of this application is shown. Fig.11 As shown, the upsampling process includes steps S2031-S2035. Exemplarily, assuming that the pixel size of the human eye iris region image (which can be referred to as the human eye iris image) in the portrait is about 20×20, the human eye iris image is firstly anti-aliased up-sampled, thereby realizing that the human eye iris image is enlarged, while avoiding the appearance of jagged edges of the iris edge and the eye light edge. For example, the pixel size of the enlarged human eye iris image is 100×100. Then, the enlarged human eye iris image is filtered and denoised to remove noise. Then, it is determined whether the enlarged human eye iris image reaches a specified size. Exemplarily, the specified size is 200×200. When the enlarged human eye iris image does not reach the specified size, the human eye iris image continues to be anti-aliased up-sampled and filtered and denoised until the enlarged human eye iris image reaches the specified size of 200×200, and then the human eye iris image is no longer anti-aliased up-sampled and filtered and denoised to obtain a human eye iris image with an enlarged size.
[0203] Through the solution of the present application, after multiple (for example, 2 to 3 times) cycles of anti-aliasing upsampling and filtering denoising processing, the human eye iris image is gradually enlarged (for example, 10 times), so that the subsequent eye light / background separation is more detailed, thereby improving the accuracy of image processing.
[0204] (2) Based on the highlight removal algorithm, the eye highlight area and eye background area are separated from the human iris image.
[0205] In the embodiments of the present application, taking into account that the highlight removal (highlight i.e., catch light) process will not produce negative effects on the background area of the human eye, such as smearing, missing, brightness change of non-catch light area, etc., the present application adopts a simple and efficient universal highlight removal algorithm, which can quickly realize the separation of highlights and background on larger-sized human iris images.
[0206] Through the highlight removal algorithm, the highlight area can be detected in the iris area of the human eye. For example, the mask of the highlight area can be generated by image threshold segmentation. Among them, the flood filling method can be used to solve the connected components of the mask. The connected component is defined as: a maximal connected subgraph of an undirected graph.
[0207] Assuming that the image imaging conforms to the dichotomous reflection model, the image I is decomposed into diffuse reflection components (denoted as V diff ) and the specular component (denoted as V spec ), the diffuse reflection component corresponds to the de-eye light map, and the specular reflection component corresponds to the eye light map. After theoretical derivation, the approximate specular reflection component is expressed as β s :
[0208] β s =I min -τ(p) (Equation 1)
[0209] Among them, I min It represents a single-channel image obtained by taking the minimum value of the RGB three channels of each pixel in image I; τ(p) represents an offset related to the mean and variance of the image pixels.
[0210] Denote the approximate diffuse component as V' diff :
[0211] V d ' iff =I-kβ s (Equation 2)
[0212] In order to solve the proportional coefficient k, the image is first normalized to a pixel interval of [0,1], and the area with a pixel value greater than 0.1 is regarded as a possible highlight area. Then the flood fill method (also known as the closed area filling algorithm) is used to obtain the dominant highlight connected component area (the first M largest) of the area. It should be noted that the diffuse reflection color intensity of the highlight connected component area and its edge area should be consistent, that is, the mean of the diffuse reflection component is equal. Under this smooth color transition criterion, the k value is obtained by the least squares method. Then substitute the k value into equation 2 to obtain the de-eye catch light map V′. diff Then, the eye light map is obtained by the following equation 3:
[0213] V s ' pec =IVd ' iff (Equation 3)
[0214] Among them, V′ diff represents the removed eye-catching light image, V′ spec Indicates the eye light pattern.
[0215] It should be noted that when the present application uses the de-highlighting algorithm to perform eye light / background separation, considering that the eye light area in the actual scene may not be continuous and there may be multiple connected components, the first M largest connected components that "may be eye lights" are determined as highlight areas that need to be separated.
[0216] For example, if M is 2, assuming that there are four connected components in the iris region of the human eye: 7, 20, 9, 5, then Fig.12 As shown, the area where the first two largest connected components "20 and 9" of "possibly catch light" in the iris area of the human eye are located can be determined as the highlight area that needs to be separated.
[0217] In the embodiment of the present application, the catch light area mask is obtained by an automatic highlight removal algorithm, which can support the separation of discontinuous catch light areas of arbitrary shapes.
[0218] Through this application solution, the iris, sclera and eye light areas can be automatically identified without manual demarcation.
[0219] (3) Downsample the eye highlight area and eye background area.
[0220] Downsampling refers to reducing the size of an image. Similar to upsampling, downsampling can be achieved by using interpolation, pooling, or convolution.
[0221] Since the subsequent steps will have a great impact on whether the final eye light enhancement effect is natural, in order to prevent it from being affected by the interpolation algorithm, avoid post-processing after image fusion, and to improve the efficiency of the algorithm, this step will downsample the separated eye highlight area and eye background area to the original image size (such as reducing the current image to 1 / 10).
[0222] Exemplarily, the pixel sizes of the separated eye highlight area and eye background area are 50×50 and 200×200 respectively. After downsampling, the pixel sizes of the eye highlight area and eye background area are 5×5 and 20×20 respectively.
[0223] It should be noted that if the image is directly downsampled, for example, using nearest linear interpolation, it is found that the downsampled image will have a lot of jagged edges. In some embodiments, the present application can use anti-aliasing downsampling to scale the size to the original image size to avoid jagged edges on the iris edge and the eye light edge.
[0224] Fig.13 FIG. 1 is a schematic diagram showing an embodiment of the present invention for performing catch light / background separation on an image of the iris region of a human eye. Fig.13 As shown in the figure, taking the catch light / background separation of the human eye iris area image of a low-quality portrait as an example, the human eye iris area image is first upsampled and the human eye iris area image is enlarged; then the enlarged human eye iris area image is separated and processed by the catch light / background separation algorithm to obtain the eye highlight area and the eye background area; then the separated eye highlight area and eye background area are downsampled and the image size is reduced to the original image size, thereby obtaining the low-quality eye highlight area and eye background area.
[0225] It should be noted that the eye highlight area of a low-quality portrait may contain eye light features. If the low-light area may contain eye light features, then the eye light features will be enhanced to replace the "abnormal" eye light. Therefore, in subsequent image processing, the eye highlight area of the low-quality portrait will be used to identify the eye light features. In addition, the eye background area of the low-quality portrait is not used in subsequent image processing, so the eye background area of the low-quality portrait is discarded.
[0226] It should also be noted that, similar to the above process of performing catch light / background separation on the iris region image of the low-quality portrait, the catch light / background separation is performed on the iris region image of the high-quality portrait to obtain the eye background area and eye highlight area of the high-quality portrait. Among them, the highlight feature in the eye highlight area of the high-quality portrait may be an "abnormal" catch light feature, which needs to be replaced with a "normal" catch light feature. Therefore, the eye highlight area of the high-quality portrait will be discarded.
[0227] It should also be noted that in the embodiments of the present application, "normal" eye light refers to the distortion effect of "abnormal" eye light. For example, if the eye light in a portrait is real and natural, it can be considered as "normal" eye light; or if there is no "abnormal" eye light in the portrait, it can also be considered as "normal" eye light.
[0228] S204: judging whether the low-quality portrait has an eye light feature according to the highlight attribute of the eye highlight area of the low-quality portrait.
[0229] In an embodiment of the present application, after the eye highlight area and the eye background area are separated by the eye light / background separation algorithm, taking into account the fact that part of the eyelid may remain in the human eye iris area obtained by the human eye iris cutout processing, and taking into account the possible errors in the de-highlighting algorithm, it may happen that there is a highlight area in the human eye iris area but the highlight feature in the highlight area is not actually the eye light. In other words, the original image taken by the user does not necessarily contain the eye light, and it is necessary to identify the eye highlight area in the original image, that is, to judge whether the low-quality portrait has the eye light feature.
[0230] Considering that not all images taken by users contain eye light, this application is based on the principle of high-fidelity enhancement. If a low-quality portrait does not have eye light features, the mistakenly added eye light features can be directly removed from the corresponding high-quality portrait. Since the eye light / background separation algorithm does not change the iris / sclera background of the human eye, the detailed information of the iris / sclera area of the high-quality portrait can be retained.
[0231] In the embodiment of the present application, the area of the eye highlight area separated from the human eye iris area can be used to determine whether there is catch light in the original image. For example, if the area of the highlight area is 0, it can be directly determined that there is no catch light in the highlight area. However, due to factors such as the fact that the human eye iris portion obtained by cutting out the image may have some eyelids remaining, and the de-highlighting algorithm may have errors, even if the area of the highlight area is not 0, the highlight area may still be without catch light.
[0232] In order to improve the accuracy of eye light recognition, in the embodiment of the present application, a joint judgment can be made based on multiple highlight attributes of the eye highlight area separated from the human eye iris area, so that it can be more accurately judged whether there is eye light in the original image. Among them, the highlight attributes of the eye highlight area may include attributes such as color moment, contrast, average brightness, saturation and / or area of the highlight area.
[0233] In some embodiments, the present application selects to use multiple hyperparameters as features to train a simple binary classifier offline. Among them, the hyperparameters include color moment, contrast, average brightness, saturation, highlight area, etc. It should be noted that in the context of machine learning, hyperparameters are parameters whose values are set before starting the learning process. By optimizing the hyperparameters, a set of optimal hyperparameters is selected for the learning machine to improve the performance and effect of learning.
[0234] For example, taking the binary classifier as a support vector machine (SVM), multiple highlight attributes of the eye highlight area are input into the trained SVM, and the SVM can directly output the eye light recognition result. Through the present application scheme, it is possible to comprehensively judge whether the original image has eye light based on multiple highlight attributes of the eye highlight area, which can improve the accuracy of eye light recognition.
[0235] If the eye highlight region of the portrait has a catch light feature, then continue to execute S205 - S206 ; if the eye highlight region of the portrait does not have a catch light feature, then continue to execute S207 .
[0236] S205: If the original image has eye light features, adaptively enhance the eye light features.
[0237] It should be noted that the size, shape, brightness and other characteristics of the original area of the eye light are all random and uncertain, so the same enhancement parameters (such as fixed contrast stretching parameters) are difficult to be compatible with all situations.
[0238] In order to increase the generalization of the algorithm, this application uses an adaptive catch light contrast enhancement method. First, the length and width of the catch light area (for example, the length and width of the minimum circumscribed rectangle of the catch light area), average brightness, brightest point brightness and other parameters are calculated, and then the corresponding contrast enhancement algorithm parameters are calculated. Among them, the parameters of the contrast enhancement algorithm include the gamma value for gamma correction, and / or the partition block size and contrast limit for limiting contrast adaptive histogram equalization. The following introduces the application of the parameters of the contrast enhancement algorithm in this application.
[0239] First, let's explain the gamma value. The gamma value is used to perform gamma correction on the highlight area to stretch the image contrast in the highlight area. The image corrected by a certain gamma value can be obtained through the gamma correction function. The gamma correction function can be expressed as:
[0240] I′=I γ
[0241] Among them, γ represents the gamma value, I represents the original image, and I′ represents the image after contrast enhancement.
[0242] For example, the gamma value corresponding to the highlight area may be calculated according to the ratio of the brightness of the brightest point in the highlight area to 255. The gamma correction is performed on the highlight area according to the gamma value to stretch the image contrast of the highlight area.
[0243] The following is an explanation of the block size and contrast limit of contrast-limited adaptive histogram equalization.
[0244] First of all, it should be noted that histogram equalization is a method to enhance image contrast, which changes the histogram distribution of an image into an approximately uniform distribution, thereby enhancing the contrast of the image.
[0245] Ordinary histogram equalization transforms all pixels using the same transformation derived from the image histogram. This works well when the distribution of pixel values is similar across the image. However, when the image contains areas that are significantly brighter or darker than the bulk of the image, the contrast in these areas will not be adequately enhanced.
[0246] Adaptive histogram equalization (AHE) transforms each pixel by using a transformation function obtained from the histogram calculated from the neighborhood. Therefore, it is suitable for improving the local contrast of an image. However, AHE tends to over-amplify noise in relatively uniform areas of the image.
[0247] In order to avoid discontinuity and over-enhancement caused by adaptive histogram equalization, a method of limiting the histogram distribution is introduced, namely, contrast-limited adaptive histogram equalization (CLAHE). Contrast limitation is achieved by limiting the ratio of the number of pixels per column of each local area histogram to the average grayscale distribution of the entire image.
[0248] OpenCV's CLAHE implementation includes two important parameters:
[0249] (1) The block size, or window size, is to divide the input image into non-overlapping sub-blocks of equal size, each containing a certain number of pixels, for example, 4 × 4. This division can effectively avoid the problem of block discontinuity.
[0250] (2) Clipping threshold (clipLimit), also known as contrast limit, for example, a value of 20.
[0251] clipLimit is the clipping limit value of CLAHE. When the pixel data of the histogram group of each block image of the image exceeds the limit value, clipping is required. In other words, for the histogram of the original image, a threshold is set. If a grayscale level of the histogram exceeds the threshold, it is clipped, and then the part exceeding the threshold is evenly distributed to each grayscale level, so that the result of the mapping function becomes smoother. If there is still an excess after the distribution, the above steps can be recursively repeated.
[0252] When it is necessary to explain, in addition to the partition block size and the clipping threshold, CLAHE can also include parameters such as the number of partition blocks, contrast stretching, and the number of histogram bins.
[0253] The specific implementation of CLAHE mainly includes the following steps:
[0254] Step 1: Block division: Divide the input image into non-overlapping sub-blocks of equal size, for example, each sub-block contains 4×4 pixels.
[0255] Step 2: Calculate the sub-block histogram.
[0256] Step 3: Calculate the clipping threshold. Redundant pixels can be clipped based on the clipping threshold.
[0257] Step 4: Pixel redistribution: For each sub-block, use the redundant pixels in step 3 to redistribute.
[0258] Step 5: Reconstruct the grayscale value of the pixel. By using bilinear interpolation and Gaussian smoothing, the noise generation can be reduced.
[0259] When this method is used to enhance the eye light, each channel of the eye light map needs to be processed. For example, the side length of the partition block (i.e., the partition block size) is min(L / 2,3), and the clipping threshold is max(D / 2,2). Where L represents the shortest side of the eye light area, and D represents the maximum value of the histogram in the circumscribed rectangle of the maximum highlight area. min() means taking the minimum value, and max() means taking the maximum value.
[0260] In the embodiment of the present application, each eye light area can be adaptively enhanced in a targeted manner. Specifically, the enhancement parameters corresponding to a certain eye light area can be calculated based on the size, shape, brightness and other characteristics of the eye light area, and the eye light area can be enhanced in a targeted manner using the enhancement parameters corresponding to the eye light area. Therefore, the embodiment of the present application can adaptively enhance different eye light areas.
[0261] For example, Table 1 shows parameters of the catch light area, such as length, width, average brightness, and brightest point brightness. As shown in Table 1, the catch light area includes a catch light area 1 and a catch light area 2.
[0262] Table 1
[0263] length width Average brightness Brightest point brightness Catch light area 1 8 pixels 7 pixels 100 gray levels 155 gray levels Catch light area 2 3 pixels 3 pixels 80 gray levels 115 gray levels
[0264] As shown in Table 1, in the case of a 256-grayscale image signal, the size of eye light area 1 is larger than the size of eye light area 2, the average brightness of eye light area 1 is larger than the average brightness of eye light area 2, and the brightness of the brightest point in eye light area 1 is larger than the brightness of the brightest point in eye light area 2.
[0265] After calculating the length, width, average brightness, and brightness of the brightest point in the catch light area, the parameters of the corresponding contrast enhancement algorithm are calculated. Here, the calculation of gamma values for different areas is used as an example for illustrative description. As mentioned above, the gamma value can be calculated based on the ratio of the brightness of the brightest point to 255. The following Table 2 shows the gamma values corresponding to different catch light areas.
[0266] Table 2
[0267] Brightest point brightness Calculated gamma value Catch light area 1 155 gray levels 0.61 Catch light area 2 115 gray levels 0.45
[0268] Through the solution of the present application, for the catch light area 1, a gamma value of 0.61 is used for gamma correction to stretch the image contrast of the catch light area 1. For the catch light area 2, a gamma value of 0.45 is used for gamma correction to stretch the image contrast of the catch light area 2. That is to say, in the embodiment of the present application, corresponding gamma values can be used for correction for different highlight areas.
[0269] For catch light areas with different characteristics such as size, shape, brightness, etc., the embodiment of the present application can perform targeted enhancement processing on each catch light area respectively. Specifically, the enhancement parameter A corresponding to catch light area 1 can be calculated according to the size, shape, brightness and other characteristics of catch light area 1, and the enhancement parameter A corresponding to catch light area 1 can be used to enhance the catch light area 1 in a targeted manner; similarly, the enhancement parameter B corresponding to catch light area 2 can be calculated according to the size, shape, brightness and other characteristics of catch light area 2, and the enhancement parameter B corresponding to catch light area 2 can be used to enhance the catch light area 2 in a targeted manner. Therefore, the embodiment of the present application can make different catch light areas adaptively enhanced.
[0270] It should be noted that the above embodiment uses the example of calculating the parameters of the contrast enhancement algorithm according to the length, width, average brightness, and brightest point brightness of the eye light area as an example for illustrative description. It can be understood that in actual implementation, the parameters of the contrast enhancement algorithm can also be calculated according to other parameters of the eye light area. The specific parameters can be determined according to actual use requirements and are not limited in the embodiments of this application.
[0271] It should be noted that the present application can use the above-mentioned catch light enhancement algorithm to adaptively enhance the eye light in the original image. The catch light enhancement algorithm uses an adaptive catch light contrast enhancement method to controllably enhance the eye light. The catch light enhancement algorithm provided by the present application is not a deep model, and can avoid the problem of abnormalities after the eye light enhancement caused by the deep model, and meet the user's demand for the fidelity of the face area in scenes such as low-light shooting and telephoto shooting.
[0272] S206: performing image fusion of the enhanced catch light feature and the eye background area of the high-quality portrait.
[0273] In order to controllably enhance the eye light, first, based on the low-quality eye light features in the low-quality portrait, the embodiment of the present application adopts the eye light enhancement algorithm provided by the present application to achieve eye light enhancement and obtain enhanced eye light features. Then, the enhanced eye light features are used to replace the "abnormal" eye light features in the high-quality portrait, and the eye light features are corrected, so that a high-quality portrait with "normal" eye light after enhancement can be obtained. This satisfies the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0274] It should be noted that for low-quality portraits and high-quality portraits, the positions and relative positions of the iris and eye light are not exactly the same, and the edge of the eye light feature of the low-quality portrait may not be softly attenuated after adaptive enhancement. If the enhanced eye light feature is directly copied and pasted onto the iris / sclera background of the high-quality portrait, the eye light effect is unnatural. In order to make the enhanced eye light blend naturally with the iris / sclera background of the high-quality portrait, the embodiment of the present application adopts the method of affine transformation and eye light area edge feathering.
[0275] For example, Fig.14 FIG. 1 is a schematic diagram showing the process of merging the eye light with the human eye iris background provided by an embodiment of the present application. Fig.14 As shown, first, an affine transformation matrix is determined based on the edge key points of the iris background area of the low-quality portrait and the edge key points of the iris background area of the high-quality portrait. Then, the enhanced eye light is multiplied by the affine transformation matrix, and an affine transformation is performed to obtain an aligned enhanced eye light. Then, the aligned enhanced eye light is adaptively feathered to obtain a feathered enhanced eye light. Then, the feathered enhanced eye light is image-fused with the iris background area. As a result, the controllably enhanced eye light of the present application can be naturally fused to the eye background area of the high-quality portrait.
[0276] It should be noted that here, affine transformation and image fusion based on the iris background area are used as an example for explanation. It can be understood that in actual implementation, affine transformation and image fusion can also be performed based on the sclera background area. The specific implementation method can be determined according to actual usage requirements, and the embodiments of this application are not limited to this.
[0277] The following first introduces the principle of affine transformation, and then explains how this application performs affine transformation.
[0278] During the affine transformation of an image, it is necessary to operate on each pixel in the image. Assuming that the coordinates of the pixel point X in the image are (x, y), the pixel point X can be expressed in matrix form as follows:
[0279]
[0280] like Fig.15 As shown, an affine transformation may include rotation, scaling, and translation. An affine transformation is a linear transformation plus a translation.
[0281] The matrix of the rotation parameter R and the matrix of the translation parameter T are given as examples below.
[0282]
[0283]
[0284] Among them, R is an orthogonal matrix, θ represents the rotation angle, and T represents the translation distance. In addition, s can be used to represent the scaling ratio.
[0285] In some embodiments, after rotation, the pixel point X can be expressed in matrix form as:
[0286]
[0287] In some embodiments, after translation, the pixel point X can be expressed in matrix form as:
[0288]
[0289] In some embodiments, after scaling, the pixel point X can be expressed in matrix form as:
[0290]
[0291] In other embodiments, after image affine transformations such as rotation, translation, and scaling, the coordinates of pixel point X become (u, v), and pixel point X can be expressed in matrix form as follows:
[0292]
[0293] Based on the above principles, the process of performing affine transformation based on the enhanced eye light features and the human eye background area in the embodiment of the present application is described below.
[0294] For example, if Fig.16 As shown, the edge feature points of the eye background area of the low-quality portrait can be calibrated, and the corresponding pixel matrix is recorded as F1, and the edge feature points of the eye background area of the high-quality portrait can be calibrated, and the corresponding pixel matrix is recorded as F2.
[0295] Assume that the affine transformation matrix is M, and the pixel matrix corresponding to the enhanced eye light feature is X. According to F1*M=F2, the affine transformation matrix M between the low-quality portrait and the high-quality portrait can be converted. Exemplarily, the converted affine transformation matrix M can be expressed as:
[0296]
[0297]
[0298] Among them, matrix M represents the complete affine transformation matrix, matrix A controls rotation and scaling, and matrix B controls translation.
[0299] Then, the pixel matrix X corresponding to the enhanced catch light feature is multiplied by the affine transformation matrix M to obtain the aligned enhanced catch light region X'. In addition, it is known that the affine transformation is a linear transformation plus a translation. After the affine transformation, the aligned enhanced catch light region X' can be expressed by a matrix as follows:
[0300]
[0301]
[0302] Through affine transformation processing, the enhanced eye light feature is aligned with the eye background area of the high-quality portrait in position, so as to ensure that the positions of the enhanced eye light feature and the eye background area are accurately aligned during image fusion. Since the unchanged position of the eye light feature relative to the iris is an important factor in the unchanged eye expression, after the eye light feature is aligned with the iris, a natural and realistic eye light effect can be presented.
[0303] After completing the affine transformation, the edge of the catch light can also be feathered. Fig.17 As shown, by feathering the catch light, the inner and outer connecting parts of the edge of the catch light are blurred to achieve a gradual change, thereby achieving a natural connection effect and improving the image display effect.
[0304] It should be noted that the present application utilizes affine transformation and feathering processing to naturally blend the enhanced eye light into the eye area of the high-quality abnormal image, rather than simply copying and pasting the eye light.
[0305] The eye light fusion algorithm provided in the embodiment of the present application can naturally fuse the enhanced eye light feature with the human eye background area, so that the enhanced eye light feature can be aligned with the human eye background area in position, and the edge of the enhanced eye light feature is smooth, which can present a natural and realistic eye light effect.
[0306] S207, discarding the eye highlight area of the low-quality portrait and retaining only the eye background area of the high-quality portrait.
[0307] Through the above S205-S206 or S207, a high-quality portrait with normal eye light can be finally obtained, and the defect of unnatural eye light caused by full face enhancement through the face enhancement depth model can be corrected, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting that require algorithm-enhanced low-quality portrait photos.
[0308] Through the application scheme, the application focuses on the user's demand for the fidelity of the face area in scenes such as low-light shooting and telephoto shooting, and can controllably enhance the eye light of the human eye to make up for the unnatural defect of the eye light caused by the full face enhancement through the face enhancement depth model. In addition, in the case of low-quality portraits without eye light features, the eye light features that were added by mistake can be directly removed from the corresponding high-quality portraits, thereby correcting the unnatural eye light caused by the full face enhancement through the face enhancement depth model. That is, the defect of mistaken addition.
[0309] In summary, the embodiments of the present application can select a corresponding eye light processing strategy according to whether the low-quality portrait has eye light features.
[0310] Eye light processing strategy 1: If the original image has eye light features, the eye light is enhanced according to the eye light features, and then the enhanced eye light features are fused with the eye background area of the high-quality portrait.
[0311] For example, in some embodiments, when the eye highlight area of a low-quality portrait meets a preset eye light condition, it is determined that the original low-quality portrait image has eye light, and eye light enhancement processing is performed based on the eye highlight area of the low-quality portrait to obtain enhanced eye light features. The enhanced eye light features are image fused with the eye background area of the high-quality portrait to obtain a high-quality portrait with normal eye light after enhancement, thereby achieving controllable enhancement of eye light. In this case, the portrait in the photo obtained by image processing has eye light, and the eye light is normal, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm enhancement of low-quality portrait photos.
[0312] Eye light processing strategy 2: If there is no eye light in the original image, no eye light enhancement or image fusion will be performed, and only the eye background area of the high-quality portrait will be retained.
[0313] For example, in some embodiments, when the eye highlight area of a low-quality portrait does not meet the preset eye light condition, it is determined that the original image has no eye light, and then the eye light enhancement process is not performed, the low-quality eye light features are discarded, and only the eye background area of the high-quality portrait is retained. In this way, there is no "abnormal" eye light in the high-quality portrait output at the end, thereby correcting the eye light and meeting the user's demand for the fidelity of the face area in scenes such as low-light shooting. In this case, the portrait in the photo obtained by image processing does not contain "abnormal" eye light, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm enhancement of low-quality portraits.
[0314] Fig.18 FIG. 1 is a schematic diagram showing an implementation process of the method for processing the eye light image feature provided in an embodiment of the present application. Fig.18 As shown, when the electronic device has turned on the automatic correction function of the eye light, when the user triggers the photo, the electronic device will simultaneously obtain the original photo (low-quality portrait) and the high-quality portrait obtained through the face enhancement depth model, and the eye light of the high-quality portrait is abnormal. The electronic device performs iris cutout processing and eye light / background separation processing on the low-quality portrait to obtain the eye background and eye light of the low-quality portrait. At the same time, the electronic device performs iris cutout processing and eye light / background separation processing on the high-quality portrait to obtain the eye background and abnormal eye light of the high-quality portrait.
[0315] In this case, the eye background of the low-quality portrait is discarded, and the abnormal eye light of the high-quality portrait is discarded. The eye light of the low-quality portrait is adaptively enhanced to obtain enhanced eye light, and then the enhanced eye light is affine transformed and feathered, thereby achieving controllable enhancement of the eye light. Then, the processed enhanced eye light is image fused with the eye background of the high-quality portrait to obtain a high-quality portrait with normal eye light.
[0316] In this implementation, the portrait in the photo obtained through image processing has eye light, and the eye light is normal, thereby meeting the user's demand for the fidelity of the face area in scenes such as low-light shooting and telephoto shooting.
[0317] Fig.19 FIG. 2 is a schematic diagram showing another implementation process of the method for processing the eye light image feature provided in an embodiment of the present application. Fig.19As shown, when the electronic device has turned on the automatic correction function of the catch light, when the user triggers the photo, the electronic device will simultaneously obtain the original photo (low-quality portrait) and the high-quality portrait obtained through the face enhancement depth model, and the catch light of the high-quality portrait is abnormal. The electronic device performs iris cutout processing and catch light / background separation processing on the low-quality portrait to obtain the eye background of the low-quality portrait, and the original photo has no catch light. At the same time, the electronic device performs iris cutout processing and catch light / background separation processing on the high-quality portrait to obtain the eye background of the high-quality portrait and abnormal catch light.
[0318] In this case, the eye background of the low-quality portrait is discarded, and the abnormal eye light of the high-quality portrait is discarded. In the case where the original photo has no eye light, this application can directly remove the mistakenly added eye light features from the corresponding high-quality portrait based on the principle of high-fidelity enhancement. Since the eye light / background separation algorithm does not change the iris / sclera background of the human eye, the detailed information of the iris / sclera area of the high-quality portrait can be retained.
[0319] In this implementation, the high-quality portrait outputted at the end does not contain "abnormal" eye light, thus correcting the eye light and meeting the user's demand for the fidelity of the face area in scenes such as low-light shooting. In this case, the portrait obtained through image processing does not contain "abnormal" eye light, thus meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0320] Through the processing method of the eye light image feature provided by the embodiment of the present application, different eye light processing strategies can be adopted according to whether the original image has eye light: when the original image has eye light, the eye light of the original image is enhanced and the enhanced eye light replaces the "abnormal" eye light in the high-quality portrait, thereby correcting the "abnormal" eye light in the high-quality portrait; when the original image has no eye light, the "abnormal" eye light in the high-quality portrait is discarded to correct the "abnormal" eye light in the high-quality portrait. Therefore, the embodiment of the present application can correct the unnatural eye light defect caused by full face enhancement through the face enhancement depth model, thereby improving the user experience.
[0321] The above describes the overall process of the method for processing eye light image features provided in the embodiment of the present application. The following describes in detail the human eye iris cutout algorithm (also called human eye iris extraction algorithm) in the method for processing eye light image features provided in the embodiment of the present application.
[0322] Fig. 20 A schematic flowchart of human eye iris cutout in an embodiment of the present application is shown. Fig. 20 It includes the following steps S301-S309.
[0323] S301, input a face image.
[0324] In the embodiment of the present application, while performing iris cutout processing on the low-quality original portrait image, the iris cutout processing is also performed on the high-quality original portrait image. The high-quality portrait is a face image obtained by inputting the original image into the face enhancement deep model for enhancement.
[0325] S302, detecting the eye area.
[0326] For example, a face image to be processed is input into the first network model, and a mask of the eye area and key points of the eye center are output. The first network model may include functions such as face parsing and face key point detection. The mask is a binary image composed of 0 and 1.
[0327] For example, Fig.21 (a) in FIG. 1 shows a schematic diagram of a mask of an eye region, wherein a white region represents an eye region. For example, Fig.21 (b) in FIG. 1 shows a schematic diagram of the eye center key point, where point O represents the eye center key point.
[0328] S303: Determine the iris area.
[0329] First of all, it should be noted that the palpebral fissure refers to the gap formed between the upper and lower eyelids. The length of the palpebral fissure of an adult is about 20 mm to 30 mm. The corresponding relationship between the palpebral fissure length and the iris size is known. For example, Table 3 shows the relationship between the palpebral fissure length and the iris radius (denoted as R).
[0330] Table 3
[0331] Palpebral fissure length Iris size (radius R) 20mm 5.5mm 25mm 6mm 30mm 6.5mm
[0332] In the embodiment of the present application, the length of the palpebral fissure can be determined based on the mask of the eye area. Then, based on the palpebral fissure length and the relationship between the palpebral fissure length and the iris size, the iris radius range can be roughly estimated. Then, based on the key point of the eye center and the estimated iris radius range, the iris area can be determined.
[0333] For example, the length of the palpebral fissure can be determined to be 25 mm based on the mask of the eye area. By looking up Table 1, it can be determined that the iris radius corresponding to the palpebral fissure length is 6 mm. Then, a circle is drawn with the key point O at the center of the eye as the center and a radius of 6 mm to determine the iris area in the human eye. Fig.21 (c) in FIG. 1 shows a schematic diagram of the iris region of the eye, where R represents the radius of the estimated iris region.
[0334] S304: extracting feature points of the iris area.
[0335] For example, Fig. 22 A schematic flowchart of extracting feature points of an iris area in the present application is shown. Fig. 22 Includes the following steps S3041-S3048.
[0336] S3041, input iris area image.
[0337] For the sake of convenience, it is assumed that the iris area is N*N pixels.
[0338] S3042, enhance the contrast of the iris area.
[0339] In the embodiment of the present application, after the iris region is determined, the contrast of the iris region is enhanced to highlight the edge features, so as to facilitate the detection and extraction of feature points of the iris region.
[0340] S3043, transform to the polar coordinate system with (N / 2, N / 2) as the center.
[0341] It should be noted that since the iris edge region is a curve, it is difficult to extract feature points at the iris edge. Therefore, in the embodiment of the present application, the iris region is converted to a polar coordinate system with (N / 2, N / 2) as the center to obtain a 360*N / 2 polar coordinate image, thereby converting the curve feature into a straight line feature. 360 represents 360 degrees.
[0342] Assume that the iris region is N*N pixels. In the rectangular coordinate system, the center point of the iris region can be expressed as (x, y), x=N / 2, y=N / 2. Correspondingly, in the polar coordinate system, the coordinates of the center point of the iris region can be expressed as (ρ, θ), the radius coordinate ρ=0, the angle coordinate θ=0, that is, the pole (0, 0) of the polar coordinate system is the center point of the iris region.
[0343] The ρ coordinate represents the distance from the pole, and the θ coordinate represents the angle from the 0° ray (also called the polar axis) in the counterclockwise direction. The polar axis is the positive direction of the x-axis in the plane rectangular coordinate system. The angular coordinate θ is also called the polar angle or azimuth.
[0344] The value range of ρ is [0, N / 2] and the value range of θ is [0, 359].
[0345] S3044, extracting linear features through a linear filter.
[0346] In the embodiment of the present application, a linear filter is used in the feature point extraction process, so that the feature points of the iris area can be extracted more accurately.
[0347] S3045, input image features in the θ direction.
[0348] Among them, θ is 0, 1, ..., 359 respectively.
[0349] S3046, obtaining feature points in the θ direction.
[0350] Since the edge of the iris changes significantly, optionally, for each angular feature, feature point detection and extraction can be performed by peak detection. Specifically, peak detection can be performed on the eye region in the polar coordinate system to obtain feature points (ρ, θ) in each angular feature.
[0351] S3047, when it is judged that θ≤359, return to continue executing S3045-S3046 until θ reaches 359.
[0352] The following example illustrates the process of obtaining feature points in various directions in sequence.
[0353] When θ is 0, the feature points in the 0-angle direction are obtained based on the image features in the 0-angle direction.
[0354] Then, when θ is 1, the feature points in the 1-angle direction are obtained according to the image features in the 1-angle direction.
[0355] By analogy, when θ is 359, the feature points in the 359 angle direction are obtained according to the image features in the 359 angle direction.
[0356] Thus, feature points at various angles of 0, 1, ... 359 are obtained in the iris area.
[0357] S3048, output all feature points of the iris area.
[0358] Fig.23 The specific implementation process of extracting the feature points of the iris area in the above S3041-S3048 is shown. Specifically, for the iris area image with a pixel size of N*N, it is converted to a polar coordinate system with (N / 2, N / 2) as the center, and then the linear features are extracted through a linear filter. θ starts from 0 and until θ reaches 359, feature extraction is performed in each direction. Among them, the eye area in the polar coordinate system can be peak detected (the local maximum value in the edge feature point is taken) to obtain the edge feature points in each angular direction, and the edge feature points in all angular directions form a circle or ellipse. Among them, the edge feature points and the feature points surrounded by the edge feature points constitute all the feature points of the iris area.
[0359] It should be noted that among the feature points extracted from the iris region, some feature points are actually covered or blocked by the upper and lower eyelids, so these feature points in the iris region need to be removed. Therefore, it is necessary to perform the following step S305, that is, to filter the feature points in the iris region.
[0360] S305, screening feature points of the iris area.
[0361] For example, Fig.24 A schematic flow chart of screening feature points of an iris region in the present application is shown. Fig.24 The method includes steps S3051-S3055. After extracting all feature points in the iris area, feature points in the periphery and pupil area can be eliminated according to the roughly estimated iris radius range, feature points covered by the upper and lower eyelids in the iris area can be eliminated according to the mask of the eye area, and isolated points (outlier feature points) can be eliminated according to the connected length or area to achieve the screening of iris edge feature points.
[0362] The following describes the process of eliminating isolated points based on the connected length or area. First, edge detection is performed on the iris area, and the identification point rectangle is extracted. Rectangles with an area smaller than a certain threshold are determined to be noise points, and the pixel values in the noise point area are changed to background color pixel values.
[0363] It should be noted that the embodiment of the present application performs peak detection on the eye area in the polar coordinate system to obtain feature points, and introduces the results of face analysis and key point detection results to perform feature point screening, which can improve the accuracy of eye detection.
[0364] It should be noted that, for the sake of convenience, the screening of iris edge feature points in S305 is referred to as the first screening.
[0365] S306, obtaining an ellipse with the largest support by ellipse fitting, and generating a mask 1 of the iris region according to the ellipse.
[0366] Fig.25 A schematic flow chart of ellipse fitting of the present application is shown. Fig.25 Includes steps S3061-S3069.
[0367] Specifically, after the first screening of iris edge feature points is completed, the initial estimation of the ellipse is firstly realized by least square fitting for the first screened iris edge feature points, that is, the ellipse is initialized to obtain the initialized ellipse.
[0368] Then, the distance error is calculated based on the distance between each feature point of the iris edge and the initialization ellipse.
[0369] Then, the feature points are screened according to the size of the distance error. For example, if the distance error corresponding to feature point A in the iris area is greater than the error threshold (e.g., 1 pixel), feature point A is removed; if the distance error corresponding to feature point B in the iris area is less than or equal to the error threshold, feature point B is retained. The total number of feature points removed in this cycle calculation (referred to as the number of feature points removed) is recorded, and the number of feature points removed is recorded as N. Among them, the screening by distance error is called the second screening of iris edge feature points.
[0370] It should be noted that after continuous screening of the distance error, the number of feature points to be eliminated N is getting smaller and smaller. In order to improve the operation efficiency, a threshold value of elimination can be preset to determine whether the number of feature points to be eliminated in this cycle calculation meets the requirement. For example, the threshold value of elimination can be 2. The specific value of the threshold value of elimination can be set according to actual use requirements, and the embodiment of the present application is not limited thereto.
[0371] Then, determine whether the number of feature points to be eliminated N is less than or equal to the elimination threshold. If the change of some feature points on the iris edge is greater than the elimination threshold, fit the ellipse Z based on the feature points obtained by distance error screening, and calculate the support of all feature points on the iris edge (referred to as points) based on the distance from the feature point to the ellipse Z and the gradient of the feature point pixel:
[0372]
[0373] Among them, S represents the support of all feature points of the iris edge, p represents any feature point among the feature points screened by distance error, represents the gradient of the ellipse at the feature point p, Represents the gradient of the image at feature point p.
[0374] It should be noted that the calculation process of the gradient of the ellipse and the gradient of the image are different. The coordinate value corresponding to the feature point p is (x, y).
[0375] Ellipse Gradient The cone equation corresponding to the obtained fitting ellipse is obtained by deriving it:
[0376] ellipse(x,y)=Ax 2 +Bxy+Cy 2 +Dx+Ey+F
[0377] Among them, x and y represent the coordinate values of the X-axis and the Y-axis respectively, and A, B, C, D, E, and F are coefficients respectively. The changes of the X-axis and Y-axis are the derivatives of the cone equation about the X and Y directions respectively.
[0378] Different from the gradient of an ellipse, the pixel gradient changes as follows: It refers to the rate of change of a pixel (i.e., feature point p) in the image in the X and Y directions (compared with adjacent pixels). It is a two-dimensional vector consisting of two components: the change in the X axis and the change in the Y axis. (1) The change in the X axis refers to the pixel value to the right of the current pixel (X plus 1) minus the pixel value to the left of the current pixel (X minus 1). (2) The change in the Y axis is the pixel value below the current pixel (Y plus 1) minus the pixel value above the current pixel (Y minus 1). By calculating these two components to form a two-dimensional vector, the image gradient of the pixel point (i.e., feature point p) is obtained.
[0379] It should be noted that when the gradient directions of the feature point on the ellipse and the image are consistent, the contribution of the feature point will increase the weight, and the greater the gradient of the feature point on the image, the greater the weight.
[0380] The process of S3063-S3067 is repeated until the changes of all feature points are less than the preset change threshold, and the ellipse with the highest support is obtained.
[0381] The following is an example to illustrate the cycle process of S3063-S3067 and the process of ellipse fitting.
[0382] First cycle: Calculate the distance error between all feature points of the iris edge and the initialized ellipse. If the distance error of a feature point among all feature points of the iris edge is greater than the error threshold, then remove the feature point. In this way, by judging the distance error, the feature point set Y1 of the iris edge can be screened out from all feature points of the iris edge, and the number of feature points removed can be recorded. Then, determine whether the number of feature points removed from the feature point set A1 is less than the removal threshold. When the change of the feature point set Y1 is greater than or equal to the removal threshold, perform ellipse fitting based on the feature point set Y1 of the iris edge to obtain the ellipse Z1, and calculate the support of the feature point set Y1 of the iris edge based on the distance and gradient from the feature point set Y1 of the iris edge to the ellipse Z1 to obtain the support S1.
[0383] Second cycle: Calculate the distance error between the feature point set Y1 of the iris edge and the ellipse Z1. If the distance error of a feature point in the feature point set Y1 of the iris edge is greater than the error threshold, then the feature point is removed. In this way, the feature point set Y2 of the iris edge is screened out from the feature point set Y1 of the iris edge by judging the distance error, and the number of feature points removed is recorded. Then, it is determined whether the number of feature points removed from the feature point set Y2 is less than the removal threshold. When the change of the feature point set Y2 is greater than or equal to the removal threshold, an ellipse is fitted based on the feature point set Y2 of the iris edge to obtain the ellipse Z2, and the support of the feature point set Y2 of the iris edge is calculated based on the distance and gradient from the feature point set Y2 of the iris edge to the ellipse Z2 to obtain the support S2.
[0384] The third cycle: Calculate the distance error between the feature point set Y2 of the iris edge and the ellipse Z2. If the distance error of a feature point in the feature point set Y2 of the iris edge is greater than the error threshold, then remove the feature point. In this way, by judging the distance error, the feature point set Y3 of the iris edge is screened out from the feature point set Y2 of the iris edge, and the number of feature points removed is recorded. Then, it is determined whether the number of feature points removed from the feature point set Y2 is less than the removal threshold.
[0385] When the number of feature points eliminated from the feature point set Y3 is less than the elimination threshold, the support S1 and the support S2 are compared. If the support S2 is greater than the support S1, the ellipse Z2 corresponding to the support S2 is output.
[0386] In the embodiment of the present application, a mask 1 of the iris region is generated according to the ellipse with the highest support.
[0387] S307: finding the intersection of the mask 1 of the iris region and the mask of the eye region to obtain the mask 2 of the iris region.
[0388] S308, fine-tune the edge of the mask 2 of the iris area to obtain the mask 3 of the iris area.
[0389] Alternatively, if Fig.26 As shown, the mask of the iris region is intersected with the mask of the eye region obtained by face analysis to obtain the mask 1 of the iris region, and then the edge of the mask 2 of the iris region is fine-tuned to obtain the mask 3 of the iris region.
[0390] In the embodiment of the present application, the edge of the mask 2 of the iris region can be fine-tuned by using a graph cut algorithm to obtain a mask 3 of the iris region. Among them, image cutting refers to the technology and process of dividing an image into regions with different characteristics and extracting objects of interest. It is a key step from image processing to image analysis. The basic idea of image segmentation is that similar and close pixels should be divided into the same region.
[0391] It should be noted that the present application uses an image segmentation algorithm to fine-tune the extracted rough iris region, so that the final extracted iris region is more accurate.
[0392] S308, outputting mask 3 of the iris area.
[0393] Through the above steps, the human eye iris cutout processing is completed. After the human eye iris cutout processing, the obtained materials include: the human eye sclera area and iris area of the low-quality portrait, and the human eye sclera area and iris area of the high-quality portrait with abnormal eye light.
[0394] It should be noted that the present application performs ellipse fine-tuning through repeated iterations: all feature points are used to fit the ellipse, and the feature points are screened according to the distance from the feature points to the ellipse and the gradient of the feature points, so as to improve the accuracy of the ellipse fitting.
[0395] It should be noted that the processing method of the eye light image feature provided in the embodiments of the present application can be applied to user photo shooting scenarios. For example, the above embodiments are described by taking the automatic processing of the eye light of a portrait by an electronic device in a user photo shooting scenario as an example. Specifically, after the user triggers the photo shooting, the background of the electronic device identifies whether the original image meets certain conditions (such as the original image contains a face and the image quality is lower than a certain threshold), and if the original image meets certain conditions, the background automatically executes the processing method of the eye light image feature provided in the present application, so that the eye light of the portrait in the final generated photo is normal or does not contain "abnormal" eye light, and corrects the defect of unnatural eye light caused by full face enhancement through the face enhancement depth model.
[0396] It can be understood that in actual implementation, the application scenarios of the method for processing the eye light image feature provided by the embodiment of the present application include but are not limited to user photo shooting scenarios. For example, the method for processing the eye light image feature can also be applied to the gallery photo editing scenario. For example, the gallery application of the electronic device stores the original image and the high-quality portrait obtained by processing the face enhancement depth model. In the photo editing interface of the gallery of the electronic device, in response to the user operation, the electronic device retrieves the user-selected photo from the album or gallery and executes the method for processing the eye light image feature provided by the present application. The portrait eye light in the processed photo is normal or does not contain "abnormal" eye light, which corrects the unnatural defect of the eye light caused by the full face enhancement through the face enhancement depth model, thereby meeting the user's demand for the fidelity of the face area in shooting scenarios such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0397] The processing method of eye light image features provided in the embodiments of the present application can also be applied to eye light editing of portraits in mobile phone photo galleries, eye light replacement of specified images, replacement of specified eye light templates, eye light enhancement / weakening / removal, etc.
[0398] For example, in the photo editing interface of the mobile phone gallery, one or more eye light templates can be provided for eye light replacement; eye light enhancement / reduction controls, eye light replacement controls, and eye light removal controls can also be provided. Users can select one or more photos in the gallery according to their personal needs, and click different function controls to trigger the mobile phone to edit the eye light of the portrait in the photo, such as enhancing / reducing / replacing / removing the eye light, by clicking on different function controls.
[0399] It should be noted that the method for processing eye light image features provided in the embodiment of the present application can be used not only to correct the eye light of portraits in photos, but also to modify the eye light of animals in photos.
[0400] It should also be noted that in the embodiments of the present application, "greater than" can be replaced by "greater than or equal to", "less than or equal to" can be replaced by "less than", or "greater than or equal to" can be replaced by "greater than", and "less than" can be replaced by "less than or equal to".
[0401] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the protection scope of this application.
[0402] The above mainly describes the solution provided by the embodiment of the present application from the perspective of method steps. It is understandable that, in order to achieve the above functions, the electronic device implementing the method includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should be aware that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of the present application.
[0403] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other feasible division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.
[0404] Fig. 27 A schematic block diagram of a device 400 for processing catch light image features provided in an embodiment of the present application. The device 400 can be used to perform the actions performed by the electronic device in the above method embodiment. The device 400 includes an image acquisition unit 410, a highlight separation unit 420, an eye light enhancement unit 430, and an image fusion unit 430.
[0405] An image acquisition unit 410 is used to acquire a first image and a second image containing facial features when a preset condition is met, wherein the second image is an image obtained by enhancing the first image using a face enhancement depth model;
[0406] A highlight separation unit 420, configured to separate a first iris background region and a first highlight region from the iris region of the first image, and to separate a second iris background region and a second highlight region from the iris region of the second image by using a highlight separation algorithm;
[0407] The catch light enhancement unit 430 is used to determine a catch light contrast enhancement parameter according to the attribute parameters of the first highlight area; the attribute parameters include size, shape, average brightness and / or brightest point brightness, and the catch light contrast enhancement parameter includes a gamma value, and / or a partition block size and a contrast limit of a contrast-limited adaptive histogram equalization (CLAHE); the image features of the first highlight area are enhanced using the catch light contrast enhancement parameter to obtain the enhanced catch light feature;
[0408] The image fusion unit 440 is used to perform background fusion on the enhanced catch light feature and the image feature of the second iris background area to obtain a third image.
[0409] Through the processing device of the eye light image feature provided by the embodiment of the present application, for the scene where the eye light of a low-quality portrait is unnatural after being enhanced by the face enhancement depth model, the highlight area is first separated from the eye area of the low-quality portrait (first image), and then the eye light contrast enhancement parameter is determined according to the attribute parameters of the highlight area, and then the eye light is controllably enhanced according to the eye light contrast enhancement parameter. Then, the "abnormal" eye light feature in the eye area of the high-quality portrait (second image) is replaced with the enhanced eye light feature, so as to correct the eye light feature, thereby obtaining a high-quality portrait with "normal" eye light after enhancement, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0410] According to the embodiment of the present application, the device 400 may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of the units in the device 400 are respectively for implementing the corresponding processes of the method, which will not be repeated here for the sake of brevity.
[0411] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments.
[0412] The present application also provides an electronic device, which includes a chip, and the chip is used to read and execute a computer program or instruction stored in a memory, so that the method in each embodiment is executed.
[0413] This embodiment also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method for processing eye light image features in the above-mentioned embodiment.
[0414] This embodiment further provides a computer program product, wherein the computer-readable storage medium stores program code. When the computer program product is run on a computer, the computer executes the above-mentioned related steps to implement the method for processing eye light image features in the above-mentioned embodiment.
[0415] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the processing method of the eye light image features in the above-mentioned method embodiments.
[0416] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.
[0417] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0418] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for processing eye light image features, characterized in that: include: When the preset conditions are met, a first image and a second image containing facial features are obtained, where the second image is an image obtained by enhancing the first image using a face enhancement deep model; Using a highlight separation algorithm, separating a first iris background region and a first highlight region from the iris region of the first image, and separating a second iris background region and a second highlight region from the iris region of the second image; Determining a catch light contrast enhancement parameter according to the attribute parameters of the first highlight area; The attribute parameters include size, shape, average brightness and / or brightest point brightness, and the catch light contrast enhancement parameters include gamma value, and / or partition block size and contrast limit of contrast adaptive histogram equalization (CLAHE); Using the catch light contrast enhancement parameter, the image feature of the first highlight area is enhanced to obtain the enhanced catch light feature; The enhanced catch light feature is background-fused with the image feature of the second iris background area to obtain a third image.
2. The method according to claim 1, characterized in that The method of separating a first iris background region and a first highlight region from the iris region of the first image and separating a second iris background region and a second highlight region from the iris region of the second image by using a highlight separation algorithm comprises: The iris background area and highlight area are determined by the following equations: V s ′ pec =IV d ′ iff V d ′ iff =I-kβ s b s =I min -τ(p) Among them, V′ spec Represents the image component of the highlight area; V′ diff represents the image component of the iris background area, I represents the image of the iris area, and I min represents a single-channel image component obtained by taking the minimum value of the RGB three channels of each pixel point of the image I, τ(p) represents an offset related to the mean and variance of the image pixels, and k is a proportional coefficient; The iris background area is a first iris background area, and the highlight area is a first highlight area; or, the iris background area is a second iris background area, and the highlight area is a second highlight area.
3. The method according to claim 2, characterized in that The step of separating a first iris background area and a first highlight area from the iris area of the first image comprises: Normalizing the pixel values of the iris region of the first image to the interval [0,1]; The area in the iris area of the first image where the pixel value is greater than a threshold α is regarded as a possible highlight area, where α is a value in the interval [0,1]; By using a flood filling method, the first M maximum connected component regions are extracted from the possible highlight regions as the first highlight regions, where M is a positive integer.
4. The method according to claim 3, characterized in that The method further comprises: The k value is calculated using the least squares method according to the diffuse reflection color intensity of the first M largest connected component regions and their edge regions.
5. The method according to any one of claims 1 to 4, characterized in that Before obtaining the attribute parameters of the first highlight area, the method further includes: Determining whether the catch light feature exists in the iris region according to a set of attribute parameters of the first highlight region; wherein the set of attribute parameters includes at least two items of color moment, contrast, average brightness, saturation, and area; The step of obtaining the attribute parameters of the first highlight area includes: obtaining the attribute parameters of the first highlight area when the catch light feature exists in the iris area.
6. The method according to claim 1, characterized in that The attribute parameter includes the brightness of the brightest point; the eye light contrast enhancement parameter includes the gamma value; Determining the eye light contrast enhancement parameter according to the attribute parameter of the first highlight area includes: determining the ratio between the brightness of the brightest point in the first highlight area and 255 as the gamma value; The method of using the catch light contrast enhancement parameter to enhance the image features of the first highlight area to obtain enhanced catch light features includes: using the gamma value to perform gamma correction on the image features of the first highlight area to stretch the image contrast of the first highlight area.
7. The method according to claim 1, characterized in that The attribute parameters include the length and width of the minimum circumscribed rectangle of the first highlight area; the catch light contrast enhancement parameters include the block size and contrast limit of the CLAHE; the first highlight area includes one or more highlight areas; The block size is min(L / 2,3), and the contrast limit is max(D / 2,2); L represents the shortest side of the first highlight area, D represents the maximum value of the histogram in the circumscribed rectangle of the highlight area with the largest area in the first highlight area, min() represents the minimum value, and max() represents the maximum value; The method of using the catch light contrast enhancement parameter to enhance the image features of the first highlight area to obtain enhanced catch light features includes: using the block size and the contrast limit to redistribute pixels and reconstruct the grayscale values of pixels in the first highlight area to enhance the contrast of the first highlight area.
8. The method according to any one of claims 1 to 7, characterized in that The preset condition is: in response to a first operation of a shutter button by a user, the first image is captured, the first image includes facial features, and the image quality of the first image is lower than or equal to a preset image quality; or, The preset condition is: receiving a second operation of the user on the second image in the gallery, the second image includes facial features and eye light features, and the second operation is used to adjust the eye light features.
9. The method according to any one of claims 1 to 8, characterized in that The step of separating a first iris background area and a first highlight area from the iris area of the first image by using a highlight separation algorithm includes: Upsampling the eye region of the first image to obtain an enlarged eye region; Separating the first iris background area and the first highlight area from the enlarged iris area by using the highlight separation algorithm; Down-sampling the first iris background area and the first highlight area respectively; The reduction ratio of the downsampling is the inverse of the magnification ratio of the upsampling.
10. The method according to claim 9, characterized in that The step of upsampling the eye region of the first image to obtain an enlarged eye region includes: Performing anti-aliasing upsampling and filtering denoising on the eye region of the first image; When the enlarged size of the eye region of the first image is smaller than a preset size threshold, performing anti-aliasing upsampling and filtering denoising on the eye region of the first image again; When the enlarged size of the eye region of the first image is greater than or equal to the preset size threshold, the enlarged eye region is obtained.
11. The method according to any one of claims 1 to 10, characterized in that The determining, based on the attribute parameter set of the first highlight area, whether the catch light feature exists in the iris area includes: Obtaining a set of attribute parameters of the first highlight area; The attribute parameter set of the first highlight area is input into a preset binary classification model to obtain a judgment result, wherein the judgment result is that the eye light feature exists in the iris area or the eye light feature does not exist in the iris area.
12. The method according to claim 11, characterized in that The method further comprises: The attribute parameter set is used as a hyperparameter to train the preset binary classification model offline.
13. The method according to any one of claims 1 to 12, characterized in that The step of performing background fusion on the enhanced catch light feature and the image feature of the second iris background area to obtain a third image comprises: Determine an affine transformation matrix according to edge key points of the first iris background area and edge key points of the second iris background area; Performing an affine transformation on the enhanced catch light feature according to the affine transformation matrix to obtain an enhanced catch light feature aligned with the position of the second iris background area; Background fusion is performed on the aligned enhanced catch light features and the image features of the second iris background area to obtain the third image.
14. The method according to claim 13, characterized in that After obtaining the enhanced catch light feature aligned with the position of the second iris background region, the method further includes: performing adaptive feathering processing on the edge of the aligned enhanced catch light feature to obtain the feathered enhanced catch light feature; Among them, the step of performing background fusion on the aligned enhanced eye light features and the image features of the second iris background area to obtain the third image includes: performing background fusion on the feathered enhanced eye light features and the image features of the second iris background area to obtain the third image.
15. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory, wherein the processor is configured to execute the computer program so that the electronic device implements the method according to any one of claims 1 to 14.
16. A chip comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: The processor is configured to execute the computer program to implement the method according to any one of claims 1 to 14.
17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 14.
Citation Information
Patent Citations
Image processing method and device, electronic equipment and computer readable storage medium
CN107909058A
Image processing method, image processing device, and program
US20220398701A1
Cited By
Light effect control method, related device and computer storage medium
CN122597187A
