Iris feature extraction method, electronic equipment and storage medium
The iris feature extraction method iteratively refines ellipse fitting and uses image segmentation to improve iris feature extraction accuracy and natural eye light processing, addressing deformation issues in low-light and long-distance portraits.
Patent Information
- Application Number
- CN202310644364.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
Existing iris feature extraction methods struggle to accurately process eye light due to the small size and susceptibility of the iris to noise, especially in low-light and long-distance shooting conditions, leading to deformation and unnatural eye light enhancements.
An iris feature extraction method that involves identifying the iris area, extracting feature points in a polar coordinate system, screening points based on distance errors, iteratively fitting ellipses, and using image segmentation to enhance edge features, thereby improving accuracy and naturalness of eye light processing.
The method enhances the accuracy of iris feature extraction and naturalness of eye light, addressing the unnatural enhancements caused by AI depth models in low-light and long-distance portraits, ensuring fidelity in image quality.
Smart Images

Figure CN119067861A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an iris feature extraction method, electronic equipment and storage medium. Background Art
[0002] In the correction technology for abnormal eye light in an image, since the eye light is usually displayed in the iris area, it is usually necessary to extract the iris features and then correct the eye light in the area where the iris features are located.
[0003] However, since the iris is very small and is easily disturbed by noise such as light spots, eyelids, and eyelashes, and in shooting scenes such as low light and long distances, the iris image is easily deformed due to the different user shapes. Therefore, how to extract iris features is the key and difficulty of eye light processing. Summary of the invention
[0004] The present application provides an iris feature extraction method, an electronic device, and a storage medium, which can accurately extract iris image features and help to more accurately process eye light.
[0005] In a first aspect, the present application provides an iris feature extraction method, the method comprising:
[0006] When a preset condition is met, a first image containing facial features and a second image are obtained, wherein the first image contains facial features, and the second image is an image obtained by applying a facial enhancement depth model to the first image;
[0007] Perform feature recognition on the image to be processed to determine the iris area; the image to be processed is the first image or the second image;
[0008] Extracting feature points at the edge of the iris region in a polar coordinate system;
[0009] Calculating the distance error between each feature point of the G feature points and the first ellipse; wherein the G feature points are feature points extracted from the edge of the iris region for the first time, and the first ellipse is an ellipse fitted based on the feature points extracted from the edge of the iris region for the first time; or the G feature points are feature points remaining after being eliminated last time, and the first ellipse is an ellipse fitted based on the feature points remaining after being eliminated last time;
[0010] Eliminate K feature points from the G feature points, and the distance error between each feature point in the K feature points and the first ellipse is greater than a preset error threshold;
[0011] When K is greater than a preset rejection threshold, a second ellipse is obtained by fitting according to the GK feature points, and the support corresponding to the second ellipse is calculated; wherein the GK feature points and the second ellipse are used together for the next calculation of the distance error between the feature points and the ellipse;
[0012] Circularly calculating the distance error between the feature point and the ellipse and eliminating the feature point according to the distance error, fitting to obtain multiple ellipses, and calculating the support corresponding to each of the multiple ellipses;
[0013] When the number of removed feature points is less than or equal to the preset removal threshold, obtaining a target ellipse with the largest support among the multiple ellipses;
[0014] The image features corresponding to the target ellipse are extracted from the image to be processed as the image features of the iris area.
[0015] According to the iris feature extraction method provided by the embodiment of the present application, after the iris area in the image is identified, the feature points at the edge of the iris area are extracted in the polar coordinate system, the feature points are first screened according to the result of face analysis and then the ellipse is fitted, and then some feature points are eliminated according to the distance error between the feature points and the ellipse; then the ellipse is fitted again according to the feature points remaining after the elimination and the corresponding support is calculated. Then the distance error is calculated again according to the feature points remaining after the elimination and the fitted ellipse. The distance error is calculated cyclically and the feature points are eliminated according to the distance error, thereby fitting multiple ellipses and calculating multiple support. When the number of eliminated feature points is less than or equal to the threshold, the image features of the iris area are extracted according to the ellipse with the largest support. The present application performs ellipse fine-tuning by repeated iterations: all feature points are used to fit the ellipse, and the feature points are screened according to the distance from the feature points to the ellipse, so as to improve the accuracy of ellipse fitting and improve the accuracy of iris extraction.
[0016] It should be noted that, when the first ellipse is an ellipse fitted according to feature points extracted from the edge of the iris region for the first time, the first ellipse may be referred to as an initialization ellipse.
[0017] It should be noted that after continuous screening of the distance error, the number of feature points to be eliminated N is getting smaller and smaller. In order to improve the operation efficiency, a threshold value of elimination can be preset to determine whether the number of feature points to be eliminated in this cycle calculation meets the requirement. For example, the threshold value of elimination can be 2. The specific value of the threshold value of elimination can be set according to actual use requirements, and the embodiment of the present application is not limited thereto.
[0018] The following is an exemplary description of the cyclic calculation process.
[0019] First cycle: Calculate the distance error between all feature points of the iris edge and the initialized ellipse. If the distance error of a feature point among all feature points of the iris edge is greater than the error threshold, then remove the feature point. In this way, by judging the distance error, the feature point set Y1 of the iris edge can be screened out from all feature points of the iris edge, and the number of feature points removed can be recorded. Then, determine whether the number of feature points removed from the feature point set A1 is less than or equal to the removal threshold. When the change of the feature point set Y1 is greater than the removal threshold, perform ellipse fitting based on the feature point set Y1 of the iris edge to obtain the ellipse Z1, and calculate the support of the feature point set Y1 of the iris edge based on the distance and gradient from the feature point set Y1 of the iris edge to the ellipse Z1 to obtain the support S1.
[0020] Second cycle: Calculate the distance error between the feature point set Y1 of the iris edge and the ellipse Z1. If the distance error of a feature point in the feature point set Y1 of the iris edge is greater than the error threshold, then the feature point is removed. In this way, the feature point set Y2 of the iris edge is screened out from the feature point set Y1 of the iris edge by judging the distance error, and the number of feature points removed is recorded. Then, it is determined whether the number of feature points removed from the feature point set Y2 is less than or equal to the removal threshold. When the change of the feature point set Y2 is greater than the removal threshold, an ellipse is fitted based on the feature point set Y2 of the iris edge to obtain the ellipse Z2, and the support of the feature point set Y2 of the iris edge is calculated based on the distance and gradient from the feature point set Y2 of the iris edge to the ellipse Z2 to obtain the support S2.
[0021] The third cycle: Calculate the distance error between the feature point set Y2 of the iris edge and the ellipse Z2. If the distance error of a feature point in the feature point set Y2 of the iris edge is greater than the error threshold, then remove the feature point. In this way, by judging the distance error, the feature point set Y3 of the iris edge is screened out from the feature point set Y2 of the iris edge, and the number of feature points removed is recorded. Then, it is determined whether the number of feature points removed from the feature point set Y2 is less than or equal to the removal threshold.
[0022] When the number of feature points to be eliminated in the feature point set Y3 is less than or equal to the elimination threshold, the support S1 and the support S2 are compared. If the support S2 is greater than the support S1, the ellipse Z2 corresponding to the support S2 is output.
[0023] It should be noted that the present application performs ellipse fine-tuning through repeated iterations: all feature points are used to fit the ellipse, and the feature points are screened according to the distance from the feature points to the ellipse and the gradient of the feature points, so as to improve the accuracy of the ellipse fitting.
[0024] Through the solution of the present application, multiple cycles are performed to continuously optimize the ellipse fitting, so that the iris image features can be accurately extracted.
[0025] In some possible implementations, the preset condition is: in response to a first operation of a shutter button by a user, the first image is captured, the first image includes facial features, and an image quality of the first image is lower than or equal to a preset image quality.
[0026] In some possible implementations, the preset condition is: receiving a second operation of the user on the second image in the gallery, the second image including facial features and eye light features, and the second operation is used to adjust the eye light features.
[0027] In some possible implementations, the calculating the support corresponding to the second ellipse includes: calculating the support corresponding to the second ellipse based on the gradient of the second ellipse at the GK feature points and the gradient of the image to be processed at the GK feature points.
[0028] In some possible implementations, calculating the support corresponding to the second ellipse according to the gradient of the second ellipse at the GK feature points and the gradient of the image to be processed at the GK feature points includes:
[0029] The support corresponding to the second ellipse is calculated using the following equation:
[0030]
[0031] Where S represents the support corresponding to the second ellipse, represents the gradient of the second ellipse at the feature point p, represents the gradient of the image to be processed at the feature point p, where p is each feature point among the GK feature points.
[0032] It should be noted that when the gradient direction of the feature point on the ellipse and the image is consistent, the contribution of the feature point will increase the weight, and the greater the gradient of the feature point on the image, the greater the weight, and accordingly, the greater the support, and the better the ellipse fitting effect.
[0033] In some possible implementations, after acquiring the target ellipse with the largest support among the multiple ellipses and before extracting the image features corresponding to the target ellipse from the image to be processed as the image features of the iris region, the method further includes:
[0034] A first mask of the iris region is generated according to the target ellipse; an intersection between the first mask and the mask of the eye region is obtained to obtain a second mask of the iris region; and an image segmentation algorithm is used to fine-tune the edge of the second mask of the iris region to obtain a third mask of the iris region.
[0035] Among them, the image segmentation (Graph Cut) algorithm refers to the technology and process of dividing an image into regions with different characteristics and extracting the target of interest. It is a key step from image processing to image analysis. The basic idea of image segmentation is that similar and close pixels should be divided into the same region.
[0036] Through the solution of the present application, the extracted rough iris region is fine-tuned using an image segmentation algorithm, so that the iris region finally extracted is more accurate.
[0037] The step of extracting image features corresponding to the target ellipse from the image to be processed as image features of the iris region includes: extracting image features from the image to be processed according to a third mask of the iris region as image features of the iris region.
[0038] In some possible implementations, after extracting the feature points at the edge of the iris area in the polar coordinate system, the method further includes: removing the feature points covered by the upper and lower eyelids from the feature points at the edge of the iris area based on a mask of the eye area; and removing the feature points located outside the iris from the feature points at the edge of the iris area based on a roughly estimated iris radius length.
[0039] In some possible implementations, the step of performing feature recognition on the image to be processed and determining the iris area includes:
[0040] Input the image to be processed into a first preset network model, and output a mask of the eye area and key points of the eye center; wherein the first preset network model is a network model for analyzing faces and detecting key points of faces;
[0041] determining a first palpebral fissure length based on the mask of the eye region;
[0042] Determine the iris radius length according to the first palpebral fissure length and a preset first corresponding relationship; wherein the first corresponding relationship includes a corresponding relationship between the first palpebral fissure length and the iris radius length;
[0043] The iris region is determined in the image to be processed according to the iris radius length and the eye center key point; wherein the iris region is a circular region with the eye center key point as the center and the iris radius length as the radius.
[0044] In some possible implementations, after performing feature recognition on the image to be processed and determining the iris region, the method further includes: enhancing the contrast of the iris region to highlight edge features of the iris region.
[0045] Through the solution of the present application, after the iris area is determined, the contrast of the iris area is enhanced to highlight the edge features, thereby facilitating the detection and extraction of feature points of the iris area.
[0046] In some possible implementations, after performing feature recognition on the processed image to determine the iris region, and before extracting feature points at the edge of the iris region in the polar coordinate system, the method further includes: converting the iris region from a rectangular coordinate system to the polar coordinate system. In the rectangular coordinate system, the coordinates of the pixel points in the iris region are expressed as (x, y), and the value ranges of x and y are both [0, N]; in the polar coordinate system, the coordinates of the pixel points in the iris region are expressed as (ρ, θ), and the value range of ρ is [0, N / 2], and the value range of θ is [0, 360).
[0047] It should be noted that since the iris edge region is a curve, it is difficult to extract feature points at the iris edge. Therefore, in the embodiment of the present application, the iris region is converted to a polar coordinate system with (N / 2, N / 2) as the center to obtain a 360*N / 2 polar coordinate image, thereby converting the curve feature into a straight line feature. 360 represents 360 degrees.
[0048] Assume that the iris region is N*N pixels. In the rectangular coordinate system, the center point of the iris region can be expressed as (x, y), x=N / 2, y=N / 2. Correspondingly, in the polar coordinate system, the coordinates of the center point of the iris region can be expressed as (ρ, θ), the radius coordinate ρ=0, the angle coordinate θ=0, that is, the pole (0, 0) of the polar coordinate system is the center point of the iris region.
[0049] In some possible implementations, extracting the feature points at the edge of the iris area in a polar coordinate system includes: extracting linear features of the image to be processed by a linear filter in the polar coordinate system; performing peak detection on the linear features of the image to be processed to obtain each feature point with a polar radius ρ of N / 2 and a polar angle θ in the range of [0, 360), thereby obtaining the feature points at the edge of the iris area.
[0050] In the embodiment of the present application, a linear filter is used in the feature point extraction process, so that the feature points of the iris area can be extracted more accurately.
[0051] The following example illustrates the process of obtaining feature points in various directions in sequence.
[0052] When θ is 0, the feature points in the 0-angle direction are obtained according to the image features in the 0-angle direction. Then, when θ is 1, the feature points in the 1-angle direction are obtained according to the image features in the 1-angle direction. By analogy, when θ is 359, the feature points in the 359-angle direction are obtained according to the image features in the 359-angle direction. Since the edge changes of the iris edge are more significant, optionally, for the features in each angular direction, feature point detection and extraction can be performed by peak detection. Specifically, peak detection can be performed on the eye area in the polar coordinate system to obtain feature points (ρ, θ) in each angular direction.
[0053] In some possible implementations, after extracting the image features corresponding to the target ellipse from the image to be processed as the image features of the iris region, the method further includes:
[0054] Separating a first background region and a first highlight region from the eye region of the first image, and separating a second background region and a second highlight region from the eye region of the second image;
[0055] In the case where there is a catch light feature in the eye area, enhancing the image feature of the first highlight area to obtain an enhanced catch light feature; and background fusion of the enhanced catch light feature with the image feature of the second background area to obtain a third image;
[0056] When the catch light feature does not exist in the eye region, a fourth image is obtained according to the image feature of the second background region.
[0057] In some possible implementations, separating the first background region and the first highlight region from the eye region of the first image includes: upsampling the eye region of the first image to obtain an enlarged eye region; separating the first background region and the first highlight region from the enlarged eye region based on a highlight removal algorithm; and downsampling the first background region and the first highlight region, respectively. The reduction ratio of the downsampling is the inverse of the enlargement ratio of the upsampling.
[0058] In some possible implementations, upsampling the eye region of the first image to obtain an enlarged eye region includes: performing anti-aliasing upsampling and filtering denoising on the eye region of the first image; when the size of the enlarged eye region of the first image is smaller than a preset size threshold, performing anti-aliasing upsampling and filtering denoising on the eye region of the first image again; and when the size of the enlarged eye region of the first image is greater than or equal to the preset size threshold, obtaining the enlarged eye region.
[0059] In some possible implementations, the step of enhancing the image features of the first highlight area to obtain enhanced catch light features includes:
[0060] Acquire attribute parameters of the first highlight area, where the attribute parameters include size, shape, average brightness and / or brightest point brightness;
[0061] determining a first enhancement parameter according to the attribute parameter of the first highlight area;
[0062] The first enhancement parameter is used to enhance the image feature of the first highlight area to obtain an enhanced eye light feature.
[0063] In some possible implementations, performing background fusion on the enhanced catch light feature and the image feature of the second background area to obtain a third image includes:
[0064] An affine transformation matrix is determined according to edge key points of the first background area and edge key points of the second background area; an affine transformation is performed on the enhanced eye light feature according to the affine transformation matrix to obtain an enhanced eye light feature aligned with the position of the second background area; and background fusion is performed on the aligned enhanced eye light feature and the image feature of the second background area to obtain the third image.
[0065] It should be noted that the iris feature extraction method provided in the embodiments of the present application can be applied to user photo-taking scenarios. For example, the above embodiments are described by taking the automatic processing of portrait eye light by an electronic device in a user photo-taking scenario as an example. Specifically, after the user triggers the photo-taking, the background of the electronic device identifies whether the original image meets certain conditions (such as the original image contains a face and the image quality is lower than a certain threshold), and if the original image meets certain conditions, the background automatically executes the iris feature extraction method provided in the present application, so that the portrait eye light in the final generated photo is normal or does not contain "abnormal" eye light, correcting the defect of unnatural eye light caused by full face enhancement through the face enhancement depth model.
[0066] In actual implementation, the application scenarios of the iris feature extraction method provided by the embodiment of the present application include but are not limited to user photo-taking scenarios. For example, the iris feature extraction method can also be applied to gallery photo editing scenarios. For example, the gallery application of the electronic device stores the original image and the high-quality portrait obtained by processing the face enhancement depth model. In the photo editing interface of the gallery of the electronic device, in response to the user operation, the electronic device retrieves the user-selected photo from the album or gallery and executes the iris feature extraction method provided by the present application. The portrait eye light in the processed photo is normal or does not contain "abnormal" eye light, which corrects the unnatural defect of eye light caused by full face enhancement through the face enhancement depth model, thereby meeting the user's demand for the fidelity of the face area in shooting scenarios such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0067] It should be noted that the algorithm scenarios involved in this application are mainly scenarios where mobile phones take low-quality images, including but not limited to low-light and telephoto scenes, and may also be any other scenarios that may take low-quality images.
[0068] In a second aspect, the present application provides an iris feature extraction device, the device comprising a unit for executing the method in the first aspect. The device may correspond to executing the method described in the first aspect, and the relevant description of the unit in the device can refer to the description of the first aspect, which will not be repeated here for the sake of brevity.
[0069] The method described in the first aspect above can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.
[0070] In a third aspect, the present application provides an electronic device, comprising a processor, a computer program or instructions stored in the processor and a memory, wherein the processor is configured to execute the computer program or instructions so that the method in the first aspect is executed.
[0071] In a fourth aspect, the present application provides a computer-readable storage medium on which a computer program (also referred to as instructions or codes) for implementing the method in the first aspect is stored. For example, when the computer program is executed by a computer, the computer can execute the method in the first aspect.
[0072] In a fifth aspect, the present application provides a chip, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof. Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.
[0073] In a sixth aspect, the present application provides a chip system, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof. Optionally, the chip system also includes a memory, and the memory is connected to the processor via a circuit or wire.
[0074] In a seventh aspect, the present application provides a computer program product, which includes a computer program (also referred to as instructions or codes). When the computer program is executed by an electronic device, the electronic device implements the method in the first aspect.
[0075] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0076] Figure 1 A schematic diagram of the sclera and iris in the eye region involved in an embodiment of the present application;
[0077] Figure 2 An illustration of the distorted eye light of an image taken by a user in the related art after being processed by a face enhancement depth model Figure 1 ;
[0078] Figure 3 An illustration of the distorted eye light of an image taken by a user in the related art after being processed by a face enhancement depth model Figure 2 ;
[0079] Figure 4 Schematic diagram of the process of iris feature extraction method provided in the embodiment of the present application Figure 1 ;
[0080] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0081] Figure 6 A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;
[0082] Figure 7 Schematic diagram of the process of iris feature extraction method provided in the embodiment of the present application Figure 2 ;
[0083] Figure 8 Schematic diagram of the process of iris feature extraction method provided in the embodiment of the present application Figure 3 ;
[0084] Fig. 9 A schematic diagram of a human eye iris cutout in an iris feature extraction method provided in an embodiment of the present application;
[0085] Fig.10 A schematic diagram of catch light / background separation in the iris feature extraction method provided in an embodiment of the present application;
[0086] Fig.11 A schematic diagram of image upsampling in the iris feature extraction method provided in an embodiment of the present application;
[0087] Fig.12 A schematic diagram of extracting eye highlight areas in the iris feature extraction method provided in an embodiment of the present application;
[0088] Fig.13 A schematic diagram of upsampling, catch light / background separation, and downsampling in the iris feature extraction method provided in an embodiment of the present application;
[0089] Fig.14 A schematic diagram of the fusion of eye light and background in the iris feature extraction method provided in an embodiment of the present application;
[0090] Fig.15 A schematic diagram of three forms of scaling, rotation and translation of affine transformation in the iris feature extraction method provided in an embodiment of the present application;
[0091] Fig.16 A schematic diagram of the iris feature extraction method provided in an embodiment of the present application using affine transformation to enhance the eye light feature;
[0092] Fig.17 A schematic diagram of feathering processing in the iris feature extraction method provided in an embodiment of the present application;
[0093] Fig.18 Schematic diagram of the implementation process of the iris feature extraction method provided in the embodiment of the present application Figure 1 ;
[0094] Fig.19 Schematic diagram of the implementation process of the iris feature extraction method provided in the embodiment of the present application Figure 2 ;
[0095] Fig. 20 A schematic diagram of the iris cutout process in the iris feature extraction method provided in an embodiment of the present application;
[0096] Fig.21 A schematic diagram of a mask of an eye region, a key point of an eye center, and an iris region in an iris cutout process of an iris feature extraction method provided in an embodiment of the present application;
[0097] Fig. 22 Schematic diagram of the process of extracting feature points of the iris area in the iris feature extraction method provided in the embodiment of the present application Figure 1 ;
[0098] Fig.23Schematic diagram of the process of extracting feature points of the iris area in the iris feature extraction method provided in the embodiment of the present application Figure 2 ;
[0099] Fig.24 A schematic diagram of a process for selecting feature points of an iris area in an iris feature extraction method provided in an embodiment of the present application;
[0100] Fig.25 A schematic diagram of the process of ellipse fitting in the iris feature extraction method provided in an embodiment of the present application;
[0101] Fig.26 A schematic diagram of obtaining a mask of an iris region in the iris feature extraction method provided in an embodiment of the present application;
[0102] Fig. 27 A schematic diagram of the structure of an iris feature extraction device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0103] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0104] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.
[0105] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of objects. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two, for example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.
[0106] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0107] The catch light is used to create a bright spot of light in the eyes in portrait photography. The catch light usually creates a small round spot of light in the black pupil, which can make the eyes look lively. Figure 1 As shown, eye features may include sclera and iris, and the iris area is the black pupil. Among them, the catch light is usually displayed in the iris area.
[0108] The current portrait imaging effect of mobile phones in low light and long distance conditions is significantly lower than that in good light and close-up conditions. In order to improve the quality of portrait images taken in low light and long distance conditions, the mainstream method in the industry is usually to use an artificial intelligence (AI) depth model to enhance the face image on the original image after the user triggers the photo. The enhanced photo is clearer and has richer face details. For ease of explanation, the AI depth model for enhancing face images is referred to as the face enhancement depth model.
[0109] However, due to the low interpretability and controllability of the deep model for face image enhancement, the eye light features are often misprocessed after enhancement, and the processed eye light does not look natural enough, which does not meet the basic needs of users for the authenticity of the camera function. Eyes and eye light are one of the most prominent features of the face area. Changes in eye light are easily noticed by users. If the eye light enhancement effect is not realistic, it will greatly affect the overall effect of the face.
[0110] For example, if there is eye light in the original image, the face enhancement depth model will enhance the eye light, but the enhanced eye light does not look natural enough. Figure 2 (a) in the figure shows a low-quality portrait (original image) taken by a mobile phone under conditions such as low light and long distance. The original image has poor imaging effect, and there is catch light in the eye area. After the low-quality portrait is input into the face enhancement deep model, the face enhancement deep model performs face enhancement on the low-quality portrait and outputs a high-quality portrait. Figure 2 (b) in the figure shows a high-quality portrait with face enhancement completed by a face enhancement deep model, in which the eye light in the eye area is enhanced, but the shape, size, intensity, etc. of the eye light are modified, so the enhanced eye light is unnatural and does not meet the user's basic demand for the authenticity of the photo-taking function.
[0111] For another example, when the original image does not have eye light, the face enhancement depth model will add eye light in the eye area, but the added eye light does not look natural enough. Figure 3(a) in the figure shows a low-quality portrait (original image) taken by a mobile phone under conditions of dim light and long distance, where there is no catch light in the eye area. After the low-quality portrait is input into the face enhancement deep model, the face enhancement deep model performs face enhancement on the low-quality portrait and outputs a high-quality portrait. Figure 3 (b) in the figure shows a high-quality portrait with face enhancement completed by the face enhancement deep model. Compared with the original image, eye light is added to the eye area. However, since there was no eye light originally, adding eye light will modify the original expression of the face. Therefore, adding eye light will appear unnatural and fail to meet the user's basic demand for the authenticity of the camera function.
[0112] In the above-mentioned correction technology for abnormal eye light in an image, since the eye light is usually displayed in the iris area, it is usually necessary to extract the iris features and then correct the eye light in the area where the iris features are located.
[0113] However, since the iris is very small and is easily disturbed by noise such as light spots, eyelids, and eyelashes, and in shooting scenes such as low light and long distances, the iris image is easily deformed due to the different user shapes. Therefore, how to extract iris features is the key and difficulty of eye light processing.
[0114] In view of the above technical problems, the embodiments of the present application provide an iris feature extraction method and electronic device, which can be implemented based on a computer vision algorithm, and can controllably enhance the natural enhancement of the eye light by using the eye and eye light information in a low-quality portrait. In the embodiments of the present application, through improvements in the underlying system of the mobile phone, the eye light of a portrait in a low-quality image can be controllably enhanced without any additional hardware, so as to improve the user experience.
[0115] Through the solution of this application, after the iris area in the image is identified, the feature points at the edge of the iris area are extracted in the polar coordinate system, the feature points are first screened according to the results of face analysis, and then the ellipse is fitted, and then some feature points are eliminated according to the distance error between the feature points and the ellipse; then the ellipse is fitted again according to the remaining feature points after elimination and the corresponding support is calculated. This application screens the feature points according to the distance error between the feature points and the ellipse, and iterates repeatedly to fine-tune the ellipse to improve the accuracy of the ellipse fitting, thereby improving the accuracy of iris extraction and facilitating accurate processing of eye light.
[0116] In order to better understand the embodiments of the present application, Figure 4 The embodiments of the present application are briefly described as follows:
[0117] Figure 4 The simplified algorithm flow of the present application is shown. Figure 4 As shown, in the embodiment of the present application, the eye light is controllably enhanced by performing image processing on low-quality portraits and high-quality portraits respectively.
[0118] The low-quality portrait may be a photo containing facial features taken by a user after pressing the shutter button of the camera of an electronic device under conditions such as dim light and long distance, or may be a photo containing facial features stored in an album or gallery of the electronic device. After the electronic device performs face enhancement on the low-quality portrait using a preset face enhancement depth model, a high-quality portrait is obtained. In a high-quality portrait, since the eye light after enhancement is unnatural, the image may be referred to as a high-quality portrait with "abnormal" eye light after enhancement.
[0119] In order to controllably enhance the eye light, first, based on the low-quality eye light features in the low-quality portrait, the embodiment of the present application adopts the eye light enhancement algorithm provided by the present application to achieve eye light enhancement and obtain enhanced eye light features. Then, the enhanced eye light features are used to replace the "abnormal" eye light features in the high-quality portrait, and the eye light features are corrected, so that a high-quality portrait with "normal" eye light after enhancement can be obtained. This satisfies the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0120] It should be noted that in the embodiments of the present application, "normal" eye light refers to the distortion effect of "abnormal" eye light. For example, if the eye light in a portrait is real and natural, it can be considered as "normal" eye light; or if there is no "abnormal" eye light in a portrait, it can also be considered as "normal" eye light.
[0121] The embodiments of the present application focus on the user's demand for the fidelity of the facial area in scenarios such as low-light shooting and telephoto shooting, and can controllably enhance the human eye's catch light to compensate for the unnatural defect of the eye light caused by full-face enhancement through the AI depth model.
[0122] Combination of the above Figure 4 The present invention describes a scheme for correcting "abnormal" eye light in a high-quality portrait when the original image has eye light: based on the low-quality portrait, it is determined that the original image has eye light, and then the eye light features are controllably enhanced based on the low-quality portrait. The enhanced eye light features are then used to replace the "abnormal" eye light features in the high-quality portrait, thereby correcting the "abnormal" eye light in the high-quality portrait to "normal" eye light, thereby satisfying the user's demand for the fidelity of the face area in shooting scenarios such as low-light shooting and telephoto shooting where low-quality portraits need to be enhanced by algorithms.
[0123] In addition, the embodiment of the present application also provides a solution for correcting the "abnormal" eye light in the high-quality portrait when the original image has no eye light: based on the low-quality portrait, it is judged that the original image has no eye light, and based on the high-quality portrait, the high-quality eye light feature and the high-quality human eye background are separated. Since the high-quality eye light feature appears as "abnormal" eye light, the high-quality eye light feature can be deleted and the high-quality human eye background can be retained, thereby correcting the "abnormal" eye light in the high-quality portrait, so that the high-quality portrait does not contain "abnormal" eye light, thereby meeting the user's demand for the fidelity of the face area in shooting scenarios such as low-light shooting and telephoto shooting where low-quality portraits need to be enhanced by algorithms.
[0124] The iris feature extraction method provided in the embodiment of the present application can be applied to electronic devices with a shooting function. Figure 5 , is a structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, an ambient light sensor 180L, and the like.
[0125] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0126] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. For example, the processor 110 is used to execute the iris feature extraction method in the embodiment of the present application.
[0127] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0128] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0129] The external memory 120 generally refers to an external memory. In the embodiment of the present application, the external memory refers to a memory other than the memory of the electronic device and the cache of the processor, and the memory is generally a non-volatile memory.
[0130] The internal memory 121, which may also be referred to as "memory", may be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), and the like.
[0131] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel may be an organic light-emitting diode (OLED). In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0132] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[0133] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0134] The electronic device 100 also includes various sensors that can convert various physical signals into electrical signals.
[0135] The above is a specific description of the embodiment of the present application using the electronic device 100 as an example. It should be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.
[0136] The electronic device provided in the embodiments of the present application may be a user equipment (UE), for example, a mobile terminal (such as a user's mobile phone), a tablet computer, a desktop computer, a laptop computer, a handheld computer, a netbook, a personal digital assistant (PDA), and the like.
[0137] In addition, an operating system is running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on the operating system.
[0138] The operating system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.
[0139] Figure 6 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.
[0140] The layered architecture divides the system into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: application layer (applications), application framework layer (application framework), hardware abstraction layer (HAL), and kernel layer (kernel).
[0141] The application layer may include a series of application packages. In the embodiment of the present application, the application package may include a camera, a gallery, etc.
[0142] The application framework layer provides an application programming interface (API) and a programming framework for the application programs of the application layer. The application framework layer includes some predefined functions. In an embodiment of the present application, the application framework layer may include a camera access interface, wherein the camera access interface may include a human eye iris cutout service, a catch light / background separation service, a catch light enhancement service, and a catch light fusion service. The camera access interface is used to provide an application programming interface and a programming framework for the camera application.
[0143] The hardware abstraction layer is an interface layer located between the application framework layer and the kernel layer, and provides a virtual hardware platform for the operating system. In the embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.
[0144] The camera hardware abstraction layer can provide virtual hardware (referred to as camera HAL) for camera devices such as the front camera and the rear camera. The camera algorithm library can include a human eye iris cutout algorithm, a catch light / background separation algorithm, a catch light enhancement algorithm, and a catch light fusion algorithm. In other words, the camera algorithm library contains the operating code and data for implementing the iris feature extraction method provided in the embodiment of the present application.
[0145] The kernel layer is a layer between hardware and software. The kernel layer includes drivers for various hardware. The kernel layer may include camera device drivers, digital signal processor drivers, and image processor drivers. Among them, the camera device driver is used to drive the camera sensor to capture images and drive the image signal processor to pre-process the image. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.
[0146] In addition, for the sake of convenience, Figure 6 The hardware layers that interact with the software layers are also shown. For example, the hardware layer includes the camera sensor, image signal processor, data signal processor, and image processor.
[0147] The following is an overview of the iris feature extraction method in the embodiment of the present application in combination with the above hardware structure and system structure:
[0148] (1) Image acquisition stage:
[0149] like Figure 7 As shown, the electronic device captures a first image (original image), and uses a face enhancement depth model to perform face enhancement processing on the first image to obtain a second image.
[0150] First, in response to the user's operation on the camera application icon (such as a click operation), the camera application calls the camera access interface of the application framework layer to start the camera application, and then sends an instruction to start the camera application by calling the camera device (such as the rear camera) in the camera hardware abstraction layer. The camera hardware abstraction layer sends the instruction to the camera device driver of the kernel layer. The camera device driver can start the sensor of the rear camera, that is, the electronic device starts the shooting function.
[0151] Then, in response to the user pressing the camera shutter button, the first image signal is collected by the rear camera, and the first image light signal is transmitted to the image signal processor for preprocessing to obtain a first image, which includes facial features. Then, the first image is transmitted to the camera hardware abstraction layer through the camera device driver. Then, the camera hardware abstraction layer can send the first image to the camera algorithm library.
[0152] Based on the support of digital signal processors and image processors, the camera algorithm library can first input the first image into a face enhancement depth model, perform face enhancement processing on the first image, and obtain a second image whose image quality is better than that of the first image.
[0153] For the sake of convenience, the first image is referred to as a low-quality portrait, and the second image is referred to as a high-quality portrait.
[0154] (2) Image processing stage:
[0155] like Figure 7 As shown, in the image processing stage, the image processing provided by the present application includes: ① human eye iris cutout; ② eye light / background separation; ③ eye light enhancement; ④ image fusion.
[0156] On the one hand, from the perspective of processing high-quality portraits with abnormal eye light after enhancement, the camera algorithm library first uses the human iris cutout algorithm provided in the present application to cut out the high-quality portrait, and extracts the human iris area image from the high-quality portrait; then the camera algorithm library uses the eye light / background separation algorithm provided in the present application to separate the eye highlight area and the eye background area of the high-quality portrait from the human iris area image, wherein the highlight feature in the eye highlight area of the high-quality portrait can be an "abnormal" eye light feature.
[0157] On the other hand, from the perspective of processing low-quality portraits, the camera algorithm library first uses the human eye iris cutout algorithm provided in this application to cut out the low-quality portrait, and extracts the human eye iris area image from the low-quality portrait; then the camera algorithm library uses the eye light / background separation algorithm provided in this application to separate the eye light features from the human eye iris area image, and obtains the eye highlight area and eye background area of the low-quality portrait.
[0158] In the embodiment of the present application, the catch light / background separation algorithm may include: first upsampling the human eye iris region image, then using a highlight removal algorithm to separate the eye highlight region (which may include the catch light feature) and the eye background region from the human eye iris region image, and then downsampling the separated eye highlight region and eye background region. The specific implementation process of the catch light / background separation algorithm will be described in detail below and will not be repeated here.
[0159] It should be noted that after downsampling the eye highlight area and the eye background area, taking into account the fact that part of the eyelid may remain in the iris area obtained by the iris cutout processing, and considering that the de-highlighting algorithm may have errors, it may appear that there is a highlight area in the iris area, but the highlight feature in the highlight area is not actually the catch light. In other words, the images taken by the user do not necessarily contain catch light, and the eye highlight area in the original image needs to be identified.
[0160] This application is based on the principle of high-fidelity enhancement, and determines whether there is eye light in the original image according to the parameters of the eye highlight area separated from the iris area of the human eye. The parameters of the highlight area may include the color moment, contrast, average brightness, saturation and / or area of the highlight area. Then, according to whether there is eye light in the original image, the corresponding eye light processing strategy is selected.
[0161] For example, in some scenarios, such as Figure 7As shown, if there is eye light in the original image, the eye light enhancement algorithm provided by this application is used to perform "eye light enhancement" based on the low-quality eye light features. Then, the eye light fusion algorithm provided by this application is used to perform "image fusion" on the enhanced eye light features and the eye background area of the high-quality portrait.
[0162] For example, in some other scenarios, if there is no eye light in the original image, eye light enhancement and image fusion will not be performed, and the eye highlight area and eye background area of the low-quality portrait will be discarded, as well as the "abnormal" eye light features of the high-quality portrait. Only the eye background area of the high-quality portrait will be retained to meet the user's demand for the fidelity of the facial area in shooting scenarios such as low-light shooting and telephoto shooting where low-quality portraits need to be enhanced by algorithms.
[0163] It should be noted that after obtaining the eye highlight area and eye background area of the low-quality portrait and the eye highlight area and eye background area of the high-quality portrait, the eye background area of the low-quality portrait and the eye highlight area of the high-quality portrait are discarded.
[0164] (3) Image generation stage:
[0165] On the one hand, when there is eye light in the original image, the electronic device will perform "image fusion" on the enhanced eye light features with the eye background area of the high-quality portrait to obtain a high-quality portrait with controllable enhancement of the eye light.
[0166] That is to say, in order to controllably enhance the eye light, first, the embodiment of the present application uses the eye light enhancement algorithm provided by the present application based on the low-quality eye light features in the low-quality portrait to achieve eye light enhancement and obtain enhanced eye light features. Then, the enhanced eye light features are used to replace the "abnormal" eye light features in the high-quality portrait, and the eye light features are corrected, so that a high-quality portrait with "normal" eye light after enhancement can be obtained. In this case, the portrait in the photo obtained by image processing has eye light, and the eye light is normal, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0167] On the other hand, when there is no eye light in the original image, the electronic device discards the "abnormal" eye light features and only retains the eye background area of the high-quality portrait. In this way, there is no "abnormal" eye light in the high-quality portrait output at the end, thereby correcting the eye light. In this case, the portrait in the photo obtained by image processing does not contain "abnormal" eye light, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portrait photos.
[0168] After obtaining the photo through image processing, the camera algorithm library can send the photo obtained through image processing to the camera hardware abstraction layer. Then, the camera hardware abstraction layer can send the photo for display.
[0169] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.
[0170] The execution subject of the iris feature extraction method provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the iris feature extraction method, and the present application scheme can be implemented in hardware and / or software, which can be determined according to actual use requirements and is not limited in the embodiment of the present application. The following is an exemplary description of the iris feature extraction method provided in the embodiment of the present application using an electronic device as an example.
[0171] Combine the following Figure 8 Describe the implementation process of the overall solution of this application. Figure 8 As shown, the iris feature extraction method provided in the embodiment of the present application includes the following steps S201-S208.
[0172] First of all, it should be noted that in the embodiment of the present application, when the face enhancement function of the electronic device is turned on, the electronic device will identify whether the photographed photo contains facial features during the process of taking the photo. When the electronic device identifies that the photographed photo contains facial features and the image quality does not meet the preset image quality requirements, the electronic device uses the face enhancement depth model to process the photo to achieve full face enhancement. For the sake of convenience, the face enhancement function is turned on by default in the embodiment of the present application.
[0173] The embodiments of the present application focus on the user's demand for the fidelity of the face area in scenes such as low-light shooting and telephoto shooting, and can controllably enhance the eye light of the human eye to compensate for the defect of unnatural eye light caused by full-face enhancement through a face enhancement depth model.
[0174] S201A, obtain low-quality portrait.
[0175] S201B, obtain high-quality portraits.
[0176] The high-quality portrait is an image obtained by processing the low-quality portrait with the face enhancement deep model. After the low-quality portrait is processed with the face enhancement deep model, the eye light feature is often mistakenly enhanced, and the enhanced eye light looks very unnatural.
[0177] Among them, the low-quality portrait can also be called a low-quality facial image or a low-quality portrait original image or a low-quality original image, and the high-quality portrait can also be called a high-quality facial image or a high-quality portrait abnormal image or a high-quality abnormal image.
[0178] In the embodiment of the present application, in order to compensate for the unnatural defect of eye light caused by full face enhancement through the face enhancement depth model, the electronic device provides an automatic eye light correction function, which can implement the iris feature extraction method provided in the embodiment of the present application. The electronic device can turn on or off the automatic eye light correction function in response to user operation.
[0179] In the case where the electronic device does not have the automatic correction function for eye light turned on, when the user triggers the photo shooting, the electronic device will input the original photo taken (called a low-quality portrait) into the preset face enhancement depth model to obtain a high-quality portrait with face enhancement. At this time, in the high-quality portrait, while the face is enhanced, the eye light is also enhanced, and the enhanced eye light may be unnatural. It should be noted that in the high-quality portrait, since the enhanced eye light is unnatural, the image can be called a high-quality portrait with "abnormal" enhanced eye light.
[0180] When the automatic eye light correction function of the electronic device is turned on, when the user triggers the photo taking function, the electronic device will simultaneously obtain the original photo (low-quality portrait) and the high-quality portrait obtained through the face enhancement depth model.
[0181] In some embodiments, a low-quality portrait is obtained by: under conditions of dim light, long distance, etc., after a user presses a camera shutter button of the electronic device, the electronic device takes a photo containing facial features.
[0182] S202A-S203A, perform iris cutout processing on the low-quality portrait to obtain the iris region of the low-quality portrait. Perform catch light / background separation processing on the iris region of the low-quality portrait to obtain the eye highlight region and eye background region of the low-quality portrait.
[0183] S202B-S203B, perform iris cutout processing on the high-quality portrait to obtain the iris / sclera region of the high-quality portrait. Perform catch light / background separation processing on the iris / sclera region of the high-quality portrait to obtain the eye highlight region and eye background region of the high-quality portrait.
[0184] S202A and S203A are for image processing of low-quality portraits. S202B and S203B are for image processing of high-quality portraits. The image processing of low-quality portraits is similar to the image processing of high-quality portraits. The following is a detailed description of the eye light processing method for portraits provided by the present application in conjunction with the accompanying drawings.
[0185] For example, Fig. 9 As shown, the eye region in the portrait is first detected, such as the part circled by the black dotted line; then the eye region is subjected to eye cutout processing to obtain the eye image. Next, the iris region is detected in the eye image, such as the part circled by the white dotted line; then the iris cutout processing is performed on the iris region, and the iris image can be extracted from the portrait. The specific implementation process of the iris cutout algorithm will be described in detail below, which will not be repeated here.
[0186] In addition, the human eye image can also be processed by cutting out the sclera of the human eye. Fig. 9 As shown, the human eye sclera area is detected in the human eye image, and the annular part circled by the white dotted line is the human eye sclera area; then after performing human eye sclera cutout processing on the human eye sclera area, the human eye sclera image can be extracted from the portrait.
[0187] In some embodiments, if the iris region of a human eye is extracted from a human portrait, the portion of the iris region of the human eye excluding the highlight region may be referred to as the eye background region.
[0188] In other embodiments, if the sclera region of the human eye is extracted from the portrait, the portion of the sclera region of the human eye except the highlight region may also be referred to as the eye background region.
[0189] In some further embodiments, if the iris region and the sclera region are extracted from the portrait, the portions of the iris region and the sclera region except the highlight region may be referred to as the eye background region.
[0190] In the embodiment of the present application, the “eye background area” may also be referred to as the human eye iris / sclera background area, which covers the following three situations:
[0191] (1) “Eye background area” refers to the part of the iris area of the human eye excluding the highlight area.
[0192] (2) “Eye background area” refers to the part of the sclera area of the human eye excluding the highlight area.
[0193] (3) “Eye background area” refers to the iris area and the sclera area of the human eye excluding the highlight area.
[0194] It should be noted that the embodiments of the present application support eye light / background separation processing for the iris area of the human eye, and can also perform eye light / background separation processing on the sclera area of the human eye. It also supports eye light / background separation processing for the iris area of the human eye and the sclera area of the human eye.
[0195] For ease of explanation, each embodiment of the present application is exemplified by taking the eye light / background separation processing of the iris area of the human eye as an example.
[0196] For example, if Fig.10 As shown, first, the human eye iris cutout algorithm is used to process the portrait, and the human eye iris area is determined in the portrait. Then, the eye light / background separation algorithm is used to process the human eye iris area image to obtain the eye highlight area and the eye background area. In addition, the application will also perform eye light recognition based on whether the eye highlight area contains eye light, and the specific implementation process will be described in detail below.
[0197] In an embodiment of the present application, the catch light / background separation algorithm includes upsampling the human eye iris image, separating the eye highlight area and the eye background area from the human eye iris image based on a highlight removal algorithm, and downsampling the eye highlight area and the eye background area.
[0198] (1) Upsample the human iris image.
[0199] Here, upsampling refers to enlarging the image.
[0200] In the embodiment of the present application, upsampling can be achieved by using an interpolation method or a deconvolution (also called a transposed convolution) method. For example, the resize() function in the open source computer vision library OpenCV can be called to adjust the image size by an interpolation algorithm (such as a bilinear interpolation algorithm).
[0201] In order to make the subsequent catch light / background separation more detailed, the iris / sclera area of the human eye is first upsampled to achieve image magnification, and then the upsampled iris / sclera area of the human eye is subjected to catch light / background separation processing.
[0202] In order to reduce the noise amplification caused by upsampling and the jagged edges of the iris and the eye light, the present application uses filtering and anti-aliasing upsampling alternately to achieve image magnification.
[0203] Fig.11 Schematic diagram of the upsampling process of this application is shown. Fig.11As shown, the upsampling process includes steps S2031-S2035. Exemplarily, assuming that the pixel size of the human eye iris region image (which can be referred to as the human eye iris image) in the portrait is about 20×20, the human eye iris image is firstly anti-aliased up-sampled, thereby realizing that the human eye iris image is enlarged, while avoiding the appearance of jagged edges of the iris edge and the eye light edge. For example, the pixel size of the enlarged human eye iris image is 100×100. Then, the enlarged human eye iris image is filtered and denoised to remove noise. Then, it is determined whether the enlarged human eye iris image reaches a specified size. Exemplarily, the specified size is 200×200. When the enlarged human eye iris image does not reach the specified size, the human eye iris image continues to be anti-aliased up-sampled and filtered and denoised until the enlarged human eye iris image reaches the specified size of 200×200, and then the human eye iris image is no longer anti-aliased up-sampled and filtered and denoised to obtain a human eye iris image with an enlarged size.
[0204] Through the solution of the present application, after multiple (for example, 2 to 3 times) cycles of anti-aliasing upsampling and filtering denoising processing, the human eye iris image is gradually enlarged (for example, 10 times), so that the subsequent eye light / background separation is more detailed, thereby improving the accuracy of image processing.
[0205] (2) Based on the highlight removal algorithm, the eye highlight area and eye background area are separated from the human iris image.
[0206] In the embodiments of the present application, taking into account that the highlight removal (highlight i.e., catch light) process will not produce negative effects on the background area of the human eye, such as smearing, missing, brightness change of non-catch light area, etc., the present application adopts a simple and efficient universal highlight removal algorithm, which can quickly realize the separation of highlights and background on larger-sized human iris images.
[0207] Through the highlight removal algorithm, the highlight area can be detected in the iris area of the human eye. For example, the mask of the highlight area can be generated by image threshold segmentation. Among them, the flood filling method can be used to solve the connected components of the mask. The connected component is defined as: a maximal connected subgraph of an undirected graph.
[0208] Assuming that the image imaging conforms to the dichotomous reflection model, the image I is decomposed into diffuse reflection components (denoted as V diff ) and the specular component (denoted as V spec ), the diffuse reflection component corresponds to the de-eye light map, and the specular reflection component corresponds to the eye light map. After theoretical derivation, the approximate specular reflection component is expressed as β s :
[0209] β s =I min -τ(p) (Formula 1)
[0210] Among them, I min It represents a single-channel image obtained by taking the minimum value of the RGB three channels of each pixel in image I; τ(p) represents an offset related to the mean and variance of the image pixels.
[0211] Denote the approximate diffuse component as V' diff :
[0212] V d ' iff = I-kβ s (Formula 2)
[0213] In order to solve the proportional coefficient k, first normalize the image to a pixel interval of [0,1], and take the area with a pixel value greater than 0.1 as a possible highlight area. Then use the flood fill method to obtain the dominant highlight connected component area (the first M largest) of the area. It should be noted that the diffuse color intensity of the highlight connected component area and its edge area should be consistent, that is, the mean of the diffuse component is equal. Further, under this smooth color transition criterion, the least squares method is used to obtain the k value. Substituting the k value into formula 2, the de-eye catch light map V′ can be obtained. diff Then, the eye light map is obtained by the following formula 3:
[0214] V s ' pec = IV d ' iff (Formula 3)
[0215] Among them, V′ diff represents the removed eye-catching light image, V′ spec Indicates the eye light pattern.
[0216] It should be noted that when the present application uses the de-highlighting algorithm to perform eye light / background separation, considering that the eye light area in the actual scene may not be continuous and there may be multiple connected components, the first M largest connected components that "may be eye lights" are determined as highlight areas that need to be separated.
[0217] For example, if M is 2, assuming that there are four connected components in the iris region of the human eye: 7, 20, 9, 5, then Fig.12 As shown, the area where the first two largest connected components "20 and 9" of "possibly catch light" in the iris area of the human eye are located can be determined as the highlight area that needs to be separated.
[0218] In the embodiment of the present application, the catch light area mask is obtained by an automatic highlight removal algorithm, which can support the separation of discontinuous catch light areas of arbitrary shapes.
[0219] Through this application solution, the iris, sclera and eye light areas can be automatically identified without manual demarcation.
[0220] (3) Downsample the eye highlight area and eye background area.
[0221] Downsampling refers to reducing the size of an image. Similar to upsampling, downsampling can be achieved by using interpolation, pooling, or convolution.
[0222] Since the subsequent steps will have a great impact on whether the final eye light enhancement effect is natural, in order to prevent it from being affected by the interpolation algorithm, avoid post-processing after image fusion, and to improve the efficiency of the algorithm, this step will downsample the separated eye highlight area and eye background area to the original image size (such as reducing the current image to 1 / 10).
[0223] Exemplarily, the pixel sizes of the separated eye highlight area and eye background area are 50×50 and 200×200 respectively. After downsampling, the pixel sizes of the eye highlight area and eye background area are 5×5 and 20×20 respectively.
[0224] It should be noted that if the image is directly downsampled, for example, using nearest linear interpolation, it is found that the downsampled image will have a lot of jagged edges. In some embodiments, the present application can use anti-aliasing downsampling to scale the size to the original image size to avoid jagged edges on the iris edge and the eye light edge.
[0225] Fig.13 FIG. 1 is a schematic diagram showing an embodiment of the present invention for performing catch light / background separation on an image of the iris region of a human eye. Fig.13 As shown in the figure, taking the catch light / background separation of the human eye iris area image of a low-quality portrait as an example, the human eye iris area image is first upsampled and the human eye iris area image is enlarged; then the enlarged human eye iris area image is separated and processed by the catch light / background separation algorithm to obtain the eye highlight area and the eye background area; then the separated eye highlight area and eye background area are downsampled and the image size is reduced to the original image size, thereby obtaining the low-quality eye highlight area and eye background area.
[0226] It should be noted that the eye highlight area of a low-quality portrait may contain eye light features. If the low-light area may contain eye light features, then the eye light features will be enhanced to replace the "abnormal" eye light. Therefore, in subsequent image processing, the eye highlight area of the low-quality portrait will be used to identify the eye light features. In addition, the eye background area of the low-quality portrait is not used in subsequent image processing, so the eye background area of the low-quality portrait is discarded.
[0227] It should also be noted that, similar to the above process of performing catch light / background separation on the iris region image of the low-quality portrait, the catch light / background separation is performed on the iris region image of the high-quality portrait to obtain the eye background area and eye highlight area of the high-quality portrait. Among them, the highlight feature in the eye highlight area of the high-quality portrait may be an "abnormal" catch light feature, which needs to be replaced with a "normal" catch light feature. Therefore, the eye highlight area of the high-quality portrait will be discarded.
[0228] It should also be noted that in the embodiments of the present application, "normal" eye light refers to the distortion effect of "abnormal" eye light. For example, if the eye light in a portrait is real and natural, it can be considered as "normal" eye light; or if there is no "abnormal" eye light in the portrait, it can also be considered as "normal" eye light.
[0229] S204: judging whether the low-quality portrait has an eye light feature according to the highlight attribute of the eye highlight area of the low-quality portrait.
[0230] In an embodiment of the present application, after the eye highlight area and the eye background area are separated by the eye light / background separation algorithm, taking into account the fact that part of the eyelid may remain in the human eye iris area obtained by the human eye iris cutout processing, and taking into account the possible errors in the de-highlighting algorithm, it may happen that there is a highlight area in the human eye iris area but the highlight feature in the highlight area is not actually the eye light. In other words, the original image taken by the user does not necessarily contain the eye light, and it is necessary to identify the eye highlight area in the original image, that is, to judge whether the low-quality portrait has the eye light feature.
[0231] Considering that not all images taken by users contain eye light, this application is based on the principle of high-fidelity enhancement. If a low-quality portrait does not have eye light features, the mistakenly added eye light features can be directly removed from the corresponding high-quality portrait. Since the eye light / background separation algorithm does not change the iris / sclera background of the human eye, the detailed information of the iris / sclera area of the high-quality portrait can be retained.
[0232] In the embodiment of the present application, the area of the eye highlight area separated from the human eye iris area can be used to determine whether there is catch light in the original image. For example, if the area of the highlight area is 0, it can be directly determined that there is no catch light in the highlight area. However, due to factors such as the fact that the human eye iris portion obtained by cutting out the image may have some eyelids remaining, and the de-highlighting algorithm may have errors, even if the area of the highlight area is not 0, the highlight area may still be without catch light.
[0233] In order to improve the accuracy of eye light recognition, in the embodiment of the present application, a joint judgment can be made based on multiple highlight attributes of the eye highlight area separated from the human eye iris area, so that it can be more accurately judged whether there is eye light in the original image. Among them, the highlight attributes of the eye highlight area may include attributes such as color moment, contrast, average brightness, saturation and / or area of the highlight area.
[0234] In some embodiments, the present application selects to use multiple hyperparameters as features to train a simple binary classifier offline. Among them, the hyperparameters include color moment, contrast, average brightness, saturation, highlight area, etc. It should be noted that in the context of machine learning, hyperparameters are parameters whose values are set before starting the learning process. By optimizing the hyperparameters, a set of optimal hyperparameters is selected for the learning machine to improve the performance and effect of learning.
[0235] For example, taking the binary classifier as a support vector machine (SVM), multiple highlight attributes of the eye highlight area are input into the trained SVM, and the SVM can directly output the eye light recognition result. Through the present application scheme, it is possible to comprehensively judge whether the original image has eye light based on multiple highlight attributes of the eye highlight area, which can improve the accuracy of eye light recognition.
[0236] If the eye highlight region of the portrait has a catch light feature, then continue to execute S205 - S206 ; if the eye highlight region of the portrait does not have a catch light feature, then continue to execute S207 .
[0237] S205: If the original image has eye light features, adaptively enhance the eye light features.
[0238] It should be noted that the size, shape, brightness and other characteristics of the original area of the eye light are all random and uncertain, so the same enhancement parameters (such as fixed contrast stretching parameters) are difficult to be compatible with all situations.
[0239] In order to increase the generalization of the algorithm, this application uses an adaptive catch light contrast enhancement method. First, the length and width of the catch light area (for example, the length and width of the minimum circumscribed rectangle of the catch light area), average brightness, brightest point brightness and other parameters are calculated, and then the corresponding contrast enhancement algorithm parameters are calculated. Among them, the parameters of the contrast enhancement algorithm include the gamma value used for gamma correction, and the partition block size and contrast limit for limiting the contrast adaptive histogram equalization. The following introduces the application of the parameters of the contrast enhancement algorithm in this application.
[0240] First, let's explain the gamma value. The gamma value is used to perform gamma correction on the highlight area to stretch the image contrast in the highlight area. The image corrected by a certain gamma value can be obtained through the gamma correction function. The gamma correction function can be expressed as:
[0241] I′=I γ
[0242] Among them, γ represents the gamma value, I represents the original image, and I′ represents the image after contrast enhancement.
[0243] For example, the gamma value corresponding to the highlight area may be calculated according to the ratio of the brightness of the brightest point in the highlight area to 255. The gamma correction is performed on the highlight area according to the gamma value to stretch the image contrast of the highlight area.
[0244] The following is an explanation of the block size and contrast limit of contrast-limited adaptive histogram equalization.
[0245] First of all, it is important to note that ordinary histogram equalization transforms all pixels using the same transformation derived from the image histogram. This works well when the pixel values throughout the image are similarly distributed. However, when the image contains areas that are significantly brighter or darker than the bulk of the image, the contrast in these areas will not be adequately enhanced.
[0246] Adaptive histogram equalization (AHE) transforms each pixel by using a transformation function obtained from the histogram calculated from the neighborhood. Therefore, it is suitable for improving the local contrast of an image. However, AHE tends to over-amplify noise in relatively uniform areas of the image.
[0247] Contrast-limited adaptive histogram equalization (CLAHE) further prevents this from happening by limiting the degree of amplification. Contrast limitation is achieved by limiting the ratio of the number of pixels per column of each local area histogram to the average grayscale distribution of the entire image.
[0248] OpenCV's CLAHE implementation includes two important parameters:
[0249] (1) Local area size, such as 4×4.
[0250] (2) Contrast limit (ClipLimit), such as 20.
[0251] ClipLimit is the clipping limit value of CLAHE. When the pixel data of the histogram grouping of each block image of the image exceeds this limit, clipping is required. When clipping, the part of each grouping that exceeds the limit is clipped, and the excess part of all groups is accumulated and evenly distributed to each group. In other words, the part of the histogram bins with a height greater than the calculated ClipLimit is clipped, and then evenly distributed to the entire histogram. If there is still excess after distribution, the above steps can be recursively repeated.
[0252] When this method is used to enhance the eye light, each channel of the eye light map needs to be processed. For example, the side length of the local area size is: min(L / 2, 3), and the clipping limit parameter is: max(D / 2, 2). Among them, L represents the shortest side of the eye light area, and D represents the maximum value of the histogram in the circumscribed rectangle of the maximum highlight area. min() means taking the minimum value, and max() means taking the maximum value.
[0253] In the embodiment of the present application, each eye light area can be adaptively enhanced in a targeted manner. Specifically, the enhancement parameters corresponding to a certain eye light area can be calculated based on the size, shape, brightness and other characteristics of the eye light area, and the eye light area can be enhanced in a targeted manner using the enhancement parameters corresponding to the eye light area. Therefore, the embodiment of the present application can adaptively enhance different eye light areas.
[0254] For example, Table 1 shows parameters of the catch light area, such as length, width, average brightness, and brightest point brightness. As shown in Table 1, the catch light area includes a catch light area 1 and a catch light area 2.
[0255] Table 1
[0256] length width Average brightness Brightest point brightness Catch light area 1 8 pixels 7 pixels 100 gray levels 155 gray levels Catch light area 2 3 pixels 3 pixels 80 gray levels 115 gray levels
[0257] As shown in Table 1, in the case of a 256-grayscale image signal, the size of eye light area 1 is larger than the size of eye light area 2, the average brightness of eye light area 1 is larger than the average brightness of eye light area 2, and the brightness of the brightest point in eye light area 1 is larger than the brightness of the brightest point in eye light area 2.
[0258] After calculating the length, width, average brightness, and brightness of the brightest point in the catch light area, the parameters of the corresponding contrast enhancement algorithm are calculated. Here, the calculation of gamma values for different areas is used as an example for illustrative description. As mentioned above, the gamma value can be calculated based on the ratio of the brightness of the brightest point to 255. The following Table 2 shows the gamma values corresponding to different catch light areas.
[0259] Table 2
[0260] Brightest point brightness Calculated gamma value Catch light area 1 155 gray levels 0.61 Catch light area 2 115 gray levels 0.45
[0261] Through the solution of the present application, for the catch light area 1, a gamma value of 0.61 is used for gamma correction to stretch the image contrast of the catch light area 1. For the catch light area 2, a gamma value of 0.45 is used for gamma correction to stretch the image contrast of the catch light area 2. That is to say, in the embodiment of the present application, corresponding gamma values can be used for correction for different highlight areas.
[0262] For catch light areas with different characteristics such as size, shape, brightness, etc., the embodiment of the present application can perform targeted enhancement processing on each catch light area respectively. Specifically, the enhancement parameter A corresponding to catch light area 1 can be calculated according to the size, shape, brightness and other characteristics of catch light area 1, and the enhancement parameter A corresponding to catch light area 1 can be used to enhance the catch light area 1 in a targeted manner; similarly, the enhancement parameter B corresponding to catch light area 2 can be calculated according to the size, shape, brightness and other characteristics of catch light area 2, and the enhancement parameter B corresponding to catch light area 2 can be used to enhance the catch light area 2 in a targeted manner. Therefore, the embodiment of the present application can make different catch light areas adaptively enhanced.
[0263] It should be noted that the above embodiment uses the example of calculating the parameters of the contrast enhancement algorithm according to the length, width, average brightness, and brightest point brightness of the eye light area as an example for illustrative description. It can be understood that in actual implementation, the parameters of the contrast enhancement algorithm can also be calculated according to other parameters of the eye light area. The specific parameters can be determined according to actual use requirements and are not limited in the embodiments of this application.
[0264] It should be noted that the present application can use the above-mentioned catch light enhancement algorithm to adaptively enhance the eye light in the original image. The catch light enhancement algorithm uses an adaptive eye light contrast enhancement method to controllably enhance the eye light. The catch light enhancement algorithm provided by the present application is not a depth model, and can avoid the problem of abnormalities after the eye light enhancement caused by the depth model, and meet the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting that require algorithm-enhanced low-quality portraits.
[0265] S206: performing image fusion of the enhanced catch light feature and the eye background area of the high-quality portrait.
[0266] In order to controllably enhance the eye light, first, based on the low-quality eye light features in the low-quality portrait, the embodiment of the present application adopts the eye light enhancement algorithm provided by the present application to achieve eye light enhancement and obtain enhanced eye light features. Then, the enhanced eye light features are used to replace the "abnormal" eye light features in the high-quality portrait, and the eye light features are corrected, so that a high-quality portrait with "normal" eye light after enhancement can be obtained. This satisfies the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0267] It should be noted that for low-quality portraits and high-quality portraits, the positions and relative positions of the iris and eye light are not exactly the same, and the edge of the eye light feature of the low-quality portrait may not be softly attenuated after adaptive enhancement. If the enhanced eye light feature is directly copied and pasted onto the iris / sclera background of the high-quality portrait, the eye light effect is unnatural. In order to make the enhanced eye light blend naturally with the iris / sclera background of the high-quality portrait, the embodiment of the present application adopts the method of affine transformation and eye light area edge feathering.
[0268] For example, Fig.14 FIG. 1 is a schematic diagram showing the process of fusing the catch light with the human eye iris background provided by an embodiment of the present application. Fig.14 As shown, first, an affine transformation matrix is determined based on the edge key points of the iris background area of the low-quality portrait and the edge key points of the iris background area of the high-quality portrait. Then, the enhanced eye light is multiplied by the affine transformation matrix, and an affine transformation is performed to obtain an aligned enhanced eye light. Then, the aligned enhanced eye light is adaptively feathered to obtain a feathered enhanced eye light. Then, the feathered enhanced eye light is image-fused with the iris background area. As a result, the controllably enhanced eye light of the present application can be naturally fused to the eye background area of the high-quality portrait.
[0269] It should be noted that here, affine transformation and image fusion based on the iris background area are used as an example for explanation. It can be understood that in actual implementation, affine transformation and image fusion can also be performed based on the sclera background area. The specific implementation method can be determined according to actual usage requirements, and the embodiments of this application are not limited to this.
[0270] The following first introduces the principle of affine transformation, and then explains how this application performs affine transformation.
[0271] During the affine transformation of an image, it is necessary to operate on each pixel in the image. Assuming that the coordinates of the pixel point X in the image are (x, y), the pixel point X can be expressed in matrix form as follows:
[0272]
[0273] like Fig.15 As shown, an affine transformation may include rotation, scaling, and translation. An affine transformation is a linear transformation plus a translation.
[0274] The matrix of the rotation parameter R and the matrix of the translation parameter T are given as examples below.
[0275]
[0276]
[0277] Among them, R is an orthogonal matrix, θ represents the rotation angle, and T represents the translation distance. In addition, s can be used to represent the scaling ratio.
[0278] In some embodiments, after rotation, the pixel point X can be expressed in matrix form as:
[0279]
[0280] In some embodiments, after translation, the pixel point X can be expressed in matrix form as:
[0281]
[0282] In some embodiments, after scaling, the pixel point X can be expressed in matrix form as:
[0283]
[0284] In other embodiments, after image affine transformations such as rotation, translation, and scaling, the coordinates of pixel point X become (u, v), and pixel point X can be expressed in matrix form as follows:
[0285]
[0286] Based on the above principles, the process of performing affine transformation based on the enhanced eye light features and the human eye background area in the embodiment of the present application is described below.
[0287] For example, if Fig.16 As shown, the edge feature points of the eye background area of the low-quality portrait can be calibrated, and the corresponding pixel matrix is recorded as F1, and the edge feature points of the eye background area of the high-quality portrait can be calibrated, and the corresponding pixel matrix is recorded as F2.
[0288] Assume that the affine transformation matrix is M, and the pixel matrix corresponding to the enhanced eye light feature is X. According to F1*M=F2, the affine transformation matrix M between the low-quality portrait and the high-quality portrait can be converted. Exemplarily, the converted affine transformation matrix M can be expressed as:
[0289]
[0290]
[0291] Among them, matrix M represents the complete affine transformation matrix, matrix A controls rotation and scaling, and matrix B controls translation.
[0292] Then, the pixel matrix X corresponding to the enhanced catch light feature is multiplied by the affine transformation matrix M to obtain the aligned enhanced catch light region X'. In addition, it is known that the affine transformation is a linear transformation plus a translation. After the affine transformation, the aligned enhanced catch light region X' can be expressed by a matrix as follows:
[0293]
[0294]
[0295] Through affine transformation processing, the enhanced eye light feature is aligned with the eye background area of the high-quality portrait in position, so as to ensure that the positions of the enhanced eye light feature and the eye background area are accurately aligned during image fusion. Since the unchanged position of the eye light feature relative to the iris is an important factor in the unchanged eye expression, the alignment of the eye light feature relative to the iris can present a natural and realistic eye light effect.
[0296] After completing the affine transformation, the edge of the catch light can also be feathered. Fig.17 As shown, by feathering the catch light, the inner and outer connecting parts of the edge of the catch light are blurred to achieve a gradual change, thereby achieving a natural connection effect and improving the image display effect.
[0297] It should be noted that the present application utilizes affine transformation and feathering processing to naturally blend the enhanced eye light into the eye area of the high-quality abnormal image, rather than simply copying and pasting the eye light.
[0298] The eye light fusion algorithm provided in the embodiment of the present application can naturally fuse the enhanced eye light feature with the human eye background area, so that the enhanced eye light feature can be aligned with the human eye background area in position, and the edge of the enhanced eye light feature is smooth, which can present a natural and realistic eye light effect.
[0299] S207, discarding the eye highlight area of the low-quality portrait and retaining only the eye background area of the high-quality portrait.
[0300] Through the above S205-S206 or S207, a high-quality portrait with normal eye light can be finally obtained, and the defect of unnatural eye light caused by full face enhancement through the face enhancement depth model can be corrected, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting that require algorithm-enhanced low-quality portrait photos.
[0301] Through the application scheme, the application focuses on the user's demand for the fidelity of the face area in scenes such as low-light shooting and telephoto shooting, and can controllably enhance the eye light of the human eye to make up for the unnatural defect of the eye light caused by the full face enhancement through the face enhancement depth model. In addition, for the case where the low-quality portrait has no eye light feature, the mistakenly added eye light feature can be directly removed from the corresponding high-quality portrait, which can also correct the unnatural eye light caused by the full face enhancement through the face enhancement depth model.
[0302] In summary, the embodiments of the present application can select a corresponding eye light processing strategy according to whether the low-quality portrait has eye light features.
[0303] Eye light processing strategy 1: If the original image has eye light features, the eye light is enhanced according to the eye light features, and then the enhanced eye light features are fused with the eye background area of the high-quality portrait.
[0304] For example, in some embodiments, when the eye highlight area of a low-quality portrait meets a preset eye light condition, it is determined that the original low-quality portrait image has eye light, and eye light enhancement processing is performed based on the eye highlight area of the low-quality portrait to obtain enhanced eye light features. The enhanced eye light features are image fused with the eye background area of the high-quality portrait to obtain a high-quality portrait with normal eye light after enhancement, thereby achieving controllable enhancement of eye light. In this case, the portrait in the photo obtained by image processing has eye light, and the eye light is normal, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm enhancement of low-quality portrait photos.
[0305] Eye light processing strategy 2: If there is no eye light in the original image, no eye light enhancement or image fusion will be performed, and only the eye background area of the high-quality portrait will be retained.
[0306] For example, in some embodiments, when the eye highlight area of a low-quality portrait does not meet the preset eye light condition, it is determined that the original image has no eye light, and then the eye light enhancement process is not performed, the low-quality eye light features are discarded, and only the eye background area of the high-quality portrait is retained. In this way, there is no "abnormal" eye light in the high-quality portrait output at the end, thereby correcting the eye light. In this case, the portrait in the photo obtained by image processing does not contain "abnormal" eye light, thereby meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm enhancement of low-quality portrait photos.
[0307] Fig.18 FIG. 1 is a schematic diagram showing an implementation process of the iris feature extraction method provided in an embodiment of the present application. Fig.18 As shown, when the electronic device has turned on the automatic correction function of the eye light, when the user triggers the photo, the electronic device will simultaneously obtain the original photo (low-quality portrait) and the high-quality portrait obtained through the face enhancement depth model, and the eye light of the high-quality portrait is abnormal. The electronic device performs iris cutout processing and eye light / background separation processing on the low-quality portrait to obtain the eye background and eye light of the low-quality portrait. At the same time, the electronic device performs iris cutout processing and eye light / background separation processing on the high-quality portrait to obtain the eye background and abnormal eye light of the high-quality portrait.
[0308] In this case, the eye background of the low-quality portrait is discarded, and the abnormal eye light of the high-quality portrait is discarded. The eye light of the low-quality portrait is adaptively enhanced to obtain enhanced eye light, and then the enhanced eye light is affine transformed and feathered, thereby achieving controllable enhancement of the eye light. Then, the processed enhanced eye light is image fused with the eye background of the high-quality portrait to obtain a high-quality portrait with normal eye light.
[0309] In this implementation, the portrait in the photo obtained through image processing has eye light, and the eye light is normal, thereby meeting the user's demand for the fidelity of the face area in shooting scenarios such as low-light shooting and telephoto shooting that require algorithm-enhanced low-quality portrait photos.
[0310] Fig.19 FIG. 2 shows another implementation process of the iris feature extraction method provided in the embodiment of the present application. Fig.19As shown, when the electronic device has turned on the automatic correction function of the catch light, when the user triggers the photo, the electronic device will simultaneously obtain the original photo (low-quality portrait) and the high-quality portrait obtained through the face enhancement depth model, and the catch light of the high-quality portrait is abnormal. The electronic device performs iris cutout processing and catch light / background separation processing on the low-quality portrait to obtain the eye background of the low-quality portrait, and the original photo has no catch light. At the same time, the electronic device performs iris cutout processing and catch light / background separation processing on the high-quality portrait to obtain the eye background of the high-quality portrait and abnormal catch light.
[0311] In this case, the eye background of the low-quality portrait is discarded, and the abnormal eye light of the high-quality portrait is discarded. In the case where the original photo has no eye light, this application can directly remove the mistakenly added eye light features from the corresponding high-quality portrait based on the principle of high-fidelity enhancement. Since the eye light / background separation algorithm does not change the iris / sclera background of the human eye, the detailed information of the iris / sclera area of the high-quality portrait can be retained.
[0312] In this implementation, the high-quality portrait outputted at the end does not contain "abnormal" eye light, thus correcting the eye light. In this case, the portrait obtained through image processing does not contain "abnormal" eye light, thus meeting the user's demand for the fidelity of the face area in shooting scenes such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0313] Through the iris feature extraction method provided in the embodiment of the present application, different eye light processing strategies can be adopted according to whether the original image has eye light: when the original image has eye light, the eye light of the original image is enhanced and the enhanced eye light replaces the "abnormal" eye light in the high-quality portrait, thereby correcting the "abnormal" eye light in the high-quality portrait; when the original image has no eye light, the "abnormal" eye light in the high-quality portrait is discarded to correct the "abnormal" eye light in the high-quality portrait. Therefore, the embodiment of the present application can correct the unnatural eye light defect caused by full face enhancement through the face enhancement depth model, thereby improving the user experience.
[0314] Human eye iris cutout algorithm
[0315] The above describes the overall process of the iris feature extraction method provided in the embodiment of the present application. The following describes in detail the human eye iris cutout algorithm (also referred to as the human eye iris extraction algorithm) in the iris feature extraction method provided in the embodiment of the present application.
[0316] Fig. 20 A schematic flowchart of human eye iris cutout in an embodiment of the present application is shown. Fig. 20 It includes the following steps S301-S309.
[0317] S301, input a face image.
[0318] In the embodiment of the present application, while the low-quality original portrait image is subjected to the human eye iris cutout processing, the high-quality original portrait image is also subjected to the human eye iris cutout processing. The high-quality portrait image is a face image obtained after the original image is input into the face enhancement depth model.
[0319] S302, detecting the eye area.
[0320] For example, a face image to be processed is input into the first network model, and a mask of the eye area and key points of the eye center are output. The first network model may include functions such as face parsing and face key point detection. The mask is a binary image composed of 0 and 1.
[0321] For example, Fig.21 (a) in FIG. 1 shows a schematic diagram of a mask of an eye region, wherein a white region represents an eye region. For example, Fig.21 (b) in FIG. 1 shows a schematic diagram of the eye center key point, where point O represents the eye center key point.
[0322] S303: Determine the iris area.
[0323] First of all, it should be noted that the palpebral fissure refers to the gap formed between the upper and lower eyelids. The length of the palpebral fissure of an adult is about 20 mm to 30 mm. The corresponding relationship between the palpebral fissure length and the iris size is known. For example, Table 3 shows the relationship between the palpebral fissure length and the iris radius (denoted as R).
[0324] Table 3
[0325] Palpebral fissure length Iris size (radius R) 20mm 5.5mm 25mm 6mm 30mm 6.5mm
[0326] In the embodiment of the present application, the length of the palpebral fissure can be determined based on the mask of the eye area. Then, based on the palpebral fissure length and the relationship between the palpebral fissure length and the iris size, the iris radius range can be roughly estimated. Then, based on the key point of the eye center and the estimated iris radius range, the iris area can be determined.
[0327] For example, the length of the palpebral fissure can be determined to be 25 mm based on the mask of the eye area. By looking up Table 1, it can be determined that the iris radius corresponding to the palpebral fissure length is 6 mm. Then, a circle is drawn with the key point O at the center of the eye as the center and a radius of 6 mm to determine the iris area in the human eye. Fig.21 (c) in FIG. 1 shows a schematic diagram of the iris region of the eye, where R represents the radius of the estimated iris region.
[0328] S304: extracting feature points of the iris area.
[0329] For example, Fig. 22 A schematic flowchart of extracting feature points of an iris area in the present application is shown. Fig. 22 Includes the following steps S3041-S3048.
[0330] S3041, input iris area image.
[0331] For the sake of convenience, it is assumed that the iris area is N*N pixels.
[0332] S3042, enhance the contrast of the iris area.
[0333] In the embodiment of the present application, after the iris region is determined, the contrast of the iris region is enhanced to highlight the edge features, so as to facilitate the detection and extraction of feature points of the iris region.
[0334] S3043, transform to the polar coordinate system with (N / 2, N / 2) as the center.
[0335] It should be noted that since the iris edge region is a curve, it is difficult to extract feature points at the iris edge. Therefore, in the embodiment of the present application, the iris region is converted to a polar coordinate system with (N / 2, N / 2) as the center to obtain a 360*N / 2 polar coordinate image, thereby converting the curve feature into a straight line feature. 360 represents 360 degrees.
[0336] Assume that the iris region is N*N pixels. In the rectangular coordinate system, the center point of the iris region can be expressed as (x, y), x=N / 2, y=N / 2. Correspondingly, in the polar coordinate system, the coordinates of the center point of the iris region can be expressed as (ρ, θ), the radius coordinate ρ=0, the angle coordinate θ=0, that is, the pole (0, 0) of the polar coordinate system is the center point of the iris region.
[0337] The ρ coordinate represents the distance from the pole, and the θ coordinate represents the angle from the 0° ray (also called the polar axis) in the counterclockwise direction. The polar axis is the positive direction of the x-axis in the plane rectangular coordinate system. The angular coordinate θ is also called the polar angle or azimuth.
[0338] The value range of ρ is [0, N / 2] and the value range of θ is [0, 359].
[0339] S3044, extracting linear features through a linear filter.
[0340] In the embodiment of the present application, a linear filter is used in the feature point extraction process, so that the feature points of the iris area can be extracted more accurately.
[0341] S3045, input image features in the θ direction.
[0342] Among them, θ is 0, 1, ..., 359 respectively.
[0343] S3046, obtaining feature points in the θ direction.
[0344] Since the edge of the iris changes significantly, optionally, for each angular feature, feature point detection and extraction can be performed by peak detection. Specifically, peak detection can be performed on the eye region in the polar coordinate system to obtain feature points (ρ, θ) in each angular feature.
[0345] S3047, when it is judged that θ≤359, return to continue executing S3045-S3046 until θ reaches 359.
[0346] The following example illustrates the process of obtaining feature points in various directions in sequence.
[0347] When θ is 0, the feature points in the 0-angle direction are obtained based on the image features in the 0-angle direction.
[0348] Then, when θ is 1, the feature points in the 1-angle direction are obtained according to the image features in the 1-angle direction.
[0349] By analogy, when θ is 359, the feature points in the 359 angle direction are obtained according to the image features in the 359 angle direction.
[0350] Thus, feature points at various angles of 0, 1, ... 359 are obtained in the iris area.
[0351] S3048, output all feature points of the iris area.
[0352] Fig.23 The specific implementation process of extracting the feature points of the iris area in the above S3041-S3048 is shown. Specifically, for the iris area image with a pixel size of N*N, it is converted to a polar coordinate system with (N / 2, N / 2) as the center, and then the linear features are extracted through a linear filter. θ starts from 0 and until θ reaches 359, feature extraction is performed in each direction. Among them, the eye area in the polar coordinate system can be peak detected (the local maximum value in the edge feature point is taken) to obtain the edge feature points in each angular direction, and the edge feature points in all angular directions form a circle or ellipse. Among them, the edge feature points and the feature points surrounded by the edge feature points constitute all the feature points of the iris area.
[0353] It should be noted that among the feature points extracted from the iris region, some feature points are actually covered or blocked by the upper and lower eyelids, so these feature points in the iris region need to be removed. Therefore, it is necessary to perform the following step S305, that is, to filter the feature points in the iris region.
[0354] S305, screening feature points of the iris area.
[0355] For example, Fig.24 A schematic flow chart of screening feature points of an iris region in the present application is shown. Fig.24 The method includes steps S3051-S3055. After extracting all feature points in the iris area, feature points in the periphery and pupil area can be eliminated according to the roughly estimated iris radius range, feature points covered by the upper and lower eyelids in the iris area can be eliminated according to the mask of the eye area, and isolated points (outlier feature points) can be eliminated according to the connected length or area to achieve the screening of iris edge feature points.
[0356] The following describes the process of eliminating isolated points based on the connected length or area. First, edge detection is performed on the iris area, and the identification point rectangle is extracted. Rectangles with an area smaller than a certain threshold are determined to be noise points, and the pixel values in the noise point area are changed to background color pixel values.
[0357] It should be noted that the embodiment of the present application performs peak detection on the eye area in the polar coordinate system to obtain feature points, and introduces the results of face analysis and key point detection results to perform feature point screening, which can improve the accuracy of eye detection.
[0358] It should be noted that, for the sake of convenience, the screening of iris edge feature points in S305 is referred to as the first screening.
[0359] S306, obtaining an ellipse with the largest support by ellipse fitting, and generating a mask 1 of the iris region according to the ellipse.
[0360] Fig.25 A schematic flow chart of ellipse fitting of the present application is shown. Fig.25 Includes steps S3061-S3069.
[0361] Specifically, after the first screening of iris edge feature points is completed, the initial estimation of the ellipse is firstly realized by least square fitting for the first screened iris edge feature points, that is, the ellipse is initialized to obtain the initialized ellipse.
[0362] Then, the distance error is calculated based on the distance between each feature point of the iris edge and the initialization ellipse.
[0363] Then, the feature points are screened according to the size of the distance error. For example, if the distance error corresponding to feature point A in the iris area is greater than the error threshold (e.g., 1 pixel), feature point A is removed; if the distance error corresponding to feature point B in the iris area is less than or equal to the error threshold, feature point B is retained. The total number of feature points removed in this cycle calculation (referred to as the number of feature points removed) is recorded, and the number of feature points removed is recorded as N. Among them, the screening by distance error is called the second screening of iris edge feature points.
[0364] It should be noted that after continuous screening of the distance error, the number of feature points to be eliminated N is getting smaller and smaller. In order to improve the operation efficiency, a threshold value of elimination can be preset to determine whether the number of feature points to be eliminated in this cycle calculation meets the requirement. For example, the threshold value of elimination can be 2. The specific value of the threshold value of elimination can be set according to actual use requirements, and the embodiment of the present application is not limited thereto.
[0365] Then, determine whether the number of feature points to be eliminated N is less than or equal to the elimination threshold. If the change of some feature points on the iris edge is greater than the elimination threshold, fit the ellipse Z based on the feature points obtained by distance error screening, and calculate the support of all feature points (denoted as points) on the iris edge based on the gradient of the feature point pixels:
[0366]
[0367] Among them, S represents the support of all feature points of the iris edge, p represents any feature point among the feature points screened by distance error, represents the gradient of the ellipse at the feature point p, Represents the gradient of the image at feature point p.
[0368] It should be noted that the calculation process of the gradient of the ellipse and the gradient of the image are different. The coordinate value corresponding to the feature point p is (x, y).
[0369] Ellipse Gradient The cone equation corresponding to the obtained fitting ellipse is obtained by deriving it:
[0370] ellipse(x,y)=Ax 2 +Bxy+Cy 2 +Dx+Ey+F
[0371] Among them, x and y represent the coordinate values of the X-axis and the Y-axis respectively, and A, B, C, D, E, and F are coefficients respectively. The changes of the X-axis and Y-axis are the derivatives of the cone equation about the X and Y directions respectively.
[0372] Different from the gradient of an ellipse, the pixel gradient changes as follows: It refers to the rate of change of a pixel (i.e., feature point p) in the image in the X and Y directions (compared with adjacent pixels). It is a two-dimensional vector consisting of two components: the change in the X axis and the change in the Y axis. (1) The change in the X axis refers to the pixel value to the right of the current pixel (X plus 1) minus the pixel value to the left of the current pixel (X minus 1). (2) The change in the Y axis is the pixel value below the current pixel (Y plus 1) minus the pixel value above the current pixel (Y minus 1). By calculating these two components to form a two-dimensional vector, the image gradient of the pixel point (i.e., feature point p) is obtained.
[0373] It should be noted that when the gradient direction of the feature point on the ellipse and the image is consistent, the contribution of the feature point will increase the weight, and the greater the gradient of the feature point on the image, the greater the weight, and accordingly, the greater the support, and the better the ellipse fitting effect.
[0374] The process of S3063-S3067 is repeated until the changes of all feature points are less than the preset change threshold, and the ellipse with the highest support is obtained.
[0375] The following is an example to illustrate the cycle process of S3063-S3067 and the process of ellipse fitting.
[0376] First cycle: Calculate the distance error between all feature points of the iris edge and the initialized ellipse. If the distance error of a feature point among all feature points of the iris edge is greater than the error threshold, then remove the feature point. In this way, by judging the distance error, the feature point set Y1 of the iris edge can be screened out from all feature points of the iris edge, and the number of feature points removed can be recorded. Then, determine whether the number of feature points removed from the feature point set A1 is less than or equal to the removal threshold. When the change of the feature point set Y1 is greater than the removal threshold, perform ellipse fitting based on the feature point set Y1 of the iris edge to obtain the ellipse Z1, and calculate the support of the feature point set Y1 of the iris edge based on the distance and gradient from the feature point set Y1 of the iris edge to the ellipse Z1 to obtain the support S1.
[0377] Second cycle: Calculate the distance error between the feature point set Y1 of the iris edge and the ellipse Z1. If the distance error of a feature point in the feature point set Y1 of the iris edge is greater than the error threshold, then the feature point is removed. In this way, the feature point set Y2 of the iris edge is screened out from the feature point set Y1 of the iris edge by judging the distance error, and the number of feature points removed is recorded. Then, it is determined whether the number of feature points removed from the feature point set Y2 is less than or equal to the removal threshold. When the change of the feature point set Y2 is greater than the removal threshold, an ellipse is fitted based on the feature point set Y2 of the iris edge to obtain the ellipse Z2, and the support of the feature point set Y2 of the iris edge is calculated based on the distance and gradient from the feature point set Y2 of the iris edge to the ellipse Z2 to obtain the support S2.
[0378] The third cycle: Calculate the distance error between the feature point set Y2 of the iris edge and the ellipse Z2. If the distance error of a feature point in the feature point set Y2 of the iris edge is greater than the error threshold, then remove the feature point. In this way, by judging the distance error, the feature point set Y3 of the iris edge is screened out from the feature point set Y2 of the iris edge, and the number of feature points removed is recorded. Then, it is determined whether the number of feature points removed from the feature point set Y2 is less than or equal to the removal threshold.
[0379] When the number of feature points to be eliminated in the feature point set Y3 is less than or equal to the elimination threshold, the support S1 and the support S2 are compared. If the support S2 is greater than the support S1, the ellipse Z2 corresponding to the support S2 is output.
[0380] In the embodiment of the present application, a mask 1 of the iris region is generated according to the ellipse with the highest support.
[0381] S307: finding the intersection of the mask 1 of the iris region and the mask of the eye region to obtain the mask 2 of the iris region.
[0382] S308, fine-tune the edge of the mask 2 of the iris area to obtain the mask 3 of the iris area.
[0383] Alternatively, if Fig.26 As shown, the mask of the iris region is intersected with the mask of the eye region obtained by face analysis to obtain the mask 1 of the iris region, and then the edge of the mask 2 of the iris region is fine-tuned to obtain the mask 3 of the iris region.
[0384] In the embodiment of the present application, the edge of the mask 2 of the iris region can be fine-tuned by using a graph cut algorithm to obtain a mask 3 of the iris region. Image segmentation refers to the technology and process of dividing an image into regions with different characteristics and extracting objects of interest. It is a key step from image processing to image analysis. The basic idea of image segmentation is that similar and close pixels should be divided into the same region.
[0385] It should be noted that the present application uses an image segmentation algorithm to fine-tune the extracted rough iris region, so that the final extracted iris region is more accurate.
[0386] S308, outputting mask 3 of the iris area.
[0387] Through the above steps, the human eye iris cutout processing is completed. After the human eye iris cutout processing, the obtained materials include: the human eye sclera area and iris area of the low-quality portrait, and the human eye sclera area and iris area of the high-quality portrait with abnormal eye light.
[0388] It should be noted that the present application performs ellipse fine-tuning through repeated iterations: all feature points are used to fit the ellipse, and the feature points are screened according to the distance from the feature points to the ellipse and the gradient of the feature points, so as to improve the accuracy of the ellipse fitting.
[0389] It should be noted that the iris feature extraction method provided in the embodiments of the present application can be applied to user photo-taking scenarios. For example, the above embodiments are described by taking the automatic processing of portrait eye light by an electronic device in a user photo-taking scenario as an example. Specifically, after the user triggers the photo-taking, the background of the electronic device identifies whether the original image meets certain conditions (such as the original image contains a face and the image quality is lower than a certain threshold), and if the original image meets certain conditions, the background automatically executes the iris feature extraction method provided in the present application, so that the portrait eye light in the final generated photo is normal or does not contain "abnormal" eye light, correcting the defect of unnatural eye light caused by full face enhancement through the face enhancement depth model.
[0390] It can be understood that in actual implementation, the application scenarios of the iris feature extraction method provided by the embodiment of the present application include but are not limited to user photo-taking scenarios. For example, the iris feature extraction method can also be applied to gallery photo editing scenarios. For example, the gallery application of the electronic device stores the original image and the high-quality portrait obtained by processing the face enhancement depth model. In the photo editing interface of the gallery of the electronic device, in response to the user operation, the electronic device retrieves the user-selected photo from the album or gallery and executes the iris feature extraction method provided by the present application. The portrait eye light in the processed photo is normal or does not contain "abnormal" eye light, which corrects the unnatural defect of eye light caused by full face enhancement through the face enhancement depth model, thereby meeting the user's demand for the fidelity of the face area in shooting scenarios such as low-light shooting and telephoto shooting, which require algorithm-enhanced low-quality portraits.
[0391] The iris feature extraction method provided in the embodiment of the present application can also be applied to eye light editing of portraits in mobile phone photo galleries, eye light replacement of specified images, replacement of specified eye light templates, eye light enhancement / weakening / removal, etc.
[0392] For example, in the photo editing interface of the mobile phone gallery, one or more eye light templates can be provided for eye light replacement, or two pictures can be directly used for replacement; eye light enhancement / weakening controls, eye light replacement controls, and eye light removal controls can also be provided. Users can select one or more photos in the gallery according to their personal needs, and click different function controls to trigger the phone to edit the eye light of the portrait in the photo, such as eye light enhancement / weakening / replacement / removal.
[0393] It should be noted that the iris feature extraction method provided in the embodiment of the present application can be used not only to correct the eye light of a portrait in a photo, but also to modify the eye light of an animal in a photo.
[0394] It should also be noted that in the embodiments of the present application, "greater than" can be replaced by "greater than or equal to", "less than or equal to" can be replaced by "less than", or "greater than or equal to" can be replaced by "greater than", and "less than" can be replaced by "less than or equal to".
[0395] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the protection scope of this application.
[0396] The above mainly describes the solution provided by the embodiment of the present application from the perspective of method steps. It is understandable that, in order to achieve the above functions, the electronic device implementing the method includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should be aware that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of the present application.
[0397] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other feasible division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.
[0398] Fig. 27 The schematic block diagram of the iris feature extraction device 400 provided in the embodiment of the present application. The device 400 can be used to perform the actions performed by the electronic device in the above method embodiment. The device 400 includes an image acquisition unit 410, an iris recognition unit 420, a feature point extraction unit 430 and an image processing unit 440.
[0399] An image acquisition unit 410 is used to acquire a first image and a second image when a preset condition is met, wherein the second image is an image obtained by applying a facial enhancement depth model to the first image;
[0400] An iris recognition unit 420 is used to perform feature recognition on the image to be processed and determine the iris area; the image to be processed is the first image or the second image;
[0401] A feature point extraction unit 430, used to extract feature points at the edge of the iris region in a polar coordinate system;
[0402] The image processing unit 440 is configured to:
[0403] Calculating the distance error between each feature point of the G feature points and the first ellipse; wherein the G feature points are feature points extracted from the edge of the iris region for the first time, and the first ellipse is an ellipse fitted based on the feature points extracted from the edge of the iris region for the first time; or the G feature points are feature points remaining after being eliminated last time, and the first ellipse is an ellipse fitted based on the feature points remaining after being eliminated last time;
[0404] Eliminate K feature points from the G feature points, and the distance error between each feature point in the K feature points and the first ellipse is greater than a preset error threshold;
[0405] When K is greater than a preset rejection threshold, a second ellipse is obtained by fitting according to the GK feature points, and the support corresponding to the second ellipse is calculated; wherein the GK feature points and the second ellipse are used together for the next calculation of the distance error between the feature points and the ellipse;
[0406] Circularly calculating the distance error between the feature point and the ellipse and eliminating the feature point according to the distance error, fitting to obtain multiple ellipses, and calculating the support corresponding to each of the multiple ellipses;
[0407] When the number of removed feature points is less than or equal to the preset removal threshold, obtaining a target ellipse with the largest support among the multiple ellipses;
[0408] The image features corresponding to the target ellipse are extracted from the image to be processed as the image features of the iris area.
[0409] Through the iris feature extraction device provided by the embodiment of the present application, after the iris area in the image is identified, the feature points at the edge of the iris area are extracted in the polar coordinate system, the feature points are first screened according to the result of face analysis and then the ellipse is fitted, and then a part of the feature points are eliminated according to the distance error between the feature points and the ellipse; then the ellipse is fitted again according to the feature points remaining after the elimination and the corresponding support is calculated. Then the distance error is calculated again according to the feature points remaining after the elimination and the fitted ellipse. The distance error is calculated cyclically and the feature points are eliminated according to the distance error, thereby fitting multiple ellipses and calculating multiple support. When the number of eliminated feature points is less than or equal to the threshold, the image features of the iris area are extracted according to the ellipse with the largest support. The present application screens the feature points according to the distance error from the feature points to the ellipse, and iterates repeatedly to fine-tune the ellipse to improve the accuracy of the ellipse fitting, so as to improve the accuracy of the iris extraction, which is helpful for the subsequent accurate processing of the eye light.
[0410] According to the embodiment of the present application, the device 400 may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of the units in the device 400 are respectively for implementing the corresponding processes of the method, which will not be repeated here for the sake of brevity.
[0411] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments.
[0412] The present application also provides an electronic device, which includes a chip, and the chip is used to read and execute a computer program or instruction stored in a memory, so that the method in each embodiment is executed.
[0413] This embodiment also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the iris feature extraction method in the above-mentioned embodiment.
[0414] This embodiment further provides a computer program product, wherein the computer-readable storage medium stores program codes. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the iris feature extraction method in the above-mentioned embodiment.
[0415] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the iris feature extraction method in the above-mentioned method embodiments.
[0416] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.
[0417] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0418] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for extracting iris features, characterized in that: include: When a preset condition is met, a first image and a second image containing facial features are obtained, where the second image is an image obtained by applying a facial enhancement depth model to the first image; Perform feature recognition on the image to be processed to determine the iris area; the image to be processed is the first image or the second image; Extracting feature points at the edge of the iris region in a polar coordinate system; Calculating the distance error between each feature point of the G feature points and the first ellipse; wherein the G feature points are feature points extracted from the edge of the iris region for the first time, and the first ellipse is an ellipse fitted based on the feature points extracted from the edge of the iris region for the first time; or the G feature points are feature points remaining after being eliminated last time, and the first ellipse is an ellipse fitted based on the feature points remaining after being eliminated last time; Eliminate K feature points from the G feature points, and the distance error between each feature point in the K feature points and the first ellipse is greater than a preset error threshold; When K is greater than a preset rejection threshold, a second ellipse is obtained by fitting according to the GK feature points, and the support corresponding to the second ellipse is calculated; wherein the GK feature points and the second ellipse are used together for the next calculation of the distance error between the feature points and the ellipse; Circularly calculating the distance error between the feature point and the ellipse and eliminating the feature point according to the distance error, fitting to obtain multiple ellipses, and calculating the support corresponding to each of the multiple ellipses; When the number of removed feature points is less than or equal to the preset removal threshold, obtaining a target ellipse with the largest support among the multiple ellipses; The image features corresponding to the target ellipse are extracted from the image to be processed as the image features of the iris area.
2. The method according to claim 1, characterized in that The preset condition is: in response to a first operation of a shutter button by a user, the first image is captured, the first image includes facial features, and the image quality of the first image is lower than or equal to a preset image quality; Alternatively, the preset condition is: receiving a second operation of the user on the second image in the gallery, the second image includes facial features and eye light features, and the second operation is used to adjust the eye light features.
3. The method according to claim 1 or 2, characterized in that: The calculating the support corresponding to the second ellipse includes: The support corresponding to the second ellipse is calculated according to the gradient of the second ellipse at the GK feature points and the gradient of the image to be processed at the GK feature points.
4. The method according to claim 3, characterized in that Calculating the support corresponding to the second ellipse according to the gradient of the second ellipse at the GK feature points and the gradient of the image to be processed at the GK feature points includes: The support corresponding to the second ellipse is calculated using the following equation: Where S represents the support corresponding to the second ellipse, represents the gradient of the second ellipse at the feature point p, represents the gradient of the image to be processed at the feature point p, where p is each feature point among the GK feature points.
5. The method according to any one of claims 1 to 4, characterized in that After acquiring the target ellipse with the largest support among the multiple ellipses, and before extracting the image features corresponding to the target ellipse from the image to be processed as the image features of the iris area, the method further includes: Generating a first mask of the iris region according to the target ellipse; Obtaining an intersection between the first mask and a mask of the eye region to obtain a second mask of the iris region; Using an image segmentation algorithm to fine-tune the edge of the second mask of the iris region to obtain a third mask of the iris region; The step of extracting image features corresponding to the target ellipse from the image to be processed as image features of the iris region includes: extracting image features from the image to be processed according to a third mask of the iris region as image features of the iris region.
6. The method according to any one of claims 1 to 5, characterized in that After extracting the feature points at the edge of the iris region in the polar coordinate system, the method further includes: According to the mask of the eye area, removing the feature points covered by the upper and lower eyelids from the feature points at the edge of the iris area; According to the length of the iris radius, the feature points located at the periphery of the iris are removed from the feature points at the edge of the iris area.
7. The method according to any one of claims 1 to 6, characterized in that The step of performing feature recognition on the image to be processed and determining the iris area includes: Input the image to be processed into a first preset network model, and output a mask of the eye area and key points of the eye center; wherein the first preset network model is a network model for analyzing faces and detecting key points of faces; determining a first palpebral fissure length based on the mask of the eye region; Determine the iris radius length according to the first palpebral fissure length and a preset first corresponding relationship; wherein the first corresponding relationship includes a corresponding relationship between the first palpebral fissure length and the iris radius length; The iris region is determined in the image to be processed according to the iris radius length and the eye center key point; wherein the iris region is a circular region with the eye center key point as the center and the iris radius length as the radius.
8. The method according to any one of claims 1 to 7, characterized in that After performing feature recognition on the image to be processed and determining the iris area, the method further includes: The contrast of the iris region is enhanced to highlight the edge features of the iris region.
9. The method according to any one of claims 1 to 8, characterized in that After performing feature recognition on the image to be processed to determine the iris area and before extracting feature points at the edge of the iris area in a polar coordinate system, the method further includes: Converting the iris region from a rectangular coordinate system to the polar coordinate system; Among them, in the rectangular coordinate system, the coordinates of the pixel points in the iris area are expressed as (x, y), and the value ranges of x and y are both [0, N]; in the polar coordinate system, the coordinates of the pixel points in the iris area are expressed as (ρ, θ), the value range of the polar radius ρ is [0, N / 2], and the value range of the polar angle θ is [0, 360).
10. The method according to any one of claims 1 to 9, characterized in that The step of extracting feature points at the edge of the iris region in a polar coordinate system includes: In the polar coordinate system, extracting linear features of the image to be processed by a linear filter; By performing peak detection on the linear features of the image to be processed, each feature point with a polar radius ρ of N / 2 and a polar angle θ in the range of [0, 360) is obtained, and the feature points at the edge of the iris area are obtained.
11. The method according to any one of claims 1 to 10, characterized in that After extracting the image features corresponding to the target ellipse from the image to be processed as the image features of the iris area, the method further includes: Separating a first background region and a first highlight region from the eye region of the first image, and separating a second background region and a second highlight region from the eye region of the second image; In the case where there is a catch light feature in the eye area, enhancing the image feature of the first highlight area to obtain an enhanced catch light feature; and background fusion is performed on the enhanced catch light feature and the image feature of the second background area to obtain a third image; When the catch light feature does not exist in the eye region, a fourth image is obtained according to the image feature of the second background region.
12. The method according to claim 11, characterized in that The step of separating a first background area and a first highlight area from the eye area of the first image comprises: Upsampling the eye region of the first image to obtain an enlarged eye region; Separating the first background region and the first highlight region from the enlarged eye region based on a highlight removal algorithm; Downsampling the first background area and the first highlight area respectively; The reduction ratio of the downsampling is the inverse of the magnification ratio of the upsampling.
13. The method according to claim 12, characterized in that The step of upsampling the eye region of the first image to obtain an enlarged eye region includes: Performing anti-aliasing upsampling and filtering denoising on the eye region of the first image; When the enlarged size of the eye region of the first image is smaller than a preset size threshold, performing anti-aliasing upsampling and filtering denoising on the eye region of the first image again; When the enlarged size of the eye region of the first image is greater than or equal to the preset size threshold, the enlarged eye region is obtained.
14. The method according to any one of claims 11 to 13, characterized in that The step of enhancing the image feature of the first highlight area to obtain an enhanced catch light feature includes: Acquire attribute parameters of the first highlight area, where the attribute parameters include size, shape, average brightness and / or brightest point brightness; determining a first enhancement parameter according to the attribute parameter of the first highlight area; The first enhancement parameter is used to enhance the image feature of the first highlight area to obtain an enhanced eye light feature.
15. The method according to any one of claims 11 to 14, characterized in that The step of performing background fusion on the enhanced catch light feature and the image feature of the second background area to obtain a third image includes: Determining an affine transformation matrix according to edge key points of the first background area and edge key points of the second background area; Performing an affine transformation on the enhanced eye light feature according to the affine transformation matrix to obtain an enhanced eye light feature aligned with the position of the second background area; Background fusion is performed on the aligned enhanced catch light features and the image features of the second background area to obtain the third image.
16. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory, wherein the processor is configured to execute the computer program so that the electronic device implements the method according to any one of claims 1 to 15.
17. A chip comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: The processor is configured to execute the computer program to implement the method according to any one of claims 1 to 15.
18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 15.
Citation Information
Patent Citations
Ellipse-detection-based iris detection method
CN106485210A
Iris recognition method and terminal
CN107292242A
Iris recognition method and apparatus, storage medium, and device
WO2023088069A1
