Iris feature extraction methods, electronic devices and storage media
By extracting edge feature points of the iris region in polar coordinates and repeatedly iterating ellipse fitting, the problem of iris feature extraction in low light and long-distance shooting is solved, achieving accurate extraction of the iris region and natural correction of catchlight, thus improving image quality.
Patent Information
- Application Number
- CN202310644364.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-05-31
AI Technical Summary
In low-light and long-distance shooting scenarios, iris images are easily affected by noise and distortion. Existing technologies struggle to accurately extract iris features, resulting in unnatural processing of catchlights.
In polar coordinates, edge feature points of the iris region are extracted. Feature points are selected by ellipse fitting and distance error. The ellipse fitting is repeatedly iterated and optimized. Combined with image segmentation algorithm, the iris region is fine-tuned to improve the accuracy of iris extraction.
It enables accurate extraction of iris features and correction of catchlight in low-light and long-distance shooting scenarios, thereby improving the naturalness of image quality and user experience.
Smart Images

Figure CN119067861B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to an iris feature extraction method, electronic device, and storage medium. Background Technology
[0002] In techniques for correcting abnormal catchlights in images, since catchlights are usually displayed in the iris region, it is often necessary to extract iris features and then correct the catchlights in the region where the iris features are located.
[0003] However, due to the small size of the iris and its susceptibility to noise interference from light spots, eyelids, eyelashes, etc., and the fact that iris images are easily distorted in low-light and long-distance shooting scenarios due to the different shapes of users, how to extract iris features is the key and difficult point in eye catchlight processing. Summary of the Invention
[0004] This application provides an iris feature extraction method, electronic device, and storage medium, which can accurately extract iris image features and help to process eye light more accurately.
[0005] In a first aspect, this application provides an iris feature extraction method, the method comprising:
[0006] When preset conditions are met, a first image and a second image containing facial features are acquired. The first image contains facial features, and the second image is an image obtained by applying a facial enhancement depth model to the first image.
[0007] The image to be processed is subjected to feature recognition to determine the iris region; the image to be processed is either the first image or the second image.
[0008] Extract feature points at the edge of the iris region in polar coordinates;
[0009] Calculate the distance error between each of the G feature points and the first ellipse; wherein the G feature points are the feature points extracted for the first time from the edge of the iris region, and the first ellipse is an ellipse fitted based on the feature points extracted for the first time from the edge of the iris region; or the G feature points are the feature points remaining after the previous removal, and the first ellipse is an ellipse fitted based on the feature points remaining after the previous removal.
[0010] K feature points are removed from the G feature points, and the distance error between each of the K feature points and the first ellipse is greater than a preset error threshold.
[0011] When K is greater than the preset rejection threshold, a second ellipse is obtained by fitting GK feature points, and the support corresponding to the second ellipse is calculated; wherein, the GK feature points and the second ellipse are used together to calculate the distance error between the feature points and the ellipse in the next calculation.
[0012] The distance error between feature points and ellipses is calculated iteratively, and feature points are removed based on the distance error. Multiple ellipses are then fitted, and the support of each of the multiple ellipses is calculated.
[0013] When the number of removed feature points is less than or equal to the preset removal threshold, the target ellipse with the highest support among the multiple ellipses is obtained.
[0014] The image features corresponding to the target ellipse are extracted from the image to be processed and used as the image features of the iris region.
[0015] The iris feature extraction method provided in this application, after identifying the iris region in an image, extracts feature points at the edge of the iris region in a polar coordinate system. First, feature points are filtered based on the face analysis results, then ellipse fitting is performed. Next, some feature points are removed based on the distance error between the feature points and the ellipse. Then, an ellipse is fitted again based on the remaining feature points, and the corresponding support is calculated. The distance error is then calculated again based on the remaining feature points and the fitted ellipse. This process of calculating the distance error and removing feature points based on the distance error is repeated, resulting in multiple fitted ellipses and multiple support values. When the number of removed feature points is less than or equal to a threshold, the image features of the iris region are extracted based on the ellipse with the highest support. This application performs ellipse fine-tuning through repeated iterations: ellipse fitting is performed using all feature points, and feature points are filtered based on the distance from the feature points to the ellipse, improving the accuracy of ellipse fitting and thus improving the accuracy of iris extraction.
[0016] It should be noted that when the first ellipse is an ellipse fitted based on the feature points initially extracted from the edge of the iris region, the first ellipse can be called the initialization ellipse.
[0017] It should be noted that, through continuous filtering of distance errors, the number of feature points N removed becomes smaller and smaller. To improve computational efficiency, a removal threshold can be preset to determine whether the number of feature points removed in the current iteration meets the requirements. For example, the removal threshold can be 2. The specific value of the removal threshold can be set according to actual usage needs, and this embodiment does not limit it.
[0018] The following example illustrates the process of loop calculation.
[0019] First loop: Calculate the distance error between all feature points on the iris edge and the initialized ellipse. If the distance error of a feature point in the iris edge is greater than the error threshold, then that feature point is removed. By judging the distance error, a feature point set Y1 for the iris edge can be selected from all feature points on the iris edge, and the number of removed feature points is recorded. Then, it is determined whether the number of removed feature points in feature point set Y1 is less than or equal to the removal threshold. If the change in feature point set Y1 is greater than the removal threshold, an ellipse is fitted based on the feature point set Y1 of the iris edge to obtain ellipse Z1. Based on the distance and gradient from the feature point set Y1 to ellipse Z1, the support of the feature point set Y1 of the iris edge is calculated to obtain the support S1.
[0020] The second iteration calculates the distance error between the feature point set Y1 of the iris edge and the ellipse Z1. If the distance error of a feature point in the feature point set Y1 is greater than the error threshold, that feature point is removed. This distance error determination filters the feature point set Y2 of the iris edge from the feature point set Y1, and the number of removed feature points is recorded. Then, it is determined whether the number of removed feature points in feature point set Y2 is less than or equal to the removal threshold. If the change in feature point set Y2 is greater than the removal threshold, an ellipse is fitted based on the feature point set Y2 of the iris edge to obtain ellipse Z2. The support of the feature point set Y2 of the iris edge is calculated based on the distance and gradient from the feature point set Y2 to ellipse Z2, yielding the support S2.
[0021] The third iteration calculates the distance error between the feature point set Y2 of the iris edge and the ellipse Z2. If the distance error of a feature point in the feature point set Y2 of the iris edge is greater than the error threshold, then that feature point is removed. By judging the distance error, a feature point set Y3 of the iris edge is selected from the feature point set Y2, and the number of removed feature points is recorded. Then, it is determined whether the number of removed feature points in the feature point set Y2 is less than or equal to the removal threshold.
[0022] If the number of feature points removed from feature point set Y3 is less than or equal to the removal threshold, compare the support S1 and support S2. If support S2 is greater than support S1, then output the ellipse Z2 corresponding to support S2.
[0023] It should be noted that this application performs ellipse fine-tuning through repeated iterations: ellipse fitting is performed using all feature points, and feature points are selected based on their distance from the ellipse and the gradient of the feature point, thereby improving the accuracy of ellipse fitting.
[0024] Through this proposed solution, by repeatedly iterating and continuously optimizing the ellipse fitting, iris image features can be accurately extracted.
[0025] In some possible implementations, the preset condition is: in response to a user's first operation on the shutter button, the first image is captured, the first image includes facial features, and the image quality of the first image is lower than or equal to a preset image quality.
[0026] In some possible implementations, the preset condition is: receiving a second operation from a user on the second image in the image library, the second image including facial features and eye catchlight features, the second operation being used to adjust the eye catchlight features.
[0027] In some possible implementations, calculating the support corresponding to the second ellipse includes: calculating the support corresponding to the second ellipse based on the gradient of the second ellipse at the GK feature points and the gradient of the image to be processed at the GK feature points.
[0028] In some possible implementations, calculating the support of the second ellipse based on the gradient of the second ellipse at the GK feature points and the gradient of the image to be processed at the GK feature points includes:
[0029] The support of the second ellipse is calculated using the following equation:
[0030]
[0031] Where S represents the support corresponding to the second ellipse. This represents the gradient of the second ellipse at feature point p. This represents the gradient of the image to be processed at feature point p, where p is any one of the GK feature points.
[0032] It should be noted that when the gradient direction of a feature point is consistent with that of the ellipse and the image, the contribution of the feature point will increase the weight. Furthermore, the larger the gradient of the feature point in the image, the larger the weight, and consequently, the greater the support and the better the ellipse fitting effect.
[0033] In some possible implementations, after obtaining the target ellipse with the highest support among the plurality of ellipses, and before extracting the image features corresponding to the target ellipse from the image to be processed as the image features of the iris region, the method further includes:
[0034] A first mask for the iris region is generated based on the target ellipse; the intersection between the first mask and the mask for the eye region is calculated to obtain a second mask for the iris region; the edges of the second mask for the iris region are fine-tuned using an image segmentation algorithm to obtain a third mask for the iris region.
[0035] Image segmentation (Graph Cut) algorithms refer to the techniques and processes of dividing an image into regions with distinct characteristics and extracting the target of interest. It is a crucial step in the transition from image processing to image analysis. The basic idea of image segmentation is that similar and closely spaced pixels should be grouped into the same region.
[0036] The proposed solution uses an image segmentation algorithm to fine-tune the extracted coarse iris region, resulting in a more accurate final extracted iris region.
[0037] The step of extracting the image features corresponding to the target ellipse from the image to be processed as the image features of the iris region includes: extracting image features from the image to be processed according to the third mask of the iris region as the image features of the iris region.
[0038] In some possible implementations, after extracting feature points at the edge of the iris region in polar coordinates, the method further includes: removing feature points covered by the upper and lower eyelids from the feature points at the edge of the iris region based on a mask of the eye region; and removing feature points located outside the iris region from the feature points at the edge of the iris region based on a roughly estimated iris radius length.
[0039] In some possible implementations, the step of performing feature recognition on the image to be processed to determine the iris region includes:
[0040] The image to be processed is input into a first preset network model, which outputs a mask for the eye region and key points at the center of the eye; wherein, the first preset network model is a network model used for parsing the face and detecting facial key points;
[0041] The length of the first palpebral fissure is determined based on the mask of the eye region;
[0042] The iris radius length is determined based on the first palpebral fissure length and a preset first correspondence relationship; wherein, the first correspondence relationship includes the correspondence between the first palpebral fissure length and the iris radius length;
[0043] Based on the iris radius and the eye center key point, the iris region is determined in the image to be processed; wherein the iris region is a circular area centered on the eye center key point and with the iris radius as the radius.
[0044] In some possible implementations, after performing feature recognition on the image to be processed to determine the iris region, the method further includes: enhancing the contrast of the iris region to highlight the edge features of the iris region.
[0045] The proposed solution enhances the contrast of the iris region after it has been determined, thereby highlighting edge features and facilitating the detection and extraction of feature points in the iris region.
[0046] In some possible implementations, after performing feature recognition on the image to be processed and determining the iris region, and before extracting feature points at the edge of the iris region in polar coordinates, the method further includes: transforming the iris region from a Cartesian coordinate system to the polar coordinate system. Wherein, in the Cartesian coordinate system, the pixel coordinates in the iris region are represented as (x, y), where x and y both range from [0, N]; in the polar coordinate system, the pixel coordinates in the iris region are represented as (ρ, θ), where ρ ranges from [0, N / 2] and θ ranges from [0, 360].
[0047] It should be noted that, since the iris edge region is curved, feature point extraction at the iris edge is relatively difficult. Therefore, in this embodiment, the iris region is transformed to a polar coordinate system centered at (N / 2, N / 2) to obtain a 360*N / 2 polar coordinate image, thereby transforming the curved features into straight line features. Here, 360 represents 360 degrees.
[0048] Assume the iris region consists of N*N pixels. In a Cartesian coordinate system, the center point of the iris region can be represented as (x, y), where x = N / 2 and y = N / 2. Correspondingly, in a polar coordinate system, the coordinates of the center point of the iris region can be represented as (ρ, θ), where the radius coordinate ρ = 0 and the angular coordinate θ = 0. That is, the pole (0, 0) of the polar coordinate system is the center point of the iris region.
[0049] In some possible implementations, the step of extracting feature points at the edge of the iris region in polar coordinates includes: extracting linear features of the image to be processed by a linear filter in polar coordinates; obtaining each feature point with a polar radius ρ of N / 2 and a polar angle θ in the range of [0, 360) by performing peak detection on the linear features of the image to be processed, thereby obtaining the feature points at the edge of the iris region.
[0050] In the embodiments of this application, a linear filter is used in the feature point extraction process, which can extract feature points of the iris region more accurately.
[0051] The following example illustrates the process of sequentially acquiring feature points in each direction.
[0052] When θ is 0, feature points in the 0-angle direction are obtained based on the image features in the 0-angle direction. Then, when θ is 1, feature points in the 1-angle direction are obtained based on the image features in the 1-angle direction. Similarly, when θ is 359, feature points in the 359-angle direction are obtained based on the image features in the 359-angle direction. Since the edge changes of the iris are significant, feature points in each angular direction can optionally be detected and extracted using peak detection. Specifically, peak detection can be performed on the eye region in polar coordinates to obtain feature points (ρ, θ) in each angular direction.
[0053] In some possible implementations, after extracting the image features corresponding to the target ellipse from the image to be processed as image features of the iris region, the method further includes:
[0054] A first background region and a first highlight region are separated from the eye region of the first image, and a second background region and a second highlight region are separated from the eye region of the second image;
[0055] If catchlight features exist in the eye region, the image features of the first highlight region are enhanced to obtain enhanced catchlight features; and the enhanced catchlight features are then fused with the image features of the second background region to obtain a third image.
[0056] If the eye light feature is not present in the eye region, a fourth image is obtained based on the image features of the second background region.
[0057] In some possible implementations, separating the first background region and the first highlight region from the eye region of the first image includes: upsampling the eye region of the first image to obtain an enlarged eye region; separating the first background region and the first highlight region from the enlarged eye region based on a highlight removal algorithm; and downsampling the first background region and the first highlight region respectively. The downsampling reduction ratio is the reciprocal of the upsampling magnification ratio.
[0058] In some possible implementations, the step of upsampling the eye region of the first image to obtain an enlarged eye region includes: performing anti-aliasing upsampling and filtering noise reduction on the eye region of the first image; if the enlarged size of the eye region of the first image is less than a preset size threshold, performing anti-aliasing upsampling and filtering noise reduction on the eye region of the first image again; and obtaining the enlarged eye region if the enlarged size of the eye region of the first image is greater than or equal to the preset size threshold.
[0059] In some possible implementations, the enhancement processing of the image features of the first highlight region to obtain enhanced catchlight features includes:
[0060] Obtain the attribute parameters of the first highlight region, including size, shape, average brightness, and / or the brightness of the brightest point;
[0061] The first enhancement parameter is determined based on the attribute parameters of the first highlight region;
[0062] Using the first enhancement parameter, the image features of the first highlight region are enhanced to obtain the enhanced catchlight features.
[0063] In some possible implementations, the step of fusing the enhanced gaze features with the image features of the second background region to obtain a third image includes:
[0064] Based on the edge key points of the first background region and the edge key points of the second background region, an affine transformation matrix is determined; the enhanced catchlight feature is subjected to an affine transformation based on the affine transformation matrix to obtain an enhanced catchlight feature aligned with the position of the second background region; the aligned enhanced catchlight feature is then fused with the image features of the second background region to obtain the third image.
[0065] It should be noted that the iris feature extraction method provided in this application can be applied to user photography scenarios. For example, the above embodiment is illustrated by the example of an electronic device automatically processing the catchlight of a person's eyes in a user photography scenario. Specifically, after the user triggers the photo, the background of the electronic device identifies whether the original image meets certain conditions (such as the original image containing a face and the image quality being below a certain threshold). If the original image meets certain conditions, the background automatically executes the iris feature extraction method provided in this application, so that the catchlight of the person's eyes in the final generated photo is normal or does not contain "abnormal" catchlight, thus correcting the unnatural catchlight caused by full-face enhancement through a face enhancement depth model.
[0066] In practical implementation, the iris feature extraction method provided in this application embodiment can be applied to scenarios including but not limited to user photography scenarios. For example, the iris feature extraction method can also be applied to photo editing scenarios in a photo library. For instance, the photo library application of an electronic device stores the original image and a high-quality portrait processed by a face enhancement depth model. In the photo editing interface of the electronic device's photo library, in response to user operation, the electronic device retrieves the photo selected by the user from the album or photo library and executes the iris feature extraction method provided in this application. The processed photo has normal or no "abnormal" catchlight in the eyes, correcting the unnatural catchlight caused by full-face enhancement through a face enhancement depth model. This satisfies the user's need for facial region fidelity in shooting scenarios such as low-light shooting and telephoto shooting, where low-quality portrait photos need to be enhanced by algorithms.
[0067] It should be noted that the algorithm scenarios involved in this application are mainly those in which mobile phones capture low-quality images, including but not limited to low-light and telephoto scenarios, and may also be any other scenarios that may capture low-quality images.
[0068] Secondly, this application provides an iris feature extraction apparatus, which includes units for performing the method described in the first aspect above. This apparatus is capable of performing the method described in the first aspect above, and the relevant descriptions of the units within this apparatus are provided in the description of the first aspect above; for the sake of brevity, they will not be repeated here.
[0069] The method described in the first aspect above can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.
[0070] Thirdly, this application provides an electronic device, which includes a processor, a computer program or instructions stored in a memory, wherein the processor is used to execute the computer program or instructions to cause the method in the first aspect to be performed.
[0071] Fourthly, this application provides a computer-readable storage medium having a computer program (also referred to as instructions or code) stored thereon for implementing the method of the first aspect. For example, when the computer program is executed by a computer, it enables the computer to perform the method of the first aspect.
[0072] Fifthly, this application provides a chip including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof. Optionally, the chip further includes a memory connected to the processor via a circuit or wire.
[0073] Sixthly, this application provides a chip system including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof. Optionally, the chip system further includes a memory connected to the processor via a circuit or wire.
[0074] In a seventh aspect, this application provides a computer program product comprising a computer program (also referred to as instructions or code), which, when executed by an electronic device, causes the electronic device to implement the method in the first aspect.
[0075] It is understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description
[0076] Figure 1 This is a schematic diagram of the sclera and iris in the eye region involved in the embodiments of this application;
[0077] Figure 2 This illustration shows how the catchlight in the eyes of a user-captured image becomes distorted after being processed by a face enhancement depth model in a related technology. Figure 1 ;
[0078] Figure 3 This illustration shows how the catchlight in the eyes of a user-captured image becomes distorted after being processed by a face enhancement depth model in a related technology. Figure 2 ;
[0079] Figure 4 A flowchart illustrating the iris feature extraction method provided in this application embodiment. Figure 1 ;
[0080] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;
[0081] Figure 6 A schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;
[0082] Figure 7 A flowchart illustrating the iris feature extraction method provided in this application embodiment. Figure 2 ;
[0083] Figure 8 A flowchart illustrating the iris feature extraction method provided in this application embodiment. Figure 3 ;
[0084] Figure 9 This is a schematic diagram of human eye iris matting in the iris feature extraction method provided in the embodiments of this application;
[0085] Figure 10 This is a schematic diagram of eye light / background separation in the iris feature extraction method provided in the embodiments of this application;
[0086] Figure 11 This is a schematic diagram of image upsampling in the iris feature extraction method provided in the embodiments of this application;
[0087] Figure 12 A schematic diagram illustrating the extraction of the highlight region of the eye using the iris feature extraction method provided in this application embodiment;
[0088] Figure 13 A schematic diagram illustrating upsampling, eye light / background separation, and downsampling in the iris feature extraction method provided in the embodiments of this application;
[0089] Figure 14 This is a schematic diagram illustrating the fusion of eye light and background in the iris feature extraction method provided in this application embodiment;
[0090] Figure 15 A schematic diagram illustrating the scaling, rotation, and translation forms of affine transformation in the iris feature extraction method provided in this application embodiment;
[0091] Figure 16 A schematic diagram illustrating the iris feature extraction method provided in this application for applying affine transformation to enhanced eye light features;
[0092] Figure 17 This is a schematic diagram of feathering processing in the iris feature extraction method provided in the embodiments of this application;
[0093] Figure 18 A schematic diagram of the implementation process of the iris feature extraction method provided in the embodiments of this application. Figure 1 ;
[0094] Figure 19 A schematic diagram of the implementation process of the iris feature extraction method provided in the embodiments of this application. Figure 2 ;
[0095] Figure 20 This is a schematic diagram of the iris matting process in the iris feature extraction method provided in the embodiments of this application;
[0096] Figure 21 This is a schematic diagram of the mask for the eye region, the key points at the center of the eye, and the iris region in the iris matting process of the iris feature extraction method provided in this application embodiment;
[0097] Figure 22 A flowchart illustrating the process of extracting feature points of the iris region using the iris feature extraction method provided in this application embodiment. Figure 1 ;
[0098] Figure 23A flowchart illustrating the process of extracting feature points of the iris region using the iris feature extraction method provided in this application embodiment. Figure 2 ;
[0099] Figure 24 A flowchart illustrating the process of selecting feature points in the iris region in the iris feature extraction method provided in this embodiment of the application;
[0100] Figure 25 This is a schematic diagram of the ellipse fitting process in the iris feature extraction method provided in the embodiments of this application;
[0101] Figure 26 This is a schematic diagram of the mask used to obtain the iris region in the iris feature extraction method provided in this application embodiment;
[0102] Figure 27 This is a schematic diagram of an iris feature extraction device provided in an embodiment of this application. Detailed Implementation
[0103] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0104] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.
[0105] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. In the description of embodiments in this application, unless otherwise stated, "multiple" means two or more; for example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.
[0106] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0107] Catchlight is a light source used in portrait photography to create a bright spot within the eyes. It typically produces a small, round point of light within the iris, making the eyes appear more expressive. For example... Figure 1 As shown, eye features can include the sclera and iris, with the iris area being the black part of the eye. Catchlight is typically displayed in the iris area.
[0108] Currently, portrait image quality taken by mobile phones in low light and at long distances shows a significant decline compared to good lighting and close-up shots. To improve the image quality of portraits taken in low light and at long distances, the mainstream industry approach is to apply an artificial intelligence (AI) depth model to the original image after the user triggers the shutter button. The enhanced photo is clearer and has richer facial details. For ease of explanation, this AI depth model for enhancing facial images will be referred to as the face enhancement depth model.
[0109] However, due to the low interpretability and weak controllability of deep models for facial image enhancement, there are frequent instances where enhanced catchlight features are misprocessed, resulting in unnatural-looking catchlights that fail to meet users' basic requirement for realism in photo capture. As one of the most prominent features of the face, changes in catchlights are easily noticeable to users; if the catchlight enhancement is unrealistic, it will significantly impact the overall facial appearance.
[0110] For example, if the original image has catchlights, a face enhancement depth model will enhance the catchlights, but the enhanced catchlights will not look natural. For instance, Figure 2 (a) shows a low-quality portrait (original image) taken by a mobile phone under conditions such as low light and long distance. The original image has poor imaging quality, with catchlights in the eye area. After the low-quality portrait is input into the face enhancement depth model, the face enhancement depth model performs face enhancement on the low-quality portrait and outputs a high-quality portrait. Figure 2 (b) shows a high-quality portrait with enhanced face using a face enhancement depth model. The catchlight in the eye area is enhanced, but the shape, size, and intensity of the catchlight are modified. Therefore, the enhanced catchlight is unnatural and does not meet the user's basic requirement for the realism of the photo function.
[0111] For another example, in an image without catchlights, a face enhancement depth model might add catchlights to the eye area, but the added catchlights may not look natural. For instance, Figure 3(a) shows a low-quality portrait (original image) taken by a mobile phone under conditions such as low light and long distance, in which there is no catchlight in the eye area. After the low-quality portrait is input into the face enhancement depth model, the face enhancement depth model performs face enhancement on the low-quality portrait and outputs a high-quality portrait. Figure 3 (b) shows a high-quality portrait with enhanced face using a face enhancement depth model. Compared to the original image, catchlights have been added to the eye area. However, since there were no catchlights originally, adding catchlights would modify the original expression of the face, making it look unnatural and failing to meet the user's basic requirement for the realism of the photo function.
[0112] In the above-mentioned techniques for correcting abnormal catchlights in images, since catchlights are usually displayed in the iris region, it is usually necessary to extract iris features and then correct the catchlights in the region where the iris features are located.
[0113] However, due to the small size of the iris and its susceptibility to noise interference from light spots, eyelids, eyelashes, etc., and the fact that iris images are easily distorted in low-light and long-distance shooting scenarios due to the different shapes of users, how to extract iris features is the key and difficult point in eye catchlight processing.
[0114] In view of the above-mentioned technical problems, this application provides an iris feature extraction method and electronic device, which can be implemented based on computer vision algorithms. Utilizing eye and catchlight information in low-quality portraits, it can controllably and naturally enhance the catchlight. In this application embodiment, through improvements at the underlying level of the mobile phone system, the catchlight in portraits of low-quality images can be controllably enhanced without any additional hardware, thereby improving the user experience.
[0115] This application's solution, after identifying the iris region in an image, extracts feature points at the edge of the iris region in polar coordinates. First, feature points are filtered based on the face analysis results, then an ellipse is fitted. Next, some feature points are removed based on the distance error between the feature points and the ellipse. Finally, an ellipse is fitted again based on the remaining feature points, and the corresponding support is calculated. This application filters feature points based on the distance error to the ellipse, iteratively fine-tuning the ellipse to improve the accuracy of ellipse fitting, thereby enhancing the accuracy of iris extraction and facilitating accurate processing of eye light.
[0116] To better understand the embodiments of this application, the following is combined with... Figure 4 A brief description of the embodiments of this application is provided below:
[0117] Figure 4 A simplified algorithm flow for the scheme in this application is shown. For example... Figure 4 As shown in the embodiments of this application, the catchlight in the eyes is enhanced in a controllable manner by performing image processing on low-quality portraits and high-quality portraits respectively.
[0118] Low-quality portraits can be photos containing facial features taken by a user pressing the camera shutter button on an electronic device under conditions such as low light and distance, or photos containing facial features stored in the electronic device's album or gallery. After the electronic device enhances the low-quality portrait using a preset face enhancement depth model, a high-quality portrait is obtained. In high-quality portraits, because the catchlights in the eyes appear unnatural after enhancement, the image can be described as a high-quality portrait with "abnormal" catchlights in the eyes after enhancement.
[0119] To controllably enhance catchlights in the eyes, this embodiment first employs the catchlight enhancement algorithm provided in this application, based on the low-quality catchlight features in low-quality portraits, to enhance the catchlights and obtain enhanced catchlight features. Then, the enhanced catchlight features replace the "abnormal" catchlight features in high-quality portraits, correcting the catchlight features and resulting in high-quality portraits with "normal" catchlights after enhancement. This satisfies the user's need for facial region fidelity in shooting scenarios such as low-light shooting and telephoto shooting, where algorithmic enhancement of low-quality portrait photos is required.
[0120] It should be noted that in the embodiments of this application, "normal" catchlights are defined in relation to the distorted effect of "abnormal" catchlights. For example, if the catchlights in a portrait are realistic and natural, they can be considered "normal"; or if a portrait does not contain "abnormal" catchlights, it can also be considered "normal".
[0121] This application focuses on the user's need for facial region fidelity in scenarios such as low-light shooting and telephoto shooting. It can controllably enhance the catchlight in the human eye and make up for the unnaturalness of the catchlight caused by full-face enhancement through AI deep model.
[0122] The above combination Figure 4 This paper describes a scheme to correct "abnormal" catchlights in high-quality portraits when the original image already has catchlights: The scheme identifies catchlights in the original image based on a low-quality portrait, then performs controllable enhancement on the catchlight features based on the low-quality portrait, and finally replaces the "abnormal" catchlight features in the high-quality portrait with the enhanced catchlight features. This corrects the "abnormal" catchlights in the high-quality portrait to "normal" catchlights, meeting the user's need for facial fidelity in shooting scenarios such as low-light shooting and telephoto shooting, where algorithms are needed to enhance low-quality portrait photos.
[0123] Furthermore, this application also provides a scheme for correcting "abnormal" catchlights in high-quality portraits when the original image lacks catchlights: based on the low-quality portrait, it is determined that the original image lacks catchlights, and based on the high-quality portrait, high-quality catchlight features and high-quality human eye backgrounds are separated. Since the high-quality catchlight features are "abnormal" catchlights, they can be deleted while retaining the high-quality human eye backgrounds. This achieves the correction of "abnormal" catchlights in high-quality portraits, ensuring that high-quality portraits do not contain "abnormal" catchlights. This satisfies the user's need for facial region fidelity in shooting scenarios such as low-light shooting and telephoto shooting, where algorithms are needed to enhance low-quality portrait photos.
[0124] The iris feature extraction method provided in this application can be applied to electronic devices with camera functionality. See also... Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, an ambient light sensor 180L, etc.
[0125] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0126] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. For example, processor 110 is used to execute the iris feature extraction method in the embodiments of this application.
[0127] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.
[0128] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0129] External memory 120 generally refers to external storage. In the embodiments of this application, external storage refers to storage other than the memory of electronic devices and the cache of processors. This storage is generally non-volatile memory.
[0130] Internal memory 121, also known as "RAM," can be used to store executable program code for a computer, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a given function (such as sound playback, image playback, etc.).
[0131] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be an organic light-emitting diode (OLED). In some embodiments, electronic device 100 may include one or N displays screens 194, where N is a positive integer greater than 1.
[0132] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.
[0133] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.
[0134] Electronic device 100 also includes various sensors that can convert different physical signals into electrical signals.
[0135] The above is a detailed description of the embodiments of this application using electronic device 100 as an example. It should be understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on electronic device 100. Electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have different component configurations. The various components shown in the figures can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.
[0136] The electronic device provided in this application embodiment can be user equipment (UE), such as a mobile terminal (e.g., a user's mobile phone), tablet computer, desktop computer, laptop computer, handheld computer, netbook, personal digital assistant (PDA), etc.
[0137] In addition, an operating system runs on top of these components. Examples include Apple's iOS, Google's Android, and Microsoft's Windows. Applications can be installed and run on this operating system.
[0138] The operating system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.
[0139] Figure 6 This is a software structure block diagram of an electronic device 100 according to an embodiment of this application.
[0140] A layered architecture divides the system into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: applications, application framework, hardware abstraction layer (HAL), and kernel.
[0141] The application layer may include a series of application packages. In this embodiment, the application package may include a camera, a gallery, etc.
[0142] The application framework layer provides an application programming interface (API) and programming framework for applications within the application layer. The application framework layer includes some predefined functions. In this embodiment, the application framework layer may include a camera access interface, which may include services such as human eye iris matting, eye catchlight / background separation, eye catchlight enhancement, and eye catchlight fusion. The camera access interface is used to provide an API and programming framework for camera applications.
[0143] The hardware abstraction layer (HAL) is an interface layer located between the application framework layer and the kernel layer, providing a virtual hardware platform for the operating system. In this embodiment, the HAL may include a camera hardware abstraction layer and a camera algorithm library.
[0144] The camera hardware abstraction layer (HAL) can provide virtual hardware for camera devices such as the front and rear cameras (referred to as camera HAL). The camera algorithm library may include algorithms for iris matting, eye catchlight / background separation, eye catchlight enhancement, and eye catchlight fusion. In other words, the camera algorithm library contains the code and data to implement the iris feature extraction method provided in the embodiments of this application.
[0145] The kernel layer is the layer between hardware and software. It includes drivers for various hardware components. Specifically, the kernel layer can include camera device drivers, digital signal processor (DSP) drivers, and image processor (IPC) drivers. Camera device drivers are used to drive the camera sensor to acquire images and to drive the IPC to preprocess the images. DSP drivers are used to drive the DSP to process images. IPC drivers are used to drive the graphics processor to process images.
[0146] Furthermore, for ease of explanation, in Figure 6 The document also demonstrates the hardware layer that interacts with the various software layers. For example, the hardware layer includes the camera sensor, image signal processor, data signal processor, and image processor.
[0147] The iris feature extraction method in this application embodiment is summarized below in conjunction with the above hardware and system structures:
[0148] (1) Image acquisition stage:
[0149] like Figure 7 As shown, the electronic device acquires the first image (original image) and uses a face enhancement depth model to perform face enhancement processing on the first image to obtain the second image.
[0150] First, in response to the user's action on the camera application icon (such as a click), the camera application calls the camera access interface in the application framework layer to launch the camera application. Then, it sends a command to launch the camera application by calling the camera device (such as the rear camera) in the camera hardware abstraction layer. The camera hardware abstraction layer sends this command to the camera device driver in the kernel layer, which can then activate the rear camera sensor, i.e., the electronic device starts the shooting function.
[0151] Then, in response to the user pressing the camera shutter button, a first image signal is acquired through the rear camera, and the first image light signal is transmitted to the image signal processor for preprocessing to obtain a first image, which includes facial features. This first image is then transmitted to the camera hardware abstraction layer via the camera device driver. The camera hardware abstraction layer can then send the first image to the camera algorithm library.
[0152] With the support of digital signal processors and image processors, the camera algorithm library can first input the first image into the face enhancement depth model, perform face enhancement processing on the first image, and obtain the second image, which has better image quality than the first image.
[0153] For ease of explanation, the first image will be referred to as a low-quality portrait, and the second image will be referred to as a high-quality portrait.
[0154] (2) Image processing stage:
[0155] like Figure 7 As shown, in the image processing stage, the image processing provided by this application includes: ① human eye iris matting; ② eye catchlight / background separation; ③ eye catchlight enhancement; ④ image fusion.
[0156] On the one hand, from the perspective of processing high-quality portraits with abnormal catchlights after enhancement, the camera algorithm library first uses the human eye iris matting algorithm provided in this application to perform matting processing on the high-quality portrait, and extracts the human eye iris region image from the high-quality portrait; then the camera algorithm library uses the catchlight / background separation algorithm provided in this application to separate the eye highlight region and eye background region of the high-quality portrait from the human eye iris region image, wherein the highlight features in the eye highlight region of the high-quality portrait can be "abnormal" catchlight features.
[0157] On the other hand, from the perspective of processing low-quality portraits, the camera algorithm library first uses the human eye iris matting algorithm provided in this application to perform matting processing on the low-quality portrait, and extracts the human eye iris region image from the low-quality portrait; then the camera algorithm library uses the eye catchlight / background separation algorithm provided in this application to separate the eye catchlight features from the human eye iris region image, and obtain the eye highlight area and eye background area of the low-quality portrait.
[0158] In this embodiment, the catchlight / background separation algorithm may include: first, upsampling the image of the human eye's iris region; then, using a highlight removal algorithm to separate the eye highlight region (which may contain catchlight features) and the eye background region from the image of the human eye's iris region; and finally, downsampling the separated eye highlight region and eye background region. The specific implementation process of the catchlight / background separation algorithm will be described in detail below and will not be repeated here.
[0159] It should be noted that after downsampling the highlight area and background area of the eye, considering that the iris area obtained by the human eye iris matting process may have residual eyelids, and considering that the highlight removal algorithm may have errors, there may be a situation where there is a highlight area in the human eye iris area, but the highlight features at the highlight area are not actually the catchlight. In other words, the image taken by the user may not necessarily contain the catchlight, and it is necessary to identify the highlight area of the eye in the original image.
[0160] This application, based on the principle of high-fidelity enhancement, determines whether catchlights exist in the original image by using parameters of the highlight region of the eye separated from the iris region. The parameters of the highlight region may include its color moment, contrast, average brightness, saturation, and / or area. Then, based on whether catchlights exist in the original image, a corresponding catchlight processing strategy is selected.
[0161] For example, in some scenarios, such as Figure 7As shown, if the original image contains catchlights, the catchlight enhancement algorithm provided in this application is used to "enhance catchlights" based on the low-quality catchlight features. Then, the catchlight fusion algorithm provided in this application is used to "fuse" the enhanced catchlight features with the eye background area of the high-quality portrait.
[0162] For example, in other scenarios, if the original image does not have catchlights, then catchlight enhancement and image fusion are not performed. The highlight and background areas of the eyes in low-quality portraits are discarded, as are the "abnormal" catchlight features in high-quality portraits. Only the background area of the eyes in high-quality portraits is retained to meet the user's need for facial region fidelity in shooting scenarios such as low-light shooting and telephoto shooting, where algorithms are needed to enhance low-quality portrait photos.
[0163] It should be noted that after obtaining the eye highlight area and eye background area of a low-quality portrait, and the eye highlight area and eye background area of a high-quality portrait, the eye background area of the low-quality portrait and the eye highlight area of the high-quality portrait are discarded.
[0164] (3) Image generation stage:
[0165] On the one hand, when the original image has catchlights, the electronic device will perform "image fusion" with the enhanced catchlight features and the background area of the eyes of the high-quality portrait to obtain a high-quality portrait with controllable enhancement of catchlights.
[0166] In other words, to controllably enhance catchlights, this application first employs the catchlight enhancement algorithm provided in this application based on the low-quality catchlight features in low-quality portraits to enhance the catchlights and obtain enhanced catchlight features. Then, the enhanced catchlight features replace the "abnormal" catchlight features in high-quality portraits, correcting the catchlight features and resulting in a high-quality portrait with "normal" catchlights after enhancement. In this case, the image processed in the photograph has normal catchlights, thus meeting the user's need for facial fidelity in low-light shooting, telephoto shooting, and other scenarios requiring algorithmic enhancement of low-quality portrait photos.
[0167] On the other hand, if the original image lacks catchlights, the electronic device discards the "abnormal" catchlight features, retaining only the background area around the eyes in the high-quality portrait. This results in a high-quality portrait output without "abnormal" catchlights, thus correcting the catchlight issue. In this case, the image processed image does not contain "abnormal" catchlights, meeting the user's need for facial fidelity in low-light shooting, telephoto shooting, and other scenarios requiring algorithmic enhancement of low-quality portrait photos.
[0168] After obtaining a photograph through image processing, the camera algorithm library can send the resulting photograph to the camera hardware abstraction layer. The camera hardware abstraction layer can then display the photograph.
[0169] It should be noted that although the embodiments of this application are described using the Android system as an example, the basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.
[0170] The iris feature extraction method provided in this application can be executed by the aforementioned electronic device, or by a functional module and / or entity within that electronic device capable of implementing the iris feature extraction method. Furthermore, the solution in this application can be implemented through hardware and / or software, the specific implementation depending on actual usage requirements, and this application does not impose any limitations. The following description uses an electronic device as an example to illustrate the iris feature extraction method provided in this application.
[0171] The following is combined Figure 8 Describe the overall implementation process of this application. For example... Figure 8 As shown, the iris feature extraction method provided in this application embodiment includes the following steps S201-S208.
[0172] First, it should be noted that in this embodiment, when the electronic device has its face enhancement function enabled, the electronic device will identify whether the captured photo contains facial features during the photo-taking process. When the electronic device identifies that the captured photo contains facial features and the image quality does not meet the preset image quality requirements, the electronic device will use a face enhancement depth model to process the photo to achieve full-face enhancement. For ease of explanation, the electronic device in this embodiment is assumed to have its face enhancement function enabled by default.
[0173] This application focuses on the user's need for facial region fidelity in scenarios such as low-light shooting and telephoto shooting. It can controllably enhance the catchlight of the human eye and make up for the unnaturalness of the catchlight caused by full-face enhancement through the face enhancement depth model.
[0174] S201A, acquires low-quality portraits.
[0175] S201B, for acquiring high-quality portraits.
[0176] High-quality portraits are images obtained by processing low-quality portraits using a face enhancement depth model. However, when processing low-quality portraits using this model, the catchlight features in the eyes are often mistakenly enhanced, resulting in an unnatural-looking catchlight.
[0177] Low-quality portraits can also be called low-quality face images, low-quality original portraits, or low-quality original images, while high-quality portraits can also be called high-quality face images, high-quality portrait anomaly images, or high-quality anomaly images.
[0178] In this embodiment of the application, to compensate for the unnatural appearance of the eye light caused by full-face enhancement using a face enhancement depth model, the electronic device provides an automatic eye light correction function. This automatic eye light correction function enables the iris feature extraction method provided in this embodiment of the application. The electronic device can turn the automatic eye light correction function on or off in response to user operation.
[0179] When an electronic device does not have its automatic catchlight correction function enabled, when a user takes a photo, the device inputs the original image (referred to as a low-quality portrait) into a preset face enhancement depth model to obtain a high-quality portrait with enhanced face. In this high-quality portrait, while the face is enhanced, the catchlights in the eyes are also enhanced, but these enhanced catchlights may appear unnatural. It should be noted that because the enhanced catchlights in the eyes are unnatural, this image can be described as a high-quality portrait with "abnormal" enhanced catchlights.
[0180] When the electronic device has the automatic catchlight correction function enabled, when the user triggers the photo, the electronic device will simultaneously acquire the original photo (low-quality portrait) and the high-quality portrait obtained after face enhancement depth model.
[0181] In some embodiments, low-quality portraits are acquired by the electronic device taking a photo containing facial features after the user presses the camera shutter button of the electronic device under conditions such as low light and distance.
[0182] S202A-S203A performs iris masking on low-quality portraits to obtain the iris region of the eyes. Then, it performs catchlight / background separation processing on the iris region of the eyes to obtain the highlight and background areas of the eyes.
[0183] S202B-S203B performs iris masking on high-quality portraits to obtain the iris / sclera region of the eyes. Then, it performs catchlight / background separation processing on the iris / sclera region of the eyes to obtain the highlight and background areas of the eyes.
[0184] S202A and S203A are image processing methods for low-quality portraits. S202B and S203B are image processing methods for high-quality portraits. Image processing for low-quality portraits is similar to image processing for high-quality portraits. The portrait catchlight processing method provided in this application is described in detail below with reference to the accompanying drawings.
[0185] For example, such as Figure 9 As shown, the eye region is first detected in the portrait, as indicated by the black dotted line. Then, eye matting is performed on the eye region to obtain the eye image. Next, the iris region is detected in the eye image, as indicated by the white dotted line. Then, iris matting is performed on the iris region, allowing the extraction of the iris image from the portrait. The specific implementation process of the iris matting algorithm will be detailed below and will not be elaborated here.
[0186] In addition, the sclera of the human eye can be cut out from images of the human eye. For example... Figure 9 As shown, the sclera region of the human eye is detected in the human eye image, as the ring-shaped part circled by the white dotted line represents the sclera region of the human eye; then, after performing sclera matting processing on the sclera region, the sclera image of the human eye can be extracted from the portrait.
[0187] In some embodiments, if the iris region of a human eye is extracted from a portrait, the portion of the iris region excluding the highlight area can be referred to as the eye background region.
[0188] In other embodiments, if the sclera region of the human eye is extracted from the portrait, the portion of the sclera region excluding the highlight area can be referred to as the background region of the eye.
[0189] In some other embodiments, if the iris region and sclera region of the human eye are extracted from the portrait, then the portion of the iris region and sclera region excluding the highlight area can be referred to as the eye background region.
[0190] In the embodiments of this application, the "eye background area" can also be referred to as the human eye iris / sclera background area, which includes the following three situations:
[0191] (1) "Background area of the eye" refers to the part of the iris of the human eye other than the highlight area.
[0192] (2) “Background area of the eye” refers to the part of the sclera of the human eye excluding the highlight area.
[0193] (3) “Background area of the eye” refers to the part of the iris and sclera of the human eye excluding the highlight area.
[0194] It should be noted that the embodiments of this application support eye light / background separation processing for the iris region of the human eye, and can also support eye light / background separation processing for the sclera region of the human eye, as well as eye light / background separation processing for both the iris region and the sclera region of the human eye.
[0195] For ease of explanation, each embodiment of this application uses the process of separating the light from the background in the iris region of the human eye as an example for illustrative purposes.
[0196] For example, such as Figure 10 As shown, firstly, an iris matting algorithm is used to process the portrait, identifying the iris region of the human eye. Then, an eye catchlight / background separation algorithm is used to process the iris region image, obtaining the highlight region and background region of the eye. Furthermore, this application also performs eye catchlight recognition to determine whether the highlight region of the eye contains a catchlight; the specific implementation process will be described in detail below.
[0197] In the embodiments of this application, the eye light / background separation algorithm includes upsampling the human eye iris image, separating the eye highlight region and the eye background region from the human eye iris image based on the highlight removal algorithm, and downsampling the eye highlight region and the eye background region.
[0198] (1) Upsample the human eye iris image.
[0199] Upsampling refers to magnifying the image.
[0200] In this embodiment, upsampling can be achieved using interpolation or deconvolution (also known as transposed convolution). For example, the resize() function in the open-source computer vision library OpenCV can be called to adjust the image size using an interpolation algorithm (such as bilinear interpolation).
[0201] To make the subsequent eye light / background separation more detailed, the iris / sclera region of the human eye is first upsampled to achieve image magnification, and then the upsampled iris / sclera region of the human eye is processed for eye light / background separation.
[0202] To reduce noise amplification and jagged edges on the iris and catchlight edges caused by upsampling, this application uses an alternating method of filtering and anti-aliasing upsampling to achieve image magnification.
[0203] Figure 11 A schematic diagram of the sampling process in this application is shown. Figure 11As shown, the upsampling process includes steps S2031-S2035. For example, assuming the pixel size of the iris region image in a portrait (hereinafter referred to as the iris image) is approximately 20×20, anti-aliasing upsampling is first performed on the iris image. This enlarges the iris image while avoiding jagged edges on the iris and catchlight edges; for example, the enlarged iris image has a pixel size of 100×100. Then, the enlarged iris image is filtered and denoised to remove noise. Next, it is determined whether the enlarged iris image has reached a specified size. For example, the specified size is 200×200. If the enlarged iris image has not reached the specified size, anti-aliasing upsampling and filtering / denoising are continued until the enlarged iris image reaches the specified size of 200×200. At this point, anti-aliasing upsampling and filtering / denoising are no longer performed, resulting in an enlarged iris image.
[0204] The proposed solution, through multiple (e.g., 2 to 3) cycles of anti-aliasing upsampling and filtering denoising, gradually magnifies the human eye iris image (e.g., 10x magnification), making subsequent eye light / background separation more detailed, thereby improving the accuracy of image processing.
[0205] (2) The highlight region and background region of the eye are separated from the human eye iris image based on the highlight removal algorithm.
[0206] In this application embodiment, considering that the highlight removal process (highlights are also known as catchlights) will not produce negative effects on the background area of the human eye, such as smearing, missing parts, or changes in the brightness of non-catchlight areas, this application adopts a simple and efficient general highlight removal algorithm, which can quickly separate highlights from the background on large-sized human eye iris images.
[0207] Highlight removal algorithms can detect highlight areas in the iris region of the human eye. For example, image thresholding can be used to generate a mask for the highlight areas. The flood fill method can be used to solve for the connected components of the mask. A connected component is defined as a maximal connected subgraph of an undirected graph.
[0208] Assuming the image imaging follows a dichotomous reflectance model, image I is decomposed into diffuse reflectance components (denoted as V). diff ) and specular reflection component (denoted as V) spec The diffuse component corresponds to the catchlight pattern, and the specular component corresponds to the catchlight pattern. Through theoretical derivation, the approximate specular component is expressed as β. s :
[0209] β s =I min -τ(p) (Formula 1)
[0210] Among them, I min τ(p) represents a single-channel image obtained by taking the minimum value of the RGB three channels of each pixel in image I; τ(p) represents an offset related to the mean and variance of the image pixels.
[0211] The approximate diffuse reflection component is represented as V' diff :
[0212] V d ′ iff = I-kβ s (Formula 2)
[0213] To solve for the scaling factor k, the image is first normalized to the pixel range [0,1], and regions with pixel values greater than 0.1 are considered potential specular regions. Then, the flood fill method is used to determine the dominant (M largest) specular connected component regions within these regions. It should be noted that the diffuse color intensity of the specular connected component regions and their edge regions should be consistent, i.e., the mean values of the diffuse components should be equal. Further, under this smooth color transition criterion, the least squares method is used to obtain the value of k. Substituting the value of k into Formula 2 yields the catchlight-removed image V′. diff Then, using the following formula 3, the catchlight pattern is obtained:
[0214] V s ′ pec = IV d ′ iff (Formula 3)
[0215] Among them, V′ diff V′ represents the removal of catchlight from the eye image. spec This represents a gaze pattern.
[0216] It should be noted that when using the specular removal algorithm to separate catchlights from the background, this application considers that the catchlight region in the actual scene may not be continuous and may have multiple connected components. Therefore, the top M largest connected components that "may be catchlights" are identified as the specular regions that need to be separated.
[0217] For example, if M is 2, and assuming there are four connected components in the iris region of the human eye: 7, 20, 9, 5, then... Figure 12 As shown, the region containing the two largest connected components "20 and 9" that are "possible catchlights" in the iris region of the human eye can be identified as the highlight region that needs to be separated.
[0218] In this embodiment of the application, the eye light region mask is obtained by an automatic specular removal algorithm, which can support the separation of eye light regions with arbitrary shapes and discontinuities.
[0219] The proposed solution can automatically identify the iris, sclera, and catchlight areas of the eyes without the need for manual delineation.
[0220] (3) Downsample the highlight area and background area of the eye.
[0221] Downsampling refers to reducing the size of an image. Similar to upsampling, downsampling can be achieved using interpolation, pooling, or convolution.
[0222] Since subsequent steps have a significant impact on whether the final catchlight enhancement effect is natural, in order to prevent it from being affected by the interpolation algorithm, avoid post-processing after image fusion, and improve algorithm efficiency, this step downsamples both the separated eye highlight area and eye background area to the original image size (e.g., shrinking the current image to 1 / 10).
[0223] For example, the pixel sizes of the separated eye highlight area and eye background area are 50×50 and 200×200, respectively. After downsampling, the pixel sizes of the eye highlight area and eye background area are 5×5 and 20×20, respectively.
[0224] It should be noted that if the image is directly downsampled, for example using nearest-neighbor linear interpolation, the downsampled image will have many jagged edges. In some embodiments, this application can use anti-aliasing downsampling, which scales the dimensions to the original image size to avoid jagged edges at the edges of the iris and catchlight.
[0225] Figure 13 This illustration shows a schematic diagram of eye light / background separation in an embodiment of this application for an image of the human eye's iris region. For example... Figure 13 As shown, taking the separation of eye highlight / background in a low-quality human iris image as an example, the human iris image is first upsampled and enlarged; then, the eye highlight / background separation algorithm is used to separate the enlarged human iris image to obtain the eye highlight area and the eye background area; then, the separated eye highlight area and eye background area are downsampled to reduce the image size to the original size, thus obtaining the low-quality eye highlight area and eye background area.
[0226] It should be noted that the highlight areas around the eyes in low-quality portraits may contain catchlight features. If the low-light areas may contain catchlight features, these features will be enhanced to replace "abnormal" catchlights. Therefore, in subsequent image processing, the highlight areas around the eyes of low-quality portraits will be the primary focus for catchlight feature recognition. Furthermore, the background areas around the eyes of low-quality portraits will not be used in subsequent image processing and will be discarded.
[0227] It should also be noted that, similar to the process of performing catchlight / background separation on the iris region image of low-quality portraits described above, the same catchlight / background separation is performed on the iris region image of high-quality portraits to obtain the background and highlight regions of the eyes. The highlight features in the highlight region of the eyes in high-quality portraits may be "abnormal" catchlight features and need to be replaced with "normal" catchlight features. Therefore, the highlight regions of the eyes in high-quality portraits will be discarded.
[0228] It should also be noted that in the embodiments of this application, "normal" catchlights are defined in relation to the distorted effect of "abnormal" catchlights. For example, a natural and realistic catchlight in a portrait can be considered "normal"; or a portrait that does not contain "abnormal" catchlights can also be considered "normal".
[0229] S204. Based on the highlight properties of the highlight area of the eyes in a low-quality portrait, determine whether the low-quality portrait has catchlight features.
[0230] In this embodiment, after separating the highlight area and background area of the eyes using the catchlight / background separation algorithm, considering that the iris area obtained by the human eye iris matting process may have residual eyelids, and considering that the highlight removal algorithm may have errors, it is possible that there is a highlight area in the human eye iris area, but the highlight features at the highlight area are not actually catchlights. In other words, the original image taken by the user may not necessarily contain catchlights, and it is necessary to identify the highlight area of the eyes in the original image, that is, to determine whether low-quality portraits have catchlight features.
[0231] Considering that not all user-captured images contain catchlights, this application, based on the principle of high-fidelity enhancement, removes incorrectly added catchlight features directly from the corresponding high-quality portrait if low-quality portraits lack catchlight characteristics. Since the catchlight / background separation algorithm does not alter the iris / sclera background, it preserves the detailed information of the iris / sclera region in high-quality portraits.
[0232] In this embodiment, the presence of catchlight in the original image can be determined based on the area of the highlight region separated from the iris region of the human eye. For example, if the area of the highlight region is 0, it can be directly determined that there is no catchlight in the highlight region. However, due to factors such as the possibility of residual eyelids in the iris portion obtained through image matting and potential errors in the highlight removal algorithm, even if the area of the highlight region is not 0, the highlight region may still lack catchlight.
[0233] To improve the accuracy of catchlight recognition, in this embodiment, multiple highlight attributes of the highlight region of the eye separated from the iris region can be jointly judged, thus enabling a more accurate determination of whether a catchlight exists in the original image. The highlight attributes of the highlight region of the eye may include attributes such as color moment, contrast, average brightness, saturation, and / or area.
[0234] In some embodiments, this application selects multiple hyperparameters as features to train a simple binary classifier offline. These hyperparameters include color moments, contrast, average brightness, saturation, and highlight area. It should be noted that, in the context of machine learning, hyperparameters are parameters whose values are set before the learning process begins. By optimizing these hyperparameters, an optimal set of hyperparameters is selected for the learning machine to improve learning performance and effectiveness.
[0235] For example, taking a binary classifier as a support vector machine (SVM), multiple highlight attributes of the highlight region of the eye are input into a trained SVM, which can directly output the catchlight recognition result. Through the solution of this application, the presence of catchlights in the original image is determined comprehensively based on multiple highlight attributes of the highlight region of the eye, thus improving the accuracy of catchlight recognition.
[0236] If the highlight area of the eyes in the portrait has catchlight characteristics, then continue to execute S205-S206; if the highlight area of the eyes in the portrait does not have catchlight characteristics, then continue to execute S207.
[0237] S205. If the original image contains catchlight features, then adaptively enhance the catchlight features.
[0238] It should be noted that the size, shape, brightness, and other characteristics of the original area of the catchlight are all random and unpredictable. Therefore, the same enhancement parameters (such as fixed contrast stretching parameters) are difficult to be compatible with all situations.
[0239] To enhance the generalization ability of the algorithm, this application employs an adaptive catchlight contrast enhancement method. First, it calculates parameters such as the length and width of the catchlight region (e.g., the length and width of the minimum bounding rectangle of the catchlight region), average brightness, and maximum brightness. Then, it calculates the corresponding parameters for the contrast enhancement algorithm. These parameters include the gamma value used for gamma correction, as well as the block size and contrast limit for restricting the adaptive histogram equalization. The application of these contrast enhancement algorithm parameters in this application is described below.
[0240] First, let's explain gamma values. Gamma values are used for gamma correction of highlight areas to enhance image contrast in those areas. The gamma correction function can be used to obtain an image corrected for a specific gamma value. The gamma correction function can be expressed as:
[0241] I′=I γ
[0242] Where γ represents the gamma value, I represents the original image, and I′ represents the image after contrast enhancement.
[0243] For example, the gamma value corresponding to the highlight area can be calculated based on the ratio of the brightness of the brightest point in the highlight area to 255. Gamma correction is then performed on the highlight area using this gamma value to enhance the image contrast of the highlight area.
[0244] The following section explains the block size and contrast limit for limiting contrast adaptive histogram equalization.
[0245] First, it's important to note that standard histogram equalization uses the same transformation derived from the image histogram to transform all pixels. This works well when the pixel value distribution across the entire image is similar. However, when the image contains areas that are significantly brighter or darker than most areas, the contrast in those areas will not be adequately enhanced.
[0246] Adaptive histogram equalization (AHE) transforms each pixel using a transformation function derived from a histogram calculated from its neighborhood. Therefore, it is suitable for improving local contrast in an image. However, AHE tends to over-amplify noise in relatively uniform regions of the image.
[0247] Contrast-limited adaptive histogram equalization (CLAHE) further prevents this from happening by limiting the magnification. Contrast limitation is achieved by limiting the ratio of the number of pixels per bar in the histogram of each local region to the average number of gray levels in the entire image.
[0248] OpenCV's CLAHE implementation includes two important parameters:
[0249] (1) Local area size, such as 4×4.
[0250] (2) Contrast limit (ClipLimit), such as 20.
[0251] ClipLimit is the clipping limit of CLAHE. Clipping is required when the pixel data of the histogram groups in each block of the image exceeds this limit. During clipping, the portion of each group with more pixels than the limit is clipped, and the excess portions from all groups are summed and then evenly distributed across the groups. In other words, portions of the histogram bins with a height greater than the calculated ClipLimit are clipped and then evenly distributed across the entire histogram. If there are still excess pixels after distribution, the above steps can be recursively repeated.
[0252] When using this method to enhance catchlights, each channel of the catchlight map needs to be processed. For example, the side length of the local region size is min(L / 2, 3), and the clipping constraint parameter is max(D / 2, 2). Here, L represents the shortest side of the catchlight region, and D represents the maximum value of the histogram within the bounding rectangle of the maximum highlight region. min() represents taking the minimum value, and max() represents taking the maximum value.
[0253] In this embodiment, each catchlight region can be adaptively enhanced. Specifically, enhancement parameters corresponding to a catchlight region can be calculated based on its size, shape, brightness, and other characteristics, and the catchlight region can be enhanced in a targeted manner using the enhancement parameters corresponding to that catchlight region. Therefore, this embodiment enables different catchlight regions to be adaptively enhanced.
[0254] For example, Table 1 shows parameters such as the length, width, average brightness, and maximum brightness of the catchlight region. As shown in Table 1, the catchlight region includes catchlight region 1 and catchlight region 2.
[0255] Table 1
[0256] length width Average brightness Brightest point Eye light area 1 8 pixels 7 pixels 100 gray levels 155 gray levels Eye Light Area 2 3 pixels 3 pixels 80 gray levels 115 gray levels
[0257] As shown in Table 1, in the case of a 256 grayscale image signal, the size of the catchlight region 1 is larger than that of the catchlight region 2, the average brightness of the catchlight region 1 is greater than that of the catchlight region 2, and the brightness of the brightest point of the catchlight region 1 is greater than that of the brightest point of the catchlight region 2.
[0258] After calculating parameters such as the length, width, average brightness, and peak brightness of the catchlight region, the parameters of the corresponding contrast enhancement algorithm are calculated. Here, we will use the calculation of gamma values for different regions as an example for illustration. As mentioned above, the gamma value can be calculated based on the ratio of the peak brightness to 255. Table 2 below shows the gamma values corresponding to different catchlight regions.
[0259] Table 2
[0260] Brightest point Calculated gamma value Eye light area 1 155 gray levels 0.61 Eye Light Area 2 115 gray levels 0.45
[0261] According to the solution of this application, a gamma value of 0.61 is used for gamma correction of catchlight region 1 to enhance the image contrast of catchlight region 1. A gamma value of 0.45 is used for gamma correction of catchlight region 2 to enhance the image contrast of catchlight region 2. That is to say, in this embodiment of the application, different gamma values can be used for correction for different highlight regions.
[0262] For catchlight regions with different characteristics such as size, shape, and brightness, the embodiments of this application can perform targeted enhancement processing on each catchlight region. Specifically, enhancement parameter A corresponding to catchlight region 1 can be calculated based on the size, shape, brightness, and other characteristics of catchlight region 1, and catchlight region 1 can be enhanced specifically using enhancement parameter A. Similarly, enhancement parameter B corresponding to catchlight region 2 can be calculated based on the size, shape, brightness, and other characteristics of catchlight region 2, and catchlight region 2 can be enhanced specifically using enhancement parameter B. Therefore, the embodiments of this application can enable adaptive enhancement of different catchlight regions.
[0263] It should be noted that the above embodiments are exemplified by calculating the parameters of the contrast enhancement algorithm based on parameters such as the length, width, average brightness, and brightness of the brightest point of the catchlight region. It can be understood that in actual implementation, the parameters of the contrast enhancement algorithm can also be calculated based on other parameters of the catchlight region. The specific calculation can be determined according to actual usage requirements, and this application embodiment does not impose any limitations.
[0264] It should be noted that this application can adaptively enhance the catchlights in the original image using the aforementioned catchlight enhancement algorithm. This catchlight enhancement algorithm uses an adaptive catchlight contrast enhancement method, which can controllably achieve catchlight enhancement. The catchlight enhancement algorithm provided in this application is not a depth model, thus avoiding the abnormalities that occur after catchlight enhancement caused by depth models. It meets the user's need for facial region fidelity in shooting scenarios such as low-light shooting and telephoto shooting, where low-quality portrait photos need to be enhanced by algorithms.
[0265] S206 performs image fusion of enhanced catchlight features with the eye background area of a high-quality portrait.
[0266] To controllably enhance catchlights in the eyes, this embodiment first employs the catchlight enhancement algorithm provided in this application, based on the low-quality catchlight features in low-quality portraits, to enhance the catchlights and obtain enhanced catchlight features. Then, the enhanced catchlight features replace the "abnormal" catchlight features in high-quality portraits, correcting the catchlight features and resulting in high-quality portraits with "normal" catchlights after enhancement. This satisfies the user's need for facial region fidelity in shooting scenarios such as low-light shooting and telephoto shooting, where algorithmic enhancement of low-quality portrait photos is required.
[0267] It should be noted that the positions and relative positions of the iris and catchlights are not entirely the same for low-quality and high-quality portraits. Furthermore, the edges of the catchlight features in a low-quality portrait may not have a smooth decay after adaptive enhancement. If the enhanced catchlight features are directly copied and pasted onto the iris / sclera background of a high-quality portrait, the catchlight effect will be unnatural. To ensure a natural blend between the enhanced catchlights and the iris / sclera background of a high-quality portrait, this embodiment employs affine transformation and catchlight region edge feathering.
[0268] For example, Figure 14 This illustration shows a schematic diagram of the process for fusing eye light with the background of the human eye iris, as provided in an embodiment of this application. Figure 14 As shown, firstly, affine transformation matrices are determined based on the edge key points of the iris background region of a low-quality portrait and those of a high-quality portrait. Then, the enhanced catchlight is multiplied by the affine transformation matrix to perform an affine transformation, resulting in an aligned enhanced catchlight. Next, the aligned enhanced catchlight is adaptively feathered to obtain a feathered enhanced catchlight. Finally, the feathered enhanced catchlight is fused with the iris background region. Thus, the controllable enhanced catchlight of this application can be naturally integrated into the eye background region of a high-quality portrait.
[0269] It should be noted that this explanation uses affine transformation and image fusion based on the iris background area as an example. It can be understood that in actual implementation, affine transformation and image fusion can also be performed based on the sclera background area. The specific implementation method can be determined according to the actual usage requirements, and this application embodiment does not limit it.
[0270] The principle of affine transformation will be introduced first, and then this application will explain how to perform affine transformation.
[0271] In affine transformations of an image, it is necessary to operate on each pixel in the image. Assuming that the coordinates of pixel X in the image are (x, y), then pixel X can be represented in matrix form as follows:
[0272]
[0273] like Figure 15 As shown, affine transformations can include rotation, scaling, and translation. An affine transformation is a linear transformation with added translation.
[0274] The following examples provide the matrices for the rotation parameter R and the translation parameter T.
[0275]
[0276]
[0277] Where R is an orthogonal matrix, θ represents the rotation angle, and T represents the translation distance. Additionally, s can be used to represent the scaling factor.
[0278] In some embodiments, after rotation, pixel X can be represented in matrix form as follows:
[0279]
[0280] In some embodiments, after translation, pixel X can be represented in matrix form as follows:
[0281]
[0282] In some embodiments, after scaling, pixel X can be represented in matrix form as follows:
[0283]
[0284] In other embodiments, after affine transformations such as rotation, translation, and scaling, the coordinates of pixel X become (u, v), and pixel X can be represented in matrix form as follows:
[0285]
[0286] The following describes the process of affine transformation based on enhanced eye light features and the human eye background region in the embodiments of this application, based on the above principles.
[0287] For example, such as Figure 16 As shown, the edge feature points of the eye background region of a low-quality portrait can be identified, and the corresponding pixel matrix is denoted as F1. The edge feature points of the eye background region of a high-quality portrait can be identified, and the corresponding pixel matrix is denoted as F2.
[0288] Assuming the affine transformation matrix is M, and the pixel matrix corresponding to the enhanced catchlight feature is X, the affine transformation matrix M between low-quality and high-quality portraits can be calculated using F1*M = F2. For example, the calculated affine transformation matrix M can be expressed as:
[0289]
[0290]
[0291] In this matrix, M represents the complete affine transformation matrix, matrix A controls rotation and scaling, and matrix B controls translation.
[0292] Then, the pixel matrix X corresponding to the enhanced catchlight feature is multiplied by the affine transformation matrix M to obtain the aligned enhanced catchlight region X'. Furthermore, since an affine transformation is a linear transformation plus translation, the aligned enhanced catchlight region X' after the affine transformation can be represented by a matrix as follows:
[0293]
[0294]
[0295] By using affine transformation, the enhanced catchlight features are aligned with the background area of the eyes in the high-quality portrait. This ensures that the enhanced catchlight features and the background area of the eyes are accurately aligned during image fusion. Since the position of the catchlight features relative to the iris remains unchanged, which is an important factor in keeping the gaze unchanged, aligning the catchlight features with the iris results in a natural and realistic catchlight effect.
[0296] After completing the affine transformation, the edges of the catchlight can also be feathered. For example... Figure 17 As shown, by feathering the catchlight, the transition between the inner and outer edges of the catchlight is blurred, creating a gradient effect and achieving a natural transition, thus improving the image display effect.
[0297] It should be noted that this application uses affine transformation and feathering to naturally blend the enhanced catchlight into the eye region of the high-quality anomaly image, rather than simply copying and pasting the catchlight.
[0298] The eye-light fusion algorithm provided in this application embodiment can naturally blend the enhanced eye-light features with the background area of the human eye, so that the enhanced eye-light features can be aligned with the background area of the human eye in position, and the edges of the enhanced eye-light features are smooth, so as to present a natural and realistic eye-light effect.
[0299] S207 discards the highlight area around the eyes of low-quality portraits, retaining only the background area around the eyes of high-quality portraits.
[0300] Through S205-S206 or S207, a high-quality portrait with normal catchlights can be obtained. This can correct the unnatural catchlights caused by full-face enhancement through a face enhancement depth model, and meet the user's need for facial region fidelity in shooting scenarios such as low-light shooting and telephoto shooting, where low-quality portrait photos need to be enhanced by algorithms.
[0301] This application addresses the user's need for facial fidelity in low-light and telephoto shooting scenarios by controllably enhancing catchlights in the eyes to compensate for the unnaturalness caused by full-face enhancement using a face enhancement depth model. Furthermore, for low-quality portraits lacking catchlight features, the incorrectly added catchlight features can be directly removed from the corresponding high-quality portrait, thus correcting the unnaturalness and incorrect addition of catchlights caused by full-face enhancement using a face enhancement depth model.
[0302] In summary, the embodiments of this application can select the corresponding catchlight processing strategy based on whether the low-quality portrait has catchlight features.
[0303] Catchlight processing strategy 1: If the original image has catchlight features, then enhance the catchlight based on these features, and then fuse the enhanced catchlight features with the background area of the eyes in the high-quality portrait.
[0304] For example, in some embodiments, when the highlight area around the eyes of a low-quality portrait meets preset catchlight conditions, it is determined that the original low-quality portrait image has catchlights. Then, catchlight enhancement processing is performed on the highlight area around the eyes of the low-quality portrait to obtain enhanced catchlight features. These enhanced catchlight features are then fused with the background area around the eyes of a high-quality portrait to obtain a high-quality portrait with normal enhanced catchlights, thus achieving controllable catchlight enhancement. In this case, the portrait in the image processed has normal catchlights, thereby meeting the user's need for facial region fidelity in shooting scenarios such as low-light shooting and telephoto shooting, where algorithmic enhancement of low-quality portrait photos is required.
[0305] Catchlight processing strategy two: If the original image does not have catchlights, then neither catchlight enhancement nor image fusion will be performed, and only the eye background area of the high-quality portrait will be preserved.
[0306] For example, in some embodiments, when the highlight area around the eyes of a low-quality portrait does not meet the preset catchlight conditions, it is determined that the original image lacks catchlight. Therefore, catchlight enhancement processing is not performed, the low-quality catchlight features are discarded, and only the background area around the eyes of the high-quality portrait is retained. In this way, the final high-quality portrait output does not contain "abnormal" catchlights, thus correcting the catchlights. In this case, the portrait obtained through image processing does not contain "abnormal" catchlights, thereby meeting the user's need for facial region fidelity in shooting scenarios such as low-light shooting and telephoto shooting, where algorithms are needed to enhance low-quality portrait photos.
[0307] Figure 18 This illustration shows a schematic diagram of one implementation process of the iris feature extraction method provided in an embodiment of this application. For example... Figure 18 As shown, when the electronic device has its automatic catchlight correction function enabled, when the user triggers a photo capture, the device simultaneously acquires the original image (low-quality portrait) and a high-quality portrait obtained through a face enhancement depth model. The high-quality portrait exhibits abnormal catchlights. The electronic device performs iris matting and catchlight / background separation processing on the low-quality portrait, obtaining the eye background and catchlight of the low-quality image. Simultaneously, the electronic device performs iris matting and catchlight / background separation processing on the high-quality portrait, obtaining the eye background and abnormal catchlight of the high-quality image.
[0308] In this case, the eye background of the low-quality portrait is discarded, and the abnormal catchlights of the high-quality portrait are also discarded. The catchlights of the low-quality portrait are adaptively enhanced to obtain enhanced catchlights. Then, affine transformation and feathering are applied to the enhanced catchlights to achieve controllable catchlight enhancement. Finally, the enhanced catchlights are fused with the eye background of the high-quality portrait to obtain a high-quality portrait with normal catchlights.
[0309] In this implementation, the portrait in the photo obtained through image processing has catchlights in the eyes, and the catchlights are normal, thus meeting the user's need for the fidelity of the face area in shooting scenarios such as low light shooting and telephoto shooting, which require algorithmic enhancement of low-quality portrait photos.
[0310] Figure 19 This diagram illustrates another implementation process of the iris feature extraction method provided in this application embodiment. For example... Figure 19As shown, when the electronic device has its automatic catchlight correction function enabled, when the user triggers a photo, the device simultaneously acquires the original image (low-quality portrait) and a high-quality portrait obtained through a face enhancement depth model. The high-quality portrait exhibits abnormal catchlights. The electronic device performs iris masking and catchlight / background separation on the low-quality portrait, resulting in the background around the eyes, while the original image lacks catchlights. Simultaneously, the electronic device performs iris masking and catchlight / background separation on the high-quality portrait, resulting in the background around the eyes and the abnormal catchlights.
[0311] In this case, the eye background of low-quality portraits is discarded, and abnormal catchlights in high-quality portraits are also discarded. If the original image lacks catchlights, this application, based on the principle of high-fidelity enhancement, directly removes the incorrectly added catchlight features from the corresponding high-quality portrait. Since the catchlight / background separation algorithm does not alter the iris / sclera background of the human eye, it can preserve the detailed information of the iris / sclera region of the human eye in high-quality portraits.
[0312] In this implementation, the final high-quality portrait output does not contain any "abnormal" catchlights, thus correcting the catchlights. In this case, the portrait obtained through image processing does not contain "abnormal" catchlights, thereby meeting the user's need for facial region fidelity in shooting scenarios such as low-light shooting and telephoto shooting, where algorithms are needed to enhance low-quality portrait photos.
[0313] The iris feature extraction method provided in this application can employ different catchlight processing strategies depending on whether the original image has catchlights: if the original image has catchlights, the catchlights in the original image are enhanced and replaced with the enhanced catchlights to correct the "abnormal" catchlights in the high-quality portrait; if the original image does not have catchlights, the "abnormal" catchlights in the high-quality portrait are discarded to correct them. Therefore, this application can correct the unnatural catchlights caused by full-face enhancement using a face enhancement depth model, thus improving the user experience.
[0314] Human eye iris matting algorithm
[0315] The above describes the overall process of the iris feature extraction method provided in the embodiments of this application. The following describes in detail the human eye iris matting algorithm (also known as the human eye iris extraction algorithm) in the iris feature extraction method provided in the embodiments of this application.
[0316] Figure 20 A schematic flowchart of human eye iris matting is shown in an embodiment of this application. Figure 20 This includes the following steps S301-S309.
[0317] S301, Input a face image.
[0318] In this embodiment, while performing iris matting on the low-quality original portrait image, iris matting is also performed on the high-quality original portrait image. The high-quality portrait image is the face image obtained by inputting the original image into a face enhancement depth model.
[0319] S302, detects the eye area.
[0320] For example, the first network model inputs a face image to be processed, and outputs a mask for the eye region and key points at the center of the eye. The first network model can include functions such as face parsing and facial landmark detection. The mask is a binary image composed of 0s and 1s.
[0321] For example, Figure 21 Image (a) shows a schematic diagram of a mask for the eye region, where the white area represents the eye region. For example, Figure 21 (b) shows a schematic diagram of the central key point of the eye, where point O represents the central key point of the eye.
[0322] S303, Determine the iris region.
[0323] First, it should be noted that the palpebral fissure refers to the slit formed between the upper and lower eyelids. The length of the palpebral fissure in adults is approximately 20mm to 30mm. The correspondence between palpebral fissure length and iris size is known. For example, Table 3 shows the relationship between palpebral fissure length and iris radius (denoted as R).
[0324] Table 3
[0325] palpebral fissure length Iris size (radius R) 20mm 5.5mm 25mm 6mm 30mm 6.5mm
[0326] In this embodiment, the length of the palpebral fissure can be determined based on a mask of the eye region. Then, based on the palpebral fissure length and the relationship between the palpebral fissure length and the iris size, the radius of the iris can be roughly estimated. Finally, based on the key points at the center of the eye and the estimated radius of the iris, the iris region can be determined.
[0327] For example, the eye fissure length can be determined to be 25mm based on the mask of the eye area. By referring to Table 1, the corresponding iris radius can be determined to be 6mm. Then, a circle with a radius of 6mm is drawn with the eye's central key point O as the center, thus determining the iris area of the human eye. For example, Figure 21 (c) shows a schematic diagram of the iris region of the eye, where R represents the estimated radius of the iris region.
[0328] S304, extract feature points of the iris region.
[0329] For example, Figure 22 A schematic flowchart illustrating the extraction of feature points from the iris region in this application is shown. Figure 22 This includes the following steps S3041-S3048.
[0330] S3041, Input iris region image.
[0331] For ease of explanation, assume that the iris region consists of N*N pixels.
[0332] S3042 enhances contrast in the iris region.
[0333] In this embodiment of the application, after the iris region is determined, the contrast of the iris region is enhanced to highlight the edge features, which facilitates the detection and extraction of feature points of the iris region.
[0334] S3043, transform to polar coordinates with (N / 2, N / 2) as the center.
[0335] It should be noted that, since the iris edge region is curved, feature point extraction at the iris edge is relatively difficult. Therefore, in this embodiment, the iris region is transformed to a polar coordinate system centered at (N / 2, N / 2) to obtain a 360*N / 2 polar coordinate image, thereby transforming the curved features into straight line features. Here, 360 represents 360 degrees.
[0336] Assume the iris region consists of N*N pixels. In a Cartesian coordinate system, the center point of the iris region can be represented as (x, y), where x = N / 2 and y = N / 2. Correspondingly, in a polar coordinate system, the coordinates of the center point of the iris region can be represented as (ρ, θ), where the radius coordinate ρ = 0 and the angular coordinate θ = 0. That is, the pole (0, 0) of the polar coordinate system is the center point of the iris region.
[0337] In this system, the ρ coordinate represents the distance from the pole, and the θ coordinate represents the angle from the 0° ray (also called the polar axis) in a counterclockwise direction. The polar axis is the positive x-axis in a Cartesian coordinate system. The angular coordinate θ is also called the polar angle or azimuth angle.
[0338] The value of ρ ranges from [0, N / 2]. The value of θ ranges from [0, 359].
[0339] S3044 extracts linear features using a linear filter.
[0340] In the embodiments of this application, a linear filter is used in the feature point extraction process, which can extract feature points of the iris region more accurately.
[0341] S3045, Input image features in the θ direction.
[0342] Where θ takes values of 0, 1, ..., 359 in sequence.
[0343] S3046, obtain feature points in the θ direction.
[0344] Since the edge variations of the iris are quite significant, feature points can be detected and extracted using peak detection for each angular direction. Specifically, peak detection can be performed on the eye region in polar coordinates to obtain feature points (ρ, θ) for each angular direction.
[0345] S3047, if θ≤359, return to continue executing S3045-S3046 until θ is 359.
[0346] The following example illustrates the process of sequentially acquiring feature points in each direction.
[0347] When θ is 0, feature points in the 0-angle direction are obtained based on the image features in the 0-angle direction.
[0348] Then, when θ is 1, feature points in the 1-angle direction are obtained based on the image features in the 1-angle direction.
[0349] Similarly, when θ is 359, feature points in the 359-degree angle direction are obtained based on the image features in the 359-degree angle direction.
[0350] This yields feature points in the iris region at various angles from 0 to 359.
[0351] S3048 outputs all feature points of the iris region.
[0352] Figure 23 The specific implementation process of extracting feature points of the iris region in steps S3041-S3048 is shown. Specifically, for an iris region image of size N*N pixels, it is transformed to a polar coordinate system centered at (N / 2, N / 2), and then linear features are extracted using a linear filter. θ starts from 0 and continues until it reaches 359, performing feature extraction in each direction. Peak detection (taking the local maximum value among edge feature points) can be performed on the eye region in the polar coordinate system to obtain edge feature points in each angular direction. All edge feature points in all angular directions form a circle or ellipse. These edge feature points, along with the feature points surrounded by them, constitute all the feature points of the iris region.
[0353] It should be noted that some of the feature points extracted from the iris region are actually covered or obscured by the upper and lower eyelids, therefore these feature points in the iris region need to be removed. Therefore, it is necessary to perform step S305 below, which involves filtering the feature points of the iris region.
[0354] S305, filter feature points in the iris region.
[0355] For example, Figure 24 A schematic flowchart illustrating the screening of feature points in the iris region in this application is shown. Figure 24 This includes steps S3051-S3055. After extracting all feature points in the iris region, feature points in the outer periphery and pupil region can be removed based on the roughly estimated iris radius. Feature points covered by the upper and lower eyelids in the iris region can be removed based on the mask of the eye region. Isolated points (outliers) can also be removed based on the connection length or area, thus achieving the screening of iris edge feature points.
[0356] The following describes the process of removing isolated points based on connectivity length or area. First, edge detection is performed on the iris region to extract the recognition point rectangles. Rectangles with an area less than a certain threshold are identified as noise points, and the pixel values within the noise point region are changed to the background color pixel values.
[0357] It should be noted that, in this embodiment, peak detection is performed on the eye region in polar coordinates to obtain feature points, and the results of face analysis and key point detection are introduced to filter feature points, which can improve the accuracy of eye detection.
[0358] It should be noted that, for ease of explanation, the screening of iris edge feature points by S305 is referred to as the first screening.
[0359] S306, by fitting an ellipse, the ellipse with the highest support is obtained, and a mask 1 for the iris region is generated based on the ellipse.
[0360] Figure 25 A schematic flowchart of the ellipse fitting method of this application is shown. Figure 25 This includes steps S3061-S3069.
[0361] Specifically, after completing the first screening of iris edge feature points, the initial estimation of the ellipse is achieved by least squares fitting for the iris edge feature points selected in the first screening. That is, the ellipse is initialized first to obtain the initialized ellipse.
[0362] Then, the distance error is calculated based on the distance between each feature point on the iris edge and the initial ellipse.
[0363] Then, feature points are selected based on the magnitude of the distance error. For example, if the distance error corresponding to feature point A in the iris region is greater than the error threshold (e.g., 1 pixel), feature point A is discarded; if the distance error corresponding to feature point B in the iris region is less than or equal to the error threshold, feature point B is retained. The total number of feature points discarded in this iteration is recorded (referred to as the feature point discard count), and denoted as N. The selection based on distance error is referred to as the second selection of iris edge feature points.
[0364] It should be noted that, through continuous filtering of distance errors, the number of feature points N removed becomes smaller and smaller. To improve computational efficiency, a removal threshold can be preset to determine whether the number of feature points removed in the current iteration meets the requirements. For example, the removal threshold can be 2. The specific value of the removal threshold can be set according to actual usage needs, and this embodiment does not limit it.
[0365] Then, it is determined whether the number of feature points to be removed, N, is less than or equal to the removal threshold. If the changes in some feature points at the iris edge exceed the removal threshold, then based on the feature points obtained after distance error filtering, an ellipse Z is fitted, and the gradient of the feature point pixels is used to calculate the support of all feature points (denoted as points) at the iris edge.
[0366]
[0367] Where S represents the support of all feature points at the iris edge, and p represents any one feature point among the feature points selected by distance error filtering. This represents the gradient of the ellipse at the feature point p. This represents the gradient of the image at feature point p.
[0368] It should be noted that the calculation process for the gradient of an ellipse is different from that for the gradient of an image. In this case, the coordinates of feature point p are (x, y).
[0369] gradient of an ellipse The cone equation is obtained by differentiating the corresponding conical equation of the fitted ellipse:
[0370] ellipse(x,y)=Ax 2 +Bxy+Cy 2 +Dx+Ey+F
[0371] Where x and y represent the coordinates of the X-axis and Y-axis, respectively, and A, B, C, D, E, and F are coefficients. The changes in the X-axis and Y-axis are the derivatives of the cone equation with respect to the X and Y directions, respectively.
[0372] Unlike the gradient of an ellipse, the pixel gradient change is as follows: The gradient of an image is the rate of change of a pixel (i.e., feature point p) in the X and Y directions (compared to neighboring pixels). It is a two-dimensional vector consisting of two components: the change along the X-axis and the change along the Y-axis. (1) The change along the X-axis is the difference between the pixel value to the right (X+1) and the pixel value to the left (X-1). (2) The change along the Y-axis is the difference between the pixel value below (Y+1) and the pixel value above (Y-1). By calculating these two components and forming a two-dimensional vector, the image gradient of the pixel (i.e., feature point p) is obtained.
[0373] It should be noted that when the gradient direction of a feature point is consistent with that of the ellipse and the image, the contribution of the feature point will increase the weight. Furthermore, the larger the gradient of the feature point in the image, the larger the weight, and consequently, the greater the support and the better the ellipse fitting effect.
[0374] Repeat the process from S3063 to S3067 until the changes in all feature points are less than the preset change threshold, and obtain the ellipse with the highest support.
[0375] The following examples illustrate the iterative process of S3063-S3067 and the ellipse fitting process.
[0376] First loop: Calculate the distance error between all feature points on the iris edge and the initialized ellipse. If the distance error of a feature point in the iris edge is greater than the error threshold, then that feature point is removed. By judging the distance error, a feature point set Y1 for the iris edge can be selected from all feature points on the iris edge, and the number of removed feature points is recorded. Then, it is determined whether the number of removed feature points in feature point set Y1 is less than or equal to the removal threshold. If the change in feature point set Y1 is greater than the removal threshold, an ellipse is fitted based on the feature point set Y1 of the iris edge to obtain ellipse Z1. Based on the distance and gradient from the feature point set Y1 to ellipse Z1, the support of the feature point set Y1 of the iris edge is calculated to obtain the support S1.
[0377] The second iteration calculates the distance error between the feature point set Y1 of the iris edge and the ellipse Z1. If the distance error of a feature point in the feature point set Y1 is greater than the error threshold, that feature point is removed. This distance error determination filters the feature point set Y2 of the iris edge from the feature point set Y1, and the number of removed feature points is recorded. Then, it is determined whether the number of removed feature points in feature point set Y2 is less than or equal to the removal threshold. If the change in feature point set Y2 is greater than the removal threshold, an ellipse is fitted based on the feature point set Y2 of the iris edge to obtain ellipse Z2. The support of the feature point set Y2 of the iris edge is calculated based on the distance and gradient from the feature point set Y2 to ellipse Z2, yielding the support S2.
[0378] The third iteration calculates the distance error between the feature point set Y2 of the iris edge and the ellipse Z2. If the distance error of a feature point in the feature point set Y2 of the iris edge is greater than the error threshold, then that feature point is removed. By judging the distance error, a feature point set Y3 of the iris edge is selected from the feature point set Y2, and the number of removed feature points is recorded. Then, it is determined whether the number of removed feature points in the feature point set Y2 is less than or equal to the removal threshold.
[0379] If the number of feature points removed from feature point set Y3 is less than or equal to the removal threshold, compare the support S1 and support S2. If support S2 is greater than support S1, then output the ellipse Z2 corresponding to support S2.
[0380] In this embodiment, a mask 1 for the iris region is generated based on the ellipse with the highest support.
[0381] S307, find the intersection of the iris region mask 1 and the eye region mask to obtain the iris region mask 2.
[0382] S308, perform edge fine-tuning on the mask 2 for the iris region to obtain the mask 3 for the iris region.
[0383] Optionally, such as Figure 26 As shown, the intersection of the iris region mask and the eye region mask obtained through face analysis is calculated to obtain iris region mask 1. Then, the edges of iris region mask 2 are fine-tuned to obtain iris region mask 3.
[0384] In this embodiment, an image segmentation (Graph Cut) algorithm can be used to fine-tune the edges of the iris region mask 2 to obtain the iris region mask 3. Image segmentation refers to the technique and process of dividing an image into regions with distinct characteristics and extracting the target of interest; it is a crucial step from image processing to image analysis. The basic idea of image segmentation is that similar and closely spaced pixels should be grouped into the same region.
[0385] It should be noted that this application uses an image segmentation algorithm to fine-tune the extracted coarse iris region, making the final extracted iris region more accurate.
[0386] S308, outputs the mask 3 for the iris region.
[0387] The above steps complete the iris masking process. After iris masking, the resulting materials include: the sclera and iris areas of a low-quality portrait, and the sclera and iris areas of a high-quality portrait with abnormal catchlights.
[0388] It should be noted that this application performs ellipse fine-tuning through repeated iterations: ellipse fitting is performed using all feature points, and feature points are selected based on their distance from the ellipse and the gradient of the feature point, thereby improving the accuracy of ellipse fitting.
[0389] It should be noted that the iris feature extraction method provided in this application can be applied to user photography scenarios. For example, the above embodiment is illustrated by the example of an electronic device automatically processing the catchlight of a person's eyes in a user photography scenario. Specifically, after the user triggers the photo, the background of the electronic device identifies whether the original image meets certain conditions (such as the original image containing a face and the image quality being below a certain threshold). If the original image meets certain conditions, the background automatically executes the iris feature extraction method provided in this application, so that the catchlight of the person's eyes in the final generated photo is normal or does not contain "abnormal" catchlight, thus correcting the unnatural catchlight caused by full-face enhancement through a face enhancement depth model.
[0390] It is understood that, in actual implementation, the application scenarios of the iris feature extraction method provided in this application embodiment include, but are not limited to, user photo-taking scenarios. For example, the iris feature extraction method can also be applied to photo editing scenarios in a photo library. For instance, the photo library application of an electronic device stores the original image and a high-quality portrait processed by a face enhancement depth model. In the photo editing interface of the electronic device's photo library, in response to user operation, the electronic device retrieves the photo selected by the user from the album or photo library and executes the iris feature extraction method provided in this application. The processed photo has normal or no "abnormal" catchlight in the eyes, correcting the unnatural catchlight caused by full-face enhancement through a face enhancement depth model. This satisfies the user's need for facial region fidelity in shooting scenarios such as low-light shooting and telephoto shooting, where low-quality portrait photos need to be enhanced by algorithms.
[0391] The iris feature extraction method provided in this application can also be applied to editing the eye light of portraits in mobile phone photo libraries, replacing the eye light of a specified image, replacing a specified eye light template, and enhancing / weakening / removing the eye light.
[0392] For example, the photo editing interface in the phone's gallery can provide one or more catchlight templates for replacing catchlights, or users can directly use two images for replacement. It can also provide controls for enhancing / weakening catchlights, replacing catchlights, and removing catchlights. Users can select one or more photos in the gallery and then click on different function controls to trigger the phone to edit the catchlights in the portraits in the photos, such as enhancing / weakening / replacing / removing catchlights.
[0393] It should be noted that the iris feature extraction method provided in this application embodiment can not only be applied to correct the catchlight in the eyes of human figures in photos, but also to modify the catchlight in the eyes of animals in photos.
[0394] It should also be noted that in the embodiments of this application, "greater than" can be replaced with "greater than or equal to", "less than or equal to" can be replaced with "less than", or "greater than or equal to" can be replaced with "greater than", and "less than" can be replaced with "less than or equal to".
[0395] The various embodiments described herein can be independent solutions or combinations thereof based on their inherent logic, and all such solutions fall within the protection scope of this application.
[0396] The foregoing mainly describes the solutions provided by the embodiments of this application from the perspective of method steps. It is understood that, in order to achieve the above functions, the electronic device implementing this method includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of protection of this application.
[0397] This application embodiment can divide an electronic device into functional modules based on the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, other feasible division methods may exist. The following description uses the division of functional modules according to each function as an example.
[0398] Figure 27 This is a schematic block diagram of an iris feature extraction device 400 provided in an embodiment of this application. The device 400 can be used to perform the actions performed by the electronic device in the above method embodiments. The device 400 includes an image acquisition unit 410, an iris recognition unit 420, a feature point extraction unit 430, and an image processing unit 440.
[0399] The image acquisition unit 410 is used to acquire a first image and a second image when a preset condition is met, wherein the second image is an image obtained by applying a facial enhancement depth model to the first image;
[0400] The iris recognition unit 420 is used to perform feature recognition on the image to be processed and determine the iris region; the image to be processed is the first image or the second image.
[0401] The feature point extraction unit 430 is used to extract feature points at the edge of the iris region in polar coordinates.
[0402] Image processing unit 440, for:
[0403] Calculate the distance error between each of the G feature points and the first ellipse; wherein the G feature points are the feature points extracted for the first time from the edge of the iris region, and the first ellipse is an ellipse fitted based on the feature points extracted for the first time from the edge of the iris region; or the G feature points are the feature points remaining after the previous removal, and the first ellipse is an ellipse fitted based on the feature points remaining after the previous removal.
[0404] K feature points are removed from the G feature points, and the distance error between each of the K feature points and the first ellipse is greater than a preset error threshold.
[0405] When K is greater than the preset rejection threshold, a second ellipse is obtained by fitting GK feature points, and the support corresponding to the second ellipse is calculated; wherein, the GK feature points and the second ellipse are used together to calculate the distance error between the feature points and the ellipse in the next calculation.
[0406] The distance error between feature points and ellipses is calculated iteratively, and feature points are removed based on the distance error. Multiple ellipses are then fitted, and the support of each of the multiple ellipses is calculated.
[0407] When the number of removed feature points is less than or equal to the preset removal threshold, the target ellipse with the highest support among the multiple ellipses is obtained.
[0408] The image features corresponding to the target ellipse are extracted from the image to be processed and used as the image features of the iris region.
[0409] The iris feature extraction device provided in this application, after identifying the iris region in an image, extracts feature points at the edge of the iris region in a polar coordinate system. First, feature points are filtered based on the face analysis results, then ellipse fitting is performed. Next, some feature points are removed based on the distance error between the feature points and the ellipse. Then, an ellipse is fitted again based on the remaining feature points, and the corresponding support is calculated. The distance error is then calculated again based on the remaining feature points and the fitted ellipse. This process of calculating the distance error and removing feature points based on the distance error is repeated, resulting in multiple fitted ellipses and multiple support values. When the number of removed feature points is less than or equal to a threshold, the image features of the iris region are extracted based on the ellipse with the highest support. This application filters feature points based on the distance error from the feature points to the ellipse, iteratively fine-tuning the ellipse to improve the accuracy of ellipse fitting, thereby improving the accuracy of iris extraction and facilitating accurate processing of eye light in subsequent steps.
[0410] The apparatus 400 according to the embodiments of this application can correspond to the execution of the method described in the embodiments of this application, and the above and other operations and / or functions of the units in the apparatus 400 are respectively for implementing the corresponding process of the method, which will not be repeated here for the sake of brevity.
[0411] This application also provides a chip coupled to a memory, which is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments.
[0412] This application also provides an electronic device including a chip for reading and executing computer programs or instructions stored in a memory, causing the methods in the various embodiments to be performed.
[0413] This embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the iris feature extraction method in the above embodiment.
[0414] This embodiment also provides a computer program product. The computer-readable storage medium stores program code. When the computer program product is run on a computer, it causes the computer to perform the above-mentioned related steps to realize the iris feature extraction method in the above embodiment.
[0415] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to execute the iris feature extraction method in the above-described method embodiments.
[0416] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.
[0417] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0418] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. An iris feature extraction method, characterized by, include: When preset conditions are met, a first image and a second image containing facial features are acquired. The second image is an image obtained by performing facial enhancement processing on the first image using a facial enhancement depth model. The image to be processed is subjected to feature recognition to determine the iris region; the image to be processed is either the first image or the second image. Extract feature points at the edge of the iris region in polar coordinates; Calculate the distance error between each of the G feature points and the first ellipse; wherein the G feature points are the feature points extracted for the first time from the edge of the iris region, and the first ellipse is an ellipse fitted based on the feature points extracted for the first time from the edge of the iris region; or the G feature points are the feature points remaining after the previous removal, and the first ellipse is an ellipse fitted based on the feature points remaining after the previous removal. K feature points are removed from the G feature points, and the distance error between each of the K feature points and the first ellipse is greater than a preset error threshold. When K is greater than the preset rejection threshold, a second ellipse is obtained by fitting GK feature points, and the support corresponding to the second ellipse is calculated based on the gradient of the second ellipse at the GK feature points and the gradient of the image to be processed at the GK feature points. Where S represents the support corresponding to the second ellipse. (p) represents the gradient of the second ellipse at feature point p. (p) represents the gradient of the image to be processed at feature point p, where p is each of the GK feature points; wherein the GK feature points and the second ellipse are used together for the next calculation of the distance error between the feature point and the ellipse; The distance error between feature points and ellipses is calculated iteratively, and feature points are removed based on the distance error. Multiple ellipses are then fitted, and the support of each of the multiple ellipses is calculated. When the number of removed feature points is less than or equal to the preset removal threshold, the target ellipse with the highest support among the multiple ellipses is obtained. The image features corresponding to the target ellipse are extracted from the image to be processed and used as the image features of the iris region.
2. The method according to claim 1, characterized in that, The preset condition is: in response to the user's first operation on the shutter button, the first image is captured, the first image includes facial features, and the image quality of the first image is lower than or equal to the preset image quality. Alternatively, the preset condition is: receiving a second operation from a user on the second image in the image library, the second image including facial features and eye catchlight features, the second operation being used to adjust the eye catchlight features.
3. The method according to claim 1, characterized in that, After obtaining the target ellipse with the highest support among the plurality of ellipses, and before extracting the image features corresponding to the target ellipse from the image to be processed as the image features of the iris region, the method further includes: A first mask for the iris region is generated based on the target ellipse; The intersection between the first mask and the mask for the eye region is obtained to obtain the second mask for the iris region; An image segmentation algorithm is used to fine-tune the edges of the second mask of the iris region to obtain the third mask of the iris region; The step of extracting the image features corresponding to the target ellipse from the image to be processed as the image features of the iris region includes: extracting image features from the image to be processed according to the third mask of the iris region as the image features of the iris region.
4. The method according to claim 1, characterized in that, After extracting feature points at the edge of the iris region in polar coordinates, the method further includes: Based on the mask of the eye region, remove the feature points covered by the upper and lower eyelids from the feature points at the edge of the iris region; Based on the iris radius length, feature points located on the periphery of the iris are removed from the feature points at the edge of the iris region.
5. The method according to claim 1, characterized in that, The step of performing feature recognition on the image to be processed to determine the iris region includes: The image to be processed is input into a first preset network model, which outputs a mask for the eye region and key points at the center of the eye; wherein, the first preset network model is a network model used for parsing the face and detecting facial key points; The length of the first palpebral fissure is determined based on the mask of the eye region; The iris radius length is determined based on the first palpebral fissure length and a preset first correspondence relationship; wherein, the first correspondence relationship includes the correspondence between the first palpebral fissure length and the iris radius length; Based on the iris radius and the eye center key point, the iris region is determined in the image to be processed; wherein the iris region is a circular area centered on the eye center key point and with the iris radius as the radius.
6. The method according to any one of claims 1 to 5, characterized in that, After performing feature recognition on the image to be processed and determining the iris region, the method further includes: Enhance the contrast of the iris region to highlight its edge features.
7. The method according to any one of claims 1 to 5, characterized in that, After performing feature recognition on the image to be processed and determining the iris region, and before extracting feature points at the edge of the iris region in polar coordinates, the method further includes: Transform the iris region from the Cartesian coordinate system to the polar coordinate system; In the rectangular coordinate system, the pixel coordinates in the iris region are represented as (x, y), where the values of x and y are both in the range of [0, N]. In the polar coordinate system, the pixel coordinates in the iris region are represented as (ρ, θ), where the polar radius ρ ranges from [0, N / 2] and the polar angle θ ranges from [0, 360].
8. The method according to any one of claims 1 to 5, characterized in that, The step of extracting feature points at the edge of the iris region in polar coordinates includes: In the polar coordinate system, linear features of the image to be processed are extracted using a linear filter; By performing peak detection on the linear features of the image to be processed, each feature point with a polar radius ρ of N / 2 and a polar angle θ in the range of [0, 360) is obtained, thus obtaining the feature points at the edge of the iris region.
9. The method according to any one of claims 1 to 5, characterized in that, After extracting the image features corresponding to the target ellipse from the image to be processed as the image features of the iris region, the method further includes: A first background region and a first highlight region are separated from the eye region of the first image, and a second background region and a second highlight region are separated from the eye region of the second image; If catchlight features exist in the eye region, the image features of the first highlight region are enhanced to obtain enhanced catchlight features; and the enhanced catchlight features are then fused with the image features of the second background region to obtain a third image. If the eye light feature is not present in the eye region, a fourth image is obtained based on the image features of the second background region.
10. The method according to claim 9, characterized in that, The step of separating the first background region and the first highlight region from the eye region of the first image includes: Upsample the eye region of the first image to obtain an enlarged eye region; Based on the highlight removal algorithm, the first background region and the first highlight region are separated from the enlarged eye region; The first background region and the first highlight region are downsampled respectively; The downsampling reduction ratio is the reciprocal of the upsampling magnification ratio.
11. The method according to claim 10, characterized in that, The upsampling of the eye region in the first image to obtain an enlarged eye region includes: Anti-aliased upsampling and filtering noise reduction are performed on the eye region of the first image; If the enlarged size of the eye region of the first image is smaller than a preset size threshold, the eye region of the first image is again subjected to anti-aliasing upsampling and filtering for noise reduction. The enlarged eye region is obtained when the size of the enlarged eye region in the first image is greater than or equal to the preset size threshold.
12. The method according to claim 9, characterized in that, The enhancement process for the image features of the first highlight region to obtain enhanced catchlight features includes: Obtain the attribute parameters of the first highlight region, including size, shape, average brightness, and / or the brightness of the brightest point; The first enhancement parameter is determined based on the attribute parameters of the first highlight region; Using the first enhancement parameter, the image features of the first highlight region are enhanced to obtain the enhanced catchlight features.
13. The method according to claim 9, characterized in that, The step of fusing the enhanced eye light features with the image features of the second background region to obtain a third image includes: The affine transformation matrix is determined based on the edge key points of the first background region and the edge key points of the second background region. The enhanced catchlight feature is subjected to an affine transformation based on the affine transformation matrix to obtain an enhanced catchlight feature aligned with the position of the second background region. The aligned enhanced catchlight features are then fused with the image features of the second background region to obtain the third image.
14. An electronic device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory, the processor executing the computer program to cause the electronic device to perform the method as described in any one of claims 1 to 13.
15. A chip comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, The processor is used to execute the computer program to implement the method as described in any one of claims 1 to 13.
16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 13.
Citation Information
Patent Citations
Ellipse-detection-based iris detection method
CN106485210A
Iris recognition method and terminal
CN107292242A
