Eye position determination method, image processing device and image processing system therefor

Abstract

The present application provides an eye positioning method, an image processing device and an image processing system. The method comprises the following steps. A face image of a user is obtained, wherein the face image comprises an unobstructed face region and an obstructed face region, and the unobstructed face region comprises an eye region. At least one unobstructed face feature is detected from the unobstructed face region, and at least one obstructed face feature is estimated from the obstructed face region. Then, the position of the eye is located according to the unobstructed face feature and the obstructed face feature.

Description

Technical Field

[0001] This invention relates to a technique for eyeball localization. Background Technology

[0002] During the COVID-19 pandemic or seasonal flu, people wear masks to reduce the risk of infection. However, facial recognition cannot achieve its intended recognition effect because facial features are obscured, making it impossible to accurately perform eye tracking afterwards. Summary of the Invention

[0003] This invention provides an eyeball localization method, an image processing device, and an image processing system thereof.

[0004] In an exemplary embodiment of the present invention, the method described above includes the following steps: Obtaining a user's face image, wherein the face image includes an unoccluded face region and an occluded face region, the unoccluded face region including an eye region; detecting at least one unoccluded face feature from the unoccluded face region, and estimating at least one occluded face feature from the occluded face region; and then locating the eyeball position based on the unoccluded face feature and the occluded face feature.

[0005] In an exemplary embodiment of the present invention, the image processing apparatus described above includes a memory and a processor, wherein the processor is connected to the memory. The memory is used to store data. The processor is used to: acquire a user's face image, wherein the face image includes an unoccluded face region and an occluded face region, wherein the unoccluded face region includes an eye region; detect at least one unoccluded face feature from the unoccluded face region; estimate at least one occluded face feature from the occluded face region; and locate the eyeball position based on the unoccluded face feature and the occluded face feature.

[0006] In an exemplary embodiment of the present invention, the image processing system described above includes an image capture device and an image processing apparatus, wherein the image processing apparatus is connected to the image capture device. The image capture device is used to capture a user's face image, wherein the face image includes an unoccluded face region and an occluded face region, wherein the unoccluded face region includes an eye region. The image processing apparatus is used to: acquire a face image from the image capture device; detect at least one unoccluded face feature from the unoccluded face region; estimate at least one occluded face feature from the occluded face region; and locate the eyeball position based on the unoccluded face feature and the occluded face feature.

[0007] To make the present invention more apparent and understandable, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings. Attached Figure Description

[0008] The accompanying drawings are included to further illustrate the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0009] Figure 1 This is a schematic diagram of an image processing system according to an exemplary embodiment of the present invention.

[0010] Figure 2 This is a flowchart illustrating an eye positioning method according to an exemplary embodiment of the present invention.

[0011] Figure 3 This is a flowchart illustrating an eye positioning method according to an exemplary embodiment of the present invention.

[0012] Figure 4 This is a functional flowchart illustrating facial feature restoration according to an exemplary embodiment of the present invention.

[0013] Figure 5 This is a functional flowchart illustrating the process of generating a repaired image according to an exemplary embodiment of the present invention.

[0014] Figure 6 This is a functional flowchart illustrating eye localization for image restoration according to an exemplary embodiment of the present invention.

[0015] Figure 7 This is a functional flowchart illustrating an eye localization method according to another exemplary embodiment of the present invention.

[0016] Explanation of icon numbers

[0017] 100: Image Processing System

[0018] 110: Image capture equipment

[0019] 120: Image processing device

[0020] 122: Memory

[0021] 124: Processor

[0022] S202~S208, S302~S314, S702~S712: Steps

[0023] I41, I42, I43, I51, I52, I61, I63, I65~I68, I7: Images

[0024] M1~M5: Simulation Models

[0025] MB: Simulation Model Database

[0026] D6: Test Results

[0027] A6: Eye Feature Anchor Points

[0028] B6: Outer frame

[0029] E6, E7: Center of the eyeball

[0030] B7: Eye area Detailed Implementation

[0031] The following description will detail some exemplary embodiments of the present invention with reference to the accompanying drawings. Component symbols used in the following description, when appearing in different drawings, are considered to be the same or similar components. These exemplary embodiments are only a part of the present invention and do not disclose all possible implementations of the invention. More precisely, these exemplary embodiments are merely examples of the methods, apparatus, and systems described in the claims of the present invention.

[0032] Figure 1 This is a schematic diagram of an image processing system according to an exemplary embodiment of the present invention. First Figure 1 First, we will introduce the various components of the system and their configuration relationships. Detailed functions will be explained in conjunction with the flowcharts in the subsequent example embodiments. Figure 1 And reveal.

[0033] Please refer to Figure 1 The image processing system 100 in this exemplary embodiment includes an image capture device 110 and an image processing device 120. The image capture device 110 can be, for example, a digital camera, digital camcorder, webcam, surveillance camera, etc. The image processing device 120 can be, for example, a personal computer, laptop computer, server computer, tablet computer, smartphone, workstation, or other computer system or platform that can be wirelessly or wiredly connected to the image capture device 110 via a communication interface, wherein the communication interface can be a transmission interface that supports any wired connection or wireless communication standard for data transmission with other devices. In another exemplary embodiment, the image processing system 100 can be an electronic device that integrates all components into a single electronic device, such as a smartphone, tablet computer, laptop computer, or other electronic device with image capture and image processing functions. In another exemplary embodiment, the image processing device 120 can be an embedded system built into the image capture device 110; this invention is not limited thereto.

[0034] Image capturing device 110 is used to capture images and includes a camera lens having a lens and a photosensitive component. The photosensitive component is used to sense the intensity of light entering the lens, thereby generating an image. The photosensitive component can be, for example, a charge-coupled device (CCD), a complementary metal-oxide-semiconductor (CMOS) component, or other components.

[0035] The image processing apparatus 120 includes a memory 122 and a processor 124. The memory 122 stores data such as images and program code, and can be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or other similar devices, integrated circuits, or combinations thereof. The processor 124 controls the operation between the components of the image processing system 100, and can be, for example, a central processing unit (CPU), an application processor (AP), or other programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), image signal processors (ISPs), graphics processing units (GPUs), or other similar devices, integrated circuits, or combinations thereof.

[0036] Figure 2 This is a flowchart illustrating an eye localization method according to an exemplary embodiment of the present invention. Figure 2 The method and process can be derived from Figure 1 The image processing system 100 is used for implementation. The following process will be described using a user wearing a mask, but the invention is not limited thereto. In other exemplary embodiments, eye positioning can also be performed on users wearing sun protection devices or protective equipment such as face shields, helmets, or head coverings. That is, the following method can be used as long as the user's eyes are not obstructed.

[0037] Please refer to the following at the same time Figure 1 as well as Figure 2The processor 124 of the image processing device 120 of the image processing system 100 acquires the user's face image from the image capture device 110 (step S202). The face image here includes a face area covered by a mask (hereinafter referred to as "covered face area") and a face area not covered by a mask (hereinafter referred to as "uncovered face area"), wherein the uncovered face area includes the user's eye area.

[0038] Next, the processor 124 of the image processing device 120 detects at least one unmasked face feature from the unmasked face region (step S204), and estimates at least one masked face feature from the masked face region (step S206). The unmasked face feature can be identified using general facial feature detection methods, while the masked face feature, which is covered by a mask, needs to be estimated through feature restoration or feature estimation.

[0039] Subsequently, the processor 124 of the image processing device 120 locates the eyeball position based on the unmasked face features and the masked face features (step S208). In other words, the processor 124 can use the detected unmasked face features and the estimated masked face features to determine the eyeball position, and then calculate the center of the eyeball to achieve accurate positioning.

[0040] For clarity, the following exemplary embodiments will be used to illustrate different application scenarios of the image processing system 100. Figure 2 Details of the process. In the following exemplary embodiments, the description also uses a user wearing a mask, however, the present invention is not limited thereto.

[0041] Figure 3 This is a flowchart illustrating an eye localization method according to an exemplary embodiment of the present invention. It should be noted that before proceeding to this process, the memory 122 of the image processing device 120 has pre-stored a face detection model to assist in subsequent face feature detection. This face detection model is trained using a large number of face images (e.g., 5000) through a face feature detection algorithm via machine learning. These face images include unobstructed face images taken from different angles and occluded face images. Furthermore, the memory 122 of the image processing device 120 can store a pre-established simulation model database for repairing occluded face features; details of this will be explained later.

[0042] Please refer to the following at the same time Figure 1 as well as Figure 3The processor 124 of the image processing device 120 of the image processing system 100 acquires the user's face image captured by the image capture device 110 (step S302). Next, the processor 124 uses the trained face detection model to perform facial feature detection on the face image (step S304), and then detects information of unmasked face features from the unmasked face region in the face image.

[0043] Furthermore, the processor 124 can select the most appropriate simulation model from the simulation model database based on the user's head orientation angle in the face image (step S306). Then, the processor 124 will perform feature repair based on the unoccluded face features (step S308) to repair the occluded face region for subsequent full-face feature detection (step S310).

[0044] Specifically Figure 4 This is a functional flowchart illustrating facial feature restoration according to an exemplary embodiment of the present invention.

[0045] Please refer to the following at the same time Figure 1 as well as Figure 4 Here, in the face image I41 obtained by processor 124, the user's head orientation angle is "20 degrees to the left". On the other hand, the simulation model database MB stores multiple simulation models based on face images with different skin tones and face shapes at multiple different head orientation angles. For example, simulation model M1 is "10 degrees to the left", simulation model M2 is "40 degrees to the left", simulation model M3 is "0 degrees to the front", simulation model M4 is "25 degrees to the left", and simulation model M5 is "40 degrees to the right". Processor 124 can determine that the head orientation angle of simulation model M4 is closest to that of face image I41. Therefore, processor 124 will repair the occluded face area of ​​face image I41 based on simulation model M4 to generate repaired image I42. Then, processor 124 can use any facial feature detection algorithm to perform full-face feature detection on repaired image I42 to obtain unoccluded face features and occluded face features, as shown in image I43. From another perspective, unmasked facial features come from face image I41, while masked facial features come from restored image I42.

[0046] It should be noted that the process of generating the repaired image I42 can be achieved through translation and scaling steps. Specifically, Figure 5 This is a functional block diagram illustrating the process of generating a repaired image according to an exemplary embodiment of the present invention.

[0047] Please refer to the following at the same time Figure 1 as well as Figure 5The processor 124 obtains the coordinates P1(x1,y1) and P2(x2,y2) of the bounding box of the eye region from the face image I51. The processor 124 then selects the simulation model M3 (with a resolution of W×H) corresponding to the face image I51 from the simulation model database, and obtains the coordinates P3(x3,y3) and P4(x4,y4) of the bounding box of the eye region from the simulation model M3. The processor 124 then repairs the occluded face region of the face image I51 based on the coordinates P7(x7,y7) and P8(x8,y8) of the simulation repair region in the simulation model M3, to generate the coordinates P5(x5,y5) and P6(x6,y6) of the feature repair region in the repaired image I52. Here, each coordinate value must satisfy the following relationship:

[0048] x5=xl, x6=x2, x7=x3, x8=x4

[0049] y5 = y2, y7 = y4

[0050] y8=H

[0051] From the proportional relationship, we can obtain:

[0052]

[0053] so:

[0054]

[0055] This yields the following translation and / or scaling relationships:

[0056]

[0057] It should be further explained that the head orientation angle includes both the head rotation direction and the head rotation angle. In an exemplary embodiment, the processor 124 determines the head orientation angle of the face image I51 based on the bounding box of the eye region. Specifically, the processor 124 can calculate the turning index based on the grayscale distribution of the bounding box of the eye region to determine whether the user's head is facing left or right. When the head is facing left, the distance between the right eye and the face boundary (e.g., the temple) becomes shorter, while the distance between the left eye and the right eye does not change significantly. This slight difference can be calculated using a weighted mechanism to determine the head orientation. Furthermore, the processor 124 estimates the user's head rotation angle based on the aspect ratio of the bounding box of the eye region, where the head rotation angle and the aspect ratio of the bounding box of the eye region can be related as follows:

[0058] Head rotation angle = Reference angle ratio × (Depth-to-width ratio - Depth-to-width ratio when facing forward) 幂次

[0059] When facing forward, the aspect ratio of the bounding box for the eye area is small, and the head angle is 0 degrees. When facing sideways, as the head angle increases, the aspect ratio of the bounding box for the eye area also increases. Therefore, in an exemplary embodiment, the above parameters can be set as follows:

[0060] 250 degrees < reference angle ratio < 300 degrees, with an optimal value of 270 degrees;

[0061] When the aspect ratio is less than 0.3 for a frontal view, it should be less than 0.35, with an optimal value of 0.333.

[0062] 0.8 < power < 1.2, the optimal value is 1.0.

[0063] In another exemplary embodiment, processor 124 estimates the user's head rotation angle based on a mask (or other occluding object) on the occluded face area of ​​face image I51. For example, processor 124 can estimate the head rotation angle by the outline of the mask and the distribution of the shadow areas of the mask. Processor 124 can also determine the bridge of the nose position by the connectivity characteristics of the bright areas of the mask, and then estimate the head rotation angle.

[0064] Please return Figure 3 The processor 124 will perform full-face feature detection, and then detect eye feature anchor points. Next, the processor 124 can use a clustering algorithm (step S312) to perform region clustering (e.g., black eye region, white eye region, skin region), and calculate the eye center from the clustered eye position (step S314). In this embodiment, the clustering algorithm used first converts the image from RGB color space to HSV color space, and then converts the HSV color space to cone space to generate a custom HSV color space (hereinafter referred to as HSV+) image, and then performs clustering calculation.

[0065] In detail, the conversion method between the RGB color space and the HSV color space used in this example embodiment is as follows:

[0066]

[0067] v = max

[0068] The conversion methods between HSV color space and HSV+ color space are as follows:

[0069] x = s × cos h

[0070] y = S × sin h

[0071]

[0072] Where (r, g, b) are the coordinates of a color in the RGB color space for red, green, and blue, respectively, with values ​​between 0 and 1. max and min are the maximum and minimum values ​​among r, g, and b, respectively. h ∈ [0, 360°) is the hue angle, and s and v ∈ [0, 1] are the saturation and lightness, respectively. (x, y, z) are the coordinates in the HSV+ color space.

[0073] In this example embodiment, after conversion between the RGB and HSV+ color spaces, the eye region data can be clustered using the K-means clustering algorithm to generate the black area, white area, and skin area. The coordinates of the eyeball center are then calculated from the black area. Regarding the positioning accuracy of the eyeball center, compared to the 12-13 pixel error of the RGB color space, 12-13 pixel error of the YUV color space, 10-11 pixel error of the Lab color space, and 13-14 pixel error of the HSV color space, the HSV+ color space used in this example embodiment has only a 4-5 pixel error. This not only effectively clusters the eye region captured in bright or dim environments but also supports close-range applications at 15-30 cm.

[0074] Overall, Figure 6 This is a functional flowchart illustrating eye localization for image restoration according to an exemplary embodiment of the present invention. Figure 6 The process can be regarded as Figure 5 The next steps.

[0075] Please refer to the following at the same time Figure 1 as well as Figure 6 In this exemplary embodiment, after acquiring the repaired image I61, the processor 124 will detect facial feature anchor points, generating a detection result D6 for facial feature anchor points, and capture an image I63 of the eye feature anchor point A6 from it. The minimum bounding box B6 is then defined by the image I63 of the eye feature anchor point A6. Next, the processor 124 will capture the image I65 of the minimum bounding box and perform HSV+ color space conversion to generate an image I66 of the minimum bounding box in HSV+ color space, for grouping including the black area of ​​the eye, the white area of ​​the eye, and the skin area. The processor 124 will filter out the image I67 of the black area of ​​the eye to calculate the coordinates of the eyeball center, thereby obtaining the coordinates of the eyeball center E6 of the image I68 of the eye region as the positioning result. It should be noted that, assuming the size of the aforementioned minimum bounding box B6 is W×H, the minimum bounding box B6 can be replaced by a bounding box with a size between W×H and (W+2H)×(3H), and this invention is not limited thereto.

[0076] Figure 7This is a functional flowchart illustrating an eye localization method according to another exemplary embodiment of the present invention. In this exemplary embodiment, feature repair of the occluded facial region can be replaced by analysis of the mask outline and connectivity properties to directly calculate the eye center.

[0077] Please refer to the following at the same time Figure 1 as well as Figure 7 In this exemplary embodiment, the processor 124 will acquire the face image I7 (step S702) and perform mask contour detection and mask connectivity analysis (step S704). The processor 124 can first obtain the mask contour through mask contour analysis, then obtain multiple corner points of the mask, and then perform connectivity analysis on the mask covering the face area to predict the occluded face features. For example, the processor 124 can deduce the position of the bridge of the nose point through the connectivity characteristics of the bright area outside the mask, and then calculate the position of the center of the eyebrows. After the processor 124 has completed the calculation of the bridge of the nose point, the center of the eyebrows, and the corner points of the mask (step S706), the processor 124 can calculate the eye region B7 (step S708). Here, the processor 124 can calculate the coordinates of the same height position based on the height of the center of the eyebrows, extending vertically upward from the corner points of the mask on both sides, and then calculate the eye region B7 from these corner points. Next, the processor 124 processes the eye region B7 to calculate the corner point of the eye (step S710), and then calculates the center of the eyeball E7 (step S712).

[0078] In one exemplary embodiment, the above-described eye-tracking method can be implemented, for example, in a medical setting with a transparent display. Even when the surgeon is wearing a mask, the information needed during the surgery can still be rendered on the transparent display based on the surgeon's line of sight. In another exemplary embodiment, the above-described eye-tracking method can also be implemented, for example, in a retail or display setting with a transparent display. Product information or narration can be presented on the transparent display according to the user's line of sight to provide an immersive experience. However, the above scenarios are merely illustrative and the present invention is not intended to be limiting.

[0079] The eye positioning method, image processing device, and image processing system proposed in the exemplary embodiments of the present invention can determine the position of the eyeballs even when the user's face is covered by an object such as a mask. They can use the detected uncovered facial features and the estimated occluded facial features to determine the position of the eyeballs and then calculate the center of the eyeballs to achieve accurate eye positioning.

[0080] Finally, it should be noted that the above exemplary embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing exemplary embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing exemplary embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the exemplary embodiments of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims and their equivalents.

Claims

1. An eyeball localization method, characterized in that, include: A user's face image captured by an image capture device is obtained, and a face detection model is used to perform facial feature detection on the face image, wherein the face image includes an unoccluded face region and an occluded face region, wherein the unoccluded face region includes the eye region; From the uncovered face area, detect at least one uncovered face feature; From the area where the face is obscured, at least one obscured facial feature is estimated, including: Based on the outline of the occluded object within the occluded face area, obtain multiple object corner points of the occluded object; and Image connectivity analysis is performed on the occluded face region to predict the occluded face features; The eye region is located based on the multiple object corner points, the unoccluded facial features, and the occluded facial features; and The system groups multiple pixels in the eye region to generate an eye mesentery region, an eye sclera region, and a skin region. It then calculates the coordinates of the eyeball center in the eye mesentery region to locate the eyeball position.

2. The method according to claim 1, characterized in that, Also includes: A simulation model database is established, which includes multiple simulation models, each based on multiple different facial images with different head orientation angles.

3. The method according to claim 2, characterized in that, The step of estimating the occluded facial features from the occluded face area includes: Determine the angle of the user's head orientation in the face image to select the corresponding first simulation model from the simulation models; Using this first simulation model, the occluded face area is repaired to generate a repaired image; and The masked facial features are estimated from the repaired image.

4. The method according to claim 3, characterized in that, The user's head orientation angle in the face image includes the head rotation direction, which is determined based on the distance between the eyeball in the eye region and the boundary of the face.

5. The method according to claim 3, characterized in that, The user's head orientation angle in the face image includes the head rotation angle, which is determined based on the aspect ratio of the eye region.

6. The method according to claim 5, characterized in that, The head rotation angle is determined based on the aspect ratio of the eye region in the face image, the reference angle ratio, the aspect ratio of the frontal face, and the power. The reference angle ratio is between 250 degrees and 300 degrees, the aspect ratio of the frontal face is between 0.3 and 0.35, and the power is between 0.8 and 1.

2.

7. The method according to claim 3, characterized in that, The user's head orientation angle in the face image includes a head rotation angle, which is determined based on the outline of the occluded object in the occluded face area.

8. The method according to claim 3, characterized in that, The user's head orientation angle in the face image includes the head rotation angle, which is determined based on the distribution of the shadow area of ​​the occluded object on the occluded face area.

9. The method according to claim 3, characterized in that, The user's head orientation angle in the face image includes a head rotation angle, which is determined based on the connectivity characteristics of the external bright area of ​​the occluded object on the occluded face area.

10. The method according to claim 3, characterized in that, The steps for repairing the occluded face region using the first simulation model to generate the repaired image include: The simulation restoration area corresponding to the occluded face region is captured by the first simulation model; and The restored image is generated based on the unmasked face area and the simulated restoration area.

11. The method according to claim 10, characterized in that, The steps for generating the restored image based on the unmasked face region and the simulated restoration region include: The simulated repair area is translated and / or scaled to generate a feature-repaired area; and The repaired image is generated using the unmasked face region and the feature repair region.

12. The method according to claim 1, characterized in that, Before grouping the pixels in the eye region, the method further includes: The pixel is converted to a custom HSV color space, which is defined as an HSV color space converted from the HSV color space to the cone space.

13. The method according to claim 12, characterized in that, The step of grouping the pixels in the eye region includes: The pixels converted to the custom HSV color space are grouped to generate regions including the black area of ​​the eye, the white area of ​​the eye, and the skin area.

14. The method according to claim 12, characterized in that, The steps to locate the eyeball include: Calculate the center position of the black region of the eye to determine the location of the eyeball.

15. The method according to claim 1, characterized in that, The steps to locate the eyeball include: Calculate multiple eye corner points in this eye region; and The position of the eyeball is calculated based on the corner point of the eye.

16. An image processing apparatus, characterized in that, include: A memory used to store data; as well as The processor, connected to the memory, is used to: A user's face image captured by an image capture device is obtained, and a face detection model is used to perform facial feature detection on the face image, wherein the face image includes an unoccluded face region and an occluded face region, wherein the unoccluded face region includes the eye region; From the uncovered face area, detect at least one uncovered face feature; From the area where the face is obscured, at least one obscured facial feature is estimated, including: Based on the outline of the occluded object within the occluded face area, obtain multiple object corner points of the occluded object; and Image connectivity analysis is performed on the occluded face region to predict the occluded face features; The eye region is located based on the multiple object corner points, the unoccluded facial features, and the occluded facial features; and The system groups multiple pixels in the eye region to generate an eye mesentery region, an eye sclera region, and a skin region. It then calculates the coordinates of the eyeball center in the eye mesentery region to locate the eyeball position.

17. An image processing system, characterized in that, include: An image capture device is used to capture a user's face image and perform facial feature detection on the face image using a face detection model, wherein the face image includes an unoccluded face region and an occluded face region, wherein the unoccluded face region includes an eye region. An image processing device, connected to the image capturing device, is used for: The face image is obtained from the image capture device; From the uncovered face area, detect at least one uncovered face feature; From the area where the face is obscured, at least one obscured facial feature is estimated, including: Based on the outline of the occluded object within the occluded face area, obtain multiple object corner points of the occluded object; and Image connectivity analysis is performed on the occluded face region to predict the features of the occluded face; and The eye region is located based on the multiple object corner points, the unoccluded facial features, and the occluded facial features; and The system groups multiple pixels in the eye region to generate an eye mesentery region, an eye sclera region, and a skin region. It then calculates the coordinates of the eyeball center in the eye mesentery region to locate the eyeball position.

Images