Eyeball depth image restoration method, system and equipment

Abstract

The invention discloses an eyeball depth image restoration method, system and device. The restoration method comprises the steps of obtaining an eye opening color image, an eye opening depth image, an eye closing color image and an eye closing depth image; respectively carrying out key point extraction on the eye-opening color image and the eye-closing color image to obtain eye-opening key point information and eye-closing key point information; respectively performing depth information extraction on the eye opening depth image and the eye closing depth image according to the eye opening key point information or the eye closing key point information to obtain eye opening depth information and eye closing depth information; according to the eye opening key point information and the eye closing key point information, obtaining a position conversion matrix between the eye opening color image and the eye closing color image; and according to the eye opening key point information, the eye opening depth information, the eye closing depth information and the position conversion matrix, obtaining an eyeball depth image after the to-be-repaired eyeball is repaired. The eye closing depth information and the eye opening depth information are aligned and fused, so that the problem of incomplete depth data in the eye opening state is effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of depth image, and particularly to a method, system and device for repairing eye depth image. Background Art

[0002] With the rapid development of three-dimensional vision technology, depth cameras, as important tools for capturing three-dimensional information of the real world, play a crucial role in fields such as human-computer interaction, virtual reality, augmented reality, and biometrics. However, the eye is a unique organ in the human body, with its surface rich in moisture and having a high light absorption characteristic, which makes it difficult for existing depth sensing technologies to achieve ideal accuracy when capturing the details of the eye surface.

[0003] When existing depth cameras scan a face in the open-eye state, due to the absorption of infrared light by the eyes and the complexity of surface reflection characteristics, the point cloud data in the eye area often shows discontinuity or depression. This phenomenon leads to low accuracy and integrity of the obtained eye depth image, and further affects the performance of high-precision applications such as face recognition and iris recognition based on depth information.

[0004] Therefore, there is an urgent need for a repair method to repair the eye images obtained by depth cameras, so as to effectively improve the accuracy and integrity of depth cameras when capturing and reconstructing the depth information of the eye area. Summary of the Invention

[0005] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide a method, system and device for repairing eye depth image, which solves the problem of low accuracy and integrity of eye depth image caused by the absorption of infrared light by the eyes and the complexity of surface reflection characteristics by obtaining color images and depth images in the closed-eye state.

[0006] To solve the above problems, the present invention is implemented according to the following solutions:

[0007] A method for repairing eye depth image is provided, including:

[0008] Obtaining an open-eye color image, an open-eye depth image, a closed-eye color image and a closed-eye depth image;

[0009] Performing key point extraction on the open-eye color image and the closed-eye color image respectively to obtain open-eye key point information and closed-eye key point information;

[0010] According to the open-eye key point information or the closed-eye key point information, performing depth information extraction on the open-eye depth image and the closed-eye depth image respectively to obtain open-eye depth information and closed-eye depth information;

[0011] Based on the open - eye key - point information and the closed - eye key - point information, obtain the position transformation matrix between the open - eye color image and the closed - eye color image;

[0012] Based on the open - eye key - point information, the open - eye depth information, the closed - eye depth information, and the position transformation matrix, obtain the depth image of the repaired eyeball after repair.

[0013] Compared with the prior art, the beneficial effects of an eyeball depth - image repair method of the present invention are as follows: By acquiring the color images and depth images in the open - eye state and the closed - eye state, extracting the key points in the open - eye and closed - eye states and establishing the position transformation matrix between the open - eye and closed - eye states, aligning and fusing the closed - eye depth information that is not affected by the absorption of infrared light by the eyeball and the complexity of the surface reflection characteristics with the open - eye depth information, effectively solving the problem of incomplete depth data in the open - eye state. At the same time, the conversion deviation caused by the difference in pupil position is avoided through the position transformation matrix, significantly improving the accuracy and continuity of the eyeball depth image.

[0014] Optionally, the open - eye color image includes the target face of the eyeball to be repaired, and the open - eye key - point information includes the outer - circle key points around the open - eye, the inner - circle key points around the open - eye, and the pupil key points;

[0015] Performing key - point extraction on the open - eye color image to obtain the open - eye key - point information, including:

[0016] Performing face key - point extraction on the target face in the open - eye color image to obtain multiple to - be - confirmed open - eye key points;

[0017] Based on the characteristics of the outer - circle around the open - eye of the target face, confirm the outer - circle key points around the open - eye among the multiple to - be - confirmed open - eye key points;

[0018] Based on the characteristics of the inner - circle around the open - eye of the target face, confirm the inner - circle key points around the open - eye among the multiple to - be - confirmed open - eye key points;

[0019] Based on the characteristics of the pupil of the target face, confirm the pupil key points among the multiple to - be - confirmed open - eye key points.

[0020] Optionally, the closed - eye color image includes the target face of the eyeball to be repaired, and the closed - eye key - point information includes the outer - circle key points around the closed - eye;

[0021] Performing key - point extraction on the closed - eye color image to obtain the closed - eye key - point information, including:

[0022] Performing face key - point extraction on the target face in the closed - eye color image to obtain multiple to - be - confirmed closed - eye key points;

[0023] Based on the characteristics of the closed-eye outer eye perimeter of the target face, confirm the closed-eye outer eye perimeter key points among multiple key points to be confirmed for closing eyes.

[0024] Optionally, according to the key point information of opening eyes, perform depth information extraction on the depth image of opening eyes to obtain the depth information of opening eyes, including:

[0025] Create a blank mask image with a low luminance value area according to the depth image of opening eyes;

[0026] Determine the target area and background area of the blank mask image according to the inner eye perimeter key points of opening eyes;

[0027] Fill the target area of the blank mask image with a high luminance value to obtain a filled mask image;

[0028] Perform an inversion operation on all areas of the filled mask image to obtain an initial mask image with a low luminance value in the target area and a high luminance value in the background area;

[0029] Perform an AND operation on the depth image of opening eyes and the initial mask image to obtain the depth information of the area except the eyeball part in the open-eye state.

[0030] Optionally, according to the key point information of closing eyes, perform depth information extraction on the depth image of closing eyes to obtain the depth information of closing eyes, including:

[0031] Create a blank mask image with a low luminance value area according to the depth image of closing eyes;

[0032] Determine the target area and background area of the blank mask image according to the outer eye perimeter key points of closing eyes;

[0033] Fill the target area of the blank mask image with a high luminance value to obtain a filled mask image;

[0034] Perform an AND operation on the depth image of closing eyes and the filled mask image to obtain the depth information of the eyeball part in the closed-eye state.

[0035] Optionally, according to the key point information of opening eyes and the key point information of closing eyes, obtain the position transformation matrix between the color image of opening eyes and the color image of closing eyes, including:

[0036] Determine the open-eye coordinates of the outer eye perimeter key points of opening eyes, and the open-eye coordinates form an open-eye outer eye perimeter coordinate set;

[0037] Determine the closed-eye coordinates of the outer eye perimeter key points of closing eyes, and the closed-eye coordinates form a closed-eye outer eye perimeter coordinate set;

[0038] Based on the set of coordinates of the outer perimeter of the eye in the open - eye state and the set of coordinates of the outer perimeter of the eye in the closed - eye state, the position transformation matrix is obtained.

[0039] Optionally, based on the open - eye key - point information, the open - eye depth information, the closed - eye depth information, and the position transformation matrix, an eyeball depth image after repair for the eyeball to be repaired is obtained, including:

[0040] Based on the closed - eye depth information and the position transformation matrix, an initial depth image of the eyeball part in the closed - eye state is obtained, and the depth of the initial depth image is aligned with the depth of the open - eye depth image;

[0041] Based on the pupil key - point and the initial depth image, a translation matrix is constructed;

[0042] Based on the translation matrix, the initial depth image is aligned to obtain a target depth image;

[0043] Based on the open - eye depth information and the target depth image, the eyeball depth image is obtained.

[0044] Optionally, constructing a translation matrix based on the pupil key - point and the initial depth image includes:

[0045] Based on the pupil key - point, the pupil position in the open - eye state is obtained;

[0046] Based on the initial depth image, an estimated pupil position in the closed - eye state is obtained;

[0047] Based on the pupil position and the estimated pupil position, the translation matrix is constructed.

[0048] An eyeball depth - image repair system is also provided, which is applied to the above - mentioned eyeball depth - image repair method, including:

[0049] An image acquisition module, configured to acquire an open - eye color image, an open - eye depth image, a closed - eye color image, and a closed - eye depth image;

[0050] A key - point extraction module, configured to perform key - point extraction on the open - eye color image and the closed - eye color image respectively to obtain open - eye key - point information and closed - eye key - point information;

[0051] A depth extraction module, configured to perform depth - information extraction on the open - eye depth image and the closed - eye depth image respectively based on the open - eye key - point information or the closed - eye key - point information to obtain open - eye depth information and closed - eye depth information;

[0052] A position conversion matrix calculation module, configured to obtain a position conversion matrix between the open-eye color image and the closed-eye color image according to the open-eye key point information and the closed-eye key point information;

[0053] A repair module, configured to obtain a repaired eye depth image of the eye to be repaired according to the open-eye key point information, the open-eye depth information, the closed-eye depth information, and the position conversion matrix.

[0054] There is also provided a computer device, including a processor and a memory, where at least one instruction, at least one program, a code set, or an instruction set is stored in the memory, and the at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the described method for repairing an eye depth image. Description of the Drawings

[0055] Figure 1 It is a flowchart of the repair method of the present invention. Detailed Embodiments

[0056] The following describes the preferred embodiments of the present invention with reference to the drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0057] When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0058] See Figure 1 As shown, the present invention provides a method for repairing an eye depth image, including:

[0059] S1: Obtain an open-eye color image, an open-eye depth image, a closed-eye color image, and a closed-eye depth image; wherein, the open-eye color image and the closed-eye color image are obtained by an ordinary camera. Through the open-eye color image, visible light information of a target face including the eye to be repaired can be obtained, such as the color and texture of the target face; the open-eye depth image and the closed-eye depth image are obtained by a depth camera, and the distance from each pixel point in the target face image including the eye to be repaired to the depth camera can be obtained.

[0060] The open-eye color image can provide the visible features of the outer eye perimeter, inner eye perimeter, and pupil of the target face in the open-eye state, which are used for key point extraction of the target face and its to-be-restored eyeball; the open-eye depth image can record the original depth information of the eyeball area of the target face in the open-eye state. However, due to the absorption of infrared light by the eyeball and the complexity of the surface reflection characteristics, there will be data discontinuity or concavity in the eyeball part of the open-eye depth image; the closed-eye color image can provide the morphological features of the outer eye perimeter of the target face in the closed-eye state, which are used for key point extraction of the target face and its to-be-restored eyeball; the closed-eye depth image can record the complete depth information of the eyeball surface of the target face in the closed-eye state. In the closed-eye state, because the eyelids cover the eyeballs, they are not affected by the absorption of infrared light by the eyeballs and the complexity of the surface reflection characteristics like the eyeballs in the open-eye state, thus avoiding the problem of data discontinuity or concavity in the eye part of the closed-eye depth image from occurring.

[0061] The target face in the open-eye color image, open-eye depth image, closed-eye color image, and closed-eye depth image is the same face individual, that is, the acquisition objects of the four images are the same, and the four images respectively represent the color image and depth image of the same person in the open-eye or closed-eye state; at the same time, in order to ensure the alignment of the position and depth of the to-be-restored eyeball in the same state, it is necessary to ensure that the open-eye color image and the open-eye depth image are aligned one by one in coordinates, and the closed-eye color image and the closed-eye depth image are aligned one by one in coordinates.

[0062] S2: Perform key point extraction on the open-eye color image and the closed-eye color image respectively to obtain open-eye key point information and closed-eye key point information; among them, the open-eye color image includes the target face of the to-be-restored eyeball, and the open-eye key point information includes open-eye outer eye perimeter key points, open-eye inner eye perimeter key points, and pupil key points; the closed-eye color image includes the target face of the to-be-restored eyeball, and the closed-eye key point information includes closed-eye outer eye perimeter key points.

[0063] First, perform key point extraction on the open-eye color image to obtain open-eye key point information, including:

[0064] Perform face key point extraction on the target face in the open-eye color image to obtain multiple to-be-confirmed open-eye key points; in an embodiment of the present invention, key points of the target face in the open-eye color image are extracted based on Mediapipe, generating 478 key points with serial numbers on the face of the target face, that is, the to-be-confirmed open-eye key points include 478 key points with serial numbers.

[0065] Based on the characteristics of the outer circle of the open eyes of the target face, the key points of the outer circle of the open eyes are confirmed among multiple key points to be confirmed for open eyes. Specifically, the key points with serial numbers 446, 261, 448, 449, 450, 451, 452, 453, 464, 413, 441, 442, 443, 444, 445, 342 are selected as the key points of the outer circle of the left eye under the open-eye state; the key points with serial numbers 113, 225, 224, 223, 222, 221, 189, 244, 233, 232, 231, 230, 229, 228, 31, 226 are selected as the key points of the outer circle of the right eye under the open-eye state.

[0066] The key points of the outer circle of the open eyes of the left / right eye are used to prevent overflow problems during subsequent correction. The minimum requirement is that the circle formed by them is larger than the circle formed by the key points of the inner circle of the eye, and all key points of the outer circle of the eye must be outside the key points of the inner circle of the eye. Therefore, in an embodiment of the present invention, combining the above conditions and the key point distribution of Mediapipe, the key points corresponding to the above serial numbers are selected as the key points of the outer circle of the open eyes of the left / right eye.

[0067] Based on the characteristics of the inner circle of the open eyes of the target face, the key points of the inner circle of the open eyes are confirmed among multiple key points to be confirmed for open eyes. Specifically, the key points with serial numbers 263, 249, 390, 373, 374, 380, 381, 382, 362, 398, 384, 385, 386, 387, 388, 466 are selected as the key points of the inner circle of the left eye under the open-eye state; the key points with serial numbers 246, 161, 160, 159, 158, 157, 173, 133, 155, 154, 153, 145, 144, 163, 7, 33 are selected as the key points of the inner circle of the right eye under the open-eye state.

[0068] The key points of the inner circle of the open eyes of the left / right eye are used for image alignment. The minimum requirement is 3 key points distributed outside the eyeball. The more the number of key points of the inner circle of the eye, the closer the position of the key points of the inner circle of the eye to the position of the eyeball, and the higher the accuracy can effectively improve the alignment accuracy. Therefore, in an embodiment of the present invention, combining the above conditions and the key point distribution of Mediapipe, the key points corresponding to the above serial numbers are selected as the key points of the inner circle of the open eyes of the left / right eye.

[0069] Based on the characteristics of the pupils of the target face, the pupil key points are confirmed among multiple key points to be confirmed for open eyes. Specifically, the key points with serial numbers 473, 474, 475, 476, 477 are selected as the key points of the left eye pupil under the open-eye state; the key points with serial numbers 468, 469, 470, 471, 472 are selected as the key points of the right eye pupil under the open-eye state.

[0070] The pupil key points of the left / right eye are used to locate the pupils in the target face, that is, to locate the eyeball to be repaired. In Mediapipe, these 5 points are mainly distributed on the pupil edge. Therefore, in an embodiment of the present invention, combining the above conditions and the key point distribution of Mediapipe, the key points corresponding to the above serial numbers are selected as the pupil key points of the left / right eye.

[0071] Next, key points are extracted from the closed-eye color image to obtain closed-eye key point information, including:

[0072] Facial key points are extracted from the target face in the closed-eye color image to obtain multiple closed-eye key points to be confirmed; in an embodiment of the present invention, key points are extracted from the target face in the closed-eye color image based on Mediapipe, and 478 key points with serial numbers are generated for the face of the target face, that is, the closed-eye key points to be confirmed include 478 key points with serial numbers.

[0073] Based on the characteristics of the outer circle of the closed-eye eye area of the target face, the key points of the outer circle of the closed-eye eye area are confirmed among the multiple closed-eye key points to be confirmed. Specifically, the key points with serial numbers 446, 261, 448, 449, 450, 451, 452, 453, 464, 413, 441, 442, 443, 444, 445, 342 are selected as the key points of the outer circle of the left-eye eye area in the closed-eye state; the key points with serial numbers 113, 225, 224, 223, 222, 221, 189, 244, 233, 232, 231, 230, 229, 228, 31, 226 are selected as the key points of the outer circle of the right-eye eye area in the closed-eye state.

[0074] The key points of the outer circle of the closed-eye of the left / right eye are used to prevent overflow problems during subsequent correction. In the case of the same face, whether in the open-eye state or the closed-eye state, the position of the eye area on the same face will not change. Therefore, in the closed-eye color image, the same serial numbers as in the open-eye state are selected as the key points of the inner circle of the closed-eye of the left / right eye. Through the key points of the outer circle of the open-eye and the key points of the outer circle of the closed-eye, the positional relationship between the color image / depth image in the open-eye state and the color image / depth image in the closed-eye state can be determined.

[0075] When the target face is in the closed-eye state, the eyelids completely cover the eyeballs, and the key points of the inner circle of the eye area (such as the iris edge) and the pupil key points are not visible. Therefore, in the closed-eye state, only the key points of the outer circle of the closed-eye (such as the closed eyelid contour) need to be obtained, unlike in the open-eye state, where in addition to obtaining the key points of the outer circle of the eye area, the key points of the inner circle of the eye area and the pupil key points also need to be obtained.

[0076] By determining the key points for opening eyes, the eye region (inner circle) and pupil position in the target face can be accurately located, providing a mask basis for subsequent elimination of incorrect depth data in the depth image with eyes open; by determining the key points for closing eyes, the complete eye region (outer circle) to be extracted in the depth image with eyes closed can be defined, serving as the data source for subsequent repair of the eye depth image.

[0077] S3: According to the key point information for opening eyes or the key point information for closing eyes, depth information extraction is respectively performed on the depth image with eyes open and the depth image with eyes closed to obtain the depth information with eyes open and the depth information with eyes closed.

[0078] First, according to the key point information for opening eyes, depth information extraction is performed on the depth image with eyes open to obtain the depth information with eyes open, including:

[0079] Based on the depth image with eyes open, a blank mask image with a low brightness value is created. Specifically, according to the size of the depth image with eyes open, a blank mask image with the same size as the depth image with eyes open is created, and all regions of this blank mask image are black (low brightness value).

[0080] According to the key points of the inner circle around the eyes with eyes open, the target region and background region of the blank mask image are determined; when acquiring images, the color image with eyes open and the depth image with eyes open are aligned one by one in position. Therefore, through the key points of the inner circle around the eyes in the color image with eyes open, the inner circle around the eyes in the depth image with eyes open can be determined; where the target region is the region composed of the key points of the inner circle around the eyes, that is, the target region is the region where the inner circle around the eyes is located in the depth image with eyes open, and the region other than the target region (the region where the inner circle around the eyes is located) in the blank mask image is used as the background region.

[0081] At this time, both the target region and background region of the blank mask image are of low brightness value (black). The target region of the blank mask image is filled with a high brightness value to obtain a filled mask image, that is, at this time, the target region of the filled mask image is of high brightness value (white), and the background region is of low brightness value (black). When filling the target region with a high brightness value, the fillPoly() function in opencv can be used for filling.

[0082] An inversion operation is performed on all regions of the filled mask image to obtain an initial mask image with a low brightness value in the target region and a high brightness value in the background region, that is, the brightness value of the target region in the filled mask image is inverted, changing the target region from a high brightness value (white) to a low brightness value (black), and changing the background region from a low brightness value (black) to a high brightness value (white) to obtain the initial mask image.

[0083] When the eyes are open, the pupils (eyeballs) are exposed, and their light absorption characteristics cause problems of data discontinuity or depression in the inner circle area around the eyes (near the iris) of the depth camera. Therefore, the wrong depth information in this area needs to be removed through the initial mask image, and only reliable data in the non-eyeball area (such as eyelids and skin) is retained.

[0084] Perform an AND operation on the open-eye depth image and the initial mask image to obtain the open-eye depth information excluding the eyeball part in the open-eye state. Among them, the open-eye depth image includes all depth information of the target face in the open-eye state, and the target area of the initial mask image has a low brightness value (black), and the background area has a high brightness value (white). When performing an AND operation on the open-eye depth image and the initial mask image, the target area (the area where the inner circle around the eyes is located) with a low brightness value (black) in the initial mask image will block the inner circle area around the eyes in the open-eye depth image, while the background area with a high brightness value (white) will not block the area other than the inner circle area around the eyes in the open-eye depth image, so as to remove the inner circle area around the eyes in the open-eye depth image, and then remove the influence of infrared light on the eyeball when the eyes are open, resulting in the depth information (data discontinuity) of the eyeball part. That is, after performing an AND operation on the open-eye depth image and the initial mask image, the obtained open-eye depth information is the depth information in the open-eye depth image (the depth image in the open-eye state) excluding the eyeball part (the area where the inner circle around the eyes is located).

[0085] Next, according to the closed-eye key point information, extract the depth information from the closed-eye depth image to obtain the closed-eye depth information, including:

[0086] Create a blank mask image with a low brightness value according to the closed-eye depth image. Specifically, create a blank mask image with the same size as the closed-eye depth image according to the size of the closed-eye depth image, and all areas of this blank mask image are black (low brightness value).

[0087] Determine the target area and background area of the blank mask image according to the outer circle key points around the closed eyes; when acquiring the image, the closed-eye color image and the closed-eye depth image are aligned one by one in position. Therefore, through the outer circle key points around the closed eyes in the closed-eye color image, the outer circle around the eyes in the closed-eye depth image can be determined; among them, the target area is the area composed of the outer circle key points around the closed eyes, that is, the target area is the area where the outer circle around the eyes is located in the closed-eye depth image, and the area other than the target area (the area where the outer circle around the eyes is located) in the blank mask image is used as the background area.

[0088] At this time, both the target area and the background area of the blank mask image have low luminance values (black). The target area of the blank mask image is filled with high luminance values to obtain a filled mask image. That is, at this time, the target area of the filled mask image has high luminance values (white), and the background area has low luminance values (black). When filling the target area with high luminance values, the fillPoly() function in opencv can be used for filling.

[0089] When the eyes are closed, the eyelids cover the pupils (eyeballs), and the light absorption interference disappears. The depth camera can capture the entire surface of the eyeball (including the iris and sclera) completely. The key points on the outer circle of the eye area (such as the contour of the closed eyelids) define the entire eyeball area, and its depth data is more accurate and continuous.

[0090] The AND operation is performed on the closed-eye depth image and the filled mask image to obtain the closed-eye depth information of the eyeball part in the closed-eye state. Among them, the closed-eye depth image includes all the depth information of the target face in the closed-eye state, and the target area of the filled mask image has high luminance values (white) and the background area has low luminance values (black). When performing the AND operation on the closed-eye depth image and the filled mask image, the target area with high luminance values (white) in the filled mask image will not block the outer circle area of the eyes in the closed-eye depth image, while the background area with low luminance values (black) will block the area in the closed-eye depth image except for the outer circle area of the eyes, so as to eliminate the area in the closed-eye depth image except for the outer circle of the eyes, and only retain the outer circle area of the eyes in the closed-eye depth image, and further retain the eyeball area in the closed-eye depth image that is not affected by infrared light on the eyeball, so as to obtain continuous and accurate eyeball depth information of the target face; that is, after performing the AND operation on the closed-eye depth image and the filled mask image, the obtained closed-eye depth information is the depth information in the closed-eye depth image (depth image in the closed-eye state) that only retains the eyeball part (the area where the outer circle of the eyes is located).

[0091] In the above process, both the open-eye depth information and the closed-eye depth information include two eyes, the left eye and the right eye. The above process is the processing process of the left-eye image or the right-eye image, and the left and right eyes are not limited here.

[0092] S4: According to the open-eye key point information and the closed-eye key point information, obtain the position transformation matrix between the open-eye color image and the closed-eye color image, including:

[0093] Determine the open-eye coordinates of the key points on the outer circle of the eye area during eye opening. The open-eye coordinates form the open-eye outer-circle coordinate set of the eye area, that is, determine the key points with serial numbers 446, 261, 448, 449, 450, 451, 452, 453, 464, 413, 441, 442, 443, 444, 445, 342 (the key points on the outer circle of the left eye area in the open-eye state), and the key points with serial numbers 113, 225, 224, 223, 222, 221, 189, 244, 233, 232, 231, 230, 229, 228, 31, 226 (the key points on the outer circle of the right eye area in the open-eye state). The open-eye outer-circle coordinate set of the eye area includes 32 numbered coordinates.

[0094] Determine the closed-eye coordinates of the key points on the outer circle of the eye area during eye closing. The closed-eye coordinates form the closed-eye outer-circle coordinate set of the eye area, that is, determine the key points with serial numbers 446, 261, 448, 449, 450, 451, 452, 453, 464, 413, 441, 442, 443, 444, 445, 342 (the key points on the outer circle of the left eye area in the closed-eye state), and the key points with serial numbers 113, 225, 224, 223, 222, 221, 189, 244, 233, 232, 231, 230, 229, 228, 31, 226 (the key points on the outer circle of the right eye area in the closed-eye state). The closed-eye outer-circle coordinate set of the eye area includes 32 numbered coordinates.

[0095] According to the open-eye outer-circle coordinate set and the closed-eye outer-circle coordinate set of the eye area, obtain the position transformation matrix. When acquiring images, the positions of the open-eye color image and the open-eye depth image are aligned one by one, and the positions of the closed-eye color image and the closed-eye depth image are aligned one by one. Therefore, through the open-eye outer-circle coordinate set of the open-eye color image and the closed-eye outer-circle coordinate set of the closed-eye color image, and the coordinate positions under the same serial number, the position transformation matrix of the position transformation relationship between the open-eye color image and the closed-eye color image in the eye area can be obtained. This position transformation matrix can also be used to reflect the position transformation relationship between the open-eye depth image and the closed-eye depth image in the eye area. In an embodiment of the present invention, the findHomography() function in opencv can be used to calculate this position transformation matrix.

[0096] In the above process, the position transformation matrix includes the position transformation relationships of the left eye in the open-eye color image and the left eye in the closed-eye color image, and the right eye in the open-eye color image and the right eye in the closed-eye color image for both eyes. The above process is a processing process for the left-eye image or the right-eye image, and the left and right eyes are not limited here.

[0097] S5: Obtain the depth image of the repaired eyeball to be repaired based on the eye-opening key point information, eye-opening depth information, eye-closing depth information, and position transformation matrix, including:

[0098] First, based on the eye-closing depth information and the position transformation matrix, obtain the initial depth image of the eyeball part in the eye-closed state, and the depth of the initial depth image is aligned with the depth of the eye-opening depth image; in step S4, a position transformation matrix that simultaneously reflects the position transformation relationship between the eye-opening color image and the eye-closing color image in the eye perimeter area, and the position transformation relationship between the eye-opening depth image and the eye-closing depth image in the eye perimeter area is obtained. Therefore, through the position transformation matrix, the eye-closing depth information can be converted into an initial depth image with the same position relationship in the eye-opening depth image. At this time, the position of this initial depth image is in one-to-one alignment with the position of the eye-opening depth image (eye-opening depth information); in an embodiment of the present invention, the opencv's warpPerspective() function can be used to execute the process of converting the eye-closing depth information into an initial depth image according to the position transformation matrix.

[0099] Next, construct a translation matrix based on the pupil key point and the initial depth image, including: obtain the pupil position in the eye-opening state according to the pupil key point; obtain the estimated pupil position in the eye-closed state according to the initial depth image; wherein, when obtaining the estimated pupil position, the initial depth image (the depth image of the eyeball part in the eye-closed state) includes the depth values of multiple positions of the eyeball part and the depth camera, and the depth value is used to represent the physical distance between multiple pixel points (multiple positions) and the depth camera; sort the non-zero depth values in ascending order, select the positions corresponding to the smallest k depth values for average calculation, and use the result as the estimated pupil position; finally, construct a translation matrix according to the pupil position and the estimated pupil position, specifically, use the difference between the pupil position and the estimated pupil position as the translation matrix, and this translation matrix is used to reflect the translation deviation amount when the initial depth image and the eye-opening depth image are not in the same depth plane.

[0100] Finally, align the initial depth image according to the translation matrix to obtain a target depth image. The target depth image is in the same depth plane as the eye-opening depth image (eye-opening depth information), and the target depth information includes the depth information of the eyeball part area in the eye-closing depth image; obtain the depth image of the eyeball according to the eye-opening depth information and the target depth image, specifically, fill the area of the eye that has been removed from the eye-opening depth information with the target depth information to obtain a complete depth image of the eyeball.

[0101] In the above process, the translation matrix includes the left eye pupil position and the left eye estimated pupil position, as well as the right eye pupil position and the right eye estimated pupil position. The above process is the processing process of the left eye image or the right eye image, and the left and right eyes are not limited here.

[0102] By acquiring the color images and depth images in the open-eye state and the closed-eye state, the present invention extracts the key points in the open-eye and closed-eye states and establishes a position transformation matrix between the open-eye and closed-eye states, aligns and fuses the closed-eye depth information and the open-eye depth information that are not affected by the absorption of infrared light by the eyeball and the complexity of the surface reflection characteristics, effectively solves the problem of incomplete depth data in the open-eye state, and at the same time avoids the conversion deviation caused by the pupil position difference through the position transformation matrix, significantly improving the accuracy and continuity of the eyeball depth image.

[0103] The present invention also provides an eyeball depth image repair system, which is applied to the above-mentioned eyeball depth image repair method, and includes:

[0104] An image acquisition module, configured to acquire an open-eye color image, an open-eye depth image, a closed-eye color image, and a closed-eye depth image;

[0105] A key point extraction module, configured to respectively extract key points from the open-eye color image and the closed-eye color image to obtain open-eye key point information and closed-eye key point information;

[0106] A depth extraction module, configured to respectively extract depth information from the open-eye depth image and the closed-eye depth image according to the open-eye key point information or the closed-eye key point information to obtain open-eye depth information and closed-eye depth information;

[0107] A position transformation matrix calculation module, configured to obtain a position transformation matrix between the open-eye color image and the closed-eye color image according to the open-eye key point information and the closed-eye key point information;

[0108] A repair module, configured to obtain the repaired eyeball depth image of the eyeball to be repaired according to the open-eye key point information, the open-eye depth information, the closed-eye depth information, and the position transformation matrix.

[0109] The processing processes of the above modules for each image are all the processing processes for the left eye or the right eye, and the left and right eyes are not limited herein.

[0110] The present invention also provides a computer device, including a processor and a memory. At least one instruction, at least one program, a code set, or an instruction set is stored in the memory, and the at least one instruction, at least one program, a code set, or an instruction set is loaded and executed by the processor to implement the above-mentioned eyeball depth image repair method.

[0111] The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0112] The memory can be used to store the computer program or module. The processor realizes various functions of the eye depth image restoration method by running or executing the computer program or module stored in the memory, and by invoking the data stored in the memory. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.

[0113] The above are only the preferred embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. An eyeball depth image restoration method, characterized in that, Including: Obtain an open - eye color image, an open - eye depth image, a closed - eye color image, and a closed - eye depth image; Perform key - point extraction on the open - eye color image and the closed - eye color image respectively to obtain open - eye key - point information and closed - eye key - point information; According to the open - eye key - point information or the closed - eye key - point information, perform depth - information extraction on the open - eye depth image and the closed - eye depth image respectively to obtain open - eye depth information and closed - eye depth information; According to the open - eye key - point information and the closed - eye key - point information, obtain a position transformation matrix between the open - eye color image and the closed - eye color image; According to the open - eye key - point information, the open - eye depth information, the closed - eye depth information, and the position transformation matrix, obtain a repaired eye - ball depth image of the to - be - repaired eye - ball.

2. The method for repairing an eyeball depth image according to claim 1, wherein The open - eye color image includes a target face of the to - be - repaired eye - ball, and the open - eye key - point information includes open - eye outer - eye - socket key - points, open - eye inner - eye - socket key - points, and pupil key - points; Performing key - point extraction on the open - eye color image to obtain the open - eye key - point information includes: Performing face key - point extraction on the target face in the open - eye color image to obtain multiple to - be - confirmed open - eye key - points; Based on the open - eye outer - eye - socket characteristics of the target face, confirm the open - eye outer - eye - socket key - points among the multiple to - be - confirmed open - eye key - points; Based on the open - eye inner - eye - socket characteristics of the target face, confirm the open - eye inner - eye - socket key - points among the multiple to - be - confirmed open - eye key - points; Based on the pupil characteristics of the target face, confirm the pupil key - points among the multiple to - be - confirmed open - eye key - points.

3. A method for repairing an eye depth image according to claim 2, wherein The closed - eye color image includes a target face of the to - be - repaired eye - ball, and the closed - eye key - point information includes closed - eye outer - eye - socket key - points; Performing key - point extraction on the closed - eye color image to obtain the closed - eye key - point information includes: Performing face key - point extraction on the target face in the closed - eye color image to obtain multiple to - be - confirmed closed - eye key - points; Based on the closed - eye outer - eye - socket characteristics of the target face, confirm the closed - eye outer - eye - socket key - points among the multiple to - be - confirmed closed - eye key - points.

4. A method for repairing an eye depth image according to claim 3, wherein According to the open - eye key - point information, performing depth - information extraction on the open - eye depth image to obtain the open - eye depth information includes: According to the open - eye depth image, create a blank mask image with a low - brightness - value area; According to the open - eye inner - eye - socket key - points, determine the target area and the background area of the blank mask image; Fill the target area of the blank mask image with a high - brightness value to obtain a filled mask image; Perform an inversion operation on all areas of the filled mask image to obtain an initial mask image with a low - brightness - value target area and a high - brightness - value background area; Perform an AND operation on the open - eye depth image and the initial mask image to obtain open - eye depth information excluding the eye - ball part in the open - eye state.

5. A method for repairing an eye depth image according to claim 3, characterized in that, According to the closed - eye key - point information, performing depth - information extraction on the closed - eye depth image to obtain the closed - eye depth information includes: According to the closed - eye depth image, create a blank mask image with a low - brightness - value area; According to the closed - eye outer - eye - socket key - points, determine the target area and the background area of the blank mask image; Fill the target area of the blank mask image with high brightness values to obtain a filled mask image; Perform an AND operation on the closed-eye depth image and the filled mask image to obtain the closed-eye depth information of the eyeball part in the closed-eye state.

6. The method for repairing an eye depth image according to claim 4, wherein According to the open-eye key point information and the closed-eye key point information, obtain the position transformation matrix between the open-eye color image and the closed-eye color image, including: Determine the open-eye coordinates of the key points on the outer circle of the open-eye eye circumference, and the open-eye coordinates form a set of open-eye outer circle coordinates of the eye circumference; Determine the closed-eye coordinates of the key points on the outer circle of the closed-eye eye circumference, and the closed-eye coordinates form a set of closed-eye outer circle coordinates of the eye circumference; According to the set of open-eye outer circle coordinates of the eye circumference and the set of closed-eye outer circle coordinates of the eye circumference, obtain the position transformation matrix.

7. A method for repairing an eye depth image according to claim 6, characterized in that, According to the open-eye key point information, the open-eye depth information, the closed-eye depth information and the position transformation matrix, obtain the depth image of the repaired eyeball of the eyeball to be repaired, including: According to the closed-eye depth information and the position transformation matrix, obtain the initial depth image of the eyeball part in the closed-eye state, and the depth of the initial depth image is aligned with the depth of the open-eye depth image; Construct a translation matrix according to the pupil key point and the initial depth image; Align the initial depth image according to the translation matrix to obtain a target depth image; According to the open-eye depth information and the target depth image, obtain the depth image of the eyeball.

8. A method for repairing an eye depth image according to claim 7, characterized in that, Construct a translation matrix according to the pupil key point and the initial depth image, including: According to the pupil key point, obtain the pupil position in the open-eye state; According to the initial depth image, obtain the estimated pupil position in the closed-eye state; Construct the translation matrix according to the pupil position and the estimated pupil position.

9. An eyeball depth image restoration system, which is applied to the eyeball depth image restoration method described in the above claims 1-8, and is characterized in that, Including: An image acquisition module for acquiring an open-eye color image, an open-eye depth image, a closed-eye color image and a closed-eye depth image; A key point extraction module for respectively extracting key points from the open-eye color image and the closed-eye color image to obtain open-eye key point information and closed-eye key point information; A depth extraction module for respectively extracting depth information from the open-eye depth image and the closed-eye depth image according to the open-eye key point information or the closed-eye key point information to obtain open-eye depth information and closed-eye depth information; A position transformation matrix calculation module for obtaining the position transformation matrix between the open-eye color image and the closed-eye color image according to the open-eye key point information and the closed-eye key point information; A repair module for obtaining the depth image of the repaired eyeball of the eyeball to be repaired according to the open-eye key point information, the open-eye depth information, the closed-eye depth information and the position transformation matrix.

10. A computer device, characterized in that, A computer device includes a processor and a memory, and at least one instruction, at least one program, a code set or an instruction set is stored in the memory, and at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement an eyeball depth image repair method according to any one of claims 1 to 8.

Images