An eyeball depth image inpainting method, system and device

Abstract

The application discloses an eyeball depth image repairing method, system and device, and the repairing method comprises the following steps: acquiring an open-eye color image, an open-eye depth image, a closed-eye color image and a closed-eye depth image; respectively extracting key points from the open-eye color image and the closed-eye color image to obtain open-eye key point information and closed-eye key point information; respectively extracting depth information from the open-eye depth image and the closed-eye depth image according to the open-eye key point information or the closed-eye key point information to obtain open-eye depth information and closed-eye depth information; obtaining a position conversion matrix between the open-eye color image and the closed-eye color image according to the open-eye key point information and the closed-eye key point information; and obtaining an eyeball depth image of the repaired eyeball to be repaired according to the open-eye key point information, the open-eye depth information, the closed-eye depth information and the position conversion matrix. Through alignment and fusion of the closed-eye depth information and the open-eye depth information, the problem that the depth data is incomplete in an open-eye state is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of depth image, and particularly relates to an eyeball depth image repairing method, system and device. BACKGROUND

[0002] With the rapid development of three-dimensional vision technology, depth cameras as an important tool for capturing three-dimensional information of the real world play a vital role in human-computer interaction, virtual reality, augmented reality, biometric identification and other fields. As a unique organ in the human body, the eyeball is rich in water on the surface and has a high light absorption property, which makes it difficult for existing depth sensing technology to achieve ideal accuracy when capturing details on the surface of the eyeball.

[0003] When the existing depth camera scans the face in the open-eye state, due to the absorption of infrared light by the eyeball and the complexity of the surface reflection characteristics, the point cloud data of the eyeball region often appears discontinuous or concave. This phenomenon leads to low accuracy and completeness of the obtained eyeball depth image, which further affects the performance of high-precision applications such as face recognition and iris recognition based on depth information.

[0004] Therefore, there is an urgent need for a repairing method to repair the eyeball image obtained by the depth camera, so as to effectively improve the accuracy and completeness of the depth camera in capturing and reconstructing the depth information of the eyeball region. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide an eyeball depth image repairing method, system and device, which acquires color images and depth images in the closed-eye state, solves the problem of low accuracy and completeness of the eyeball depth image caused by the absorption of infrared light by the eyeball and the complexity of the surface reflection characteristics.

[0006] In order to solve the above problems, the present application is implemented according to the following scheme:

[0007] An eyeball depth image repairing method is provided, comprising:

[0008] acquiring an open-eye color image, an open-eye depth image, a closed-eye color image and a closed-eye depth image;

[0009] extracting key points from the open-eye color image and the closed-eye color image respectively to obtain open-eye key point information and closed-eye key point information;

[0010] extracting depth information from the open-eye depth image and the closed-eye depth image according to the open-eye key point information or the closed-eye key point information to obtain open-eye depth information and closed-eye depth information;

[0011] According to the open-eye key point information and the closed-eye key point information, a position conversion matrix between the open-eye color image and the closed-eye color image is obtained;

[0012] According to the open-eye key point information, the open-eye depth information, the closed-eye depth information and the position conversion matrix, a repaired eyeball depth image of the to-be-repaired eyeball is obtained.

[0013] Compared with the prior art, the beneficial effects of the eyeball depth image repairing method are as follows: by acquiring color images and depth images in open-eye and closed-eye states, key points in open-eye and closed-eye states are extracted, and a position conversion matrix between open-eye and closed-eye states is established, closed-eye depth information and open-eye depth information which are not affected by the absorption of infrared light by eyeballs and the complexity of surface reflection characteristics are aligned and fused, the problem of incomplete depth data in open-eye state is effectively solved, and through the position conversion matrix, conversion deviation caused by pupil position difference is avoided, and the accuracy and continuity of the eyeball depth image are significantly improved.

[0014] Optionally, the open-eye color image comprises a target human face of the to-be-repaired eyeball, and the open-eye key point information comprises an open-eye eye circumference outer ring key point, an open-eye eye circumference inner ring key point and a pupil key point.

[0015] Key points of the open-eye color image are extracted to obtain the open-eye key point information, comprising:

[0016] Face key points of the target human face in the open-eye color image are extracted to obtain a plurality of to-be-confirmed open-eye key points.

[0017] Based on open-eye eye circumference outer ring characteristics of the target human face, the open-eye eye circumference outer ring key point is confirmed from the plurality of to-be-confirmed open-eye key points.

[0018] Based on open-eye eye circumference inner ring characteristics of the target human face, the open-eye eye circumference inner ring key point is confirmed from the plurality of to-be-confirmed open-eye key points.

[0019] Based on pupil characteristics of the target human face, the pupil key point is confirmed from the plurality of to-be-confirmed open-eye key points.

[0020] Optionally, the closed-eye color image comprises a target human face of the to-be-repaired eyeball, and the closed-eye key point information comprises a closed-eye eye circumference outer ring key point.

[0021] Key points of the closed-eye color image are extracted to obtain the closed-eye key point information, comprising:

[0022] Face key points of the target human face in the closed-eye color image are extracted to obtain a plurality of to-be-confirmed closed-eye key points.

[0023] Based on the closed-eye eye periphery outer ring characteristics of the target face, the closed-eye eye periphery outer ring key point is confirmed from the plurality of to-be-confirmed closed-eye key points.

[0024] Optionally, according to the open-eye key point information, the open-eye depth image is subjected to depth information extraction to obtain the open-eye depth information, including:

[0025] According to the open-eye depth image, a blank mask image with a low brightness value region is created;

[0026] According to the open-eye eye periphery inner ring key point, a target region and a background region of the blank mask image are determined;

[0027] The target region of the blank mask image is subjected to high brightness value filling to obtain a filled mask image;

[0028] All regions of the filled mask image are subjected to an inversion operation to obtain an initial mask image with a target region of a low brightness value and a background region of a high brightness value;

[0029] The open-eye depth image and the initial mask image are subjected to an AND operation to obtain open-eye depth information of an eyeball part in an open-eye state.

[0030] Optionally, according to the closed-eye key point information, the closed-eye depth image is subjected to depth information extraction to obtain the closed-eye depth information, including:

[0031] According to the closed-eye depth image, a blank mask image with a low brightness value region is created;

[0032] According to the closed-eye eye periphery outer ring key point, a target region and a background region of the blank mask image are determined;

[0033] The target region of the blank mask image is subjected to high brightness value filling to obtain a filled mask image;

[0034] The closed-eye depth image and the filled mask image are subjected to an AND operation to obtain closed-eye depth information of an eyeball part in a closed-eye state.

[0035] Optionally, according to the open-eye key point information and the closed-eye key point information, a position conversion matrix between the open-eye color image and the closed-eye color image is obtained, including:

[0036] An open-eye coordinate of the open-eye eye periphery outer ring key point is determined, and the open-eye coordinate constitutes an open-eye eye periphery outer ring coordinate set;

[0037] A closed-eye coordinate of the closed-eye eye periphery outer ring key point is determined, and the closed-eye coordinate constitutes a closed-eye eye periphery outer ring coordinate set;

[0038] According to the open-eye eye periphery outer circle coordinate set and the closed-eye eye periphery outer circle coordinate set, the position conversion matrix is obtained.

[0039] Optionally, according to the open-eye key point information, the open-eye depth information, the closed-eye depth information and the position conversion matrix, an eye ball depth image after repair is obtained, including:

[0040] According to the closed-eye depth information and the position conversion matrix, an initial depth image of the eye ball part in the closed-eye state is obtained, and the depth of the initial depth image is aligned with the depth of the open-eye depth image.

[0041] According to the pupil key point and the initial depth image, a translation matrix is constructed.

[0042] According to the translation matrix, the initial depth image is aligned to obtain a target depth image.

[0043] According to the open-eye depth information and the target depth image, the eye ball depth image is obtained.

[0044] Optionally, according to the pupil key point and the initial depth image, a translation matrix is constructed, including:

[0045] According to the pupil key point, a pupil position in the open-eye state is obtained.

[0046] According to the initial depth image, an estimated pupil position in the closed-eye state is obtained.

[0047] According to the pupil position and the estimated pupil position, the translation matrix is constructed.

[0048] An eye ball depth image repair system is also provided, which is applied to the above-mentioned eye ball depth image repair method and includes:

[0049] An image acquisition module is configured to acquire an open-eye color image, an open-eye depth image, a closed-eye color image and a closed-eye depth image.

[0050] A key point extraction module is configured to extract key points from the open-eye color image and the closed-eye color image respectively to obtain open-eye key point information and closed-eye key point information.

[0051] A depth extraction module is configured to extract depth information from the open-eye depth image and the closed-eye depth image respectively according to the open-eye key point information or the closed-eye key point information to obtain open-eye depth information and closed-eye depth information.

[0052] A position conversion matrix calculation module is configured to obtain a position conversion matrix between the open-eye color image and the closed-eye color image according to the open-eye key point information and the closed-eye key point information.

[0053] A repairing module is configured to obtain an eye ball depth image after repairing of the eye ball to be repaired according to the open-eye key point information, the open-eye depth information, the closed-eye depth information and the position conversion matrix.

[0054] A computer device is also provided, which comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by the processor to implement the eye ball depth image repairing method. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A flow chart of the repairing method of the present application. DETAILED DESCRIPTION

[0056] The preferred embodiments of the present application will be described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are merely intended to illustrate and explain the present application, and are not intended to limit the present application.

[0057] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present application. It should be appreciated that the present application can be practiced in a variety of ways, and that the following example embodiments are not the only ways in which the present application can be practiced.

[0058] Referring to Figure 1 As shown in the drawings, the present application provides an eye ball depth image repairing method, which comprises:

[0059] S1: obtaining an open-eye color image, an open-eye depth image, a closed-eye color image and a closed-eye depth image; wherein the open-eye color image and the closed-eye color image are obtained by a normal camera, and the visible light information of a target face image including an eye ball to be repaired, such as the color and texture of the target face image, can be obtained through the open-eye color image; the open-eye depth image and the closed-eye depth image are obtained by a depth camera, and the distance from each pixel point in the target face image including the eye ball to be repaired to the depth camera can be obtained.

[0060] The open-eye color image can provide visible features of an outer circle of an eye, an inner circle of the eye and a pupil of the target face in an open-eye state, for key point extraction of the target face and the eye to be repaired; the open-eye depth image can record original depth information of the eye region of the target face in the open-eye state, but due to the absorption of infrared light by the eye and the complexity of the surface reflection characteristics, the eye part in the open-eye depth image may have discontinuous or concave data; the closed-eye color image can provide morphological features of the outer circle of the eye of the target face in the closed-eye state, for key point extraction of the target face and the eye to be repaired; the closed-eye depth image can record complete depth information of the surface of the eye in the closed-eye state, and in the closed-eye state, the eyelid covers the eye, which is not affected by the absorption of infrared light and the complexity of the surface reflection characteristics like the eye in the open-eye state, avoiding the problem of discontinuous or concave data of the eye part in the closed-eye depth image.

[0061] The target face in the open-eye color image, the open-eye depth image, the closed-eye color image and the closed-eye depth image is the same face individual, that is, the collection objects of the four images are consistent, and the four images respectively represent the color image and the depth image of the same person in the open-eye or closed-eye state; at the same time, in order to ensure the alignment of the position and the depth of the eye to be repaired in the same state, it is necessary to ensure that the open-eye color image and the open-eye depth image are aligned in coordinates, and the closed-eye color image and the closed-eye depth image are aligned in coordinates.

[0062] S2: key points are extracted from the open-eye color image and the closed-eye color image respectively to obtain open-eye key point information and closed-eye key point information; wherein the open-eye color image includes the target face of the eye to be repaired, the open-eye key point information includes open-eye outer circle key points, open-eye inner circle key points and pupil key points; the closed-eye color image includes the target face of the eye to be repaired, and the closed-eye key point information includes closed-eye outer circle key points.

[0063] First, key points are extracted from the open-eye color image to obtain open-eye key point information, including:

[0064] Face key points are extracted from the target face in the open-eye color image to obtain a plurality of open-eye key points to be confirmed; in an embodiment of the present application, Mediapipe is used to extract key points from the target face in the open-eye color image, and 478 key points with serial numbers are generated for the face of the target face, that is, the open-eye key points to be confirmed include 478 key points with serial numbers.

[0065] Based on the open-eye eye periphery outer circle characteristics of the target face, open-eye eye periphery outer circle key points are confirmed from the plurality of to-be-confirmed open-eye key points, specifically, the key points with serial numbers 446, 261, 448, 449, 450, 451, 452, 453, 464, 413, 441, 442, 443, 444, 445 and 342 are selected as the key points of the left eye periphery outer circle in the open-eye state; and the key points with serial numbers 113, 225, 224, 223, 222, 221, 189, 244, 233, 232, 231, 230, 229, 228, 31 and 226 are selected as the key points of the right eye periphery outer circle in the open-eye state.

[0066] The open-eye eye periphery outer circle key points of the left eye / right eye are used to prevent overflow problems in subsequent correction, and the minimum requirement is that the circle composed of the key points is larger than the circle composed of the eye periphery inner circle key points, and all the eye periphery outer circle key points must be outside the eye periphery inner circle key points. Therefore, in an embodiment of the present application, the key points corresponding to the above serial numbers are selected as the open-eye eye periphery outer circle key points of the left eye / right eye in combination with the above conditions and the key point distribution of Mediapipe.

[0067] Based on the open-eye eye periphery inner circle characteristics of the target face, open-eye eye periphery inner circle key points are confirmed from the plurality of to-be-confirmed open-eye key points, specifically, the key points with serial numbers 263, 249, 390, 373, 374, 380, 381, 382, 362, 398, 384, 385, 386, 387, 388 and 466 are selected as the key points of the left eye periphery inner circle in the open-eye state; and the key points with serial numbers 246, 161, 160, 159, 158, 157, 173, 133, 155, 154, 153, 145, 144, 163, 7 and 33 are selected as the key points of the right eye periphery inner circle in the open-eye state.

[0068] The open-eye eye periphery inner circle key points of the left eye / right eye are used for image alignment, and the minimum requirement is 3 key points distributed outside the eyeball. The more the number of eye periphery inner circle key points, the closer the position of the eye periphery inner circle key points to the eyeball position, and the higher the accuracy can effectively improve the alignment accuracy. Therefore, in an embodiment of the present application, the key points corresponding to the above serial numbers are selected as the open-eye eye periphery inner circle key points of the left eye / right eye in combination with the above conditions and the key point distribution of Mediapipe.

[0069] Based on the pupil characteristics of the target face, pupil key points are confirmed from the plurality of to-be-confirmed open-eye key points, specifically, the key points with serial numbers 473, 474, 475, 476 and 477 are selected as the key points of the left eye pupil in the open-eye state; and the key points with serial numbers 468, 469, 470, 471 and 472 are selected as the key points of the right eye pupil in the open-eye state.

[0070] The pupil key points of the left eye / right eye are used for positioning the pupil in the target face, that is, positioning the eye to be repaired, and in Mediapipe, the 5 points are mainly distributed on the edge of the pupil, so in an embodiment of the present application, in combination with the above conditions and the key point distribution of Mediapipe, the key points corresponding to the above serial numbers are selected as the pupil key points of the left eye / right eye.

[0071] Then, key point extraction is performed on the closed-eye color image to obtain closed-eye key point information, including:

[0072] Face key point extraction is performed on the target face in the closed-eye color image to obtain a plurality of to-be-confirmed closed-eye key points; in an embodiment of the present application, Mediapipe is used to perform key extraction on the target face in the closed-eye color image, to generate 478 key points with serial numbers for the face of the target face, that is, the to-be-confirmed closed-eye key points include 478 key points with serial numbers.

[0073] Based on the closed-eye eye periphery outer ring characteristics of the target face, the closed-eye eye periphery outer ring key points are confirmed from the plurality of to-be-confirmed closed-eye key points, specifically, the key points with serial numbers 446, 261, 448, 449, 450, 451, 452, 453, 464, 413, 441, 442, 443, 444, 445, and 342 are selected as the key points of the left eye periphery outer ring in the closed-eye state; and the key points with serial numbers 113, 225, 224, 223, 222, 221, 189, 244, 233, 232, 231, 230, 229, 228, 31, and 226 are selected as the key points of the right eye periphery outer ring in the closed-eye state.

[0074] The closed-eye eye periphery outer ring key points of the left eye / right eye are used for preventing overflow in subsequent correction, and in the case of the same face, the eye periphery will not change in position in the same face whether in an open-eye state or a closed-eye state, so in the closed-eye color image, the same serial numbers as in the open-eye state are selected as the closed-eye eye periphery inner ring key points of the left eye / right eye, and through the open-eye eye periphery outer ring key points and the closed-eye eye periphery outer ring key points, the positional relationship between the color image / depth image in the open-eye state and the color image / depth image in the closed-eye state can be determined.

[0075] The eyelid completely covers the eyeball in the closed-eye state of the target face, and the eye periphery inner ring key points (such as the iris edge) and the pupil key points are not visible, so in the closed-eye state, only the closed-eye eye periphery outer ring key points (such as the closed-eye eyelid contour) need to be obtained, and in the open-eye state, in addition to the eye periphery outer ring key points, the eye periphery inner ring key points and the pupil key points also need to be obtained.

[0076] The application can accurately position the eyeball region (inner circle) and pupil position in the target face by determining the open-eye key points, and provide mask basis for subsequent elimination of false depth data in the open-eye depth image; the complete eyeball region (outer circle) to be extracted in the closed-eye depth image can be defined by determining the closed-eye key points, as a data source for subsequent repair of the eyeball depth image.

[0077] S3: according to the open-eye key point information or the closed-eye key point information, depth information extraction is performed on the open-eye depth image and the closed-eye depth image respectively, to obtain open-eye depth information and closed-eye depth information.

[0078] Firstly, according to the open-eye key point information, depth information extraction is performed on the open-eye depth image to obtain open-eye depth information, including:

[0079] According to the open-eye depth image, a blank mask image with a low brightness value is created, specifically, according to the size of the open-eye depth image, a blank mask image with the same size as the open-eye depth image is created, and all regions of the blank mask image are black (low brightness value).

[0080] According to the open-eye eye circle inner circle key point, the target region and the background region of the blank mask image are determined; when the image is acquired, the open-eye color image and the open-eye depth image are one-to-one aligned, so the open-eye eye circle inner circle in the open-eye depth image can be determined through the open-eye eye circle inner circle key point of the open-eye color image; wherein the target region is the region composed of the open-eye eye circle inner circle key point, that is, the target region is the region where the open-eye eye circle inner circle in the open-eye depth image is located, and the region of the blank mask image except the target region (the region where the open-eye eye circle inner circle is located) is taken as the background region.

[0081] At this time, the target region and the background region of the blank mask image are both low brightness values (black), the target region of the blank mask image is filled with high brightness values to obtain a filled mask image, that is, at this time, the target region of the filled mask image is high brightness value (white), and the background region is low brightness value (black), when the target region is filled with high brightness value, the fillPoly() function in opencv can be used for filling.

[0082] The all regions of the filled mask image are subjected to an inversion operation to obtain an initial mask image with a target region of low brightness value and a background region of high brightness value, that is, the brightness value of the target region of the filled mask image is subjected to an inversion operation, the target region changes from high brightness value (white) to low brightness value (black), and the background region changes from low brightness value (black) to high brightness value (white) to obtain the initial mask image.

[0083] When the eyes are open, the pupil (eyeball) is exposed, and its light absorption characteristics cause the depth camera to generate discontinuous or concave data in the inner circle area of the eye (near the iris). Therefore, the error depth information in this area needs to be removed through the initial mask image, and only reliable data in the non-eyeball area (such as eyelids, skin) is retained.

[0084] The open-eye depth image and the initial mask image are operated to obtain open-eye depth information excluding the eyeball part in the open-eye state, wherein the open-eye depth image includes all depth information of the target face in the open-eye state, and the target area of the initial mask image is a low-brightness value (black), and the background area is a high-brightness value (white). When the open-eye depth image and the initial mask image are operated, the target area (the area where the inner circle of the eye is located) of the low-brightness value (black) in the initial mask image will shield the inner circle area of the eye in the open-eye depth image, and the background area of the high-brightness value (white) will not shield the area excluding the inner circle area of the eye in the open-eye depth image. The inner circle area of the eye in the open-eye depth image is removed, and the influence of infrared light on the eyeball when the eyes are open causes the depth information (discontinuous data) of the eyeball part. That is, after the open-eye depth image and the initial mask image are operated, the open-eye depth information obtained is the depth information excluding the eyeball part (the area where the inner circle of the eye is located) in the open-eye depth image (depth image in the open-eye state).

[0085] Then, according to the closed-eye key point information, the closed-eye depth image is operated to extract depth information, and closed-eye depth information is obtained, including:

[0086] According to the closed-eye depth image, a blank mask image with a low-brightness value is created. Specifically, according to the size of the closed-eye depth image, a blank mask image with the same size as the closed-eye depth image is created, and all areas of the blank mask image are black (low-brightness value).

[0087] According to the closed-eye eye circle outer key point, the target area and the background area of the blank mask image are determined. When the image is acquired, the positions of the closed-eye color image and the closed-eye depth image are aligned one by one. Therefore, through the closed-eye eye circle outer key point of the closed-eye color image, the eye circle in the closed-eye depth image can be determined. The target area is the area composed of the closed-eye eye circle outer key point, that is, the target area is the area where the eye circle in the closed-eye depth image is located. The area of the blank mask image excluding the target area (the area where the eye circle is located) is regarded as the background area.

[0088] At this time, the target region and the background region of the blank mask image are both low brightness values (black), the target region of the blank mask image is filled with high brightness values, and a filled mask image is obtained. At this time, the target region of the filled mask image is a high brightness value (white), and the background region is a low brightness value (black). When the target region is filled with high brightness values, the fillPoly() function in opencv can be used for filling.

[0089] When the eyes are closed, the eyelids cover the pupils (eyeballs), the light absorption interference disappears, and the depth camera can completely capture the surface of the eyeball (including the iris and the sclera). The key points outside the eye circle (such as the outline of the closed eyelids) define the entire eyeball region, and the depth data is more accurate and continuous.

[0090] The closed-eye depth image and the filled mask image are operated with each other to obtain the closed-eye depth information of the eyeball part in the closed-eye state. The closed-eye depth image includes all depth information of the target face in the closed-eye state, and the target region of the filled mask image is a high brightness value (white) and the background region is a low brightness value (black). When the closed-eye depth image and the filled mask image are operated with each other, the target region with a high brightness value (white) in the filled mask image does not shield the region outside the eye circle in the closed-eye depth image, and the background region with a low brightness value (black) shields the region outside the eye circle in the closed-eye depth image. The region outside the eye circle in the closed-eye depth image is removed, only the region outside the eye circle in the closed-eye depth image is retained, and then the eyeball region in the closed-eye depth image that is not affected by infrared light is retained, so that continuous and accurate eyeball depth information in the target face is obtained. That is, after the closed-eye depth image and the filled mask image are operated with each other, the closed-eye depth information obtained is the depth information of the closed-eye depth image (the depth image in the closed-eye state) with only the eyeball part (the region where the eye circle is located) removed.

[0091] In the above process, the open-eye depth information and the closed-eye depth information both include the left eye and the right eye, and the above process is a processing process of the left eye image or the right eye image, and the left and right eyes are not limited.

[0092] S4: According to the open-eye key point information and the closed-eye key point information, a position conversion matrix between the open-eye color image and the closed-eye color image is obtained, including:

[0093] The open-eye coordinates of the open-eye eye peripheral circle key points are determined, the open-eye coordinates constitute an open-eye eye peripheral circle coordinate set, that is, the coordinates of the key points with serial numbers 446, 261, 448, 449, 450, 451, 452, 453, 464, 413, 441, 442, 443, 444, 445, 342 (left eye peripheral circle key points in the open-eye state) and serial numbers 113, 225, 224, 223, 222, 221, 189, 244, 233, 232, 231, 230, 229, 228, 31, 226 (right eye peripheral circle key points in the open-eye state) are determined, and the open-eye eye peripheral circle coordinate set includes 32 coordinates with serial numbers.

[0094] The closed-eye coordinates of the closed-eye eye peripheral circle key points are determined, the closed-eye coordinates constitute a closed-eye eye peripheral circle coordinate set, that is, the coordinates of the key points with serial numbers 446, 261, 448, 449, 450, 451, 452, 453, 464, 413, 441, 442, 443, 444, 445, 342 (left eye peripheral circle key points in the closed-eye state) and serial numbers 113, 225, 224, 223, 222, 221, 189, 244, 233, 232, 231, 230, 229, 228, 31, 226 (right eye peripheral circle key points in the closed-eye state) are determined, and the closed-eye eye peripheral circle coordinate set includes 32 coordinates with serial numbers.

[0095] According to the open-eye eye peripheral circle coordinate set and the closed-eye eye peripheral circle coordinate set, a position conversion matrix is obtained, when the images are acquired, the positions of the open-eye color image and the open-eye depth image are aligned one by one, and the positions of the closed-eye color image and the closed-eye depth image are aligned one by one, therefore, through the coordinate positions in the same serial number under the open-eye eye peripheral circle coordinate set of the open-eye color image and the closed-eye eye peripheral circle coordinate set of the closed-eye color image, the position conversion matrix of the position conversion relationship between the open-eye color image and the closed-eye color image in the eye peripheral region can be obtained, and the position conversion matrix can also be used to reflect the position conversion relationship between the open-eye depth image and the closed-eye depth image in the eye peripheral region, in an embodiment of the present application, the findHomography() function in opencv can be used to calculate the position conversion matrix.

[0096] In the above process, the position conversion matrix includes the position conversion relationship of the left eye in the open-eye color image and the left eye in the closed-eye color image, and the position conversion relationship of the right eye in the open-eye color image and the right eye in the closed-eye color image, the above process is a processing process of the left eye image or the right eye image, and the left and right eyes are not limited.

[0097] S5: obtaining the repaired eyeball depth image of the to-be-repaired eyeball according to the open-eye key point information, the open-eye depth information, the closed-eye depth information, and the position conversion matrix, including:

[0098] Firstly, the initial depth image of the eyeball part in the closed-eye state is obtained according to the closed-eye depth information and the position conversion matrix, and the depth of the initial depth image is aligned with the depth of the open-eye depth image; in step S4, the position conversion matrix is obtained, which is used to reflect the position conversion relationship between the open-eye color image and the closed-eye color image in the eye region, and the position conversion relationship between the open-eye depth image and the closed-eye depth image in the eye region, so that the closed-eye depth information can be converted into the initial depth image with the same position relationship in the open-eye depth image through the position conversion matrix, and at this time, the position of the initial depth image is aligned with the position of the open-eye depth image (open-eye depth information); in an embodiment of the present application, the warpPerspective() function of opencv can be used to perform the process of converting the closed-eye depth information into the initial depth image according to the position conversion matrix.

[0099] Then, a translation matrix is constructed according to the pupil key points and the initial depth image, including: obtaining the pupil position in the open-eye state according to the pupil key points; obtaining the estimated pupil position in the closed-eye state according to the initial depth image; wherein, when obtaining the estimated pupil position, the initial depth image (the depth image of the eyeball part in the closed-eye state) includes a plurality of positions of the eyeball part and the depth value of the depth camera, and the depth value is used to represent the physical distance between a plurality of pixel points (a plurality of positions) and the depth camera; the non-zero depth values are sorted in ascending order, and the positions corresponding to the smallest k depth values are averaged to obtain the estimated pupil position; finally, the translation matrix is constructed according to the pupil position and the estimated pupil position, specifically, the difference between the pupil position and the estimated pupil position is taken as the translation matrix, which is used to reflect the translation deviation amount when the initial depth image and the open-eye depth image are not in the same depth plane.

[0100] Finally, the initial depth image is aligned according to the translation matrix to obtain a target depth image, and the target depth image and the open-eye depth image (open-eye depth information) are in the same depth plane, and the target depth information includes the depth information of the eyeball part region in the closed-eye depth image; the eyeball depth image is obtained according to the open-eye depth information and the target depth image, specifically, the target depth information is filled into the excluded eye part in the open-eye depth information to obtain a complete eyeball depth image.

[0101] In the above process, the translation matrix includes the left eye pupil position and the left eye estimated pupil position, and the right eye pupil position and the right eye estimated pupil position, and the above process is the processing process of the left eye image or the right eye image, and the left and right eyes are not limited here.

[0102] The present application obtains color images and depth images in open-eye and closed-eye states, extracts key points in open-eye and closed-eye states, and establishes a position conversion matrix between open-eye and closed-eye states, aligns and fuses closed-eye depth information and open-eye depth information which are not affected by the absorption of infrared light by eyeballs and the complexity of surface reflection characteristics, effectively solves the problem of incomplete depth data in open-eye state, and avoids conversion deviation caused by pupil position difference through the position conversion matrix, thereby significantly improving the accuracy and continuity of eyeball depth images.

[0103] The present application also provides an eyeball depth image repairing system applied to the eyeball depth image repairing method.

[0104] An image acquisition module is configured to acquire open-eye color images, open-eye depth images, closed-eye color images, and closed-eye depth images.

[0105] A key point extraction module is configured to extract key points from the open-eye color images and the closed-eye color images to obtain open-eye key point information and closed-eye key point information.

[0106] A depth extraction module is configured to extract depth information from the open-eye depth images and the closed-eye depth images according to the open-eye key point information or the closed-eye key point information to obtain open-eye depth information and closed-eye depth information.

[0107] A position conversion matrix calculation module is configured to obtain a position conversion matrix between the open-eye color images and the closed-eye color images according to the open-eye key point information and the closed-eye key point information.

[0108] A repairing module is configured to obtain an eyeball depth image of a repaired eyeball according to the open-eye key point information, the open-eye depth information, the closed-eye depth information, and the position conversion matrix.

[0109] The above modules are all configured to process left eyes or right eyes, and the left eyes and the right eyes are not limited.

[0110] The present application also provides a computer device including a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above eyeball depth image repairing method.

[0111] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc.

[0112] The memory can be configured to store the computer programs or modules, the processor can realize various functions of the eyeball depth image repairing method by running or executing the computer programs or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, etc.; and the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0113] The above are only the preferred embodiments of the present application, and are not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An ocular globe depth map inpainting method, characterized by, The method comprises the following steps: obtaining an open-eye color image, an open-eye depth image, a closed-eye color image and a closed-eye depth image; performing key point extraction on the open-eye color image and the closed-eye color image respectively to obtain open-eye key point information and closed-eye key point information; performing depth information extraction on the open-eye depth image and the closed-eye depth image respectively according to the open-eye key point information or the closed-eye key point information to obtain open-eye depth information and closed-eye depth information; obtaining a position conversion matrix between the open-eye color image and the closed-eye color image according to the open-eye key point information and the closed-eye key point information; obtaining an eye depth image of the repaired eyeball according to the open-eye key point information, the open-eye depth information, the closed-eye depth information and the position conversion matrix.

2. The method of claim 1, wherein, The open-eye color image comprises a target human face of the eyeball to be repaired, and the open-eye key point information comprises an open-eye eye circumference outer ring key point, an open-eye eye circumference inner ring key point and a pupil key point. The key point extraction on the open-eye color image to obtain the open-eye key point information comprises the following steps: performing face key point extraction on the target human face in the open-eye color image to obtain a plurality of open-eye key points to be confirmed; confirming the open-eye eye circumference outer ring key point from the plurality of open-eye key points to be confirmed based on the open-eye eye circumference outer ring characteristics of the target human face; confirming the open-eye eye circumference inner ring key point from the plurality of open-eye key points to be confirmed based on the open-eye eye circumference inner ring characteristics of the target human face; confirming the pupil key point from the plurality of open-eye key points to be confirmed based on the pupil characteristics of the target human face.

3. The method of claim 2, wherein, The closed-eye color image comprises a target human face of the eyeball to be repaired, and the closed-eye key point information comprises a closed-eye eye circumference outer ring key point. The key point extraction on the closed-eye color image to obtain the closed-eye key point information comprises the following steps: performing face key point extraction on the target human face in the closed-eye color image to obtain a plurality of closed-eye key points to be confirmed; confirming the closed-eye eye circumference outer ring key point from the plurality of closed-eye key points to be confirmed based on the closed-eye eye circumference outer ring characteristics of the target human face.

4. The method of claim 3, wherein, The depth information extraction on the open-eye depth image according to the open-eye key point information to obtain the open-eye depth information comprises the following steps: creating a blank mask image with a low brightness value region according to the open-eye depth image; determining a target region and a background region of the blank mask image according to the open-eye eye circumference inner ring key point; performing high brightness value filling on the target region of the blank mask image to obtain a filled mask image; performing an inversion operation on all regions of the filled mask image to obtain an initial mask image with a low brightness value target region and a high brightness value background region; performing an AND operation on the open-eye depth image and the initial mask image to obtain open-eye depth information of the eyeball part in the open-eye state.

5. The method of claim 3, wherein the step of generating the depth map of the eye is performed by a neural network. The depth information extraction on the closed-eye depth image according to the closed-eye key point information to obtain the closed-eye depth information comprises the following steps: creating a blank mask image with a low brightness value region according to the closed-eye depth image; determining a target region and a background region of the blank mask image according to the closed-eye eye circumference outer ring key point; The target region of the blank mask image is filled with high lightness values to obtain a filled mask image; The open-eye depth image and the filled mask image are subjected to an AND operation to obtain open-eye depth information of the eyeball part in the closed-eye state.

6. The method of claim 4, wherein, According to the open-eye key point information and the closed-eye key point information, a position conversion matrix between the open-eye color image and the closed-eye color image is obtained, including: An open-eye coordinate of the open-eye periorbital key point is determined, and the open-eye coordinate constitutes an open-eye periorbital coordinate set; A closed-eye coordinate of the closed-eye periorbital key point is determined, and the closed-eye coordinate constitutes a closed-eye periorbital coordinate set; According to the open-eye periorbital coordinate set and the closed-eye periorbital coordinate set, the position conversion matrix is obtained.

7. The method of claim 6, wherein, According to the open-eye key point information, the open-eye depth information, the closed-eye depth information, and the position conversion matrix, an eyeball depth image of a repaired eyeball to be repaired is obtained, including: According to the closed-eye depth information and the position conversion matrix, an initial depth image of the eyeball part in the closed-eye state is obtained, and the depth of the initial depth image is aligned with the depth of the open-eye depth image; According to the pupil key point and the initial depth image, a translation matrix is constructed; According to the translation matrix, the initial depth image is aligned to obtain a target depth image; According to the open-eye depth information and the target depth image, the eyeball depth image is obtained.

8. The method of claim 7, wherein, According to the pupil key point and the initial depth image, a translation matrix is constructed, including: According to the pupil key point, a pupil position in the open-eye state is obtained; According to the initial depth image, an estimated pupil position in the closed-eye state is obtained; According to the pupil position and the estimated pupil position, the translation matrix is constructed.

9. An ocular depth image inpainting system, applied to the ocular depth image inpainting method of any one of claims 1-8, characterized in that, including: An image acquisition module is configured to acquire an open-eye color image, an open-eye depth image, a closed-eye color image, and a closed-eye depth image; A key point extraction module is configured to extract key points from the open-eye color image and the closed-eye color image to obtain open-eye key point information and closed-eye key point information; A depth extraction module is configured to extract depth information from the open-eye depth image and the closed-eye depth image according to the open-eye key point information or the closed-eye key point information to obtain open-eye depth information and closed-eye depth information; A position conversion matrix calculation module is configured to obtain a position conversion matrix between the open-eye color image and the closed-eye color image according to the open-eye key point information and the closed-eye key point information; A repair module is configured to obtain an eyeball depth image of a repaired eyeball to be repaired according to the open-eye key point information, the open-eye depth information, the closed-eye depth information, and the position conversion matrix.

10. A computer device, comprising: A computer device includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement an eyeball depth image repair method according to any one of claims 1 to 8.

Images