Method, device and electronic device for determining eyeball model

Abstract

The present disclosure relates to a method, device and electronic device for determining an eyeball model. In at least one embodiment of the present disclosure, a first eyeball image captured by an infrared camera and a second eyeball image captured by a visible light camera are obtained; points on the edge of the iris in the first eyeball image and points on the edge of the iris in the second eyeball image are identified, and the correspondence between the points on the edge of the iris in the first eyeball image and the points on the edge of the iris in the second eyeball image is determined; thereby, based on the first eyeball image, the second eyeball image and the correspondence, the three-dimensional spatial position information of the points on the edge of the iris is determined; based on the three-dimensional spatial position information of the points on the edge of the iris, the three-dimensional spatial position information of the iris center in the eyeball model and the long axis and short axis of the iris are determined. The eyeball model determination scheme provided in the embodiments of the present disclosure can further realize related functions based on the eyeball model, such as iris recognition, pupil distance adjustment, and eye orientation estimation.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of eyeball recognition technology, and more particularly to a method, device, and electronic device for determining an eyeball model. Background Art

[0002] With the development of virtual reality (VR) technology, VR devices will be equipped with an eye recognition system to obtain the characteristics of the eye. Therefore, it is urgent to provide a solution for determining an eye model, so as to further realize related functions based on the eye model, such as iris recognition, pupil distance adjustment, and eye direction estimation. Summary of the invention

[0003] At least one embodiment of the present disclosure provides a method, device and electronic device for determining an eyeball model.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for determining an eyeball model, the method comprising:

[0005] Acquire a first eyeball image captured by an infrared camera and a second eyeball image captured by a visible light camera;

[0006] identifying a point on the edge of the iris in the first eye image and a point on the edge of the iris in the second eye image;

[0007] Determine a correspondence between a point on the edge of the iris in the first eyeball image and a point on the edge of the iris in the second eyeball image;

[0008] Determine three-dimensional spatial position information of a point on the edge of the iris based on the first eyeball image, the second eyeball image and the corresponding relationship;

[0009] Based on the three-dimensional spatial position information of the points on the edge of the iris, the three-dimensional spatial position information of the iris center in the eyeball model and the major axis and the minor axis of the iris are determined.

[0010] In some embodiments, after identifying a point on the edge of the iris in the second eye image, the method further includes:

[0011] Determine color features of points on the iris edge in the second eye image based on color information in the second eye image;

[0012] Binding an association relationship between the color features of points on the iris edge in the second eyeball image and the eyeball model;

[0013] In response to an instruction to identify an eyeball, a visible light image of the target eyeball is collected, and points on the edge of the iris in the visible light image and color features of the points are identified;

[0014] Based on the pre-bound association relationship, the target eye model associated with the color feature of the search point is used as the eye model of the target eye.

[0015] In some embodiments, after determining the eye model of the target eye, the method further includes:

[0016] In response to an instruction to track the eyeball, the visible light camera is turned off, and an image of the target eyeball is captured using the infrared camera;

[0017] Based on the image collected by the infrared camera, the three-dimensional spatial position information of the iris center in the eyeball model of the target eyeball is identified.

[0018] In some embodiments, determining a correspondence between a point on an iris edge in a first eye image and a point on an iris edge in a second eye image includes:

[0019] Extracting texture feature points of the iris in the first eyeball image and texture feature points of the iris in the second eyeball image;

[0020] Matching the texture feature points of the iris in the first eyeball image with the texture feature points of the iris in the second eyeball image to obtain a plurality of texture feature point matching pairs;

[0021] At least four texture feature point matching pairs are selected from the plurality of texture feature point matching pairs; any three texture feature points in the at least four texture feature point matching pairs are not collinear;

[0022] Determine an iris mapping relationship between the first eyeball image and the second eyeball image based on at least four texture feature point matching pairs;

[0023] Based on the iris mapping relationship, a correspondence relationship between a point on the iris edge in the first eyeball image and a point on the iris edge in the second eyeball image is determined.

[0024] In some embodiments, determining the three-dimensional spatial position information of a point on the edge of the iris based on the first eyeball image, the second eyeball image, and the correspondence relationship includes:

[0025] Based on the internal and external parameters of the infrared camera and the internal and external parameters of the visible light camera, the first eyeball image and the second eyeball image are transformed to the same plane;

[0026] After the transformation to the same plane is completed, the parallax between the infrared camera and the visible light camera is calculated based on the correspondence;

[0027] Determining depth information of points on the edge of the iris based on disparity;

[0028] Based on the depth information, the three-dimensional spatial position information of the point on the iris edge is determined.

[0029] In some embodiments, based on the three-dimensional spatial position information of the point on the edge of the iris, determining the three-dimensional spatial position information of the iris center in the eyeball model and the major axis and the minor axis of the iris includes:

[0030] Based on the three-dimensional spatial position information of the points on the edge of the iris, an ellipse representing the edge of the iris is fitted;

[0031] Determine the three-dimensional spatial position of the center of the ellipse as the three-dimensional spatial position of the center of the iris;

[0032] Determine the major and minor axes of the ellipse as the major and minor axes of the iris.

[0033] In some embodiments, the method for determining the eyeball model further includes:

[0034] Based on the major axis and minor axis of the iris, the major axis and minor axis of the pupil in the eyeball model are determined.

[0035] In some embodiments, determining the major axis and the minor axis of the pupil in the eyeball model based on the major axis and the minor axis of the iris includes:

[0036] Fitting a first ellipse representing an iris edge and a second ellipse representing a pupil edge in the same image, wherein the same image is a first eyeball image or a second eyeball image;

[0037] Determining a major axis ratio and a minor axis ratio based on the first ellipse and the second ellipse;

[0038] Based on the major and minor axes of the iris and the major and minor axis ratios, the major and minor axes of the pupil are determined.

[0039] In some embodiments, the method for determining the eyeball model further includes:

[0040] Based on the iris mapping relationship between the first eyeball image and the second eyeball image, the major axis and the minor axis of the pupil in the eyeball model are determined.

[0041] In some embodiments, determining the major axis and the minor axis of the pupil in the eyeball model based on the iris mapping relationship between the first eyeball image and the second eyeball image includes:

[0042] Determining a correspondence between a point on the pupil edge in the first eye image and a point on the pupil edge in the second eye image based on an iris mapping relationship between the first eye image and the second eye image;

[0043] After converting the first eyeball image and the second eyeball image to the same plane, the parallax between the infrared camera and the visible light camera is calculated based on the corresponding relationship;

[0044] Determining depth information of points on the pupil edge based on disparity;

[0045] Determine the three-dimensional spatial position information of the point on the pupil edge based on the depth information of the point on the pupil edge;

[0046] Based on the three-dimensional spatial position information of the points on the pupil edge, an ellipse representing the pupil edge is fitted, and the major axis and minor axis of the ellipse are determined to be the major axis and minor axis of the pupil.

[0047] In a second aspect, the present disclosure also provides a device for determining an eyeball model, the device comprising:

[0048] An acquisition unit, used for acquiring a first eyeball image acquired by an infrared camera and a second eyeball image acquired by a visible light camera;

[0049] A recognition unit, used to recognize points on the edge of the iris in the first eyeball image and points on the edge of the iris in the second eyeball image;

[0050] A first determining unit, used to determine a correspondence between a point on the edge of the iris in the first eyeball image and a point on the edge of the iris in the second eyeball image;

[0051] A second determining unit, configured to determine three-dimensional spatial position information of a point on the edge of the iris based on the first eyeball image, the second eyeball image and the corresponding relationship;

[0052] The third determining unit is used to determine the three-dimensional spatial position information of the iris center in the eyeball model and the major axis and the minor axis of the iris based on the three-dimensional spatial position information of the points on the iris edge.

[0053] In a third aspect, an embodiment of the present disclosure further proposes an electronic device, which includes a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the method for determining the eyeball model as described in the first aspect.

[0054] In a fourth aspect, an embodiment of the present disclosure further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instructions, and the program or instructions enable a computer to execute the steps of the method for determining the eyeball model as described in the first aspect.

[0055] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, wherein the computer program product comprises a computer program, the computer program is stored in a computer-readable storage medium, at least one processor of a computer reads and executes the computer program from the computer-readable storage medium, so that the computer performs the steps of the method for determining the eyeball model as described in the first aspect.

[0056] It can be seen that in at least one embodiment of the present disclosure, by acquiring a first eyeball image captured by an infrared camera and a second eyeball image captured by a visible light camera; then identifying points on the edge of the iris in the first eyeball image and points on the edge of the iris in the second eyeball image, and determining the correspondence between the points on the edge of the iris in the first eyeball image and the points on the edge of the iris in the second eyeball image; thereby determining the three-dimensional spatial position information of the points on the edge of the iris based on the first eyeball image, the second eyeball image and the correspondence; based on the three-dimensional spatial position information of the points on the edge of the iris, determining the three-dimensional spatial position information of the iris center in the eyeball model and the long axis and short axis of the iris. The eyeball model determination scheme provided in the embodiments of the present disclosure can further realize related functions based on the eyeball model, such as iris recognition, pupil distance adjustment, and eye orientation estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can also obtain other drawings based on these drawings.

[0058] Figure 1 A schematic flow chart of a method for determining an eyeball model provided in an embodiment of the present disclosure;

[0059] Figure 2 A schematic diagram of determining three-dimensional spatial position information of a point on an iris edge provided by an embodiment of the present disclosure;

[0060] Figure 3 A schematic diagram of a device for determining an eyeball model provided by an embodiment of the present disclosure;

[0061] Figure 4 An exemplary block diagram of an electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. The specific embodiments described herein are only used to explain the present disclosure, rather than to limit the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field belong to the scope of protection of the present disclosure.

[0063] It should be noted that, in this document, relational terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0064] Figure 1 A flow chart of a method for determining an eyeball model provided in an embodiment of the present disclosure, wherein the executor of the method for determining the eyeball model is an electronic device, and the electronic device includes but is not limited to VR equipment, vehicle-mounted equipment, smart phones, PDAs, tablet computers, wearable devices with display screens, desktop computers, laptop computers, all-in-one machines, smart home devices, servers, etc., wherein the server can be an independent server or a cluster of multiple servers, including servers built locally and servers set up in the cloud.

[0065] like Figure 1 As shown, the method for determining the eyeball model may include but is not limited to steps 101 to 105:

[0066] In step 101, a first eyeball image captured by an infrared camera and a second eyeball image captured by a visible light camera are acquired.

[0067] In this embodiment, by setting an infrared camera and a visible light camera, a first eyeball image captured by the infrared camera and a second eyeball image captured by the visible light camera can be obtained. Then, the first eyeball image and the second eyeball image are used to calculate the 3D feature data of the eyeball in real time (how to calculate this information is described in detail below), such as the three-dimensional spatial position information of the iris center and the long axis and short axis of the iris, so as to achieve real-time 3D modeling of the eyeball, laying the foundation for downstream functions based on the eyeball model (such as the eyeball recognition function in VR devices).

[0068] In step 102, points on the edge of the iris in the first eye image and points on the edge of the iris in the second eye image are identified.

[0069] In this embodiment, identifying points on the edge of the iris in the first eye image and points on the edge of the iris in the second eye image can be achieved through two pre-trained neural network models, one of which is used to identify points on the edge of the iris in the first eye image, and the other is used to identify points on the edge of the iris in the second eye image.

[0070] For example, through a pre-trained infrared image recognition model, the first eye image captured by the infrared camera is input into the infrared image recognition model, and the infrared image recognition model recognizes the points on the edge of the iris in the first eye image, so that the infrared image recognition model outputs the points on the edge of the iris in the first eye image.

[0071] The training process of the infrared image recognition model is as follows: obtaining infrared image samples, where the infrared image samples are infrared images with points on the edge of the iris marked; then, inputting the infrared image samples into the neural network model, and adjusting the parameters of the neural network model itself until the output of the neural network model converges to the points on the edge of the iris that have been marked, then the training is completed and the infrared image recognition model is obtained. The training process is a mature technology in the field of neural network technology and will not be described in detail.

[0072] Similarly, the second eye image captured by the visible light camera is input into the visible light image recognition model through a pre-trained visible light image recognition model, and the visible light image recognition model recognizes the points on the edge of the iris in the second eye image, so that the visible light image recognition model outputs the points on the edge of the iris in the second eye image.

[0073] The training process of the visible light image recognition model is as follows: obtain visible light image samples, where the visible light image samples are visible light images with points on the iris edge marked; then, input the visible light image samples into the neural network model, and adjust the parameters of the neural network model itself until the output of the neural network model converges to the points on the marked iris edge, then the training is completed and the visible light image recognition model is obtained. The training process is a mature technology in the field of neural network technology and will not be described in detail.

[0074] In step 103, the correspondence between the points on the edge of the iris in the first eyeball image and the points on the edge of the iris in the second eyeball image is determined.

[0075] In this embodiment, the iris mapping relationship (i.e., the homography matrix) between the first eye image and the second eye image is constructed by using the points on the iris edge in the first eye image and the points on the iris edge in the second eye image, and the texture features on the iris are utilized. Thus, the correspondence of the iris points (including the points on the iris edge and the points inside the iris) between the first eye image and the second eye image can be achieved.

[0076] For example, texture feature points of the iris in the first eye image and texture feature points of the iris in the second eye image are extracted; the texture feature points of the iris in the first eye image and the texture feature points of the iris in the second eye image are matched to obtain a plurality of texture feature point matching pairs; at least four texture feature point matching pairs are selected from the plurality of texture feature point matching pairs, and any three texture feature points in the at least four texture feature point matching pairs are not collinear; based on the at least four texture feature point matching pairs, an iris mapping relationship (i.e., a homography matrix) between the first eye image and the second eye image is determined; based on the iris mapping relationship, a correspondence relationship between points on the edge of the iris in the first eye image and points on the edge of the iris in the second eye image is determined.

[0077] In step 104, based on the first eyeball image, the second eyeball image and the correspondence, the three-dimensional spatial position information of the point on the iris edge is determined.

[0078] In this embodiment, based on the first eye image, the second eye image, and the correspondence between the points on the edge of the iris in the first eye image and the points on the edge of the iris in the second eye image, the parallax between the infrared camera and the visible light camera can be calculated, and then the three-dimensional spatial position information of the points on the edge of the iris can be determined using the parallax, thereby achieving 3D reconstruction of the iris.

[0079] For example, based on the internal and external parameters of the infrared camera and the internal and external parameters of the visible light camera, the first eye image and the second eye image are converted to the same plane; after the conversion to the same plane is completed, the parallax between the infrared camera and the visible light camera is calculated based on the correspondence between the points on the edge of the iris in the first eye image and the points on the edge of the iris in the second eye image; the depth information of the points on the edge of the iris is determined based on the parallax; and the three-dimensional spatial position information of the points on the edge of the iris is determined based on the depth information.

[0080] Figure 2 A schematic diagram of determining the three-dimensional spatial position information of a point on the edge of an iris provided by an embodiment of the present disclosure. Figure 2 In the above equation, camera L can be an infrared camera, camera R can be a visible light camera, or vice versa. b is the baseline length between the two cameras, f is the distance from the camera to the image plane, xl is the parallax of camera L, xr is the parallax of camera R, X is the x-axis of the spatial rectangular coordinate system, Z is the z-axis of the spatial rectangular coordinate system, and the y-axis of the spatial rectangular coordinate system is Figure 2 Not shown, the y axis is perpendicular to the plane formed by X and Z. P is a point in space, for example, a point on the edge of the iris. The three-dimensional spatial position information of point P is recorded as P(x, y, z), where z is the depth information of point P.

[0081] exist Figure 2 Based on the principle of similar triangles, z / f=x / xl, z / f=(xb) / xr, z / f=y / yl=y / yr. The three-dimensional spatial position information of point P can be calculated as follows: z=f×b / (xl-xr), x=(xl×z / f) or (b+xr×z / f), y=(yl×z / f) or (yr×z / f).

[0082] In step 105, based on the three-dimensional spatial position information of the points on the edge of the iris, the three-dimensional spatial position information of the iris center in the eyeball model and the major axis and the minor axis of the iris are determined.

[0083] In this embodiment, based on the three-dimensional spatial position information of the points on the edge of the iris, an ellipse representing the edge of the iris is fitted, and the three-dimensional spatial position of the center of the ellipse is determined as the three-dimensional spatial position of the iris center, and the major axis and minor axis of the ellipse are determined as the major axis and minor axis of the iris. In some embodiments, based on the three-dimensional spatial position information of the points on the edge of the iris, it can be simplified to fit a circle representing the edge of the iris, so that the center of the circle can be determined as the iris center, and the radius of the circle can be determined as the iris radius.

[0084] In this embodiment, after the iris center and the iris radius are determined, the three-dimensional spatial position of the point inside the iris can be calculated.

[0085] It can be seen that in the above embodiments, by acquiring the first eyeball image captured by the infrared camera and the second eyeball image captured by the visible light camera; then identifying the points on the edge of the iris in the first eyeball image and the points on the edge of the iris in the second eyeball image, and determining the correspondence between the points on the edge of the iris in the first eyeball image and the points on the edge of the iris in the second eyeball image; thereby determining the three-dimensional spatial position information of the points on the edge of the iris based on the first eyeball image, the second eyeball image and the correspondence; and determining the three-dimensional spatial position information of the iris center in the eyeball model and the long axis and short axis of the iris based on the three-dimensional spatial position information of the points on the edge of the iris. The eyeball model determination scheme provided in the embodiments of the present disclosure can further realize related functions based on the eyeball model, such as iris recognition, pupil distance adjustment, and eye orientation estimation.

[0086] In some embodiments, after identifying the points on the edge of the iris in the second eye image in step 102, the color features of the points on the edge of the iris in the second eye image can also be determined based on the color information in the second eye image; and then the color features of the points on the edge of the iris in the second eye image are bound to the eye model; when an instruction to identify the eye is received, in response to the instruction to identify the eye, a visible light image of the target eye is collected, and the points on the edge of the iris in the visible light image and the color features of the points are identified; based on the pre-bound association relationship, the target eye model associated with the color features of the points is searched as the eye model of the target eye.

[0087] In this embodiment, an eye model library can be pre-constructed, such as an eye model library of company employees. When the eye model of a certain employee needs to be identified, the association between the color features of the pre-bound points on the edge of the iris and the eye model can be used to capture a visible light image of the target eye through a visible light camera, identify the points on the edge of the iris in the visible light image and the color features of the points, and quickly use the color features to find a matching eye model from the eye model library, thereby quickly identifying the employee's eye model.

[0088] In some embodiments, after the eyeball model of the target eyeball is determined, if an eyeball tracking instruction is received, the visible light camera is turned off in response to the eyeball tracking instruction, and only the infrared camera is used to collect the image of the target eyeball; based on the image collected by the infrared camera, the three-dimensional spatial position information of the iris center in the eyeball model of the target eyeball is identified. Therefore, the eyeball position tracking is achieved through the change of the three-dimensional spatial position of the iris center.

[0089] It can be seen that in this embodiment, when tracking the eyeball, there is no need to use two cameras, but eyeball tracking can be achieved using a single camera.

[0090] In some embodiments, the method for determining the eyeball model may also determine pupil features, which include: pupil center, long axis and short axis of the pupil. The pupil center may use the iris center, that is, the iris center is used as the pupil center, and the three-dimensional spatial position information of the iris center is the three-dimensional spatial position information of the pupil center.

[0091] One way to determine the major axis and minor axis of the pupil in the eyeball model is: based on the major axis and minor axis of the iris, determine the major axis and minor axis of the pupil in the eyeball model, specifically: fit a first ellipse representing the edge of the iris and a second ellipse representing the edge of the pupil in the same image, wherein the same image is the first eyeball image or the second eyeball image; based on the first ellipse and the second ellipse, determine the major axis ratio and the minor axis ratio; based on the major axis and minor axis of the iris and the major axis ratio and the minor axis ratio, determine the major axis and minor axis of the pupil.

[0092] Another way to determine the major axis and minor axis of the pupil in the eyeball model is: based on the iris mapping relationship between the first eyeball image and the second eyeball image, determine the major axis and minor axis of the pupil in the eyeball model, specifically: based on the iris mapping relationship between the first eyeball image and the second eyeball image, determine the correspondence between the points on the edge of the pupil in the first eyeball image and the points on the edge of the pupil in the second eyeball image; after converting the first eyeball image and the second eyeball image to the same plane, calculate the parallax between the infrared camera and the visible light camera based on the correspondence; determine the depth information of the points on the edge of the pupil based on the parallax; determine the three-dimensional spatial position information of the points on the edge of the pupil based on the depth information of the points on the edge of the pupil; based on the three-dimensional spatial position information of the points on the edge of the pupil, fit an ellipse representing the pupil edge, and determine the major axis and minor axis of the ellipse as the major axis and minor axis of the pupil.

[0093] In some embodiments, it can be simplified to fit a circle that represents the edge of the pupil, so that the center of the circle can be determined as the pupil center and the radius of the circle can be determined as the pupil radius.

[0094] In some embodiments, since the visible light camera can capture the color information of the eyeball, the color information can be added to the eyeball model.

[0095] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all described as a series of action combinations, but those skilled in the art can understand that the embodiments of the present disclosure are not limited by the described action sequence, because according to the embodiments of the present disclosure, some steps can be performed in other sequences or simultaneously. In addition, those skilled in the art can understand that the embodiments described in the specification are all optional embodiments.

[0096] Figure 3 A schematic diagram of an eyeball model determination device provided in an embodiment of the present disclosure, the eyeball model determination device can be applied to electronic devices, including but not limited to VR devices, vehicle-mounted devices, smart phones, PDAs, tablet computers, wearable devices with display screens, desktop computers, laptop computers, all-in-one machines, smart home devices, servers, etc., wherein the server can be an independent server or a cluster of multiple servers, including servers built locally and servers built in the cloud. The eyeball model determination device provided in an embodiment of the present disclosure can execute the processing flow provided in each embodiment of the eyeball model determination method, such as Figure 3 As shown, the eyeball model determination device includes but is not limited to: an acquisition unit 31, a recognition unit 32, a first determination unit 33, a second determination unit 34 and a third determination unit 35. The functions of each unit are described as follows:

[0097] An acquisition unit 31 is used to acquire a first eyeball image acquired by an infrared camera and a second eyeball image acquired by a visible light camera;

[0098] An identification unit 32, configured to identify points on the edge of the iris in the first eye image and points on the edge of the iris in the second eye image;

[0099] A first determining unit 33, configured to determine a correspondence between a point on the edge of the iris in the first eyeball image and a point on the edge of the iris in the second eyeball image;

[0100] A second determining unit 34 is used to determine the three-dimensional spatial position information of the point on the iris edge based on the first eyeball image, the second eyeball image and the corresponding relationship;

[0101] The third determining unit 35 is used to determine the three-dimensional spatial position information of the iris center in the eyeball model and the major axis and the minor axis of the iris based on the three-dimensional spatial position information of the points on the iris edge.

[0102] In some embodiments, the apparatus for determining the eyeball model further includes a processing unit, configured to:

[0103] Determine color features of points on the iris edge in the second eye image based on color information in the second eye image;

[0104] Binding an association relationship between the color features of points on the iris edge in the second eyeball image and the eyeball model;

[0105] In response to an instruction to identify an eyeball, a visible light image of the target eyeball is collected, and points on the edge of the iris in the visible light image and color features of the points are identified;

[0106] Based on the pre-bound association relationship, the target eye model associated with the color feature of the search point is used as the eye model of the target eye.

[0107] In some embodiments, the apparatus for determining the eyeball model further includes a tracking unit, which is used to:

[0108] In response to an instruction to track the eyeball, the visible light camera is turned off, and an image of the target eyeball is captured using the infrared camera;

[0109] Based on the image collected by the infrared camera, the three-dimensional spatial position information of the iris center in the eyeball model of the target eyeball is identified.

[0110] In some embodiments, the first determining unit 33 is configured to:

[0111] Extracting texture feature points of the iris in the first eyeball image and texture feature points of the iris in the second eyeball image;

[0112] Matching the texture feature points of the iris in the first eyeball image with the texture feature points of the iris in the second eyeball image to obtain a plurality of texture feature point matching pairs;

[0113] At least four texture feature point matching pairs are selected from the plurality of texture feature point matching pairs; any three texture feature points in the at least four texture feature point matching pairs are not collinear;

[0114] Determine an iris mapping relationship between the first eyeball image and the second eyeball image based on at least four texture feature point matching pairs;

[0115] Based on the iris mapping relationship, a correspondence relationship between a point on the iris edge in the first eyeball image and a point on the iris edge in the second eyeball image is determined.

[0116] In some embodiments, the second determining unit 34 is configured to:

[0117] Based on the internal and external parameters of the infrared camera and the internal and external parameters of the visible light camera, the first eyeball image and the second eyeball image are transformed to the same plane;

[0118] After the transformation to the same plane is completed, the parallax between the infrared camera and the visible light camera is calculated based on the correspondence;

[0119] Determining depth information of points on the edge of the iris based on disparity;

[0120] Based on the depth information, the three-dimensional spatial position information of the point on the iris edge is determined.

[0121] In some embodiments, the third determining unit 35 is configured to:

[0122] Based on the three-dimensional spatial position information of the points on the edge of the iris, an ellipse representing the edge of the iris is fitted;

[0123] Determine the three-dimensional spatial position of the center of the ellipse as the three-dimensional spatial position of the center of the iris;

[0124] Determine the major and minor axes of the ellipse as the major and minor axes of the iris.

[0125] In some embodiments, the apparatus for determining the eyeball model further includes a first pupil feature determination unit for determining the major axis and minor axis of the pupil in the eyeball model based on the major axis and minor axis of the iris.

[0126] In some embodiments, the first pupil feature determination unit is configured to:

[0127] Fitting a first ellipse representing an iris edge and a second ellipse representing a pupil edge in the same image, wherein the same image is a first eyeball image or a second eyeball image;

[0128] Determining a major axis ratio and a minor axis ratio based on the first ellipse and the second ellipse;

[0129] Based on the major and minor axes of the iris and the major and minor axis ratios, the major and minor axes of the pupil are determined.

[0130] In some embodiments, the eyeball model determination device also includes a second pupil feature determination unit, which is used to determine the long axis and short axis of the pupil in the eyeball model based on the iris mapping relationship between the first eyeball image and the second eyeball image.

[0131] In some embodiments, the second pupil feature determination unit is configured to:

[0132] Determining a correspondence between a point on the pupil edge in the first eye image and a point on the pupil edge in the second eye image based on an iris mapping relationship between the first eye image and the second eye image;

[0133] After converting the first eyeball image and the second eyeball image to the same plane, the parallax between the infrared camera and the visible light camera is calculated based on the corresponding relationship;

[0134] Determining depth information of points on the pupil edge based on disparity;

[0135] Determine the three-dimensional spatial position information of the point on the pupil edge based on the depth information of the point on the pupil edge;

[0136] Based on the three-dimensional spatial position information of the points on the pupil edge, an ellipse representing the pupil edge is fitted, and the major axis and minor axis of the ellipse are determined to be the major axis and minor axis of the pupil.

[0137] It can be seen that in at least one embodiment of the device for determining an eyeball model of the present disclosure, by acquiring a first eyeball image captured by an infrared camera and a second eyeball image captured by a visible light camera; then identifying points on the edge of the iris in the first eyeball image and points on the edge of the iris in the second eyeball image, and determining the correspondence between the points on the edge of the iris in the first eyeball image and the points on the edge of the iris in the second eyeball image; thereby determining the three-dimensional spatial position information of the points on the edge of the iris based on the first eyeball image, the second eyeball image and the correspondence; based on the three-dimensional spatial position information of the points on the edge of the iris, determining the three-dimensional spatial position information of the iris center in the eyeball model and the long axis and short axis of the iris. The eyeball model determination scheme provided in the embodiment of the present disclosure can further realize related functions based on the eyeball model, such as iris recognition, pupil distance adjustment, and eye orientation estimation.

[0138] Figure 4 is an exemplary block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 4 As shown, the electronic device includes: a memory 41, a processor 42, and a computer program stored in the memory 41. It can be understood that the memory 41 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0139] In some implementations, the memory 41 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and application programs.

[0140] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic tasks and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application tasks. The program that implements the method for determining the eyeball model provided in the embodiment of the present disclosure can be included in the application.

[0141] In the embodiment of the present disclosure, at least one processor 42 calls a program or instruction stored in at least one memory 41, specifically, a program or instruction stored in an application, and at least one processor 42 is used to execute the steps of each embodiment of the method for determining the eyeball model provided in the embodiment of the present disclosure.

[0142] The method for determining the eyeball model provided in the embodiment of the present disclosure can be applied to the processor 42, or implemented by the processor 42. The processor 42 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 42 or instructions in the form of software. The above-mentioned processor 42 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0143] The steps of the method for determining the eyeball model provided in the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 41, and the processor 42 reads the information in the memory 41 and completes the steps of the method in combination with its hardware.

[0144] The present disclosure also provides a computer-readable storage medium that stores a program or instruction that enables a computer to execute the steps of each embodiment of the eyeball model determination method. To avoid repeated description, the computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0145] The embodiment of the present disclosure also proposes a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium, which may be a non-transitory computer-readable storage medium. At least one processor of a computer reads and executes the computer program from the computer-readable storage medium, so that the computer executes the steps of each embodiment of the method for determining an eyeball model, which will not be described here in detail to avoid repeated description.

[0146] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0147] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present disclosure and to form different embodiments.

[0148] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0149] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for determining an eyeball model, the method comprising: Acquire a first eyeball image captured by an infrared camera and a second eyeball image captured by a visible light camera; Identify a point on the edge of the iris in the first eye image and a point on the edge of the iris in the second eye image; Determine a correspondence between a point on the edge of the iris in the first eyeball image and a point on the edge of the iris in the second eyeball image; Determine three-dimensional spatial position information of a point on the edge of the iris based on the first eyeball image, the second eyeball image and the corresponding relationship; Based on the three-dimensional spatial position information of the points on the edge of the iris, determine the three-dimensional spatial position information of the iris center in the eyeball model and the major axis and minor axis of the iris; After identifying the point on the iris edge in the second eyeball image, the method further includes: Determine, based on the color information in the second eyeball image, a color feature of a point on the iris edge in the second eyeball image; Binding an association relationship between the color features of points on the iris edge in the second eyeball image and the eyeball model; The method further comprises: Fitting a first ellipse representing an iris edge and a second ellipse representing a pupil edge in the same image, wherein the same image is the first eyeball image or the second eyeball image; Determining a major axis ratio and a minor axis ratio based on the first ellipse and the second ellipse; Based on the major axis and minor axis of the iris and the major axis ratio and minor axis ratio, the major axis and minor axis of the pupil are determined.

2. The method according to claim 1, wherein: In response to an instruction to identify an eyeball, a visible light image of the target eyeball is collected, and points on the edge of the iris in the visible light image and color features of the points are identified; Based on the pre-bound association relationship, a target eyeball model associated with the color feature of the point is searched as the eyeball model of the target eyeball.

3. The method according to claim 2, wherein: After determining the eyeball model of the target eyeball, the method further includes: In response to an instruction to track an eyeball, the visible light camera is turned off, and an image of the target eyeball is acquired by using the infrared camera; Based on the image captured by the infrared camera, the three-dimensional spatial position information of the iris center in the eyeball model of the target eyeball is identified.

4. The method according to claim 1, wherein: The determining of the correspondence between the point on the iris edge in the first eyeball image and the point on the iris edge in the second eyeball image comprises: Extracting texture feature points of the iris in the first eyeball image and texture feature points of the iris in the second eyeball image; Matching the texture feature points of the iris in the first eyeball image with the texture feature points of the iris in the second eyeball image to obtain a plurality of texture feature point matching pairs; Select at least four texture feature point matching pairs from the plurality of texture feature point matching pairs; any three texture feature points in the at least four texture feature point matching pairs are not collinear; Determining an iris mapping relationship between the first eyeball image and the second eyeball image based on the at least four texture feature point matching pairs; Based on the iris mapping relationship, a correspondence relationship between points on the iris edge in the first eyeball image and points on the iris edge in the second eyeball image is determined.

5. The method according to claim 1, wherein: The determining the three-dimensional spatial position information of a point on the iris edge based on the first eyeball image, the second eyeball image and the corresponding relationship includes: Based on the internal and external parameters of the infrared camera and the internal and external parameters of the visible light camera, converting the first eyeball image and the second eyeball image to the same plane; After the conversion to the same plane is completed, the parallax between the infrared camera and the visible light camera is calculated based on the corresponding relationship; determining depth information of a point on the edge of the iris based on the disparity; Based on the depth information, three-dimensional spatial position information of a point on the edge of the iris is determined.

6. The method according to claim 1, wherein: The method of determining the three-dimensional spatial position information of the iris center and the major axis and the minor axis of the iris in the eyeball model based on the three-dimensional spatial position information of the point on the iris edge includes: Based on the three-dimensional spatial position information of the points on the iris edge, fitting an ellipse representing the iris edge; Determine the three-dimensional spatial position of the center of the ellipse as the three-dimensional spatial position of the iris center; The major axis and the minor axis of the ellipse are determined to be the major axis and the minor axis of the iris.

7. The method according to claim 1, wherein: The method further comprises: Based on the iris mapping relationship between the first eyeball image and the second eyeball image, the major axis and the minor axis of the pupil in the eyeball model are determined.

8. The method according to claim 7, wherein: The determining, based on the iris mapping relationship between the first eyeball image and the second eyeball image, the major axis and the minor axis of the pupil in the eyeball model comprises: Determining a correspondence between a point on an edge of a pupil in the first eyeball image and a point on an edge of a pupil in the second eyeball image based on an iris mapping relationship between the first eyeball image and the second eyeball image; After converting the first eyeball image and the second eyeball image to the same plane, calculating the parallax between the infrared camera and the visible light camera based on the corresponding relationship; determining depth information of a point on the pupil edge based on the disparity; Determining three-dimensional spatial position information of the point on the pupil edge based on the depth information of the point on the pupil edge; Based on the three-dimensional spatial position information of the points on the pupil edge, an ellipse representing the pupil edge is fitted, and the major axis and minor axis of the ellipse are determined to be the major axis and minor axis of the pupil.

9. A device for determining an eyeball model, the device comprising: An acquisition unit, used for acquiring a first eyeball image acquired by an infrared camera and a second eyeball image acquired by a visible light camera; an identification unit, configured to identify points on the edge of the iris in the first eyeball image and points on the edge of the iris in the second eyeball image; A first determining unit, configured to determine a correspondence between a point on an iris edge in the first eyeball image and a point on an iris edge in the second eyeball image; A second determining unit, configured to determine three-dimensional spatial position information of a point on an iris edge based on the first eyeball image, the second eyeball image and the corresponding relationship; A third determining unit is used to determine the three-dimensional spatial position information of the iris center in the eyeball model and the major axis and the minor axis of the iris based on the three-dimensional spatial position information of the point on the iris edge; a processing unit, configured to determine color features of points on the edge of the iris in the second eyeball image based on color information in the second eyeball image, and to associate the color features of the points on the edge of the iris in the second eyeball image with the eyeball model; The determining device further comprises a first pupil feature determining unit, which is used to fit a first ellipse representing an iris edge and a second ellipse representing a pupil edge in the same image, wherein the same image is the first eyeball image or the second eyeball image; Determining a major axis ratio and a minor axis ratio based on the first ellipse and the second ellipse; Based on the major axis and minor axis of the iris and the major axis ratio and minor axis ratio, the major axis and minor axis of the pupil are determined.

10. An electronic device, wherein: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method for determining the eyeball model as claimed in any one of claims 1 to 8.

Images