Fitting method of 3D eye model and electronic equipment

Abstract

The invention provides a fitting method of a 3D eye model and electronic equipment, and aims to improve the accuracy of the eye model. Comprising the following steps: in response to an eye model fitting instruction, obtaining eye images obtained when eyes of a user watch fixation points at different positions; obtaining eyeball parameters of the user by using the image position coordinates of the pupil center of the user in each eye image and the standard visual axis vector corresponding to each eye image; obtaining a current eyeball rotation angle based on the eyeball parameters of the user and the image position coordinates of the pupil center in the current eye image every specified duration; according to the current eyeball rotation angle and the eyeball parameters, current eyeball model parameters are obtained, and the eyeball model parameters comprise the position vector of the eyeball center, the position vector of the pupil center, the position vector of the curvature center and the visual axis vector; and updating a preset initial eye model by using the current eyeball model parameters of the user to obtain a current 3D eye model of the user.

Description

Technical Field

[0001] This application relates to the field of virtual reality, and more particularly to a method for fitting a 3D eye model and an electronic device. Background Technology

[0002] Eye tracking is one of the two key technologies in hand-eye interaction methods in virtual reality. The goal of eye tracking is to estimate the direction of the user's eye gaze, that is, based on... Figure 1 The system calculates the user's visual axis and then estimates the target the user is looking at in the virtual scene (i.e., the gaze point). Currently, there are two main methods for gaze point estimation: one is based on the flickering reflection of infrared LEDs, and the other is based on the flicker-free method of infrared LEDs.

[0003] However, existing infrared LED flashing reflection methods require complex algorithms for detecting and matching Pulchin spots (spots formed on an image by the reflection of an LED light source from the eye, hereinafter referred to as spots). For example, ... Figure 1 As shown, Figure 1 The image shows a light spot detected in the eye image. After detecting the light spot, noise needs to be removed, and an algorithm needs to determine which infrared LED emitted the light spot. On the other hand, this method places high demands on the eye-tracking hardware module; for example, the number of infrared LEDs must meet certain conditions. When the number of infrared LEDs is insufficient, no effective light spot can be detected in the eye image, making it impossible to solve for the 3D eye model parameters. This results in the inability to fit the 3D eye model.

[0004] While existing methods based on flicker-free infrared LEDs can solve the technical problems of flicker-reflection methods based on infrared LEDs, these methods only use the 3D circle corresponding to the 2D ellipse to estimate the center of the eyeball. However, this method cannot obtain eyeball parameters such as the center of curvature and radius of curvature, resulting in a low accuracy of the fitted 3D eyeball model. Summary of the Invention

[0005] This application provides a method and electronic device for fitting a 3D eye model, which can improve the accuracy of the fitted 3D eye model.

[0006] In a first aspect, embodiments of this application provide a method for fitting a 3D eye model, the method comprising:

[0007] In response to the eye model fitting command, each eye image of the user is acquired, wherein each eye image is an image obtained when the user's eyes are focused on a fixation point at different positions, and the number of each eye image is the same as the number of fixation points;

[0008] Using the image position coordinates of the user's pupil center in each eye image and the standard visual axis vector corresponding to each eye image, the user's eye parameters are obtained. The eye parameters include the position vector of the eye center, radius of curvature, first distance, second distance, and target angle. The first distance is the distance between the pupil center and the eye center, the second distance is the distance between the center of curvature and the eye center, and the target angle includes the horizontal angle and the vertical angle between the visual axis and the optical axis.

[0009] At specified time intervals, the current eye image is acquired, and based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, the user's current eye rotation angle is obtained; and,

[0010] Based on the user's current eye rotation angle and the user's eye parameters, the user's current eye model parameters are obtained, wherein the eye model parameters include the position vector of the eyeball center, the position vector of the pupil center, the position vector of the curvature center, and the visual axis vector; and,

[0011] The preset initial eye model is updated using the user's current eye model parameters to obtain the user's current 3D eye model.

[0012] A second aspect of this application provides an electronic device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0013] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0014] In response to the eye model fitting command, each eye image of the user is acquired, wherein each eye image is an image obtained when the user's eyes are focused on a fixation point at different positions, and the number of each eye image is the same as the number of fixation points;

[0015] Using the image position coordinates of the user's pupil center in each eye image and the standard visual axis vector corresponding to each eye image, the user's eye parameters are obtained. The eye parameters include the position vector of the eye center, radius of curvature, first distance, second distance, and target angle. The first distance is the distance between the pupil center and the eye center, the second distance is the distance between the center of curvature and the eye center, and the target angle includes the horizontal angle and the vertical angle between the visual axis and the optical axis.

[0016] At specified time intervals, the current eye image is acquired, and based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, the user's current eye rotation angle is obtained; and,

[0017] Based on the user's current eye rotation angle and the user's eye parameters, the user's current eye model parameters are obtained, wherein the eye model parameters include the position vector of the eyeball center, the position vector of the pupil center, the position vector of the curvature center, and the visual axis vector; and,

[0018] The preset initial eye model is updated using the user's current eye model parameters to obtain the user's current 3D eye model.

[0019] According to a third aspect of the present invention, a computer storage medium is provided, the computer storage medium storing a computer program for performing the method as described in the first aspect.

[0020] In the above embodiments of this application, the user's eye parameters are obtained by utilizing the image position coordinates of the user's pupil center in each eye image obtained when the user's eyes are focused at different positions, and the standard visual axis vector corresponding to each eye image. These parameters include the position vector of the eye center, the radius of curvature, the distance between the pupil center and the eye center, the distance between the center of curvature and the eye center, and the horizontal and vertical angles between the visual axis and the optical axis. Then, at specified intervals, the current eye image is acquired, and the user's current eye rotation angle is obtained based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image. Furthermore, based on the user's current eye rotation angle and the user's eye parameters, the user's current eye model parameters are obtained, namely the position vector of the eye center, the position vector of the pupil center, the position vector of the center of curvature, and the visual axis vector. The user's current eye model parameters are used to update the preset initial eye model to obtain the user's current 3D eye model. Therefore, this embodiment does not require infrared LEDs, and in addition to obtaining the center of the eyeball, it can also obtain other parameters such as the center of curvature and the radius of curvature, thus improving the accuracy of the 3D eyeball model. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1An exemplary schematic diagram of an eye image provided in an embodiment of this application is shown;

[0023] Figure 2 An exemplary diagram illustrating an application scenario provided in an embodiment of this application is shown;

[0024] Figure 3 An exemplary flowchart of a fitting method for a 3D eye model provided in an embodiment of this application is shown.

[0025] Figure 4 An exemplary illustration is shown in the form of a schematic diagram for acquiring an eye image, provided in an embodiment of this application.

[0026] Figure 5 An exemplary diagram illustrating the current eyeball rotation angle provided in an embodiment of this application is shown;

[0027] Figure 6 An exemplary schematic diagram of the process for determining user eye parameters provided in an embodiment of this application is shown;

[0028] Figure 7 An exemplary schematic diagram of the process for determining the current eyeball rotation angle provided in an embodiment of this application is shown;

[0029] Figure 8 An exemplary illustration shows a schematic diagram of an eye model provided in an embodiment of this application;

[0030] Figure 9 An exemplary schematic diagram of a fitting device for a 3D eye model provided in an embodiment of this application is shown;

[0031] Figure 10 An exemplary structural diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0032] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0033] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0034] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0035] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to be omnipresent but not exclusive; for example, a product or device comprising a series of components is not necessarily limited to those explicitly listed, but may include other components not explicitly listed or inherent to such product or device.

[0036] As used in this application, the term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0037] The following is an overview of the ideas behind the embodiments of this application.

[0038] Currently, while the flicker-free method based on infrared LEDs can solve the technical problems existing in the flicker reflection method based on infrared LEDs, this method only uses the 3D circle corresponding to the 2D ellipse to estimate the center of the eyeball. However, this method cannot obtain eyeball parameters such as the center of curvature and radius of curvature, resulting in a low accuracy of the fitted 3D eyeball model.

[0039] The existing technology suffers from the problem that the accuracy of the fitted 3D eyeball model is low because it cannot obtain eyeball parameters such as the center of curvature and radius of curvature. This application provides a method for fitting a 3D eye model. By utilizing the image position coordinates of the user's pupil center in various eye images obtained when the user's eyes are focused at different positions, and the corresponding standard visual axis vectors for each eye image, the user's eye parameters are obtained. These parameters include the position vector of the eye center, the radius of curvature, the distance between the pupil center and the eye center, the distance between the center of curvature and the eye center, and the horizontal and vertical angles between the visual axis and the optical axis. Then, at specified intervals, the current eye image is acquired, and based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, the user's current eye rotation angle is obtained. Furthermore, based on the user's current eye rotation angle and the user's eye parameters, the user's current eye model parameters are obtained, including the position vector of the eye center, the position vector of the pupil center, the position vector of the center of curvature, and the visual axis vector. Finally, the user's current eye model parameters are used to update a preset initial eye model to obtain the user's current 3D eye model. Therefore, this embodiment does not require infrared LEDs, and in addition to obtaining the center of the eyeball, it can also obtain other parameters such as the center of curvature and the radius of curvature, thus improving the accuracy of the 3D eyeball model.

[0040] like Figure 2 As shown, an application scenario of a 2D eye model fitting method is presented, which includes a camera 201 and a server 202.

[0041] In one possible application scenario, in response to an eye model fitting command, camera 201 acquires various eye images of the user's eyes at different fixation points and sends each eye image to server 202. Upon receiving each eye image, server 202 uses the image position coordinates of the user's pupil center in each eye image and the standard visual axis vector corresponding to each eye image to obtain the user's eye parameters. These eye parameters include the position vector of the eye center, radius of curvature, a first distance, a second distance, and a target angle. The first distance is the distance between the pupil center and the eye center, the second distance is the distance between the center of curvature and the eye center, and the target angle includes the horizontal and vertical angles between the visual axis and the optical axis. Then, server 202 acquires the current eye image at specified intervals and, based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, obtains the user's current eye rotation angle. Then, server 202 obtains the user's current eye model parameters based on the user's current eye rotation angle and the user's eye parameters. The eye model parameters include the position vector of the eye center, the position vector of the pupil center, the position vector of the curvature center, and the visual axis vector. The server 202 then updates the preset initial eye model using the user's current eye model parameters to obtain the user's current 2D eye model.

[0042] in, Figure 1 The camera 201 and the server 202 can exchange information through a communication network. The communication network can be either wireless or wired.

[0043] For example, camera 201 can access the network and communicate with server 202 via cellular mobile communication technology, wherein the cellular mobile communication technology includes, for example, 5th Generation Mobile Networks (5G) technology.

[0044] Optionally, the camera 201 can access the network and communicate with the server 202 via short-range wireless communication, wherein the short-range wireless communication method includes, for example, Wireless Fidelity (Wi-Fi) technology.

[0045] The description in this application focuses only on a single camera 201 and a single server 202. However, those skilled in the art should understand that the illustrated camera 201 and single server 202 are intended to illustrate the operation of the camera 201 and server 202 involved in the technical solution of this application, and do not imply any limitation on the number, type, or location of the camera 201 and server 202. It should be noted that adding additional modules to or removing individual modules from the illustrated environment will not change the underlying concept of the exemplary embodiments of this application.

[0046] It should be noted that the fitting method for the 3D eye model proposed in this application is not only applicable to... Figure 2 The application scenarios shown can also be applied to any fitting device with a 3D eye model.

[0047] The following describes a method for fitting a 3D eye model in an exemplary embodiment of this application, in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the methods and principles of this application, and the implementation of this application is not limited in any way in this respect.

[0048] like Figure 3 The diagram shown illustrates a process for fitting a 3D eye model, which may include the following steps:

[0049] Step 301: In response to the eye model fitting command, acquire each eye image of the user, wherein each eye image is an image obtained when the user's eyes are focused on different fixation points, and the number of each eye image is the same as the number of fixation points;

[0050] The fixation points in this embodiment are pre-set, and the number of fixation points used in this embodiment is 9. Therefore, the number of images of each eye of the user acquired in this embodiment is 9. Figure 4 The image shown is a schematic diagram for obtaining an eye image. Figure 4 As can be seen from the figure, there are 9 annotation points. The user's eyes fixate on each fixation point in sequence. Each time the user fixates on a fixation point, the current eye image of the user is obtained to obtain the eye image.

[0051] It should be noted that: the number of each fixation point in this application embodiment can be greater than or equal to 9. The number of fixation points in this application embodiment can be set according to the specific actual situation. This application embodiment does not limit the number of fixation points.

[0052] Step 302: Using the image position coordinates of the user's pupil center in each eye image and the standard visual axis vector corresponding to each eye image, obtain the user's eye parameters. The eye parameters include the position vector of the eye center, radius of curvature, first distance, second distance, and target angle. The first distance is the distance between the pupil center and the eye center, the second distance is the distance between the center of curvature and the eye center, and the target angle includes the horizontal angle and the vertical angle between the visual axis and the optical axis.

[0053] In this embodiment, the method for determining the image position coordinates of the pupil center in any eye image can be as follows:

[0054] Method 1: A target detection algorithm is used to determine the image position coordinates of the pupil's bounding box in any given eye image. Then, the image position coordinates of the pupil's center in that eye image are obtained using the bounding box's coordinates. In other words, the pupil center in this embodiment is the center of the bounding box.

[0055] Method 2: Use a pre-trained neural network model to detect the pupil center in any eye image to obtain the image position coordinates of the pupil center in any eye image.

[0056] The method for determining the image position coordinates of the pupil center in any eye image in this embodiment is only for illustrative purposes and is not intended to limit the determination method. The method for determining the image position coordinates of the pupil center in any eye image in this embodiment can be limited according to the specific actual situation.

[0057] To further improve the accuracy of the model, in one embodiment, the image position coordinates of the pupil center in any eye image are distorted.

[0058] The distortion processing method in this application embodiment is not limited, and can be set according to the actual situation.

[0059] Step 303: At specified intervals, acquire the current eye image, and based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, obtain the user's current eye rotation angle;

[0060] In this embodiment, the current eye rotation angle includes a first eye rotation angle and a second eye rotation angle. The first eye rotation angle is the angle between the vertical projection of the optical axis onto a specified plane in the eye's system coordinate system and the Z-axis. The second eye rotation angle is the angle between the vertical projection of the optical axis onto a specified plane in the eye's system coordinate system and the optical axis. Furthermore, in this embodiment, the origin of the eye's volume coordinate system is at the center of the eye, and the orientation of its XYZ axes is consistent with the world coordinate system.

[0061] In this embodiment, the specified plane is the XOZ plane, but the specified plane is not limited and can be set according to the actual situation.

[0062] like Figure 5 The image shown is a schematic diagram of the current eye rotation angle, where the origin o is the center of the user's eyeball. Figure 5 θ eye φ is the first eyeball rotation angle. eye This refers to the rotation angle of the second eyeball.

[0063] Below, we will first describe the method for determining the user's eye parameters in step 302 of this application embodiment, such as... Figure 6 The diagram shown illustrates the process for determining a user's eye parameters, which may include the following steps:

[0064] Step 601: For any eye image, based on the image position coordinates of the user's pupil center in the eye image and the standard visual axis vector corresponding to the eye image, obtain the first set of constraint equations corresponding to the eye image.

[0065] In one embodiment, the standard visual axis vector corresponding to each of the eye images is determined in the following manner:

[0066] For any one of the eye images, based on the image identifier of the eye image, a target gaze point corresponding to the eye image is determined; using the pre-set correspondence between each gaze point and each standard visual axis vector, a standard visual axis vector corresponding to the target gaze point is determined, and the determined standard visual axis vector is set as the standard visual axis vector corresponding to the eye image.

[0067] In this embodiment of the application, the gaze points also have corresponding identifiers. The gaze points whose identifiers are the same as those of any eye image are determined as the target gaze points that are the same as those of any eye image.

[0068] Step 602: Obtain the user's eye parameters using the first constraint equations corresponding to each of the eye images. The first constraint equations corresponding to any one of the eye images can be obtained using the following formula (1):

[0069]

[0070] in, The standard view axis vector, Let g′ be the predicted visual axis vector corresponding to any given eye image. x These are the horizontal position coordinates in the standard view vector. To predict the horizontal position coordinates in the view vector, g′ y This represents the ordinate coordinate value in the standard view vector. To predict the ordinate coordinates in the view vector, Let k be the position vector of the center of curvature. g k is the pre-set first scaling factor. g >0, θ eye φ is the first eyeball rotation angle corresponding to any given eye image. eye α is the second eye rotation angle corresponding to any given eye image. eye β is the horizontal angle between the visual axis and the optical axis. eye The vertical angle between the visual axis and the optical axis; Let D be the position vector of the center of the eyeball, and D be the second distance. Let k be the preset position vector of the camera's optical center. r This is the second scaling factor. Let R be the vector corresponding to the image position coordinates of the user's pupil center in any given eye image, and let R be the radius of curvature. Let K be the position vector of the center of the pupil, K be the first distance, n1 be the pre-set waterproof refractive index of the cornea, and n2 be the preset air refractive index.

[0071] The unknown parameter that needs to be solved in the first set of constraint equations in this application embodiment is: the first eyeball rotation angle θ corresponding to any eye image. eye The second eye rotation angle φ corresponding to any given eye image eye Second scaling factor k r The horizontal angle α between the visual axis and the optical axis eye The vertical angle β between the visual axis and the optical axis eye The position vector of the center of the eyeball The second distance D, the radius of curvature R, and the first distance K. Wherein, the horizontal angle α between the visual axis and the optical axis... eyeThe vertical angle β between the visual axis and the optical axis eye The position vector of the center of the eyeball The second distance D and the radius of curvature R are constant, while the first distance K remains unchanged. However, the first eyeball rotation angle θ corresponding to any given eye image is different. eye The second eye rotation angle φ corresponding to any given eye image eye Second scaling factor k r The parameters change with eye movement. Furthermore, in this embodiment, each eye image has a corresponding set of first constraint equations. Therefore, by solving a system of equations composed of these first constraint equations, the eye parameters can be obtained. The number of unknown parameters in the system of equations composed of these first constraint equations is 6 + 27 = 33, and the total number of equations in the system is 54. Since there are 33 unknowns in these 54 equations, the system of equations has a solution.

[0072] However, this application does not limit the solution method for the system of equations; it can be set according to the specific circumstances. Although this application also solves for θ corresponding to each eye image. eye φ eye and k r However, this parameter changes, so in step 303, the lower eye rotation angle and kr need to be redefined for each acquired eye image. The user's eye parameters, however, remain constant.

[0073] The following explains the method for determining the user's current eye rotation angle in step 303, such as... Figure 7 The diagram shown illustrates the process for determining the current eye rotation angle, which may include the following steps:

[0074] Step 701: Based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, obtain the second set of constraint equations corresponding to the current eye rotation angle; wherein, the second set of constraint equations corresponding to the current eye rotation angle is obtained through formula (2):

[0075]

[0076] Where, φ eye ′ represents the current second eye rotation angle, θ eye ′ represents the current first eyeball rotation angle. Let k be the preset position vector of the camera's optical center. r This is the second scaling factor. R is the vector corresponding to the image position coordinates of the user's pupil center in the current eye image, and R is the radius of curvature. Let K be the position vector of the pupil center, D be the first distance, n1 be the preset waterproof refractive index of the cornea, and n2 be the preset air refractive index. Let α be the position vector of the center of the eyeball. eye β is the horizontal angle between the visual axis and the optical axis. eye It is the vertical angle between the visual axis and the optical axis.

[0077] Step 702: Using the second set of constraint equations, obtain the user's current eye rotation angle.

[0078] In this embodiment, the user's eye parameters—the horizontal angle α between the visual axis and the optical axis—have already been determined. eye The vertical angle β between the visual axis and the optical axis eye The position vector of the center of the eyeball Given the second distance D, radius of curvature R, and the first distance K, the unknown parameter in the second set of constraint equations is: the current second eyeball rotation angle φ. eye ′, the current first eyeball rotation angle θ eye ′, second scaling factor k r Therefore, it can be concluded that the current eye rotation angle can be determined directly based on the equations in the second set of constraint equations.

[0079] It should be noted that the specific solution method for the second constraint equation is not limited in this embodiment of the application, and can be set according to the actual situation.

[0080] Step 304: Based on the user's current eye rotation angle and the user's eye parameters, obtain the user's current eye model parameters, wherein the eye model parameters include the position vector of the eye center, the position vector of the pupil center, the position vector of the curvature center, and the visual axis vector;

[0081] In this embodiment, the position vector of the eyeball center has already been determined when determining the user's eyeball parameters. Since this parameter is constant, it does not need to be determined again in this step. The following describes the method for determining the position vector of the user's current curvature center, the position vector of the current pupil center, and the current visual axis vector in this embodiment:

[0082] In one embodiment, the position vector of the user's current curvature center can be obtained through formula (3):

[0083]

[0084] in, Let be the position vector of the user's current curvature center. Let φ be the position vector of the center of the eyeball, D be the second distance, and φ be the position vector of the center of the eyeball. eye ′ represents the current second eye rotation angle, θ eye ′ represents the current first eyeball rotation angle;

[0085] In one embodiment, the position vector of the user's current pupil center can be obtained by formula (4):

[0086]

[0087] in, Let K be the position vector of the current pupil center of the user, and K be the first distance;

[0088] In one embodiment, the user's current view vector can be obtained using formula (5):

[0089]

[0090] in, Let k be the user's current view vector. g k is the pre-set first scaling factor. g >0, α eye β is the horizontal angle between the visual axis and the optical axis. eye It is the vertical angle between the visual axis and the optical axis.

[0091] Step 305: Update the preset initial eye model using the user's current eye model parameters to obtain the user's current 3D eye model.

[0092] In this embodiment, an initial eye model is pre-suggested based on the initial eye model parameters. Each time the current eye model parameters are obtained, the initial eye model is directly updated based on the current eye model parameters to update the initial eye model and obtain the current 3D eye model.

[0093] like Figure 8 The diagram shows a schematic of an eye model, where point d is the center of the eyeball, point c is the center of curvature, point p is the center of the pupil, point o is the optical center of the camera, point v is the image pixel coordinate of the pupil center in the current eye image, and point r is the intersection of the ray formed by the image pixel coordinates of the pupil center in the current eye image and the corneal surface. This embodiment can adjust the position of the eyeball center in the eye model based on the determined position vector of the eyeball center, and adjust the position of the pupil center in the eye model based on the position vector of the pupil center, and adjust the position of the center of curvature in the eye model based on the position vector of the center of curvature, and adjust the position of the visual axis in the eye model based on the visual axis vector, thus obtaining the corresponding 3D eye model.

[0094] Based on the same inventive concept, the 3D eye model fitting method described above can also be implemented by a 3D eye model fitting device. The effect of this 3D eye model fitting device is similar to that of the aforementioned method, and will not be described again here.

[0095] Figure 9 This is a schematic diagram of the structure of a fitting device for a 3D eye model according to an embodiment of the present disclosure.

[0096] like Figure 9 As shown, the 3D eye model fitting device 900 disclosed herein may include an eye image acquisition module 910, an eye parameter determination module 920, an eye rotation angle determination module 930, an eye model parameter determination module 940, and an eye model determination module 950.

[0097] The eye image acquisition module 910 is used to acquire various eye images of the user in response to an eye model fitting command, wherein each eye image is an image obtained when the user's eyes are focused on a fixation point at different positions, and the number of each eye image is the same as the number of fixation points;

[0098] The eye parameter determination module 920 is used to obtain the user's eye parameters by using the image position coordinates of the user's pupil center in each eye image and the standard visual axis vector corresponding to each eye image. The eye parameters include the position vector of the eye center, the radius of curvature, a first distance, a second distance, and a target angle. The first distance is the distance between the pupil center and the eye center, the second distance is the distance between the center of curvature and the eye center, and the target angle includes the horizontal angle and the vertical angle between the visual axis and the optical axis.

[0099] The eye rotation angle determination module 930 is used to acquire the current eye image at specified time intervals, and obtain the user's current eye rotation angle based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image; and,

[0100] The eye model parameter determination module 940 is used to obtain the user's current eye model parameters based on the user's current eye rotation angle and the user's eye parameters, wherein the eye model parameters include the position vector of the eye center, the position vector of the pupil center, the position vector of the curvature center, and the visual axis vector; and,

[0101] The eye model determination module 950 is used to update the preset initial eye model using the user's current eye model parameters to obtain the user's current 3D eye model.

[0102] In one embodiment, the eye parameter determination module 920 is specifically used for:

[0103] For any eye image, based on the image position coordinates of the user's pupil center in the eye image and the standard visual axis vector corresponding to the eye image, a first set of constraint equations corresponding to the eye image is obtained;

[0104] The user's eye parameters are obtained by using the first set of constraint equations corresponding to each of the eye images.

[0105] In one embodiment, the current eye rotation angle includes a current first eye rotation angle and a current second eye rotation angle. The first eye rotation angle is the angle between the projection of the optical axis in the XOZ plane of the eye's system coordinate system and the Z-axis. The second eye rotation angle is the angle between the projection of the optical axis in the YOZ plane of the eye's system coordinate system and the optical axis.

[0106] The eye parameter determination module 920 executes the first set of constraint equations corresponding to the arbitrary eye image, based on the image position coordinates of the user's pupil center in any eye image and the standard visual axis vector corresponding to the arbitrary eye image, specifically used for:

[0107] The first set of constraint equations corresponding to any given eye image is obtained using the following formula:

[0108]

[0109] in, The standard view axis vector, Let g′ be the predicted visual axis vector corresponding to any given eye image. x These are the horizontal position coordinates in the standard view vector. To predict the horizontal position coordinates in the view vector, g′ y This represents the ordinate coordinate value in the standard view vector. To predict the ordinate coordinates in the view vector, Let k be the position vector of the center of curvature. g k is the pre-set first scaling factor. g >0, θ eye φ is the first eyeball rotation angle corresponding to any given eye image. eye α is the second eye rotation angle corresponding to any given eye image. eye β is the horizontal angle between the visual axis and the optical axis. eye The vertical angle between the visual axis and the optical axis; Let D be the position vector of the center of the eyeball, and D be the second distance. Let k be the preset position vector of the camera's optical center. r This is the second scaling factor. Let R be the vector corresponding to the image position coordinates of the user's pupil center in any given eye image, and let R be the radius of curvature. Let K be the position vector of the center of the pupil, K be the first distance, n1 be the pre-set waterproof refractive index of the cornea, and n2 be the preset air refractive index.

[0110] In one embodiment, the apparatus further includes:

[0111] The standard visual axis vector determination module 960 is used to determine the standard visual axis vector corresponding to each of the eye images in the following manner:

[0112] For any one of the eye images, based on the image identifier of the eye image, determine the target gaze point corresponding to the eye image.

[0113] By utilizing the pre-set correspondence between each fixation point and each standard visual axis vector, the standard visual axis vector corresponding to the target fixation point is determined, and the determined standard visual axis vector is set as the standard visual axis vector corresponding to any one of the eye images.

[0114] In one embodiment, the eye rotation angle determination module 930 is specifically used for:

[0115] Based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, a second set of constraint equations corresponding to the current eye rotation angle is obtained;

[0116] Using the second set of constraint equations, the user's current eye rotation angle is obtained.

[0117] In one embodiment, the current eye rotation angle includes a current first eye rotation angle and a current second eye rotation angle. The first eye rotation angle is the angle between the projection of the optical axis onto the XOZ plane in the system coordinate system of the eye and the Z axis. The second eye rotation angle is the angle between the projection of the optical axis onto the YOZ plane in the system coordinate system of the eye and the optical axis.

[0118] The eye rotation angle determination module 930 executes the process based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image to obtain a second set of constraint equations corresponding to the current eye rotation angle, specifically used for:

[0119] The second set of constraint equations corresponding to the current eyeball rotation angle is obtained through the following formula:

[0120]

[0121] Where, φ eye ′ represents the current second eye rotation angle, θ eye ′ represents the current first eyeball rotation angle. Let k be the preset position vector of the camera's optical center. r This is the second scaling factor. R is the vector corresponding to the image position coordinates of the user's pupil center in the current eye image, and R is the radius of curvature. Let K be the position vector of the pupil center, D be the first distance, n1 be the preset waterproof refractive index of the cornea, and n2 be the preset air refractive index. Let α be the position vector of the center of the eyeball. eye β is the horizontal angle between the visual axis and the optical axis. eye It is the vertical angle between the visual axis and the optical axis.

[0122] In one embodiment, the current eye rotation angle includes a current first eye rotation angle and a current second eye rotation angle. The first eye rotation angle is the angle between the projection of the optical axis onto the XOZ plane in the system coordinate system of the eye and the Z axis. The second eye rotation angle is the angle between the projection of the optical axis onto the YOZ plane in the system coordinate system of the eye and the optical axis.

[0123] The eyeball model parameter determination module 940 is specifically used for:

[0124] The current position vector of the user's center of curvature can be obtained using the following formula:

[0125]

[0126] in, Let be the position vector of the user's current curvature center. Let φ be the position vector of the center of the eyeball, D be the second distance, and φ be the position vector of the center of the eyeball. eye ′ represents the current second eye rotation angle, θ eye ′ represents the current first eyeball rotation angle;

[0127] The current position vector of the user's pupil center can be obtained using the following formula:

[0128]

[0129] in, Let K be the position vector of the current pupil center of the user, and K be the first distance;

[0130] The user's current view vector is obtained using the following formula:

[0131]

[0132] in, Let k be the user's current view vector. g k is the pre-set first scaling factor. g >0, α eye β is the horizontal angle between the visual axis and the optical axis. eye It is the vertical angle between the visual axis and the optical axis.

[0133] After introducing a fitting method and apparatus for a 3D eye model according to an exemplary embodiment of the present invention, the following describes an electronic device according to another exemplary embodiment of the present invention.

[0134] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as "circuit", "module", or "system".

[0135] In some possible implementations, the IoT module according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code that, when executed by the processor, causes the processor to perform the steps in the 3D eye model fitting method according to various exemplary embodiments of the present invention described above. For example, the processor may perform actions such as... Figure 3 Steps 301-305 are shown in the diagram.

[0136] The following reference Figure 10 To describe an electronic device 1000 according to this embodiment of the present invention. Figure 10 The electronic device 1000 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0137] like Figure 10 As shown, the electronic device 1000 is presented in the form of a general AR device. The components of the electronic device 1000 may include, but are not limited to: at least one processor 1001, at least one computer storage medium 1002, and a bus 1003 connecting different system components (including the computer storage medium 1002 and the processor 1001).

[0138] Bus 1003 represents one or more of several bus structures, including computer storage media bus or computer storage media controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0139] Computer storage medium 1002 may include readable media in the form of volatile computer storage media, such as random access computer storage medium (RAM) 1021 and / or cache storage medium 1022, and may further include read-only computer storage medium (ROM) 1023.

[0140] The computer storage medium 1002 may also include a program / utility 1025 having a set (at least one) of program modules 1024, such program modules 1024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0141] Electronic device 1000 can also communicate with one or more external devices 1004 (e.g., keyboard, pointing device, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other AR devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1005. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1006. As shown, network adapter 1006 communicates with other modules used in electronic device 1000 via bus 1003. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0142] In some possible implementations, various aspects of the network connection method for an electronic device provided by the present invention can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in the network connection method for an electronic device according to various exemplary embodiments of the present invention described above.

[0143] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A fitting method for a 3D eye model, characterized in that, The method includes: In response to the eye model fitting command, each eye image of the user is acquired, wherein each eye image is an image obtained when the user's eyes are focused on a fixation point at different positions, and the number of each eye image is the same as the number of fixation points; Using the image position coordinates of the user's pupil center in each eye image and the standard visual axis vector corresponding to each eye image, the user's eye parameters are obtained. The eye parameters include the position vector of the eye center, radius of curvature, first distance, second distance, and target angle. The first distance is the distance between the pupil center and the eye center, the second distance is the distance between the center of curvature and the eye center, and the target angle includes the horizontal angle and the vertical angle between the visual axis and the optical axis. At specified time intervals, the current eye image is acquired, and based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, the user's current eye rotation angle is obtained; and, Based on the user's current eye rotation angle and the user's eye parameters, the user's current eye model parameters are obtained, wherein the eye model parameters include the position vector of the eyeball center, the position vector of the pupil center, the position vector of the curvature center, and the visual axis vector; and, The preset initial eye model is updated using the user's current eye model parameters to obtain the user's current 3D eye model.

2. The method according to claim 1, characterized in that, The process of obtaining the user's eye parameters by utilizing the image position coordinates of the user's pupil center in each eye image and the standard visual axis vector corresponding to each eye image includes: For any eye image, based on the image position coordinates of the user's pupil center in the eye image and the standard visual axis vector corresponding to the eye image, a first set of constraint equations corresponding to the eye image is obtained; The user's eye parameters are obtained by using the first set of constraint equations corresponding to each of the eye images.

3. The method according to claim 2, characterized in that, The current eye rotation angle includes the current first eye rotation angle and the current second eye rotation angle. The first eye rotation angle is the angle between the vertical projection of the optical axis in the specified plane of the eye's system coordinate system and the Z-axis. The second eye rotation angle is the angle between the vertical projection of the optical axis in the specified plane of the eye's system coordinate system and the optical axis. Based on the image position coordinates of the user's pupil center in any eye image and the standard visual axis vector corresponding to any eye image, a first set of constraint equations corresponding to any eye image is obtained, including: The first set of constraint equations corresponding to any given eye image is obtained using the following formula: in, The standard view axis vector, Let g′ be the predicted visual axis vector corresponding to any given eye image. x These are the horizontal position coordinates in the standard view vector. To predict the horizontal position coordinates in the view vector, g′ y This represents the ordinate coordinate value in the standard view vector. To predict the ordinate coordinates in the view vector, Let k be the position vector of the center of curvature. g k is the pre-set first scaling factor. g >0, θ eye φ is the first eyeball rotation angle corresponding to any given eye image. eye α is the second eye rotation angle corresponding to any given eye image. eye β is the horizontal angle between the visual axis and the optical axis. eye The vertical angle between the visual axis and the optical axis; Let D be the position vector of the center of the eyeball, and D be the second distance. Let k be the preset position vector of the camera's optical center. r This is the second scaling factor. Let R be the vector corresponding to the image position coordinates of the user's pupil center in any given eye image, and let R be the radius of curvature. Let K be the position vector of the center of the pupil, K be the first distance, n1 be the pre-set waterproof refractive index of the cornea, and n2 be the preset air refractive index.

4. The method according to any one of claims 1 to 3, characterized in that, The standard visual axis vector corresponding to each of the aforementioned eye images is determined in the following manner: For any one of the eye images, based on the image identifier of the eye image, determine the target gaze point corresponding to the eye image. By utilizing the pre-set correspondence between each fixation point and each standard visual axis vector, the standard visual axis vector corresponding to the target fixation point is determined, and the determined standard visual axis vector is set as the standard visual axis vector corresponding to any one of the eye images.

5. The method according to claim 1, characterized in that, The process of obtaining the user's current eye rotation angle based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image includes: Based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, a second set of constraint equations corresponding to the current eye rotation angle is obtained; Using the second set of constraint equations, the user's current eye rotation angle is obtained.

6. The method according to claim 5, characterized in that, The current eye rotation angle includes a current first eye rotation angle and a current second eye rotation angle. The first eye rotation angle is the angle between the vertical projection of the optical axis in a specified plane in the eye's system coordinate system and the Z-axis. The second eye rotation angle is the angle between the vertical projection of the optical axis in a specified plane in the eye's system coordinate system and the optical axis. Based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, a second set of constraint equations corresponding to the current eye rotation angle is obtained, including: The second set of constraint equations corresponding to the current eyeball rotation angle is obtained through the following formula: Where, φ eye ′ represents the current second eye rotation angle, θ eye ′ represents the current first eyeball rotation angle. Let k be the preset position vector of the camera's optical center. r This is the second scaling factor. R is the vector corresponding to the image position coordinates of the user's pupil center in the current eye image, and R is the radius of curvature. Let K be the position vector of the pupil center, D be the first distance, n1 be the preset waterproof refractive index of the cornea, and n2 be the preset air refractive index. Let α be the position vector of the center of the eyeball. eye β is the horizontal angle between the visual axis and the optical axis. eye It is the vertical angle between the visual axis and the optical axis.

7. The method according to claim 1, characterized in that, The current eye rotation angle includes a current first eye rotation angle and a current second eye rotation angle. The first eye rotation angle is the angle between the vertical projection of the optical axis in a specified plane in the eye's system coordinate system and the Z-axis. The second eye rotation angle is the angle between the vertical projection of the optical axis in a specified plane in the eye's system coordinate system and the optical axis. The step of obtaining the user's current eye model parameters based on the user's current eye rotation angle and the user's eye parameters includes: The current position vector of the user's center of curvature can be obtained using the following formula: in, Let be the position vector of the user's current curvature center. Let φ be the position vector of the center of the eyeball, D be the second distance, and φ be the position vector of the center of the eyeball. eye ′ represents the current second eye rotation angle, θ eye ′ represents the current first eyeball rotation angle; The current position vector of the user's pupil center can be obtained using the following formula: in, Let K be the position vector of the current pupil center of the user, and K be the first distance; The user's current view vector is obtained using the following formula: in, Let k be the user's current view vector. g k is the pre-set first scaling factor. g >0, α eye β is the horizontal angle between the visual axis and the optical axis. eye It is the vertical angle between the visual axis and the optical axis.

8. An electronic device, characterized in that, It includes a processor and a memory, which are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: In response to the eye model fitting command, each eye image of the user is acquired, wherein each eye image is an image obtained when the user's eyes are focused on a fixation point at different positions, and the number of each eye image is the same as the number of fixation points; The eye parameters of the user are obtained by using the image position coordinates of the user's pupil center in each eye image and the standard visual axis vector corresponding to each eye image. At specified time intervals, the current eye image is acquired, and based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, the user's current eye rotation angle is obtained; and, Based on the user's current eye rotation angle and the user's eye parameters, the user's current eye model parameters are obtained; and, The preset initial eye model is updated using the user's current eye model parameters to obtain the user's current 3D eye model.

9. The electronic device according to claim 8, characterized in that, The processor executes the process of obtaining the user's eye parameters by using the image position coordinates of the user's pupil center in each eye image and the standard visual axis vector corresponding to each eye image, specifically configured as follows: For any eye image, based on the image position coordinates of the user's pupil center in the eye image and the standard visual axis vector corresponding to the eye image, a first set of constraint equations corresponding to the eye image is obtained; The user's eye parameters are obtained by using the first set of constraint equations corresponding to each of the eye images.

10. The electronic device according to claim 8, characterized in that, The processor executes the process of obtaining the user's current eye rotation angle based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, specifically configured as follows: Based on the user's eye parameters and the image position coordinates of the pupil center in the current eye image, a second set of constraint equations corresponding to the current eye rotation angle is obtained; Using the second set of constraint equations, the user's current eye rotation angle is obtained.

Images