Eyeball modeling parameter determination method, sight line direction determination method and device

Abstract

The invention provides an eyeball modeling parameter determination method and device and a line-of-sight direction determination method and device, and relates to the technical field of line-of-sight tracking, and the method comprises the steps: inputting a first initial modeling parameter of an eyeball model into a light spot position constraint model, and obtaining a predicted position parameter of a target light spot; inputting the second initial modeling parameter of the eyeball model into a pupil refraction point position constraint model to obtain a predicted position parameter of a pupil refraction point; determining a comprehensive prediction error based on the prediction position parameter of the target light spot, the prediction position parameter of the pupil refraction point, the light spot detection result of the eye image and the pupil detection result; and if the comprehensive prediction error does not meet the error requirement of the eyeball model, updating a first initial modeling parameter and a second initial modeling parameter of the eyeball model, otherwise, determining the first initial modeling parameter and the second initial modeling parameter of the eyeball model as eyeball model modeling parameters. The eyeball modeling parameter determination method is used for obtaining the sight line direction of the user.

Description

Technical Field

[0001] The present invention relates to the technical field of gaze tracking, and in particular, to a method for determining eye modeling parameters, a method for determining gaze direction, and a device therefor. Background Art

[0002] Gaze tracking technology requires the algorithm to calculate gaze direction information based on the captured image, and then perform subsequent operations such as interaction and rendering through the obtained gaze direction.

[0003] However, currently, the algorithm calculates the gaze direction information by assuming the eye geometry parameters of the user, and then solves the gaze angle of the user. Since the eye set parameters of the user vary from person to person, the above assumption will introduce calculation deviation and reduce the accuracy of the obtained gaze angle of the user. Summary of the Invention

[0004] According to one aspect of the present disclosure, there is provided a method for determining eye modeling parameters, the method comprising:

[0005] Input the first initial modeling parameters of the eye model into the spot position constraint model to obtain the predicted position parameters of the target spot, where the spot position constraint model is determined by the first initial modeling parameters of the eye model, the position parameters of the target spot, the position parameters of the target camera, and the position parameters of the target light source;

[0006] Input the second initial modeling parameters of the eye model into the pupil refraction point position constraint model to obtain the predicted position parameters of the pupil refraction point, where the pupil refraction point position constraint model is determined by the second initial modeling parameters of the eye model, the position parameters of the pupil refraction point, and the position parameters of the target camera;

[0007] Determine a comprehensive prediction error based on the predicted position parameters of the target spot, the predicted position parameters of the pupil refraction point, the spot detection result and the pupil detection result of the eye image;

[0008] If the comprehensive prediction error does not meet the error requirement of the eye model, update the first initial modeling parameters and the second initial modeling parameters of the eye model based on the gradient results of the spot position constraint model and the pupil refraction point position constraint model, otherwise, determine the eye model modeling parameters based on the first initial modeling parameters and the second initial modeling parameters of the eye model.

[0009] According to another aspect of the present disclosure, there is provided a method for determining gaze direction, the method comprising:

[0010] Detect the eye image to be detected based on the eye model modeling parameters to obtain a target spot detection result and a pupil refraction point detection result;

[0011] Based on the target light spot detection result and the pupil refraction point detection result, obtain the target line-of-sight direction;

[0012] Among them, the modeling parameters of the eyeball model are determined by the method for determining the modeling parameters of the eyeball model described in the exemplary embodiments of the present disclosure.

[0013] According to another aspect of the present disclosure, there is provided an apparatus for determining modeling parameters of an eyeball model, the apparatus including:

[0014] An obtaining module, configured to input the first initial modeling parameters of the eyeball model into the light spot position constraint model to obtain the predicted position parameters of the target light spot, where the light spot position constraint model is determined by the first initial modeling parameters of the eyeball model, the position parameters of the target light spot, the position parameters of the target camera, and the position parameters of the target light source;

[0015] The obtaining module is further configured to input the second initial modeling parameters of the eyeball model into the pupil refraction point position constraint model to obtain the predicted position parameters of the pupil refraction point, where the pupil refraction point position constraint model is determined by the second initial modeling parameters of the eyeball model, the position parameters of the pupil refraction point, and the position parameters of the target camera;

[0016] A determining module, configured to determine a comprehensive prediction error based on the predicted position parameters of the target light spot, the predicted position parameters of the pupil refraction point, the light spot detection result and the pupil detection result of the eye image;

[0017] An updating module, configured to, if the comprehensive prediction error does not meet the error requirement of the eyeball model, update the first initial modeling parameters and the second initial modeling parameters of the eyeball model based on the gradient results of the light spot position constraint model and the pupil refraction point position constraint model, otherwise, determine the modeling parameters of the eyeball model based on the first initial modeling parameters and the second initial modeling parameters of the eyeball model.

[0018] According to another aspect of the present disclosure, there is provided a line-of-sight direction determining apparatus, the apparatus including:

[0019] A detection module, configured to detect the eye image to be detected based on the modeling parameters of the eyeball model to obtain the target light spot detection result and the pupil refraction point detection result;

[0020] An obtaining module, configured to obtain the target line-of-sight direction based on the target light spot detection result and the pupil refraction point detection result;

[0021] Among them, the modeling parameters of the eyeball model are determined by the method for determining the modeling parameters of the eyeball model described in the exemplary embodiments of the present disclosure.

[0022] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0023] a processor; and,

[0024] a memory storing a program;

[0025] wherein the program includes instructions that, when executed by the processor, cause the processor to execute the method according to an exemplary embodiment of the present disclosure.

[0026] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method according to an exemplary embodiment of the present disclosure.

[0027] In one or more technical solutions provided in the embodiments of the present application, by inputting the first initial modeling parameters of the eyeball model into the spot position constraint model, the predicted position parameters of the target spot are obtained, and the spot position constraint model is determined by the first initial modeling parameters of the eyeball model, the position parameters of the target spot, the position parameters of the target camera, and the position parameters of the target light source. When the position parameters of the target light source and the position parameters of the target camera are known, the predicted position parameters of the target spot can be obtained by presetting the first initial modeling parameters of the eyeball model and substituting them into the spot position constraint model for solution. At the same time, the second initial modeling parameters of the eyeball model are input into the pupil refraction point position constraint model to obtain the predicted position parameters of the pupil refraction point, where the pupil refraction point position constraint model is determined by the second initial modeling parameters of the eyeball model, the pupil refraction point position parameters, and the position parameters of the target camera. When the position parameters of the target camera are known, the predicted position parameters of the pupil refraction point can be obtained by presetting the second initial modeling parameters of the eyeball model and substituting them into the pupil refraction point position constraint model for solution. After that, by combining the predicted position parameters of the target spot, the predicted position parameters of the pupil refraction point, the spot detection result of the eye image, and the pupil detection result, the comprehensive prediction error is determined, so that the comprehensive prediction error can reflect the difference between the spot detection result and the predicted position parameters of the target spot (defined as the spot prediction loss), and the difference between the predicted position parameters of the pupil refraction point and the pupil detection result (defined as the pupil prediction loss). Based on this, when the comprehensive prediction error meets the error requirement of the eyeball model, it indicates that the errors of the first initial modeling parameters and the second initial modeling parameters are small. Therefore, the error of determining the eyeball model modeling parameters based on the first initial modeling parameters and the second initial modeling parameters is small, and when using the eyeball model modeling parameters to determine the line of sight direction, the deviation generated is small. When the comprehensive prediction error does not meet the error requirement of the eyeball model, it indicates that the errors of the first initial modeling parameters and the second initial modeling parameters are large. Therefore, the error of determining the eyeball model modeling parameters based on the first initial modeling parameters and the second initial modeling parameters is large, and when using the eyeball model modeling parameters to determine the line of sight direction, the deviation generated is large. Since the line of sight direction is determined by two factors, the pupil and the spot, based on the spot prediction loss and the pupil prediction loss as the supervision signals at the same time, the spot prediction loss and the pupil prediction loss are jointly updated to reduce the comprehensive error, so that the error of the updated eyeball modeling parameters is small, ensuring that the deviation of using the eyeball modeling parameters to determine the line of sight direction is small.

[0028] In summary, when the eyeball modeling parameter determination method provided by the exemplary embodiment of the present disclosure is used to determine the line of sight direction, the accuracy of the obtained line of sight direction is higher. Description of the Drawings

[0029] In the following description of exemplary embodiments with reference to the accompanying drawings, more details, features, and advantages of the present disclosure are disclosed. In the drawings:

[0030] Figure 1 Schematic diagram of an eyeball model provided according to an exemplary embodiment of the present disclosure;

[0031] Figure 2 Schematic flowchart showing a method for determining eyeball modeling parameters provided according to an exemplary embodiment of the present disclosure;

[0032] Figure 3 Schematic structural diagram of a world coordinate system provided according to an exemplary embodiment of the present disclosure;

[0033] Figure 4 Schematic diagram of a two-dimensional reflection plane provided according to an exemplary embodiment of the present disclosure Figure 1 ;

[0034] Figure 5 Schematic diagram of a two-dimensional reflection plane provided according to an exemplary embodiment of the present disclosure Figure 2 ;

[0035] Figure 6 Schematic flowchart showing a method for determining a second position constraint model provided according to an exemplary embodiment of the present disclosure;

[0036] Figure 7 Schematic flowchart showing a method for determining an extension line expression of an incident light ray provided according to an exemplary embodiment of the present disclosure;

[0037] Figure 8 Schematic diagram of a two-dimensional refraction plane provided according to an exemplary embodiment of the present disclosure;

[0038] Figure 9 Schematic flowchart showing a method for determining a fourth position constraint model provided according to an exemplary embodiment of the present disclosure;

[0039] Figure 10 Schematic flowchart showing a method for determining a direction expression of a refracted light ray provided according to an exemplary embodiment of the present disclosure;

[0040] Figure 11 Schematic flowchart showing a method for determining a comprehensive prediction error provided according to an exemplary embodiment of the present disclosure;

[0041] Figure 12 Schematic flowchart showing a method for determining a line of sight direction provided according to an exemplary embodiment of the present disclosure;

[0042] Figure 13The figure shows a schematic block diagram of the functional modules of an eyeball model modeling parameter determination device according to an exemplary embodiment of the present disclosure;

[0043] Figure 14 The figure shows a schematic block diagram of the functional modules of a line-of-sight direction determination device according to an exemplary embodiment of the present disclosure;

[0044] Figure 15 The figure shows a schematic block diagram of a chip according to an exemplary embodiment of the present disclosure;

[0045] Figure 16 The figure shows a structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure. Detailed implementation manners

[0046] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0047] It should be understood that the various steps recorded in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.

[0048] As used herein, the term "including" and its variants are open-ended, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0049] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly stated in the context, it should be understood as "one or more".

[0050] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0051] Before introducing the embodiments of the present disclosure, the following interpretations are made for the relevant terms involved in the embodiments of the present disclosure:

[0052] The gaze tracking technology is a technology that uses a camera and computer algorithms to obtain and analyze the user's gaze information, and can realize the tracking and recognition of the user's fixation target.

[0053] The backpropagation algorithm is a backpropagation movement dominated by the error loss, aiming to obtain the parameters of the optimal neural network model. During the training of the neural network, the backpropagation algorithm can be used to correct the magnitudes of the parameters in the initial neural network model, so that the reconstruction error loss of the neural network model becomes smaller and smaller. The forward propagation of the input signal until the output generates an error loss, and the initial neural network model parameters are updated by backpropagating the error loss information, so that the error loss converges.

[0054] In the related art, the gaze tracking technology requires the algorithm to calculate the gaze direction information based on the captured image, and then perform subsequent operations such as interaction and rendering through the obtained gaze direction. Currently, a typical gaze tracking solution is the pupil-corneal reflection solution, and the hardware system of this solution consists of several target light sources and several target cameras. The target light source illuminates the user's pupil area and forms a specular reflection spot on the cornea. The three-dimensional position of the cornea can be solved through the law of reflection and the relative positions of the target light source and the target camera. After obtaining the three-dimensional position of the cornea, the user's gaze angle is solved through the law of refraction and the imaging of the pupil. However, this solution assumes the user's eye geometry parameters, and these parameters may vary from person to person. Therefore, such an assumption will introduce a calculation deviation, resulting in a large error in the finally solved user's gaze.

[0055] To overcome the above problems, the exemplary embodiments of the present disclosure provide a method for determining eye modeling parameters and a method for determining gaze direction, so as to determine the geometric parameters of the eye through the method for determining eye modeling parameters. Thus, when calculating the user's gaze direction, the geometric parameters of the eye can be used for calculation, so as to ensure that the obtained user's gaze direction is closer to the user's actual gaze direction, improve the accuracy of the calculated user's gaze direction, and reduce the calculation deviation.

[0056] The method provided by the exemplary embodiments of the present disclosure can be applied to an electronic device, which can be installed with various image software such as image processing software and 3D modeling engines, and can also be installed with optical simulation software for optical simulation.

[0057] The above-mentioned electronic device can be a smart phone (such as an Android phone, an iOS phone), a wearable device, an AR (augmented reality) / VR (virtual reality) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a tablet computer, a palm computer, and a mobile internet device (MID), etc., which are devices using flash memory storage devices. The electronic device can be equipped with a camera, which can capture images of any scene, such as a pure black scene, a scene with a light source, etc. The camera can be a monocular camera, a binocular camera, etc. The images captured by it can be grayscale images, color images, infrared images, ultraviolet imaging images, thermal imaging images, etc., but are not limited thereto.

[0058] The execution subject of the method for determining eye modeling parameters provided by the exemplary embodiment of the present disclosure is an electronic device or a chip applied in the electronic device. The method of the exemplary embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0059] Figure 1 Fig. shows a schematic diagram of an eye model provided by the exemplary embodiment of the present disclosure. As Figure 1 shown, the eye model 100 includes two intersecting spheres, namely a first sphere 101 and a second sphere 102. The first sphere 101 represents the white part of the eye, and the second sphere 102 represents the cornea part.

[0060] The modeling parameters of the eye model 100 can include the modeling parameters of the cornea and the modeling parameters of the pupil.

[0061] The above-mentioned modeling parameters of the cornea can include the geometric modeling parameters of the cornea, such as the cornea shape parameter and the cornea position parameter, etc., and can also include the cornea refractive index parameter, for example: the cornea refractive index μ. The cornea shape parameter can include the cornea radius r, and the cornea position parameter can include the position parameter of the cornea center o. Here, the position parameter of the cornea center o can be directly represented by the position coordinates of the cornea center o of the eye, or can be represented by the parameters of other parts, for example: it can be represented by the position parameter of the eye rotation center s, the distance d between the eye rotation center s and the cornea center o (i.e., the cornea depth), etc.

[0062] The above-mentioned modeling parameters of the pupil may include geometric modeling parameters of the pupil, such as pupil shape parameters and pupil position parameters, etc. For example: the pupil shape parameter may include the pupil depth k, and the pupil position parameter may include the position parameter of the pupil center p. Here, the position parameter of the pupil center p can be directly represented by the position coordinates of the pupil center p, or can be represented by the parameters of other parts. For example, it can be represented by the distance between the corneal center o, the corneal radius r and the pupil depth k, etc.

[0063] Figure 2 FIG. shows a schematic flowchart of a method for determining eye modeling parameters according to an exemplary embodiment of the present disclosure.

[0064] As Figure 2 shown, the method for determining eye modeling parameters provided by the exemplary embodiment of the present disclosure includes:

[0065] Step 201: Input the first initial modeling parameters of the eye model into the spot position constraint model to obtain the predicted position parameters of the target spot. The spot position constraint model is determined by the first initial modeling parameters of the eye model, the position parameters of the target spot, the position parameters of the target camera, and the position parameters of the target light source. Here, the position parameters of the target camera and the position parameters of the target light source are known parameters, which can be obtained by calibration. For example: in a VR device, the position parameters of the target camera and the position parameters of the target light source can be calibrated in advance and saved in the form of a calibration file of the device.

[0066] In practical applications, as Figure 3 shown, the position parameters of the target camera and the position parameters of the target light source may be coordinates of the world coordinate system 300 (for example, the device coordinate system of the VR device). The coordinates of the two can reflect the relative position relationship between the target light source 301 and the target camera 302. According to this relative position relationship, all the target light sources 301 and the target camera 302 can be unified into a world coordinate system 300. At the same time, the eye model can also be projected into the world coordinate system 300, and the line-of-sight direction corresponding to the eye model is defined as a normalized vector (dx, dy, 1).

[0067] Exemplarily, the exemplary embodiment of the present disclosure can take the first initial modeling parameters of the eye model and the position parameters of the target spot as unknowns, combine the known position parameters of the target camera and the position parameters of the target light source, and establish a spot position constraint model through the law of light reflection and the position constraint conditions of the spot on the cornea. After establishing the first spot position constraint model in this way, the first initial modeling parameters of the eye model can be initialized, and the first initial modeling parameters of the eye model are input into the spot position constraint model, so as to obtain the predicted position parameters of the target spot.

[0068] Step 202: Input the second initial modeling parameters of the eyeball model into the pupil refraction point position constraint model to obtain the predicted position parameters of the pupil refraction point. The pupil refraction point position constraint model is determined by the second initial modeling parameters of the eyeball model, the pupil refraction point position parameters, and the position parameters of the target camera.

[0069] Exemplarily, the second initial modeling parameters of the above eyeball model may include the pupil center p, the corneal center o, the corneal refractive index μ, the pupil depth k, and the corneal depth d. At this time, the second initial modeling parameters of the eyeball model and the position parameters of the target camera can be input into the pupil refraction point position constraint model to solve the pupil refraction point position parameters.

[0070] Step 203: Determine the comprehensive prediction error based on the predicted position parameters of the target light spot, the predicted position parameters of the pupil refraction point, the light spot detection result, and the pupil detection result of the eye image.

[0071] The light spot detection result and the pupil detection result of the eye image in the exemplary embodiment of the present disclosure can be detected with reference to related technologies, such as pixel brightness retrieval, global image gradient, image heat map, etc., which are not limited thereto.

[0072] The comprehensive prediction error in the exemplary embodiment of the present disclosure can be divided into a light spot prediction loss and a pupil prediction loss. The light spot prediction loss can be determined by the predicted position parameters of the target light spot and the light spot detection result of the eye image, while the pupil prediction loss can be determined by the predicted position parameters of the pupil refraction point and the pupil detection result of the eye image. By using the two errors of the light spot prediction loss and the pupil prediction loss as supervision signals to determine the comprehensive prediction error and update the eyeball modeling parameters, the error of the eyeball modeling parameters is reduced, and the deviation of the line-of-sight direction determined by using the eyeball modeling parameters is reduced.

[0073] Step 204: Determine whether the comprehensive prediction error meets the error requirements of the eyeball model. If it meets the error requirements of the eyeball model, it means that the errors of the first initial modeling parameters and the second initial modeling parameters are small, and the error of the eyeball model modeling parameters determined based on the first initial modeling parameters and the second initial modeling parameters is small. Therefore, execute Step 205. If not, it means that the errors of the first initial modeling parameters and the second initial modeling parameters are large, and the error of the eyeball model modeling parameters determined based on the first initial modeling parameters and the second initial modeling parameters is large. When using the eyeball model modeling parameters to determine the line-of-sight direction, the generated deviation is large. Therefore, execute Step 206 to ensure that the error of the obtained eyeball modeling parameters is small and improve the accuracy of the line-of-sight direction solved based on the eyeball modeling parameters.

[0074] Exemplarily, the error requirements of the above-mentioned eyeball model may include the following two cases: First: When the error value of the comprehensive prediction error is less than or equal to the preset threshold of the error requirements of the eyeball model, it is considered that the comprehensive prediction error meets the error requirements of the eyeball model. Second: When the comprehensive prediction error does not meet the error requirements of the eyeball model and is updated, when the number of updates is greater than or equal to the update times threshold of the error requirements of the eyeball model, it is considered that the comprehensive prediction error meets the error requirements of the eyeball model.

[0075] Step 205: Update the first initial modeling parameter and the second initial modeling parameter of the eyeball model based on the gradient results of the spot position constraint model and the pupil refraction point position constraint model. Since the line-of-sight direction is determined by two factors, the pupil and the spot, therefore, using the spot prediction loss and the pupil prediction loss as supervision signals simultaneously, jointly update the spot prediction loss and the pupil prediction loss, so as to reduce the comprehensive error, thereby making the error of the updated eyeball modeling parameter smaller, and ensuring that the deviation of determining the line-of-sight direction using this eyeball modeling parameter is smaller.

[0076] In practical applications, the first modeling parameter and the second modeling parameter of the exemplary embodiment of the present disclosure may include multiple modeling parameters. For the update of each modeling parameter, if both the spot position constraint model and the pupil refraction point position constraint model involve this modeling parameter, it may be to first perform gradient calculation on the spot position constraint model and the pupil refraction point position constraint model for this modeling parameter, and then combine the gradient results of the spot position constraint model and the gradient results of the pupil refraction point position constraint model to obtain the gradient model of this modeling parameter.

[0077] Next, substitute the predicted position parameter of the target spot, the predicted position parameter of the pupil refraction point, the spot detection result and the pupil detection result of the eye image into the gradient model of this modeling parameter to obtain the gradient of the modeling parameter. Then update the modeling parameters of the spot position constraint model and the pupil refraction point position constraint model based on the gradient of this modeling parameter.

[0078] Step 206: Determine the modeling parameters of the eyeball model based on the first initial modeling parameter and the second initial modeling parameter of the eyeball model. When the comprehensive prediction error meets the error requirements of the eyeball model, it indicates that the first initial modeling parameter and the second initial modeling parameter of the eyeball model at this time are close to the actual eyeball parameters of the user. Therefore, the first initial modeling parameter and the second initial modeling parameter of the eyeball model can be determined as the modeling parameters of the eyeball model.

[0079] As a possible implementation, the first initial modeling parameters of the eyeball model include the geometric modeling parameters of the cornea, and the spot position constraint model is determined by the first position constraint model and the second position constraint model. The first position constraint model includes a position constraint model between the target spot and the cornea, and the second position constraint model includes a target spot position constraint model that conforms to the reflection mechanism. The spot position constraint model can be obtained by jointly simplifying the first position constraint model and the second position constraint model, thereby solving and obtaining the predicted position parameters of the target spot. It should be understood that the geometric modeling parameters of the cornea here may include the corneal center o and the corneal radius r.

[0080] In some optional embodiments, the first position constraint model is related to geometric modeling parameters of the cornea, and the second position constraint model is related to geometric modeling parameters of the cornea, position parameters of the target camera, and position parameters of the target light source.

[0081] In practical applications, Figure 4 A schematic diagram of a two-dimensional reflection plane provided according to an exemplary embodiment of the present disclosure is shown. Figure 1 .like Figure 4 As shown in , since the target spot satisfies the light reflection law, the first initial modeling parameters, the position parameters of the target camera and the position parameters of the target light source can be projected from the world coordinate system to the two-dimensional reflection plane. For the convenience of calculation, the corneal center o can be set as the origin of the two-dimensional reflection plane, and the line between the corneal center o and the target camera c is the y-axis of the two-dimensional reflection plane. The x-axis vector of the two-dimensional reflection plane is and the y-axis vector You can write:

[0082]

[0083]

[0084] In the formula, norm represents the normalization operation of the vector, represents the vector of the line between the cornea center o and the target camera c (the target camera c may be, for example, the center of the target camera or the optical center of the target camera), The vector representing the line between the corneal center o and the target light source l. It can be seen that by normalizing the vector, the data is mapped within the range of 0 to 1, so as to improve the processing speed and make data processing more convenient and faster.

[0085] After obtaining the vector of the two-dimensional reflection plane, the target camera c and the target light source l can be projected into the two-dimensional reflection plane to obtain the position parameter coordinates of the target camera c and the target light source l as follows:

[0086]

[0087]

[0088] In the formula, d represents the ordinate of the target camera c, u represents the abscissa of the target light source l, and v represents the ordinate of the target light source l. represents the vector of the line connecting the corneal center o and the target camera c. represents the vector of the line connecting the corneal center o and the target light source l. represents the x-axis vector of the two-dimensional reflection plane. represents the y-axis vector of the two-dimensional reflection plane.

[0089] For the first position constraint model, it includes the position constraint model between the target light spot and the cornea. Considering that the position of the target light spot on the cornea is on a circle centered at the corneal center, the coordinates of the target light spot, that is, the reflection point g, in the two-dimensional reflection plane can be set as g(x, y). Then the first position constraint model can be expressed as:

[0090] x 2 +y 2 =r 2 . Equation (5)

[0091] For the second position constraint model, it includes the position constraint model of the target light spot that conforms to the law of light reflection. From the law of light reflection, the incident angle of light is equal to the exit angle. Based on this principle, the position constraint model of the target light spot that conforms to the law of light reflection can be solved by means of auxiliary lines.

[0092] Figure 5 shows a schematic diagram of a two-dimensional reflection plane according to an exemplary embodiment of the present disclosure. Figure 2 . As Figure 5 shown, by extending the incident light lg, at the same time, a line parallel to the connection line og between the corneal center o and the predicted position g of the target light spot (i.e., the target light spot g) is drawn through the target camera c, and the intersection point of the two lines is point e. Since ∠lga = ∠cga and the line og is parallel to the line ce, therefore, ∠lga = ∠ceg, ∠cga = ∠ecg, it can be known that triangle gec is an isosceles triangle.

[0093] Figure 6 shows a schematic flowchart of a method for determining a second position constraint model according to an exemplary embodiment of the present disclosure. As Figure 5 and Figure 6 shown, the method for determining the second position constraint model of the exemplary embodiment of the present disclosure may include:

[0094] Step 601: Taking the first initial modeling parameters of the eyeball model and the position parameters of the target light spot as unknowns, based on the position information of the target light source and the position parameters of the target light spot, determine the expression of the first incident ray with the target light spot as the reflection point. The vector expression of the first incident ray gl is

[0095] Step 602: Taking the first initial modeling parameters of the eyeball model and the position parameters of the target light spot as unknowns, based on the position parameters of the target camera and the position parameters of the target light source, determine the expression of the extension line of the first incident ray with the target light spot as the reflection point. The vector expression of the extension line eg of the first incident ray is:

[0096]

[0097] In the formula, represents the vector expression of the extension line eg of the first incident ray, represents the vector of the line connecting the corneal center o and the target light spot g, represents the vector of the line connecting the target camera c and the target light spot g, and r represents the corneal radius.

[0098] Step 603: Based on the first incident ray expression and the expression of the extension line of the first incident ray, determine the second position constraint model.

[0099] From Figure 5 it can be seen that the straight line eg and the straight line gl are collinear, so Considering that the vector expression of the extension line eg of the first incident ray is known in formula (6), therefore, the expression of the second position constraint model can be:

[0100]

[0101] In the formula, represents the vector of the line connecting the target camera c and the target light spot g, represents the vector of the line connecting the corneal center o and the target light spot g, represents the vector of the line connecting the target light spot g and the target light source l, and r represents the corneal radius.

[0102] Exemplarily, as Figure 5 shown, after determining the first incident ray expression , by extending the line gl, obtain the extension line of the line gl. At the same time, draw a parallel line to the line go through the point c. The point e is the intersection point between the two auxiliary lines. According to the law of reflection, it can be deduced that the triangle △gec is an isosceles triangle. Since in the projection length in the unit vector direction is In the formula, r is The modulus length, and triangle △gec is an isosceles triangle, from which it can be deduced that The length of In Twice the projection length in the direction of the unit vector, and The direction of Is the same as the direction of the unit vector of Based on this, the expression of the extension line of the incident light can be obtained as:

[0103]

[0104] As can be seen from the above, the first position constraint model of the exemplary embodiment of the present disclosure can be represented by Equation (5), and the second position constraint model can be represented by Equation (7). Considering that the position parameters of the target camera c and the target light source l are known, therefore, the system of equations composed of the first position constraint model and the second position constraint model contains only two unknowns, that is, the coordinates g(x, y) of the predicted position g of the target light spot on the two-dimensional reflection plane. At this time, Equation (3), Equation (4), the predicted position g of the target light spot = (x, y), and the origin o = (0, 0) can be substituted into the above system of equations. After simplification, the equation satisfied by the predicted position of the target light spot is obtained, that is, the expression of the light spot position constraint model is:

[0105] u 2 (r 2 (d + y) - 2dy 2 ) 2 -(r 2 -y 2 )(r 2 (d + v) - 2dvy) 2 = 0, Equation (8)

[0106] In the formula, u represents the abscissa of the target light source l on the two-dimensional reflection plane, v represents the ordinate of the target light source l on the two-dimensional reflection plane, r represents the corneal radius, d represents the ordinate of the target camera c on the two-dimensional reflection plane, and y represents the ordinate of the target light spot g on the two-dimensional reflection plane. By solving this light spot position constraint model, four solutions are obtained, two of which are solutions that conform to the law of reflection, and two are solutions that do not conform to the law of reflection. By judging the law of reflection and the angle between the incident light and the normal, the wrong solutions are filtered to obtain the predicted position of the target light spot g, that is, the coordinates of the target light spot g on the two-dimensional reflection plane.

[0107] In some alternative ways, Figure 7 Shows a schematic flowchart of a method for determining the expression of the extension line of the incident light according to the exemplary embodiment of the present disclosure. As Figure 7As shown, the above-mentioned first initial modeling parameters of the eyeball model and the position parameters of the target light spot are unknown quantities, and the expression of the extended line of the incident light with the target light spot as the reflection point is determined based on the position parameters of the target camera and the position parameters of the target light source, which may include:

[0108] Step 701: Determine the expression of the reflected light with the target light spot as the reflection point based on the position parameters of the target camera and the position parameters of the target light spot. The vector expression of the reflected light cg is:

[0109] Step 702: Based on the position parameters of the target light spot and the first initial modeling parameters of the eyeball model, determine the first normal expression with the target light spot as the reflection point. The expression of the vector of the first normal ec is:

[0110]

[0111] Step 703: Based on the first normal line expression and the reflected light expression, determine the extended line expression of the first incident light. The extended line expression of the first incident light is the above equation (6).

[0112] In some optional embodiments, the second initial modeling parameters of the eyeball model may include: geometric modeling parameters of the cornea, corneal refractive index parameters and pupil position parameters. The geometric modeling parameters of the cornea may include the corneal center o and the corneal depth d, the pupil position parameters may include the pupil depth k and the pupil center p, and the corneal refractive index parameters may include the corneal refractive index μ.

[0113] The pupil refraction point position constraint model can be determined by a third position constraint model and a fourth position constraint model. The third position constraint model includes a position constraint model between the pupil refraction point and the cornea, and the fourth position constraint model includes a pupil refraction point position constraint model that conforms to the light refraction law. The pupil refraction point position constraint model can be obtained by jointly simplifying the third position constraint model and the fourth position constraint model, thereby solving and obtaining the pupil refraction point prediction position parameters.

[0114] In some optional embodiments, the third position constraint model is related to the geometric modeling parameters of the cornea, and the fourth position constraint model is related to the geometric modeling parameters of the cornea and the corneal refractive index parameters.

[0115] In practical applications, Figure 8 FIG. 2 shows a schematic diagram of a two-dimensional refraction plane provided according to an exemplary embodiment of the present disclosure. Figure 8As shown, the incident light ray is the straight line ch, the refracted light ray is the straight line hp, and the normal line oh is made through the corneal center o and the pupil refraction point h. Since the pupil refraction point h satisfies the refraction law, the second initial modeling parameter and the position parameter of the target camera can be projected from the world coordinate system to the two-dimensional refraction plane. For the convenience of calculation, the corneal center o can be set as the origin of this two-dimensional refraction plane, and the pupil center p is projected onto the x-axis of this two-dimensional refraction plane. At this time, the x-axis vector of this two-dimensional refraction plane and the y-axis vector can be written as:

[0116]

[0117]

[0118] In the formula, norm represents the normalization operation of the vector, represents the vector of the line connection between the corneal center o and the pupil center p, represents the vector of the line connection between the corneal center o and the target camera c, represents the vector of the line connection between the corneal center o and the target light source l. It can be seen that by normalizing the vector, the data is mapped within the range of 0 to 1, so as to improve the processing speed and make the data processing more convenient and fast.

[0119] After obtaining the vectors of the two-dimensional refraction plane, the target camera c and the pupil center p can be projected into this two-dimensional refraction plane, and the position parameter coordinates of the target camera c and the pupil center p are obtained as:

[0120]

[0121]

[0122] In the formula, u1 represents the abscissa of the target camera c, v1 represents the ordinate of the target camera c, k represents the ordinate of the pupil center p, represents the vector of the line connection between the corneal center o and the target camera c, represents the vector of the line connection between the corneal center o and the pupil center p, represents the x-axis vector of the two-dimensional reflection plane, represents the y-axis vector of the two-dimensional reflection plane.

[0123] For the third position constraint model, it includes the position constraint model between the pupil refraction point and the cornea. Considering that the position of the pupil refraction point on the cornea is on a circle with the corneal center as the center, the coordinates of the pupil refraction point h in the two-dimensional refraction plane can be set as h(x, y), then the third position constraint model can be expressed as:

[0124] x2 +y 2 = r 2 。 Equation (14)

[0125] For the fourth position constraint model, it includes a pupil refraction point position constraint model that conforms to the law of light refraction. From the law of light refraction, the angle ∠θ1 between the incident light and the normal line and the angle ∠θ2 between the refracted light and the normal line satisfy n1sinθ1 = n2sinθ2. Based on this principle, the target light spot position constraint model that conforms to the law of light reflection can be solved through the incident light, the normal line, and the corneal refractive index parameter.

[0126] In some alternative ways, Figure 9 shows a schematic flowchart of a method for determining the fourth position constraint model provided according to an exemplary embodiment of the present disclosure. As Figure 9 shown, the method for determining the fourth position constraint model of the exemplary embodiment of the present disclosure may include:

[0127] Step 901: Taking the geometric modeling parameters of the cornea and the pupil refraction point position parameters as unknowns, based on the pupil refraction point position parameters, the geometric modeling parameters of the cornea, and the position parameters of the target camera, determine the direction expression of the refracted light with the pupil refraction point as the incident point, and the expression of the direction vector of the refracted light hp

[0128] Step 902: Taking the pupil refraction point position parameters and the pupil position parameters as unknowns, based on the pupil refraction point position parameters and the pupil position parameters, determine the refracted light expression, and the expression of the vector of the refracted light hp

[0129] Step 903: Based on the direction expression of the refracted light and the refracted light expression, determine the fourth position constraint model.

[0130] It can be seen from Figure 8 that the expression of the direction vector of the refracted light hp and the expression of the vector of the refracted light line hp are collinear, so their cross product is 0. Then the expression of the fourth position constraint model can be:

[0131]

[0132] As can be seen from the above, the third position constraint model of the exemplary embodiment of the present disclosure can be represented by Equation (14), and the fourth position constraint model can be represented by Equation (15). Considering that the calibration of the target camera c and the position parameters of the pupil center p are known, therefore, the system of equations composed of the third position constraint model and the fourth position constraint model contains only two unknowns, that is, the coordinates h(x, y) of the pupil refraction point h in the two-dimensional refraction plane. At this time, Equation (12), Equation (13), the predicted position h=(x, y) of the pupil refraction point, and the origin o=(0, 0) can be substituted into the above system of equations. After simplification, the equation satisfied by the predicted position of the pupil refraction point is obtained, that is, the expression of the pupil refraction point position constraint model is:

[0133]

[0134] By solving the pupil refraction point position constraint model and verifying the refraction law, the wrong solutions can be filtered out to obtain the predicted position of the pupil refraction point h, that is, the coordinates of the pupil refraction point h on the two-dimensional refraction plane.

[0135] In some alternative ways, Figure 10 shows a schematic flowchart of a method for determining the direction expression of a refracted light ray according to the exemplary embodiment of the present disclosure. As Figure 10 shown, taking the geometric modeling parameters of the cornea and the position parameters of the pupil refraction point as unknowns, based on the position parameters of the pupil refraction point, the geometric modeling parameters of the cornea, and the position parameters of the target camera, determining the direction expression of the refracted light ray with the pupil refraction point as the incident point may include:

[0136] Step 1001: Based on the position parameters of the target camera and the position parameters of the pupil refraction point, determine the second incident light ray expression with the pupil refraction point as the incident point. The vector expression of the second incident light ray ch is

[0137] Step 1002: Based on the position parameters of the pupil refraction point and the geometric modeling parameters of the cornea, determine the second normal expression with the pupil refraction point as the incident point. The expression of the vector of the second normal oh is

[0138] Step 1003: Based on the second incident light ray expression and the second normal expression, determine the direction expression of the refracted light ray. The vector expression of the direction of the refracted light ray is:

[0139]

[0140] After solving the predicted position parameters of the target light spot and the predicted position parameters of the pupil refraction point through the light spot position constraint model and the pupil refraction point position constraint model respectively, the predicted position parameters of the target light spot and the predicted position parameters of the pupil refraction point can be projected onto, for example,Figure 3 In the shown world coordinate system, then, by using the internal and external parameters of the target camera, project the predicted position parameters of the target light spot and the predicted position parameters of the pupil refraction point in the world coordinate system onto the two-dimensional detection plane, so as to compare with the light spot detection result and pupil detection result of the eye image on the two-dimensional detection plane to determine the comprehensive prediction error.

[0141] As a possible implementation manner, Figure 11 shows a schematic flowchart of a method for determining a comprehensive prediction error according to an exemplary embodiment of the present disclosure. As Figure 11 shown, the above determining the comprehensive prediction error based on the predicted position parameters of the target light spot, the predicted position parameters of the pupil refraction point, the light spot detection result of the eye image, and the pupil detection position may include:

[0142] Step 1101: Determine the light spot prediction loss based on the predicted position parameters of the target light spot and the target light spot detection result of the eye image. So that when the comprehensive prediction error does not meet the error requirement of the eyeball model, the first initial modeling parameters of the eyeball model can be updated by performing backpropagation based on the light spot prediction loss. It should be understood that the eye image in the exemplary embodiment of the present disclosure may be a human eye image or an animal image, and the eye image sequence may only contain an eye, a facial image including the eye, or a whole body image including the eye, etc. The eye image may be a video file including the eye of the target object collected by an image acquisition device, or a photo including the eye of the target object collected by an image acquisition device. It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner according to relevant laws and regulations.

[0143] For example, in response to receiving an active request from the user, send a prompt message to the user to clearly prompt the user that the operation requested to be executed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that performs the operations of the technical solutions of the present disclosure according to the prompt message.

[0144] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving an active request from the user may be, for example, in the form of a pop-up window, and the prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0145] It can be understood that the above-mentioned notification and the process of obtaining user authorization are only illustrative and do not limit the implementation manner of the present disclosure. Other methods that comply with relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0146] Exemplarily, the target spot detection result of the above-mentioned eye image can be obtained based on the spot and pupil detection module. The spot and pupil detection module can be a spot and pupil detection module based on an image detection algorithm or a spot and pupil detection module based on a deep learning model.

[0147] When the target spot detection result of the above-mentioned eye image is determined by a spot and pupil detection module based on an image detection algorithm, the eye image can be grayscaled first, and then spot detection can be performed on the eye image. The method of spot detection can be any method such as pixel brightness retrieval, gradient of the whole image, image heat map, etc., and the present disclosure does not limit this.

[0148] When the target spot detection result of the above-mentioned eye image is determined by a spot and pupil detection module based on a deep learning model, multiple eye images can be used as samples to train an image detection model. When the model converges, the image detection model can be used to implement the target spot detection of the eye image.

[0149] Step 1102: Determine the pupil prediction loss based on the predicted position parameter of the pupil refraction point and the pupil detection result of the eye image. In order to facilitate the case where the comprehensive prediction error does not meet the error requirement of the eye model, the second initial modeling parameter of the eye model can be updated by performing backpropagation based on the pupil prediction loss. The method for determining the pupil detection result of the eye image can refer to the relevant description of the target spot detection result of the eye image above, and will not be elaborated here.

[0150] Step 1103: Determine the comprehensive prediction error based on the spot prediction loss and the pupil prediction loss. By determining the comprehensive prediction error based on the two supervision signals of the spot prediction loss and the pupil prediction loss, the prediction accuracy of the comprehensive prediction error can be improved, so that the finally obtained eye modeling parameters can better simulate the actual eye parameters of the user, thereby making the line-of-sight direction calculated according to the eye modeling parameters more accurate and reducing the calculation deviation of the line-of-sight direction.

[0151] In some optional ways, the above-mentioned spot prediction loss can be determined according to the correspondence between the target spot detection result of the eye image and the target light source. When the target spot detection result of the eye image corresponds to the target light source, the spot prediction loss can be determined based on the predicted position parameter of the target spot corresponding to the same target light source and the target spot detection result of the eye image. At this time, the expression of the spot prediction loss can be:

[0152]

[0153] In the formula, g i represents the coordinates of the spot corresponding to the i-th light source calculated by the eyeball modeling parameter determination method of the exemplary embodiment of the present disclosure, and g di represents the coordinates of the spot reflected by the i-th light source detected by the target spot detection of the eye image. By adding the coordinates of the spots corresponding to multiple light sources and the coordinates of the spots reflected by multiple light sources, the spot prediction loss can be obtained.

[0154] In some alternative ways, when there is no corresponding relationship between the target spot detection result of the eye image and the target light source, and the spot detection result is multiple, the spot prediction sub-loss can be determined based on the minimum distance between the predicted position parameter of each target spot and the target spot detection results of multiple eye images. Then, based on the spot prediction sub-loss corresponding to the predicted position parameter of each target spot, the spot prediction loss is determined. At this time, the expression of the spot prediction loss can be:

[0155]

[0156] In the formula, g j represents the coordinates of the spot closest to the detected j-th spot calculated by the eyeball modeling parameter determination method of the exemplary embodiment of the present disclosure, and g dj represents the coordinates of the j-th spot detected by the target spot detection of the eye image.

[0157] Exemplarily, when the spot detection result is multiple, it is necessary to solve the error between each spot detection result and the predicted position parameter of the target spot closest to it to obtain multiple spot prediction sub-losses, and then by adding the multiple spot prediction sub-losses, the spot prediction loss can be obtained.

[0158] In some alternative ways, the above comprehensive prediction error is determined by the weighted sum of the spot prediction loss and the pupil prediction loss, where the weight of the spot prediction loss is greater than the weight of the pupil prediction loss, so that the obtained comprehensive prediction error can be more accurate.

[0159] The exemplary embodiment of the present disclosure also provides a method for determining the line of sight direction, which is used to obtain the user's line of sight direction, so as to perform subsequent operations such as interaction and rendering based on the obtained user's line of sight direction. Figure 12 shows a schematic flowchart of the method for determining the line of sight direction provided by the exemplary embodiment of the present disclosure. As Figure 12 shown, the method for determining the line of sight direction may include:

[0160] Step 1201: Detect the eye image to be detected to obtain the target spot detection result and the pupil refraction point detection result.

[0161] Step 1202: Obtain a target line-of-sight direction based on the target light spot detection result, the pupil refraction point detection result, and the eye model modeling parameters, where the eye model modeling parameters are determined by the method for determining the modeling parameters of the eye model in the exemplary embodiments of the present disclosure. Since the eye modeling parameters obtained in the method for determining the eye modeling parameters in the exemplary embodiments of the present disclosure can better simulate the geometric parameters of the user's eye, the deviation of the user's line-of-sight direction calculated through the target light spot detection result, the pupil refraction point detection result, and the eye model modeling parameters is small.

[0162] Exemplarily, the above-mentioned eye model modeling parameters may include geometric modeling parameters of the cornea, corneal refractive index parameters, and pupil position parameters. By detecting the eye image to be detected, the detection results (2D) of the light spot and the pupil refraction point on the image plane can be obtained, and then the line-of-sight information of the user can be calculated by using the detection result of the light spot on the image plane, the detection result of the pupil refraction point on the image plane, the known eye model modeling parameters, the known camera parameters, and the known light source parameters.

[0163] Exemplarily, the three-dimensional position of the cornea can be solved by using the eye model modeling parameters obtained in the method for determining the eye modeling parameters in the exemplary embodiments of the present disclosure, the law of reflection, and the relative position between the light source and the target camera. Then, the line-of-sight direction of the user can be solved by using the law of refraction and the imaging of the pupil.

[0164] The above mainly introduces the solution provided by the embodiments of the present disclosure from the perspective of the server. It can be understood that in order to implement the above functions, the server includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0165] The embodiments of the present disclosure can divide the server into functional units according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present disclosure is illustrative, and is only a logical function division. There may be other division methods in actual implementation.

[0166] In the case where each functional module is divided according to each function, an exemplary embodiment of the present disclosure provides an apparatus for determining modeling parameters of an eyeball model, and the apparatus for determining modeling parameters of the eyeball model may be a server or a chip applied to the server. Figure 13 FIG. shows a schematic block diagram of functional modules of an apparatus for determining modeling parameters of an eyeball model according to an exemplary embodiment of the present disclosure. As Figure 13 shown, the apparatus 1300 for determining modeling parameters of the eyeball model includes:

[0167] An obtaining module 1301, configured to input first initial modeling parameters of the eyeball model into a spot position constraint model to obtain predicted position parameters of a target spot, where the spot position constraint model is determined by the first initial modeling parameters of the eyeball model, the position parameters of the target spot, the position parameters of the target camera, and the position parameters of the target light source.

[0168] The obtaining module 1301 is further configured to input second initial modeling parameters of the eyeball model into a pupil refraction point position constraint model to obtain predicted position parameters of the pupil refraction point, where the pupil refraction point position constraint model is determined by the second initial modeling parameters of the eyeball model, the position parameters of the pupil refraction point, and the position parameters of the target camera;

[0169] A determining module 1302, configured to determine a comprehensive prediction error based on the predicted position parameters of the target spot, the predicted position parameters of the pupil refraction point, the spot detection result of the eye image, and the pupil detection result;

[0170] An updating module 1303, configured to, if the comprehensive prediction error does not meet the error requirement of the eyeball model, update the first initial modeling parameters and the second initial modeling parameters of the eyeball model based on the gradient results of the spot position constraint model and the pupil refraction point position constraint model, otherwise, determine the modeling parameters of the eyeball model based on the first initial modeling parameters and the second initial modeling parameters of the eyeball model.

[0171] As a possible implementation manner, the first initial modeling parameters of the above eyeball model include geometric modeling parameters of the cornea, and the spot position constraint model is determined by a first position constraint model and a second position constraint model. The first position constraint model includes a position constraint model between the target spot and the cornea, and the second position constraint model includes a target spot position constraint model that conforms to the law of light reflection.

[0172] In some optional manners, the first initial modeling parameters of the above eyeball model include geometric modeling parameters of the cornea. The first position constraint model is related to the geometric modeling parameters of the cornea, and the second position constraint model is related to the geometric modeling parameters of the cornea, the position parameters of the target camera, and the position parameters of the target light source.

[0173] In some alternative ways, with the geometric modeling parameters of the cornea and the position parameters of the target light spot as unknowns, the determination module 1302 is further configured to determine the expression of the first incident ray with the target light spot as the reflection point based on the position information of the target light source and the position parameters of the target light spot.

[0174] With the first initial modeling parameters of the eyeball model and the position parameters of the target light spot as unknowns, the determination module 1302 is further configured to determine the expression of the extension line of the first incident ray with the target light spot as the reflection point based on the position parameters of the target camera and the position parameters of the target light source;

[0175] The determination module 1302 determines the second position constraint model based on the first incident ray expression and the first incident ray extension line expression.

[0176] In some alternative ways, with the position parameters of the target light spot as unknowns, the determination module 1302 is further configured to determine the expression of the reflected ray with the target light spot as the reflection point based on the position parameters of the target camera and the position parameters of the target light spot; determine the expression of the first normal line with the target light spot as the reflection point based on the position parameters of the target light spot and the first initial modeling parameters of the eyeball model; determine the expression of the extension line of the first incident ray based on the first normal line expression and the reflected ray expression.

[0177] As a possible implementation, the second initial modeling parameters of the above-mentioned eyeball model include: geometric modeling parameters of the cornea, corneal refractive index parameters, and pupil position parameters. The pupil refraction point position constraint model is determined by the third position constraint model and the fourth position constraint model. The third position constraint model includes the position constraint model between the pupil refraction point and the cornea, and the fourth position constraint model includes the pupil refraction point position constraint model that conforms to the law of light refraction.

[0178] In some alternative ways, the above-mentioned third position constraint model is related to the geometric modeling parameters of the cornea, and the fourth position constraint model is related to the geometric modeling parameters of the cornea and the corneal refractive index parameters.

[0179] In some alternative ways, with the geometric modeling parameters of the cornea and the pupil refraction point position parameters as unknowns, the determination module 1302 is further configured to determine the direction expression of the refracted ray with the pupil refraction point as the incident point based on the pupil refraction point position parameters, the geometric modeling parameters of the cornea, and the position parameters of the target camera;

[0180] With the pupil refraction point position parameters and the pupil position parameters as unknowns, the determination module 1302 is further configured to determine the refracted ray expression based on the pupil refraction point position parameters and the pupil position parameters;

[0181] The determination module 1302 is further configured to determine the fourth position constraint model based on the direction expression of the refracted ray and the refracted ray expression.

[0182] In some alternative embodiments, the determining module 1302 is further configured to determine an expression of a second incident ray with the pupil refraction point as the incident point based on the position parameters of the target camera and the position parameters of the pupil refraction point.

[0183] The determining module 1302 is further configured to determine an expression of a second normal line with the pupil refraction point as the incident point based on the position parameters of the pupil refraction point and the geometric modeling parameters of the cornea.

[0184] The determining module 1302 is further configured to determine an expression of the direction of the refracted ray based on the expression of the second incident ray and the expression of the second normal line.

[0185] In some alternative embodiments, the above-mentioned determining module 1302 is further configured to determine a spot prediction loss based on the predicted position parameters of the target spot and the detection result of the target spot in the eye image.

[0186] The determining module 1302 is further configured to determine a pupil prediction loss based on the predicted position parameters of the pupil refraction point and the detection result of the pupil in the eye image.

[0187] The determining module 1302 is further configured to determine a comprehensive prediction error based on the spot prediction loss and the pupil prediction loss.

[0188] In some alternative embodiments, the above-mentioned determining module 1302 is further configured to determine a spot prediction loss based on the predicted position parameters of the target spot corresponding to the same target light source and the detection result of the target spot in the eye image.

[0189] In some alternative embodiments, when there are multiple spot detection results, the above-mentioned determining module 1302 is further configured to determine a spot prediction sub-loss based on the minimum distance between the predicted position parameters of each target spot and the detection results of the target spots in multiple eye images.

[0190] The above-mentioned determining module 1302 is further configured to determine a spot prediction loss based on the spot prediction sub-losses corresponding to the predicted position parameters of each target spot.

[0191] In some alternative embodiments, the above-mentioned comprehensive prediction error is determined by a weighted sum of the spot prediction loss and the pupil prediction loss, and the weight of the spot prediction loss is greater than the weight of the pupil prediction loss.

[0192] In the case of dividing each functional module according to the corresponding functions, the exemplary embodiment of the present disclosure further provides a line-of-sight direction determination device, and the line-of-sight direction determination device may be a server or a chip applied to the server. Figure 14 The schematic block diagram of the functional modules of the line-of-sight direction determination device according to the exemplary embodiment of the present disclosure is shown. As Figure 14 shown, the line-of-sight direction determination device 1400 includes:

[0193] The detection module 1401 is configured to detect the to-be-detected eye image based on the eye model modeling parameters, and obtain a target spot detection result and a pupil refraction point detection result;

[0194] The obtaining module 1402 is configured to obtain a target line-of-sight direction based on the target spot detection result and the pupil refraction point detection result, wherein the eye model modeling parameters are determined by the modeling parameter determination method of the eye model according to any one of claims 1 to 13.

[0195] In some alternative embodiments, the above line-of-sight direction determination device 1400 further includes a determination module 1403, and the determination module 1403 is configured to determine the target spot detection result based on the geometric modeling parameters of the cornea, the position parameters of the target camera, and the position parameters of the target light source.

[0196] The determination module 1403 is configured to determine the pupil refraction point detection result based on the geometric modeling parameters of the cornea, the corneal refractive index parameters, the pupil position parameters, and the position parameters of the target camera.

[0197] Figure 15 Fig. shows a schematic block diagram of a chip according to an exemplary embodiment of the present disclosure. As Figure 15 shown, the chip 1500 includes one or more than two (including two) processors 1501 and a communication interface 1502. The communication interface 1502 can support the server to execute the data sending and receiving steps in the above image processing method, and the processor 1501 can support the server to execute the data processing steps in the above image processing method.

[0198] Optionally, as Figure 15 shown, the chip 1500 further includes a memory 1503, and the memory 1503 may include a read-only memory and a random access memory, and provide operation instructions and data to the processor. A part of the memory may further include a non-volatile random access memory (NVRAM).

[0199] In some embodiments, as Figure 15As shown, the processor 1501 executes corresponding operations by invoking the operation instructions stored in the memory (the operation instructions can be stored in the operating system). The processor 1501 controls the processing operations of any one of the terminal devices, and the processor can also be referred to as a central processing unit (CPU). The memory 1503 can include a read-only memory and a random access memory, and provides instructions and data to the processor 1501. A part of the memory 1503 can also include NVRAM. For example, in the application, the memory, the communication interface, and the memory are coupled together through a bus system, where the bus system can include a power bus, a control bus, a status signal bus, etc. in addition to the data bus. However, for the sake of clarity, in Figure 15 all kinds of buses are labeled as the bus system 1504.

[0200] The method disclosed in the above embodiments of the present disclosure can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor or the instructions in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present disclosure can be directly embodied as being executed by the hardware decoding processor, or executed by the combination of the hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0201] The exemplary embodiments of the present disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program that can be executed by the at least one processor, and the computer program, when executed by the at least one processor, is used to cause the electronic device to execute the method according to the embodiments of the present disclosure.

[0202] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is configured to cause the computer to execute the method according to the embodiments of the present disclosure.

[0203] The exemplary embodiments of the present disclosure also provide a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is configured to cause the computer to execute the method according to the embodiments of the present disclosure.

[0204] Referring Figure 16 , a block diagram of an electronic device 1600 that can be used as a server or a client of the present disclosure will now be described. It is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0205] As Figure 16 shown, the electronic device 1600 includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1602 or a computer program loaded from a storage unit 1608 into a random access memory (RAM) 1603. In the RAM 1603, various programs and data required for the operation of the electronic device 1600 can also be stored. The computing unit 1601, the ROM 1602, and the RAM 1603 are connected to each other through a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.

[0206] Multiple components in the electronic device 1600 are connected to the I / O interface 1605, including: an input unit 1606, an output unit 1607, a storage unit 1608, and a communication unit 1609. The input unit 1606 can be any type of device capable of inputting information into the electronic device 1600. The input unit 1606 can receive input digital or character information and generate key signal inputs related to the user settings and / or function controls of the electronic device. The output unit 1607 can be any type of device capable of presenting information and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1608 can include, but is not limited to, magnetic disks and optical discs. The communication unit 1609 allows the electronic device 1600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0207] The computing unit 1601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1601 executes the various methods and processes described above. For example, in some embodiments, the methods of the exemplary embodiments of the present disclosure can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 1600 via the ROM 1602 and / or the communication unit 1609. In some embodiments, the computing unit 1601 can be configured to execute the methods of the exemplary embodiments of the present disclosure in any other suitable manner (e.g., by means of firmware).

[0208] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, executed partially on the machine as an independent software package and partially on a remote machine, or executed entirely on a remote machine or server.

[0209] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0210] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0211] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0212] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of the communication network include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0213] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client - server relationship is created by computer programs running on the respective computers and having a client - server relationship with each other.

[0214] In the above - mentioned embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present disclosure are executed in whole or in part. The computer can be a general - purpose computer, a special - purpose computer, a computer network, a terminal, a user device, or other programmable devices. The computer program or instructions can be stored in a computer - readable storage medium, or transmitted from one computer - readable storage medium to another computer - readable storage medium. For example, the computer program or instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The computer - readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrating one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid - state drive (SSD).

[0215] Although the present disclosure has been described in connection with specific features and their embodiments, it will be apparent that various modifications and combinations can be made without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the present disclosure. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalent technologies, the present disclosure is also intended to include these changes and modifications.

Claims

1. A method for determining eye modeling parameters, characterized in that, The method includes: Inputting the first initial modeling parameters of the eyeball model into the spot position constraint model to obtain the predicted position parameters of the target spot, where the spot position constraint model is determined by the first initial modeling parameters of the eyeball model, the position parameters of the target spot, the position parameters of the target camera, and the position parameters of the target light source; Inputting the second initial modeling parameters of the eyeball model into the pupil refraction point position constraint model to obtain the predicted position parameters of the pupil refraction point, where the pupil refraction point position constraint model is determined by the second initial modeling parameters of the eyeball model, the pupil refraction point position parameters, and the position parameters of the target camera; Determining the comprehensive prediction error based on the predicted position parameters of the target spot, the predicted position parameters of the pupil refraction point, the spot detection result of the eye image, and the pupil detection result; If the comprehensive prediction error does not meet the error requirements of the eyeball model, updating the first initial modeling parameters and the second initial modeling parameters of the eyeball model based on the gradient results of the spot position constraint model and the pupil refraction point position constraint model; otherwise, determining the eyeball modeling parameters based on the first initial modeling parameters and the second initial modeling parameters of the eyeball model.

2. The method according to claim 1, characterized in that, The first initial modeling parameters of the eyeball model include the geometric modeling parameters of the cornea. The spot position constraint model is determined by the first position constraint model and the second position constraint model. The first position constraint model includes the position constraint model between the target spot and the cornea, and the second position constraint model includes the target spot position constraint model that conforms to the law of light reflection.

3. The method according to claim 2, characterized in that, The first position constraint model is related to the geometric modeling parameters of the cornea, and the second position constraint model is related to the geometric modeling parameters of the cornea, the position parameters of the target camera, and the position parameters of the target light source.

4. The method according to claim 2, characterized in that, The determination method of the second position constraint model includes: Taking the geometric modeling parameters of the cornea and the position parameters of the target spot as unknowns, and determining the first incident light expression with the target spot as the reflection point based on the position parameters of the target light source and the position parameters of the target spot; Taking the first initial modeling parameters of the eyeball model and the position parameters of the target spot as unknowns, and determining the extension line expression of the first incident light with the target spot as the reflection point based on the position parameters of the target camera and the position parameters of the target light source; Determining the second position constraint model based on the first incident light expression and the first incident light extension line expression.

5. The method according to claim 4, characterized in that, The step of taking the first initial modeling parameters of the eyeball model and the position parameters of the target spot as unknowns, and determining the extension line expression of the incident light with the target spot as the reflection point based on the position parameters of the target camera and the position parameters of the target light source includes: Determining the reflection light expression with the target spot as the reflection point based on the position parameters of the target camera and the position parameters of the target spot; Determining the first normal expression with the target spot as the reflection point based on the position parameters of the target spot and the first initial modeling parameters of the eyeball model; Based on the first normal expression and the reflected light expression, determine the extended line expression of the first incident light ray.

6. The method according to claim 1, characterized in that, The second initial modeling parameters of the eyeball model include: geometric modeling parameters of the cornea, corneal refractive index parameters, and pupil position parameters; The pupil refraction point position constraint model is determined by a third position constraint model and a fourth position constraint model. The third position constraint model includes the position constraint model between the pupil refraction point and the cornea, and the fourth position constraint model includes the pupil refraction point position constraint model that conforms to the law of light refraction.

7. The method according to claim 6, characterized in that, The third position constraint model is related to the geometric modeling parameters of the cornea, and the fourth position constraint model is related to the geometric modeling parameters of the cornea and the corneal refractive index parameters.

8. The method according to claim 6, characterized in that, The determination method of the fourth position constraint model includes: Taking the geometric modeling parameters of the cornea and the pupil refraction point position parameters as unknowns, based on the pupil refraction point position parameters, the geometric modeling parameters of the cornea, and the position parameters of the target camera, determine the direction expression of the refracted light ray with the pupil refraction point as the incident point; Taking the pupil refraction point position parameters and the pupil position parameters as unknowns, based on the pupil refraction point position parameters and the pupil position parameters, determine the refracted light ray expression; Based on the direction expression of the refracted light ray and the refracted light ray expression, determine the fourth position constraint model.

9. The method according to claim 8, characterized in that, The step of taking the geometric modeling parameters of the cornea and the pupil refraction point position parameters as unknowns, and based on the pupil refraction point position parameters, the geometric modeling parameters of the cornea, and the position parameters of the target camera, to determine the direction expression of the refracted light ray with the pupil refraction point as the incident point includes: Based on the position parameters of the target camera and the pupil refraction point position parameters, determine the second incident light ray expression with the pupil refraction point as the incident point; Based on the pupil refraction point position parameters and the geometric modeling parameters of the cornea, determine the second normal expression with the pupil refraction point as the incident point; Based on the second incident light ray expression and the second normal expression, determine the direction expression of the refracted light ray.

10. The method according to any one of claims 1 to 9, characterized in that, The step of determining the comprehensive prediction error based on the predicted position parameters of the target light spot, the predicted position parameters of the pupil refraction point, the light spot detection result of the eye image, and the pupil detection position includes: Based on the predicted position parameters of the target light spot and the target light spot detection result of the eye image, determine the light spot prediction loss; Based on the predicted position parameters of the pupil refraction point and the pupil detection result of the eye image, determine the pupil prediction loss; Based on the light spot prediction loss and the pupil prediction loss, determine the comprehensive prediction error.

11. The method according to claim 10, characterized in that The method further includes: Based on the predicted position parameters of the target light spot corresponding to the same target light source and the target light spot detection result of the eye image, determine the light spot prediction loss.

12. The method according to claim 10, characterized in that The method further includes: When there are multiple light spot detection results, based on the minimum distance between the predicted position parameters of each target light spot and the target light spot detection results of multiple eye images, determine the light spot prediction sub-loss; Determine the spot prediction loss based on the spot prediction sub-loss corresponding to the predicted position parameters of each of the target spots.

13. The method according to claim 10, characterized in that The comprehensive prediction error is determined by weighted summation of the spot prediction loss and the pupil prediction loss, and the weight of the spot prediction loss is greater than the weight of the pupil prediction loss.

14. A method for determining a line of sight direction, characterized in that The method includes: Detect the eye image to be detected to obtain the target spot detection result and the pupil refraction point detection result; Obtain the target line-of-sight direction based on the target spot detection result, the pupil refraction point detection result, and the eye model modeling parameters, where the eye model modeling parameters are determined by the modeling parameter determination method of the eye model according to any one of claims 1-13.

15. An apparatus for determining modeling parameters of an eyeball model, characterized in that The device includes: An obtaining module, configured to input the first initial modeling parameters of the eye model into the spot position constraint model to obtain the predicted position parameters of the target spot, where the spot position constraint model is determined by the first initial modeling parameters of the eye model, the position parameters of the target spot, the position parameters of the target camera, and the position parameters of the target light source; The obtaining module is further configured to input the second initial modeling parameters of the eye model into the pupil refraction point position constraint model to obtain the predicted position parameters of the pupil refraction point, where the pupil refraction point position constraint model is determined by the second initial modeling parameters of the eye model, the pupil refraction point position parameters, and the position parameters of the target camera; A determining module, configured to determine a comprehensive prediction error based on the predicted position parameters of the target spot, the predicted position parameters of the pupil refraction point, the spot detection result of the eye image, and the pupil detection result; An updating module, configured to, if the comprehensive prediction error does not meet the error requirement of the eye model, update the first initial modeling parameters and the second initial modeling parameters of the eye model based on the gradient results of the spot position constraint model and the pupil refraction point position constraint model, otherwise, determine the eye model modeling parameters based on the first initial modeling parameters and the second initial modeling parameters of the eye model.

16. A device for determining a line of sight direction, characterized in that The device includes: A detecting module, configured to detect the eye image to be detected based on the eye model modeling parameters to obtain the target spot detection result and the pupil refraction point detection result; An obtaining module, configured to obtain the target line-of-sight direction based on the target spot detection result and the pupil refraction point detection result; Wherein, the eye model modeling parameters are determined by the modeling parameter determination method of the eye model according to any one of claims 1-13.

17. An electronic device, characterized in that Includes: A processor; And, A memory storing a program; Wherein, the program includes instructions that, when executed by the processor, cause the processor to execute the method according to any one of claims 1-14.

18. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions for causing the computer to execute the method according to any one of claims 1-14.

Images