Parameter determination method and device, equipment, medium and product

Abstract

The invention discloses a parameter determination method and device, equipment, a medium and a product, and the method comprises the steps: firstly obtaining a plurality of images collected for a target eyeball, so as to enable the collection angles of different images to be different; the predicted pupil edge of each image is back-projected to a three-dimensional space to obtain a three-dimensional elliptic cone corresponding to each image, so that the three-dimensional elliptic cones can better and more comprehensively represent the characteristics of the target eyeball in the three-dimensional space; and then, according to the three-dimensional elliptic cones corresponding to the plurality of images, determining eyeball parameters of the target eyeball, so that the eyeball parameters can more accurately represent characteristics of the target eyeball, thereby effectively avoiding defects existing in eye parameter determination processing based on the images at a single acquisition angle, and improving the accuracy of eye parameter determination processing. Therefore, the eyeball parameter accuracy can be improved, the image rendering effect can be improved, and the user experience can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a parameter determination method, apparatus, device, medium, and product. Background Technology

[0002] In certain scenarios, such as Virtual Reality (VR), Augmented Reality (AR), or Mixed Reality (MR), it is necessary to optimize image rendering on the screens involved in these scenarios based on the user's eye parameters, such as the position of the pupil center. This optimization may involve dynamic distortion correction (DDC). Therefore, the accuracy of these eye parameters directly affects the image rendering effect on the screens in these scenarios and the user's viewing experience, making the determination of these eye parameters a pressing technical problem. Summary of the Invention

[0003] This application provides a parameter determination method, apparatus, device, medium, and product, which helps to improve the accuracy of eye parameters, thereby improving image rendering effects and ultimately enhancing user experience.

[0004] To achieve the above objectives, the technical solution provided in this application is as follows:

[0005] This application provides a parameter determination method, the method comprising: acquiring multiple images of a target eyeball, the acquisition angles of the different images being different; for any one of the multiple images, back-projecting the predicted pupil edge of the image onto a three-dimensional space to obtain a three-dimensional elliptic cone corresponding to the image; and determining the eyeball parameters of the target eyeball based on the three-dimensional elliptic cone corresponding to the multiple images.

[0006] In one possible implementation, the process of determining the eyeball parameters of the target eyeball includes: determining multiple circular cross-sections from the three-dimensional elliptical cones corresponding to the multiple images, wherein the multiple circular cross-sections include at most two circular cross-sections in each of the three-dimensional elliptical cones, the normals of the different circular cross-sections in the at most two circular cross-sections are different, and the radii of each of the multiple circular cross-sections are the same; determining at least two circular cross-sections from the multiple circular cross-sections based on the normals of each of the multiple circular cross-sections, wherein the three-dimensional elliptical cones in which the different circular cross-sections of the at least two circular cross-sections are located are different, and the similarity between the normals of any two circular cross-sections in the at least two circular cross-sections is higher than the similarity between the normals of any other two circular cross-sections in the multiple circular cross-sections besides the at least two circular cross-sections; and determining the eyeball parameters of the target eyeball based on the at least two circular cross-sections.

[0007] In one possible implementation, for any one of the plurality of images, the vertex position of the three-dimensional elliptical cone corresponding to that image is determined based on the center position of the image acquisition device; the method further includes: for any one of the at least two circular cross-sections, determining the straight line corresponding to the circular cross-section based on the line connecting the center position of the circular cross-section and the vertex position of the three-dimensional elliptical cone in which the circular cross-section is located; determining the eyeball parameters of the target eyeball based on the at least two circular cross-sections includes: determining the eyeball parameters of the target eyeball based on the intersection point between the straight lines corresponding to the at least two circular cross-sections.

[0008] In one possible implementation, for any one of the plurality of images, the corresponding three-dimensional elliptical cone is determined based on the parameters of the image acquisition device, including the center position of the acquisition device.

[0009] In one possible implementation, the process of determining the eyeball parameters of the target eyeball includes: determining the predicted parameters of the target eyeball based on the three-dimensional elliptic cones corresponding to the plurality of images, wherein the predicted parameters are used to describe the state of the target eyeball in the plurality of images; constructing an eyeball coordinate system based on the predicted parameters; transforming the predicted parameters and the center position of the acquisition device of each of the images to the eyeball coordinate system to obtain a transformation result; and determining the eyeball parameters of the target eyeball based on the transformation result.

[0010] In one possible implementation, the prediction parameters include the predicted position of the pupil center of the target eye and the predicted gaze direction of the target eye; the eye coordinate system satisfies at least one of the following constraints: the origin of the eye coordinate system is determined based on the predicted position of the corneal center of the target eye, which is determined based on the predicted position of the pupil center and the predicted gaze direction of the target eye; the first coordinate axis of the eye coordinate system is determined based on the predicted gaze direction of the target eye; the second coordinate axis of the eye coordinate system is determined based on the normal to the target plane, which is determined based on the predicted position of the corneal center, the predicted gaze direction of the target eye, and the center position of the target image acquisition device; the plurality of images includes the target image; and the third coordinate axis of the eye coordinate system is determined based on the vector product of the first and second coordinate axes.

[0011] In one possible implementation, the prediction parameters include the predicted position of the pupil center of the target eye and the predicted gaze direction of the target eye; the step of converting the prediction parameters and the center positions of the image acquisition devices to the eye coordinate system to obtain conversion results includes: converting the predicted position of the pupil center, the predicted gaze direction, and the center positions of the image acquisition devices to the eye coordinate system to obtain the pupil center position conversion result, the gaze direction conversion result, and the center position conversion result of the image acquisition devices; the process of determining the eye parameters of the target eye includes: determining the pupil center position correction result based on a pre-constructed pupil center position correction function, the pupil center position conversion result, the gaze direction conversion result, and the center position conversion result of the image acquisition devices; and determining the eye parameters of the target eye based on the pupil center position correction result.

[0012] In one possible implementation, the pupil center position correction function is determined using eyeball parameters of at least one sample eyeball and virtual image parameters of each sample eyeball. For any sample eyeball, the eyeball parameters of the sample eyeball include the position of the pupil center of the sample eyeball, and the virtual image parameters of the sample eyeball include the virtual image position of the pupil center of the sample eyeball.

[0013] In one possible implementation, the prediction parameters include the predicted gaze direction of the target eye; the step of converting the prediction parameters and the center positions of the image acquisition devices to the eye coordinate system to obtain a conversion result includes: converting the predicted gaze direction and the center positions of the image acquisition devices to the eye coordinate system to obtain a gaze direction conversion result and a center position conversion result for each image acquisition device; the process of determining the eye parameters of the target eye includes: determining the gaze direction correction result of the target eye based on a pre-constructed gaze direction correction function, the gaze direction conversion result, and the center position conversion result of each image acquisition device; and determining the eye parameters of the target eye based on the gaze direction correction result of the target eye.

[0014] In one possible implementation, the gaze direction correction function is determined using the eye parameters of at least one sample eye and the virtual image parameters of each sample eye. For any sample eye, the eye parameters of the sample eye include the gaze direction of the sample eye, and the virtual image parameters of the sample eye include the gaze direction of the virtual image corresponding to the sample eye.

[0015] In one possible implementation, the prediction parameters include the predicted pupil radius of the target eyeball; the process of determining the eyeball parameters of the target eyeball includes: using a pre-constructed pupil radius correction function to correct the predicted pupil radius, thereby obtaining the corrected pupil radius of the target eyeball, wherein the pupil radius correction function is determined using the eyeball parameters of at least one sample eyeball and the virtual image parameters of each sample eyeball, and for any sample eyeball, the eyeball parameters of the sample eyeball include the pupil radius of the sample eyeball, and the virtual image parameters of the sample eyeball include the pupil radius of the virtual image corresponding to the sample eyeball; and determining the eyeball parameters of the target eyeball based on the corrected pupil radius of the target eyeball.

[0016] In one possible implementation, the eye parameters include the position of the pupil center of the target eye, the pupil radius of the target eye, and the gaze direction of the target eye; the method further includes: determining the virtual image parameters of the target eye based on the eye parameters and the parameters of the rendering screen in the target device, wherein the target device is equipped with image acquisition devices, the parameters of the rendering screen include the center position of the rendering screen, the center position of the rendering screen is different from the center position of each image acquisition device, and the virtual image parameters are used to describe the virtual image presented by the target eye relative to the center position of the rendering screen, the virtual image parameters including the virtual image position of the pupil center of the target eye.

[0017] In one possible implementation, the eyeball parameters of the target eyeball further include the position of the pupil edge of the target eyeball; the process of determining the virtual image parameters includes: determining the positions of multiple edge points based on the position of the pupil edge, each edge point being located on the pupil edge; for any edge point, determining the virtual image position of the edge point based on the eyeball parameters, the parameters of the rendering screen, and the position of the edge point; and determining the virtual image parameters of the target eyeball based on the virtual image positions of the multiple edge points.

[0018] In one possible implementation, the process of determining the virtual image position of any edge point includes: determining parameters of multiple observation devices based on parameters of the rendering screen; for any observation device, the parameters of the observation device include the center position of the observation device, and the distance between the center position of each observation device and the center position of the rendering screen is less than a preset distance threshold; for any observation device, constructing a refracted light path between the edge point and the observation device based on the eye parameters, the parameters of the observation device, and the position of the edge point, and constructing a direct light path corresponding to the observation device at the edge point based on the refracted light path, wherein the refracted light path and the direct light path partially overlap; and determining the virtual image position of the edge point based on the intersection of the direct light paths corresponding to the multiple observation devices at the edge point.

[0019] In one possible implementation, the acquisition times of the different images are the same.

[0020] In one possible implementation, the plurality of images are captured using a binocular camera targeting the target eye.

[0021] This application provides a parameter determination device, comprising: an image acquisition unit for acquiring multiple images of a target eyeball, wherein the acquisition angles of the different images are different; a back projection unit for backprojecting the predicted pupil edge of any one of the multiple images onto a three-dimensional space to obtain a three-dimensional elliptic cone corresponding to the image; and a first determination unit for determining the eyeball parameters of the target eyeball based on the three-dimensional elliptic cone corresponding to the multiple images.

[0022] This application provides an electronic device, the device comprising: a processor and a memory; the memory for storing instructions or computer programs; the processor for executing the instructions or computer programs in the memory, so that the electronic device performs the parameter determination method provided in this application.

[0023] This application provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform the parameter determination method provided in this application.

[0024] This application provides a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the parameter determination method provided in this application.

[0025] Compared with related technologies, this application has at least the following advantages:

[0026] In the technical solution provided in this application, for a target eyeball, such as a user's left eyeball, multiple images of the target eyeball are first acquired. For example, two images of the target eyeball are acquired using a binocular camera, with different acquisition angles for each image. This allows the multiple images to describe the state of the target eyeball at multiple acquisition angles, thus enabling a more comprehensive description of the characteristics of the target eyeball. Then, the predicted pupil edges of each image are back-projected into three-dimensional space to obtain a three-dimensional elliptic cone corresponding to each image. This allows the three-dimensional elliptic cone to represent the state of the eyeball in the corresponding image within three-dimensional space, thereby enabling these images to more comprehensively describe the characteristics of the target eyeball. Three-dimensional elliptical cones can better represent the state of the target eyeball at multiple acquisition angles in three-dimensional space, thus enabling these three-dimensional elliptical cones to better and more comprehensively represent the characteristics of the target eyeball in three-dimensional space. Then, based on the three-dimensional elliptical cones corresponding to these multiple images, the eyeball parameters of the target eyeball are determined, so that the eyeball parameters can more accurately represent the characteristics of the target eyeball, such as the position of the pupil center. This can effectively avoid the defects that exist when determining eye parameters based on images at a single acquisition angle, which is conducive to improving the accuracy of eyeball parameters, thereby improving image rendering effects and ultimately improving user experience. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic diagram of the pupil edge provided in an embodiment of this application;

[0029] Figure 2 A schematic diagram of another pupil edge provided in an embodiment of this application;

[0030] Figure 3 A flowchart illustrating a parameter determination method provided in an embodiment of this application;

[0031] Figure 4 A schematic diagram of an eyeball provided for an embodiment of this application;

[0032] Figure 5 A schematic diagram of a virtual image corresponding to an eyeball provided in an embodiment of this application;

[0033] Figure 6 A schematic diagram of another virtual image corresponding to the eyeball provided in an embodiment of this application;

[0034] Figure 7 A schematic diagram of another pupil edge provided in an embodiment of this application;

[0035] Figure 8 A schematic diagram of a three-dimensional elliptical cone provided for an embodiment of this application;

[0036] Figure 9 A schematic diagram of triangulation based on a binocular camera provided for an embodiment of this application;

[0037] Figure 10 A schematic diagram (e.g., front view) of the pupil refraction process from a certain perspective provided in an embodiment of this application;

[0038] Figure 11 A schematic diagram (such as a top view) of the pupil refraction process from another perspective provided for an embodiment of this application;

[0039] Figure 12 A schematic diagram illustrating a virtual image position determination process provided in an embodiment of this application;

[0040] Figure 13 A schematic diagram of an eye image processing flow provided in an embodiment of this application;

[0041] Figure 14 This is a schematic diagram of the structure of a parameter determination device provided in an embodiment of this application;

[0042] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0043] Research has found that some methods for determining eye parameters include: first capturing an image of the user's eye; then identifying the pupil's edge contour from that image; and finally calculating the geometric center of the pupil's edge contour, such as... Figure 1 The center of the circle shown or Figure 2 The center of the ellipse shown is used as the location of the pupil center of the eyeball, so that the eyeball parameters can be determined based on this location, and the eyeball parameters include this location.

[0044] The study also found that the scheme described above has the following drawbacks: the pupil is distorted in images obtained from certain shooting angles, causing a significant deviation in the position of the pupil center determined by this scheme, thus affecting the accuracy of the eye parameters. For ease of understanding, examples are provided below.

[0045] As an example, if a camera is used to photograph a user's eyeball from the front, the pupil in the resulting image will appear as a circle (e.g., ...). Figure 1 (as shown in the circle), so that the center of the circle (such as) Figure 1 The center of the circle shown) and the center of the pupil (as shown) Figure 1 The center of the circumscribed rectangle of the circle shown coincides with the center of the pupil, thus ensuring that the position of the pupil center determined based on the center of the circle is accurate; if the user's eyeball is photographed from the side of the camera, the pupil shape in the resulting image will be elliptical due to affine transformation (e.g., ...). Figure 2 The ellipse shown is such that its center is aligned with the center of the pupil (e.g., the ellipse shown). Figure 2 The centers of the circumscribed trapezoids of the ellipse do not coincide, making the position of the pupil center determined based on the center of the ellipse inaccurate, thus affecting the accuracy of the eye parameters.

[0046] Research has also found that in some scenarios, such as VR scenarios, in order to minimize the interference of the image acquisition process on the user, the devices used to achieve this image acquisition process, such as binocular cameras, are usually deployed in corners close to the edge. This makes it difficult for the device to capture the user's eyes from the front, resulting in lower accuracy of the eye parameters determined based on the images acquired by the device. This, in turn, affects the image rendering effect and thus impacts the user experience.

[0047] Based on the above research, in order to improve the accuracy of eyeball parameters, this application provides a parameter determination method. The method includes: for a target eyeball, such as a user's left eyeball, first acquiring multiple images of the target eyeball, such as two images acquired using a binocular camera, with different acquisition angles for each image, so that the multiple images can describe the state of the target eyeball at multiple acquisition angles, thereby enabling the multiple images to more comprehensively describe the characteristics of the target eyeball; then back-projecting the predicted pupil edge of each image onto three-dimensional space to obtain a three-dimensional elliptic cone corresponding to each image, so that the three-dimensional elliptic cone corresponding to each image can represent the corresponding… The image shows the state of the eyeball in three-dimensional space, allowing these three-dimensional elliptical cones to better represent the state of the target eyeball at multiple acquisition angles. This enables the three-dimensional elliptical cones to better and more comprehensively represent the characteristics of the target eyeball in three-dimensional space. Then, based on the three-dimensional elliptical cones corresponding to these multiple images, the eyeball parameters of the target eyeball are determined so that these eyeball parameters can more accurately represent the characteristics of the target eyeball, such as the position of the pupil center. This effectively avoids the adverse effects caused by inappropriate shooting angles, thereby improving the accuracy of eyeball parameters, which in turn improves image rendering effects and ultimately enhances the user experience.

[0048] Furthermore, this application does not limit the executing entity of the parameter determination method provided in the embodiments of this application. For example, the parameter determination method provided in the embodiments of this application can be applied to a terminal device or a server. Alternatively, the parameter determination method provided in the embodiments of this application can also be implemented through a data interaction process between a terminal device and a server. The terminal device can be a smartphone, computer, personal digital assistant (PDA), tablet computer, etc. The server can be a standalone server, a cluster server, or a cloud server.

[0049] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0050] To better understand the technical solution provided in this application, the parameter determination method provided in this application will be explained below with reference to some accompanying drawings. For example... Figure 3 As shown, the parameter determination method provided in this application includes S1-S3 as described below. Wherein, the... Figure 3 This is a flowchart of a parameter determination method provided in an embodiment of this application.

[0051] S1: Acquire multiple images of the target eyeball, with different acquisition angles for each image.

[0052] Among them, the target eyeball refers to the eyeball that requires eyeball parameter determination processing, such as the left eyeball of the VR device user, the right eyeball of the VR device user, etc. Figure 4 The eyeball shown Figure 5 The eyeball shown Figure 6 The eyeball shown.

[0053] It is evident that in some scenarios, such as VR scenarios, for users wearing VR devices (such as VR glasses), when the VR device includes a first-lens camera (such as... Figure 7 Shot 1) and shot 2 (e.g.) Figure 7In the case of lens 2), when the first lens is used to display an image to the user's left eye and the second lens is used to display an image to the user's right eye, the eyeball corresponding to the first lens (such as the user's left eyeball) or the eyeball corresponding to the second lens (such as the user's right eyeball) can be regarded as the target eyeball. This allows the image acquisition device (such as a binocular camera) deployed on the lens corresponding to the target eyeball to perform image acquisition and processing on the target eyeball, obtaining multiple images corresponding to the target eyeball (such as two images taken by the binocular camera). This enables the multiple images to more accurately and comprehensively represent the state of the target eyeball, so that the eyeball parameters of the target eyeball can be obtained based on the multiple images, and the image rendering optimization processing of the screen in the lens corresponding to the target eyeball can be achieved based on the eyeball parameters.

[0054] Multiple images refer to images obtained through image acquisition and processing targeting the eyeball, such as... Figure 7 The two images shown in the left column or Figure 7 The two images shown on the right are intended to describe the information that can be observed when the target eyeball is observed through the acquisition device (such as a binocular camera), thereby enabling the multiple images to represent some states of the target eyeball, such as the position of the pupil center, the pupil radius, and the direction of the line of sight.

[0055] In addition, in some scenarios, the multiple images mentioned above can at least satisfy the following constraints: different images in the multiple images are used to describe the same eyeball, and the acquisition angles of the different images in the multiple images are different, so that the multiple images can describe the state of the target eyeball under multiple acquisition angles, thereby making the eyeball parameters determined based on the multiple images more accurate. This can effectively avoid the defect of low accuracy of eyeball parameters determined based on images under a single acquisition angle.

[0056] In addition, in some scenarios, the multiple images mentioned above can also satisfy the following constraint: the acquisition time of different images in the multiple images is the same, so that the multiple images can describe the state of the target eyeball at multiple acquisition angles at the same time. This can effectively avoid the interference caused by different acquisition times of different images, thereby making the eyeball parameters determined based on the multiple images more accurate.

[0057] Furthermore, this application does not limit the implementation method of S1 above. For example, when the lens corresponding to the target eyeball (such as...) Figure 6When the VR lens shown includes M image acquisition devices, S1 can specifically be as follows: First, acquire the image acquired by the m-th image acquisition device for the target eyeball, and use it as the m-th image, so that the m-th image can represent the state of the target eyeball at the acquisition angle corresponding to the m-th image acquisition device, where m is a positive integer, m≤M, and M is a positive integer; then, based on the 1st image to the M-th image, determine the above multiple images, so that the multiple images include the 1st image to the M-th image, thereby enabling the multiple images to describe the state of the target eyeball at multiple acquisition angles.

[0058] It should be noted that this application does not limit the implementation of the "M image acquisition devices" mentioned above. For example, the M image acquisition devices can at least satisfy the following constraint: the center positions of different devices among the M image acquisition devices are different, so that the acquisition angles of different devices among the M image acquisition devices are different, which is beneficial to improving accuracy.

[0059] For example, in some scenarios, to reduce costs, the M image acquisition devices mentioned above can be binocular cameras, such as... Figure 6 The illustrated binocular camera is implemented such that the M image acquisition devices include two imaging lenses (such as camera A lens and camera B lens, etc.) of the binocular camera, thereby enabling the multiple images acquired using the M image acquisition devices to include images captured by the two imaging lenses, such as... Figure 7 The two images shown in the left column or the two images shown in the right column.

[0060] Therefore, in one possible implementation, the aforementioned multiple images can be acquired using a binocular camera targeting the target eye, such that the multiple images include two eye images captured by the binocular camera, as shown below. Figure 7 The two images shown in the left column allow the multiple images to describe the target eyeball from two acquisition angles.

[0061] Based on the above content regarding S1, in some scenarios, such as VR scenarios, when a user is wearing a VR device, which includes a first lens and a second lens, the first lens is used to display an image to the user's left eye, and the second lens is used to display an image to the user's right eye. If the user's left eyeball is taken as the target eyeball, then the image captured by the binocular camera deployed on the first lens for the left eye (such as...) can be used to display an image to the user's right eye. Figure 7 The two images shown in the left column are used as multiple images captured for the target eye, so that the eye parameters of the left eye can be determined based on these multiple images; if the user's right eye is used as the target eye, the images captured by the binocular camera deployed on the second lens for the right eye (such as...) can be used as the target eye. Figure 7The two images shown on the right are multiple images acquired for the target eyeball, so that the eyeball parameters of the right eye can be determined based on these multiple images.

[0062] S2: For any image among multiple images, backproject the predicted pupil edge of the image to three-dimensional space to obtain the corresponding three-dimensional elliptic cone.

[0063] Here, the predicted pupil edge of the m-th image refers to the result obtained by performing pupil edge detection processing on the m-th image, such as... Figure 7 The pupil edge is shown so that the predicted pupil edge is used to describe the position of the pupil edge contour in the m-th image, so that the predicted pupil edge can represent the predicted position of the pupil of the target eyeball in the m-th image, where m is a positive integer, m≤M, and M is a positive integer.

[0064] The three-dimensional elliptic cone corresponding to the m-th image is obtained by back-projecting the predicted pupil edge of the m-th image, so that the three-dimensional elliptic cone can more accurately describe the state of the pupil in the m-th image in three-dimensional (3-Dimensional, 3D) space, thereby making the eye parameters determined based on the three-dimensional elliptic cone more accurate. m is a positive integer, m≤M, and M is a positive integer.

[0065] It should be noted that this application does not limit the implementation method of the 3D space; for example, it can be implemented using any kind of 3D space. For instance, when back-projecting the predicted pupil edge of the m-th image, the 3D space involved in the back-projection can be implemented using the 3D space constructed based on the acquisition device of the m-th image.

[0066] Furthermore, this application does not limit the implementation method of the back projection in S2 above. For example, it can be implemented using any existing or future back projection method.

[0067] For example, to improve accuracy, S2 above can be specifically described as follows: after obtaining the predicted pupil edge of the m-th image, based on the parameters of the acquisition device of the m-th image (such as camera intrinsic parameters), the predicted pupil edge of the m-th image is back-projected into three-dimensional space to obtain the three-dimensional elliptic cone corresponding to the m-th image, where m≤M and M is a positive integer.

[0068] The parameters of the acquisition device for the m-th image refer to the parameters used when the acquisition device is used to acquire and process images of the target eye, such as camera intrinsic parameters, so that the parameters can indicate the state of the acquisition device when acquiring the m-th image.

[0069] Furthermore, this application does not limit the implementation of the "parameters of the acquisition device for the m-th image" mentioned above. For example, the "parameters of the acquisition device for the m-th image" may at least include the center position of the acquisition device, so that the vertex position of the three-dimensional elliptical cone corresponding to the m-th image can be determined based on the center position, so that the vertex of the three-dimensional elliptical cone corresponding to the m-th image can be located at the center of the acquisition device for the m-th image (e.g., Figure 8 The center of the camera shown is used as the vertex.

[0070] For example, in some scenarios, the "parameters of the acquisition device for the m-th image" mentioned above may also include the light direction of each pixel in the m-th image (e.g., used to construct the image). Figure 8 The direction of the green lines in the 3D elliptic cone is used to determine the direction of the ray rays from the predicted pupil edge of the m-th image to the corresponding pixels in the m-th image, so that the 3D elliptic cone corresponding to the m-th image can be obtained later. Figure 8 The 3D elliptical cone shown.

[0071] Based on the relevant content in S2 above, in some scenarios, for the target eyeball, after acquiring the image of the target eyeball, the image is first processed by pupil edge detection to obtain the predicted pupil edge. This predicted pupil edge is used to describe the position of the pupil, which is in a two-dimensional ellipse state, in the image. Then, using the parameters of the image acquisition device (such as camera intrinsics), the predicted pupil edge is projected onto 3D space to obtain the corresponding three-dimensional elliptic cone (e.g., ...). Figure 8 The 3D elliptic cone (as described in the image) is such that the vertex position of the 3D elliptic cone is determined based on the center position of the image acquisition device, and the 3D elliptic cone comprises a 2D ellipse described by the predicted pupil edge (e.g., a 3D elliptic cone). Figure 8 The pupil edge in the image is represented by a three-dimensional elliptic cone, which allows the pupil to be positioned in 3D space within the image.

[0072] S3: Determine the eyeball parameters of the target eyeball based on the three-dimensional elliptical cones corresponding to multiple images.

[0073] Among them, the eye parameters of the target eye are used to describe the state of the target eye when the image is acquired, such as the position of the pupil center, the direction of the gaze, the pupil radius, etc., so that the eye parameters can represent the characteristics of the target eye.

[0074] Furthermore, this application does not limit the implementation of the eyeball parameters of the target eyeball. For example, the eyeball parameters may include one or more of the following: the position of the pupil center of the target eyeball, the pupil radius of the target eyeball, the line of sight of the target eyeball, the position of the pupil edge of the target eyeball, and the position of the corneal center of the target eyeball.

[0075] The position of the pupil center of the target eye refers to the actual position of the pupil center in any coordinate system, such as... Figure 4-6 The location of the pupil center is shown. It should be noted that this application does not limit the implementation of this coordinate system; for example, it can be implemented using a world coordinate system, a VR coordinate system, a camera coordinate system, or an eye-tracking coordinate system.

[0076] The pupil radius of a target eyeball refers to the actual radius of the pupil of the target eyeball in any coordinate system, so that the pupil radius can represent the actual size of the target eyeball.

[0077] The gaze direction of the target eyeball refers to the pupil center normal of the target eyeball in any coordinate system when the image is acquired, so that the "gaze direction of the target eyeball" can represent the actual gaze direction of the target eyeball in that coordinate system.

[0078] The position of the pupil edge of the target eyeball refers to the actual position of the pupil edge of the target eyeball in any coordinate system, so as to represent the actual outline of the pupil of the target eyeball.

[0079] The location of the corneal center of the target eye refers to the actual location of the corneal center of the target eye in any coordinate system, so as to represent the actual state of the cornea of ​​the target eye.

[0080] Research has found that for an eyeball, the center of the cornea lies on the reverse extension of the line of sight (also known as the pupil center normal), so that both the center of the cornea and the center of the pupil are located on the straight line of the pupil center normal, and the position of the center of the cornea on this straight line is closer to the inside of the eyeball than the position of the center of the pupil on this straight line; moreover, the straight-line distance between the center of the cornea and the center of the pupil is a fixed value.

[0081] Based on the above research, in one possible implementation, the position of the corneal center of the target eyeball can be determined based on the position of the pupil center and the line of sight of the target eyeball. These three data points should at least satisfy the following eyeball structural constraints: the positions of the corneal center and the pupil center of the target eyeball are both located on a straight line described by the line of sight of the target eyeball; the distance between the corneal center and the pupil center of the target eyeball is a pre-specified first distance; and the corneal center is closer to the interior of the eyeball than the pupil center.

[0082] As can be seen, in one possible implementation, the process of determining the position of the corneal center of the target eyeball described above can be as follows: based on the position of the pupil center of the target eyeball, the line of sight of the target eyeball, and a pre-specified first distance, calculate the position of the corneal center of the target eyeball so that the position of the corneal center is located on the reverse extension line of the line of sight, the distance between the position of the corneal center and the position of the pupil center is the first distance, and the position of the corneal center is closer to the inside of the eyeball than the position of the pupil center.

[0083] It should be noted that the first distance shown in the above two paragraphs is used to describe the straight-line distance between the center of the cornea and the center of the pupil in the same eyeball; and this application does not limit the first distance, for example, it can be determined based on a large amount of experimental data.

[0084] Furthermore, this application does not limit the implementation method of S3 described above. For example, S3 can be implemented using a pre-built machine learning model with eye parameter determination capabilities. It should be noted that this application does not limit the implementation method of the machine learning model.

[0085] Research has revealed that while the pupil's edge is circular in 3D space, the image captured by the acquisition device shows it as a two-dimensional ellipse.

[0086] Based on the above research, this application also provides a possible implementation of S3 above, in which S3 may include steps 11-13 below.

[0087] Step 11: Determine multiple circular cross sections from the three-dimensional elliptical cones corresponding to multiple images. The multiple circular cross sections include at most two circular cross sections in each three-dimensional elliptical cone. The normals of the different circular cross sections in the at most two circular cross sections are different, and the radii of each circular cross section in the multiple circular cross sections are the same.

[0088] The 3D elliptical cone corresponding to the m-th image must satisfy at least the following constraint: for any radius R, there are at most two circular cross-sections with radius R on the 3D elliptical cone, such as... Figure 8 The circular cross-sections 1 and 2 are shown, and the normals of these two circular cross-sections are different, m≤M, where M is a positive integer.

[0089] It should be noted that, for the 3D elliptical cone corresponding to the m-th image, if the predicted pupil edge of the m-th image is a two-dimensional ellipse, then for any radius R, two circular cross-sections with radius R can be found on the 3D elliptical cone corresponding to the m-th image; if the predicted pupil edge of the m-th image is a two-dimensional circle, then the 3D elliptical cone corresponding to the m-th image should be a 3D cone, so that for any radius R, a circular cross-section with radius R can be found on the 3D elliptical cone corresponding to the m-th image.

[0090] Based on the relevant content of step 11 above, when the above multiple images include M images, after obtaining the three-dimensional elliptical cone corresponding to the m-th image, we can first search for all circular cross-sections with radius R in the three-dimensional elliptical cone corresponding to the m-th image to obtain the circular cross-section corresponding to the m-th image, so that the "circular cross-section corresponding to the m-th image" can represent all circular cross-sections with radius R in the three-dimensional elliptical cone corresponding to the m-th image, where m≤M and M is a positive integer; then, based on the circular cross-sections corresponding to the multiple images, we determine the above multiple circular cross-sections so that the multiple circular cross-sections include the circular cross-sections corresponding to the multiple images.

[0091] Step 12: Based on the normals of each of the multiple circular cross sections mentioned above, determine at least two circular cross sections from the multiple circular cross sections. The three-dimensional elliptical cones in which the different circular cross sections are located are different. The similarity between the normals of any two circular cross sections in the at least two circular cross sections is higher than the similarity between the normals of any two other circular cross sections in the multiple circular cross sections.

[0092] Among them, at least two circular cross sections refer to circular cross sections with high similarity among the multiple circular cross sections mentioned above, so that the at least two circular cross sections can represent the state of the pupil edge in 3D space when captured from different acquisition angles.

[0093] Furthermore, the at least two circular cross-sections mentioned above can satisfy the following constraints: the different circular cross-sections within these at least two circular cross-sections are located in different three-dimensional elliptical cones, and the similarity between the normals of any two circular cross-sections within these at least two circular cross-sections is higher than the similarity between the normals of any other two circular cross-sections besides the at least two circular cross-sections, so that the "at least two circular cross-sections" can represent circular cross-sections with a relatively high degree of similarity existing in different three-dimensional elliptical cones. For ease of understanding, examples are provided below.

[0094] As an example, suppose the above multiple images include a left-eye image and a right-eye image taken using a binocular camera; the above multiple circular cross-sections include a circular cross-section corresponding to the left-eye image and a circular cross-section corresponding to the right-eye image; the circular cross-section corresponding to the left-eye image includes a first circular cross-section and a second circular cross-section; the circular cross-section corresponding to the right-eye image includes a third circular cross-section and a fourth circular cross-section; the radii of the first circular cross-section, the second circular cross-section, the third circular cross-section, and the fourth circular cross-section are all the same.

[0095] Based on the above assumption, step 12 can be specifically described as follows: First, compare the similarity between the normals of the first circular cross section and the third circular cross section, the similarity between the normals of the first circular cross section and the fourth circular cross section, the similarity between the normals of the second circular cross section and the third circular cross section, and the similarity between the normals of the second circular cross section and the fourth circular cross section to obtain the minimum similarity, so that the two circular cross sections with the minimum similarity are as close as possible to the pupil observed by the binocular camera; then, based on the two circular cross sections with the minimum similarity, determine at least two circular cross sections mentioned above, so that the at least two circular cross sections include the two circular cross sections with the minimum similarity, so that the eyeball parameters of the target eyeball can be determined based on the two circular cross sections with the minimum similarity.

[0096] Step 13: Based on the above at least two circular cross-sections, determine the eyeball parameters of the target eyeball.

[0097] It should be noted that this application does not limit the implementation of step 13 above. For example, it can be implemented using a pre-built machine learning model capable of predicting eyeball parameters based on at least two circular cross-sections mentioned above. It should also be noted that this application does not limit the implementation of this machine learning model.

[0098] In addition, to further improve accuracy, this application also provides a possible implementation of step 13 above. In this implementation, when the vertex position of the three-dimensional elliptical cone corresponding to any one of the multiple images above is determined based on the center position of the image acquisition device, step 13 may specifically include steps 131-132 below.

[0099] Step 131: For any one of the at least two circular cross-sections mentioned above, determine the straight line corresponding to the circular cross-section based on the line connecting the center of the circular cross-section and the vertex of the three-dimensional elliptical cone in which the circular cross-section is located (e.g., ...). Figure 9 The line shown is either line 1 or line 2, such that the line includes the center position of the image acquisition device and the pupil center described by the image, so that the line can represent the line where the pupil center described by the image is located, and further so that the line can represent the optical axis of the beam involved in acquiring the image.

[0100] Step 132: Determine the eyeball parameters of the target eyeball based on the intersection points between the straight lines corresponding to at least two circular cross-sections mentioned above.

[0101] It should be noted that this application does not limit the implementation of step 132 above. For example, it can specifically be: first, take the intersection point between the straight lines corresponding to at least two circular cross sections above as the position prediction result of the pupil center of the target eyeball, so that the position prediction result can represent the position of the pupil center described by the multiple images above; then, determine the eyeball parameters of the target eyeball based on the position prediction result of the pupil center of the target eyeball.

[0102] It should also be noted that this application does not limit the implementation of the step "determine the eyeball parameters of the target eyeball based on the predicted position of the pupil center of the target eyeball" in the above paragraph. For example, in some scenarios, such as scenarios where the accuracy requirement is not very high, this step can specifically be: determine the eyeball parameters of the target eyeball based on the predicted position of the pupil center of the target eyeball, so that the eyeball parameters include the predicted position result.

[0103] Furthermore, step 132 above can specifically be: First, the intersection point between the straight lines corresponding to at least two circular cross-sections mentioned above is taken as the predicted position of the pupil center of the target eyeball, and the average value of the normals of the at least two circular cross-sections is taken as the predicted gaze direction of the target eyeball (e.g., Figure 9 The predicted gaze direction is shown in the image above, so that the predicted gaze direction can represent the gaze direction described by the multiple images above; then, based on the predicted position of the pupil center of the target eyeball and the predicted gaze direction of the target eyeball, the eyeball parameters of the target eyeball are determined.

[0104] It should be noted that this application does not limit the implementation of the step "determine the eye parameters of the target eye based on the predicted position of the pupil center of the target eye and the predicted line of sight of the target eye" in the above paragraph. For example, in some scenarios, such as scenarios where the accuracy requirement is not very high, this step can specifically be: determine the eye parameters of the target eye based on the predicted position of the pupil center of the target eye and the predicted line of sight of the target eye, so that the eye parameters include the predicted position and the predicted line of sight.

[0105] Furthermore, step 132 above can specifically be as follows: First, the intersection point between the straight lines corresponding to at least two circular cross-sections is taken as the predicted position of the pupil center of the target eyeball, and the average value of the normals of the at least two circular cross-sections is taken as the predicted direction of the gaze of the target eyeball; then, based on the predicted position of the pupil center of the target eyeball and the predicted direction of the gaze of the target eyeball, the predicted result of the pupil of the target eyeball is determined (e.g., Figure 9 The pupil prediction results shown are used to represent the state of the pupil described by the multiple images above. The pupil prediction results include parameters such as the predicted position of the pupil center of the target eye, the predicted normal of the pupil center of the target eye (such as the predicted gaze direction), and the predicted pupil radius of the target eye. Then, based on the pupil prediction results of the target eye, the eye parameters of the target eye are determined.

[0106] It should be noted that this application does not limit the implementation of the step "determine the eye parameters of the target eye based on the prediction result of the pupil of the target eye" in the above paragraph. For example, in some scenarios, such as scenarios where the accuracy requirement is not very high, this step can specifically be: determine the eye parameters of the target eye based on the prediction result of the pupil of the target eye, so that the eye parameters include the prediction result.

[0107] It should also be noted that this application does not limit the method for determining the predicted pupil radius of the target eyeball. For example, the predicted pupil radius can be determined based on the predicted position of the pupil center of the target eyeball and some or all parameters of the acquisition devices (such as center position parameters). Therefore, in one possible implementation, the predicted pupil radius of the target eyeball can be inferred from the distance between the predicted position of the pupil center of the target eyeball and the center position of any of the acquisition devices among the multiple image acquisition devices mentioned above, so that the inference process conforms to the imaging principle used by the acquisition device itself.

[0108] Based on the relevant content of steps 11 to 13 above, for some scenarios, such as scenarios with low accuracy requirements or scenarios where corneal refraction is not considered, after acquiring the three-dimensional elliptical cones corresponding to multiple images, the radius R can be fixed first, and all circular cross-sections with radius R can be found on each elliptical cone. Then, based on the normal of these circular cross-sections, at least two circular cross-sections are determined so that these at least two circular cross-sections can represent the circular cross-sections in each three-dimensional elliptical cone whose normal is closest to the viewing direction described by the multiple images. This makes the normal of these at least two circular cross-sections as parallel as possible to the normal of the pupil center described by the multiple images. Consequently, the line connecting the center position of each circular cross-section in these at least two circular cross-sections to the center position of the corresponding acquisition device can represent the straight line where the pupil center is located. This allows the pupil center position described by the multiple images to be calculated using a certain method (such as binocular camera pupil direction triangulation). This allows the eye parameters of the target eyeball to be determined based on the pupil center position described by the multiple images.

[0109] Research has found that there is a transparent cornea on the outer side of the pupil of the eye. Because the refractive index of the cornea is different from that of air, light is refracted at the surface of the cornea. Figure 5 , Figure 6 , Figure 10 as well as Figure 11 The refraction shown makes the image acquired for this eyeball a virtual image after being refracted by the cornea, thus affecting the prediction of eyeball parameters (such as...) based on this image. Figures 9 to 11 The predicted information shown) and the actual eye parameters (such as...) Figure 10 and Figure 11 The discrepancy between the actual pupil and other parameters shown in the image introduces refractive errors into the eye parameters predicted based on the image.

[0110] Based on the above research, in order to improve accuracy, this application provides a possible implementation of S3 above, in which S3 may specifically include steps 21-24 below.

[0111] Step 21: Based on the three-dimensional elliptic cones corresponding to the multiple images above, determine the prediction parameters of the target eyeball. These prediction parameters are used to describe the state of the target eyeball in the multiple images.

[0112] The predicted parameters for the target eye refer to the eye parameters predicted using the multiple images mentioned above, so that these predicted parameters can represent the eye state described by the multiple images, such as... Figure 10 or Figure 11The virtual image state shown allows the prediction parameter to represent the state of the target eyeball after refraction through the cornea, thus enabling the pupil described by the prediction parameter to represent the virtual pupil image captured by the acquisition device, such as... Figure 10 or Figure 11 The pupil prediction results are shown.

[0113] Furthermore, this application does not limit the implementation of the prediction parameters of the target eyeball mentioned above. For example, the prediction parameters may include one or more of the predicted position of the pupil center of the target eyeball, the predicted line of sight of the target eyeball, the predicted pupil radius of the target eyeball, and the predicted position of the corneal center of the target eyeball.

[0114] The predicted position of the pupil center of the target eye is used to describe the position of the pupil center described by the multiple images above in any coordinate system, so that the predicted position can represent the center position of the virtual pupil image captured by the acquisition device of the multiple images.

[0115] The predicted gaze direction of the target eyeball is used to describe the state of the gaze direction described by the multiple images above in any coordinate system, so that the predicted gaze direction can represent the center normal of the pupil virtual image captured by the acquisition device of the multiple images.

[0116] The pupil radius prediction result of the target eyeball is used to describe the radius of the pupil described by the multiple images above in any coordinate system, so that the pupil radius prediction result can represent the size of the virtual pupil image captured by the acquisition device of the multiple images.

[0117] The predicted position of the corneal center of the target eye is used to describe the position of the corneal center described by the multiple images above in any coordinate system, so that the predicted position can represent the position of the corneal center corresponding to the virtual pupil image captured by the acquisition device of the multiple images; moreover, the predicted position of the corneal center of the target eye is determined based on the predicted position of the pupil center of the target eye and the predicted line of sight of the target eye. These three data can at least satisfy the following eye structure constraints: the predicted position of the corneal center of the target eye and the predicted position of the pupil center of the target eye are both located on the straight line described by the predicted line of sight of the target eye; the distance between the predicted position of the corneal center of the target eye and the predicted position of the pupil center of the target eye is a pre-specified first distance; and the predicted position of the corneal center of the target eye is closer to the interior of the eye than the predicted position of the pupil center of the target eye. It should be noted that the relevant content of the first distance is described above.

[0118] Therefore, in one possible implementation, the process of determining the predicted position of the corneal center of the target eye can be as follows: based on the predicted position of the pupil center of the target eye, the predicted line of sight of the target eye, and a pre-specified first distance, the predicted position of the corneal center of the target eye is calculated such that the position described by the predicted position of the corneal center is located on the reverse extension line of the line of sight described by the predicted line of sight, the distance between the position described by the predicted position of the corneal center and the position described by the predicted position of the pupil center is the first distance, and the position described by the predicted position of the corneal center is closer to the interior of the eye than the position described by the predicted position of the pupil center.

[0119] Furthermore, this application does not limit the implementation of step 21 above. For example, step 21 can be implemented using a pre-built machine learning model with eye parameter prediction capabilities. It should be noted that this application does not limit the implementation of this machine learning model.

[0120] In addition, to improve accuracy, step 21 above can be implemented using any of the embodiments shown in steps 11-13 above. Simply replace the "eye parameters of the target eye" in any of the embodiments shown in steps 11-13 with the "predicted parameters of the target eye".

[0121] Based on the relevant content of step 21 above, for multiple images acquired for the target eyeball, after obtaining the three-dimensional elliptical cones corresponding to the multiple images, eyeball parameter prediction processing can be performed based on these three-dimensional elliptical cones to obtain the predicted parameters of the target eyeball, so that the predicted parameters can represent the state of the target eyeball in the multiple images, thereby enabling the predicted parameters to represent the virtual image captured by the acquisition device of the multiple images and presented after refraction through the cornea, such as the virtual image of the pupil.

[0122] Step 22: Construct an eye coordinate system based on the predicted parameters of the target eye.

[0123] It should be noted that this application does not limit the implementation of step 22 above. For example, it can be implemented using any existing or future eye coordinate system construction method.

[0124] In addition, to further improve accuracy, this application provides a possible implementation of the eye coordinate system described above, in which the eye coordinate system can satisfy at least one of the constraints described in ①-④ below.

[0125] ① The origin of the eye coordinate system is determined based on the predicted position of the corneal center of the target eye, so that the eye coordinate system can take the position described by the predicted position as the origin, thereby making the eye coordinate system take the corneal center as the origin.

[0126] ② The first coordinate axis in the eye coordinate system is determined based on the predicted direction of the target eye's gaze, so that the eye coordinate system can use the vector described by the predicted gaze direction as the first coordinate axis, thereby allowing the eye coordinate system to use the gaze direction after refraction as the first coordinate axis. Here, the first coordinate axis refers to a coordinate axis in the eye coordinate system, such as the X-axis.

[0127] ③ The second coordinate axis in the eye coordinate system is determined based on the normal of the target plane. The target plane is determined based on the predicted position of the corneal center of the target eye, the predicted line of sight of the target eye, and the center position of the target image acquisition device. The multiple images mentioned above include the target image.

[0128] The target image refers to any one of the multiple images mentioned above, such as an image captured by the left eye camera in a binocular camera; moreover, this application does not limit the target image, for example, it can be determined by random selection.

[0129] The target plane refers to a plane constructed using the predicted position of the corneal center of the target eye, the predicted line of sight of the target eye, and the center position of the target image acquisition device, such that the target plane includes the position described by the predicted position of the corneal center, the center position of the target image acquisition device, and the straight line described by the predicted line of sight.

[0130] Furthermore, this application does not limit the implementation of the second coordinate axis in the eye coordinate system described above. For example, the second coordinate axis may include the center position of the target image acquisition device, so that after the center position of the acquisition device is transformed into the eye coordinate system, it only includes a non-zero value, which helps to reduce the number of parameters involved in subsequent correction processing and thus improves efficiency.

[0131] ④ The third coordinate axis in the eye coordinate system is determined by the vector product of the first and second coordinate axes.

[0132] Based on the relevant content in ① to ④ above, after obtaining the predicted parameters of the target eyeball, an eyeball coordinate system can be constructed based on the predicted parameters and the parameters of any one of the multiple image acquisition devices mentioned above. This eyeball coordinate system should satisfy the constraints shown in ①-④, so that data from different coordinate systems can be unified into the same coordinate system. This helps to overcome the defects of processing data from different coordinate systems, thereby improving accuracy.

[0133] Step 23: Transform the predicted parameters of the target eyeball and the center position of each image acquisition device to the eyeball coordinate system to obtain the transformation result, so as to unify the data under different coordinate systems into the same coordinate system.

[0134] It should be noted that this application does not limit the implementation of step 23 above. For example, it may include at least steps 231-233 below.

[0135] Step 231: Transform the predicted position of the pupil center of the target eyeball into the eyeball coordinate system to obtain the position transformation result of the pupil center of the target eyeball, so that the position transformation result can describe the position of the pupil center described by the multiple images above in the eyeball coordinate system, such as coordinates.

[0136] Step 232: Transform the predicted gaze direction of the target eyeball into the eyeball coordinate system to obtain the gaze direction transformation result of the target eyeball, so that the gaze direction transformation result can describe the state of the gaze direction described by the multiple images above in the eyeball coordinate system, such as a vector.

[0137] Step 233: Transform the center position of the acquisition device for the m-th image to the eye coordinate system to obtain the transformation result of the center position of the acquisition device for the m-th image, so that the transformation result can describe the position of the center of the acquisition device in the eye coordinate system, such as coordinates, etc., where m is a positive integer, m≤M, and M is a positive integer.

[0138] It should be noted that this application does not limit the execution relationship between steps 231 to 233 above. For example, they can be executed simultaneously. Or, they can be executed in a pre-set order.

[0139] Step 24: Based on the conversion results above, determine the eye parameters of the target eye.

[0140] It should be noted that this application does not limit the implementation of step 24 above. For ease of understanding, some examples are provided below.

[0141] Example 1: Step 24 above may specifically include at least steps 241-242 below.

[0142] Step 241: Based on the pre-constructed pupil center position correction function, the position conversion result of the pupil center of the target eyeball, the line of sight conversion result of the target eyeball, and the center position conversion result of each image acquisition device, determine the position correction result of the pupil center of the target eyeball, so that the position correction result can represent the position of the pupil center of the target eyeball before refraction, thereby enabling the position correction result to represent the actual position of the pupil center of the target eyeball.

[0143] The pupil center position correction function is used to correct the position conversion result of the pupil center of the target eyeball, such as eliminating refraction error.

[0144] In addition, to improve accuracy, the pupil center position correction function mentioned above can be a function constructed by the least squares method, as shown in formula (1) below.

[0145] p rev =∑ 0≤i,j,k,l≤3 α ijkl p i K j R k n l +∑ 0≤i,j,k,l≤3 β ijkl g i K j R k n l +∑ 0≤i,j,k,l≤3 γ ijkl ela i K j R k n l +

[0146] ∑ 0≤i,j,k,l≤3 δ ijkl elb i K j R k n l (1)

[0147] In the formula, p rev This represents the result of correcting the position of the pupil center of the target eye; p represents the result of transforming the position of the pupil center of the target eye; p i Represents p raised to the power of i; K represents a pre-specified second distance, which is the distance between the center of eye rotation and the center of the cornea; K j K raised to the power of j; R represents the pre-specified corneal radius; R kR is represented by the power of k; n represents the pre-specified refractive index; n l Represents n raised to the power of l; g represents the result of the change in the gaze direction of the target's eye; g i ela represents g raised to the power of i; ela represents the result of the center position transformation of camera A in a stereo camera setup; ela i Elb represents ela raised to the power of i; elb represents the result of the center position transformation of camera B in the stereo camera setup; elb i α represents elb raised to the power of i; ijjkl Indicates prior to p i K j R k n l The coefficient predicted by this term, α ijjkl In this context, "ijjkl" is the serial number; β ijkl Indicates prior to g i K j R k n l The coefficient predicted by this term, β ijkl "ijjkl" in the text is the serial number; γ ijkl Indicates prior targeting of ela i K j R k n l The coefficient predicted by this term, γ ijkl "ijjkl" in the text is the serial number; δ ijkl Indicates prior targeting of elb i K j R k n l The coefficient predicted by this term, δ ijkl "ijjkl" in the text is the serial number.

[0148] It should be noted that this application does not limit the method of obtaining the second distance in the above paragraph. For example, the second distance can be obtained by analyzing a large amount of experimental data. This application also does not limit the method of obtaining the corneal radius in the above paragraph. For example, the corneal radius can be obtained by analyzing a large amount of experimental data. This application also does not limit the method of obtaining the refractive index in the above paragraph. For example, the refractive index can be obtained by analyzing a large amount of experimental data. This application also does not limit the prediction method of the coefficient in the above paragraph. For example, the coefficient can be obtained by analyzing and processing a large amount of simulation data, such as parameter fitting, so that the coefficient represents the mapping relationship between the actual position described by these simulation data and the corresponding virtual image position.

[0149] It should also be noted that this application does not limit the implementation method of the simulation data in the above paragraph. For example, the simulation data can be data obtained by performing eyeball simulation processing, so that the simulation data can describe some parameters that the actual eyeball has (such as the position of the pupil center, pupil radius, line of sight, etc.) and some parameters of the virtual image presented by the eyeball after refraction through the cornea. This allows these parameters to be substituted into the corresponding functions (such as the pupil center position correction function mentioned above, the line of sight correction function mentioned below, or the pupil radius correction function) to solve for the coefficients involved in the function, and obtain the coefficients corresponding to the simulation data. This allows the optimal values ​​of the coefficients in the function to be obtained by analyzing and processing the coefficients corresponding to all simulation data, such as fitting processing, least squares processing, etc., so that the function can be used to correct the parameters after refraction (such as the predicted parameters of the target eyeball mentioned above).

[0150] As can be seen, in one possible implementation, the pupil center position correction function mentioned above can be determined using the eyeball parameters of at least one sample eyeball and the virtual image parameters of each sample eyeball, so that the pupil center position correction function can more accurately represent the correspondence between the parameters before refraction and the parameters after refraction, which is beneficial to improving accuracy.

[0151] Here, the sample eyeball refers to an eyeball constructed through simulation, so that the eyeball parameters of the sample eyeball are set in a certain way, so that the eyeball parameters of the sample eyeball can be used as known information to solve the coefficients that are in an unknown state in the pupil center position correction function mentioned above.

[0152] Furthermore, for any sample eyeball, the eyeball parameters refer to the parameters required when constructing the sample eyeball so that the eyeball parameters can represent the actual state of the sample eyeball, such as the position of the pupil center, the pupil radius, the direction of the gaze, etc., so that the eyeball parameters can be used as known information to guide the solution process of the coefficients in the unknown state in the pupil center position correction function mentioned above.

[0153] Therefore, in one possible implementation, for any sample eyeball, the eyeball parameters may include some or all of the following parameters: the position of the pupil center, the direction of the gaze, and the pupil radius. The position of the pupil center describes the actual location of the pupil center. The direction of the gaze describes the actual direction of the gaze. The pupil radius describes the actual pupil radius of the sample eye.

[0154] Furthermore, for any sample eyeball, the virtual image parameter of that sample eyeball refers to the parameter obtained after refraction processing of that sample eyeball, so that the virtual image parameter can represent the state of the sample eyeball after refraction, thereby enabling the virtual image parameter to represent the state of the eyeball captured by some acquisition equipment, so that the virtual image parameter can be used as known information to solve the coefficients in the unknown state in the pupil center position correction function mentioned above.

[0155] Therefore, in one possible implementation, for any sample eyeball, the virtual image parameters of the sample eyeball may include some or all of the following parameters: the virtual image position of the pupil center, the line of sight of the virtual image corresponding to the sample eyeball, and the pupil radius of the virtual image of the sample eyeball. Specifically, the virtual image position of the pupil center describes the position of the pupil center after refraction. The virtual image corresponding to the sample eyeball describes the state of the sample eyeball after refraction. The line of sight of the virtual image corresponding to the sample eyeball describes the line of sight after refraction. The pupil radius of the virtual image of the sample eyeball describes the pupil radius after refraction.

[0156] Based on the above content regarding the pupil center position correction function, we can first treat the coefficients in the pupil center position correction function as unknowns. Then, using the eye parameters of some sample eyes and the virtual image parameters of these sample eyes, we can deduce the coefficients in the pupil center position correction function. This allows the pupil center position correction function with these coefficients to accurately represent the correspondence between the parameters before and after refraction. This enables us to subsequently use the pupil center position correction function with these coefficients to achieve refraction error elimination for the pupil center position.

[0157] Furthermore, this application does not limit the implementation of step 241 above. For example, it can specifically be: substituting the position conversion result of the pupil center of the target eyeball, the gaze direction conversion result of the target eyeball, and the center position conversion result of each image acquisition device into a pre-constructed pupil center position correction function to obtain the position correction result of the pupil center of the target eyeball, such as p rev This is to ensure that the position correction result can represent the position of the pupil center of the target eyeball before refraction, thereby enabling the position correction result to represent the actual position of the pupil center of the target eyeball.

[0158] Step 242: Based on the position correction result of the pupil center of the target eye, determine the eye parameters of the target eye so that the eye parameters include the position correction result of the pupil center of the target eye.

[0159] Based on the relevant content of steps 241 to 242 above, for the target eyeball, after obtaining the position transformation result of the pupil center of the target eyeball, the position transformation result can be corrected by using a pre-constructed pupil center position correction function, such as removing refraction error processing, to obtain the position correction result of the pupil center of the target eyeball. This position correction result can represent the position of the pupil center after removing refraction error, so that the position correction result can more accurately represent the position of the pupil center of the target eyeball, and thus make the eyeball parameters determined based on the position correction result more accurate.

[0160] Example 2, step 24 above may specifically include at least steps 243-244 below.

[0161] Step 243: Based on the pre-constructed gaze direction correction function, the gaze direction conversion result of the target eyeball, and the center position conversion result of each image acquisition device, determine the gaze direction correction result of the target eyeball so that the gaze direction correction result can represent the gaze direction presented by the target eyeball before refraction, thereby enabling the gaze direction correction result to represent the actual gaze direction of the target eyeball when image acquisition is performed on the target eyeball.

[0162] Among them, the gaze direction correction function is used to correct the gaze direction conversion result of the target eye, such as eliminating refraction error processing.

[0163] In addition, to improve accuracy, the line-of-sight correction function mentioned above can be a function constructed by the least squares method, as shown in formula (2) below.

[0164] g rev =∑ 0≤i,j,k,l≤3 θ ijkl g i K j R k n l +∑ 0≤i,j,k,l≤3 λ ijkl ela i K j R k n l +∑ 0≤i,j,k,l≤3 μ ijkl elb i K j R k n l (2)

[0166] In the formula, g rev This indicates the result of correcting the gaze direction of the target eye; g indicates the result of changing the gaze direction of the target eye; gi Represents g raised to the power of i; K represents the second distance mentioned above; K j K raised to the power of j; R represents the pre-specified corneal radius; R k R is represented by the power of k; n represents the pre-specified refractive index; n l ela represents n raised to the power of l; ela represents the result of the center position transformation of camera A in a stereo camera; ela i Elb represents ela raised to the power of i; elb represents the result of the center position transformation of camera B in the stereo camera setup; elb i θ represents elb raised to the power of i; ijkl Indicates prior to g i K j R k n l The coefficient predicted by this term, θ ijkl In this context, "ijjkl" represents the serial number; λ ijkl Indicates prior targeting of ela i K j R k n l The coefficient predicted by this term, λ ijkl In this context, “ijjkl” is the serial number; μ ijkl Indicates prior targeting of elb i K j R k n l The coefficient predicted by this term, μ ijkl "ijjkl" in the text is the serial number.

[0167] It should be noted that this application does not limit the prediction method of the coefficients in the above paragraph. For example, the coefficients can be obtained by analyzing and processing a large amount of simulation data, such as parameter fitting, so that the coefficients represent the mapping relationship between the actual position described by these simulation data and the corresponding virtual image position. For details regarding the simulation data, please refer to the above text.

[0168] Therefore, in one possible implementation, the aforementioned gaze direction correction function can be determined using the eye parameters of at least one sample eye and the virtual image parameters of each sample eye. This allows the gaze direction correction function to accurately represent the correspondence between the parameters before and after refraction, thus improving accuracy. Specifically, for any sample eye, the eye parameters include the gaze direction of that sample eye, and the virtual image parameters include the gaze direction of the corresponding virtual image of that sample eye.

[0169] Based on the above content regarding the line-of-sight correction function, we can first treat the coefficients in the line-of-sight correction function as unknowns. Then, using the eye parameters of some sample eyes and the virtual image parameters of these sample eyes, we can deduce the coefficients in the line-of-sight correction function. This allows the line-of-sight correction function with these coefficients to accurately represent the correspondence between the parameters before and after refraction. This enables us to subsequently use the line-of-sight correction function with these coefficients to achieve refraction error elimination for the line-of-sight direction.

[0170] Furthermore, this application does not limit the implementation of step 243 above. For example, it can specifically be: substituting the gaze direction conversion result of the target eyeball and the center position conversion result of each image acquisition device into a pre-constructed gaze direction correction function to obtain the gaze direction correction result of the target eyeball, such as g. rev This is to enable the line-of-sight correction result to represent the line-of-sight direction of the target eye before refraction, thereby enabling the line-of-sight correction result to represent the actual line-of-sight direction of the target eye when image acquisition is performed on the target eye.

[0171] Step 244: Based on the correction result of the target eye's gaze direction, determine the eye parameters of the target eye so that the eye parameters include the gaze direction correction result.

[0172] Based on the relevant content of steps 243 to 244 above, for the target eyeball, after obtaining the gaze direction conversion result of the target eyeball, the gaze direction conversion result can be corrected by using a pre-constructed pupil center position correction function, such as removing refraction error processing, to obtain the gaze direction correction result of the target eyeball, so that the gaze direction correction result can represent the gaze direction after removing refraction error, thereby making the gaze direction correction result more accurately represent the gaze direction of the target eyeball, and thus making the eyeball parameters determined based on the gaze direction correction result more accurate.

[0173] Example 4, step 24 above may specifically include at least steps 245-246 below.

[0174] Step 245: Using a pre-constructed pupil radius correction function, the predicted pupil radius of the target eyeball is corrected to obtain the corrected pupil radius of the target eyeball. The pupil radius correction function is determined using the eyeball parameters of at least one sample eyeball and the virtual image parameters of each sample eyeball. For any sample eyeball, the eyeball parameters of the sample eyeball include the pupil radius of the sample eyeball, and the virtual image parameters of the sample eyeball include the pupil radius of the virtual image of the sample eyeball.

[0175] The pupil radius correction function is used to correct the predicted pupil radius of the target eyeball, such as eliminating refraction errors.

[0176] In addition, to improve accuracy, the pupil radius correction function mentioned above can be a function constructed by the least squares method, as shown in formula (3) below.

[0177] r rev =∑ 0≤i,j,k,l≤3 σ ijkl r i K j R k n l (3)

[0178] In the formula, r rev This represents the corrected pupil radius of the target eye; r represents the predicted pupil radius of the target eye. i K represents r raised to the power of i; K represents the second distance mentioned above; K j K raised to the power of j; R represents the pre-specified corneal radius; R k R is represented by the power of k; n represents the pre-specified refractive index; n l σ represents n raised to the power of l; ijkl Indicates prior to r i K j R k n l The coefficient predicted by this term, σ ijkl "ijjkl" in the text is the serial number.

[0179] It should be noted that this application does not limit the prediction method of the coefficients in the above paragraph. For example, the coefficients can be obtained by analyzing and processing a large amount of simulation data, such as parameter fitting, so that the coefficients represent the mapping relationship between the actual position described by these simulation data and the corresponding virtual image position. For details regarding the simulation data, please refer to the above text.

[0180] Therefore, in one possible implementation, the pupil radius correction function mentioned above can be determined using the eyeball parameters of at least one sample eyeball and the virtual image parameters of each sample eyeball. This allows the pupil radius correction function to accurately represent the correspondence between the parameters before and after refraction, thus improving accuracy. Specifically, for any sample eyeball, the eyeball parameters include the pupil radius, and the virtual image parameters include the pupil radius of the corresponding virtual image.

[0181] Based on the above content regarding the pupil radius correction function, we can first treat the coefficients in the pupil radius correction function as unknowns; then, using the eye parameters of some sample eyes and the virtual image parameters of these sample eyes, we can deduce the coefficients in the pupil radius correction function so that the pupil radius correction function with these coefficients can accurately represent the correspondence between the parameters before and after refraction. This will allow us to subsequently use the pupil radius correction function with these coefficients to achieve refraction error elimination for the pupil radius.

[0182] Furthermore, this application does not limit the implementation of step 245 above. For example, it can specifically be: substituting the predicted pupil radius of the target eyeball into a pre-constructed pupil radius correction function to obtain the corrected pupil radius of the target eyeball, such as r rev This is to ensure that the pupil radius correction result can represent the pupil radius of the target eye before refraction, thereby enabling the pupil radius correction result to represent the actual pupil radius of the target eye when the image is acquired.

[0183] Step 246: Based on the pupil radius correction result of the target eyeball, determine the eyeball parameters of the target eyeball so that the eyeball parameters include the pupil radius correction result.

[0184] Based on the relevant content of steps 243 to 244 above, for the target eyeball, after obtaining the pupil radius prediction result of the target eyeball, the pupil radius prediction result can be corrected by using a pre-constructed pupil center position correction function, such as removing refraction error processing, to obtain the corrected pupil radius result of the target eyeball. This allows the corrected pupil radius result to represent the pupil radius after removing refraction error, thereby enabling the corrected pupil radius result to more accurately represent the pupil radius of the target eyeball, and thus making the eyeball parameters determined based on the corrected pupil radius result more accurate.

[0185] Based on the relevant content in S1 to S3 above, the parameter determination method provided in this application first acquires multiple images of the target eyeball, such as two images acquired using a binocular camera, with different acquisition angles for each image. This allows the multiple images to describe the state of the target eyeball at multiple acquisition angles, thus enabling a more comprehensive description of the characteristics of the target eyeball. Then, the predicted pupil edges of each image are back-projected into three-dimensional space to obtain a three-dimensional elliptic cone corresponding to each image. This allows the three-dimensional elliptic cone to represent the state of the eyeball in the corresponding image within three-dimensional space, thereby enabling... These three-dimensional elliptical cones can better represent the state of the target eyeball at multiple acquisition angles in three-dimensional space, thus enabling them to better and more comprehensively represent the characteristics of the target eyeball in three-dimensional space. Then, based on the three-dimensional elliptical cones corresponding to these multiple images, the eyeball parameters of the target eyeball are determined so that these eyeball parameters can more accurately represent the characteristics of the target eyeball, such as the position of the pupil center. This effectively avoids the defects that exist when determining eye parameters based on images from a single acquisition angle, thereby improving the accuracy of eyeball parameters, which in turn improves image rendering effects and ultimately enhances the user experience.

[0186] Research has revealed that, for some scenarios, such as VR scenarios, the distortion in image rendering is directly related to the parameters of the virtual image obtained by observing the eyeball in reverse along the light path from the center of the VR device's screen. This allows for image rendering optimization based on these parameters, such as DDC (Distributed Dynamics Control).

[0187] The study also revealed that the deployment location of the image acquisition devices in the above images on the VR device (e.g., ...) Figure 6 The distance between the location of the binocular camera shown and the center of the VR device's screen is relatively large, so that the virtual eye image described by these images and the virtual eye image observed in reverse along the optical path through the center of the screen (such as...) Figure 6 There are significant differences between the virtual images shown.

[0188] Based on the above two studies, in order to better improve the image rendering optimization effect, this application also provides a method for determining the virtual image parameters of the target eyeball mentioned above. In this method, when the eyeball parameters of the target eyeball include at least the position of the pupil center of the target eyeball, the pupil radius of the target eyeball, and the gaze direction of the target eyeball, the process of determining the virtual image parameters of the target eyeball may include step 31 below.

[0189] Step 31: Determine the virtual image parameters of the target eye based on the eye parameters of the target eye and the parameters of the rendering screen in the target device. The target device is equipped with image acquisition devices. The parameters of the rendering screen include the center position of the rendering screen, which is different from the center position of each image acquisition device. The virtual image parameters are used to describe the virtual image presented by the target eye relative to the center position of the rendering screen. The virtual image parameters include the virtual image position of the center of the pupil of the target eye.

[0190] Among them, the target device refers to the device used to display images to the user's target eyeball in a real application scenario, such as the first lens mentioned above.

[0191] Additionally, for the target device, the rendering screen in the target device (such as...) Figure 6 The screen shown is used to perform image rendering processing so that the user's target eye can view some images through the rendering screen. The parameters of the rendering screen describe its characteristics, such as size and center position; moreover, this application does not limit the implementation of the rendering screen parameters. For example, the rendering screen parameters may at least include the center position of the rendering screen (e.g., the center position of the screen). Figure 6 (The location of the center of the screen shown). Here, the center location of the rendered screen is used to describe the actual location of the center of the rendered screen.

[0192] The virtual image parameters of the target eyeball are used to describe the virtual image obtained by viewing the target eyeball backward along the optical path from the center position of the rendering screen in the target device, such as... Figure 6 The virtual image shown is configured such that the virtual image parameters can represent the virtual image presented by the target eyeball relative to the center position of the rendering screen.

[0193] Furthermore, this application does not limit the implementation of the virtual image parameters of the target eyeball described above. For example, the virtual image parameters may include at least the virtual image position of the pupil center of the target eyeball. Here, the virtual image position refers to the position of the virtual image obtained by observing the pupil center of the target eyeball backwards along the optical path from the center position of the rendering screen in the target device, so that the virtual image position can represent the pupil center position after refraction as observed backwards along the optical path from the center position of the rendering screen.

[0194] Furthermore, this application does not limit the implementation of step 31 above. For example, it can be implemented using a pre-built machine learning model with virtual image parameter determination processing. It should be noted that this application does not limit the implementation of this machine learning model.

[0195] Furthermore, in order to further improve accuracy, this application also provides an implementation of step 31 above. In this implementation, when the eye parameters of the target eyeball also include the position of the pupil edge of the target eyeball, step 31 may specifically include steps 311-313 below.

[0196] Step 311: Based on the position of the pupil edge of the target eyeball, determine the positions of multiple edge points, each of which is located on the pupil edge.

[0197] The position of the pupil edge of the target eyeball is used to describe the actual edge contour of the pupil of the target eyeball, so that the "position of the pupil edge of the target eyeball" can indicate the position of the unrefracted pupil edge.

[0198] The d-th edge point refers to a pixel located at the edge of the pupil of the target eyeball; and the position of the d-th edge point is used to describe the location of the d-th edge point within the pupil edge, where d is a positive integer and d ≤ the number of edge points mentioned above. It should be noted that this application does not limit the implementation method of the number of edge points; for example, to better improve accuracy, the number of edge points may be ≥ 6.

[0199] Furthermore, this application does not limit the method of obtaining the multiple edge points mentioned above. For example, it can be implemented through random sampling. Alternatively, to improve accuracy, the process of obtaining the multiple edge points can be as follows: uniformly sampling the multiple edge points from the edge of the pupil of the target eyeball.

[0200] Based on the relevant content of step 311 above, for the target eyeball, after obtaining the position of the pupil edge of the target eyeball, the position of the pupil edge can be uniformly sampled to obtain the position of multiple edge points, so that these edge points are all located on the pupil edge, thereby enabling these edge points to represent the pupil edge for subsequent processing, which helps to reduce the difficulty of virtual image determination.

[0201] Step 312: For any edge point, determine the virtual image position of the edge point based on the eye parameters of the target eye, the parameters of the rendering screen in the target device, and the position of the edge point.

[0202] Wherein, the virtual image position of the d-th edge point refers to the position of the virtual image obtained by observing the d-th edge point in reverse along the light path from the center position of the rendering screen in the target device, so that the virtual image position can represent the position of the d-th edge point after refraction processing as observed in reverse along the light path from the center position of the rendering screen, where d is a positive integer and d≤ the number of multiple edge points mentioned above.

[0203] Furthermore, this application does not limit the implementation of step 312 above. For example, it can be implemented using a pre-built machine learning model with virtual image position determination processing. It should be noted that this application does not limit the implementation of this machine learning model.

[0204] In addition, to improve accuracy, the process of determining the virtual image position of the d-th edge point mentioned above may include steps 3121-3123 below.

[0205] Step 3121: Based on the parameters of the rendering screen in the target device, determine the parameters of multiple observation devices. For any observation device, the parameters of the observation device include the center position of the observation device. The distance between the center position of each observation device and the center position of the rendering screen is less than a preset distance threshold.

[0206] Among them, multiple observation devices are used to simulate (or realize) reverse observation along the optical path through the center position of the rendering screen in the target device, so that the eyeball observed by the multiple observation devices is as close as possible to the virtual image obtained by reverse observation of the target eyeball through the center position of the rendering screen along the optical path.

[0207] Furthermore, this application does not limit the implementation method of the observation device described above. For example, in some scenarios, the observation device is not present in the target device, so that the observation device is only used to simulate reverse observation along the optical path through the center position of the rendering screen in the target device. Alternatively, in some scenarios, the observation device is present in the target device, so that the observation device is used to realize reverse observation along the optical path through the center position of the rendering screen in the target device.

[0208] The parameters of the q-th observation device are used to describe the characteristics of the q-th observation device, such as the camera center position and camera intrinsic parameters. Furthermore, this application does not limit the implementation method of the parameters of the q-th observation device; for example, the parameters of the q-th observation device may include at least the center position of the q-th observation device (such as the camera center position). The center position of the q-th observation device describes the location of the center of the q-th observation device. q is a positive integer, and q ≤ the number of devices in the plurality of observation devices mentioned above.

[0209] In addition, the above-mentioned multiple observation devices can at least satisfy the following constraints: the distance between the center position of each observation device and the center position of the rendering screen in the target device is less than a preset distance threshold, so that the multiple observation devices can represent the observation device as close as possible to the center position of the rendering screen, thereby making the eyeball observed by the multiple observation devices as close as possible to the virtual image obtained by observing the target eyeball in reverse along the light path through the center position of the rendering screen.

[0210] Furthermore, to further improve accuracy, the aforementioned observation devices should at least meet the following constraints: these observation devices include a main device (such as...) Figure 12 The observation camera 4 shown) and at least one auxiliary device (such as Figure 12 The observation cameras 1-3 and 5-7 shown have their center positions coincided with the center positions of the rendering screen in the target device, and the distance between the center positions of each auxiliary device and the center position of the rendering screen is less than a preset distance threshold. This makes the optical path starting from the center position of the main device the main optical path, and the optical path starting from the center position of the auxiliary device the auxiliary optical path, so that the final determined virtual image position is located on the main optical path.

[0211] Step 3122: For any observation device, based on the eye parameters of the target eye, the parameters of the observation device, and the position of the d-th edge point, construct the refracted light path between the d-th edge point and the observation device, and based on the refracted light path, construct the direct light path corresponding to the observation device at the d-th edge point, with the refracted light path and the direct light path partially overlapping.

[0212] Specifically, for the q-th observation device, the refracted optical path between the d-th edge point and the q-th observation device is used to describe the light refraction state between the d-th edge point and the q-th observation device. Furthermore, this application does not limit the construction method of the refracted optical path. For example, the refracted optical path can be determined based on the law of refraction, the position of the corneal center of the target eye, the corneal refractive index of the target eye, the position of the d-th edge point, and the center position of the q-th observation device, where q is a positive integer and q ≤ the number of devices among the aforementioned multiple observation devices. It should be noted that this application does not limit the implementation method of this determination process. For example, it can be implemented using any existing or future optical path determination method, such as a machine learning model with optical path determination functionality.

[0213] Furthermore, regarding the refracted light path between the d-th edge point and the q-th observation device, the intersection of this refracted light path and the cornea of ​​the target eyeball is the refraction point, so that the refracted light path includes a first segment and a second segment, and the first segment and the second segment are not located on the same straight line. Since one end of the first segment is the center position of the q-th observation device, and the other end of the first segment is the refraction point, the first segment can represent the direct light stage; and since one end of the second segment is the refraction point, and the other end of the second segment is the position of the d-th edge point, the second segment can represent the light refraction stage. Thus, the refracted light path including the first segment and the second segment can represent the actual light propagation path between the d-th edge point and the q-th observation device.

[0214] Furthermore, for the q-th observation device, the direct light path corresponding to the d-th edge point is used to describe the direct light path originating from the center position of the q-th observation device, without considering refraction. This direct light path includes a first segment and a third segment, with the first and third segments lying on the same straight line. Since one end of the third segment is the refraction point, and the third segment lies on the same straight line as the first segment, the third segment represents the path of light continuing to propagate in a straight line after the refraction point. This results in a virtual image appearing on the third segment, observed by the q-th observation device at the d-th edge point.

[0215] Based on the relevant content of step 3122 above, for the d edge points on the pupil edge of the target eyeball, the refracted light path between the d edge point and the q observation device can be deduced first based on the law of refraction, the position of the corneal center of the target eyeball, the refractive index of the cornea of ​​the target eyeball, the position of the d edge point, and the center position of the q observation device, so that the refracted light path can represent the actual light propagation path between the d edge point and the q observation device; then, the segment of the refracted light path in which the light is directly incident is extended into the interior of the target eyeball to obtain the direct light path corresponding to the q observation device under the d edge point, so that the direct light path can produce the virtual image observed by the q observation device for the d edge point, where q is a positive integer and q≤ the number of devices in the above multiple observation devices.

[0216] Step 3123: Based on the intersection of the direct light paths corresponding to the d-th edge point of the multiple observation devices mentioned above, determine the virtual image position of the d-th edge point.

[0217] It should be noted that this application does not limit the implementation of step 3123 above. For example, it can be implemented using any existing or future method that can determine the position of the virtual image based on multiple optical path intersections.

[0218] In addition, to further improve accuracy, step 3123 above may specifically include steps one through three below.

[0219] Step 1: Select multiple reference devices from the observation devices mentioned above (such as...) Figure 12The observation cameras 3-5 shown are configured such that the distance between the center position of each of the plurality of reference devices and the center position of the rendering screen in the target device is not greater than the distance between the center position of the other devices in the plurality of observation devices besides the plurality of reference devices and the center position of the rendering screen, thereby making the plurality of reference devices include the main device and some auxiliary devices closest to the main device.

[0220] Step 2: Based on the direct optical paths corresponding to each of the multiple reference devices mentioned above at the d-th edge point, perform least-squares intersection processing to obtain the target intersection point (e.g., ...). Figure 12 virtual image point shown).

[0221] It should be noted that this application does not limit the implementation method of the least squares intersection process in the above paragraph. For example, it can be specifically as follows: first, obtain the intersection points between the direct optical path corresponding to each of the multiple reference devices at the d-th edge point and the direct optical path corresponding to other reference devices at the d-th edge point; then, perform least squares processing on these intersection points to obtain the target intersection point, so that the sum of the distances from the target intersection point to the other intersection points among these intersection points is minimized, and the target intersection point is located on the main optical path with the center position of the main device as the starting point.

[0222] Step 3: If the target intersection point satisfies the following constraint: the positions of the intersection points between the direct light paths of all the observation devices other than the main device at the d-th edge point and the direct light paths of the main device at the d-th edge point can approach the target intersection point from two directions, then the target intersection point is a reasonable result. Therefore, the target intersection point can be used to represent the virtual image of the d-th edge point in the observation device. Thus, the target intersection point can be used as the virtual image position of the d-th edge point.

[0223] Based on the content of steps one through three above, it is clear that in some scenarios, when the multiple observation devices mentioned above include... Figure 12 When the observation cameras 1-7 are arranged sequentially as shown, after obtaining the direct light path corresponding to each observation camera at the d-th edge point, the least squares intersection of the direct light paths corresponding to the three middle cameras at the d-th edge point can be performed to obtain the target intersection point, as shown below. Figure 12 The virtual image points shown; then, the intersection points between the direct light paths corresponding to the d-th edge points of all observation cameras except observation camera 4 and the direct light paths corresponding to the d-th edge points of observation camera 4 are determined as the verification intersection points, such as... Figure 12The blue intersection point is shown; then, it is determined whether the positions of all verified intersection points approach the target intersection point from two directions. If so, the target intersection point is a reasonable result, and it can be determined that the target intersection point can be used to represent the virtual image of the d-th edge point in the observation device. Therefore, the target intersection point can be used as the virtual image position of the d-th edge point.

[0224] Based on the relevant content of steps 3121 to 3123 above, it can be seen that, for the target eyeball, after obtaining the eyeball parameters of the target eyeball, the virtual image positions of multiple edge points on the pupil edge of the target eyeball can be determined according to the parameters of the rendering screen in the target device, so that the virtual image positions of these edge points can represent the state of the pupil edge of the virtual image corresponding to the target eyeball.

[0225] Step 313: Determine the virtual image parameters of the target eyeball based on the virtual image positions of multiple edge points mentioned above.

[0226] It should be noted that this application does not limit the implementation of step 313 above. For example, it can specifically be: first, ellipse fitting is performed on the virtual image positions of the multiple edge points mentioned above to obtain the pupil edge of the virtual image corresponding to the target eyeball, so that the "pupil edge of the virtual image corresponding to the target eyeball" can represent the pupil edge observed in reverse along the light path through the center position of the rendering screen in the target device; then, based on the "pupil edge of the virtual image corresponding to the target eyeball", the virtual image parameters of the target eyeball, such as the pupil center position of the virtual image corresponding to the target eyeball, are determined.

[0227] Based on the content of step 31 above, it is known that in some scenarios, for the target eyeball, after obtaining the eyeball parameters, the virtual image parameters of the target eyeball can be determined according to the parameters of the rendering screen in the target device, such as... Figure 13 The position of the virtual pupil image is shown so that the virtual image parameter can represent the state of the eyeball that can be observed when the target eyeball is viewed in reverse along the light path through the center position of the rendering screen. This allows for subsequent image rendering optimization processing for the rendering screen based on the virtual image parameter, such as DDC, which is beneficial to improving the image rendering effect.

[0228] Based on the parameter determination method provided in the embodiments of this application, the embodiments of this application also provide a parameter determination device, which is described below in conjunction with... Figure 14 Explanation and clarification will be provided. Among them, Figure 14 This is a schematic diagram of a parameter determination device provided in an embodiment of this application. It should be noted that for technical details of the parameter determination device provided in this embodiment, please refer to the relevant content of the parameter determination method above.

[0229] like Figure 14As shown, the parameter determination device 1400 provided in this application embodiment includes:

[0230] The image acquisition unit 1401 is used to acquire multiple images of the target eyeball, with different acquisition angles for each image;

[0231] The back projection unit 1402 is used to back project the predicted pupil edge of any one of the plurality of images to a three-dimensional space to obtain the three-dimensional elliptic cone corresponding to the image.

[0232] The first determining unit 1403 is used to determine the eyeball parameters of the target eyeball based on the three-dimensional elliptical cones corresponding to the plurality of images.

[0233] In one possible implementation, the first determining unit 1403 is specifically configured to: determine a plurality of circular cross sections from the three-dimensional elliptical cones corresponding to the plurality of images, wherein the plurality of circular cross sections include at most two circular cross sections in each of the three-dimensional elliptical cones, the normals of the different circular cross sections in the at most two circular cross sections are different, and the radii of each of the plurality of circular cross sections are the same; determine at least two circular cross sections from the plurality of circular cross sections based on the normals of each of the plurality of circular cross sections, wherein the three-dimensional elliptical cones in which the different circular cross sections in the at least two circular cross sections are located are different, and the similarity between the normals of any two circular cross sections in the at least two circular cross sections is higher than the similarity between the normals of any two other circular cross sections in the plurality of circular cross sections besides the at least two circular cross sections; and determine the eyeball parameters of the target eyeball based on the at least two circular cross sections.

[0234] In one possible implementation, for any one of the plurality of images, the vertex position of the three-dimensional elliptical cone corresponding to that image is determined based on the center position of the image acquisition device;

[0235] The first determining unit 1403 is specifically used for: for any one of the at least two circular cross-sections, determining the straight line corresponding to the circular cross-section based on the line connecting the center position of the circular cross-section and the vertex position of the three-dimensional elliptical cone where the circular cross-section is located; and determining the eyeball parameters of the target eyeball based on the intersection point between the straight lines corresponding to the at least two circular cross-sections.

[0236] In one possible implementation, for any one of the plurality of images, the corresponding three-dimensional elliptical cone is determined based on the parameters of the image acquisition device, including the center position of the acquisition device.

[0237] In one possible implementation, the first determining unit 1403 is specifically configured to: determine the prediction parameters of the target eyeball based on the three-dimensional elliptic cones corresponding to the plurality of images, wherein the prediction parameters are used to describe the state of the target eyeball in the plurality of images; construct an eyeball coordinate system based on the prediction parameters; transform the prediction parameters and the center position of the acquisition device of each of the images to the eyeball coordinate system to obtain a transformation result; and determine the eyeball parameters of the target eyeball based on the transformation result.

[0238] In one possible implementation, the prediction parameters include the predicted position of the pupil center of the target eyeball and the predicted gaze direction of the target eyeball.

[0239] The eye coordinate system satisfies at least one of the following constraints: the origin of the eye coordinate system is determined based on the predicted position of the corneal center of the target eye, which is determined based on the predicted position of the pupil center of the target eye and the predicted gaze direction of the target eye; the first coordinate axis of the eye coordinate system is determined based on the predicted gaze direction of the target eye; the second coordinate axis of the eye coordinate system is determined based on the normal to the target plane, which is determined based on the predicted position of the corneal center of the target eye, the predicted gaze direction of the target eye, and the center position of the target image acquisition device, and the plurality of images include the target image; the third coordinate axis of the eye coordinate system is determined based on the vector product of the first coordinate axis and the second coordinate axis.

[0240] In one possible implementation, the prediction parameters include the predicted position of the pupil center of the target eyeball and the predicted gaze direction of the target eyeball.

[0241] The first determining unit 1403 is specifically used for: converting the pupil center position prediction result, the gaze direction prediction result, and the center position of each image acquisition device to the eye coordinate system to obtain the pupil center position conversion result, the gaze direction conversion result, and the center position conversion result of each image acquisition device; determining the pupil center position correction result based on a pre-constructed pupil center position correction function, the pupil center position conversion result, the gaze direction conversion result, and the center position conversion result of each image acquisition device; and determining the eye parameters of the target eye based on the pupil center position correction result.

[0242] In one possible implementation, the pupil center position correction function is determined using eyeball parameters of at least one sample eyeball and virtual image parameters of each sample eyeball. For any sample eyeball, the eyeball parameters of the sample eyeball include the position of the pupil center of the sample eyeball, and the virtual image parameters of the sample eyeball include the virtual image position of the pupil center of the sample eyeball.

[0243] In one possible implementation, the prediction parameters include the predicted gaze direction of the target eye.

[0244] The first determining unit 1403 is specifically used for: converting the gaze direction prediction result and the center position of each image acquisition device to the eye coordinate system to obtain the gaze direction conversion result and the center position conversion result of each image acquisition device; determining the gaze direction correction result of the target eye based on the pre-constructed gaze direction correction function, the gaze direction conversion result, and the center position conversion result of each image acquisition device; and determining the eye parameters of the target eye based on the gaze direction correction result of the target eye.

[0245] In one possible implementation, the gaze direction correction function is determined using the eye parameters of at least one sample eye and the virtual image parameters of each sample eye. For any sample eye, the eye parameters of the sample eye include the gaze direction of the sample eye, and the virtual image parameters of the sample eye include the gaze direction of the virtual image corresponding to the sample eye.

[0246] In one possible implementation, the prediction parameters include the predicted pupil radius of the target eyeball;

[0247] The first determining unit 1403 is specifically used to: correct the pupil radius prediction result using a pre-constructed pupil radius correction function to obtain the corrected pupil radius result of the target eyeball. The pupil radius correction function is determined using the eyeball parameters of at least one sample eyeball and the virtual image parameters of each sample eyeball. For any sample eyeball, the eyeball parameters of the sample eyeball include the pupil radius of the sample eyeball, and the virtual image parameters of the sample eyeball include the pupil radius of the virtual image corresponding to the sample eyeball. Based on the corrected pupil radius result of the target eyeball, the eyeball parameters of the target eyeball are determined.

[0248] In one possible implementation, the eye parameters include the position of the pupil center of the target eye, the pupil radius of the target eye, and the line of sight of the target eye;

[0249] The parameter determining device 1400 further includes:

[0250] The second determining unit is used to determine the virtual image parameters of the target eyeball based on the eyeball parameters and the parameters of the rendering screen in the target device. The target device is equipped with image acquisition devices for each of the images. The parameters of the rendering screen include the center position of the rendering screen, which is different from the center position of each image acquisition device. The virtual image parameters are used to describe the virtual image presented by the target eyeball relative to the center position of the rendering screen. The virtual image parameters include the virtual image position of the center of the pupil of the target eyeball.

[0251] In one possible implementation, the eyeball parameters of the target eyeball also include the position of the pupil edge of the target eyeball;

[0252] The second determining unit is specifically used for: determining the positions of multiple edge points based on the position of the pupil edge, wherein each edge point is located on the pupil edge; for any edge point, determining the virtual image position of the edge point based on the eye parameters, the parameters of the rendering screen, and the position of the edge point; and determining the virtual image parameters of the target eyeball based on the virtual image positions of the multiple edge points.

[0253] In one possible implementation, the process of determining the virtual image position of any edge point includes: determining parameters of multiple observation devices based on parameters of the rendering screen; for any observation device, the parameters of the observation device include the center position of the observation device, and the distance between the center position of each observation device and the center position of the rendering screen is less than a preset distance threshold; for any observation device, constructing a refracted light path between the edge point and the observation device based on the eye parameters, the parameters of the observation device, and the position of the edge point, and constructing a direct light path corresponding to the observation device at the edge point based on the refracted light path, wherein the refracted light path and the direct light path partially overlap; and determining the virtual image position of the edge point based on the intersection of the direct light paths corresponding to the multiple observation devices at the edge point.

[0254] In one possible implementation, the acquisition times of the different images are the same.

[0255] In one possible implementation, the plurality of images are captured using a binocular camera targeting the target eye.

[0256] Based on the aforementioned content regarding the parameter determination device 1400, the working principle of the parameter determination device 1400 provided in this application includes: firstly, acquiring multiple images of the target eyeball, such as two images acquired using a binocular camera, with different acquisition angles for each image, so that the multiple images can describe the state of the target eyeball at multiple acquisition angles, thereby enabling the multiple images to more comprehensively describe the characteristics of the target eyeball; then, back-projecting the predicted pupil edge of each image onto three-dimensional space to obtain a three-dimensional elliptic cone corresponding to each image, so that the three-dimensional elliptic cone corresponding to each image can respectively represent the state of the eyeball in three-dimensional space in the corresponding image. This allows these 3D elliptical cones to better represent the state of the target eyeball at multiple acquisition angles in 3D space, thus enabling them to better and more comprehensively represent the characteristics of the target eyeball in 3D space. Then, based on the 3D elliptical cones corresponding to these multiple images, the eyeball parameters of the target eyeball are determined so that these eyeball parameters can more accurately represent the characteristics of the target eyeball, such as the position of the pupil center. This effectively avoids the defects that exist when determining eye parameters based on images from a single acquisition angle, which helps improve the accuracy of eyeball parameters, thereby improving image rendering effects and ultimately improving the user experience.

[0257] In addition, this application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory so that the electronic device performs any implementation of the parameter determination method provided in this application.

[0258] See Figure 15 The diagram illustrates a structural schematic of an electronic device 1500 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 15 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0259] like Figure 15As shown, electronic device 1500 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1502 or a program loaded from storage device 1508 into random access memory (RAM) 1503. RAM 1503 also stores various programs and data required for the operation of electronic device 1500. Processing device 1501, ROM 1502, and RAM 1503 are interconnected via bus 1504. Input / output (I / O) interface 1505 is also connected to bus 1504.

[0260] Typically, the following devices can be connected to I / O interface 1505: input devices 1506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1509. Communication device 1509 allows electronic device 1500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 15 An electronic device 1500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0261] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1509, or installed from storage device 1508, or installed from ROM 1502. When the computer program is executed by processing device 1501, it performs the functions defined in the methods of embodiments of this disclosure.

[0262] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0263] This application also provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform any implementation of the parameter determination method provided in this application.

[0264] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0265] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0266] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0267] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0268] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0269] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0270] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0271] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0272] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0273] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0274] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0275] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0276] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0277] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A parameter determination method characterized by, The method comprises: acquiring a plurality of images collected for a target eyeball, different images having different collection angles; for any image in the plurality of images, projecting a predicted pupil edge of the image back to a three-dimensional space to obtain a three-dimensional elliptical cone corresponding to the image; determining eyeball parameters of the target eyeball according to the three-dimensional elliptical cones corresponding to the plurality of images.

2. The method of claim 1, wherein, The process of determining the eyeball parameters of the target eyeball comprises: determining a plurality of circular sections from the three-dimensional elliptical cones corresponding to the plurality of images, the plurality of circular sections comprising at most two circular sections in each of the three-dimensional elliptical cones, normals of different circular sections in the at most two circular sections being different, and radii of each of the circular sections in the plurality of circular sections being the same; determining at least two circular sections from the plurality of circular sections according to the normals of each of the circular sections in the plurality of circular sections, different circular sections in the at least two circular sections being in different three-dimensional elliptical cones, and a similarity between normals of any two circular sections in the at least two circular sections being higher than a similarity between normals of any two circular sections other than the at least two circular sections in the plurality of circular sections; determining the eyeball parameters of the target eyeball according to the at least two circular sections.

3. The method of claim 2, wherein, For any image in the plurality of images, a vertex position of the three-dimensional elliptical cone corresponding to the image is determined according to a center position of a collection device of the image. The method further comprises: for any circular section in the at least two circular sections, determining a straight line corresponding to the circular section according to a line connecting a center position of the circular section and a vertex position of the three-dimensional elliptical cone in which the circular section is located; the determining of the eyeball parameters of the target eyeball according to the at least two circular sections comprises: determining the eyeball parameters of the target eyeball according to an intersection between the straight lines corresponding to the at least two circular sections.

4. The method of claim 1, wherein, For any image in the plurality of images, the three-dimensional elliptical cone corresponding to the image is determined according to parameters of a collection device of the image, the parameters of the collection device comprising the center position of the collection device.

5. The method of claim 1, wherein, The process of determining the eyeball parameters of the target eyeball comprises: determining predicted parameters of the target eyeball according to the three-dimensional elliptical cones corresponding to the plurality of images, the predicted parameters being used to describe states of the target eyeball in the plurality of images; constructing an eyeball coordinate system according to the predicted parameters; converting the predicted parameters and the center positions of the collection devices of the images into the eyeball coordinate system to obtain a conversion result; determining the eyeball parameters of the target eyeball according to the conversion result.

6. The method of claim 5, wherein, The predicted parameters comprise a position prediction result of a pupil center of the target eyeball and a gaze direction prediction result of the target eyeball; The eyeball coordinate system satisfies at least one constraint: an origin of the eyeball coordinate system is determined according to a position prediction result of a corneal center of the target eyeball, the position prediction result of the corneal center being determined according to the position prediction result of the pupil center of the target eyeball and the gaze direction prediction result of the target eyeball. The first coordinate axis in the eye coordinate system is determined according to a gaze direction prediction result of the target eye; The second coordinate axis in the eye coordinate system is determined according to a normal line of a target plane, the target plane being determined according to a corneal center position prediction result of the target eye, the gaze direction prediction result of the target eye, and a center position of an image acquisition device of the target image, the plurality of images including the target image; The third coordinate axis in the eye coordinate system is determined according to a vector product of the first coordinate axis and the second coordinate axis.

7. The method of claim 5, wherein, The prediction parameters include a corneal center position prediction result of the target eye and a gaze direction prediction result of the target eye; The conversion of the prediction parameters and the center positions of the image acquisition devices of the images into the eye coordinate system includes: The conversion of the corneal center position prediction result, the gaze direction prediction result, and the center positions of the image acquisition devices of the images into the eye coordinate system includes conversion of the corneal center position prediction result, the gaze direction prediction result, and the center positions of the image acquisition devices of the images into the eye coordinate system to obtain a corneal center position conversion result, a gaze direction conversion result, and center position conversion results of the image acquisition devices of the images; The determination of the eye parameters of the target eye includes: The determination of the gaze direction correction result of the target eye according to the gaze direction correction function and the gaze direction conversion result and the center position conversion results of the image acquisition devices of the images; The gaze direction correction function is determined by using eye parameters of at least one sample eye and virtual image parameters of the sample eyes, the eye parameters of any sample eye including a gaze direction of the sample eye, and the virtual image parameters of the sample eye including a gaze direction of a virtual image corresponding to the sample eye.

8. The method of claim 7, wherein, The prediction parameters include a gaze direction prediction result of the target eye; 9. The method of claim 5, wherein, The conversion of the prediction parameters and the center positions of the image acquisition devices of the images into the eye coordinate system includes: The conversion of the gaze direction prediction result and the center positions of the image acquisition devices of the images into the eye coordinate system includes conversion of the gaze direction prediction result and the center positions of the image acquisition devices of the images into the eye coordinate system to obtain a gaze direction conversion result and center position conversion results of the image acquisition devices of the images; The determination of the eye parameters of the target eye includes: The determination of the gaze direction correction result of the target eye according to the gaze direction correction function and the gaze direction conversion result and the center position conversion results of the image acquisition devices of the images; The gaze direction correction function is determined by using eye parameters of at least one sample eye and virtual image parameters of the sample eyes, the eye parameters of any sample eye including a gaze direction of the sample eye, and the virtual image parameters of the sample eye including a gaze direction of a virtual image corresponding to the sample eye. ​ 10. The method of claim 9, wherein, ​ 11. The method of claim 5, wherein, The prediction parameter comprises a pupil radius prediction result of the target eyeball; The determination process of the eyeball parameter of the target eyeball comprises: The pupil radius prediction result is corrected by using a pre-constructed pupil radius correction function to obtain a pupil radius correction result of the target eyeball, wherein the pupil radius correction function is determined by using eyeball parameters of at least one sample eyeball and virtual image parameters of each sample eyeball, and for any sample eyeball, the eyeball parameter of the sample eyeball comprises a pupil radius of the sample eyeball, and the virtual image parameter of the sample eyeball comprises a pupil radius of a virtual image corresponding to the sample eyeball; The eyeball parameter of the target eyeball is determined according to the pupil radius correction result of the target eyeball.

12. The method of claim 1, wherein, The eyeball parameter comprises a position of a pupil center of the target eyeball, a pupil radius of the target eyeball, and a line-of-sight direction of the target eyeball; The method further comprises: The virtual image parameter of the target eyeball is determined according to the eyeball parameter and a parameter of a rendering screen of a target device, wherein the target device is provided with an image acquisition device for each image, the parameter of the rendering screen comprises a center position of the rendering screen, the center position of the rendering screen is different from a center position of the image acquisition device, the virtual image parameter is used to describe a virtual image presented by the target eyeball relative to the center position of the rendering screen, and the virtual image parameter comprises a virtual image position of the pupil center of the target eyeball.

13. The method of claim 12, wherein, The eyeball parameter of the target eyeball further comprises a position of a pupil edge of the target eyeball. The determination process of the virtual image parameter comprises: The positions of a plurality of edge points are determined according to the position of the pupil edge, and each edge point is located on the pupil edge. For any edge point, a virtual image position of the edge point is determined according to the eyeball parameter, the parameter of the rendering screen, and the position of the edge point. The virtual image parameter of the target eyeball is determined according to the virtual image positions of the plurality of edge points.

14. The method of claim 13, wherein, For any edge point, the determination process of the virtual image position of the edge point comprises: The parameters of a plurality of observation devices are determined according to the parameter of the rendering screen, and for any observation device, the parameter of the observation device comprises a center position of the observation device, and the distance between the center position of each observation device and the center position of the rendering screen is less than a preset distance threshold. For any observation device, a refractive light path between the edge point and the observation device is constructed according to the eyeball parameter, the parameter of the observation device, and the position of the edge point, a direct light path corresponding to the observation device at the edge point is constructed according to the refractive light path, and the refractive light path and the direct light path partially overlap. The virtual image position of the edge point is determined according to the intersection point between the direct light paths corresponding to the plurality of observation devices at the edge point.

15. The method according to any one of claims 1 to 14, characterized in that, The acquisition time instants of different images are the same. And / or, The plurality of images are acquired by using a binocular camera for the target eyeball.

16. A parameter determination apparatus characterized by comprising: It comprises: An image acquisition unit is configured to acquire a plurality of images acquired for a target eyeball, and different images are acquired at different angles. A back projection unit is configured to, for any image of the plurality of images, back project a predicted pupil edge of the image to a three-dimensional space to obtain a three-dimensional elliptical cone corresponding to the image; A first determination unit is configured to determine an eye parameter of the target eye according to the three-dimensional elliptical cones corresponding to the plurality of images.

17. An electronic device, comprising: The device comprises a processor and a memory; The memory is configured to store instructions or a computer program; The processor is configured to execute the instructions or the computer program in the memory, so that the electronic device executes the method in any one of claims 1-15.

18. A computer readable medium characterized by The computer readable medium stores instructions or a computer program, when the instructions or the computer program run on a device, cause the device to execute the method in any one of claims 1-15.

19. A computer program product, characterised in that, It comprises a computer program carried on a non-transitory computer readable medium, and the computer program comprises program codes for executing the method in any one of claims 1-15.

Images