Pupil center and pupil determination method based on wearable device and related device

Abstract

The invention provides a pupil center determination method based on wearable equipment and related equipment, and the wearable equipment comprises a first camera and a second camera which are used for collecting eye images of the same eye. The method comprises the following steps: acquiring a first eye image acquired by the first camera and a second eye image acquired by the second camera; respectively determining a first pixel point and a second pixel point corresponding to the pupil center from the first eye image and the second eye image; determining a first target camera parameter of the first camera according to the position of the first pixel point in the first eye image; determining a second target camera parameter of the second camera according to the position of the second pixel point in the second eye image; and determining the spatial position of the pupil center according to the first target camera parameter and the second target camera parameter.

Description

Technical Field

[0001] The present disclosure relates to the field of extended reality technologies, and in particular, to a method for determining a pupil center and a pupil based on a wearable device and related devices. Background Art

[0002] Extended Reality (XR) refers to combining the real and the virtual through a computer to create a virtual environment for human-computer interaction. XR (Extended Reality) technology can further include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), which use hardware devices combined with various technical means to integrate virtual content with the real scene.

[0003] Generally, an extended reality system provides a wearable device for a user to achieve human-computer interaction, and the wearable device can be a head-mounted wearable device. In some scenarios, the wearable device can calculate by collecting eye images to implement functions such as gaze tracking or interpupillary distance estimation.

[0004] However, the inventors of the present disclosure have found that there are certain limitations in the method for calculating the pupil center in the related art. Summary of the Invention

[0005] The present disclosure provides a method for determining a pupil center and a pupil based on a wearable device and related devices to solve or partially solve the above problems.

[0006] In a first aspect of the present disclosure, there is provided a method for determining a pupil center based on a wearable device, where the wearable device includes a first camera and a second camera for collecting eye images of the same eye, and the method includes:

[0007] Obtaining a first eye image collected by the first camera and a second eye image collected by the second camera;

[0008] Respectively determining a first pixel point corresponding to the pupil center and a second pixel point corresponding to the pupil center from the first eye image and the second eye image;

[0009] Determining a first target camera parameter of the first camera according to the position of the first pixel point in the first eye image, where the first target camera parameter includes a first spatial projection direction of the first pixel point;

[0010] Determining a second target camera parameter of the second camera according to the position of the second pixel point in the second eye image, where the second target camera parameter includes a second spatial projection direction of the second pixel point;

[0011] Determine the spatial position of the pupil center according to the first target camera parameters and the second target camera parameters.

[0012] In a second aspect of the present disclosure, a method for determining interpupillary distance based on a wearable device is provided. The wearable device includes a first target camera and a second target camera for collecting eye images of a first eye, and a third target camera and a fourth target camera for collecting eye images of a second eye. The method includes:

[0013] Obtain the first spatial position of the first pupil center of the first eye and the second spatial position of the second pupil center of the second eye determined by the method described in the first aspect;

[0014] Determine the interpupillary distance according to the first spatial position and the second spatial position.

[0015] In a third aspect of the present disclosure, a device for determining the pupil center based on a wearable device is provided. The wearable device includes a first camera and a second camera for collecting eye images of the same eye. The device includes:

[0016] An acquisition module configured to: acquire a first eye image acquired by the first camera and a second eye image acquired by the second camera;

[0017] A first determination module configured to: respectively determine a first pixel point and a second pixel point corresponding to the pupil center from the first eye image and the second eye image;

[0018] A second determination module configured to: determine first target camera parameters of the first camera according to the position of the first pixel point in the first eye image, where the first target camera parameters include a first spatial projection direction of the first pixel point;

[0019] A third determination module configured to: determine second target camera parameters of the second camera according to the position of the second pixel point in the second eye image, where the second target camera parameters include a second spatial projection direction of the second pixel point;

[0020] A fourth determination module configured to: determine the spatial position of the pupil center according to the first target camera parameters and the second target camera parameters.

[0021] In a fourth aspect of the present disclosure, a device for determining interpupillary distance based on a wearable device is provided. The wearable device includes a first target camera and a second target camera for collecting eye images of a first eye, and a third target camera and a fourth target camera for collecting eye images of a second eye. The device includes:

[0022] An acquisition module, configured to: acquire a first spatial position of a first pupil center of the first eye and a second spatial position of a second pupil center of the second eye determined by the method according to the first aspect;

[0023] A determination module, configured to: determine an interpupillary distance according to the first spatial position and the second spatial position.

[0024] In a fifth aspect of the present disclosure, there is provided a computer device, including one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to the first aspect or the second aspect.

[0025] In a sixth aspect of the present disclosure, there is provided a non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to execute the method according to the first aspect or the second aspect.

[0026] In a seventh aspect of the present disclosure, there is provided a computer program product, including computer program instructions, which, when running on a computer, cause the computer to execute the method according to the first aspect or the second aspect.

[0027] The method for determining a pupil center and a pupil based on a wearable device and related devices provided by the embodiments of the present disclosure can accurately calculate the pupil center without using the pupil corneal reflection method by using images collected by at least two cameras provided in the wearable device for collecting eye images of the same eye. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following will briefly introduce the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings in the following description are only the embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0029] Figure 1A The schematic diagram of an exemplary system provided by the embodiments of the present disclosure is shown.

[0030] Figure 1B The schematic diagram of an exemplary head-mounted wearable device is shown.

[0031] Figure 2 The schematic diagram of an exemplary wearable device provided by the embodiments of the present disclosure is shown.

[0032] Figure 3AShows a schematic diagram of another exemplary wearable device provided by an embodiment of the present disclosure.

[0033] Figure 3B Shows a schematic diagram of yet another exemplary wearable device provided by an embodiment of the present disclosure.

[0034] Figure 3C Shows a schematic diagram of exemplary camera parameters according to an embodiment of the present disclosure.

[0035] Figure 4A Shows a schematic flowchart of an exemplary method provided by an embodiment of the present disclosure.

[0036] Figure 4B Shows a schematic flowchart of an exemplary method for determining the spatial position of the pupil center according to an embodiment of the present disclosure.

[0037] Figure 4C Shows a partial schematic flowchart of an exemplary method provided by an embodiment of the present disclosure.

[0038] Figure 4D Shows a schematic flowchart of an exemplary method for determining correction parameters according to an embodiment of the present disclosure.

[0039] Figure 4E Shows a partial schematic flowchart of an exemplary method provided by an embodiment of the present disclosure.

[0040] Figure 4F Shows a schematic flowchart of another exemplary method provided by an embodiment of the present disclosure.

[0041] Figure 5A Shows a schematic diagram of an exemplary first eye image according to an embodiment of the present disclosure.

[0042] Figure 5B Shows a schematic diagram of an exemplary second eye image according to an embodiment of the present disclosure.

[0043] Figure 5C Shows a schematic diagram of a calculation model for determining the pupil center based on camera parameters according to an embodiment of the present disclosure.

[0044] Figure 5D Shows a schematic diagram of an exemplary simulation model according to an embodiment of the present disclosure.

[0045] Figure 5E Shows a schematic diagram of a first target image according to an embodiment of the present disclosure.

[0046] Figure 5F Shows a schematic diagram of a second target image according to an embodiment of the present disclosure.

[0047] Figure 6Shows a schematic diagram of the hardware structure of an exemplary computer device provided by an embodiment of the present disclosure.

[0048] Figure 7 Shows a schematic diagram of an exemplary device provided by an embodiment of the present disclosure.

[0049] Figure 8 Shows a schematic diagram of another exemplary device provided by an embodiment of the present disclosure. Detailed implementation manners

[0050] To make the objectives, technical solutions and advantages of the present disclosure more clear and understandable, the following further describes the present disclosure in detail with reference to specific embodiments and the accompanying drawings.

[0051] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the ordinary meanings understood by those of ordinary skill in the field to which the present disclosure belongs. The "first", "second" and similar terms used in the embodiments of the present disclosure do not denote any order, quantity or importance, but are only used to distinguish different components. The terms such as "include" or "comprise" mean that the elements or objects appearing before this term cover the elements or objects listed after this term and their equivalents, without excluding other elements or objects. The terms such as "connect" or "couple" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left" and "right" are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0052] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0053] For example, when receiving the user's active request, a prompt message is sent to the user to clearly prompt the user that the operation requested to be executed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server or a storage medium that executes the operation of the technical solution of the present disclosure according to the prompt message.

[0054] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving the user's active request may be, for example, in the form of a pop-up window, and the prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0055] It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not limit the implementation manner of the present disclosure. Other manners that comply with relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0056] It is understandable that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of corresponding laws, regulations and related provisions.

[0057] Figure 1A FIG. shows a schematic diagram of an exemplary extended reality system 100 provided by an embodiment of the present disclosure.

[0058] Extended Reality (XR for short) refers to combining the real and the virtual through a computer to create a virtual environment for human-computer interaction. XR (Extended Reality) technology can further include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), which use hardware devices combined with various technical means to integrate virtual content with real scenes.

[0059] As Figure 1A shown, the system 100 may include various types of wearable devices. For example, a head-mounted wearable device (such as a VR / AR glasses or a Head-Mounted Display (HMD)) 104, an operation handle 108, etc. In some scenarios, a camera / camera 110 for taking pictures of the operator (user) 102 may also be provided. In some embodiments, when the foregoing devices do not have a processing function or when necessary, the system 100 may further include an external control device 112 for providing a processing function. The control device 112 may be, for example, a computer device such as a mobile phone or a computer. In some embodiments, when any one of the foregoing devices serves as a control device or a main control device, information interaction with other devices in the system 100 may be achieved through wired or wireless communication means.

[0060] In the system 100, the user 102 may use the head-mounted wearable device 104 and the operation handle 108 to interact with the extended reality system 100. In some scenarios, the system 100 may use the images captured by the camera / camera 110 to recognize the posture, gestures, etc. of the user 102, and then complete the interaction with the user 102 based on the recognized posture and gestures. In some embodiments, the user 130 may also implement gesture input with bare hands. The head-mounted wearable device 104 may collect the front image in real time through a camera or a camera provided in front of the head-mounted wearable device 104, and recognize the gestures of the user 130 by recognizing the image.

[0061] In some embodiments, asFigure 1A As shown, the system 100 can also communicate with the server 114 and obtain data from the server 114, such as pictures, audio, video, etc., and can output these data through the head-mounted wearable device 104. For example, display pictures or videos on the display screen of the head-mounted wearable device 104, play the audio carried by the audio and video using the speaker of the head-mounted wearable device 104, and so on. In some embodiments, as Figure 1A shown, the server 114 can retrieve the required data from the database server 116 for storing data, such as pictures, audio, video, etc.

[0062] In some embodiments, a collection unit for collecting information can be provided on the head-mounted wearable device 104. The types of the collection unit can be various.

[0063] In some embodiments, the collection unit can further include an environment acquisition unit and a positioning and tracking unit. The environment acquisition unit can be used to collect the environmental information around the wearable device 104 (e.g., in front), and the positioning and tracking unit can be used to position and track the wearable device 104. Optionally, the environment acquisition unit can include, but is not limited to, photosensitive elements such as a three-color camera (e.g., RGB camera), a depth camera, a binocular camera, a laser, etc., and the positioning and tracking unit can include, but is not limited to, modules such as visual simultaneous localization and mapping (Visual SLAM), an inertial measurement unit (IMU), a global positioning system (GPS), an ultra-wideband wireless communication technology (UWB), a laser, etc.

[0064] In some embodiments, the head-mounted wearable device 104 can also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (e.g., a gyroscope), etc. for collecting the speed information or acceleration information of the head-mounted wearable device 104. Also, for example, the operation handle 108 can also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (e.g., a gyroscope), etc. for collecting the speed information or acceleration information of the operation handle 108. It should be noted that, in addition to being provided on the head-mounted wearable device 104 and the operation handle 108, the aforementioned collection unit can also be directly attached to the body part of the interacting user 102 without relying on hardware devices, so as to collect the relevant information of this body part, such as speed or acceleration or angular velocity information, or the information collected by other sensors or collection units (e.g., eye images (including pupil images), etc.).

[0065] In some embodiments, the head-mounted wearable device 104 can also be provided with a camera or a camera for taking pictures of the operator (user) 102 (e.g., pictures of hands or feet) and environmental images.

[0066] In some embodiments, the system 100 can identify the posture and gestures of the user 102 through the collected information, and then can perform corresponding interactions based on the identified user posture and gestures.

[0067] Figure 1B A schematic diagram of an exemplary head-mounted wearable device 104 is shown.

[0068] like Figure 1B As shown, the head-mounted wearable device 104 may include a lens barrel 1042, and a display screen 1044 for displaying images and an optical component 1046 for processing the light path may be arranged inside the lens barrel 1042. Optionally, the optical component 1046 may further include a plurality of lenses (e.g., lenses 1046A and 1046B), and the combination of the plurality of lenses may project the light emitted by the display screen 1044 into the eye 1022, so that the eye 1022 may view the image displayed on the display screen 1044. It is understood that Figure 1B Only a single-side structure of the head-mounted wearable device 104 is exemplarily shown. In order to achieve binocular display, the head-mounted wearable device 104 may include two lens barrel structures arranged in parallel.

[0069] In some embodiments, Figure 1B As shown, the head-mounted wearable device 104 may also be provided with a camera 1048 for collecting eye images, and the camera 1048 may be a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, or the like.

[0070] Optionally, the camera 1048 may be an eye tracking (ET) camera, and the eye images collected by the camera may be used to implement functions such as pupil distance estimation and eye tracking.

[0071] like Figure 1B As shown, in the related art, the camera 1048 is usually arranged outside the lens barrel and usually only one camera is arranged corresponding to each lens barrel. In addition, in order to better capture the complete eye image and not affect the eye's observation of the screen 1022, the common deployment position of the camera 1048 is generally at the outer corner of the eye or the nose wing. Figure 1B As shown, if the camera 1048 is close to the outside of the device, then Figure 1B The camera deployment position shown is the outer corner of the eye. If the camera 1048 is close to the inside of the device, then Figure 1B The camera deployment position shown is the nose wing position.

[0072] However, the inventors of the present disclosure have discovered that the method of installing the camera in the related art easily causes the camera 1048 to be installed at a large inclination angle relative to the eye 1022, resulting in a large angle γ between the orientation of the camera 1048 and the direct viewing direction of the eye 1022, making it difficult for the collected eye image to reflect the image at the direct viewing angle of the eye.

[0073] In some cases, the user may need to wear glasses before using the head-mounted wearable device 104. However, since the camera 1048 is disposed outside the lens barrel 1042, the camera 1048 is higher than the lens barrel 1042, which may easily squeeze the glasses and affect the wearing comfort of the head-mounted wearable device 104. At the same time, the imaging of the camera 1048 is easily affected by the edge of the glasses. The light refracts through the edge of the glasses, which reduces the clarity of the image of the camera 1048 and forms many refracted light spots in the image, thereby affecting the accuracy of the subsequent algorithm.

[0074] In view of this, an embodiment of the present disclosure provides a wearable device, which can solve or partially solve the above-mentioned problems to a certain extent by setting a camera inside the lens barrel.

[0075] Figure 2 A schematic diagram of an exemplary wearable device 200 provided by an embodiment of the present disclosure is shown.

[0076] like Figure 2 As shown, similarly, the wearable device 200 may also include a lens barrel 202, a display screen 204 and an optical component 206 disposed inside the lens barrel 202. The optical component 206 may further include a plurality of lenses (e.g., lenses 206A and 206B), and the combination of the plurality of lenses may project light emitted by the display screen 204 into the eye 1022, so that the eye 1022 may view the image displayed on the display screen 204.

[0077] and Figure 1B The difference between the wearable device 104 shown is that the camera 208 of the wearable device 200 is arranged inside the lens barrel 202 and faces the light-emitting side of the lens barrel. Since the camera 208 is placed inside the lens barrel 202, it will not affect the wearing of glasses, thereby improving the comfort of the wearable device 200. Figure 2As shown, since the camera 208 is disposed inside the lens barrel 202, the distance from the camera 208 to the eye 1022 is extended, causing the mounting inclination angle of the camera 208 relative to the eye 1022 to become smaller. Consequently, the angle β between the orientation of the camera 208 and the frontal direction of the eye 1022 is reduced, enabling the camera 208 to have a better observation angle. The captured eye image can better reflect the image at the frontal angle of the eye, resulting in better imaging quality. Additionally, since the camera 208 is placed inside the lens barrel 202, the glasses do not interfere with the imaging of the camera 208, further improving the imaging quality. The improvement in imaging quality enhances the accuracy of algorithms such as pupil distance estimation or gaze tracking.

[0078] In some embodiments, as Figure 2 shown, the wearable device 200 may further include a set point light source 212. The light emitted by the point light source 212 can form a reflected light on the cornea of the eye 1022 and thus be captured by the camera 208. Subsequently, a light spot is formed in the image obtained by the imaging of the camera 208. Based on the position of the light spot in the image, it can be used to assist in positioning the center of the cornea and the position of the pupil. In some embodiments, to make the positioning more accurate, the number of point light sources 212 can be two or more. Optionally, the point light source 212 can be a light-emitting diode (LED). In some embodiments, the light emitted by the point light source 212 can be non-visible light (e.g., infrared light) or visible light with relatively low brightness, thereby improving the user's comfort.

[0079] It can be understood that Figure 2 only the unilateral structure of the wearable device 200 is exemplarily shown in []. To achieve binocular display, the wearable device 200 may include two lens barrel structures arranged side by side.

[0080] For a wearable device that has completed camera parameter calibration, during use, based on the images captured by the built-in camera of the wearable device and in combination with the camera parameters, processing operations such as determining the pupil position, estimating the pupil distance, and tracking the gaze can be performed.

[0081] As Figure 2 shown, an exemplary method for determining the pupil position can be to generate a light spot in the image through the point light source 212. Then, based on data such as the position of the light spot in the image and the camera parameters, and in combination with optical principles, the pupil position is calculated. This calculation method is also known as the corneal reflection method for the pupil. However, the inventors of the present disclosure have found that there are certain problems with this calculation method.

[0082] First, the pupil corneal reflex method has certain requirements for the characteristics of the camera. As mentioned above, when the camera 208 is placed inside the lens barrel 202, due to the influence of the optical component 206 on the optical path, distortion occurs, making the images captured by the camera 208 and the camera parameters no longer able to be approximately calculated using the central camera model. However, it is difficult to directly apply the pupil corneal reflex method to a non-central camera. In other words, if one wants to use the pupil corneal reflex method to determine the pupil position, the camera 208 can only be placed outside the lens barrel 202.

[0083] Second, the pupil corneal reflex method has certain requirements for the number of light spots formed on the eye. As wearable devices become thinner and lighter, the exit pupil distance of the eye becomes smaller, and it becomes increasingly difficult for the point light source 212 to form an effective light spot on the eye, resulting in a significant reduction in the success rate and accuracy of this solution.

[0084] Third, this solution is not very applicable to the situation where the user wears glasses. Glasses will cause deviations in the light emitted by the point light source 212 and the light received by the camera. These two deflections will cause a very serious decrease in the accuracy of pupil position and pupil distance estimation.

[0085] In view of this, the embodiments of the present disclosure provide a method for determining the pupil center. By using the images captured by at least two cameras in the wearable device for capturing eye images of the same eye, it is possible to accurately calculate the pupil center without using the pupil corneal reflex method (without setting a light source for generating light spots).

[0086] Figure 3A The schematic diagram of an exemplary wearable device 300 provided by the embodiments of the present disclosure is shown.

[0087] As Figure 3A shown, the wearable device 300 may include a first lens barrel 302A and a second lens barrel 302B, corresponding to the first eye (for example, the left eye) and the second eye (for example, the right eye) respectively. Among them, corresponding to the first lens barrel 302A, the wearable device 300 may be provided with a first camera 304A and a second camera 306A, and corresponding to the second lens barrel 302B, the wearable device 300 may be provided with a third camera 304B and a fourth camera 306B. Optionally, the first camera 304A and the second camera 306A may be disposed inside or outside the first lens barrel 302A and may be used to capture eye images of the first eye. Similarly, the third camera 304B and the fourth camera 306B may be disposed inside or outside the second lens barrel 302B and may be used to capture eye images of the second eye.

[0088] Embodiments of the present disclosure can use images captured by different cameras to determine the pupil positions of corresponding eyes. Therefore, to ensure the accuracy of the algorithm, the two cameras corresponding to the same eye should not be too close to each other. Also, try to avoid placing the cameras in the area above the eyes, as this area is easily blocked by eyelashes, which will also affect the accuracy. For example, as Figure 3A shown, the first camera 304A is disposed at a position near the lower part of the first barrel 302A, and the second camera 306A is disposed at a position near the left side of the first barrel 302A. Similarly, the third camera 304B is disposed at a position near the lower part of the second barrel 302B, and the fourth camera 306B is disposed at a position near the right side of the second barrel 302B.

[0089] In some embodiments, to make the captured eye images clearer, the wearable device 300 can be provided with a first light source and a second light source respectively corresponding to the first barrel 302A and the second barrel 302B. It can be understood that the first light source and the second light source only need to illuminate the corresponding eye area and can be placed at any position of the wearable device 300. Therefore, there are no restrictions on their positions, and the specific positions are not shown in the schematic diagram.

[0090] In some embodiments, to ensure comfort, the first light source and the second light source can use infrared light sources. Correspondingly, the first camera 304A, the second camera 306A, the third camera 304B, and the fourth camera 306B can use infrared cameras.

[0091] In some embodiments, to improve the accuracy of the algorithm, a larger number of cameras can be provided. As Figure 3B shown, the wearable device 300 is further provided with a fifth camera 308A corresponding to the first barrel 302A and a sixth camera 308B corresponding to the second barrel 302B. To ensure the accuracy of the algorithm, there is also a certain distance between the fifth camera 308A and the first camera 304A and the second camera 306A. For example, as Figure 3B shown, the fifth camera 308A can be disposed at a position near the right side of the first barrel 302A. Similarly, there is also a certain distance between the sixth camera 308B and the third camera 304B and the fourth camera 306B. For example, as Figure 3B shown, the sixth camera 308B can be disposed at a position near the left side of the second barrel 302B.

[0092] Furthermore, embodiments of the present disclosure provide a method for determining the pupil center based on a wearable device. By placing at least two cameras inside or outside the barrel corresponding to each eye, in cooperation with the light source inside or outside each eye barrel, the position of the pupil center can be determined with high success rate and high accuracy using the eye images captured by at least two cameras, and the function of interpupillary distance measurement can be further realized.

[0093] This method for determining the pupil center does not require calibration of the light source, but only requires calibration of the camera parameters.

[0094] In some embodiments, if the camera is placed inside the lens barrel, the camera will pass through a complex and difficult-to-parametrically-describe optical system, and the centrality of the camera will be damaged. The traditional camera model is no longer applicable. Therefore, a non-parametric camera model can be used to calibrate the camera parameters.

[0095] The calibrated camera parameters are as Figure 3C shown. These camera parameters represent the spatial straight-line equation passing through the pixel points in the image captured by this camera, reflecting the spatial projection direction of the image pixel points. Specifically, the camera parameters are used to characterize the target pixel points in the image captured by the camera and the projection direction corresponding to the target pixel points. Optionally, what the camera parameters characterize can be the one-to-one correspondence between the pixel points and the projection directions. The specific calibration method is not limited here, and any implementation manner that can calibrate to obtain the camera parameters of the present disclosure is applicable.

[0096] In some embodiments, if the camera is placed outside the lens barrel, the same method can also be used to calibrate to obtain similar camera parameters, which represent the spatial straight-line equation passing through the pixel points in the image captured by this camera.

[0097] In this way, after the camera parameters are calibrated, each camera (the first camera 304A, the second camera 306A, the third camera 304B, the fourth camera 306B, the fifth camera 308A, the sixth camera 308B) corresponds to a set of camera parameters, and these camera parameters represent the spatial projection equation of the pixel points in the image captured by this camera.

[0098] Figure 4A FIG. shows a schematic flowchart of an exemplary method 400 provided by an embodiment of the present disclosure.

[0099] This method 400 can be used to determine the pupil center and can further determine the interpupillary distance. As Figure 4A shown, this method 400 may include the following steps.

[0100] Taking the determination of the position of the pupil center of the first eye (for example, the left eye) as an example, in step 402, a first eye image 502 captured by the first camera 304A (as Figure 5A shown) and a second eye image 504 captured by the second camera 306A (as Figure 5B shown) can be obtained.

[0101] In some embodiments, an eye-opening judgment model may be utilized to determine whether the eyes in the first eye image 502 and the second eye image 504 are open. If the eyes are in a closed state, the images are repeatedly acquired until valid eye-opening images are obtained.

[0102] In step 404, a first pixel point 5022 corresponding to the pupil center and a second pixel point 5042 corresponding to the pupil center are respectively determined from the first eye image 502 and the second eye image 504.

[0103] As Figure 5A shown, the first eye image 502 is an image of the first eye captured by the first camera 304A at its position. In the figure, the eye image is schematically shown as an ellipse. Optionally, the first pixel point 5022 corresponding to the pupil center of the first eye may be identified from the first eye image 502 through an image recognition algorithm. As Figure 5B shown, the second eye image 504 is an image of the first eye captured by the second camera 306A at its position. Similarly, the second pixel point 5042 corresponding to the pupil center of the first eye may also be identified from the second eye image 504.

[0104] In step 406, according to the position of the first pixel point in the first eye image, a first target camera parameter of the first camera is determined. Among them, the first target camera parameter includes a first spatial projection direction of the first pixel point.

[0105] As described above, the camera parameter represents the spatial straight-line equation passing through the corresponding pixel point, reflecting the spatial projection direction of the image pixel point. Therefore, after determining the position of the first pixel point 5022 in the first eye image 502, the corresponding first target camera parameter can be determined from the calibrated camera parameter set of the first camera 304A accordingly. The first target camera parameter represents the spatial straight-line equation passing through the first pixel point 5022.

[0106] In step 408, according to the position of the second pixel point in the second eye image, a second target camera parameter of the second camera is determined. Among them, the second target camera parameter includes a second spatial projection direction of the second pixel point.

[0107] Similarly, after determining the position of the second pixel point 5042 in the second eye image 504, the corresponding second target camera parameter can be determined from the calibrated camera parameter set of the second camera 306A accordingly. The second target camera parameter represents the spatial straight-line equation passing through the second pixel point 5042.

[0108] In step 410, according to the first target camera parameter and the second target camera parameter, the spatial position of the pupil center is determined.

[0109] Figure 5C A schematic diagram showing a calculation model 506 for determining the pupil center based on camera parameters according to an embodiment of the present disclosure.

[0110] As Figure 5C shown, the first target camera parameter can be mapped to a straight line 5062 in a three-dimensional space, and the second target camera parameter can be mapped to a straight line 5064 in the three-dimensional space.

[0111] It can be understood that the straight line 5062 is a spatial projection straight line passing through the first pixel point 5022, representing the projection relationship of the first pixel point 5022 from the camera coordinate system of the first camera 304A to the spatial coordinate system. The straight line 5064 is a spatial projection straight line passing through the second pixel point 5042, representing the projection relationship of the second pixel point 5042 from the camera coordinate system of the second camera 306A to the spatial coordinate system. The first pixel point 5022 and the second pixel point 5042 are the pupil centers observed by the first camera 304A and the second camera 306A respectively. Therefore, the intersection point Q of the straight line 5062 and the straight line 5064 can be approximated as the pupil center.

[0112] In this step, when the accuracy permits, the spatial coordinates of the intersection point Q can be determined as the spatial position of the pupil center of the first eye.

[0113] In some embodiments, the straight line 5062 and the straight line 5064 may not intersect strictly (i.e., there is no actual intersection point), and the point with the minimum sum of the squares of the straight line distances from the two can be calculated as the intersection point Q.

[0114] In some embodiments, the user may wear glasses. Since the glasses will affect the light path, correction is required. Therefore, the method 400 may further include obtaining adjustment parameters input by the user (such as information on glasses power, astigmatism power / axis, etc.);

[0115] Step 410 may further include: adjusting the first target camera parameter and the second target camera parameter according to the adjustment parameter; determining the spatial position of the pupil center according to the adjusted first target camera parameter and the second target camera parameter.

[0116] In this step, the calibrated camera parameter (or spatial straight line) is used as the incident vector and is shot from the outside of the glasses. Then, according to the optical parameters of the glasses themselves (such as information on glasses power, astigmatism power / axis, etc.), the outgoing vector on the eye side of the glasses can be solved, and the outgoing vector is used as the new observation result, that is: the adjusted first target camera parameter and the second target camera parameter.

[0117] Essentially, the user's glasses lenses, the optical components of the wearable device, and the camera together form a new camera model, which can greatly eliminate the influence of the user's glasses on the algorithm accuracy in this way.

[0118] In some embodiments, a detection device may also be provided in the wearable device 300 to detect whether the user is wearing glasses. Specifically, the detection device emits and receives detection light. Since the glasses lenses themselves are a kind of lens, thus, according to the angular difference between the emitted light and the received light, it can be determined whether the optical path of the detection light has changed, and further it can be determined whether the user is wearing glasses. And, according to the emitted light and the received light for corresponding optical calculations, information such as the glasses degree, astigmatism degree / axis, etc. can also be determined. Therefore, the adjustment parameter may not be input by the user, but can be detected by the detection device.

[0119] Since the pupil is inside the cornea, the light will refract when passing through the corneal surface. As Figure 5C shown, the straight lines 5062 and 5064 refract at the refraction points A and B on the corneal surface, and the refracted light rays 5066 and 5068 intersect at point P, which is the position of the real pupil center. Therefore, the position P of the real pupil center and the intersection point Q of the two straight lines usually do not coincide. Therefore, in some embodiments, the position deviation between position P and intersection point Q can be corrected.

[0120] In some embodiments, as Figure 4B shown, according to the first target camera parameter and the second target camera parameter, determining the spatial position of the pupil center may further include:

[0121] In step 4102, according to the first target camera parameter and the second target camera parameter, determine the spatial position to be corrected of the pupil center (i.e., intersection point Q).

[0122] In step 4104, obtain a correction parameter, which is used to correct the spatial position to be corrected of the pupil center, so as to obtain the position P of the corrected pupil center.

[0123] In some embodiments, the correction parameter can be determined through preprocessing. For example, by pre-using a simulation model to simulate the actual situation and determining according to the simulation results.

[0124] In some embodiments, the correction parameter is determined based on a mapping relationship, which is obtained by fitting a plurality of spatial projection direction pairs corresponding to a plurality of first spatial projection directions and a plurality of second spatial projection directions and a plurality of correction parameters. The spatial projection direction pair may refer to the direction pair formed by a single first spatial projection direction and its corresponding single second spatial projection direction. That is, there may be a one-to-one correspondence between the first spatial projection direction and the second spatial projection direction.

[0125] Since after wearing the wearable device, the area where the displayed content is relatively clear is within a certain space around the eyes. Therefore, in some embodiments, the correction parameter is determined according to the corneal radius probability distribution, the distance probability distribution from the pupil center to the corneal center, the line-of-sight angle distribution, and the refractive index distribution of the eyeball in the target space. By using these parameters to establish a simulation model and then determining the correction parameter according to the simulation result, the position correction of the pupil center can be better achieved. Among them, the target space may be an area close to the eyes and the clarity of the eyes observing the display screen is greater than the clarity threshold. As an alternative embodiment, the target space may be determined according to the Eyebox (dynamic eye box). Since the Eyebox is a conical area with the clearest displayed content between the optical module and the eyes, establishing a simulation model based on the parameters of the Eyebox can better achieve the position correction of the pupil center.

[0126] The following will introduce an embodiment of determining the trimming parameter through a simulation model in combination with Figure 5C the calculation model 506 shown.

[0127] In this embodiment, the deviation correction vector from Q to P can be defined in the following manner, where x and y are correction parameters that are negative real numbers.

[0128]

[0129] According to the above formula, the correction parameter is related to the corneal radius, the distance from the pupil center to the corneal center, and the observation direction of the camera, and it is difficult to directly solve.

[0130] However, the inventors of the present disclosure found that a set of approximate x and y can be obtained through simulation to be used as the correction parameter.

[0131] Assume that x and y are functions related to the angle α between the straight lines 5062 and 5064 and the spatial position of the intersection point Q (Q x , Q y , Q z ), that is: x, y = f(α, Q x , Q y , Q z ).

[0132] As Figure 4C shown, in some embodiments, the method 400 may include the following steps.

[0133] In step 412, within the Eyebox, multiple corneal center positions and multiple pupil center positions are sampled according to the corneal radius probability distribution, the distance probability distribution from the pupil center to the corneal center, and the line-of-sight angle distribution.

[0134] Among them, the Eyebox generally refers to a conical area between the optical module and the eyeball, which is also the area where the displayed content is the clearest. The larger the Eyebox, the better the mechanical tolerance of the wearable device and the more adaptable it is to people with different interpupillary distances. Generally, when designing the barrel of the wearable device, the Eyebox can be determined according to the design parameters. Therefore, during simulation, the design parameters of the barrel can be directly input into the simulation system to determine the Eyebox.

[0135] The corneal radius probability distribution can be determined based on multiple different eyeball models, and this probability distribution can reflect the distribution of corneal radii in most populations. Similarly, the distance probability distribution from the pupil center to the corneal center can also be determined in the same way.

[0136] The line-of-sight angle distribution can be determined according to the distribution of line-of-sight angles that may exist when the user uses the wearable device, and can be determined by statistically analyzing the actual usage of a certain group of people. In some embodiments, it can be determined according to the field of view (FOV) of the wearable device.

[0137] In this step, after having the probability distributions of the above parameters, these parameters can be sampled, and the sampling method can be arbitrary and is not limited here.

[0138] Then, based on the sampling results, the corneal center position and pupil center position corresponding to each set of sampling data can be calculated. In this way, multiple corneal center positions and multiple pupil center positions corresponding to the corneal center positions can be obtained according to multiple sets of sampling data.

[0139] In step 414, multiple corneal refractive indices are sampled according to the eyeball refractive index distribution.

[0140] Optionally, the eyeball refractive index distribution can be determined by statistically analyzing the eyeball refractive index distribution of a certain group of people.

[0141] In this step, according to the eyeball refractive index distribution, any sampling method can be used to sample multiple corneal refractive indices at multiple positions of the eyeball.

[0142] In step 416, multiple eye images collected by the first camera 304A and the second camera 306A at different camera positions are obtained through simulation.

[0143] In this step, for each simulation, according to the parameters obtained by sampling previously, a set of parameters (for example, corneal center position m1, pupil center position m2, corneal refractive index m3) is selected and fixed, and then the camera position m4 is continuously adjusted to obtain multiple eye images collected at different camera positions.

[0144] In this way, each eye image corresponds to a set of parameters: corneal center position m1, pupil center position m2, corneal refractive index m3, and camera position m4.

[0145] It can be understood that the camera position m4 described in this step refers to the overall position of the camera group. In this embodiment, the camera position m4 can be understood as the position where the centers of the first camera 304A and the second camera 306A are located, and the relative positions of the first camera 304A and the second camera 306A with respect to this center remain unchanged. It can be understood that when the camera group consists of three or more cameras, this center can be the common center position of the three or more cameras.

[0146] In step 418, based on the multiple eye images, in combination with the multiple corneal center positions, multiple pupil center positions, and multiple corneal refractive indices, the correction parameters are calculated.

[0147] As mentioned above, each eye image corresponds to a set of fixed parameters (known parameters): corneal center position m1, pupil center position m2, corneal refractive index m3, and camera position m4. Therefore, based on the pixel point position corresponding to the pupil center in the eye image and in combination with these fixed parameters, the correction parameters can be calculated.

[0148] In this way, using the simulation model, a set of approximate x and y can be obtained as the correction parameters to correct the to-be-corrected spatial position of the pupil center (i.e., the intersection point Q) to obtain coordinates approximating the true spatial position of the pupil center, thereby improving the algorithm accuracy.

[0149] In some embodiments, as Figure 4D shown, calculating the correction parameters based on the multiple eye images, in combination with the multiple corneal center positions, multiple pupil center positions, and multiple corneal refractive indices, may further include the following steps.

[0150] In step 4182, a first target image and a second target image respectively collected by the first camera and the second camera at the target position are obtained from the multiple eye images.

[0151] Figure 5D A schematic diagram of an exemplary simulation model 508 according to an embodiment of the present disclosure is shown.

[0152] As Figure 5D shown, it is assumed that a set of parameters corresponding to the acquisition of the first target image and the second target image can constitute the simulation model 508. Under this model 508, the corneal center position, the pupil center position P', the corneal refractive index n, and the target position of the camera are all known.

[0153] Figure 5E and Figure 5F schematic diagrams of the first target image 510 and the second target image 512 according to an embodiment of the present disclosure are shown respectively.

[0154] As Figure 5E and Figure 5F shown, the first target pixel point 5102 and the second target pixel point 5122 corresponding to the pupil center in the first target image 510 and the second target image 512 can be obtained by recognition.

[0155] In step 4184, according to the corneal center position, the pupil center position P', and the corneal refractive index corresponding to the acquisition of the first target image 510 and the second target image 512, combined with the first target pixel point 5102 and the second target pixel point 5122 corresponding to the pupil center in the first target image 510 and the second target image 512, the first straight line 5082 passing through the first target pixel point 5102 and the second straight line 5084 passing through the second target pixel point 5122, and the first refraction point A' and the second refraction point B' of the first straight line 5082 and the second straight line 5084 on the corneal surface are determined.

[0156] In this step, similar to the foregoing embodiments, on the premise of known camera parameters, the first straight line 5082 and the second straight line 5084 can be determined according to the positions of the first target pixel point 5102 and the second target pixel point 5122 in the first target image 510 and the second target image 512.

[0157] And, since the corneal center position, the pupil center position P', and the corneal refractive index are known, combined with the first straight line 5082 and the second straight line 5084, the first refraction point A' and the second refraction point B' of the first straight line 5082 and the second straight line 5084 on the corneal surface can be calculated.

[0158] In step 4186, the intersection point Q' of the first straight line and the second straight line is determined.

[0159] After knowing the first straight line 5082 and the second straight line 5084, the intersection point Q' of the two in three-dimensional space can be determined.

[0160] It can be understood that when the first straight line 5082 and the second straight line 5084 do not strictly intersect, the point with the minimum sum of the squares of the straight-line distances between the two can be used as the intersection point Q'.

[0161] In step 4188, according to the first refraction point A', the second refraction point B', the intersection point Q', the first straight line 5082 and the second straight line 5084, and the pupil center position P' corresponding to when the first target image 510 and the second target image 512 are acquired, determine the reference correction parameter corresponding to the target position.

[0162] In this step, according to the following formula:

[0163]

[0164] Combining the first refraction point A', the second refraction point B', the intersection point Q', the first straight line 5082 and the second straight line 5084, and the pupil center position P' corresponding to when the first target image 510 and the second target image 512 are acquired, a set of x0 and y0 can be obtained.

[0165] In step 4190, determine the correction parameter according to the reference correction parameter.

[0166] In this step, the correction parameters x and y can be further determined according to x0 and y0.

[0167] It can be understood that since x0 and y0 are only calculated based on a set of specific parameters when the camera is in the target position, as mentioned above, there are many parameters used in the simulation model. To ensure the algorithm accuracy, the correction parameters x and y need to be further determined according to the reference correction parameters obtained by simulating with multiple sets of different parameters.

[0168] Therefore, in some embodiments, determining the correction parameter according to the reference correction parameter includes:

[0169] According to the reference correction parameter, the intersection point, the first straight line and the second straight line, determine the mapping relationship between the reference correction parameter and the included angle α between the intersection point and the straight line (this included angle can be calculated according to the first straight line 5082 and the second straight line 5084), for example, (α, q x , q y , q z ) and (x0, y0); then determine the correction parameter according to the mapping relationship.

[0170] Further, in some embodiments, the mapping relationship includes multiple mapping relationships corresponding to multiple different target positions. Determining the correction parameter according to the mapping relationship includes: fitting the correction parameter according to the multiple mapping relationships. For example, using a model (including but not limited to polynomial regression, deep neural network) to fit the mapping relationship from (α, q x , q y , q z ) to (x, y) obtained from multiple simulations.

[0171] In this way, the mapping relationship from (α, q x , q y , q z ) to (x, y) can be obtained.

[0172] In step 4106, according to the first target camera parameter, the second target camera parameter, the to-be-corrected spatial position of the pupil center, and the correction parameter, determine the spatial position of the pupil center.

[0173] Specifically, in step 4104, the included angle between lines 5062 and 5064 can be determined according to the first target camera parameter and the second target camera parameter. Then, according to the included angle and the to-be-corrected spatial position Q of the pupil center, the corresponding correction parameters x and y are found.

[0174] In this step, according to the first target camera parameter, the second target camera parameter, the to-be-corrected spatial position of the pupil center, and the correction parameter, the spatial position of the pupil center can be determined.

[0175] In some embodiments, determining the spatial position of the pupil center according to the first target camera parameter, the second target camera parameter, the to-be-corrected spatial position of the pupil center, and the correction parameter further includes:

[0176] Determining the offset vector of the to-be-corrected spatial position of the pupil center relative to the spatial position of the pupil center according to the first target camera parameter, the second target camera parameter, and the correction parameter; for example, calculating a correction vector according to the correction parameter

[0177] Determining the spatial position of the pupil center according to the to-be-corrected spatial position of the pupil center and the offset vector, that is: obtaining the estimated pupil coordinates

[0178] So far, according to the first eye image 502 and the second eye image 504 of the first eye collected by the first camera 304A and the second camera 306A, the spatial position of the pupil center of the first eye in the three-dimensional space can be determined.

[0179] For the case of three or more cameras, the pupil positions can be calculated for any combination of two of the cameras first, and then the average of all combinations can be taken to obtain the final pupil position result.

[0180] It can be understood that, following a similar idea to the above method 400, the pupil center of the second eye can also be determined.

[0181] As Figure 4E shown, in some embodiments, the method 400 may further include the following steps:

[0182] Taking the determination of the position of the pupil center of the second eye (e.g., the right eye) as an example, in step 420, the third eye image collected by the third camera 304B and the fourth eye image collected by the fourth camera 306B are obtained.

[0183] In some embodiments, an eye-opening judgment model can be used to judge whether the eyes in the third eye image and the fourth eye image are open. If it is in a closed-eye state, the images are repeatedly collected until valid eye-opening images are obtained.

[0184] In step 422, the third pixel point and the fourth pixel point corresponding to the second pupil center of the second eye are respectively determined from the third eye image and the fourth eye image. Optionally, the third pixel point and the fourth pixel point corresponding to the pupil center of the second eye can be identified from the third eye image and the fourth eye image through an image recognition algorithm.

[0185] In step 424, according to the position of the third pixel point in the third eye image, the third target camera parameter of the third camera is determined, and the third target camera parameter includes the third spatial projection direction of the third pixel point.

[0186] As described above, the camera parameter represents the spatial straight line equation passing through the corresponding pixel point, reflecting the spatial projection direction of the image pixel point. Therefore, after determining the position of the third pixel point in the third eye image, the corresponding third target camera parameter can be determined from the pre-calibrated camera parameter set of the third camera 304B accordingly, and the third target camera parameter represents the spatial straight line equation passing through the third pixel point.

[0187] In step 426, according to the position of the fourth pixel point in the fourth eye image, the fourth target camera parameter of the fourth camera is determined, and the fourth target camera parameter includes the fourth spatial projection direction of the fourth pixel point.

[0188] Similarly, after determining the position of the fourth pixel point in the fourth eye image, the corresponding fourth target camera parameters can be determined from the pre-calibrated camera parameter set of the fourth camera 306B accordingly. The fourth target camera parameters represent the spatial straight line equation passing through the fourth pixel point.

[0189] In step 428, according to the third target camera parameters and the fourth target camera parameters, determine the second spatial position of the second pupil center.

[0190] The implementation manner of step 428 is similar to that of step 410, and the process of constructing the simulation model is also similar, which will not be elaborated here.

[0191] In this way, the spatial position of the pupil center of the second eye in the three-dimensional space can be obtained.

[0192] As can be seen from the above embodiments, the embodiments of the present disclosure provide a method for determining the pupil center. By using the images collected by at least two cameras provided in the wearable device for collecting eye images of the same eye, the pupil center can be calculated more accurately without using the pupil corneal reflection method, and the pupil centers of both eyes can be determined accordingly, and then the interpupillary distance can be determined.

[0193] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In this distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiments of the present disclosure, and these multiple devices will interact with each other to complete the described method.

[0194] It should be noted that some embodiments of the present disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the above embodiments and still achieve the desired results. Additionally, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0195] The embodiments of the present disclosure also provide a method for determining the interpupillary distance based on a wearable device. The wearable device includes a first target camera and a second target camera for collecting eye images of the first eye, and a third target camera and a fourth target camera for collecting eye images of the second eye.

[0196] Figure 4F The flowchart of another exemplary method 430 provided by the embodiments of the present disclosure is shown.

[0197] The method 430 can be used to determine the interpupillary distance. As Figure 4F shown, the method 430 may further include the following steps.

[0198] In step 432, the first spatial position of the first pupil center of the first eye and the second spatial position of the second pupil center of the second eye determined by any embodiment or permutation and combination of the embodiments of the method 400 can be obtained. The determination methods of the first spatial position and the second spatial position refer to the embodiments of the method 400 and will not be elaborated here.

[0199] In step 434, the interpupillary distance is determined according to the first spatial position (the spatial position of the pupil center of the first eye in the three-dimensional space) and the second spatial position (the spatial position of the pupil center of the second eye in the three-dimensional space).

[0200] After determining the interpupillary distance, the lens barrel can be adjusted according to the interpupillary distance to adapt to the user's interpupillary distance, increase comfort, and improve the user experience.

[0201] The embodiments of the present disclosure also provide a computer device for implementing the above method 400 or method 430. Figure 6 The hardware structure diagram of an exemplary computer device 600 provided by the embodiments of the present disclosure is shown. The computer device 600 can be used to implement Figure 1A , Figure 1B the head-mounted wearable device 104, Figure 2 the wearable device 200, Figure 3A and Figure 3B the wearable device 300, and can also be used to implement Figure 1A the external device 112, and can also be used to implement Figure 1A the server 114. In some scenarios, the computer device 600 can also be used to implement Figure 1A the database server 116.

[0202] As Figure 6 shown, the computer device 600 may include: a processor 602, a memory 604, a network module 606, a peripheral interface 608, and a bus 610. Among them, the processor 602, the memory 604, the network module 606, and the peripheral interface 608 are communicatively connected to each other inside the computer device 600 through the bus 610.

[0203] The processor 602 may be a Central Processing Unit (CPU), an image processor, a Neural Processing Unit (NPU), a microcontroller (MCU), a programmable logic device, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits. The processor 602 may be used to execute functions related to the technologies described in this disclosure. In some embodiments, the processor 602 may further include multiple processors integrated as a single logic component. For example, as Figure 6 shown, the processor 602 may include multiple processors 602a, 602b, and 602c.

[0204] The memory 604 may be configured to store data (e.g., instructions, computer code, etc.). As Figure 6 shown, the data stored in the memory 604 may include program instructions (e.g., program instructions for implementing the method 400 or method 430 of the embodiments of this disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 602 may also access the program instructions and data stored in the memory 604 and execute the program instructions to operate on the data to be processed. The memory 604 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 604 may include a Random Access Memory (RAM), a Read Only Memory (ROM), an optical disc, a magnetic disk, a hard disk, a Solid State Drive (SSD), a flash memory, a memory stick, etc.

[0205] The network interface 606 may be configured to provide communication with other external devices to the computer device 600 via a network. The network may be any wired or wireless network capable of transmitting and receiving data. For example, the network may be a wired network, a local wireless network (e.g., Bluetooth, WiFi, Near Field Communication (NFC), etc.), a cellular network, the Internet, or a combination of the above. It can be understood that the type of the network is not limited to the above specific examples.

[0206] The peripheral interface 608 may be configured to connect the computer device 600 to one or more peripheral devices to enable information input and output. For example, the peripheral devices may include input devices such as a keyboard, a mouse, a touchpad, a touch screen, a microphone, various sensors, etc. and output devices such as a display, a speaker, a vibrator, an indicator light, etc.

[0207] The bus 610 can be configured to transfer information among various components of the computer device 600 (such as the processor 602, the memory 604, the network interface 606, and the peripheral interface 608), such as internal buses (e.g., the processor-memory bus), external buses (USB ports, PCI-E buses), etc.

[0208] It should be noted that although the architecture of the computer device 600 shown above only shows the processor 602, the memory 604, the network interface 606, the peripheral interface 608, and the bus 610, in the specific implementation process, the architecture of the computer device 600 may further include other components necessary for normal operation. In addition, those skilled in the art can understand that the architecture of the computer device 600 above may also only include the components necessary to implement the solution of the embodiments of the present disclosure, and do not necessarily include all the components shown in the figure.

[0209] The embodiments of the present disclosure also provide a pupil center determination device based on a wearable device. Figure 7 The schematic diagram of the exemplary device 700 provided by the embodiments of the present disclosure is shown. As Figure 7 shown, the device 700 can be used to implement the method 400 and may further include the following modules.

[0210] An acquisition module 702, configured to: acquire a first eye image acquired by the first camera and a second eye image acquired by the second camera;

[0211] A first determination module 704, configured to: respectively determine a first pixel point and a second pixel point corresponding to the pupil center from the first eye image and the second eye image;

[0212] A second determination module 706, configured to: determine a first target camera parameter of the first camera according to the position of the first pixel point in the first eye image, where the first target camera parameter includes a first spatial projection direction of the first pixel point;

[0213] A third determination module 708, configured to: determine a second target camera parameter of the second camera according to the position of the second pixel point in the second eye image, where the second target camera parameter includes a second spatial projection direction of the second pixel point;

[0214] A fourth determination module 710, configured to: determine the spatial position of the pupil center according to the first target camera parameter and the second target camera parameter.

[0215] In some embodiments, the fourth determination module 710 is configured to:

[0216] Determine the spatial position to be corrected of the pupil center according to the first target camera parameters and the second target camera parameters;

[0217] Obtain correction parameters;

[0218] Determine the spatial position of the pupil center according to the first target camera parameters, the second target camera parameters, the spatial position to be corrected of the pupil center, and the correction parameters.

[0219] In some embodiments, the correction parameters are determined based on a mapping relationship, and the mapping relationship is obtained by fitting a plurality of spatial projection direction pairs corresponding to a plurality of the first spatial projection directions and a plurality of the second spatial projection directions and a plurality of correction parameters.

[0220] In some embodiments, the fourth determination module 710 is configured to:

[0221] Determine an offset vector of the spatial position to be corrected of the pupil center relative to the spatial position of the pupil center according to the first target camera parameters, the second target camera parameters, and the correction parameters;

[0222] Determine the spatial position of the pupil center according to the spatial position to be corrected of the pupil center and the offset vector.

[0223] In some embodiments, the correction parameters are determined according to the probability distribution of the corneal radius, the probability distribution of the distance from the pupil center to the corneal center, the line-of-sight angle distribution, and the refractive index distribution of the eyeball within the moving orbital range.

[0224] In some embodiments, the device further includes a simulation module configured to:

[0225] Within the moving orbital range, sample a plurality of corneal center positions and a plurality of pupil center positions according to the probability distribution of the corneal radius, the probability distribution of the distance from the pupil center to the corneal center, and the line-of-sight angle distribution;

[0226] Sample a plurality of corneal refractive indices according to the refractive index distribution of the eyeball;

[0227] Simulate a plurality of eye images collected by the first camera and the second camera at different camera positions;

[0228] Calculate the correction parameters according to the plurality of eye images, in combination with the plurality of corneal center positions, the plurality of pupil center positions, and the plurality of corneal refractive indices.

[0229] In some embodiments, the simulation module is configured to:

[0230] Obtain a first target image and a second target image respectively captured by the first camera and the second camera at a target position from the multiple eye images;

[0231] According to the corneal center position, the pupil center position, and the corneal refractive index corresponding to the acquisition of the first target image and the second target image, and combining the first target pixel point and the second target pixel point corresponding to the pupil center in the first target image and the second target image, determine a first straight line passing through the first target pixel point and a second straight line passing through the second target pixel point, as well as a first refraction point and a second refraction point of the first straight line and the second straight line on the corneal surface;

[0232] Determine the intersection point of the first straight line and the second straight line;

[0233] According to the first refraction point, the second refraction point, the intersection point, the first straight line and the second straight line, and the pupil center position corresponding to the acquisition of the first target image and the second target image, determine a reference correction parameter corresponding to the target position;

[0234] Determine the correction parameter according to the reference correction parameter.

[0235] In some embodiments, the simulation module is configured to:

[0236] According to the reference correction parameter, the intersection point, the first straight line and the second straight line, determine a mapping relationship between the reference correction parameter and the included angle between the intersection point and the straight lines;

[0237] Determine the correction parameter according to the mapping relationship.

[0238] In some embodiments, the mapping relationship includes multiple mapping relationships corresponding to multiple different target positions, and the simulation module is configured to: Fit to obtain the correction parameter according to the multiple mapping relationships.

[0239] In some embodiments, the acquisition module 702 is configured to: Obtain an adjustment parameter input by a user;

[0240] The fourth determination module 710 is configured to: Adjust the first target camera parameter and the second target camera parameter according to the adjustment parameter; Determine the spatial position of the pupil center according to the adjusted first target camera parameter and the second target camera parameter.

[0241] For the convenience of description, the above device is described by dividing its functions into various modules. Of course, when implementing the present disclosure, the functions of each module can be realized in the same or multiple software and / or hardware.

[0242] The device of the above embodiment is used to implement the corresponding method 400 in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be elaborated here.

[0243] An embodiment of the present disclosure also provides a pupil distance determination device based on a wearable device. The wearable device includes a first target camera and a second target camera for collecting eye images of the first eye, and a third target camera and a fourth target camera for collecting eye images of the second eye.

[0244] Figure 8 A schematic diagram of an exemplary device 800 provided by an embodiment of the present disclosure is shown. As Figure 8 shown, the device 800 can be used to implement the method 430, and may further include the following modules.

[0245] An acquisition module 802, configured to: acquire a first spatial position of a first pupil center of the first eye and a second spatial position of a second pupil center of the second eye determined by any embodiment or permutation and combination of the embodiments of the method 400. The determination methods of the first spatial position and the second spatial position refer to the embodiments of the method 400, which will not be elaborated here.

[0246] A determination module 804, configured to: determine the pupil distance according to the first spatial position and the second spatial position.

[0247] For the convenience of description, when describing the above device, various modules are described separately according to their functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0248] The device of the above embodiment is used to implement the corresponding method 430 in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be elaborated here.

[0249] Based on the same inventive concept, corresponding to the method in any of the above embodiments, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method 400 or method 430 described in any of the foregoing embodiments.

[0250] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0251] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to execute the method 400 or method 430 described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0252] Based on the same inventive concept, corresponding to the method 400 or method 430 in any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of the computer to cause the computer and / or the processor to execute the method 400 or method 430. Corresponding to the execution subject of each step in the embodiments of method 400 or method 430, the processor executing the corresponding step can belong to the corresponding execution subject.

[0253] The computer program product of the above embodiment is used to cause the computer and / or the processor to execute the method 400 or method 430 described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0254] Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples; under the concept of the present disclosure, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of brevity.

[0255] In addition, for simplicity of explanation and discussion, and so as not to make the embodiments of the present disclosure difficult to understand, well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Further, the devices may be shown in block diagram form in order to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that details regarding the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be entirely within the understanding of those skilled in the art). In cases where specific details (such as circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be practiced without these specific details or with variations of these specific details. Accordingly, these descriptions should be considered illustrative rather than restrictive.

[0256] Although the present disclosure has been described in connection with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures (such as dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0257] Embodiments of the present disclosure are intended to cover all such alternatives, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A method for determining the pupil center based on a wearable device, wherein, The wearable device includes a first camera and a second camera for collecting eye images of the same eye, and the method includes: Obtaining a first eye image collected by the first camera and a second eye image collected by the second camera; Respectively determining a first pixel point and a second pixel point corresponding to the pupil center from the first eye image and the second eye image; Determining first target camera parameters of the first camera according to the position of the first pixel point in the first eye image, where the first target camera parameters include a first spatial projection direction of the first pixel point; Determining second target camera parameters of the second camera according to the position of the second pixel point in the second eye image, where the second target camera parameters include a second spatial projection direction of the second pixel point; Determining the spatial position of the pupil center according to the first target camera parameters and the second target camera parameters.

2. The method according to claim 1, wherein, Determining the spatial position of the pupil center according to the first target camera parameters and the second target camera parameters includes: Determining a to-be-corrected spatial position of the pupil center according to the first target camera parameters and the second target camera parameters; Obtaining correction parameters; Determining the spatial position of the pupil center according to the first target camera parameters, the second target camera parameters, the to-be-corrected spatial position of the pupil center, and the correction parameters.

3. The method according to claim 2, wherein The correction parameters are determined based on a mapping relationship, and the mapping relationship is obtained by fitting a plurality of spatial projection direction pairs corresponding to a plurality of the first spatial projection directions and a plurality of the second spatial projection directions and a plurality of correction parameters.

4. The method according to claim 2, wherein, Determining the spatial position of the pupil center according to the first target camera parameters, the second target camera parameters, the to-be-corrected spatial position of the pupil center, and the correction parameters includes: Determining an offset vector of the to-be-corrected spatial position of the pupil center relative to the spatial position of the pupil center according to the first target camera parameters, the second target camera parameters, and the correction parameters; Determining the spatial position of the pupil center according to the to-be-corrected spatial position of the pupil center and the offset vector.

5. The method according to claim 2, wherein, The correction parameters are determined according to the corneal radius probability distribution, the distance probability distribution from the pupil center to the corneal center, the line-of-sight angle distribution, and the eyeball refractive index distribution within the moving eye socket range.

6. The method according to claim 2, wherein, The method further includes: Sampling a plurality of corneal center positions and a plurality of pupil center positions according to the corneal radius probability distribution, the distance probability distribution from the pupil center to the corneal center, and the line-of-sight angle distribution within the moving eye socket range; Sampling a plurality of corneal refractive indices according to the eyeball refractive index distribution; Simulating a plurality of eye images collected by the first camera and the second camera at different camera positions; Calculating the correction parameters according to the plurality of eye images, in combination with the plurality of corneal center positions, the plurality of pupil center positions, and the plurality of corneal refractive indices.

7. The method according to claim 6, wherein, Calculating the correction parameters according to the plurality of eye images, in combination with the plurality of corneal center positions, the plurality of pupil center positions, and the plurality of corneal refractive indices, includes: Obtain a first target image and a second target image respectively collected by the first camera and the second camera at a target position from the multiple eye images; According to the corneal center position, the pupil center position, and the corneal refractive index corresponding to the acquisition of the first target image and the second target image, and combining the first target pixel point and the second target pixel point corresponding to the pupil center in the first target image and the second target image, determine a first straight line passing through the first target pixel point and a second straight line passing through the second target pixel point, as well as a first refraction point and a second refraction point of the first straight line and the second straight line on the corneal surface; Determine the intersection point of the first straight line and the second straight line; According to the first refraction point, the second refraction point, the intersection point, the first straight line and the second straight line, and the pupil center position corresponding to the acquisition of the first target image and the second target image, determine a reference correction parameter corresponding to the target position; Determine the correction parameter according to the reference correction parameter.

8. The method according to claim 7, wherein, Determining the correction parameter according to the reference correction parameter includes: According to the reference correction parameter, the intersection point, the first straight line and the second straight line, determine the mapping relationship between the reference correction parameter and the included angle between the intersection point and the straight lines; Determine the correction parameter according to the mapping relationship.

9. The method according to claim 8, wherein, The mapping relationship includes multiple mapping relationships corresponding to multiple different target positions. Determining the correction parameter according to the mapping relationship includes: Fit to obtain the correction parameter according to the multiple mapping relationships.

10. The method according to claim 1, wherein The method further includes: obtaining an adjustment parameter input by a user; Determine the spatial position of the pupil center according to the first target camera parameter and the second target camera parameter, including: adjusting the first target camera parameter and the second target camera parameter according to the adjustment parameter; determining the spatial position of the pupil center according to the adjusted first target camera parameter and the second target camera parameter.

11. A method for determining interpupillary distance based on a wearable device, the wearable device including a first target camera and a second target camera for collecting eye images of a first eye, and a third target camera and a fourth target camera for collecting eye images of a second eye, the method including: Obtain a first spatial position of a first pupil center of the first eye and a second spatial position of a second pupil center of the second eye determined by using the method according to any one of claims 1-10; Determine the interpupillary distance according to the first spatial position and the second spatial position.

12. A pupil center determination device based on a wearable device, wherein, The wearable device includes a first camera and a second camera for collecting eye images of the same eye, and the device includes: An acquisition module, configured to: acquire a first eye image acquired by the first camera and a second eye image acquired by the second camera; A first determination module, configured to: respectively determine a first pixel point and a second pixel point corresponding to the pupil center from the first eye image and the second eye image; A second determination module, configured to: determine first target camera parameters of the first camera according to the position of the first pixel point in the first eye image, where the first target camera parameters include a first spatial projection direction of the first pixel point; A third determination module, configured to: determine second target camera parameters of the second camera according to the position of the second pixel point in the second eye image, where the second target camera parameters include a second spatial projection direction of the second pixel point; A fourth determination module, configured to: determine a spatial position of the pupil center according to the first target camera parameters and the second target camera parameters.

13. A pupil distance determination device based on a wearable device, where the wearable device includes a first target camera and a second target camera for collecting eye images of a first eye and a third target camera and a fourth target camera for collecting eye images of a second eye, and the device includes: An acquisition module, configured to: acquire a first spatial position of a first pupil center of the first eye and a second spatial position of a second pupil center of the second eye determined by using the method according to any one of claims 1-10; A determination module, configured to: determine a pupil distance according to the first spatial position and the second spatial position.

14. A computer device, including one or more processors and a memory; and one or more programs, where the one or more programs are stored in the memory and are executed by the one or more processors, and the programs include instructions for executing the method according to any one of claims 1-10 or the method according to claim 11.

15. A non-volatile computer-readable storage medium containing a computer program, where when the computer program is executed by one or more processors, the processors are caused to execute the method according to any one of claims 1-10 or the method according to claim 11.

16. A computer program product, including computer program instructions, where when the computer program instructions run on a computer, the computer is caused to execute the method according to any one of claims 1-10 or the method according to claim 11.

Images