Method for determining pupil center and pupil based on wearable device and related device

Abstract

The present disclosure provides a wearable device-based pupil center determination method and related device, wherein the wearable device comprises a first camera and a second camera for collecting eye images of the same eye, and the method comprises: acquiring a first eye image collected by the first camera and a second eye image collected by the second camera; determining a first pixel point and a second pixel point corresponding to the pupil center from the first eye image and the second eye image respectively; determining a first target camera parameter of the first camera according to the position of the first pixel point in the first eye image; determining a second target camera parameter of the second camera according to the position of the second pixel point in the second eye image; and determining the spatial position of the pupil center according to the first target camera parameter and the second target camera parameter.

Description

Technical Field

[0001] This disclosure relates to the field of extended reality technology, and in particular to a method and related equipment for determining the pupil center and pupil based on a wearable device. Background Technology

[0002] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating an interactive virtual environment. XR technology can further include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), utilizing hardware devices and various technologies to merge virtual content with real-world scenes.

[0003] Generally, extended reality systems provide users with wearable devices to enable human-computer interaction; these wearable devices can be head-mounted. In some scenarios, wearable devices can perform calculations by acquiring eye images to achieve gaze tracking or interpupillary distance estimation.

[0004] However, the inventors of this disclosure have discovered that the methods for calculating the pupil center in the related art have certain limitations. Summary of the Invention

[0005] This disclosure proposes a method and related equipment for determining the pupil center and pupil based on a wearable device, in order to solve or partially solve the above-mentioned problems.

[0006] In a first aspect, this disclosure provides a method for determining the pupil center based on a wearable device, wherein the wearable device includes a first camera and a second camera for acquiring eye images of the same eye, and the method includes:

[0007] Acquire a first eye image captured by the first camera and a second eye image captured by the second camera;

[0008] The first pixel and the second pixel corresponding to the center of the pupil are determined from the first eye image and the second eye image, respectively.

[0009] Based on the position of the first pixel in the first eye image, the first target camera parameters of the first camera are determined, and the first target camera parameters include the first spatial projection direction of the first pixel.

[0010] Based on the position of the second pixel in the second eye image, the second target camera parameters of the second camera are determined, and the second target camera parameters include the second spatial projection direction of the second pixel;

[0011] The spatial position of the pupil center is determined based on the parameters of the first target camera and the parameters of the second target camera.

[0012] A second aspect of this disclosure provides a method for determining interpupillary distance based on a wearable device, the wearable device including a first target camera and a second target camera for acquiring eye images of a first eye, and a third target camera and a fourth target camera for acquiring eye images of a second eye, the method comprising:

[0013] Obtain the first spatial position of the center of the first pupil of the first eye and the second spatial position of the center of the second pupil of the second eye, as determined by the method described in the first aspect;

[0014] The interpupillary distance is determined based on the first spatial position and the second spatial position.

[0015] A third aspect of this disclosure provides a pupil center determination device based on a wearable device, wherein the wearable device includes a first camera and a second camera for acquiring eye images of the same eye, and the device includes:

[0016] The acquisition module is configured to acquire a first eye image captured by the first camera and a second eye image captured by the second camera;

[0017] The first determining module is configured to: determine the first pixel point and the second pixel point corresponding to the pupil center from the first eye image and the second eye image, respectively;

[0018] The second determining module is configured to: determine the first target camera parameters of the first camera based on the position of the first pixel in the first eye image, wherein the first target camera parameters include the first spatial projection direction of the first pixel;

[0019] The third determining module is configured to: determine the second target camera parameters of the second camera based on the position of the second pixel in the second eye image, wherein the second target camera parameters include the second spatial projection direction of the second pixel;

[0020] The fourth determining module is configured to: determine the spatial position of the pupil center based on the parameters of the first target camera and the parameters of the second target camera.

[0021] In a fourth aspect, this disclosure provides an interpupillary distance determination device based on a wearable device, the wearable device including a first target camera and a second target camera for acquiring eye images of a first eye, and a third target camera and a fourth target camera for acquiring eye images of a second eye, the device comprising:

[0022] The acquisition module is configured to: acquire the first spatial position of the center of the first pupil of the first eye and the second spatial position of the center of the second pupil of the second eye, as determined by the method described in the first aspect;

[0023] The determining module is configured to: determine the interpupillary distance based on the first spatial position and the second spatial position.

[0024] A fifth aspect of this disclosure provides a computer device including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, the programs including instructions for performing the method according to the first or second aspect.

[0025] A sixth aspect of this disclosure provides a non-volatile computer-readable storage medium comprising a computer program that, when executed by one or more processors, causes the processors to perform the method described in the first or second aspect.

[0026] A seventh aspect of this disclosure provides a computer program product including computer program instructions that, when executed on a computer, cause the computer to perform the method described in the first or second aspect.

[0027] The pupil center determination method and related equipment based on wearable devices provided in this disclosure can calculate the pupil center more accurately without using the pupil-corneal reflection method by utilizing images captured by at least two cameras installed in the wearable device for acquiring eye images of the same eye. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1A A schematic diagram of an exemplary system provided by an embodiment of this disclosure is shown.

[0030] Figure 1B A schematic diagram of an exemplary head-mounted wearable device is shown.

[0031] Figure 2 A schematic diagram of an exemplary wearable device provided by an embodiment of this disclosure is shown.

[0032] Figure 3AA schematic diagram of another exemplary wearable device provided in an embodiment of this disclosure is shown.

[0033] Figure 3B A schematic diagram of yet another exemplary wearable device provided in an embodiment of this disclosure is shown.

[0034] Figure 3C A schematic diagram of exemplary camera parameters according to an embodiment of the present disclosure is shown.

[0035] Figure 4A A flowchart illustrating an exemplary method provided in an embodiment of this disclosure is shown.

[0036] Figure 4B A flowchart illustrating an exemplary method for determining the spatial location of the pupil center according to an embodiment of the present disclosure is shown.

[0037] Figure 4C A partial flowchart of an exemplary method provided by an embodiment of this disclosure is shown.

[0038] Figure 4D A flowchart illustrating an exemplary method for determining correction parameters according to an embodiment of this disclosure is shown.

[0039] Figure 4E A partial flowchart of an exemplary method provided by an embodiment of this disclosure is shown.

[0040] Figure 4F A flowchart illustrating another exemplary method provided by an embodiment of this disclosure is shown.

[0041] Figure 5A A schematic diagram of an exemplary first eye image according to an embodiment of the present disclosure is shown.

[0042] Figure 5B A schematic diagram of an exemplary second eye image according to an embodiment of the present disclosure is shown.

[0043] Figure 5C A schematic diagram of a computational model for determining the pupil center based on camera parameters according to an embodiment of the present disclosure is shown.

[0044] Figure 5D A schematic diagram of an exemplary simulation model according to an embodiment of the present disclosure is shown.

[0045] Figure 5E A schematic diagram of a first target image according to an embodiment of the present disclosure is shown.

[0046] Figure 5F A schematic diagram of a second target image according to an embodiment of the present disclosure is shown.

[0047] Figure 6A schematic diagram of the hardware structure of an exemplary computer device provided in an embodiment of this disclosure is shown.

[0048] Figure 7 A schematic diagram of an exemplary apparatus provided in an embodiment of this disclosure is shown.

[0049] Figure 8 A schematic diagram of another exemplary apparatus provided by an embodiment of this disclosure is shown. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0051] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0052] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0053] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0054] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0055] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0056] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0057] Figure 1A A schematic diagram of an exemplary extended reality system 100 provided in an embodiment of this disclosure is shown.

[0058] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating an interactive virtual environment. XR technology can further include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), utilizing hardware devices and various technologies to merge virtual content with real-world scenes.

[0059] like Figure 1A As shown, the system 100 may include various types of wearable devices, such as head-mounted wearable devices (e.g., VR / AR glasses or head-mounted displays (HMDs)) 104, control handles 108, etc. In some scenarios, a camera / camera 110 may also be provided for taking photos of the operator (user) 102. In some embodiments, when the aforementioned devices do not have processing functions or when necessary, the system 100 may also include an external control device 112 for providing processing functions. The control device 112 may be, for example, a mobile phone, computer, or other computer device. In some embodiments, when any of the aforementioned devices acts as a control device or a main control device, it can interact with other devices in the system 100 through wired or wireless communication methods to achieve information exchange.

[0060] In system 100, user 102 can interact with extended reality system 100 using head-mounted wearable device 104 and control handle 108. In some scenarios, system 100 can use images captured by camera / video camera 110 to recognize user 102's posture, gestures, etc., and then complete the interaction with user 102 based on the recognized posture and gestures. In some embodiments, user 130 can also perform gesture input using bare hands. Head-mounted wearable device 104 can capture images in front of it in real time using a camera or other camera positioned in front of the head-mounted wearable device 104, and recognize user 130's gestures by recognizing these images.

[0061] In some embodiments, such as Figure 1A As shown, system 100 can also communicate with server 114 and obtain data from server 114, such as images, audio, and video, and can output this data through head-mounted wearable device 104, for example, displaying images or videos on the display screen of head-mounted wearable device 104, playing audio and video audio through the speaker of head-mounted wearable device 104, etc. In some embodiments, such as Figure 1A As shown, server 114 can retrieve the required data, such as images, audio, and video, from database server 116, which is used to store data.

[0062] In some embodiments, a data acquisition unit for collecting information may be provided on the head-mounted wearable device 104. The type of data acquisition unit can be varied.

[0063] In some embodiments, the acquisition unit may further include an environment acquisition unit and a positioning and tracking unit. The environment acquisition unit can be used to acquire environmental information around (e.g., in front of) the wearable device 104, and the positioning and tracking unit can be used to perform positioning and tracking of the wearable device 104. Optionally, the environment acquisition unit may include, but is not limited to, photosensitive elements such as a three-color camera (e.g., an RGB camera), a depth camera, a binocular camera, or a laser. The positioning and tracking unit may include, but is not limited to, modules such as visual real-time localization and mapping (visual SLAM), an inertial measurement unit (IMU), a global positioning system (GPS), ultra-wideband wireless communication technology (UWB), and a laser.

[0064] In some embodiments, the head-mounted wearable device 104 may also be equipped with a velocity sensor, acceleration sensor, angular velocity sensor (e.g., gyroscope), etc., for collecting velocity or acceleration information of the head-mounted wearable device 104. Similarly, the operating handle 108 may also be equipped with a velocity sensor, acceleration sensor, angular velocity sensor (e.g., gyroscope), etc., for collecting velocity or acceleration information of the operating handle 108. It should be noted that, in addition to being mounted on the head-mounted wearable device 104 and the operating handle 108, the aforementioned acquisition units may also be directly attached to the body parts of the interactive user 102 without relying on hardware devices, thereby collecting relevant information about that body part, such as velocity, acceleration, or angular velocity information, or information collected by other sensors or acquisition units (e.g., eye images (including pupil images), etc.).

[0065] In some embodiments, the head-mounted wearable device 104 may also be equipped with a camera or webcam for taking photos of the operator (user) 102 (e.g., photos of hands or feet) and environmental images.

[0066] In some embodiments, the system 100 can identify the posture, gestures, etc. of the user 102 by collecting information, and then perform corresponding interactions based on the identified user posture and gestures.

[0067] Figure 1B A schematic diagram of an exemplary head-mounted wearable device 104 is shown.

[0068] like Figure 1B As shown, the head-mounted wearable device 104 may include a lens barrel 1042, which may house a display screen 1044 for displaying images and an optical component 1046 for processing light. Optionally, the optical component 1046 may further include multiple lenses (e.g., lenses 1046A and 1046B). The combination of multiple lenses can project the light emitted from the display screen 1044 into the eye 1022, allowing the eye 1022 to view the image displayed on the display screen 1044. It is understood that... Figure 1B The diagram only shows a single-sided structure of the head-mounted wearable device 104 as an example. In order to achieve binocular display, the head-mounted wearable device 104 may include two lens barrel structures arranged side by side.

[0069] In some embodiments, such as Figure 1B As shown, the head-mounted wearable device 104 may also be equipped with a camera 1048 for acquiring images of the eyes. The camera 1048 may be a charge-coupled device (CCD) image sensor, a complementary metal-oxide-semiconductor (CMOS) image sensor, etc.

[0070] Optionally, the camera 1048 can be an eye-tracking (ET) camera, whose captured eye images can be used to achieve functions such as interpupillary distance estimation and eye-tracking.

[0071] like Figure 1B As shown, in related technologies, the camera 1048 is typically mounted outside the lens barrel, and usually only one camera is installed per lens barrel. Furthermore, in order to better capture a complete image of the eye without interfering with the eye's viewing of the display screen 1022, the camera 1048 is commonly deployed at the outer corner of the eye or the wing of the nose. (Reference) Figure 1B As shown, if camera 1048 is close to the outside of the device, then Figure 1B The camera deployment position shown is at the outer corner of the eye. If camera 1048 is closer to the inside of the device, then... Figure 1B The camera deployment location shown is at the nose wing position.

[0072] However, the inventors of this disclosure have discovered that the way the camera is mounted in the related art tends to result in a large mounting angle of the camera 1048 relative to the eye 1022, which leads to a large angle γ between the orientation of the camera 1048 and the frontal viewing direction of the eye 1022, making it difficult for the acquired eye image to reflect the frontal viewing angle of the eye.

[0073] In some cases, users may need to wear glasses before using the head-mounted wearable device 104. However, because the camera 1048 is positioned outside the lens barrel 1042, it protrudes above the lens barrel 1042, which can easily compress the glasses and affect the wearing comfort of the head-mounted wearable device 104. Simultaneously, the image captured by the camera 1048 is easily affected by the edges of the glasses. Light refraction at the edges of the glasses reduces the image sharpness of the camera 1048 and creates numerous refracted light spots in the image, thus affecting the accuracy of subsequent algorithms.

[0074] In view of this, the present disclosure provides a wearable device that can solve or partially solve the above problems by setting a camera inside the lens barrel.

[0075] Figure 2 A schematic diagram of an exemplary wearable device 200 provided in an embodiment of this disclosure is shown.

[0076] like Figure 2 As shown, similarly, the wearable device 200 may also include a lens barrel 202, and a display screen 204 and an optical component 206 disposed inside the lens barrel 202. The optical component 206 may further include multiple lenses (e.g., lenses 206A and 206B), and the combination of multiple lenses can project the light emitted from the display screen 204 into the eye 1022, so that the eye 1022 can view the image displayed on the display screen 204.

[0077] and Figure 1B Unlike the wearable device 104 shown, the camera 208 of the wearable device 200 is located inside the lens barrel 202 and faces the light-emitting side of the lens barrel. Because the camera 208 is placed inside the lens barrel 202, it does not interfere with the wearing of the glasses, thereby improving the comfort of the wearable device 200. Meanwhile, as... Figure 2As shown, because the camera 208 is located inside the lens barrel 202, the distance between the camera 208 and the eye 1022 is increased, resulting in a smaller mounting angle of the camera 208 relative to the eye 1022. Consequently, the angle β between the orientation of the camera 208 and the frontal gaze direction of the eye 1022 is reduced, giving the camera 208 a better viewing angle. The acquired eye image better reflects the eye's frontal gaze angle, resulting in better image quality. Furthermore, since the camera 208 is placed inside the lens barrel 202, the glasses do not interfere with the imaging of the camera 208, further improving image quality. This improved image quality also enhances the accuracy of algorithms such as interpupillary distance estimation or gaze tracking.

[0078] In some embodiments, such as Figure 2 As shown, the wearable device 200 may further include a point light source 212. The light emitted by the point light source 212 can form reflected light on the cornea of ​​the eye 1022, which is then collected by the camera 208 and forms a light spot in the image obtained by the camera 208. Based on the position of the light spot in the image, it can be used to assist in locating the center of the cornea and the position of the pupil. In some embodiments, to make the positioning more accurate, the number of point light sources 212 may be two or more. Optionally, the point light source 212 may be a light-emitting diode (LED). In some embodiments, the light emitted by the point light source 212 may not be visible light (e.g., infrared light) or may be relatively weak visible light, thereby improving user comfort.

[0079] Understandable. Figure 2 The image only shows a single-sided structure of the wearable device 200 as an example. In order to achieve binocular display, the wearable device 200 may include two lens barrel structures arranged side by side.

[0080] Wearable devices with calibrated camera parameters can perform processing operations such as determining pupil position, estimating interpupillary distance, and tracking gaze by combining images captured by the built-in camera with camera parameters.

[0081] like Figure 2 As shown, an exemplary method for determining the pupil position is to generate a light spot in the image using a point light source 212, and then calculate the pupil position based on the position of the light spot in the image, camera parameters, and other data, combined with optical principles. This calculation method is also known as the pupil-corneal reflex method. However, the inventors of this disclosure have discovered certain problems with this calculation method.

[0082] First, the pupillary-corneal reflection method has certain requirements regarding camera characteristics. As mentioned earlier, when the camera 208 is placed inside the lens barrel 202, the optical components 206 cause distortion due to their influence on the optical path, making it impossible to approximate the image and camera parameters acquired by the camera 208 using a central camera model. However, it is difficult to directly apply the pupillary-corneal reflection method to off-center cameras. In other words, to determine the pupil position using the pupillary-corneal reflection method, the camera 208 must be placed outside the lens barrel 202.

[0083] Secondly, the pupillary corneal reflex method has certain requirements on the number of light spots formed on the eye. As wearable devices become thinner and lighter, the exit pupil distance of the eye becomes smaller and smaller, and it becomes increasingly difficult for the point light source 212 to form an effective light spot on the eye. The success rate and accuracy of this method are significantly reduced.

[0084] Third, this solution is not suitable for users wearing glasses, as glasses cause deviations in the light emitted by the point light source 212 and the light received by the camera. These two deviations result in a very serious decrease in the accuracy of pupil position and interpupillary distance estimation.

[0085] In view of this, the present disclosure provides a method for determining the pupil center, which uses images captured by at least two cameras installed in a wearable device to capture images of the same eye, and can calculate the pupil center more accurately without using the pupil-corneal reflection method (without setting a light source to generate a light spot).

[0086] Figure 3A A schematic diagram of an exemplary wearable device 300 provided in an embodiment of the present disclosure is shown.

[0087] like Figure 3A As shown, the wearable device 300 may include a first lens barrel 302A and a second lens barrel 302B, corresponding to a first eye (e.g., the left eye) and a second eye (e.g., the right eye), respectively. Corresponding to the first lens barrel 302A, the wearable device 300 may be equipped with a first camera 304A and a second camera 306A; corresponding to the second lens barrel 302B, the wearable device 300 may be equipped with a third camera 304B and a fourth camera 306B. Optionally, the first camera 304A and the second camera 306A may be located inside or outside the first lens barrel 302A and can be used to capture images of the first eye. Similarly, the third camera 304B and the fourth camera 306B may be located inside or outside the second lens barrel 302B and can be used to capture images of the second eye.

[0088] This disclosure embodiment can determine the pupil position of a corresponding eye using images captured by different cameras. Therefore, to ensure algorithm accuracy, the two cameras corresponding to the same eye should not be too close together. It is also best to avoid placing the camera in the area above the eye, as this area is easily obscured by eyelashes, which will also affect accuracy. For example, as... Figure 3A As shown, the first camera 304A is positioned near the lower part of the first lens barrel 302A, and the second camera 306A is positioned near the left side of the first lens barrel 302A. Similarly, the third camera 304B is positioned near the lower part of the second lens barrel 302B, and the fourth camera 306B is positioned near the right side of the second lens barrel 302B.

[0089] In some embodiments, to make the captured eye images clearer, the wearable device 300 may be provided with a first light source and a second light source corresponding to the first lens barrel 302A and the second lens barrel 302B, respectively. It is understood that the first light source and the second light source only need to illuminate their corresponding eye areas and can be placed anywhere on the wearable device 300; therefore, their positions are not limited and are not shown in the schematic diagram.

[0090] In some embodiments, to ensure comfort, the first light source and the second light source may be infrared light sources, and correspondingly, the first camera 304A, the second camera 306A, the third camera 304B and the fourth camera 306B may be infrared cameras.

[0091] In some embodiments, a larger number of cameras can be used to improve algorithm accuracy. For example... Figure 3B As shown, the wearable device 300 also includes a fifth camera 308A corresponding to the first lens barrel 302A and a sixth camera 308B corresponding to the second lens barrel 302B. To ensure algorithm accuracy, the fifth camera 308A is also spaced a certain distance from the first camera 304A and the second camera 306A, for example, as... Figure 3B As shown, the fifth camera 308A can be positioned near the right side of the first lens barrel 302A. Similarly, the sixth camera 308B also has a certain distance from the third camera 304B and the fourth camera 306B, for example, as... Figure 3B As shown, the sixth camera 308B can be positioned near the left side of the second lens barrel 302B.

[0092] Furthermore, this disclosure provides a method for determining the pupil center based on a wearable device. By placing at least two cameras inside or outside the lens barrel corresponding to each eye, and using the eye images acquired by the at least two cameras in conjunction with the light source inside or outside the lens barrel of each eye, the position of the pupil center can be determined with high success rate and high accuracy, and the pupillary distance measurement function can be realized.

[0093] This method for determining the pupil center does not require calibration of the light source; it only requires calibration of the camera parameters.

[0094] In some embodiments, if the camera is placed inside the lens barrel, the camera will pass through a complex and difficult-to-parameterize optical system, the centrality of the camera will be destroyed, and traditional camera models will no longer be applicable. Therefore, non-parametric camera models can be used to calibrate the camera parameters.

[0095] The calibrated camera parameters are as follows: Figure 3C As shown, the camera parameters represent the spatial equations of straight lines passing through pixels in the image captured by the camera, reflecting the spatial projection direction of the image pixels. Specifically, the camera parameters characterize the target pixels in the image captured by the camera and the corresponding projection direction. Optionally, the camera parameters can characterize a one-to-one correspondence between pixels and projection directions. Specific calibration methods are not limited here; any implementation that can calibrate the camera parameters of this disclosure is applicable.

[0096] In some embodiments, if the camera is placed outside the lens barrel, similar camera parameters can be calibrated in the same way to represent the spatial linear equations of pixels in the image captured by the camera.

[0097] Thus, after the camera parameters are calibrated, each camera (first camera 304A, second camera 306A, third camera 304B, fourth camera 306B, fifth camera 308A, and sixth camera 308B) corresponds to a set of camera parameters, which represent the spatial projection equation of the pixels in the image acquired by the camera.

[0098] Figure 4A A flowchart illustrating an exemplary method 400 provided in an embodiment of this disclosure is shown.

[0099] This method 400 can be used to determine the pupil center and can further determine the interpupillary distance. For example... Figure 4A As shown, the method 400 may include the following steps.

[0100] Taking the determination of the position of the pupil center of the first eye (e.g., the left eye) as an example, in step 402, the first eye image 502 captured by the first camera 304A can be obtained (e.g., ...). Figure 5A (as shown) and the second eye image 504 acquired by the second camera 306A (as shown) Figure 5B (As shown).

[0101] In some embodiments, an eye-opening judgment model can be used to determine whether the eyes in the first eye image 502 and the second eye image 504 are open. If the eyes are closed, the images are repeatedly acquired until a valid eye-opening image is obtained.

[0102] In step 404, the first pixel 5022 and the second pixel 5042 corresponding to the center of the pupil are determined from the first eye image 502 and the second eye image 504, respectively.

[0103] like Figure 5A As shown, the first eye image 502 is an image of the first eye captured by the first camera 304A at its position. The eye image is represented by an ellipse in the figure. Optionally, the first pixel 5022 corresponding to the center of the pupil of the first eye can be identified from the first eye image 502 using an image recognition algorithm. Figure 5B As shown, the second eye image 504 is an image of the first eye taken by the second camera 306A at its position. Similarly, the second pixel 5042 corresponding to the center of the pupil of the first eye can also be identified from the second eye image 504.

[0104] In step 406, based on the position of the first pixel in the first eye image, the first target camera parameters of the first camera are determined. The first target camera parameters include the first spatial projection direction of the first pixel.

[0105] As mentioned earlier, camera parameters represent the equation of a straight line passing through the corresponding pixel, reflecting the spatial projection direction of the image pixel. Therefore, after determining the position of the first pixel 5022 in the first eye image 502, the corresponding first target camera parameters, which represent the equation of a straight line passing through the first pixel 5022, can be determined from the pre-calibrated camera parameter set of the first camera 304A.

[0106] In step 408, based on the position of the second pixel in the second eye image, the second target camera parameters of the second camera are determined. The second target camera parameters include the second spatial projection direction of the second pixel.

[0107] Similarly, once the position of the second pixel 5042 in the second eye image 504 is determined, the corresponding second target camera parameters can be determined from the camera parameter set that has been calibrated by the second camera 306A. These second target camera parameters represent the equation of a straight line passing through the second pixel 5042.

[0108] In step 410, the spatial position of the pupil center is determined based on the first target camera parameters and the second target camera parameters.

[0109] Figure 5C A schematic diagram of a calculation model 506 for determining the pupil center based on camera parameters according to an embodiment of the present disclosure is shown.

[0110] like Figure 5C As shown, the parameters of the first target camera can be mapped to a straight line 5062 in three-dimensional space, and the parameters of the second target camera can be mapped to a straight line 5064 in three-dimensional space.

[0111] It can be understood that line 5062 is a spatial projection line passing through the first pixel 5022, representing the projection relationship of the first pixel 5022 from the camera coordinate system of the first camera 304A to the spatial coordinate system. Line 5064 is a spatial projection line passing through the second pixel 5042, representing the projection relationship of the second pixel 5042 from the camera coordinate system of the second camera 306A to the spatial coordinate system. Since the first pixel 5022 and the second pixel 5042 are the pupil centers observed by the first camera 304A and the second camera 306A respectively, the intersection point Q of lines 5062 and 5064 can be approximated as the pupil center.

[0112] In this step, if the accuracy allows, the spatial coordinates of the intersection point Q can be determined as the spatial location of the center of the pupil of the first eye.

[0113] In some embodiments, lines 5062 and 5064 may not strictly intersect (i.e., there is no actual intersection point), and the point that minimizes the sum of the squares of the straight-line distances from them can be used as the intersection point Q.

[0114] In some embodiments, the user may wear glasses. Since glasses can affect the optical path, they need to be corrected. Therefore, the method 400 may also include obtaining adjustment parameters input by the user (e.g., glasses power, astigmatism power / axis, etc.).

[0115] Step 410 may further include: adjusting the parameters of the first target camera and the second target camera according to the adjustment parameters; and determining the spatial position of the pupil center according to the adjusted parameters of the first target camera and the second target camera.

[0116] In this step, the calibrated camera parameters (or spatial straight line) are used as the incident vector, which is then incident from the outside of the glasses. The exit vector of the glasses on the eye side can be obtained based on the optical parameters of the glasses themselves (e.g., glasses power, astigmatism power / axis, etc.). The exit vector is used as the new observation result, namely: the adjusted first target camera parameters and the second target camera parameters.

[0117] Essentially, a new camera model is formed by the user's glasses lenses, the optical components of the wearable device, and the camera. This approach can largely eliminate the impact of the user's glasses on the accuracy of the algorithm.

[0118] In some embodiments, a detection device may be included in the wearable device 300 to detect whether the user is wearing glasses. Specifically, this detection device emits and receives detection light. Since eyeglass lenses are lenses, the difference in angle between the emitted and received light can be used to determine whether the optical path of the detection light has changed, thereby determining whether the user is wearing glasses. Furthermore, by performing corresponding optical calculations based on the emitted and received light, information such as the eyeglass prescription and astigmatism / axis can be determined. Therefore, the adjustment parameters may not be input by the user but can be obtained by the detection device.

[0119] Because the pupil is inside the cornea, light is refracted as it passes through the surface of the cornea. For example... Figure 5C As shown, lines 5062 and 5064 refract at points A and B on the corneal surface, and the refracted rays 5066 and 5068 intersect at point P, which is the location of the true pupil center. Therefore, the location P of the true pupil center and the intersection point Q of the two lines usually do not coincide. Therefore, in some embodiments, the positional deviation between location P and intersection point Q can be corrected.

[0120] In some embodiments, such as Figure 4B As shown, determining the spatial position of the pupil center based on the parameters of the first target camera and the parameters of the second target camera may further include:

[0121] In step 4102, the spatial position to be corrected (i.e., intersection point Q) of the pupil center is determined based on the first target camera parameters and the second target camera parameters.

[0122] In step 4104, a correction parameter is obtained. This correction parameter is used to correct the spatial position of the pupil center to be corrected, thereby obtaining the corrected position P of the pupil center.

[0123] In some embodiments, the correction parameters may be determined through preprocessing, for example, by simulating the actual situation using a simulation model in advance and determining them based on the simulation results.

[0124] In some embodiments, the correction parameters are determined based on a mapping relationship, which is obtained by fitting multiple pairs of spatial projection directions corresponding to multiple first spatial projection directions and multiple second spatial projection directions, as well as multiple correction parameters. The spatial projection direction pair can refer to a direction pair formed by a single first spatial projection direction and its corresponding single second spatial projection direction; that is, the first spatial projection direction and the second spatial projection direction can have a one-to-one correspondence.

[0125] Since the area around the eyes where the displayed content is clearest after wearing a wearable device, in some embodiments, the correction parameters are determined based on the probability distribution of corneal radius, the probability distribution of distance from the pupil center to the corneal center, the viewing angle distribution, and the refractive index distribution of the eyeball within the target space. Using these parameters to build a simulation model and then determining the correction parameters based on the simulation results allows for better correction of the pupil center position. The target space can be an area close to the eye where the clarity of the displayed image observed by the eye is greater than a clarity threshold. As an optional embodiment, the target space can be determined based on the eyebox. Since the eyebox is the cone-shaped area between the optical module and the eye where the displayed content is clearest, building a simulation model based on the eyebox parameters allows for better correction of the pupil center position.

[0126] The following is combined with Figure 5C The computational model 506 shown illustrates an example of determining trimming parameters through a simulation model.

[0127] In this embodiment, the deviation correction vector from Q to P can be defined as follows, where x and y are negative real correction parameters.

[0128]

[0129] According to the above formula, the correction parameters are related to the corneal radius, the distance from the pupil center to the corneal center, and the camera's observation direction, making them difficult to solve directly.

[0130] However, the inventors of this disclosure have discovered that a set of approximate x and y values ​​can be obtained through simulation and used as correction parameters.

[0131] Assume x and y are the angle α between lines 5062 and 5064, and the spatial location of the intersection point Q (Q x Q y Q z The related function is: x,y=f(α,Q) x Q y Q z ).

[0132] like Figure 4C As shown, in some embodiments, the method 400 may include the following steps.

[0133] In step 412, within the moving orbit area (Eyebox), multiple corneal center positions and multiple pupil center positions are sampled according to the probability distribution of corneal radius, the probability distribution of distance from pupil center to corneal center, and the distribution of viewing angle.

[0134] The eyebox typically refers to a cone-shaped area between the optical module and the eyeball, which is also the area where the displayed content is clearest. A larger eyebox results in better mechanical tolerance for the wearable device, allowing it to accommodate people with different interpupillary distances. Generally, the eyebox can be determined based on design parameters during the design of the wearable device's lens barrel. Therefore, during simulation, the lens barrel design parameters can be directly input into the simulation system to determine the eyebox.

[0135] The probability distribution of corneal radius can be determined based on multiple different eyeball models, and this probability distribution can reflect the distribution of corneal radius in most groups. Similarly, the probability distribution of the distance from the pupil center to the corneal center can be determined in the same way.

[0136] The distribution of viewing angles can be determined based on the distribution of viewing angles that may exist when a user uses a wearable device, and can be determined by statistically analyzing the actual usage of a certain group. In some embodiments, it can be determined based on the field of view (FOV) of the wearable device.

[0137] In this step, after obtaining the probability distribution of the above parameters, these parameters can be sampled. The sampling method can be arbitrary and is not restricted here.

[0138] Then, the corneal center position and pupil center position corresponding to each set of sampling data can be calculated based on the sampling results. In this way, multiple corneal center positions and multiple pupil center positions corresponding to the corneal center positions can be obtained based on multiple sets of sampling data.

[0139] In step 414, multiple corneal refractive indices are sampled according to the refractive index distribution of the eyeball.

[0140] Alternatively, the refractive index distribution of the eyeball can be determined by statistically analyzing the refractive index distribution of the eyeballs of a certain population.

[0141] In this step, based on the refractive index distribution of the eyeball, any sampling method can be used to sample multiple corneal refractive indices at multiple locations on the eyeball.

[0142] In step 416, simulations are performed to obtain multiple eye images captured by the first camera 304A and the second camera 306A at different camera positions.

[0143] In this step, for each simulation, based on the parameters obtained from the previous sampling, a set of parameters (e.g., corneal center position m1, pupil center position m2, corneal refractive index m3) is selected and fixed. Then, the camera position m4 is continuously adjusted to acquire multiple eye images at different camera positions.

[0144] In this way, each eye image corresponds to a set of parameters: corneal center position m1, pupil center position m2, corneal refractive index m3, and camera position m4.

[0145] It is understood that the camera position m4 mentioned in this step refers to the overall position of the camera group. In this embodiment, the camera position m4 can be understood as the location of the center of the first camera 304A and the second camera 306A, and the relative positions of the first camera 304A and the second camera 306A with respect to this center remain unchanged. It is understood that when the camera group consists of three or more cameras, this center can be the common center position of the three or more cameras.

[0146] In step 418, the correction parameters are calculated based on the multiple eye images, combined with the multiple corneal center positions, multiple pupil center positions, and multiple corneal refractive indices.

[0147] As mentioned earlier, each eye image corresponds to a set of fixed parameters (known parameters): corneal center position m1, pupil center position m2, corneal refractive index m3, and camera position m4. Therefore, based on the pixel position corresponding to the pupil center in the eye image, combined with these fixed parameters, the correction parameters can be calculated.

[0148] In this way, by using the simulation model, a set of approximate x and y can be obtained as correction parameters to correct the spatial position of the pupil center (i.e., the intersection point Q) to obtain coordinates that approximate the true spatial position of the pupil center, thereby improving the accuracy of the algorithm.

[0149] In some embodiments, such as Figure 4D As shown, the correction parameters are calculated based on the multiple eye images, combined with the multiple corneal center positions, multiple pupil center positions, and multiple corneal refractive indices, and may further include the following steps.

[0150] In step 4182, the first target image and the second target image captured by the first camera and the second camera at the target position are obtained from the multiple eye images.

[0151] Figure 5D A schematic diagram of an exemplary simulation model 508 according to an embodiment of the present disclosure is shown.

[0152] like Figure 5D As shown, it is assumed that a set of parameters corresponding to the acquisition of the first target image and the second target image can constitute a simulation model 508. Under this model 508, the corneal center position, the pupil center position P', the corneal refractive index n, and the target position of the camera are all known.

[0153] Figure 5E and Figure 5F Schematic diagrams of a first target image 510 and a second target image 512 according to embodiments of the present disclosure are shown respectively.

[0154] like Figure 5E and Figure 5F As shown, the first target pixel 5102 and the second target pixel 5122 corresponding to the pupil center in the first target image 510 and the second target image 512 can be obtained by identification.

[0155] In step 4184, based on the corneal center position, pupil center position P', and corneal refractive index corresponding to the acquisition of the first target image 510 and the second target image 512, and combining the first target pixel 5102 and the second target pixel 5122 corresponding to the pupil center in the first target image 510 and the second target image 5122, a first straight line 5082 passing through the first target pixel 5102 and a second straight line 5084 passing through the second target pixel 5122, as well as the first refraction point A' and the second refraction point B' of the first straight line 5082 and the second straight line 5084 on the corneal surface, are determined.

[0156] In this step, similar to the previous embodiment, given the known camera parameters, the first straight line 5082 and the second straight line 5084 can be determined based on the positions of the first target pixel 5102 and the second target pixel 5122 in the first target image 510 and the second target image 512.

[0157] Furthermore, given the known corneal center position, pupil center position P', and corneal refractive index, and combining the first straight line 5082 and the second straight line 5084, the first refraction point A' and the second refraction point B' on the corneal surface of the first straight line 5082 and the second straight line 5084 can be calculated.

[0158] In step 4186, the intersection point Q' of the first line and the second line is determined.

[0159] Knowing the first line 5082 and the second line 5084, we can determine their intersection point Q' in three-dimensional space.

[0160] It is understandable that when the first line 5082 and the second line 5084 do not strictly intersect, the point where the sum of the squares of their linear distances is minimized can be taken as the intersection point Q'.

[0161] In step 4188, the reference correction parameters corresponding to the target position are determined based on the first refraction point A', the second refraction point B', the intersection point Q', the first straight line 5082 and the second straight line 5084, and the pupil center position P' corresponding to the acquisition of the first target image 510 and the second target image 512.

[0162] In this step, the following formula is used:

[0163]

[0164] By combining the first refraction point A', the second refraction point B', the intersection point Q', the first straight line 5082 and the second straight line 5084, and the pupil center position P' corresponding to the acquisition of the first target image 510 and the second target image 512, a set of x0 and y0 can be obtained.

[0165] In step 4190, the correction parameter is determined based on the reference correction parameter.

[0166] In this step, the correction parameters x and y can be further determined based on x0 and y0.

[0167] It is understandable that since x0 and y0 are only calculated based on a specific set of parameters when the camera is at the target position, as mentioned earlier, the simulation model uses many parameters. In order to ensure the accuracy of the algorithm, it is necessary to further determine the correction parameters x and y based on the reference correction parameters obtained from simulations of multiple sets of different parameters.

[0168] Therefore, in some embodiments, determining the correction parameter based on the reference correction parameter includes:

[0169] Based on the reference correction parameter, the intersection point, the first line, and the second line, determine the mapping relationship between the reference correction parameter, the intersection point, and the angle α between the lines (which can be calculated based on the first line 5082 and the second line 5084). For example, (α, q x ,q y ,q z The mapping relationship between (x0, y0) and (x0, y0) is determined; then, the correction parameter is determined based on the mapping relationship.

[0170] Furthermore, in some embodiments, the mapping relationship includes multiple mapping relationships corresponding to multiple different target locations. Determining the correction parameter based on the mapping relationship includes: fitting the correction parameter based on the multiple mapping relationships. For example, using a model (including but not limited to multinomial regression, deep neural network) to fit (α, q) obtained from multiple simulations. x ,q y ,q z The mapping relationship from (x,y) to (x,y).

[0171] Thus, we can obtain (α, q) x ,q y ,q z The mapping relationship from (x,y) to (x,y).

[0172] In step 4106, the spatial position of the pupil center is determined based on the first target camera parameters, the second target camera parameters, the spatial position to be corrected of the pupil center, and the correction parameters.

[0173] Specifically, in step 4104, the angle between lines 5062 and 5064 can be determined based on the parameters of the first target camera and the parameters of the second target camera. Then, the corresponding correction parameters x and y are found based on the angle and the spatial position Q of the pupil center to be corrected.

[0174] In this step, the spatial position of the pupil center can be determined based on the parameters of the first target camera, the parameters of the second target camera, the spatial position to be corrected of the pupil center, and the correction parameters.

[0175] In some embodiments, determining the spatial position of the pupil center based on the first target camera parameters, the second target camera parameters, the spatial position to be corrected of the pupil center, and the correction parameters further includes:

[0176] Based on the first target camera parameters, the second target camera parameters, and the correction parameters, determine the offset vector of the spatial position to be corrected of the pupil center relative to the spatial position of the pupil center; for example, calculate the correction vector based on the correction parameters.

[0177] Based on the spatial position to be corrected of the pupil center and the offset vector, the spatial position of the pupil center is determined, that is, the estimated pupil coordinates are obtained.

[0178] Therefore, based on the first eye image 502 and the second eye image 504 acquired by the first camera 304A and the second camera 306A, the spatial position of the pupil center of the first eye in three-dimensional space can be determined.

[0179] For cases with three or more cameras, the pupil position can be calculated first for any two camera combinations, and then the average of all combinations can be taken to obtain the final pupil position result.

[0180] It is understandable that, following a similar approach to method 400, the center of the pupil of the second eye can also be determined.

[0181] like Figure 4E As shown, in some embodiments, the method 400 may further include the following steps:

[0182] Taking the determination of the position of the pupil center of the second eye (e.g., the right eye) as an example, in step 420, the third eye image captured by the third camera 304B and the fourth eye image captured by the fourth camera 306B are acquired.

[0183] In some embodiments, an eye-opening judgment model can be used to determine whether the eyes in the third and fourth eye images are open. If the eyes are closed, the images are repeatedly acquired until a valid eye-opening image is obtained.

[0184] In step 422, the third and fourth pixels corresponding to the center of the second pupil of the second eye are determined from the third and fourth eye images, respectively. Optionally, the third and fourth pixels corresponding to the center of the pupil of the second eye can be identified from the third and fourth eye images using an image recognition algorithm.

[0185] In step 424, the third target camera parameters of the third camera are determined based on the position of the third pixel in the third eye image. The third target camera parameters include the third spatial projection direction of the third pixel.

[0186] As mentioned earlier, camera parameters represent the equation of a spatial straight line passing through the corresponding pixel, reflecting the spatial projection direction of the image pixel. Therefore, once the position of the third pixel in the third eye image is determined, the corresponding third target camera parameters, which represent the equation of a spatial straight line passing through the third pixel, can be determined from the pre-calibrated camera parameter set of the third camera 304B.

[0187] In step 426, the fourth target camera parameters of the fourth camera are determined based on the position of the fourth pixel in the fourth eye image. The fourth target camera parameters include the fourth spatial projection direction of the fourth pixel.

[0188] Similarly, once the position of the fourth pixel in the fourth eye image is determined, the corresponding fourth target camera parameters can be determined from the pre-calibrated camera parameter set of the fourth camera 306B. These fourth target camera parameters represent the equation of a spatial straight line passing through the fourth pixel.

[0189] In step 428, the second spatial position of the center of the second pupil is determined based on the parameters of the third target camera and the parameters of the fourth target camera.

[0190] The implementation of step 428 is similar to that of step 410, and the process of building the simulation model is also similar, so it will not be described again here.

[0191] In this way, the spatial position of the center of the second eye's pupil in three-dimensional space can be obtained.

[0192] As can be seen from the above embodiments, the present disclosure provides a method for determining the pupil center. By using images captured by at least two cameras installed in a wearable device for acquiring eye images of the same eye, the pupil center can be calculated relatively accurately without using the pupil-corneal reflection method. Based on this, the pupil centers of the two eyes can be determined, and thus the interpupillary distance can be determined.

[0193] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0194] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0195] This disclosure also provides a method for determining interpupillary distance based on a wearable device. The wearable device includes a first target camera and a second target camera for acquiring eye images of a first eye, and a third target camera and a fourth target camera for acquiring eye images of a second eye.

[0196] Figure 4F A flowchart illustrating another exemplary method 430 provided in an embodiment of this disclosure is shown.

[0197] This method 430 can be used to determine interpupillary distance. For example... Figure 4F As shown, the method 430 may further include the following steps.

[0198] In step 432, the first spatial position of the first pupil center of the first eye and the second spatial position of the second pupil center of the second eye, determined by any embodiment or arrangement / combination of embodiments of method 400, can be obtained. The method for determining the first and second spatial positions refers to the embodiments of method 400 and will not be repeated here.

[0199] In step 434, the interpupillary distance is determined based on the first spatial position (the spatial position of the center of the pupil of the first eye in three-dimensional space) and the second spatial position (the spatial position of the center of the pupil of the second eye in three-dimensional space).

[0200] Once the interpupillary distance (IPD) is determined, the lens barrel can be adjusted to match the user's IPD, increasing comfort and improving the user experience.

[0201] This disclosure also provides a computer device for implementing the above-described method 400 or method 430. Figure 6 A schematic diagram of the hardware structure of an exemplary computer device 600 provided in an embodiment of this disclosure is shown. The computer device 600 can be used to implement... Figure 1A , Figure 1B 104. Head-mounted wearable devices Figure 2 Wearable devices 200 Figure 3A and Figure 3B The wearable device 300 can also be used to achieve Figure 1A External device 112 can also be used to achieve Figure 1A Server 114. In some scenarios, this computer device 600 can also be used to implement... Figure 1A Database server 116.

[0202] like Figure 6 As shown, the computer device 600 may include: a processor 602, a memory 604, a network module 606, a peripheral interface 608, and a bus 610. The processor 602, memory 604, network module 606, and peripheral interface 608 are interconnected within the computer device 600 via the bus 610.

[0203] Processor 602 may be a central processing unit (CPU), image processor, neural network processor (NPU), microcontroller (MCU), programmable logic device, digital signal processor (DSP), application-specific integrated circuit (ASIC), or one or more integrated circuits. Processor 602 can be used to perform functions related to the techniques described in this disclosure. In some embodiments, processor 602 may also include multiple processors integrated as a single logic component. For example, such as... Figure 6 As shown, processor 602 may include multiple processors 602a, 602b and 602c.

[0204] Memory 604 can be configured to store data (e.g., instructions, computer code, etc.). Figure 6 As shown, the data stored in memory 604 may include program instructions (e.g., program instructions for implementing method 400 or method 430 of the embodiments of this disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). Processor 602 may also access the program instructions and data stored in memory 604 and execute the program instructions to operate on the data to be processed. Memory 604 may include volatile storage devices or non-volatile storage devices. In some embodiments, memory 604 may include random access memory (RAM), read-only memory (ROM), optical disk, magnetic disk, hard disk, solid-state drive (SSD), flash memory, memory stick, etc.

[0205] Network interface 606 can be configured to provide communication with other external devices to computer device 600 via a network. This network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, Near Field Communication (NFC), etc.), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the specific examples described above.

[0206] The peripheral interface 608 can be configured to connect the computer device 600 to one or more peripheral devices to enable information input and output. For example, peripheral devices may include input devices such as keyboards, mice, touchpads, touch screens, microphones, and various sensors, as well as output devices such as displays, speakers, vibrators, and indicator lights.

[0207] Bus 610 can be configured to transfer information between various components of computer device 600 (e.g., processor 602, memory 604, network interface 606, and peripheral interface 608), such as internal buses (e.g., processor-memory bus), external buses (USB port, PCI-E bus), etc.

[0208] It should be noted that although the architecture of the computer device 600 described above only shows the processor 602, memory 604, network interface 606, peripheral interface 608, and bus 610, in specific implementations, the architecture of the computer device 600 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the architecture of the computer device 600 described above may only include the components necessary for implementing the embodiments of this disclosure, and does not necessarily include all the components shown in the figures.

[0209] This disclosure also provides a pupil center determination device based on a wearable device. Figure 7 A schematic diagram of an exemplary apparatus 700 provided in an embodiment of this disclosure is shown. For example... Figure 7 As shown, the device 700 can be used to implement method 400 and may further include the following modules.

[0210] The acquisition module 702 is configured to acquire a first eye image captured by the first camera and a second eye image captured by the second camera;

[0211] The first determining module 704 is configured to: determine the first pixel point and the second pixel point corresponding to the pupil center from the first eye image and the second eye image, respectively;

[0212] The second determining module 706 is configured to: determine the first target camera parameters of the first camera based on the position of the first pixel in the first eye image, wherein the first target camera parameters include the first spatial projection direction of the first pixel;

[0213] The third determining module 708 is configured to: determine the second target camera parameters of the second camera based on the position of the second pixel in the second eye image, wherein the second target camera parameters include the second spatial projection direction of the second pixel;

[0214] The fourth determining module 710 is configured to: determine the spatial position of the pupil center based on the parameters of the first target camera and the parameters of the second target camera.

[0215] In some embodiments, the fourth determining module 710 is configured to:

[0216] Based on the parameters of the first target camera and the parameters of the second target camera, determine the spatial position to be corrected for the center of the pupil;

[0217] Obtain the correction parameters;

[0218] The spatial position of the pupil center is determined based on the parameters of the first target camera, the parameters of the second target camera, the spatial position of the pupil center to be corrected, and the correction parameters.

[0219] In some embodiments, the correction parameters are determined based on a mapping relationship, which is obtained by fitting multiple pairs of spatial projection directions corresponding to multiple first spatial projection directions and multiple second spatial projection directions, as well as multiple correction parameters.

[0220] In some embodiments, the fourth determining module 710 is configured to:

[0221] Based on the first target camera parameters, the second target camera parameters, and the correction parameters, determine the offset vector of the spatial position to be corrected of the pupil center relative to the spatial position of the pupil center;

[0222] The spatial position of the pupil center is determined based on the spatial position to be corrected of the pupil center and the offset vector.

[0223] In some embodiments, the correction parameters are determined based on the probability distribution of corneal radius within the orbital region, the probability distribution of distance from pupil center to corneal center, the distribution of viewing angle, and the distribution of ocular refractive index.

[0224] In some embodiments, the apparatus further includes a simulation module configured to:

[0225] Within the orbital region, multiple corneal center locations and multiple pupil center locations are sampled according to the probability distribution of corneal radius, the probability distribution of distance from pupil center to corneal center, and the distribution of viewing angle.

[0226] Based on the refractive index distribution of the eyeball, multiple corneal refractive indices were sampled;

[0227] Simulation yielded multiple eye images captured by the first and second cameras at different camera positions;

[0228] The correction parameters are calculated based on the multiple eye images, combined with the multiple corneal center positions, multiple pupil center positions, and multiple corneal refractive indices.

[0229] In some embodiments, the simulation module is configured as follows:

[0230] Obtain the first target image and the second target image captured by the first camera and the second camera respectively at the target position from the multiple eye images;

[0231] Based on the corneal center position, pupil center position, and corneal refractive index corresponding to the acquisition of the first target image and the second target image, and combined with the first target pixel and the second target pixel corresponding to the pupil center in the first target image and the second target image, a first straight line passing through the first target pixel and a second straight line passing through the second target pixel, as well as the first refraction point and the second refraction point of the first straight line and the second straight line on the corneal surface are determined;

[0232] Determine the intersection point of the first line and the second line;

[0233] Based on the first refraction point, the second refraction point, the intersection point, the first straight line and the second straight line, and the pupil center position corresponding to the acquisition of the first target image and the second target image, the reference correction parameter corresponding to the target position is determined;

[0234] The correction parameters are determined based on the reference correction parameters.

[0235] In some embodiments, the simulation module is configured as follows:

[0236] Based on the reference correction parameter, the intersection point, the first line, and the second line, determine the mapping relationship between the reference correction parameter and the intersection point and the angle between the lines;

[0237] The correction parameters are determined based on the mapping relationship.

[0238] In some embodiments, the mapping relationship includes multiple mapping relationships corresponding to multiple different target locations, and the simulation module is configured to: fit the correction parameters according to the multiple mapping relationships.

[0239] In some embodiments, the acquisition module 702 is configured to: acquire adjustment parameters input by the user;

[0240] The fourth determining module 710 is configured to: adjust the parameters of the first target camera and the second target camera according to the adjustment parameters; and determine the spatial position of the pupil center according to the adjusted parameters of the first target camera and the second target camera.

[0241] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0242] The apparatus of the above embodiments is used to implement the corresponding method 400 in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0243] This disclosure also provides an interpupillary distance determination device based on a wearable device. The wearable device includes a first target camera and a second target camera for acquiring eye images of a first eye, and a third target camera and a fourth target camera for acquiring eye images of a second eye.

[0244] Figure 8 A schematic diagram of an exemplary apparatus 800 provided in an embodiment of this disclosure is shown. For example... Figure 8 As shown, the device 800 can be used to implement method 430 and may further include the following modules.

[0245] The acquisition module 802 is configured to acquire the first spatial position of the first pupil center of the first eye and the second spatial position of the second pupil center of the second eye, determined by any embodiment or arrangement / combination of the embodiments of method 400. The method for determining the first and second spatial positions refers to the embodiments of method 400 and will not be repeated here.

[0246] The determining module 804 is configured to determine the interpupillary distance based on the first spatial position and the second spatial position.

[0247] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0248] The apparatus of the above embodiments is used to implement the corresponding method 430 in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0249] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method 400 or method 430 as described in any of the above embodiments.

[0250] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0251] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the method 400 or method 430 as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0252] Based on the same inventive concept, corresponding to any of the above-described embodiments of method 400 or method 430, this disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processors to perform the method 400 or method 430. Corresponding to the execution entity for each step in each embodiment of method 400 or method 430, the processor executing the corresponding step may belong to the corresponding execution entity.

[0253] The computer program product of the above embodiments is used to cause the computer and / or the processor to perform the method 400 or method 430 as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0254] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0255] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuitry) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0256] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0257] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A wearable device based pupil center determination method, wherein, The wearable device includes a first camera and a second camera for acquiring eye images of the same eye, and the method includes: Acquire a first eye image captured by the first camera and a second eye image captured by the second camera; The first pixel and the second pixel corresponding to the center of the pupil are determined from the first eye image and the second eye image, respectively. Based on the position of the first pixel in the first eye image, the first target camera parameters of the first camera are determined, and the first target camera parameters include the first spatial projection direction of the first pixel. Based on the position of the second pixel in the second eye image, the second target camera parameters of the second camera are determined, and the second target camera parameters include the second spatial projection direction of the second pixel; The spatial position of the pupil center is determined based on the parameters of the first target camera and the parameters of the second target camera. The method further includes: obtaining adjustment parameters input by the user; Determining the spatial position of the pupil center based on the first target camera parameters and the second target camera parameters includes: adjusting the first target camera parameters and the second target camera parameters according to the adjustment parameters; and determining the spatial position of the pupil center based on the adjusted first target camera parameters and the second target camera parameters.

2. The method as described in claim 1, wherein, Determining the spatial position of the pupil center based on the parameters of the first target camera and the parameters of the second target camera includes: Based on the parameters of the first target camera and the parameters of the second target camera, determine the spatial position to be corrected for the center of the pupil; Obtain the correction parameters; The spatial position of the pupil center is determined based on the parameters of the first target camera, the parameters of the second target camera, the spatial position of the pupil center to be corrected, and the correction parameters.

3. The method as described in claim 2, wherein, The correction parameters are determined based on a mapping relationship, which is obtained by fitting multiple pairs of spatial projection directions corresponding to multiple first spatial projection directions and multiple second spatial projection directions, as well as multiple correction parameters.

4. The method of claim 2, wherein, Determining the spatial position of the pupil center based on the parameters of the first target camera, the parameters of the second target camera, the spatial position of the pupil center to be corrected, and the correction parameters includes: Based on the first target camera parameters, the second target camera parameters, and the correction parameters, determine the offset vector of the spatial position to be corrected of the pupil center relative to the spatial position of the pupil center; The spatial position of the pupil center is determined based on the spatial position to be corrected of the pupil center and the offset vector.

5. The method of claim 2, wherein, The correction parameters are determined based on the probability distribution of corneal radius, the probability distribution of distance from pupil center to corneal center, the distribution of viewing angle, and the distribution of eye refractive index within the orbital region.

6. The method of claim 2, wherein, The method further includes: Within the orbital region, multiple corneal center locations and multiple pupil center locations are sampled according to the probability distribution of corneal radius, the probability distribution of distance from pupil center to corneal center, and the distribution of viewing angle. Based on the refractive index distribution of the eyeball, multiple corneal refractive indices were sampled; Simulation yielded multiple eye images captured by the first and second cameras at different camera positions; The correction parameters are calculated based on the multiple eye images, combined with the multiple corneal center positions, multiple pupil center positions, and multiple corneal refractive indices.

7. The method of claim 6, wherein, Based on the multiple eye images, combined with the multiple corneal center positions, multiple pupil center positions, and multiple corneal refractive indices, the correction parameters are calculated, including: Obtain the first target image and the second target image captured by the first camera and the second camera respectively at the target position from the multiple eye images; Based on the corneal center position, pupil center position, and corneal refractive index corresponding to the acquisition of the first target image and the second target image, and combined with the first target pixel and the second target pixel corresponding to the pupil center in the first target image and the second target image, a first straight line passing through the first target pixel and a second straight line passing through the second target pixel, as well as the first refraction point and the second refraction point of the first straight line and the second straight line on the corneal surface are determined; Determine the intersection point of the first line and the second line; Based on the first refraction point, the second refraction point, the intersection point, the first straight line and the second straight line, and the pupil center position corresponding to the acquisition of the first target image and the second target image, the reference correction parameter corresponding to the target position is determined; The correction parameters are determined based on the reference correction parameters.

8. The method of claim 7, wherein, Determining the correction parameters based on the reference correction parameters includes: Based on the reference correction parameter, the intersection point, the first line, and the second line, determine the mapping relationship between the reference correction parameter and the intersection point and the angle between the lines; The correction parameters are determined based on the mapping relationship.

9. The method of claim 8, wherein, The mapping relationship includes multiple mapping relationships corresponding to multiple different target locations. Based on the mapping relationship, the correction parameter is determined, including: The correction parameters are obtained by fitting based on the multiple mapping relationships.

10. A method for determining interpupillary distance based on a wearable device, the wearable device comprising a first target camera and a second target camera for acquiring eye images of a first eye, and a third target camera and a fourth target camera for acquiring eye images of a second eye, the method comprising: Obtain the first spatial position of the first pupil center of the first eye and the second spatial position of the second pupil center of the second eye, as determined by the method as described in any one of claims 1-9; The interpupillary distance is determined based on the first spatial position and the second spatial position.

11. A pupil center determination device based on a wearable device, wherein, The wearable device includes a first camera and a second camera for acquiring images of the same eye, and the device includes: The acquisition module is configured to acquire a first eye image captured by the first camera and a second eye image captured by the second camera; The first determining module is configured to: determine the first pixel point and the second pixel point corresponding to the pupil center from the first eye image and the second eye image, respectively; The second determining module is configured to: determine the first target camera parameters of the first camera based on the position of the first pixel in the first eye image, wherein the first target camera parameters include the first spatial projection direction of the first pixel; The third determining module is configured to: determine the second target camera parameters of the second camera based on the position of the second pixel in the second eye image, wherein the second target camera parameters include the second spatial projection direction of the second pixel; The fourth determining module is configured to: determine the spatial position of the pupil center based on the parameters of the first target camera and the parameters of the second target camera; The acquisition module is configured to acquire adjustment parameters input by the user. The fourth determining module is configured to: adjust the parameters of the first target camera and the second target camera according to the adjustment parameters; and determine the spatial position of the pupil center according to the adjusted parameters of the first target camera and the second target camera.

12. A device for determining interpupillary distance based on a wearable device, the wearable device comprising a first target camera and a second target camera for acquiring eye images of a first eye, and a third target camera and a fourth target camera for acquiring eye images of a second eye, the device comprising: The acquisition module is configured to: acquire the first spatial position of the first pupil center of the first eye and the second spatial position of the second pupil center of the second eye, as determined by the method described in any one of claims 1-9; The determining module is configured to: determine the interpupillary distance based on the first spatial position and the second spatial position.

13. A computer device comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, the programs comprising instructions for performing the method of any one of claims 1-9 or the method of claim 10.

14. A non-volatile computer-readable storage medium comprising a computer program, which, when executed by one or more processors, causes the processors to perform the method as claimed in any one of claims 1-9 or the method as claimed in claim 10.

15. A computer program product comprising computer program instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1-9 or the method as claimed in claim 10.

Images