Method for determining pupil center and pupil on basis of wearable device, and related device

Abstract

The present disclosure provides a method for determining a pupil center on the basis of a wearable device, and a related device, wherein the wearable device comprises a first camera and a second camera which are configured for collecting eye images of the same eye, and the method comprises: acquiring a first eye image collected by the first camera and a second eye image collected by the second camera; determining from the first eye image and the second eye image a first pixel point and a second pixel point corresponding to the pupil center, respectively; determining, according to a position of the first pixel point in the first eye image, a first target camera parameter of the first camera; determining, according to a position of the second pixel point in the second eye image, a second target camera parameter of the second camera; and determining, according to the first target camera parameter and the second target camera parameter, a spatial position of the pupil center.

Description

Method for determining pupil center and pupil based on wearable device and related equipment

[0001] This application claims priority to the Chinese invention patent application with application number 202311774989.7 filed on December 21, 2023, entitled “Method for determining pupil center and pupil based on wearable device and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of extended reality technology, and in particular to a method for determining a pupil center and a pupil based on a wearable device and related equipment. Background Art

[0003] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds to create a virtual environment where humans and machines can interact. XR technology can further encompass augmented reality (AR), virtual reality (VR), and mixed reality (MR), leveraging hardware devices combined with various technologies to fuse virtual content with real-world scenes.

[0004] Generally, an extended reality system provides users with wearable devices to enable human-computer interaction. The wearable device can be a head-mounted wearable device. In some scenarios, the wearable device can collect eye images to perform calculations to achieve gaze tracking or pupil distance estimation. Summary of the Invention

[0005] The present disclosure proposes a method for determining the pupil center and pupil based on a wearable device and related devices to solve or partially solve the above problems.

[0006] According to a first aspect of the present disclosure, a method for determining a pupil center based on a wearable device is provided, wherein the wearable device includes a first camera and a second camera for capturing eye images of the same eye, and the method includes: acquiring a first eye image captured by the first camera and a second eye image captured by the second camera; determining a first pixel point and a second pixel point corresponding to the pupil center from the first eye image and the second eye image, respectively; determining a first target camera parameter of the first camera according to a position of the first pixel point in the first eye image, the first target camera parameter including a first spatial projection direction of the first pixel point; determining a second target camera parameter of the second camera according to a position of the second pixel point in the second eye image, the second target camera parameter including a second spatial projection direction of the second pixel point; and determining the spatial position of the pupil center according to the first target camera parameter and the second target camera parameter.

[0007] In a second aspect of the present disclosure, a method for determining pupillary distance based on a wearable device is provided, the wearable device including a first target camera and a second target camera for capturing an eye image of a first eye, and a third target camera and a fourth target camera for capturing an eye image of a second eye. The method includes: obtaining a first spatial position of a first pupil center of the first eye and a second spatial position of a second pupil center of the second eye determined using the method described in the first aspect; and determining the pupillary distance based on the first spatial position and the second spatial position.

[0008] According to a third aspect of the present disclosure, a pupil center determination apparatus based on a wearable device is provided, wherein the wearable device includes a first camera and a second camera for capturing eye images of the same eye, and the apparatus includes: an acquisition module configured to acquire a first eye image captured by the first camera and a second eye image captured by the second camera; a first determination module configured to respectively determine a first pixel and a second pixel corresponding to the pupil center from the first eye image and the second eye image; a second determination module configured to determine a first target camera parameter of the first camera based on a position of the first pixel in the first eye image, the first target camera parameter including a first spatial projection direction of the first pixel; a third determination module configured to determine a second target camera parameter of the second camera based on a position of the second pixel in the second eye image, the second target camera parameter including a second spatial projection direction of the second pixel; and a fourth determination module configured to determine the spatial position of the pupil center based on the first target camera parameter and the second target camera parameter.

[0009] In a fourth aspect of the present disclosure, a pupillary distance determination apparatus based on a wearable device is provided, the wearable device including a first target camera and a second target camera for acquiring an eye image of a first eye, and a third target camera and a fourth target camera for acquiring an eye image of a second eye, the apparatus including: an acquisition module configured to acquire a first spatial position of a first pupil center of the first eye and a second spatial position of a second pupil center of the second eye determined using the method described in the first aspect; and a determination module configured to determine the pupillary distance based on the first spatial position and the second spatial position.

[0010] In a fifth aspect of the present disclosure, a computer device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to the first aspect or the second aspect.

[0011] In a sixth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided, which, when executed by one or more processors, enables the processors to execute the method described in the first aspect or the second aspect.

[0012] In a seventh aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method described in the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] FIG1A shows a schematic diagram of an exemplary system provided by an embodiment of the present disclosure.

[0015] FIG1B shows a schematic diagram of an exemplary head-mounted wearable device.

[0016] FIG2 shows a schematic diagram of an exemplary wearable device provided by an embodiment of the present disclosure.

[0017] FIG3A shows a schematic diagram of another exemplary wearable device provided by an embodiment of the present disclosure.

[0018] FIG3B shows a schematic diagram of another exemplary wearable device provided by an embodiment of the present disclosure.

[0019] FIG3C shows a schematic diagram of exemplary camera parameters according to an embodiment of the present disclosure.

[0020] FIG4A shows a flowchart of an exemplary method provided by an embodiment of the present disclosure.

[0021] FIG4B shows a flowchart of an exemplary method for determining the spatial position of the pupil center according to an embodiment of the present disclosure.

[0022] FIG4C shows a partial flowchart of an exemplary method provided by an embodiment of the present disclosure.

[0023] FIG4D shows a flowchart of an exemplary method for determining correction parameters according to an embodiment of the present disclosure.

[0024] FIG4E shows a partial flowchart of an exemplary method provided by an embodiment of the present disclosure.

[0025] FIG4F shows a flowchart of another exemplary method provided by an embodiment of the present disclosure.

[0026] FIG5A shows a schematic diagram of an exemplary first eye image according to an embodiment of the present disclosure.

[0027] FIG5B shows a schematic diagram of an exemplary second eye image according to an embodiment of the present disclosure.

[0028] FIG5C shows a schematic diagram of a computational model for determining the pupil center based on camera parameters according to an embodiment of the present disclosure.

[0029] FIG5D shows a schematic diagram of an exemplary simulation model according to an embodiment of the present disclosure.

[0030] FIG5E shows a schematic diagram of a first target image according to an embodiment of the present disclosure.

[0031] FIG5F shows a schematic diagram of a second target image according to an embodiment of the present disclosure.

[0032] FIG6 shows a schematic diagram of the hardware structure of an exemplary computer device provided by an embodiment of the present disclosure.

[0033] FIG7 shows a schematic diagram of an exemplary device provided by an embodiment of the present disclosure.

[0034] FIG8 shows a schematic diagram of another exemplary device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0036] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the described object changes, the relative position relationship may also change accordingly.

[0037] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0038] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0039] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0040] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0041] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0042] FIG1A shows a schematic diagram of an exemplary augmented reality system 100 provided by an embodiment of the present disclosure.

[0043] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds to create a virtual environment where humans and machines can interact. XR technology can further encompass augmented reality (AR), virtual reality (VR), and mixed reality (MR), leveraging hardware devices combined with various technologies to fuse virtual content with real-world scenes.

[0044] As shown in Figure 1A, the system 100 may include various types of wearable devices, such as head-mounted wearable devices (e.g., VR / AR glasses or head-mounted displays (HMD)) 104, operating handles 108, etc. In some scenarios, a camera / camera 110 for taking photos of the operator (user) 102 may also be provided. In some embodiments, when the aforementioned device does not have a processing function or when necessary, the system 100 may further include an external control device 112 for providing a processing function. The control device 112, for example, may be a computer device such as a mobile phone or a computer. In some embodiments, when any of the aforementioned devices serves as a control device or a main control device, it may communicate with other devices in the system 100 through wired or wireless communication to achieve information interaction.

[0045] In system 100, user 102 can interact with augmented reality system 100 using head-mounted wearable device 104 and operating handle 108. In some scenarios, system 100 can use images captured by camera 110 to identify user 102's posture and gestures, and then interact with user 102 based on the identified posture and gestures. In some embodiments, user 130 can also implement gesture input using bare hands. Head-mounted wearable device 104 can capture front-facing images in real time using a camera or other device located in front of head-mounted wearable device 104, and recognize user 130's gestures by recognizing these images.

[0046] In some embodiments, as shown in FIG1A , the system 100 may further communicate with a server 114 and obtain data, such as images, audio, and video, from the server 114 and output the data through the head-mounted wearable device 104 , for example, by displaying images or videos on a display screen of the head-mounted wearable device 104 , playing audio and video audio using a speaker of the head-mounted wearable device 104 , etc. In some embodiments, as shown in FIG1A , the server 114 may retrieve the required data, such as images, audio, and video, from a database server 116 for storing data.

[0047] In some embodiments, a collection unit for collecting information may be provided on the head-mounted wearable device 104. The collection unit may be of various types.

[0048] In some embodiments, the acquisition unit may further include an environment acquisition unit and a positioning tracking unit, wherein the environment acquisition unit may be used to acquire environmental information around (for example, in front of) the wearable device 104, and the positioning tracking unit may be used to track the wearable device 104. Optionally, the environment acquisition unit may include but is not limited to a three-color camera (for example, an RGB camera), a depth camera, a binocular camera, a laser or other photosensitive element, and the positioning tracking unit may include but is not limited to visual simultaneous localization and mapping (visual SLAM), an inertial measurement unit (IMU), a global positioning system (GPS), ultra-wideband wireless communication technology (UWB), a laser or other modules.

[0049] In some embodiments, the head-mounted wearable device 104 may also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (e.g., a gyroscope), etc. for collecting speed information or acceleration information of the head-mounted wearable device 104. For another example, the operating handle 108 may also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (e.g., a gyroscope), etc. for collecting speed information or acceleration information of the operating handle 108. It should be noted that, in addition to being provided on the head-mounted wearable device 104 and the operating handle 108, the aforementioned collection unit may also be provided directly on a body part of the interactive user 102 by attachment without relying on a hardware device, thereby collecting relevant information of the body part, such as speed, acceleration, or angular velocity information, or information collected by other sensors or collection units (e.g., eye images (including pupil images), etc.).

[0050] In some embodiments, the head-mounted wearable device 104 may also be provided with a camera or a camera for taking photos of the operator (user) 102 (eg, photos of hands or feet) and images of the environment.

[0051] In some embodiments, the system 100 can identify the posture and gestures of the user 102 through the collected information, and then can perform corresponding interactions based on the identified user posture and gestures.

[0052] FIG1B shows a schematic diagram of an exemplary head-mounted wearable device 104 .

[0053] As shown in FIG1B , the head-mounted wearable device 104 may include a lens barrel 1042, inside which a display screen 1044 for displaying an image and an optical component 1046 for processing the light path may be provided. Optionally, the optical component 1046 may further include a plurality of lenses (e.g., lenses 1046A and 1046B). The combination of the plurality of lenses may project light emitted by the display screen 1044 into the eye 1022, so that the eye 1022 may view the image displayed on the display screen 1044. It will be understood that FIG1B only exemplarily shows a single-sided structure of the head-mounted wearable device 104. In order to achieve binocular display, the head-mounted wearable device 104 may include two lens barrel structures arranged side by side.

[0054] In some embodiments, as shown in FIG1B , the head-mounted wearable device 104 may also be provided with a camera 1048 for capturing eye images. The camera 1048 may be a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, or the like.

[0055] Optionally, the camera 1048 may be an eye tracking (ET) camera, and the eye images collected by the camera may be used to implement functions such as pupil distance estimation and eye tracking.

[0056] As shown in FIG1B , in the related art, camera 1048 is typically positioned outside the lens barrel, and typically only one camera is positioned for each lens barrel. Furthermore, in order to better capture a complete eye image without affecting the eye's observation of the display screen 1022, the common deployment location of camera 1048 is generally at the outer canthus or nose wing. Referring to FIG1B , if camera 1048 is close to the outside of the device, the camera deployment location shown in FIG1B is at the outer canthus; if camera 1048 is close to the inside of the device, the camera deployment location shown in FIG1B is at the nose wing.

[0057] However, the inventors of the present disclosure discovered that the method of installing the camera in the related art easily causes the camera 1048 to be installed at a large inclination angle relative to the eye 1022, resulting in a large angle γ between the orientation of the camera 1048 and the normal viewing direction of the eye 1022, making it difficult for the collected eye image to reflect the image of the normal viewing angle of the eye.

[0058] In some cases, users may need to wear glasses before using the head-mounted wearable device 104. However, because the camera 1048 is located outside the lens barrel 1042, it is higher than the lens barrel 1042, which can easily squeeze the glasses and affect the wearing comfort of the head-mounted wearable device 104. Furthermore, the imaging of the camera 1048 is easily affected by the edges of the glasses. The refraction of light passing through the edges of the glasses reduces the clarity of the image produced by the camera 1048 and creates many refracted light spots in the image, which in turn affects the accuracy of subsequent algorithms.

[0059] In view of this, an embodiment of the present disclosure provides a wearable device, which can solve or partially solve the above-mentioned problems to a certain extent by setting a camera inside the lens barrel.

[0060] FIG2 shows a schematic diagram of an exemplary wearable device 200 provided by an embodiment of the present disclosure.

[0061] As shown in FIG2 , similarly, the wearable device 200 may also include a lens barrel 202, a display screen 204 disposed inside the lens barrel 202, and an optical assembly 206. The optical assembly 206 may further include a plurality of lenses (e.g., lenses 206A and 206B). The combination of the plurality of lenses may project light emitted by the display screen 204 into the eye 1022, so that the eye 1022 can view the image displayed on the display screen 204.

[0062] Unlike the wearable device 104 shown in FIG1B , the camera 208 of the wearable device 200 is arranged inside the lens barrel 202 and toward the light-emitting side of the lens barrel. Since the camera 208 is placed inside the lens barrel 202, it will not affect the wearing of glasses, thereby improving the comfort of the wearable device 200. At the same time, as shown in FIG2 , since the camera 208 is arranged inside the lens barrel 202, the distance between the camera 208 and the eye 1022 is extended, so that the installation angle of the camera 208 relative to the eye 1022 becomes smaller, and then the angle β between the direction of the camera 208 and the normal view direction of the eye 1022 is reduced, so that the camera 208 has a better observation angle, and the eye image collected can better reflect the image of the normal view angle of the eye, and the imaging quality is better. In addition, since the camera 208 is placed inside the lens barrel 202, the glasses will not affect the imaging of the camera 208, further improving the imaging quality. The improvement of imaging quality also improves the accuracy of algorithms such as pupil distance estimation or line of sight tracking.

[0063] In some embodiments, as shown in Figure 2, the wearable device 200 may further include a point light source 212. The light emitted by the point light source 212 can form reflected light on the cornea of ​​the eye 1022 and thus be collected by the camera 208, thereby forming a light spot in the image obtained by the camera 208. According to the position of the light spot in the image, it can be used to assist in locating the center of the cornea and the position of the pupil. In some embodiments, in order to make the positioning more accurate, the number of point light sources 212 can be two or more. Optionally, the point light source 212 can be a light emitting diode (LED). In some embodiments, the light emitted by the point light source 212 may not be visible light (for example, it may be infrared light) or it may be visible light with relatively weak brightness, thereby improving the comfort of the user.

[0064] It is understandable that FIG2 only exemplarily shows a single-side structure of the wearable device 200. In order to achieve binocular display, the wearable device 200 may include two lens barrel structures arranged in parallel.

[0065] When a wearable device that has completed camera parameter calibration is used, it can perform processing operations such as determining pupil position, estimating pupil distance, and tracking gaze based on the images captured by the built-in camera of the wearable device and the camera parameters.

[0066] As shown in FIG2 , an exemplary method for determining pupil position is to generate a light spot in an image using a point light source 212. The pupil position is then calculated based on the light spot's position in the image, camera parameters, and other data, combined with optical principles. This calculation method is also known as the pupil-corneal reflection method. However, the inventors of the present disclosure have discovered that this calculation method has certain problems.

[0067] First, the pupil-corneal reflection method places certain demands on camera characteristics. As previously mentioned, when camera 208 is placed inside lens barrel 202, the optical path is affected by optical assembly 206, causing distortion. This makes it impossible to approximate the image and camera parameters captured by camera 208 using the central camera model. However, the pupil-corneal reflection method is difficult to directly apply to non-central cameras. In other words, to determine pupil position using the pupil-corneal reflection method, camera 208 must be placed outside lens barrel 202.

[0068] Secondly, the pupil-corneal reflection method has certain requirements on the number of light spots formed on the eye. As wearable devices become thinner and lighter, the pupil distance of the eye becomes smaller and smaller, and it becomes increasingly difficult for the point light source 212 to form an effective light spot on the eye. The success rate and accuracy of this solution are significantly reduced.

[0069] Third, this solution is not suitable for users wearing glasses, as glasses can cause deviations in both the light emitted by the point light source 212 and the light received by the camera. These two deviations can lead to a significant decrease in the accuracy of pupil position and pupil distance estimation.

[0070] In view of this, an embodiment of the present disclosure provides a method for determining the pupil center. By using images captured by at least two cameras set in a wearable device for capturing eye images of the same eye, the pupil center can be calculated more accurately without using the pupil corneal reflection method (without setting a light source for generating a light spot).

[0071] FIG3A shows a schematic diagram of an exemplary wearable device 300 provided by an embodiment of the present disclosure.

[0072] As shown in FIG3A , the wearable device 300 may include a first lens barrel 302A and a second lens barrel 302B, corresponding to a first eye (e.g., left eye) and a second eye (e.g., right eye), respectively. In some embodiments, the wearable device 300 may be provided with a first camera 304A and a second camera 306A corresponding to the first lens barrel 302A, and a third camera 304B and a fourth camera 306B corresponding to the second lens barrel 302B. Optionally, the first camera 304A and the second camera 306A may be provided inside or outside the first lens barrel 302A and may be used to capture an ocular image of the first eye. Similarly, the third camera 304B and the fourth camera 306B may be provided inside or outside the second lens barrel 302B and may be used to capture an ocular image of the second eye.

[0073] The disclosed embodiments can use images captured by different cameras to determine the pupil position of the corresponding eye. Therefore, to ensure algorithm accuracy, the two cameras corresponding to the same eye should be placed as close together as possible. Also, try to avoid placing the camera in the area above the eye, as this area is easily blocked by eyelashes, which also affects accuracy. For example, as shown in Figure 3A, the first camera 304A is set near the bottom of the first lens barrel 302A, and the second camera 306A is set near the left side of the first lens barrel 302A. Similarly, the third camera 304B is set near the bottom of the second lens barrel 302B, and the fourth camera 306B is set near the right side of the second lens barrel 302B.

[0074] In some embodiments, to make the captured eye images clearer, the wearable device 300 can be provided with a first light source and a second light source corresponding to the first lens barrel 302A and the second lens barrel 302B, respectively. It will be appreciated that the first and second light sources only need to illuminate the corresponding eye regions and can be placed anywhere on the wearable device 300. Therefore, there is no restriction on their placement, and their specific locations are not shown in the schematic diagram.

[0075] In some embodiments, to ensure comfort, the first light source and the second light source may be infrared light sources. Accordingly, the first camera 304A, the second camera 306A, the third camera 304B, and the fourth camera 306B may be infrared cameras.

[0076] In some embodiments, to improve algorithm accuracy, a larger number of cameras may be provided. As shown in FIG3B , the wearable device 300 is further provided with a fifth camera 308A corresponding to the first lens barrel 302A and a sixth camera 308B corresponding to the second lens barrel 302B. To ensure algorithm accuracy, the fifth camera 308A is also provided with a certain distance from the first camera 304A and the second camera 306A. For example, as shown in FIG3B , the fifth camera 308A may be provided near the right side of the first lens barrel 302A. Similarly, the sixth camera 308B is also provided with a certain distance from the third camera 304B and the fourth camera 306B. For example, as shown in FIG3B , the sixth camera 308B may be provided near the left side of the second lens barrel 302B.

[0077] Furthermore, an embodiment of the present disclosure provides a method for determining the pupil center based on a wearable device. By placing at least two cameras inside or outside the lens barrel corresponding to each eye, and coordinating with a light source inside or outside the lens barrel of each eye, the eye images captured by the at least two cameras can be used to determine the position of the pupil center with a high success rate and high precision, and the pupil distance measurement function can be further realized.

[0078] This pupil center determination method does not require calibration of the light source, but only requires calibration of the camera parameters.

[0079] In some embodiments, if the camera is placed inside the lens barrel, the camera will pass through a complex optical system that is difficult to describe parameterically. The centrality of the camera will be destroyed, and the traditional camera model will no longer be applicable. Therefore, a non-parametric camera model can be used to calibrate the camera parameters.

[0080] The calibrated camera parameters are shown in FIG3C . The camera parameters represent the spatial straight line equations passing through the pixels in the image captured by the camera, reflecting the spatial projection directions of the image pixels. Specifically, the camera parameters are used to characterize the target pixels in the image captured by the camera and the projection directions corresponding to the target pixels. Optionally, the camera parameters may represent a one-to-one correspondence between pixels and projection directions. The specific calibration method is not limited here, and any implementation method that can calibrate the camera parameters disclosed herein is applicable.

[0081] In some embodiments, if the camera is placed outside the lens barrel, similar camera parameters can be calibrated in the same manner to represent the equation of a spatial line passing through pixel points in an image captured by the camera.

[0082] In this way, after the camera parameter calibration is completed, each camera (the first camera 304A, the second camera 306A, the third camera 304B, the fourth camera 306B, the fifth camera 308A, and the sixth camera 308B) corresponds to a set of camera parameters, which represent the spatial projection equation of the pixel points in the image captured by the camera.

[0083] FIG4A shows a flow chart of an exemplary method 400 provided in an embodiment of the present disclosure.

[0084] The method 400 may be used to determine the pupil center and may further determine the pupil distance. As shown in FIG4A , the method 400 may include the following steps 402 , 404 , 406 , 408 and 410 .

[0085] Taking determining the position of the pupil center of the first eye (e.g., the left eye) as an example, in step 402, a first eye image 502 (as shown in FIG. 5A ) captured by the first camera 304A and a second eye image 504 (as shown in FIG. 5B ) captured by the second camera 306A may be obtained.

[0086] In some embodiments, an eye-opening judgment model may be used to determine whether the eyes in the first eye image 502 and the second eye image 504 are open. If the eyes are closed, the images are repeatedly captured until a valid eye-opening image is obtained.

[0087] In step 404 , a first pixel point 5022 and a second pixel point 5042 corresponding to the pupil center are determined from the first eye image 502 and the second eye image 504 , respectively.

[0088] As shown in FIG5A , first eye image 502 is an image of the first eye captured by first camera 304A at its position. The eye image is illustrated by an ellipse. Optionally, an image recognition algorithm can be used to identify a first pixel 5022 corresponding to the center of the pupil of the first eye from first eye image 502. As shown in FIG5B , second eye image 504 is an image of the first eye captured by second camera 306A at its position. Similarly, a second pixel 5042 corresponding to the center of the pupil of the first eye can also be identified from second eye image 504.

[0089] At step 406, first target camera parameters of the first camera are determined based on the position of the first pixel in the first eye image. In some embodiments, the first target camera parameters include a first spatial projection direction of the first pixel.

[0090] As previously described, camera parameters represent the equation of a line passing through corresponding pixels, reflecting the spatial projection direction of the image pixels. Therefore, once the position of first pixel 5022 in first eye image 502 is determined, corresponding first target camera parameters can be determined from the calibrated camera parameter set for first camera 304A. These first target camera parameters represent the equation of a line passing through first pixel 5022.

[0091] At step 408, second target camera parameters of the second camera are determined based on the position of the second pixel in the second eye image. In some embodiments, the second target camera parameters include a second spatial projection direction of the second pixel.

[0092] Similarly, after determining the position of the second pixel point 5042 in the second eye image 504, the corresponding second target camera parameters can be determined from the camera parameter set that has been calibrated for the second camera 306A. The second target camera parameters represent the equation of the spatial straight line passing through the second pixel point 5042.

[0093] In step 410 , the spatial position of the pupil center is determined according to the first target camera parameters and the second target camera parameters.

[0094] FIG5C shows a schematic diagram of a computational model 506 for determining a pupil center based on camera parameters according to an embodiment of the present disclosure.

[0095] As shown in FIG. 5C , the first target camera parameter may be mapped to a straight line 5062 in the three-dimensional space, and the second target camera parameter may be mapped to a straight line 5064 in the three-dimensional space.

[0096] It can be understood that straight line 5062 is a spatial projection straight line passing through the first pixel point 5022, which represents the projection relationship of the first pixel point 5022 from the camera coordinate system of the first camera 304A to the spatial coordinate system. Straight line 5064 is a spatial projection straight line passing through the second pixel point 5042, which represents the projection relationship of the second pixel point 5042 from the camera coordinate system of the second camera 306A to the spatial coordinate system. The first pixel point 5022 and the second pixel point 5042 are the pupil centers observed by the first camera 304A and the second camera 306A respectively. Therefore, the intersection Q of straight line 5062 and straight line 5064 can be approximated as the pupil center.

[0097] In this step, if the accuracy allows, the spatial coordinates of the intersection point Q can be determined as the spatial position of the pupil center of the first eye.

[0098] In some embodiments, straight line 5062 and straight line 5064 may not strictly intersect (ie, there is no actual intersection point), and the point where the sum of the squares of the straight line distances from the two is minimized may be calculated as the intersection point Q.

[0099] In some embodiments, the user may wear glasses, and since the glasses may affect the optical path, correction is required. Therefore, the method 400 may also include obtaining adjustment parameters input by the user (for example, glasses degree, astigmatism degree / axis, etc.).

[0100] Step 410 may further include: adjusting the first target camera parameters and the second target camera parameters according to the adjustment parameters; and determining the spatial position of the pupil center according to the adjusted first target camera parameters and the second target camera parameters.

[0101] In this step, the calibrated camera parameters (or spatial straight lines) are used as the incident vector and incident from the outside of the glasses. Then, based on the optical parameters of the glasses themselves (for example, the glasses degree, astigmatism degree / axis, etc.), the exit vector of the glasses on the eye side can be solved and used as the new observation result, that is, the adjusted first target camera parameters and the second target camera parameters.

[0102] In essence, the user's glasses lenses + the optical components of the wearable device + the camera together form a new camera model. This way, the impact of the user's glasses on the accuracy of the algorithm can be largely eliminated.

[0103] In some embodiments, a detection device may be provided in the wearable device 300 to detect whether the user is wearing glasses. Specifically, the detection device emits and receives detection light. Since the glasses lens itself is a lens, the angle difference between the emitted light and the received light is used to determine whether the optical path of the detection light has changed, thereby determining whether the user is wearing glasses. In addition, by performing corresponding optical calculations based on the emitted light and the received light, information such as the glasses degree, astigmatism degree / axis, etc. can also be determined. Therefore, the adjustment parameters may not be input by the user, but may be detected by the detection device.

[0104] Because the pupil is located inside the cornea, light is refracted when passing through the corneal surface. As shown in Figure 5C, straight lines 5062 and 5064 are refracted at refraction points A and B on the corneal surface. Refracted light rays 5066 and 5068 intersect at point P, the true pupil center. Therefore, the true pupil center position P and the intersection point Q of the two straight lines typically do not coincide. Therefore, in some embodiments, the positional deviation between position P and intersection point Q can be corrected.

[0105] In some embodiments, as shown in FIG. 4B , determining the spatial position of the pupil center according to the first target camera parameters and the second target camera parameters may further include steps 4102 , 4104 , and 4106 .

[0106] In step 4102, the spatial position of the pupil center to be corrected (ie, the intersection point Q) is determined based on the first target camera parameters and the second target camera parameters.

[0107] In step 4104, a correction parameter is obtained, and the correction parameter is used to correct the spatial position of the pupil center to be corrected, thereby obtaining the corrected position P of the pupil center.

[0108] In some embodiments, the correction parameters may be determined through preprocessing, for example, by simulating the actual situation using a simulation model in advance and determining the correction parameters based on the simulation results.

[0109] In some embodiments, the correction parameter is determined based on a mapping relationship, wherein the mapping relationship is obtained by fitting a plurality of spatial projection direction pairs corresponding to a plurality of first spatial projection directions and a plurality of second spatial projection directions, and a plurality of correction parameters. The spatial projection direction pair may refer to a direction pair consisting of a single first spatial projection direction and a single corresponding second spatial projection direction, that is, the first spatial projection direction and the second spatial projection direction may have a one-to-one correspondence.

[0110] Since after wearing the wearable device, the area with clearer display content is within a certain spatial range around the eyes, in some embodiments, the correction parameters are determined based on the probability distribution of the corneal radius, the probability distribution of the distance from the pupil center to the corneal center, the line of sight angle distribution, and the refractive index distribution of the eyeball in the target space. By using these parameters to establish a simulation model and then determining the correction parameters based on the simulation results, the position correction of the pupil center can be better achieved. In some embodiments, the target space can be an area close to the eyes and the clarity of the display image observed by the eyes is greater than the clarity threshold. As an optional embodiment, the target space can be determined based on the orbital range (Eyebox). Since the Eyebox is a conical area between the optical module and the eye where the display content is clearest, establishing a simulation model based on the parameters of the Eyebox can better achieve the position correction of the pupil center.

[0111] The following describes an embodiment of determining trimming parameters through a simulation model in conjunction with the calculation model 506 shown in FIG. 5C .

[0112] In this embodiment, the deviation correction vector from Q to P may be defined as follows, where x and y are correction parameters of negative real numbers.

[0113] According to the above formula, the correction parameters are related to the corneal radius, the distance from the pupil center to the corneal center, and the observation direction of the camera, and are difficult to solve directly.

[0114] However, the inventors of the present disclosure have discovered that a set of approximate x and y values ​​can be obtained through simulation to serve as correction parameters.

[0115] Assume that x and y are the angle α between the straight line 5062 and the straight line 5064 and the spatial position of the intersection point Q (Q x ,Q y ,Q z ) related functions, namely: x,y=f(α,Q x ,Q y ,Q z ).

[0116] As shown in FIG. 4C , in some embodiments, the method 400 may include the following steps 412 , 414 , 416 , and 418 .

[0117] In step 412, within the eyebox, multiple corneal center positions and multiple pupil center positions are sampled according to the probability distribution of corneal radius, the probability distribution of the distance from the pupil center to the corneal center, and the line of sight angle distribution.

[0118] In some embodiments, the eyebox generally refers to a conical area between the optical module and the eyeball, where the displayed content is clearest. The larger the eyebox, the better the mechanical tolerance of the wearable device, and the more adaptable it is to people with different pupil distances. Generally, when designing the lens barrel of a wearable device, the eyebox can be determined based on the design parameters. Therefore, during simulation, the lens barrel design parameters can be directly input into the simulation system to determine the eyebox.

[0119] The probability distribution of corneal radius can be determined based on multiple different eyeball models, and this probability distribution can reflect the distribution of corneal radius for most groups. Similarly, the probability distribution of the distance from the pupil center to the corneal center can also be determined in the same way.

[0120] The distribution of sight angles can be determined based on the distribution of possible sight angles when users use the wearable device, and can be determined by statistically analyzing actual usage of a certain group. In some embodiments, the distribution can be determined based on the field of view (FOV) of the wearable device.

[0121] In this step, after obtaining the probability distribution of the above parameters, these parameters can be sampled. The sampling method can be arbitrary and is not limited here.

[0122] Then, the cornea center position and pupil center position corresponding to each set of sampling data can be calculated based on the sampling results. In this way, multiple cornea center positions and multiple pupil center positions corresponding to the cornea center positions can be obtained based on multiple sets of sampling data.

[0123] In step 414, a plurality of corneal refractive indices are sampled according to the refractive index distribution of the eyeball.

[0124] Optionally, the refractive index distribution of the eyeball can be determined by statistically analyzing the refractive index distribution of the eyeballs of a certain group.

[0125] In this step, any sampling method can be used to obtain multiple corneal refractive indices at multiple positions of the eyeball according to the refractive index distribution of the eyeball.

[0126] In step 416 , a plurality of eye images captured by the first camera 304A and the second camera 306A at different camera positions are obtained through simulation.

[0127] In this step, for each simulation, a set of parameters (for example, corneal center position m1, pupil center position m2, corneal refractive index m3) is selected based on the parameters obtained by the previous sampling and fixed. Then, the camera position m4 is continuously adjusted to obtain multiple eye images collected at different camera positions.

[0128] In this way, each eye image corresponds to a set of parameters: corneal center position m1, pupil center position m2, corneal refractive index m3, and camera position m4.

[0129] It will be understood that the camera position m4 described in this step refers to the position of the entire camera group. In this embodiment, camera position m4 can be understood as the location of the center of the first camera 304A and the second camera 306A. The relative positions of the first camera 304A and the second camera 306A to this center remain unchanged. It will be understood that when the camera group consists of three or more cameras, the center can be the common center position of the three or more cameras.

[0130] In step 418, the correction parameters are calculated based on the multiple eye images in combination with the multiple corneal center positions, the multiple pupil center positions, and the multiple corneal refractive indices.

[0131] As mentioned above, each eye image corresponds to a set of fixed parameters (known parameters): corneal center position m1, pupil center position m2, corneal refractive index m3, and camera position m4. Therefore, based on the pixel position corresponding to the pupil center in the eye image and combined with these fixed parameters, the correction parameters can be calculated.

[0132] In this way, using the simulation model, a set of approximate x and y can be obtained as correction parameters to correct the spatial position of the pupil center to be corrected (i.e., the intersection Q) to obtain coordinates that are approximate to the real spatial position of the pupil center, thereby improving the accuracy of the algorithm.

[0133] In some embodiments, as shown in FIG4D , calculating the correction parameters based on the multiple eye images, combined with the multiple corneal center positions, multiple pupil center positions, and multiple corneal refractive indices, may further include the following steps 4182 , 4184 , 4186 , 4188 , and 4190 .

[0134] In step 4182, a first target image and a second target image respectively captured by the first camera and the second camera at the target position are obtained from the multiple eye images.

[0135] FIG5D shows a schematic diagram of an exemplary simulation model 508 according to an embodiment of the present disclosure.

[0136] As shown in FIG5D , assuming that a set of parameters corresponding to the acquisition of the first target image and the second target image can constitute a simulation model 508 , under which the corneal center position, pupil center position P′, corneal refractive index n, and target position of the camera are all known.

[0137] 5E and 5F respectively show schematic diagrams of a first target image 510 and a second target image 512 according to an embodiment of the present disclosure.

[0138] As shown in FIG. 5E and FIG. 5F , a first target pixel point 5102 and a second target pixel point 5122 corresponding to the pupil center in the first target image 510 and the second target image 512 can be obtained by identification.

[0139] In step 4184, based on the corneal center position, the pupil center position P' and the corneal refractive index corresponding to the first target image 510 and the second target image 512 when the first target image 510 and the second target pixel point 5122 corresponding to the pupil center in the first target image 510 and the second target image 512, the first straight line 5082 passing through the first target pixel point 5102 and the second straight line 5084 passing through the second target pixel point 5122, as well as the first refraction point A' and the second refraction point B' of the first straight line 5082 and the second straight line 5084 on the corneal surface are determined.

[0140] In this step, similar to the previous embodiment, under the premise of known camera parameters, the first straight line 5082 and the second straight line 5084 can be determined according to the positions of the first target pixel point 5102 and the second target pixel point 5122 in the first target image 510 and the second target image 512.

[0141] Furthermore, since the corneal center position, pupil center position P' and corneal refractive index are known, the first refraction point A' and the second refraction point B' of the first straight line 5082 and the second straight line 5084 on the corneal surface can be calculated by combining the first straight line 5082 and the second straight line 5084.

[0142] In step 4186, the intersection point Q' of the first straight line and the second straight line is determined.

[0143] After knowing the first straight line 5082 and the second straight line 5084 , the intersection point Q′ between the two in the three-dimensional space can be determined.

[0144] It can be understood that when the first straight line 5082 and the second straight line 5084 do not strictly intersect, the point where the sum of the squares of the straight-line distances between the two is the smallest can be used as the intersection point Q'.

[0145] In step 4188, the reference correction parameters corresponding to the target position are determined based on the first refraction point A', the second refraction point B', the intersection point Q', the first straight line 5082 and the second straight line 5084, and the pupil center position P' corresponding to when the first target image 510 and the second target image 512 are acquired.

[0146] In this step, according to the following formula:

[0147] By combining the first refraction point A', the second refraction point B', the intersection Q', the first straight line 5082 and the second straight line 5084, and the pupil center position P' corresponding to when the first target image 510 and the second target image 512 are collected, a set of x0 and y0 can be obtained.

[0148] In step 4190, the correction parameter is determined based on the reference correction parameter.

[0149] In this step, correction parameters x and y can be further determined based on x0 and y0.

[0150] It can be understood that since x0 and y0 are only calculated based on a specific set of parameters when the camera is at the target position, as mentioned earlier, the simulation model uses many parameters. In order to ensure the accuracy of the algorithm, it is necessary to further determine the correction parameters x and y based on the reference correction parameters obtained by simulating multiple sets of different parameters.

[0151] Therefore, in some embodiments, determining the correction parameter according to the reference correction parameter includes:

[0152] According to the reference correction parameter, the intersection point, the first straight line and the second straight line, a mapping relationship between the reference correction parameter and the intersection point and the straight line angle α (the angle can be calculated based on the first straight line 5082 and the second straight line 5084) is determined. For example, (α,q x ,q y ,q z ) and (x0, y0); and then determining the correction parameter according to the mapping relationship.

[0153] Furthermore, in some embodiments, the mapping relationship includes multiple mapping relationships corresponding to multiple different target positions, and determining the correction parameter based on the mapping relationship includes: fitting the correction parameter based on the multiple mapping relationships. For example, a model (including but not limited to polynomial regression and deep neural network) is used to fit the (α,q x ,q y ,q z ) to (x,y).

[0154] In this way, we can get (α,q x ,q y ,q z ) to (x,y).

[0155] In step 4106, the spatial position of the pupil center is determined based on the first target camera parameters, the second target camera parameters, the spatial position of the pupil center to be corrected, and the correction parameters.

[0156] Specifically, in step 4104, the angle between straight lines 5062 and 5064 can be determined based on the first target camera parameters and the second target camera parameters, and then the corresponding correction parameters x and y can be found based on the angle and the spatial position Q of the pupil center to be corrected.

[0157] In this step, the spatial position of the pupil center can be determined according to the first target camera parameters, the second target camera parameters, the spatial position of the pupil center to be corrected, and the correction parameters.

[0158] In some embodiments, determining the spatial position of the pupil center according to the first target camera parameters, the second target camera parameters, the to-be-corrected spatial position of the pupil center, and the correction parameters further includes:

[0159] According to the first target camera parameter, the second target camera parameter and the correction parameter, an offset vector of the to-be-corrected spatial position of the pupil center relative to the spatial position of the pupil center is determined; for example, a correction vector is calculated according to the correction parameter

[0160] According to the spatial position to be corrected of the pupil center and the offset vector, the spatial position of the pupil center is determined, that is, the estimated pupil coordinates are obtained.

[0161] At this point, the spatial position of the pupil center of the first eye in the three-dimensional space can be determined based on the first eye image 502 and the second eye image 504 of the first eye captured by the first camera 304A and the second camera 306A.

[0162] For the case of three or more cameras, the pupil position can be calculated for any combination of two cameras, and then the average of all combinations is taken to obtain the final pupil position result.

[0163] It is understandable that according to a similar approach to the above method 400 , the pupil center of the second eye can also be determined.

[0164] As shown in FIG. 4E , in some embodiments, the method 400 may further include the following steps 420 , 422 , 424 , 426 , and 428 .

[0165] Taking the determination of the position of the pupil center of the second eye (eg, the right eye) as an example, in step 420 , a third eye image captured by the third camera 304B and a fourth eye image captured by the fourth camera 306B are acquired.

[0166] In some embodiments, an eye-opening judgment model may be used to judge whether the eyes in the third eye image and the fourth eye image are open. If the eyes are closed, the images are repeatedly captured until a valid eye-opening image is obtained.

[0167] At step 422, a third pixel and a fourth pixel corresponding to the second pupil center of the second eye are determined from the third eye image and the fourth eye image, respectively. Alternatively, the third pixel and the fourth pixel corresponding to the second pupil center can be identified from the third eye image and the fourth eye image using an image recognition algorithm.

[0168] In step 424 , third target camera parameters of the third camera are determined according to the position of the third pixel in the third eye image. The third target camera parameters include a third spatial projection direction of the third pixel.

[0169] As previously described, camera parameters represent the equation of a line passing through corresponding pixels, reflecting the spatial projection direction of the image pixels. Therefore, once the position of the third pixel in the third eye image is determined, corresponding third target camera parameters can be determined from the calibrated camera parameter set of the third camera 304B. These third target camera parameters represent the equation of a line passing through the third pixel.

[0170] In step 426 , fourth target camera parameters of the fourth camera are determined according to the position of the fourth pixel in the fourth eye image. The fourth target camera parameters include a fourth spatial projection direction of the fourth pixel.

[0171] Similarly, after determining the position of the fourth pixel in the fourth eye image, the corresponding fourth target camera parameters can be determined from the calibrated camera parameter set of the fourth camera 306B. The fourth target camera parameters represent the equation of the spatial straight line passing through the fourth pixel.

[0172] In step 428 , a second spatial position of the second pupil center is determined based on the third target camera parameters and the fourth target camera parameters.

[0173] The implementation of step 428 is similar to that of step 410 , and the process of building the simulation model is also similar, which will not be repeated here.

[0174] In this way, the spatial position of the pupil center of the second eye in the three-dimensional space can be obtained.

[0175] It can be seen from the above embodiments that the embodiments of the present disclosure provide a method for determining the pupil center. By using images captured by at least two cameras provided in a wearable device for capturing eye images of the same eye, the pupil center can be calculated more accurately without using the pupil corneal reflection method. The pupil centers of the two eyes can be determined based on this, and the pupil distance can be determined.

[0176] It should be noted that the method of the embodiments of the present disclosure can be performed by a single device, such as a computer or server. The method of the embodiments of the present disclosure can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0177] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0178] The present disclosure also provides a method for determining pupillary distance based on a wearable device, wherein the wearable device includes a first target camera and a second target camera for capturing an eye image of a first eye, and a third target camera and a fourth target camera for capturing an eye image of a second eye.

[0179] FIG4F shows a flowchart of another exemplary method 430 provided in an embodiment of the present disclosure.

[0180] The method 430 may be used to determine the pupil distance. As shown in FIG4F , the method 430 may further include the following steps 432 and 434 .

[0181] At step 432, a first spatial position of the first pupil center of the first eye and a second spatial position of the second pupil center of the second eye, determined using any embodiment, permutation, or combination of the embodiments of method 400, may be obtained. The method for determining the first spatial position and the second spatial position is described in detail in the embodiments of method 400 and will not be further described here.

[0182] In step 434, the pupil distance is determined based on the first spatial position (the spatial position of the pupil center of the first eye in the three-dimensional space) and the second spatial position (the spatial position of the pupil center of the second eye in the three-dimensional space).

[0183] After determining the pupil distance, the lens barrel can be adjusted according to the pupil distance to adapt to the user's pupil distance, increase comfort, and enhance user experience.

[0184] The embodiments of the present disclosure also provide a computer device for implementing the above-mentioned method 400 or method 430. FIG6 shows a schematic diagram of the hardware structure of an exemplary computer device 600 provided in an embodiment of the present disclosure. The computer device 600 can be used to implement the head-mounted wearable device 104 of FIG1A and FIG1B, the wearable device 200 of FIG2, and the wearable device 300 of FIG3A and FIG3B. It can also be used to implement the external device 112 of FIG1A, and can also be used to implement the server 114 of FIG1A. In some scenarios, the computer device 600 can also be used to implement the database server 116 of FIG1A.

[0185] 6 , computer device 600 may include a processor 602, a memory 604, a network module 606, a peripheral interface 608, and a bus 610. The processor 602, the memory 604, the network module 606, and the peripheral interface 608 are connected to each other in communication within the computer device 600 via the bus 610.

[0186] The processor 602 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device (PLD), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or one or more integrated circuits. The processor 602 may be used to perform functions related to the technology described in this disclosure. In some embodiments, the processor 602 may also include multiple processors integrated into a single logical component. For example, as shown in FIG6 , the processor 602 may include multiple processors 602a, 602b, and 602c.

[0187] The memory 604 can be configured to store data (e.g., instructions, computer codes, etc.). As shown in Figure 6, the data stored in the memory 604 can include program instructions (e.g., program instructions for implementing the method 400 or method 430 of the embodiment of the present disclosure) and data to be processed (e.g., the memory can store configuration files of other modules, etc.). The processor 602 can also access the program instructions and data stored in the memory 604, and execute the program instructions to operate on the data to be processed. The memory 604 can include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 604 can include a random access memory (RAM), a read-only memory (ROM), an optical disc, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0188] The network interface 606 can be configured to provide the computer device 600 with communication with other external devices via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC)), a cellular network, the Internet, or a combination thereof. It will be understood that the type of network is not limited to the specific examples above.

[0189] The peripheral interface 608 can be configured to connect the computer device 600 to one or more peripheral devices to enable information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touchpad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0190] The bus 610 may be configured to transmit information between various components of the computer device 600 (e.g., the processor 602, the memory 604, the network interface 606, and the peripheral interface 608), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0191] It should be noted that although the architecture of the computer device 600 shown above only includes the processor 602, memory 604, network interface 606, peripheral interface 608, and bus 610, in a specific implementation, the architecture of the computer device 600 may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the architecture of the computer device 600 may only include the components necessary to implement the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0192] The present disclosure also provides an apparatus for determining pupil center based on a wearable device. FIG7 shows a schematic diagram of an exemplary apparatus 700 provided by the present disclosure. As shown in FIG7 , the apparatus 700 can be used to implement method 400 and can further include the following modules.

[0193] The acquisition module 702 is configured to acquire a first eye image captured by the first camera and a second eye image captured by the second camera; the first determination module 704 is configured to determine a first pixel and a second pixel corresponding to the pupil center from the first eye image and the second eye image, respectively; the second determination module 706 is configured to determine a first target camera parameter of the first camera according to the position of the first pixel in the first eye image, the first target camera parameter including a first spatial projection direction of the first pixel; the third determination module 708 is configured to determine a second target camera parameter of the second camera according to the position of the second pixel in the second eye image, the second target camera parameter including a second spatial projection direction of the second pixel; the fourth determination module 710 is configured to determine the spatial position of the pupil center according to the first target camera parameter and the second target camera parameter.

[0194] In some embodiments, the fourth determination module 710 is configured to: determine the spatial position of the pupil center to be corrected based on the first target camera parameters and the second target camera parameters; obtain correction parameters; and determine the spatial position of the pupil center based on the first target camera parameters, the second target camera parameters, the spatial position of the pupil center to be corrected, and the correction parameters.

[0195] In some embodiments, the correction parameter is determined based on a mapping relationship, and the mapping relationship is obtained by fitting a plurality of spatial projection direction pairs corresponding to a plurality of first spatial projection directions and a plurality of second spatial projection directions and a plurality of correction parameters.

[0196] In some embodiments, the fourth determination module 710 is configured to: determine the offset vector of the to-be-corrected spatial position of the pupil center relative to the spatial position of the pupil center based on the first target camera parameters, the second target camera parameters and the correction parameters; and determine the spatial position of the pupil center based on the to-be-corrected spatial position of the pupil center and the offset vector.

[0197] In some embodiments, the correction parameter is determined based on the probability distribution of the corneal radius within the dynamic orbital range, the probability distribution of the distance from the pupil center to the corneal center, the line of sight angle distribution, and the refractive index distribution of the eyeball.

[0198] In some embodiments, the device also includes a simulation module, which is configured to: sample multiple corneal center positions and multiple pupil center positions within the dynamic orbital range according to the probability distribution of corneal radius, the probability distribution of the distance from the pupil center to the corneal center, and the line of sight angle distribution; sample multiple corneal refractive indices according to the refractive index distribution of the eyeball; simulate and obtain multiple eye images captured by the first camera and the second camera at different camera positions; and calculate the correction parameters based on the multiple eye images in combination with the multiple corneal center positions, multiple pupil center positions, and multiple corneal refractive indices.

[0199] In some embodiments, the simulation module is configured to: obtain the first target image and the second target image captured by the first camera and the second camera at the target position respectively from the multiple eye images; determine the first straight line passing through the first target pixel point and the second straight line passing through the second target pixel point and the first refraction point and the second refraction point of the first straight line and the second straight line on the corneal surface according to the corneal center position, the pupil center position and the corneal refractive index corresponding to the first target image and the second target image, combined with the first target pixel point and the second target pixel point corresponding to the pupil center in the first target image and the second target image; determine the intersection of the first straight line and the second straight line; determine the reference correction parameters corresponding to the target position according to the first refraction point, the second refraction point, the intersection, the first straight line and the second straight line and the pupil center position corresponding to the first target image and the second target image; determine the correction parameters according to the reference correction parameters.

[0200] In some embodiments, the simulation module is configured to: determine the mapping relationship between the reference correction parameter and the intersection point and the angle between the straight lines based on the reference correction parameter, the intersection point, the first straight line and the second straight line; and determine the correction parameter based on the mapping relationship.

[0201] In some embodiments, the mapping relationship includes multiple mapping relationships corresponding to multiple different target positions, and the simulation module is configured to fit the correction parameters according to the multiple mapping relationships.

[0202] In some embodiments, the acquisition module 702 is configured to: obtain adjustment parameters input by the user; the fourth determination module 710 is configured to: adjust the first target camera parameters and the second target camera parameters according to the adjustment parameters; and determine the spatial position of the pupil center based on the adjusted first target camera parameters and the second target camera parameters.

[0203] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0204] The apparatus of the above embodiment is used to implement the corresponding method 400 in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0205] The present disclosure also provides an apparatus for determining pupillary distance based on a wearable device, wherein the wearable device includes a first target camera and a second target camera for capturing an eye image of a first eye, and a third target camera and a fourth target camera for capturing an eye image of a second eye.

[0206] Fig. 8 shows a schematic diagram of an exemplary apparatus 800 provided by an embodiment of the present disclosure. As shown in Fig. 8 , the apparatus 800 may be used to implement the method 430 and may further include the following modules.

[0207] Acquisition module 802 is configured to acquire a first spatial position of the first pupil center of the first eye and a second spatial position of the second pupil center of the second eye, determined using any embodiment, or a permutation or combination of embodiments of method 400. The method for determining the first spatial position and the second spatial position is described in the embodiment of method 400 and is not further described here.

[0208] The determination module 804 is configured to determine the pupil distance according to the first spatial position and the second spatial position.

[0209] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0210] The apparatus of the above embodiment is used to implement the corresponding method 430 in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0211] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute method 400 or method 430 as described in any of the above embodiments.

[0212] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0213] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute method 400 or method 430 as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0214] Based on the same inventive concept, corresponding to method 400 or method 430 in any of the above embodiments, the present disclosure further provides a computer program product comprising computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to perform method 400 or method 430. For the execution entities corresponding to the steps in each embodiment of method 400 or method 430, the processors that execute the corresponding steps can belong to the corresponding execution entities.

[0215] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute method 400 or method 430 as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0216] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0217] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0218] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0219] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for determining pupil center based on a wearable device, wherein: The wearable device includes a first camera and a second camera for acquiring an eye image of the same eye, and the method includes: Acquire a first eye image captured by the first camera and a second eye image captured by the second camera; Determine a first pixel point and a second pixel point corresponding to the pupil center from the first eye image and the second eye image respectively; determining, according to a position of the first pixel in the first eye image, first target camera parameters of the first camera, where the first target camera parameters include a first spatial projection direction of the first pixel; determining, according to a position of the second pixel in the second eye image, second target camera parameters of the second camera, where the second target camera parameters include a second spatial projection direction of the second pixel; The spatial position of the pupil center is determined according to the first target camera parameters and the second target camera parameters.

2. The method of claim 1, wherein: Determining the spatial position of the pupil center according to the first target camera parameter and the second target camera parameter includes: Determining a spatial position of the pupil center to be corrected according to the first target camera parameter and the second target camera parameter; Get correction parameters; The spatial position of the pupil center is determined according to the first target camera parameters, the second target camera parameters, the to-be-corrected spatial position of the pupil center, and the correction parameters.

3. The method of claim 2, wherein: The correction parameter is determined based on a mapping relationship, and the mapping relationship is obtained by fitting a plurality of spatial projection direction pairs corresponding to a plurality of the first spatial projection directions and a plurality of the second spatial projection directions and a plurality of correction parameters.

4. The method of claim 2, wherein: Determining the spatial position of the pupil center according to the first target camera parameter, the second target camera parameter, the to-be-corrected spatial position of the pupil center, and the correction parameter includes: Determine, according to the first target camera parameter, the second target camera parameter and the correction parameter, an offset vector of the to-be-corrected spatial position of the pupil center relative to the spatial position of the pupil center; The spatial position of the pupil center is determined according to the to-be-corrected spatial position of the pupil center and the offset vector.

5. The method of claim 2, wherein: The correction parameters are determined based on the probability distribution of the corneal radius within the dynamic orbit, the probability distribution of the distance from the pupil center to the corneal center, the line of sight angle distribution, and the refractive index distribution of the eyeball.

6. The method of claim 2, wherein: The method further comprises: Within the orbital range, multiple corneal center positions and multiple pupil center positions are sampled according to the probability distribution of corneal radius, the probability distribution of the distance from the pupil center to the corneal center, and the line of sight angle distribution; According to the refractive index distribution of the eyeball, multiple corneal refractive indices are sampled; Simulating to obtain a plurality of eye images captured by the first camera and the second camera at different camera positions; The correction parameters are calculated based on the multiple eye images in combination with the multiple corneal center positions, the multiple pupil center positions and the multiple corneal refractive indices.

7. The method of claim 6, wherein: Calculating the correction parameter according to the multiple eye images, in combination with the multiple corneal center positions, the multiple pupil center positions and the multiple corneal refractive indices, includes: Acquire, from the plurality of eye images, a first target image and a second target image respectively captured by the first camera and the second camera at a target position; According to the corneal center position, the pupil center position and the corneal refractive index corresponding to when the first target image and the second target image are acquired, in combination with the first target pixel point and the second target pixel point corresponding to the pupil center in the first target image and the second target image, determine a first straight line passing through the first target pixel point and a second straight line passing through the second target pixel point, as well as a first refraction point and a second refraction point of the first straight line and the second straight line on the corneal surface; determining an intersection point of the first straight line and the second straight line; Determine a reference correction parameter corresponding to the target position according to the first refraction point, the second refraction point, the intersection point, the first straight line, the second straight line, and the pupil center position corresponding to when the first target image and the second target image are acquired; The correction parameter is determined according to the reference correction parameter.

8. The method of claim 7, wherein: Determining the correction parameter according to the reference correction parameter includes: Determine a mapping relationship between the reference correction parameter and the intersection point and the straight line angle according to the reference correction parameter, the intersection point, the first straight line and the second straight line; The correction parameter is determined according to the mapping relationship.

9. The method of claim 8, wherein: The mapping relationship includes a plurality of mapping relationships corresponding to a plurality of different target positions, and determining the correction parameter according to the mapping relationship includes: The correction parameters are obtained by fitting according to the multiple mapping relationships.

10. The method of claim 1, wherein: The method further includes: obtaining adjustment parameters input by a user; Determining the spatial position of the pupil center according to the first target camera parameters and the second target camera parameters includes: adjusting the first target camera parameters and the second target camera parameters according to the adjustment parameters; and determining the spatial position of the pupil center according to the adjusted first target camera parameters and the second target camera parameters.

11. A method for determining pupil distance based on a wearable device, the wearable device comprising a first target camera and a second target camera for acquiring an eye image of a first eye and a third target camera and a fourth target camera for acquiring an eye image of a second eye, the method comprising: Acquire a first spatial position of a first pupil center of the first eye and a second spatial position of a second pupil center of the second eye determined by the method according to any one of claims 1 to 10; The pupil distance is determined according to the first spatial position and the second spatial position.

12. A pupil center determination device based on a wearable device, wherein: The wearable device includes a first camera and a second camera for acquiring an eye image of the same eye, and the apparatus includes: An acquisition module is configured to: acquire a first eye image acquired by the first camera and a second eye image acquired by the second camera; A first determining module is configured to: determine a first pixel point and a second pixel point corresponding to the pupil center from the first eye image and the second eye image respectively; A second determination module is configured to: determine a first target camera parameter of the first camera according to a position of the first pixel in the first eye image, where the first target camera parameter includes a first spatial projection direction of the first pixel; A third determination module is configured to: determine a second target camera parameter of the second camera according to a position of the second pixel in the second eye image, where the second target camera parameter includes a second spatial projection direction of the second pixel; The fourth determination module is configured to determine the spatial position of the pupil center according to the first target camera parameters and the second target camera parameters.

13. A pupil distance determination device based on a wearable device, the wearable device comprising a first target camera and a second target camera for acquiring an eye image of a first eye and a third target camera and a fourth target camera for acquiring an eye image of a second eye, the device comprising: An acquisition module, configured to: acquire a first spatial position of a first pupil center of the first eye and a second spatial position of a second pupil center of the second eye determined by the method according to any one of claims 1 to 10; The determination module is configured to determine the pupil distance according to the first spatial position and the second spatial position.

14. A computer device comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to any one of claims 1 to 10 or the method according to claim 11.

15. A non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to perform the method according to any one of claims 1 to 10 or the method according to claim 11.

16. A computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 10 or the method according to claim 11.

Images