Eye movement tracking method, readable storage medium, program product and electronic device
By dynamically adjusting the light emission parameters of the light source in electronic devices, the high power consumption problem caused by the continuous opening of the light source in extended real-life technology is solved, and the equipment's battery life and eye tracking accuracy are improved.
Patent Information
- Application Number
- PCT/CN2024/071604
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-08
- Filing Date
- 2024-01-10
- Publication Date
- 2024-11-14
AI Technical Summary
In electronic devices that use extended reality technology, the light source required for continuous eye tracking is always turned on, resulting in high power consumption and affecting the battery life of the device.
By collecting the user's eye image, dynamically adjust the light luminous parameters of the light source, such as reducing the brightness or turning off some light sources, and optimizing the light source's opening state according to the eye movement method and image parameters to reduce power consumption.
It effectively reduces the power consumption of light sources, improves the battery life of electronic devices, and improves the accuracy of eye tracking and the overall performance of the equipment.
Smart Images

Figure CN2024071604_14112024_PF_FP_ABST
Abstract
Description
Eye tracking method, readable storage medium, program product, and electronic device
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 8, 2023, with application number 202310511960.3 and application name “Eye tracking method, readable storage medium, program product and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of terminal technology, and in particular to an eye tracking method, a readable storage medium, a program product, and an electronic device. Background Art
[0003] Extended range (XR) technology, which can include virtual reality (VR), augmented reality (AR), and mixed reality (MR), is widely used in displays on various electronic devices, such as head-mounted displays (HMDs) like VR glasses and VR helmets. Electronic devices that use XR technology are typically equipped with a light source that can emit light toward the user's eyes and a camera for capturing images of the user's eyes. The camera detects the user's eye movements and gaze point by measuring the relationship between the projection of the light source in the eye image and the position of the user's eyeballs. This allows the display parameters of the electronic device to be adjusted to match the image displayed by the electronic device with the user's eye movements or gaze point.
[0004] However, since eye tracking is required to be performed continuously when users use electronic devices that adopt XR technology, the light source used to emit light to the user's eyes remains on during the user's use, which consumes high power and affects the battery life of the electronic device.
[0005] Summary of the Invention
[0006] In view of this, the present application provides an eye tracking method, a readable storage medium, a program product, and an electronic device. During eye tracking, the electronic device can dynamically adjust the luminous parameters of each light source, thereby reducing the power consumption of the light source and improving the battery life of the electronic device.
[0007] In a first aspect, the present application provides an eye tracking method, which is applied to an electronic device, wherein the electronic device includes multiple light sources, and the method includes: acquiring a first eye image of a user, the first eye image including a projection of at least one light source among the multiple light sources in the user's eye; adjusting the luminous parameters of at least some of the multiple light sources according to the first eye image; acquiring a second eye image, and performing eye tracking based on the second eye image.
[0008] That is, during eye tracking, the electronic device can adjust the lighting parameters of its light sources based on a first eye image captured at a certain moment (e.g., the current eye image frame below) when the electronic device captures a second eye image after that moment (e.g., the next eye image frame below). This can be done by, for example, reducing the number of light sources that emit light or lowering the brightness of the light sources. This can reduce the power consumed by the light sources and improve the battery life of the electronic device.
[0009] In a possible implementation of the first aspect, the lighting parameter includes at least one of the following parameters: lighting brightness, on / off.
[0010] In a possible implementation of the first aspect above, the above-mentioned adjusting the luminous parameters of at least some of the multiple light sources according to the first eye image includes: adjusting the luminous parameters of at least some of the multiple light sources in at least one of the following ways: adjusting the luminous brightness of at least some of the multiple light sources according to the image parameters of the first eye image, or turning on / off at least some of the light sources, wherein the image parameters include contrast; controlling the turning on or off of at least some of the multiple light sources according to the projection parameters of each projection, wherein the projection parameters include at least one of distribution, area, brightness, and shape; controlling the turning on or off of at least some of the multiple light sources according to the movement state of the user's eyes, wherein the movement state is obtained based on the first eye image.
[0011] In a possible implementation of the first aspect above, the above-mentioned adjusting the luminous brightness of at least some of the multiple light sources, or turning on / off at least some of the light sources according to the image parameters of the first eye image, includes: when the contrast of the first eye image is greater than the first contrast, reducing the luminous brightness of at least one light source, or turning off some of the at least one light source; when the contrast of the first eye image is less than the second contrast, increasing the luminous brightness of at least one light source, or increasing the number of turned-on light sources.
[0012] That is, if the contrast of the first eye image is greater than the first contrast, the electronic device can reduce the brightness of at least one light source, or turn off some of the at least one light source, thereby reducing the power consumed by the light source and improving the battery life of the electronic device. If the contrast of the first eye image is less than the second contrast, the electronic device can increase the brightness of at least one light source, or increase the number of turned-on light sources, thereby reducing or increasing the contrast of the second eye image and improving the accuracy of eye tracking.
[0013] In a possible implementation of the first aspect, the controlling of turning on or off at least some of the multiple light sources according to the projection parameters of each projection includes: determining invalid projections in the first eye image and turning off the light sources corresponding to the invalid projections.
[0014] That is to say, when there is an invalid projection in the first eye image, the electronic device can turn off the light source corresponding to the invalid projection, which can not only reduce the power consumption of the light source, but also prevent the invalid projection from affecting the accuracy of eye tracking.
[0015] In a possible implementation of the first aspect above, the above-mentioned determination of an invalid projection in the first eye image includes: determining the first projection as an invalid projection when at least one of the following conditions of the first projection is met: in the first eye image, the distance between the first projection and the center of the user's eye is greater than the first distance; in the first eye image, the area of the connected domain formed by the first projection and the second projection corresponding to at least one second light source is greater than the first area, or the distance between the connected domain and the center of the user's eye is greater than the second distance; the area of the first projection is smaller than the second area; the brightness of the first projection is smaller than the first brightness; the similarity between the shape of the first projection and the first shape is less than the first similarity.
[0016] In a possible implementation of the first aspect above, the above-mentioned adjusting the luminous parameters of at least part of the multiple light sources according to the movement state of the user's eyes includes: when the user's eye movement is gaze, adjusting the number of turned-on light sources among the multiple light sources to a first number; when the user's eye movement is saccade or following movement, adjusting the number of turned-on light sources among the multiple light sources to a second number, wherein the first number is smaller than the second number.
[0017] That is to say, when the user's eye movement is gaze, the electronic device can reduce the number of turned-on light sources, thereby reducing the power consumption of the light sources and improving the battery life of the electronic device.
[0018] In a possible implementation of the first aspect above, the method further includes: when the distance between the first position of the user's pupil center in the first eye image and the second position of the user's pupil center in the third eye image is less than a third distance, adjusting the number of turned-on light sources among the multiple light sources to a third number; when the distance between the first position and the second position is greater than or equal to the third distance, adjusting the number of turned-on light sources among the multiple light sources to a fourth number; wherein the third number is less than the fourth number, and the third eye image is an eye image captured before the first eye image.
[0019] That is to say, the electronic device can adjust the number of turned-on light sources according to the changes in the center of the user's pupil (the changes reflect the movement state of the user's eyes), so that when the user's eyes move slowly, the number of turned-on light sources can be reduced, thereby reducing the power consumption of the light sources and improving the battery life of the electronic device.
[0020] In a possible implementation of the first aspect above, the second eye image includes multiple frames of eye images, and the method also includes: in the process of acquiring multiple frames of eye images, controlling multiple light sources to flash, and determining a model of the user's eye based on the second eye image.
[0021] That is to say, when the second eye image includes multiple frames of eye images for determining a model of the user's eyes, the electronic device can control the flashing of multiple light sources during the process of collecting multiple frames of eye images to collect eye images with different contrasts, which is conducive to improving the accuracy of the determined model of the user's eyes, thereby improving the accuracy of eye tracking.
[0022] In a possible implementation of the first aspect, a flashing frequency of the plurality of light sources is less than a capture frequency of a plurality of frames of eye images.
[0023] In a second aspect, the present application provides a readable storage medium, which includes instructions. When the instructions are executed by an electronic device, the electronic device implements any one of the eye tracking methods provided by the above-mentioned first aspect and various possible implementations of the above-mentioned first aspect.
[0024] In a third aspect, the present application provides an electronic device comprising: multiple light sources; a memory for storing instructions; and a processor for executing instructions so that the electronic device controls the multiple light sources to implement any one of the eye tracking methods provided by the first aspect and various possible implementations of the first aspect.
[0025] In a fourth aspect, the present application provides a program product, which, when running on an electronic device, enables the electronic device to implement any one of the eye tracking methods provided by the first aspect and various possible implementations of the first aspect.
[0026] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be referred to the description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] FIG1 shows a schematic diagram of an eye tracking scenario according to some embodiments of the present application;
[0028] FIG2A shows a schematic diagram of an eye image according to some embodiments of the present application;
[0029] FIG2B shows a schematic diagram of another eye image according to some embodiments of the present application;
[0030] FIG2C shows a schematic diagram of another eye image according to some embodiments of the present application;
[0031] FIG3 is a schematic diagram showing a process of adjusting light emission parameters of a light source by user eye movement according to some embodiments of the present application;
[0032] FIG4A is a schematic diagram showing a current frame eye image and a next frame eye image according to some embodiments of the present application;
[0033] FIG4B shows another schematic diagram of a current frame eye image and a next frame eye image according to some embodiments of the present application;
[0034] FIG4C shows a schematic diagram of another current frame eye image and a next frame eye image according to some embodiments of the present application;
[0035] FIG4D shows another schematic diagram of a current frame eye image and a next frame eye image according to some embodiments of the present application;
[0036] FIG5 shows a schematic structural diagram of a VR glasses 10 according to some embodiments of the present application;
[0037] FIG6A shows a schematic diagram of the distribution of light sources in a light source group 11L / light source group 11R according to some embodiments of the present application;
[0038] FIG6B shows another schematic diagram of the distribution of light sources in the light source group 11L / light source group 11R according to some embodiments of the present application;
[0039] FIG7 is a schematic diagram showing a flow chart of an eye tracking method according to some embodiments of the present application;
[0040] FIG8A shows a schematic diagram of a light source switch state for the situation shown in FIG2B according to some embodiments of the present application;
[0041] FIG8B shows a schematic diagram of a light source switch state for the situation shown in FIG2C according to some embodiments of the present application;
[0042] FIG9 is a schematic diagram showing a process of adjusting light source parameters according to a user's pupil position according to some embodiments of the present application;
[0043] FIG10A is a schematic diagram showing a flow chart of another eye tracking method according to some embodiments of the present application;
[0044] FIG10B shows a schematic diagram of acquiring a calibration eye image according to some embodiments of the present application;
[0045] FIG11A is a schematic diagram showing a process of an eye tracking method according to some embodiments of the present application;
[0046] FIG11B shows a control timing diagram of an eye tracking method according to some embodiments of the present application. DETAILED DESCRIPTION
[0047] Illustrative embodiments of the present application include, but are not limited to, an eye tracking method, a readable storage medium, a program product, and an electronic device.
[0048] The technical solution of this application is introduced below with reference to the accompanying drawings.
[0049] It can be understood that eye tracking, also known as eye tracking, is to detect the movement state of the user's eyes relative to the user's head or electronic device and / or the gaze point of the user's eyes, so that the electronic device can adjust the display parameters according to the tracking results, so that the picture displayed by the electronic device matches the movement state or gaze point of the user's eyes.
[0050] It is understood that the technical solution of this application is applicable to any electronic device that performs eye tracking by capturing the projections of multiple light sources on the user's eyes, including but not limited to HMDs, head-up displays (HUDs), and VR / AR / MR devices such as headphones, helmets, goggles, or glasses. For ease of description, the technical solution of this application is described below using VR glasses as an example.
[0051] FIG1 shows a schematic diagram of an eye tracking scenario according to some embodiments of the present application.
[0052] As shown in Figure 1, the VR glasses 10 include a light source group 11L and a light source group 11R. The light source group 11L includes a plurality of light sources for emitting light to the user's left eye when the user wears the VR glasses 10, so as to project the light source of the light source group 11L into the user's left eye. The light source group 11R includes a plurality of light sources for emitting light to the user's right eye when the user wears the VR glasses 10, so as to project the light source of the light source group 11R into the user's right eye. The VR glasses 10 also include cameras for capturing images of the user's eyes, such as a camera 13L (not shown) for capturing images of the user's left eye and a camera 13R for capturing images of the user's right eye, as well as a display screen 12L (not shown) and a display screen 12R for displaying images.
[0053] It can be understood that the eye tracking device for the user's left eye in the VR glasses 10 (including at least the light source group 11L and the camera 13L) and the eye tracking device for the user's left eye (including at least the light source group 11R and the camera 13R) can adopt a symmetrical structure. The following is an introduction based on the eye tracking device for the user's right eye.
[0054] Referring to Figure 1 , for ease of description, when the VR glasses 10 are worn by a user, the horizontal direction is defined as the X direction, the vertical direction is defined as the Y direction, and the direction perpendicular to the XY plane is defined as the Z direction. The display screen 12R, camera 13R, and light source group 11R are arranged in sequence along the Z direction.
[0055] When a user wears the VR glasses 10 to view an image on the display screen 12R, the light sources in the light source group 11R emit light toward the user's right eye, projecting at least some of the light sources in the light source group 11R into the user's right eye. After receiving the light from the light source group 11R, the user's eye can reflect the received light to the camera 13R, so that the camera 13R captures an eye image P1 of the user's right eye. The eye image P1 includes the user's eye image and a projection R1 of at least some of the light sources in the light source group 11R on the user's eye. After acquiring the eye image P1, the VR glasses 10 can determine the movement state of the user's eye relative to the user's head or the VR glasses 10 based on the positional relationship between the projection R1 and the user's eye, that is, perform eye tracking on the user's eye.
[0056] It can be understood that the arrangement of the display screen 12R, camera 13R, and light source group 11R shown in Figure 1 is only an example. In other implementations, the display screen 12R, camera 13R, and light source group 11R may also be arranged in other forms, which is not limited here.
[0057] However, when the user uses the VR glasses 10, the VR glasses 10 usually keep all the light sources in the light source group 11L and the light source group 11R turned on, which increases the power consumption of the light source group 11L and the light source group 11R and reduces the battery life of the VR glasses 10.
[0058] It is understood that in some scenarios, changes in the image parameters of the eye image (such as contrast) within the allowable range of the eye tracking algorithm will not affect the accuracy of eye tracking. When other factors are the same, the greater the brightness of the light source, the greater the contrast of the image captured by the camera, and the higher the power consumption of the light source. Therefore, when the contrast of the captured eye image is high, the VR glasses 10 can reduce the brightness of the light source to reduce the power consumption of light source group 11L and light source group 11R.
[0059] It is understood that in some scenarios, due to differences in eye shape and size between users, the projection of some light sources on the user's eyes may not improve the accuracy of eye tracking or may even reduce the accuracy of eye tracking (hereinafter referred to as invalid projection). Therefore, the VR glasses 10 can turn off the light sources corresponding to the invalid projections to reduce the power consumption of the light source group 11L and the light source group 11R.
[0060] It is understood that in some scenarios, the number of reference objects (light source projections) required by the eye tracking algorithm to achieve the same tracking accuracy varies depending on the user's eye movement state (e.g., running speed, eye movement pattern, etc.). For example, the faster the user's eye movement, the more reference objects are required. Therefore, the VR glasses 10 can activate different numbers of light sources based on the user's eye movement state, reducing the number of light sources activated when the user's eye dynamic speed is slower, thereby reducing the power consumption of light source group 11L and light source group 11R.
[0061] In view of this, an embodiment of the present application provides an eye tracking method. During the eye tracking process, after the VR glasses 10 capture a frame of eye image (hereinafter referred to as the current frame eye image, as the first eye image), the VR glasses 10 can dynamically adjust the light-emitting parameters of at least some of the light sources in the light source group 11L and the light source group 11R when the VR glasses 10 capture the next frame or multiple frames of eye image (as the second eye image) based on at least one of the image parameters of the current frame eye image, the user's motion state determined based on the current frame eye image, and the projection parameters of the light source projection in the current frame eye image (such as projection distribution, projection area, projection brightness, projection shape, etc.), such as reducing the brightness of the light source, turning off some light sources, etc.
[0062] In this way, since the higher the brightness of the light source, the higher the power consumed, in some scenarios, the power consumption of the light source group 11L and the light source group 11R can be reduced, which is beneficial to improving the battery life of the VR glasses 10.
[0063] In some embodiments, the VR glasses 10 can reduce the brightness of at least some light sources in light source group 11L and light source group 11R when the contrast of the current frame eye image is greater than the first contrast, so as to reduce the contrast of the next frame eye image collected, thereby reducing the power consumption of light source group 11L and light source group 11R.
[0064] Referring to Figure 2A , assuming the contrast of the current eye image frame P2A captured by camera 13R is greater than the first contrast ratio, the VR glasses 10 can reduce the brightness of at least some of the light sources in light source group 11R when camera 13R captures the next eye image frame. As shown in Figure 2A , the contrast of the next eye image frame P2B captured by camera 13R is lower than that of the current eye image frame P2A. This reduces the power consumption of light source group 11R and improves the battery life of the VR glasses 10.
[0065] It is understood that when the contrast of the eye image is too high, the accuracy of the light source projection determined by the VR glasses 10 will be reduced, thereby affecting the accuracy of eye tracking. Therefore, when the contrast of the eye image is greater than the first contrast, reducing the brightness of the light source is beneficial to improving the accuracy of eye tracking.
[0066] It is understood that in some embodiments, when the contrast of the eye image is too low, the accuracy of the light source projection determined by the VR glasses 10 may be reduced, thereby affecting the accuracy of eye tracking. Therefore, when the contrast of the eye image is less than the second contrast, the VR glasses 10 may also increase the brightness of the light source or activate more light sources to improve the accuracy of eye tracking.
[0067] In some implementations, if an invalid projection exists in the current frame of the eye image, the VR glasses 10 may turn off the light source corresponding to the invalid projection to improve the battery life of the VR glasses 10. In other words, the VR glasses 10 may first determine the invalid projection in the current frame of the eye image, and then turn off the light source corresponding to the invalid projection when capturing the next frame or multiple frames of eye images.
[0068] It is understood that in some embodiments, if a projection in the eye image meets at least one of the following closing conditions, the VR glasses 10 may determine the projection as an invalid projection:
[0069] The distance between the projection and the center of the user's eye in the eye image is greater than the first distance.
[0070] The area of the connected domain formed by the projection and at least one other projection in the eye image is greater than the first area, or the distance between the center of the connected domain and the center of the user's eye is greater than the second distance. It can be understood that a connected domain is a collection of multiple pixels with the same or similar values in the image.
[0071] The area of the projection in the eye image is smaller than the second area.
[0072] The brightness of the projection in the eye image is less than the second brightness.
[0073] The similarity between the shape in the eye image and the first shape is less than the first similarity. It is understood that the first shape is a valid projected shape, such as a circle, an ellipse, a polygon, etc., and the first shape can be obtained through experience or experimentation. The embodiments of the present application do not limit the specific shape of the first shape.
[0074] For example, referring to Figure 2B , if the distance between projections NR1 and NR2 in the current eye image P3A captured by camera 13R and the center of the human eye is greater than a first distance, the VR glasses 10 may disable the light sources corresponding to projections NR1 and NR2. As shown in Figure 2B , compared to the current eye image P3A, the eye image P3B captured by camera 13R after capturing the current eye image P3A no longer includes invalid projections. In other words, when camera 13R captures eye image P3B, the light sources corresponding to projections NR1 and NR2 are disabled, reducing the power consumption of light source group 11R.
[0075] For another example, referring to Figure 2C , the current eye image P4A captured by camera 13R contains multiple projections forming a connected domain CD1. If the area of connected domain CD1 is larger than the first area, the VR glasses 10 may determine these multiple projections as invalid projections and shut down the light sources corresponding to connected domain CD1. As shown in Figure 2C , compared to the current eye image P4A, the eye image P4B captured by camera 13R after capturing the current eye image P4A no longer includes invalid projections. In other words, when camera 13R captures eye image P4B, the light sources corresponding to the projections in connected domain CD1 are shut down, reducing the power consumption of light source group 11R.
[0076] It is understandable that in other embodiments, the VR glasses 10 may also determine invalid projections by other means, such as by using the judgment logic of the eye tracking algorithm adopted by the VR glasses 10. The method for determining invalid projections is not limited here.
[0077] It can be understood that in some implementations, the movement state of the user's eyes can be represented by eye movement patterns, wherein eye movement patterns can include fixation, saccade, and smooth pursuit. In this way, the VR glasses 10 can determine the user's eye movement pattern based on the current frame eye image, and adjust the brightness of at least some of the light sources in the light source group 11L and the light source group 11R according to the determined eye movement pattern. For example, assuming that the light source group 11L and the light source group 11R each include N (N>1) light sources, when the user's eye movement pattern is fixation, each light source group can turn on M light sources, and when the user's eye movement pattern is saccade or pursuit, each light source group can turn on P light sources, wherein M <P≤N。
[0078] In some implementations, the VR glasses 10 can determine the eye movement pattern through an eye movement classification module. Referring to FIG3 , the VR glasses 10 can input the current frame eye image into the eye movement classification module to obtain the user's eye movement pattern and / or pupil position, and send the user's eye movement pattern and / or pupil position to the light source control module. The light source control module can determine the number of light sources to turn on when capturing the next frame of eye image based on the user's eye movement pattern and / or pupil position, and control the switching of each light source when capturing the next frame of eye image through light source driving.
[0079] For example, when the user's eye movement is gaze, each light source group may turn on M light sources; when the user's eye movement is saccade or follow-up, each light source group may turn on P light sources.
[0080] It is understood that the movement of the user's eyes can also be represented by changes in the user's pupils. Therefore, in some embodiments, the VR glasses 10 can adjust the lighting parameters of each light source when capturing the next eye image frame based on the distance between the position of the user's pupils in the current eye image frame and the position of the user's pupils in the previous eye image frame. For example, when the distance is less than a third distance, each light source group can activate M light sources; when the distance is greater than or equal to the third distance, each light source group can activate P light sources.
[0081] It is understood that the eye movement classification module may include a speed detection module, a dispersion detection module, and a synthesis module. The speed detection module is used to determine the speed of the user's eye movement; the dispersion detection module is used to determine the dispersion of the projections in the current frame of the eye image; and the synthesis module is used to determine the user's eye movement pattern or pupil position based on the user's eye movement speed and the dispersion of the projections.
[0082] In some implementations, when determining the model of the human eye, the VR glasses 10 can also control the flashing of the light sources in the light source group 11L and the light source group 11R to improve the effectiveness of the light source projection in the collected eye image, thereby improving the accuracy of the determined human eye model.
[0083] It is understandable that in other embodiments, the VR glasses 10 can also combine the above-mentioned multiple methods to adjust the light-emitting parameters of the light sources in the light source group 11L and the light source group 11R.
[0084] For example, the VR glasses 10 can adjust the brightness of the light sources in light source group 11L and light source group 11R based on the image parameters of the current frame eye image, and can also turn off the light sources corresponding to invalid projections based on invalid projections in the current frame eye image. Referring to Figure 4A, the distance between projections NR3 and NR4 in the current frame eye image P5A captured by camera 13R and the center of the human eye is greater than the first distance, and the contrast of the current frame eye image P5A is greater than the first contrast. When capturing the next frame eye image P5B, camera 13R can turn off the light sources corresponding to projections NR3 and NR4 and reduce the brightness of the turned-on light sources.
[0085] For another example, the VR glasses 10 can adjust the brightness of the light sources in the light source group 11L and the light source group 11R according to the image parameters of the current frame eye image, and turn on or off the corresponding number of light sources according to the movement state of the user's eyes. Referring to Figure 4B, the contrast of the current frame eye image P6A captured by the camera 13R is greater than the first contrast, and it is determined that the user's eye movement mode is gaze. When the camera 13R captures the next frame eye image P6B, the number of turned-on light sources can be reduced to the number corresponding to gaze (for example, 6) and the brightness of the turned-on light sources can be reduced.
[0086] For another example, the VR glasses 10 can not only turn off the light sources corresponding to invalid projections based on the invalid projections in the current frame eye image, but also turn on or off the corresponding number of light sources based on the movement state of the user's eyes. Referring to Figure 4C, the distance between projection NR5 and projection NR6 in the current frame eye image P7A captured by the camera 13R and the center of the human eye is greater than the first distance, and it is determined that the user's eye movement mode is gaze. When the camera 13R captures the next frame eye image P7B, it can turn off the light sources corresponding to projection NR5 and projection NR6, reduce the number of turned-on light sources to the number corresponding to gaze (for example, 6), and reduce the brightness of the turned-on light sources.
[0087] For example, the VR glasses 10 can adjust the brightness of the light sources in the light source group 11L and the light source group 11R according to the image parameters of the current frame eye image, turn off the light sources corresponding to the invalid projections according to the invalid projections in the current frame eye image, and turn on or off the corresponding number of light sources according to the movement state of the user's eyes. Referring to Figure 4D, the distance between projection NR7 and projection NR8 in the current frame eye image P8A captured by the camera 13R and the center of the human eye is greater than the first distance, the contrast of the current frame eye image P8A is greater than the first contrast, and it is determined that the user's eye movement mode is gaze. When the camera 13R captures the next frame eye image P8B, it can turn off the light sources corresponding to projection NR7 and projection NR8, reduce the number of turned-on light sources to the number corresponding to gaze (for example, 6), and reduce the brightness of the turned-on light sources.
[0088] It is understandable that in other embodiments, the VR glasses 10 can also dynamically adjust the luminous parameters of some light sources in the light source group 11L and the light source group 11R when capturing the next frame or multiple frames of eye images based on the current frame eye image in other ways, which is not limited here.
[0089] The following describes the technical solution of the present application in conjunction with the structure of the VR glasses 10.
[0090] First, the structure of the VR glasses 10 is introduced.
[0091] FIG5 shows a structural block diagram of a VR glasses 10 according to some embodiments of the present application.
[0092] As shown in Figure 5, the VR glasses 10 may include a processor 110, a memory 120, a sensor module 130, an audio module 140, a button 150, an input and output interface 160, a communication module 170, a power supply module 180, an eye tracking module 11, a display module 12, etc.
[0093] The processor 110 is generally used to control the overall operation of the VR glasses 10 and may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video processing unit (VPU) controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.
[0094] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.
[0095] In some embodiments, the processor 110 may be configured to provide instructions for adjusting the lighting parameters of the light sources in the light source groups 11L and 11R when the cameras 13L and 13R capture the next frame of eye images based on the current frame of eye images captured by the cameras 13L and 13R.
[0096] In some embodiments, the processor 10 can also be used to run the instructions of the aforementioned eye movement classification module and light source control module to determine the user's eye movement pattern and pupil position based on the current frame eye image, and adjust the light-emitting parameters of the light sources in light source group 11L and light source group 11R when capturing the next frame eye image according to the determined eye movement pattern or pupil position.
[0097] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input / output (GPIO) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.
[0098] For example, in some embodiments, the processor 110 can be connected to the light source driver 14 through an I2C interface or a GPIO interface, and send instructions to the light source driver through the I2C interface or the GPIO interface to adjust the light emission parameters of the light sources in the light source group 11L and the light source group 11R.
[0099] The memory 120 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the VR glasses 10 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as an image display function, a video playback function), etc. The data storage area may store data created during the use of the VR glasses 10 (such as eye images captured during eye tracking of the VR glasses 10), etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0100] The sensor module 130 may include sensors for detecting the working state and usage state of the VR glasses 10. These sensors include, but are not limited to, proximity sensors / contact sensors for detecting whether the user is wearing the VR glasses 10, and inertial sensors for detecting the user's head movement state.
[0101] The audio module 140 may include a speaker, a microphone, etc., for implementing audio functions.
[0102] The buttons 150 may be one or more. The buttons 150 may be in the form of buttons, switches, dials, and touch or near-touch sensing devices (such as touch sensors). The VR glasses 10 may trigger corresponding functions based on the user's operation of one or more of the buttons 150, such as adjusting the volume, controlling the VR glasses 10 to enter a calibration mode, etc.
[0103] The input / output interface 160 can connect other devices to the VR glasses 10 via appropriate components. Components may include, for example, audio / video jacks, data connectors, etc. For example, in some embodiments, the eye tracking module 11 may not be part of the VR glasses, but rather an external device connected to the VR glasses 10 via the input / output interface 160.
[0104] The communication module 170 may include a wireless communication module. The wireless communication function may be implemented through an antenna (not shown), a modem processor (not shown), and a baseband processor (not shown). The antenna is used to transmit and receive electromagnetic wave signals. The VR glasses 10 may include multiple antennas, each of which may be used to cover a single or multiple communication frequency bands. Different antennas may also be reused to improve antenna utilization. For example, antenna 1 may be reused as a diversity antenna for a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.
[0105] The wireless communication module can provide wireless communication solutions for the VR glasses 10, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module can be one or more devices that integrate at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module can also receive signals to be transmitted from the processor 110, frequency modulate them, amplify them, and convert them into electromagnetic waves for radiation through the antenna.
[0106] The power supply module 180 may include a battery for supplying power to various modules of the VR glasses 10 .
[0107] The eye tracking module 11 may include a light source group 11L, a light source group 11R, a camera 13L, a camera 13R, a light source driver 14 , and the like.
[0108] Light source group 11L is configured to emit light toward the user's left eye, allowing camera 13L to capture an image of the user's left eye and the projection of the light sources in light source group 11L on the user's eye. Light source group 11R is configured to emit light toward the user's right eye, allowing camera 13R to capture an image of the user's right eye and the projection of the light sources in light source group 11R on the user's eye.
[0109] It can be understood that the light source group 11L / the light source group 11R may include multiple light sources, and the number of light sources in each light source group may be, for example, 6, 7, 8, 10, etc., or other numbers, which are not limited here.
[0110] It is understandable that the light emission wavelength of the light source may be in the infrared band, the visible light band, or other bands, which are not limited here.
[0111] It is understood that the light sources in light source group 11L or light source group 11R may be light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes or active-matrix organic light-emitting diodes (AMOLEDs), flexible light-emitting diodes (FLEDs), Mini-LEDs, Micro-LEDs, Micro-OLEDs, quantum dot light-emitting diodes (QLEDs), etc. In other embodiments, the light sources in light source group 11L and light source group 11R may also be other forms of light sources, which are not limited here.
[0112] It can be understood that the light sources in the light source group 11L or the light source group 11R can be distributed on the edge of a circle, an ellipse, a polygon, or other irregular shapes, which is not limited here.
[0113] For example, in some embodiments, referring to FIG6A , the light sources in light source group 11L / light source group 11R may each include eight light sources, and the eight light sources may be arranged on the same circumference. For example, the eight light sources in each light source group may be arranged at positions of 5°, 30°, 150°, 175°, 200°, 225°, 315°, and 340° on the circumference, with the positive half axis of the X-axis being 0°.
[0114] For another example, in some embodiments, referring to FIG6B , the light sources in light source group 11L / light source group 11R may each include 10 light sources, and the 10 light sources may be arranged on the same circumference. For example, with the positive half axis of the X-axis on the circumference being 0°, the 10 light sources in each light source group may be arranged at positions of 9°, 36°, 60°, 120°, 144°, 171°, 198°, 225°, 315°, and 342° on the circumference.
[0115] Camera 13L / camera 13R can be a camera that matches the wavelength of light emitted by the light sources in light source group 11L / light source group 11R, and is used to capture images of the user's eyes. For example, when the light source is a visible light source, camera 13L / camera 13R can be a camera that operates in the visible light band; when the light source is an infrared light source, camera 13L / camera 13R can be a visible infrared camera.
[0116] It can be understood that the camera 13L / camera 13R can be set at any position where the user's eye image can be captured.
[0117] It can be understood that in some embodiments, the VR glasses 10 may include only one camera that can simultaneously capture the user's left and right eye images, or may use a larger number of cameras (for example, greater than 2) to capture the user's eye images, which is not limited here.
[0118] The light source driver 14 is used to adjust the luminous parameters of each light source according to the instructions or signals of the processor 110, for example, according to the instructions or signals of the processor 110, adjust the voltage or current applied to each light source to adjust the brightness of each light source, flicker, turn on or off each light source, etc.
[0119] It can be understood that the structure of the VR glasses 10 shown in FIG5 is only an example. In other embodiments, the VR glasses 10 may include more or fewer modules, or some modules may be merged or split, which is not limited here.
[0120] The following describes the technical solution of the present application in conjunction with the structure of the VR glasses 10 shown in FIG5 .
[0121] FIG7 is a flow chart illustrating an eye tracking method according to some embodiments of the present application. The method may be performed by the VR glasses 10, such as the processor 110 or other computing unit in the VR glasses. As shown in FIG7 , the method may include the following steps:
[0122] S701: Acquire the eye image of the current frame.
[0123] During the eye tracking process, the VR glasses 10 collect the eye image of the current frame.
[0124] It can be understood that in some embodiments, when the VR glasses 10 capture the current frame of eye image, the light-emitting parameters of the light sources in the light source group 11L / light source group 11R can be determined by the previous frame of eye image.
[0125] It can be understood that in some embodiments, if the current frame eye image is the first frame eye image collected during the use of the VR glasses 10, when the VR glasses 10 collect the current frame eye image, the light-emitting parameters of the light sources in the light source group 11L / light source group 11R can be the light-emitting parameters preset by the VR glasses, for example, all light sources use the same preset brightness to emit light.
[0126] It will be appreciated that in some embodiments, when the VR glasses 10 are in use, the cameras 13L and 13R may capture eye images of the user at the same sampling frequency, for example, at a sampling frequency of 30 to 160 Hz. For example, assuming the sampling frequency of the cameras 13L and 13R is 45 Hz, the cameras 13L and 13R will capture a frame of eye images every 1000 ms / 45 = 22.2 ms. The current frame of eye image can be any frame of eye image captured by either camera 13L or camera 13R.
[0127] It can be understood that in some embodiments, after capturing the current frame eye image, the VR glasses 10 can also perform eye tracking based on the current frame eye image to determine the movement state of the user's eyes or the user's gaze point.
[0128] S702: Determine the light emission parameters of each light source in the light source group 11L / light source group 11R when acquiring the next frame of eye image according to the current frame of eye image.
[0129] After capturing the current frame eye image, the VR glasses 10 can determine the luminous parameters of each light source in the light source group 11L / light source group 11R when capturing the next frame eye image based on at least one of the image parameters of the current frame eye image, the movement state of the user's eyes determined based on the current frame eye image, and the projection parameters of the light source projection in the current frame eye image (such as projection distribution, projection area, projection brightness, projection shape, etc.).
[0130] In some implementations, after capturing a current eye image frame, the VR glasses 10 may first determine image parameters, such as contrast, for the current eye image frame. The VR glasses 10 may then compare the determined image parameters with preset image parameters and, based on the comparison results, determine the luminous parameters of each light source in the light source group 11L / light source group 11R when capturing the next eye image frame.
[0131] For example, when the contrast of the eye image of the current frame is greater than the first contrast, the brightness of the currently turned-on light sources in light source group 11L and light source group 11R is reduced or some of the currently turned-on light sources are turned off to reduce the contrast of the next frame of eye image collected, thereby reducing the power consumption of the light source group.
[0132] For another example, when the contrast of the eye image of the current frame is less than the second contrast, the brightness of the currently turned-on light sources in light source group 11L and light source group 11R is increased or the number of turned-on light sources is increased to increase the contrast of the next frame of eye image collected, thereby improving the accuracy of eye tracking.
[0133] In some implementations, after capturing a current frame of eye imagery, the VR glasses 10 may first determine whether there are any invalid projections in the current frame. If an invalid projection is determined to be present, the light source corresponding to the invalid projection is turned off. This can reduce the power consumption of the VR glasses 10 and improve eye tracking accuracy.
[0134] It is understood that in some embodiments, if a projection in the eye image meets at least one of the following closing conditions, the VR glasses 10 may determine the projection as an invalid projection:
[0135] The distance between the projection and the center of the user's eye in the eye image is greater than the first distance.
[0136] The area of the connected domain formed by the projection and at least one other projection in the eye image is greater than the first area, or the distance between the center of the connected domain and the center of the human eye is greater than the second distance. It can be understood that the connected domain is a collection of multiple pixels with the same or similar values in the image.
[0137] The area of the projection in the eye image is smaller than the second area.
[0138] The brightness of the projection in the eye image is less than the second brightness.
[0139] The similarity between the shape in the eye image and the first shape is less than the first similarity. It is understood that the first shape is a valid projected shape, such as a circle, an ellipse, a polygon, etc., and the first shape can be obtained through experience or experimentation. The embodiments of the present application do not limit the specific shape of the first shape.
[0140] For example, the light sources in the light source group 11R are arranged in the manner shown in FIG. 6B .
[0141] For the scenario shown in FIG2B , referring to FIG8A , all light sources in light source group 11R are turned on when capturing the current eye image frame P3A. Based on the presence of invalid projections NR1 and NR2 in the current eye image frame P3A, which correspond to the light sources at 225° and 315°, respectively, the VR glasses 10 can determine that the light sources at 225° and 315° should be turned off when capturing the next eye image frame P3B.
[0142] Regarding the scenario shown in FIG2C , referring to FIG8B , when capturing the current frame eye image P4A, all light sources in light source group 11R except the light source at the 225° position are turned on. Based on the presence of a connected domain CD1 in the current frame eye image P4A, the area of the connected domain CD1 being greater than the first area, and the projections within the connected domain CD1 corresponding to the light sources at the 9° position, the 315° position, and the 342° position, respectively, the VR glasses 10 can determine that when capturing the next frame eye image P4B, the light sources at the 9° position, the 225° position, the 315° position, and the 342° position are all turned off.
[0143] In some implementations, after the VR glasses 10 capture the eye image of the current frame, they can first determine the eye movement pattern of the user and adjust the number of light sources turned on in the light source group 11L / light source group 11R to the corresponding number according to the user's eye movement pattern. For example, assuming that the light source group 11L and the light source group 11R each include N (N>1) light sources, when the user's eye movement pattern is gaze, each light source group can turn on M light sources. When the user's eye movement pattern is eye saccade or tracking movement, each light source group can turn on P light sources, where M <P≤N。
[0144] In some implementations, after the VR glasses 10 capture the current frame eye image, they can determine the number of turned-on light sources in the light source group 11L / light source group 11R based on the position of the user's pupils in the current frame eye image and the distance between the user's pupils in the previous frame eye image.
[0145] For example, referring to FIG9 , when the current eye image is the K-th frame eye image captured by the camera 13R, the VR glasses 10 can determine that the pupil center position of the user's right eye in the K-th frame eye image is (x1, y1). Then, the pupil center position (x1, y1) is compared with the pupil center position of the user's right eye in the K-1-th frame eye image (x0, y0), and the change amount of the pupil center position in the X direction Δx = |x1-x0| and the change amount Δy in the Y direction Δx = |y1-y0| are determined. Then, Δx and Δy are respectively compared with the preset change amount D X 、D Y For comparison, when Δx≤D X Or Δy≤D Y, it is determined that N light sources are turned on when acquiring the K+1th frame of eye image.
[0146] Continuing to refer to FIG9 , when the eye image before the frame is the K+1 frame eye image captured by the camera 13R, the VR glasses 10 can determine that the pupil center position of the user's right eye in the K+1 frame eye image is (x2, y2). Then, the pupil center position (x1, y1) is compared with the pupil center position of the user's right eye in the K frame eye image (x1, y1), and the change amount of the pupil center position in the X direction Δx = |x2-x1| and the change amount Δy in the Y direction Δy = |y2-y1| are determined. Then, Δx and Δy are respectively compared with the preset change amount D X 、D Y For comparison, when Δx>D X And Δy>D Y , it is determined that P light sources are turned on when acquiring the K+2-th frame of eye image.
[0147] It is understood that in other embodiments, the VR glasses 10 may also be in the range of Δx≤D X And Δy≤D Y When Δx>D X Or Δy>D Y When the K+2th frame of eye image is collected, it is determined that P light sources are turned on, and no limitation is made here.
[0148] It can be understood that in other embodiments, the VR glasses 10 may also combine multiple methods of the above-mentioned adjustment methods to determine the luminous parameters of each light source in the light source group 11L / light source group 11R when capturing the next frame of eye image, or may determine the luminous parameters of each light source in the light source group 11L / light source group 11R when capturing the next frame of eye image based on the current frame of eye image through other methods.
[0149] S703: Adjust the lighting parameters of at least part of the light sources in the light source group 11L / the light source group 11R according to the determined lighting parameters and capture the next frame of eye image.
[0150] After determining the luminous parameters of each light source in light source group 11L / light source group 11R when capturing the next frame of eye image, the VR glasses 10 adjust the luminous parameters of the light sources in light source group 11L / light source group 11R to the determined luminous parameters and capture the next frame of eye image.
[0151] For example, when the VR glasses 10 determine to adjust the brightness of some light sources, they can input a digital or analog signal corresponding to the brightness to the light source driver 14, so that the light source driver 14 can output a corresponding current or voltage to each light source according to the received signal.
[0152] For another example, when the VR glasses determine whether to turn certain light sources on or off, they can input corresponding digital or analog levels or on / off instructions to the light source driver 14, so that the light source driver 14 can turn the corresponding light source on or off according to the received levels or instructions. For example, referring to FIG9 , when capturing the K+1th frame of the eye image, the processor 110 can output a high level corresponding to turning on N light sources and a high level corresponding to turning off PN light sources to the light source driver 14.
[0153] Exemplarily, for the situation shown in FIG2A , the VR glasses 10 may reduce the brightness of at least some of the light sources in the light source group 11R when the camera 13R captures the next frame of eye image and captures the next frame of eye image P2B.
[0154] Exemplarily, for the situation shown in FIG2B , the VR glasses 10 may turn off the light sources corresponding to projection NR1 and projection NR2 and capture the next frame of eye image P3B.
[0155] Exemplarily, corresponding to the situation shown in FIG2C , the VR glasses 10 may turn off the light source corresponding to the projection within the connected domain CD1 and capture the next frame of eye image P4B.
[0156] It can be understood that in some embodiments, the VR glasses 10 can send the determined lighting parameters to the light source driver 14 through the I2C bus, etc., to adjust the lighting parameters of at least some light sources in the light source group 11L / light source group 11R.
[0157] S704: Perform eye tracking based on the next frame of eye image collected.
[0158] After acquiring the next frame of eye image, the VR glasses 10 perform eye tracking based on the acquired eye image. For example, the VR glasses can track the user's eyes using an eye tracking algorithm.
[0159] Through the method provided in the application embodiment, the VR glasses 10 can reduce the brightness of the light source and reduce the number of turned-on light sources in some scenarios, thereby reducing the power consumption of each light source group 11L / light source group 11R and improving the battery life of the VR glasses 10.
[0160] It is understood that in some embodiments, when a user wears the VR glasses 10, the VR glasses 10 can calibrate the human eye, that is, determine the user's eye model, such as the shape, size, pupil distance, refractive power, astigmatism, etc. of the user's eyes. During the process of calibrating the human eye, the VR glasses 10 generally utilize a calibration algorithm based on multiple eye images. In some embodiments, during the process of collecting eye images for human eye calibration, the VR glasses 10 can also adjust the light parameters of the light sources in the light source group 11L / light source group 11R, such as controlling the light source flickering, to improve the effectiveness of the collected eye images, thereby improving the accuracy of the determined eye model.
[0161] Specifically, FIG10A shows a flow chart of another eye tracking method according to some embodiments of the present application. The method may be performed by the VR glasses 10, such as the processor 110 of the VR glasses 10. As shown in FIG10A , the method includes the following steps:
[0162] S1001: It is detected that the VR glasses 10 enter the calibration mode.
[0163] For example, when the VR glasses 10 detect that the user has put on the VR glasses 10, the VR glasses 10 can detect that the VR glasses have entered a calibration mode, thereby triggering the eye tracking method provided in an embodiment of the present application.
[0164] For another example, when the VR glasses 10 detect the user's operation through the button 150, they can enter the calibration mode and trigger the eye tracking method provided in the embodiment of the present application.
[0165] For another example, the VR glasses 10 may enter a calibration mode and trigger the eye tracking method provided in an embodiment of the present application when receiving a calibration instruction sent by other electronic devices, such as a calibration instruction sent by an electronic device connected to the VR glasses 10 through an application corresponding to the VR glasses 10.
[0166] S1002: Acquire the eye image of the current frame.
[0167] After entering the calibration mode, the VR glasses 10 collect the eye image of the current frame.
[0168] It can be understood that in some embodiments, if the current frame eye image is the first frame eye image collected during the calibration process of the VR glasses 10, when the VR glasses 10 collect the current frame eye image, the light-emitting parameters of the light sources in the light source group 11L / light source group 11R can be the light-emitting parameters preset by the VR glasses, for example, all light sources use the same preset brightness to emit light.
[0169] S1003: Determine the lighting parameters of each light source in each light source group 11L / light source group 11R during the calibration process according to the eye image of the current frame.
[0170] After capturing the current frame of eye image, the VR glasses 10 determine the luminous parameters of each light source in each light source group 11L / light source group 11R during the calibration process based on the current frame of eye image. The specific determination method can be referred to the aforementioned step S702 and will not be repeated here.
[0171] For example, if there is an invalid projection in the current frame eye image, the VR glasses 10 can turn off the light source corresponding to the invalid projection during the calibration process.
[0172] For example, referring to FIG10B , for the current frame eye image P3A shown in FIG2B , the VR glasses 10 can determine that the light source at the 225° position and the light source at the 315° position are turned off during the calibration process based on the presence of invalid projections NR1 and NR2 in the current frame eye image P3A, and the invalid projections NR1 and NR2 correspond to the light source at the 225° position and the light source at the 315° position, respectively.
[0173] S1004: Adjust the lighting parameters of at least part of the light sources in the light source group 11L / the light source group 11R according to the determined lighting parameters.
[0174] After determining the lighting parameters of each light source in the light source group 11L / light source group 11R during the calibration process, the VR glasses 10 adjust the lighting parameters of the light sources in the light source group 11L / light source group 11R to the determined lighting parameters.
[0175] For example, referring to FIG10B , the VR glasses 10 may turn off the light source at the 225° position and the light source at the 315° position.
[0176] S1005: Acquire multiple frames of calibration eye images, and control the light sources in the light source group 11L / the light source group 11R to flash during the acquisition process.
[0177] The VR glasses 10 collect multiple frames of calibration eye images and control the turned-on light sources in the light source group 11L / light source group 11R to flash during the collection process.
[0178] It is understood that in some embodiments, the VR glasses 10 can control the flickering of the turned-on light sources in the light source group 11L / light source group 11R by periodically adjusting the voltage or current applied to the turned-on light sources through the light source driver 14. For example, within a flickering cycle, the voltage or current applied to the turned-on light sources can be controlled to first gradually increase and then gradually decrease, or first gradually decrease and then gradually increase, to achieve flickering of the turned-on light sources in the light source group 11L / light source group 11R.
[0179] It is understood that in some embodiments, the flashing frequency of the light sources in light source group 11L / light source group 11R can be lower than the frequency of collecting the calibration eye image. In this way, the collected multiple frames of calibration eye images can include the user's eye images under different brightness levels of the light sources, which improves the effectiveness of the collected eye images and helps improve the accuracy of human eye calibration. For example, assuming that the sampling frequency of camera 13L and camera 13R is a, the flashing frequency of the light sources in light source group 11L / light source group 11R is b (b <a)。
[0180] For example, referring to FIG10B , by controlling the flickering of light sources other than the light source at the 225° position and the light source at the 315° position, the VR glasses 10 can capture multiple frames of calibration eye images with varying brightness and contrast. This prevents calibration accuracy from being affected by the brightness or contrast of the captured calibration eye images not meeting the calibration algorithm's requirements.
[0181] S1006: Determine the user's eye model based on the collected calibration eye image.
[0182] The VR glasses 10 determine the user's eye model based on the collected calibration eye images.
[0183] Through the methods provided in the application examples, the VR glasses 10 can reduce the brightness of the light sources and the number of active light sources in some scenarios, thereby reducing the power consumption of each light source group 11L / light source group 11R and improving the battery life of the VR glasses 10. Furthermore, because the VR glasses 10 capture multiple frames of calibration eye images, the active light sources in the light source groups 11L / light source groups 11R flicker, and the user's eye images captured at different light brightness levels can be improved, which helps improve the accuracy of the determined eye model.
[0184] It is understood that in some embodiments, camera 13L and camera 13R periodically capture eye images. The following uses the example of camera 13R periodically capturing an image of the user's right eye to describe the control timing of the eye tracking method provided by the embodiments of the present application.
[0185] Specifically, Figure 11A shows a process diagram of an eye tracking method provided by the present application according to some embodiments of the present application; Figure 11B shows a control timing diagram of an eye tracking method provided by the present application according to some embodiments of the present application.
[0186] As shown in FIG11A , the eye tracking method includes:
[0187] S11.1: The camera 13R performs exposure while the camera sampling signal is at a high level.
[0188] Exemplarily, the camera 13R performs exposure during the high level period of each cycle of the camera sampling signal, such as the period from time 0 to time T1 shown in FIG. 11B , to acquire image data corresponding to the eye image.
[0189] It can be understood that when the camera sampling signal is at a high level, the driving signals corresponding to the light sources that need to be turned on are also at a high level. For example, referring to FIG11B , for the scenarios shown in FIG2B and FIG8A , during the period from 0 to T1, the driving signals of all light sources in light source group 11R are at a high level, i.e., all light sources are in the turned-on state.
[0190] It will be appreciated that after exposure is complete, or when the camera sampling signal transitions from a high level to a low level, camera 13R may transition the drive signals for all light sources from a high level to a low level, thereby turning off all light sources. Thus, during periods when camera 13R is not exposing, such as from time T1 to time T shown in FIG. 11B , all light sources are turned off, thereby saving energy.
[0191] S11.2: The camera 13R obtains the eye image of the current frame and transmits it to the processor 110 through the SDK.
[0192] After the camera 13R completes exposure, for example, at time T2 shown in FIG. 11B , the camera 13R may acquire the eye image of the current frame and transmit the image to the processor 110 via a software development kit (SDK).
[0193] S11.3: The processor 110 determines the light emission parameters when acquiring the next frame of eye image.
[0194] After receiving the current eye image frame, for example, at time T3 shown in FIG. 11B , the processor 110 can determine, based on the current eye image frame, the lighting parameters of the light sources in light source groups 11L and 11R, such as brightness and on / off status, when capturing the next eye image frame. The specific determination method is described in steps S702 and S803 above and is not further described here.
[0195] S11.4: The processor 110 configures the lighting parameters of the light source when collecting the next frame of eye image via the I2C bus.
[0196] After determining the lighting parameters of the light sources in light source group 11L and light source group 11R when capturing the next frame of the eye image, the processor 110 configures the determined lighting parameters to the light source driver 14 via the I2C bus, for example, at time T4 shown in FIG11B . For example, the processor 110 can write the lighting brightness and on / off parameters to the registers of the light source driver 14 via the I2C bus.
[0197] It is understandable that in other embodiments, the processor 110 may also configure the light parameters of the light source when capturing the next frame of eye image through other buses or interfaces, such as the GPIO of the processor 110, which is not limited here.
[0198] S11.5: The light source driver 14 drives the light source to emit light according to the light-emitting parameters while the camera sampling signal is at a high level.
[0199] Exemplarily, when the processor 110 detects that the camera sampling signal corresponding to the next frame of the eye image transitions from a low level to a high level, such as at time T shown in FIG11B , the processor 110 sets the drive signal for the light source to be turned on, as determined in step S11.4, to a high level. Upon detecting the high-level light source drive signal, the light source driver 14 drives the corresponding light source to emit light according to the light emission parameters determined in step S11.4.
[0200] For example, corresponding to the situations shown in Figures 2B and 8A, the light source driver 14 can detect that the driving signals of the light sources at the 225° position and the 315° position are converted from low level to high level and turn on the other light sources in the light source group 11R except the light source at the 225° position and the light source at the 315° position according to the driving signals of the light sources at other positions.
[0201] S11.6: Camera 13R triggers an interrupt.
[0202] When the camera 13R completes exposure or when the camera sampling signal transitions from high to low, it can trigger an interrupt to transition the drive signals of each light source from high to low. Thus, upon receiving the interrupt, the light source driver 14 can respond to it and turn off all light sources to save energy.
[0203] The process of adjusting the luminous parameters of the light sources in the light source group 11L based on the current frame eye image of the user's left eye captured by the camera 13L can refer to the aforementioned steps S11.1 to S11.6 and will not be repeated here.
[0204] The method provided in this application allows for lowering the brightness of light sources and reducing the number of active light sources in some scenarios, thereby reducing the power consumption of each light source group 11L / light source group 11R and improving the battery life of the VR glasses 10. Furthermore, because the VR glasses 10 capture multiple frames of eye calibration images, the active light sources in light source group 11L / light source group 11R flicker, allowing for the capture of user eye images at varying brightness levels, which helps improve the accuracy of eye model calibration.
[0205] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program product or program code executed on a programmable system or device, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. When the computer program product or program code is executed on a programmable system or device, it can enable the programmable system or device to implement the eye tracking method provided in the various embodiments of the present application.
[0206] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit, or a microprocessor.
[0207] Program code can be implemented with a high-level programming language or an object-oriented programming language to communicate with the processing system. Where necessary, program code can also be implemented in assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any particular programming language. In either case, the language can be a compiled language or an interpreted language.
[0208] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to a floppy disk, an optical disk, an optical disc, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic, or other form of propagation signal. Accordingly, machine-readable media includes any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).
[0209] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and in some embodiments, such features may not be included or may be combined with other features.
[0210] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems raised by this application. In addition, in order to highlight the innovative part of this application, the above-mentioned device embodiments of this application do not introduce units / modules that are not closely related to solving the technical problems raised by this application. This does not mean that other units / modules do not exist in the above-mentioned device embodiments.
[0211] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a" do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.
Claims
1. An eye tracking method, applied to an electronic device, characterized in that: The electronic device includes a plurality of light sources, and the method includes: Acquire a first eye image of the user, where the first eye image includes a projection of at least one light source among the multiple light sources in the user's eye; adjusting light emission parameters of at least part of the plurality of light sources according to the first eye image; A second eye image is acquired, and eye tracking is performed based on the second eye image.
2. The method according to claim 1, characterized in that The light-emitting parameters include at least one of the following parameters: light-emitting brightness, on / off.
3. The method according to claim 1, characterized in that The step of adjusting the light emission parameters of at least part of the plurality of light sources according to the first eye image comprises: Adjusting the light emission parameters of at least part of the plurality of light sources by at least one of the following methods: adjusting the light luminance of at least some of the multiple light sources, or turning on / off at least some of the light sources according to image parameters of the first eye image, wherein the image parameters include contrast; Controlling turning on or off at least some of the plurality of light sources according to projection parameters of each of the projections, wherein the projection parameters include at least one of distribution, area, brightness, and shape; Controlling turning on or off of at least some of the multiple light sources according to a movement state of the user's eyes, wherein the movement state is obtained based on the first eye image.
4. The method according to claim 3, characterized in that The adjusting the light luminance of at least part of the plurality of light sources according to the image parameters of the first eye image, or turning on / off at least part of the light sources, comprises: When the contrast of the first eye image is greater than a first contrast, reducing the light luminance of the at least one light source, or turning off part of the at least one light source; When the contrast of the first eye image is smaller than the second contrast, the light brightness of the at least one light source is increased, or the number of turned-on light sources is increased.
5. The method according to claim 3, characterized in that: The controlling the turning on or off of at least part of the plurality of light sources according to the projection parameters of each projection comprises: An invalid projection in the first eye image is determined, and a light source corresponding to the invalid projection is turned off.
6. The method according to claim 5, characterized in that The determining of the invalid projection in the first eye image comprises: When the first projection meets at least one of the following conditions, the first projection is determined as an invalid projection: In the first eye image, the distance between the first projection and the center of the user's eye is greater than a first distance; In the first eye image, an area of a connected domain formed by the first projection and a second projection corresponding to at least one second light source is larger than the first area, or a distance of the connected domain from a center of a user's eye is larger than a second distance; The area of the first projection is smaller than the second area; The brightness of the first projection is less than the first brightness; The similarity between the shape of the first projection and the first shape is less than the first similarity.
7. The method according to claim 3, characterized in that The step of adjusting the light emitting parameters of at least part of the plurality of light sources according to the movement state of the user's eyes comprises: When the eye movement mode of the user is gaze, adjusting the number of turned-on light sources among the plurality of light sources to a first number; When the eye movement of the user is saccade or tracking movement, the number of turned-on light sources among the plurality of light sources is adjusted to a second number, wherein the first number is smaller than the second number.
8. The method according to claim 3, characterized in that The method further comprises: When a distance between a first position of a center of a pupil of the user in the first eye image and a second position of a center of a pupil of the user in the third eye image is less than a third distance, adjusting the number of turned-on light sources among the plurality of light sources to a third number; When the distance between the first position and the second position is greater than or equal to the third distance, adjusting the number of turned-on light sources among the plurality of light sources to a fourth number; The third number is smaller than the fourth number, and the third eye image is an eye image captured before the first eye image.
9. The method according to claim 1, characterized in that: The second eye image includes a plurality of frames of eye images, and the method further includes: During the process of acquiring the multiple frames of eye images, the multiple light sources are controlled to flash, and a model of the user's eyes is determined based on the second eye images.
10. The method according to claim 9, characterized in that The flashing frequency of the multiple light sources is less than the acquisition frequency of the multiple frames of eye images.
11. A readable storage medium, characterized in that: The readable storage medium includes instructions, and when the instructions are executed by the electronic device, the electronic device implements the eye tracking method according to any one of claims 1 to 10.
12. An electronic device, characterized in that: The electronic device comprises: Multiple light sources; A memory for storing instructions; A processor, configured to execute the instructions so that the electronic device controls the multiple light sources to implement the eye tracking method according to any one of claims 1 to 10.
13. A program product, characterized in that When the program product is run on an electronic device, the electronic device implements the eye tracking method according to any one of claims 1 to 10.
Citation Information
Patent Citations
Eye movement tracking method, readable storage medium, program product and electronic equipment
CN118915903A
Method For Operating An Eye Tracking Device And Eye Tracking Device For Providing An Active Illumination Control For Improved Eye Tracking Robustness
CN105828700A
Eyeball tracking sensor adjusting method, device and apparatus and storage medium
CN110051319A
Shooting parameter adjusting method and device of eye movement tracking equipment, equipment and medium
CN110062168A
Dynamic camera or light operation
US20160195927A1
