Eye movement tracking method, readable storage medium, program product and electronic equipment
By dynamically adjusting the light emission parameters of the light source, the high power consumption problem caused by the continuous operation of the light source in XR devices is solved, thereby improving the device's battery life and eye-tracking accuracy.
Patent Information
- Application Number
- CN202310511960.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-08
- Publication Date
- 2024-11-08
AI Technical Summary
When users use electronic devices with XR technology, continuously keeping the light source on leads to high power consumption and affects the device's battery life.
By acquiring images of the user's eyes, the emission parameters of the light source are dynamically adjusted, such as reducing brightness, turning off some light sources, or adjusting the number of light sources, to match the user's eye movement state and image parameters and optimize the use of the light source.
It reduces the power consumption of the light source, improves the battery life of electronic devices, and enhances the accuracy and efficiency of eye tracking.
Smart Images

Figure CN118915903A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal technology, and in particular to an eye tracking method, a readable storage medium, a program product, and an electronic device. Background Art
[0002] Extended range (XR) technology, which can include virtual reality (VR), augmented reality (AR), mixed reality (MR), etc., is widely used in the display of various electronic devices, such as VR glasses, VR helmets and other head-mounted displays (HMD). In electronic devices using XR technology, a light source that can emit light to the user's eyes and a camera for collecting user eye images are usually configured, and the user's eye movement pattern, gaze point, etc. are detected through the projection of the light source in the eye image and the position relationship of the user's eyeball, so as to adjust the display parameters of the electronic device so that the image displayed by the electronic device matches the movement pattern or gaze point of the user's eyes.
[0003] However, since eye tracking needs to be performed continuously when users use electronic devices that adopt XR technology, the light source used to emit light to the user's eyes remains on during the user's use, which consumes high power and affects the battery life of the electronic device. Summary of the invention
[0004] In view of this, the present application provides an eye tracking method, a readable storage medium, a program product, and an electronic device. During the eye tracking process, the electronic device can dynamically adjust the light parameters of each light source, thereby reducing the power consumption of the light source and improving the endurance of the electronic device.
[0005] In a first aspect, the present application provides an eye tracking method, which is applied to an electronic device, wherein the electronic device includes multiple light sources, and the method includes: acquiring a first eye image of a user, the first eye image including a projection of at least one light source among the multiple light sources in the user's eye; adjusting the light emission parameters of at least some of the multiple light sources according to the first eye image; acquiring a second eye image, and performing eye tracking based on the second eye image.
[0006] That is to say, during the eye tracking process, the electronic device can adjust the light emitting parameters of the light source of the electronic device when the electronic device acquires the second eye image (such as the next frame of eye image below) after the moment according to the first eye image acquired at a certain moment (such as the current frame of eye image below), such as reducing the number of light sources that emit light, reducing the brightness of the light sources that emit light, etc. In this way, the power consumed by the light source can be reduced and the endurance of the electronic device can be improved.
[0007] In a possible implementation of the first aspect, the light-emitting parameter includes at least one of the following parameters: light brightness, on / off.
[0008] In a possible implementation of the first aspect above, the adjusting the luminous parameters of at least some of the multiple light sources according to the first eye image includes: adjusting the luminous parameters of at least some of the multiple light sources in at least one of the following ways: adjusting the luminous brightness of at least some of the multiple light sources according to the image parameters of the first eye image, or turning on / off at least some of the light sources, wherein the image parameters include contrast; controlling turning on or off at least some of the multiple light sources according to the projection parameters of each projection, wherein the projection parameters include at least one of distribution, area, brightness, and shape; controlling turning on or off at least some of the multiple light sources according to the movement state of the user's eyes, wherein the movement state is obtained based on the first eye image.
[0009] In a possible implementation of the first aspect, the above-mentioned adjusting the luminous brightness of at least some of the multiple light sources according to the image parameters of the first eye image, or turning on / off at least some of the light sources, includes: when the contrast of the first eye image is greater than the first contrast, reducing the luminous brightness of at least one light source, or turning off some of the at least one light source; when the contrast of the first eye image is less than the second contrast, increasing the luminous brightness of at least one light source, or increasing the number of turned-on light sources.
[0010] That is to say, if the contrast of the first eye image is greater than the first contrast, the electronic device can reduce the brightness of at least one light source, or turn off part of the at least one light source, so as to reduce the power consumed by the light source and improve the endurance of the electronic device. If the contrast of the first eye image is less than the second contrast, the electronic device can increase the brightness of at least one light source, or increase the number of turned-on light sources, so as to reduce the contrast of the second eye image and improve the accuracy of eye tracking.
[0011] In a possible implementation of the first aspect, the controlling of turning on or off at least some of the multiple light sources according to the projection parameters of each projection includes: determining invalid projections in the first eye image, and turning off the light sources corresponding to the invalid projections.
[0012] That is to say, when there is an invalid projection in the first eye image, the electronic device can turn off the light source corresponding to the invalid projection, which can not only reduce the power consumption of the light source, but also avoid the invalid projection affecting the accuracy of eye tracking.
[0013] In a possible implementation of the first aspect above, the above-mentioned determination of an invalid projection in the first eye image includes: determining the first projection as an invalid projection when at least one of the following conditions of the first projection is met: in the first eye image, the distance between the first projection and the center of the user's eye is greater than the first distance; in the first eye image, the area of a connected domain formed by the first projection and the second projection corresponding to at least one second light source is greater than the first area, or the distance between the connected domain and the center of the user's eye is greater than the second distance; the area of the first projection is smaller than the second area; the brightness of the first projection is smaller than the first brightness; the similarity between the shape of the first projection and the first shape is less than the first similarity.
[0014] In a possible implementation of the first aspect, the method of adjusting the light emitting parameters of at least some of the multiple light sources according to the movement state of the user's eyes includes: when the user's eye movement is gaze, adjusting the number of turned-on light sources among the multiple light sources to a first number; when the user's eye movement is eye saccade or following movement, adjusting the number of turned-on light sources among the multiple light sources to a second number, wherein the first number is smaller than the second number.
[0015] That is to say, when the user's eye movement is gaze, the electronic device can reduce the number of turned-on light sources, thereby reducing the power consumption of the light sources and improving the battery life of the electronic device.
[0016] In a possible implementation of the first aspect above, the method also includes: when the distance between a first position of the user's pupil center in the first eye image and a second position of the user's pupil center in the third eye image is less than a third distance, adjusting the number of turned-on light sources among the multiple light sources to a third number; when the distance between the first position and the second position is greater than or equal to the third distance, adjusting the number of turned-on light sources among the multiple light sources to a fourth number; wherein the third number is less than the fourth number, and the third eye image is an eye image captured before the first eye image.
[0017] That is to say, the electronic device can adjust the number of turned-on light sources according to the changes in the user's pupil center (the changes reflect the movement state of the user's eyes), so that when the user's eyes move slowly, the number of turned-on light sources can be reduced, thereby reducing the power consumption of the light sources and improving the battery life of the electronic device.
[0018] In a possible implementation of the first aspect, the second eye image includes multiple frames of eye images, and the method further includes: in the process of acquiring multiple frames of eye images, controlling multiple light sources to flash, and determining a model of the user's eye based on the second eye image.
[0019] That is to say, when the second eye image includes multiple frames of eye images for determining a model of the user's eyes, the electronic device can control the flashing of multiple light sources during the process of capturing the multiple frames of eye images to capture eye images with different contrasts, which is beneficial to improving the accuracy of the determined model of the user's eyes, thereby improving the accuracy of eye tracking.
[0020] In a possible implementation of the first aspect, a flashing frequency of the multiple light sources is less than a capture frequency of multiple frames of eye images.
[0021] In a second aspect, the present application provides a readable storage medium, which includes instructions. When the instructions are executed by an electronic device, the electronic device implements any eye tracking method provided by the first aspect and various possible implementations of the first aspect.
[0022] In a third aspect, the present application provides an electronic device, comprising: multiple light sources; a memory for storing instructions; and a processor for executing instructions so that the electronic device controls the multiple light sources to implement any one of the eye tracking methods provided by the first aspect and various possible implementations of the first aspect.
[0023] In a fourth aspect, the present application provides a program product, which, when running on an electronic device, enables the electronic device to implement any one of the eye tracking methods provided by the first aspect and various possible implementations of the first aspect.
[0024] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be referred to the description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 According to some embodiments of the present application, a schematic diagram of an eye tracking scenario is shown;
[0026] Figure 2A According to some embodiments of the present application, a schematic diagram of an eye image is shown;
[0027] Figure 2B According to some embodiments of the present application, another schematic diagram of an eye image is shown;
[0028] Figure 2C According to some embodiments of the present application, a schematic diagram of another eye image is shown;
[0029] Figure 3 According to some embodiments of the present application, a schematic diagram of a process of adjusting light source lighting parameters by user eye movement is shown;
[0030] Figure 4A According to some embodiments of the present application, a schematic diagram of a current frame eye image and a next frame eye image is shown;
[0031] Figure 4B According to some embodiments of the present application, another schematic diagram of a current frame eye image and a next frame eye image is shown;
[0032] Figure 4C According to some embodiments of the present application, a schematic diagram of another current frame eye image and a next frame eye image is shown;
[0033] Figure 4D According to some embodiments of the present application, another schematic diagram of a current frame eye image and a next frame eye image is shown;
[0034] Figure 5 According to some embodiments of the present application, a structural schematic diagram of a VR glasses 10 is shown;
[0035] Fig. 6A According to some embodiments of the present application, a schematic diagram of the distribution of light sources in a light source group 11L / light source group 11R is shown;
[0036] Figure 6B According to some embodiments of the present application, another schematic diagram of the distribution of light sources in the light source group 11L / light source group 11R is shown;
[0037] Figure 7 According to some embodiments of the present application, a flow chart of an eye tracking method is shown;
[0038] Fig. 8A According to some embodiments of the present application, a method for Figure 2B Schematic diagram of the light source switch state in the situation shown;
[0039] Figure 8B According to some embodiments of the present application, a method for Figure 2C Schematic diagram of the light source switch state in the situation shown;
[0040] Fig. 9According to some embodiments of the present application, a schematic diagram of a process of adjusting light source lighting parameters according to the user's pupil position is shown;
[0041] Fig. 10A According to some embodiments of the present application, a flowchart of another eye tracking method is shown;
[0042] Fig. 10B According to some embodiments of the present application, a schematic diagram of acquiring a calibration eye image is shown;
[0043] Fig.11A According to some embodiments of the present application, a schematic diagram of a process of an eye tracking method is shown;
[0044] Fig. 11B According to some embodiments of the present application, a control timing diagram of an eye tracking method is shown. DETAILED DESCRIPTION
[0045] Illustrative embodiments of the present application include, but are not limited to, an eye tracking method, a readable storage medium, a program product, and an electronic device.
[0046] The technical solution of the present application is described below in conjunction with the accompanying drawings.
[0047] It can be understood that eye tracking, also known as eye tracking, is to detect the movement state of the user's eyes relative to the user's head or electronic device and / or the gaze point of the user's eyes, so that the electronic device can adjust the display parameters according to the tracking results, so that the picture displayed by the electronic device matches the movement state or gaze point of the user's eyes.
[0048] It can be understood that the technical solution of the present application is applicable to any electronic device that performs eye tracking by collecting the projections of multiple light sources in the user's eyes, including but not limited to HMD, head-up display (HUD), etc., such as headsets, helmets, goggles or glasses VR / AR / MR devices. For ease of description, the following takes the electronic device as VR glasses as an example to introduce the technical solution of the present application.
[0049] Figure 1 According to some embodiments of the present application, a schematic diagram of an eye tracking scene is shown.
[0050] like Figure 1As shown, the VR glasses 10 include a light source group 11L and a light source group 11R. The light source group 11L includes a plurality of light sources for emitting light to the user's left eye when the user wears the VR glasses 10, so as to project the light source of the light source group 11L into the user's left eye. The light source group 11R includes a plurality of light sources for emitting light to the user's right eye when the user wears the VR glasses 10, so as to project the light source of the light source group 11R into the user's right eye. The VR glasses 10 also include a camera for capturing eye images of the user, such as a camera 13L (not shown) for capturing eye images of the user's left eye and a camera 13R for capturing eye images of the user's right eye, as well as a display screen 12L (not shown) and a display screen 12R for displaying display images.
[0051] It can be understood that the eye tracking device for the user's left eye (including at least the light source group 11L and the camera 13L) and the eye tracking device for the user's left eye (including at least the light source group 11R and the camera 13R) in the VR glasses 10 can adopt a symmetrical structure. The eye tracking device for the user's right eye is introduced below for convenience.
[0052] refer to Figure 1 For ease of description, when the VR glasses 10 are worn by a user, the horizontal direction is defined as the X direction, the vertical direction is defined as the Y direction, and the direction perpendicular to the XY plane is defined as the Z direction. The display screen 12R, the camera 13R, and the light source group 11R are sequentially arranged along the Z direction.
[0053] When the user wears the VR glasses 10 to watch the image in the display screen 12R, the light sources in the light source group 11R emit light to the user's right eye to project at least part of the light sources in the light source group 11R into the user's right eye. After receiving the light sent by the light source group 11R, the user's eye can reflect the received light to the camera 13R, so that the camera 13R captures the eye image P1 of the user's right eye, and the eye image P1 includes the user's eye image and the projection R1 of at least part of the light sources in the light source group 11R in the user's eye. After acquiring the eye image P1, the VR glasses 10 can determine the movement state of the user's eye relative to the user's head or the VR glasses 10 based on the positional relationship between the projection R1 and the user's eye, that is, perform eye tracking on the user's eye.
[0054] Understandably, Figure 1 The arrangement of the display screen 12R, the camera 13R, and the light source group 11R shown is only an example. In other implementations, the display screen 12R, the camera 13R, and the light source group 11R may also be arranged in other forms, which are not limited here.
[0055] However, when the user uses the VR glasses 10, the VR glasses 10 usually keep all the light sources in the light source group 11L and the light source group 11R turned on, which increases the power consumption of the light source group 11L and the light source group 11R and reduces the battery life of the VR glasses 10.
[0056] It can be understood that in some scenarios, changes in the image parameters (such as contrast) of the eye image within the allowable range of the eye tracking algorithm will not affect the accuracy of eye tracking. When other factors are the same, the greater the brightness of the light source, the greater the contrast of the image captured by the camera, and the higher the power consumption of the light source. Therefore, the VR glasses 10 can reduce the brightness of the light source when the contrast of the captured eye image is high, so as to reduce the power consumption of the light source group 11L and the light source group 11R.
[0057] It is understandable that in some scenarios, due to the different shapes and sizes of eyes of different users, the projection of some light sources in the eyes of users cannot improve the accuracy of eye tracking or reduce the accuracy of eye tracking (hereinafter, such projection is referred to as invalid projection). Therefore, the VR glasses 10 can turn off the light sources corresponding to the invalid projections to reduce the power consumption of the light source group 11L and the light source group 11R.
[0058] It can be understood that in some scenarios, the number of reference objects (projections of light sources) required by the eye tracking algorithm to achieve the same tracking accuracy is different depending on the movement state of the user's eyes (such as running speed, eye movement pattern, etc.). For example, the greater the movement speed of the user's eyes, the more reference objects are required. Therefore, the VR glasses 10 can turn on different numbers of light sources according to the movement state of the user's eyes, so as to reduce the number of light sources turned on when the dynamic speed of the user's eyes is relatively low, thereby reducing the power consumption of the light source group 11L and the light source group 11R.
[0059] In view of this, an embodiment of the present application provides an eye tracking method. During the eye tracking process, after the VR glasses 10 capture a frame of eye image (hereinafter referred to as the current frame eye image, as the first eye image), the VR glasses 10 can dynamically adjust the light-emitting parameters of at least some light sources in the light source group 11L and the light source group 11R when the VR glasses 10 capture the next frame or multiple frames of eye images (as the second eye image) according to at least one of the image parameters of the current frame eye image, the movement state of the user determined based on the current frame eye image, and the projection parameters of the light source projection in the current frame eye image (such as projection distribution, projection area, projection brightness, projection shape, etc.), such as reducing the brightness of the light source, turning off some light sources, etc.
[0060] In this way, since the higher the brightness of the light source, the higher the power consumed, in some scenarios, the power consumption of the light source group 11L and the light source group 11R can be reduced, which is beneficial to improving the battery life of the VR glasses 10.
[0061] In some embodiments, the VR glasses 10 can reduce the brightness of at least some light sources in light source group 11L and light source group 11R when the contrast of the current frame eye image is greater than the first contrast, so as to reduce the contrast of the next frame eye image collected, thereby reducing the power consumption of light source group 11L and light source group 11R.
[0062] refer to Figure 2A , assuming that the contrast of the current frame eye image P2A captured by the camera 13R is greater than the first contrast, the VR glasses 10 can reduce the brightness of at least part of the light sources in the light source group 11R when the camera 13R captures the next frame eye image. Figure 2A As shown, the contrast of the next frame of eye image P2B collected by the camera 13R is lower than that of the current frame of eye image P2A. In this way, the power consumption of the light source group 11R can be reduced, and the endurance of the VR glasses 10 can be improved.
[0063] It is understandable that when the contrast of the eye image is too high, the accuracy of the light source projection determined by the VR glasses 10 will be reduced, thereby affecting the accuracy of eye tracking. Therefore, when the contrast of the eye image is greater than the first contrast, reducing the brightness of the light source is conducive to improving the accuracy of eye tracking.
[0064] It is understood that in some embodiments, when the contrast of the eye image is too low, the accuracy of the light source projection determined by the VR glasses 10 will be reduced, thereby affecting the accuracy of eye tracking. Therefore, when the contrast of the eye image is less than the second contrast, the VR glasses 10 can also increase the brightness of the light source or turn on more light sources to improve the accuracy of eye tracking.
[0065] In some implementations, if there is an invalid projection in the current frame of the eye image, the VR glasses 10 can turn off the light source corresponding to the invalid projection to improve the endurance of the VR glasses 10. In other words, the VR glasses 10 can first determine the invalid projection in the current frame of the eye image, and then turn off the light source corresponding to the invalid projection when collecting the next frame or multiple frames of eye images.
[0066] It can be understood that in some embodiments, if a projection in the eye image meets at least one of the following closing conditions, the VR glasses 10 may determine the projection as an invalid projection:
[0067] The distance between the projection and the center of the user's eye in the eye image is greater than the first distance.
[0068] The area of the connected domain formed by the projection and at least one other projection in the eye image is greater than the first area or the distance between the center of the connected domain and the center of the user's eye is greater than the second distance. It can be understood that the connected domain is a collection of multiple pixels with the same or similar values in the image.
[0069] The area of the projection in the eye image is smaller than the second area.
[0070] The brightness of the projection in the eye image is less than the second brightness.
[0071] The similarity between the shape in the eye image and the first shape is less than the first similarity. It can be understood that the first shape is a shape of an effective projection, such as a circle, an ellipse, a polygon, etc., and the first shape can be obtained through experience or experiment. The embodiment of the present application does not limit the specific shape of the first shape.
[0072] For example, refer to Figure 2B , the distance between the projection NR1 and the projection NR2 in the current frame eye image P3A collected by the camera 13R and the center of the human eye is greater than the first distance, and the VR glasses 10 can turn off the light sources corresponding to the projection NR1 and the projection NR2. Figure 2B As shown, compared with the current frame eye image P3A, the eye image P3B captured by the camera 13R after capturing the current frame eye image P3A no longer includes invalid projections. That is, when the camera 13R captures the eye image P3B, the light sources corresponding to the projections NR1 and NR2 are turned off, reducing the power consumption of the light source group 11R.
[0073] For example, refer to Figure 2C , the current frame eye image P4A collected by the camera 13R has multiple projections forming a connected domain CD1. If the area of the connected domain CD1 is larger than the first area, the VR glasses 10 may determine the multiple projections as invalid projections and turn off the light source corresponding to the connected domain CD1. Figure 2C As shown, compared with the current frame eye image P4A, the eye image P4B captured by the camera 13R after capturing the current frame eye image P4A no longer includes invalid projections. In other words, when the camera 13R captures the eye image P4B, the light source corresponding to the projection in the connected domain CD1 is turned off, reducing the power consumption of the light source group 11R.
[0074] It is understandable that in other embodiments, the VR glasses 10 may also determine invalid projections by other means, such as by using the judgment logic of the eye tracking algorithm adopted by the VR glasses 10. The method for determining invalid projections is not limited herein.
[0075] It can be understood that in some implementations, the movement state of the user's eyes can be represented by eye movement patterns, where eye movement patterns can include fixation, saccade, and smooth pursuit. In this way, the VR glasses 10 can determine the user's eye movement pattern based on the current frame eye image, and adjust the brightness of at least some of the light sources in the light source group 11L and the light source group 11R according to the determined eye movement pattern. For example, assuming that the light source group 11L and the light source group 11R each include N (N>1) light sources, when the user's eye movement pattern is fixation, each light source group can turn on M light sources, and when the user's eye movement pattern is saccade or pursuit, each light source group can turn on P light sources, where M <P≤N。
[0076] In some implementations, the VR glasses 10 can implement the determination of eye movement patterns through an eye movement classification module. Figure 3 , the VR glasses 10 can input the current frame eye image into the eye movement classification module to obtain the user's eye movement pattern and / or pupil position, and send the user's eye movement pattern and / or pupil position to the light source control module. The light source control module can determine the number of light sources to be turned on when collecting the next frame eye image based on the user's eye movement pattern and / or pupil position, and control the switching of each light source when collecting the next frame eye image through the light source driver.
[0077] For example, when the user's eye movement is gaze, each light source group may turn on M light sources; and when the user's eye movement is saccade or follow-up movement, each light source group may turn on P light sources.
[0078] It can be understood that the movement state of the user's eyes can also be represented by the change of the user's pupil. Therefore, in some embodiments, the VR glasses 10 can adjust the luminous parameters of each light source when collecting the next frame of eye image based on the distance between the position of the user's pupil in the current frame of eye image and the position of the user's pupil in the previous frame of eye image. For example, when the distance is less than the third distance, each light source group can turn on M light sources; when the distance is greater than or equal to the third distance, each light source group can turn on P light sources.
[0079] It can be understood that the eye movement classification module may include a speed detection module, a discreteness detection module and a comprehensive module. The speed detection module is used to determine the speed of the user's eye movement; the discreteness detection module is used to determine the discreteness of the projection in the eye image of the current frame; and the comprehensive module is used to determine the eye movement mode or pupil position of the user's eye according to the speed of the user's eye movement and the discreteness of the projection.
[0080] In some implementations, when determining the model of the human eye, the VR glasses 10 can also control the flashing of the light sources in the light source group 11L and the light source group 11R to improve the effectiveness of the light source projection in the collected eye image, so as to improve the accuracy of the determined human eye model.
[0081] It can be understood that in other embodiments, the VR glasses 10 can also combine the above-mentioned methods to adjust the light-emitting parameters of the light sources in the light source group 11L and the light source group 11R.
[0082] For example, the VR glasses 10 can adjust the brightness of the light sources in the light source group 11L and the light source group 11R according to the image parameters of the current frame eye image, and turn off the light sources corresponding to the invalid projections according to the invalid projections in the current frame eye image. Figure 4A , the distances between the projection NR3 and the projection NR4 in the current frame eye image P5A collected by the camera 13R and the center of the human eye are greater than the first distance, and the contrast of the current frame eye image P5A is greater than the first contrast. When the camera 13R collects the next frame eye image P5B, the light sources corresponding to the projection NR3 and the projection NR4 can be turned off and the brightness of the turned-on light sources can be reduced.
[0083] For another example, the VR glasses 10 can adjust the brightness of the light sources in the light source group 11L and the light source group 11R according to the image parameters of the eye image of the current frame, and turn on or off the corresponding number of light sources according to the movement state of the user's eyes. Figure 4B , the contrast of the current frame eye image P6A collected by the camera 13R is greater than the first contrast, and it is determined that the user's eye movement mode is gaze. When the camera 13R collects the next frame eye image P6B, the number of turned-on light sources can be reduced to the number corresponding to gaze (for example, 6), and the brightness of the turned-on light sources can be reduced.
[0084] For another example, the VR glasses 10 can not only turn off the light sources corresponding to the invalid projections in the eye image of the current frame, but also turn on or off the corresponding number of light sources according to the movement state of the user's eyes. Figure 4C , the distances between projection NR5 and projection NR6 in the current frame eye image P7A collected by the camera 13R and the center of the human eye are greater than the first distance, and it is determined that the user's eye movement mode is gaze. When the camera 13R collects the next frame eye image P7B, the light sources corresponding to projection NR5 and projection NR6 can be turned off, and the number of turned-on light sources can be reduced to the number corresponding to gaze (for example, 6) and the brightness of the turned-on light sources can be reduced.
[0085] For another example, the VR glasses 10 can adjust the brightness of the light sources in the light source group 11L and the light source group 11R according to the image parameters of the current frame eye image, turn off the light sources corresponding to the invalid projections according to the invalid projections in the current frame eye image, and turn on or off the corresponding number of light sources according to the movement state of the user's eyes. Figure 4D , the distances between projection NR7 and projection NR8 in the current frame eye image P8A collected by the camera 13R and the center of the human eye are greater than the first distance, the contrast of the current frame eye image P8A is greater than the first contrast, and it is determined that the user's eye movement mode is gaze. When the camera 13R collects the next frame eye image P8B, the light sources corresponding to projection NR7 and projection NR8 can be turned off, the number of turned-on light sources can be reduced to the number corresponding to gaze (e.g., 6), and the brightness of the turned-on light sources can be reduced.
[0086] It can be understood that in other embodiments, the VR glasses 10 can also dynamically adjust the light-emitting parameters of some light sources in the light source group 11L and the light source group 11R when capturing the next frame or multiple frames of eye images according to the current frame eye image in other ways, which is not limited here.
[0087] The technical solution of the present application is introduced below in conjunction with the structure of the VR glasses 10.
[0088] First, the structure of the VR glasses 10 is introduced.
[0089] Figure 5 According to some embodiments of the present application, a structural block diagram of VR glasses 10 is shown.
[0090] like Figure 5 As shown, the VR glasses 10 may include a processor 110, a memory 120, a sensor module 130, an audio module 140, a button 150, an input and output interface 160, a communication module 170, a power supply module 180, an eye tracking module 11, a display module 12, and the like.
[0091] The processor 110 is generally used to control the overall operation of the VR glasses 10, and may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a video processing unit (video processing unit, VPU) controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units may be independent devices, or may be integrated in one or more processors.
[0092] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0093] In some embodiments, processor 110 may be used to issue instructions for adjusting the lighting parameters of light sources in light source group 11L and light source group 11R when camera 13L and camera 13R capture the next frame of eye image based on the current frame of eye image captured by camera 13L and camera 13R.
[0094] In some embodiments, the processor 10 can also be used to run the instructions of the aforementioned eye movement classification module and light source control module to determine the user's eye movement pattern and pupil position based on the current frame eye image, and adjust the light-emitting parameters of the light sources in light source group 11L and light source group 11R when capturing the next frame eye image according to the determined eye movement pattern or pupil position.
[0095] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input / output (GPIO) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.
[0096] For example, in some embodiments, the processor 110 can be connected to the light source driver 14 via an I2C interface or a GPIO interface, and send instructions to adjust the light emission parameters of the light sources in the light source group 11L and the light source group 11R to the light source driver via the I2C interface or the GPIO interface.
[0097] The memory 120 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the VR glasses 10 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as an image display function, a video playback function), etc. The storage data area may store data created during the use of the VR glasses 10 (such as eye images captured during eye tracking of the VR glasses 10), etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0098] The sensor module 130 may include sensors for detecting the working state and use state of the VR glasses 10, including but not limited to proximity light sensors / contact sensors for detecting whether the user is wearing the VR glasses 10, inertial sensors for detecting the movement state of the user's head, and the like.
[0099] The audio module 140 may include a speaker, a microphone, etc., for implementing audio functions.
[0100] The button 150 may be one or more. The button 150 may be in the form of a button, a switch, a dial, and a touch or near-touch sensing device (such as a touch sensor). The VR glasses 10 may trigger corresponding functions according to the user's operation on one or more of the buttons 150, such as adjusting the volume, controlling the VR glasses 10 to enter a calibration mode, etc.
[0101] The input / output interface 160 may connect other devices to the VR glasses 10 through suitable components. The components may include, for example, audio / video jacks, data connectors, etc. For example, in some embodiments, the eye tracking module 11 may not be part of the VR glasses, but an external device connected to the VR glasses 10 through the input / output interface 160.
[0102] The communication module 170 may include a wireless communication module. The wireless communication function may be implemented through an antenna (not shown), a modem processor (not shown), and a baseband processor (not shown). The antenna is used to transmit and receive electromagnetic wave signals. The VR glasses 10 may include multiple antennas, each of which may be used to cover a single or multiple communication frequency bands. Different antennas may also be reused to improve the utilization of the antenna. For example, antenna 1 may be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna may be used in conjunction with a tuning switch.
[0103] The wireless communication module can provide wireless communication solutions for the VR glasses 10, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module can be one or more devices that integrate at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna.
[0104] The power supply module 180 may include a battery for supplying power to various modules of the VR glasses 10 .
[0105] The eye tracking module 11 may include a light source group 11L, a light source group 11R, a camera 13L, a camera 13R, a light source driver 14 and the like.
[0106] The light source group 11L is used to emit light to the user's left eye, so that the camera 13L can capture the eye image of the user's left eye and the projection of the light sources in the light source group 11L in the user's eye. The light source group 11R is used to emit light to the user's right eye, so that the camera 13R can capture the eye image of the user's right eye and the projection of the light sources in the light source group 11R in the user's eye.
[0107] It can be understood that the light source group 11L / the light source group 11R may include multiple light sources, and the number of light sources in each light source group may be, for example, 6, 7, 8, 10, etc., or other numbers, which are not limited here.
[0108] It can be understood that the light emission wavelength of the light source can be in the infrared band, in the visible light band, or in other bands, which is not limited here.
[0109] It can be understood that the light sources in the light source group 11L or the light source group 11R can be light-emitting diodes (LED), organic light-emitting diodes (OLED), active-matrix organic light-emitting diodes or active-matrix organic light-emitting diodes (AMOLED), flexible light-emitting diodes (FLED), Mini-LED, Micro-LED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In other embodiments, the light sources of the light source group 11L and the light source group 11R can also be other forms of light sources, which are not limited here.
[0110] It can be understood that the light sources in the light source group 11L or the light source group 11R can be distributed on the edge of a circle, an ellipse, a polygon, or other irregular shapes, which is not limited here.
[0111] For example, in some embodiments, reference Fig. 6A , the light sources in the light source group 11L / the light source group 11R may include 8 light sources respectively, and the 8 light sources are arranged on the same circumference. Exemplarily, the 8 light sources in each light source group may be arranged at positions of 5°, 30°, 150°, 175°, 200°, 225°, 315°, and 340° on the circumference with the positive semi-axis of the X-axis being 0°.
[0112] For example, in some embodiments, reference Figure 6B , the light sources in the light source group 11L / the light source group 11R may include 10 light sources respectively, and the 10 light sources are arranged on the same circumference. For example, with the positive half axis of the X-axis on the circumference as 0°, the 10 light sources in each light source group may be arranged at 9°, 36°, 60°, 120°, 144°, 171°, 198°, 225°, 315°, and 342° on the circumference.
[0113] The camera 13L / camera 13R may be a camera that matches the wavelength of the light source in the light source group 11L / light source group 11R, and is used to collect the user's eye image. For example, when the light source is a visible light source, the camera 13L / camera 13R may be a camera in the visible light band; when the light source is an infrared light source, the camera 13L / camera 13R may be a visible infrared camera.
[0114] It can be understood that the camera 13L / camera 13R can be set at any position where the user's eye image can be captured.
[0115] It can be understood that in some embodiments, the VR glasses 10 may include only one camera that can simultaneously capture the left and right eye images of the user, or may use a greater number (for example, greater than 2) of cameras to capture the user's eye images, which is not limited here.
[0116] The light source driver 14 is used to adjust the light-emitting parameters of each light source according to the instructions or signals of the processor 110, for example, according to the instructions or signals of the processor 110, adjust the voltage or current applied to each light source to adjust the brightness, flickering, turn on or off each light source, etc.
[0117] Understandably, Figure 5 The structure of the VR glasses 10 shown is only an example. In other embodiments, the VR glasses 10 may include more or fewer modules, and some modules may be merged or split, which is not limited here.
[0118] Combine the following Figure 5 The structure of the VR glasses 10 shown introduces the technical solution of the present application.
[0119] Figure 7 According to some embodiments of the present application, a flow chart of an eye tracking method is shown. The method may be performed by the VR glasses 10, such as the processor 110 or other computing unit in the VR glasses. Figure 7 As shown, the method may include the following steps:
[0120] S701: Acquire the eye image of the current frame.
[0121] During the eye tracking process, the VR glasses 10 collect the eye image of the current frame.
[0122] It can be understood that in some embodiments, when the VR glasses 10 capture the current frame of eye image, the light emitting parameters of the light sources in the light source group 11L / light source group 11R can be determined by the previous frame of eye image.
[0123] It can be understood that in some embodiments, if the current frame of eye image is the first frame of eye image collected during the use of VR glasses 10, when VR glasses 10 collect the current frame of eye image, the light-emitting parameters of the light sources in light source group 11L / light source group 11R can be the light-emitting parameters preset by VR glasses, for example, all light sources emit light with the same preset brightness.
[0124] It can be understood that in some embodiments, when the VR glasses 10 are in use, the camera 13L and the camera 13R can collect the user's eye images at the same sampling frequency, for example, collecting eye images at a sampling frequency of 30 to 160 Hz. For example, assuming that the sampling frequency of the camera 13L and the camera 13R is 45 Hz, the camera 13L and the camera 13R will collect a frame of eye images every 1000 ms / 45=22.2 ms. The current frame of eye image can be any frame of eye image collected by the camera 13L or the camera 13R.
[0125] It can be understood that in some embodiments, after capturing the current frame eye image, the VR glasses 10 can also perform eye tracking based on the current frame eye image to determine the movement state of the user's eyes or the user's gaze point.
[0126] S702: Determine the light emitting parameters of each light source in the light source group 11L / the light source group 11R when acquiring the next frame of eye image according to the current frame of eye image.
[0127] After capturing the current frame of eye image, the VR glasses 10 can determine the luminous parameters of each light source in the light source group 11L / light source group 11R when capturing the next frame of eye image based on at least one of the image parameters of the current frame of eye image, the movement state of the user's eyes determined based on the current frame of eye image, and the projection parameters of the light source projection in the current frame of eye image (such as projection distribution, projection area, projection brightness, projection shape, etc.).
[0128] In some implementations, after acquiring the current frame of eye image, the VR glasses 10 may first determine the image parameters of the current frame of eye image, such as contrast. Then, the VR glasses 10 may compare the determined image parameters with the preset image parameters, and determine the light emitting parameters of each light source in the light source group 11L / light source group 11R when acquiring the next frame of eye image based on the comparison result.
[0129] For example, when the contrast of the eye image in the current frame is greater than the first contrast, the brightness of the currently turned-on light sources in light source group 11L and light source group 11R is reduced or some of the currently turned-on light sources are turned off to reduce the contrast of the next frame of eye image collected, thereby reducing the power consumption of the light source group.
[0130] For another example, when the contrast of the eye image of the current frame is less than the second contrast, the brightness of the currently turned-on light sources in light source group 11L and light source group 11R is increased or the number of turned-on light sources is increased to increase the contrast of the next frame of eye image collected, thereby improving the accuracy of eye tracking.
[0131] In some implementations, after the VR glasses 10 acquire the current frame of eye image, they can first determine whether there is an invalid projection in the current frame of eye image. If it is determined that there is an invalid projection in the current frame of eye image, the light source corresponding to the invalid projection is set to off. In this way, the power consumption of the VR glasses 10 can be reduced and the accuracy of eye tracking can be improved.
[0132] It can be understood that in some embodiments, if a projection in the eye image meets at least one of the following closing conditions, the VR glasses 10 may determine the projection as an invalid projection:
[0133] The distance between the projection and the center of the user's eye in the eye image is greater than the first distance.
[0134] The area of the connected domain formed by the projection and at least one other projection in the eye image is greater than the first area or the distance between the center of the connected domain and the center of the human eye is greater than the second distance. It can be understood that the connected domain is a collection of multiple pixels with the same or similar values in the image.
[0135] The area of the projection in the eye image is smaller than the second area.
[0136] The brightness of the projection in the eye image is less than the second brightness.
[0137] The similarity between the shape in the eye image and the first shape is less than the first similarity. It can be understood that the first shape is a shape of an effective projection, such as a circle, an ellipse, a polygon, etc., and the first shape can be obtained through experience or experiment. The embodiment of the present application does not limit the specific shape of the first shape.
[0138] For example, the light sources in the light source group 11R adopt Figure 6B The setup method shown.
[0139] for Figure 2B The situation shown, refer to Fig. 8A, all the light sources in the light source group 11R are turned on when the current frame eye image P3A is collected. Based on the presence of invalid projections NR1 and NR2 in the current frame eye image P3A, and the invalid projections NR1 and NR2 correspond to the light source at the 225° position and the light source at the 315° position, respectively, the VR glasses 10 can determine that when collecting the next frame eye image P3B, the light source at the 225° position and the light source at the 315° position are turned off.
[0140] for Figure 2C The situation shown, refer to Figure 8B , when collecting the current frame eye image P4A, all light sources in the light source group 11R except the light source at the 225° position are turned on. Based on the existence of the connected domain CD1 in the current frame eye image P4A, the area of the connected domain CD1 is greater than the first area, and the projections in the connected domain CD1 correspond to the light source at the 9° position, the light source at the 315° position, and the light source at the 342° position, respectively, the VR glasses 10 can determine that when collecting the next frame eye image P4B, the light source at the 9° position, the light source at the 225° position, the light source at the 315° position, and the light source at the 342° position are all turned off.
[0141] In some implementations, after the VR glasses 10 acquire the eye image of the current frame, they can first determine the eye movement pattern of the user, and adjust the light sources turned on in the light source group 11L / light source group 11R to the corresponding number according to the eye movement pattern of the user. For example, assuming that the light source group 11L and the light source group 11R each include N (N>1) light sources, when the user's eye movement pattern is gaze, each light source group can turn on M light sources, and when the user's eye movement pattern is eye saccade or tracking movement, each light source group can turn on P light sources, where M <P≤N。
[0142] In some implementations, after the VR glasses 10 capture the current frame eye image, they can determine the number of light sources turned on in light source group 11L / light source group 11R based on the position of the user's pupil in the current frame eye image and the distance between the user's pupil in the previous frame eye image.
[0143] For example, refer to Fig. 9 , when the current eye image is the K-th eye image captured by the camera 13R, the VR glasses 10 can determine that the pupil center position of the user's right eye in the K-th eye image is (x1, y1). Then compare the pupil center position (x1, y1) with the pupil center position of the user's right eye in the K-1-th eye image (x0, y0), and determine the change in the pupil center position in the X direction Δx = |x1-x0| and the change in the Y direction Δy = |y1-y0|. Then, compare Δx and Δy with the preset change amount D X , D Y For comparison, when Δx≤D X Or Δy≤DY When the K+1th frame of eye image is collected, it is determined that N light sources are turned on.
[0144] Continue to refer Fig. 9 , when the eye image before the frame is the K+1 frame eye image captured by the camera 13R, the VR glasses 10 can determine that the pupil center position of the user's right eye in the K+1 frame eye image is (x2, y2). Then compare the pupil center position (x1, y1) with the pupil center position of the user's right eye in the K frame eye image (x1, y1), and determine the change in the pupil center position in the X direction Δx = |x2-x1| and the change in the Y direction Δy = |y2-y1|. Then, compare Δx and Δy with the preset change amount D X , D Y For comparison, when Δx>D X And Δy>D Y , it is determined that P light sources are turned on when acquiring the K+2th frame of eye image.
[0145] It is understood that in other embodiments, the VR glasses 10 may also be in the range of Δx≤D X And Δy≤D Y When Δx>D X Or Δy>D Y When the K+2th frame of eye image is collected, it is determined that P light sources are turned on, and no limitation is made here.
[0146] It can be understood that in other embodiments, the VR glasses 10 may also combine multiple methods in the above-mentioned adjustment methods to determine the light-emitting parameters of each light source in the light source group 11L / light source group 11R when capturing the next frame of eye image, or may determine the light-emitting parameters of each light source in the light source group 11L / light source group 11R when capturing the next frame of eye image based on the current frame of eye image through other methods.
[0147] S703: adjusting the light emission parameters of at least part of the light sources in the light source group 11L / the light source group 11R according to the determined light emission parameters and capturing the next frame of eye image.
[0148] After determining the light-emitting parameters of each light source in light source group 11L / light source group 11R when capturing the next frame of eye image, the VR glasses 10 adjust the light-emitting parameters of the light sources in light source group 11L / light source group 11R to the determined light-emitting parameters and capture the next frame of eye image.
[0149] For example, when the VR glasses 10 determine to adjust the brightness of some light sources, they can input a digital or analog signal corresponding to the brightness to the light source driver 14, so that the light source driver 14 can output a corresponding current or voltage to each light source according to the received signal.
[0150] For another example, when the VR glasses determine to turn on or off certain light sources, they can input the corresponding digital or analog level or the on or off instruction to the light source driver 14, so that the light source driver 14 can turn on or off the corresponding light source according to the received level or instruction. Fig. 9 When capturing the K+1th frame of eye image, the processor 110 may output to the light source driver 14 a high level corresponding to turning on N light sources and a high level corresponding to turning off PN light sources.
[0151] For example, for Figure 2A In the situation shown, the VR glasses 10 can reduce the brightness of at least part of the light sources in the light source group 11R when the camera 13R captures the next frame of eye image and captures the next frame of eye image P2B.
[0152] For example, for Figure 2B In the situation shown, the VR glasses 10 can turn off the light sources corresponding to projection NR1 and projection NR2 and collect the next frame of eye image P3B.
[0153] For example, corresponding to Figure 2C In the situation shown, the VR glasses 10 can turn off the light source corresponding to the projection in the connected domain CD1 and collect the next frame of eye image P4B.
[0154] It can be understood that in some embodiments, the VR glasses 10 can send the determined lighting parameters to the light source driver 14 through the I2C bus, etc., to adjust the lighting parameters of at least some light sources in the light source group 11L / light source group 11R.
[0155] S704: Perform eye tracking based on the next frame of eye image collected.
[0156] After collecting the next frame of eye image, the VR glasses 10 perform eye tracking based on the collected eye image. For example, the VR glasses can track the user's eyes through an eye tracking algorithm.
[0157] Through the method provided in the application embodiment, the VR glasses 10 can reduce the brightness of the light source and reduce the number of turned-on light sources in some scenarios, thereby reducing the power consumption of each light source group 11L / light source group 11R and improving the battery life of the VR glasses 10.
[0158] It can be understood that in some embodiments, when the user wears the VR glasses 10, the VR glasses 10 can calibrate the human eye, that is, determine the user's eye model, such as the shape, size, pupil distance, diopter, astigmatism, etc. of the user's eyes. In the process of calibrating the human eye, the VR glasses 10 usually use a calibration algorithm based on multiple eye images. In some embodiments, in the process of collecting eye images for human eye calibration, the VR glasses 10 can also adjust the light parameters of the light sources in the light source group 11L / light source group 11R, such as controlling the light source to flicker, to improve the effectiveness of the collected eye images, thereby improving the accuracy of the determined eye model.
[0159] Specifically, Fig. 10A According to some embodiments of the present application, a flowchart of another eye tracking method is shown. The execution subject of the method may be the VR glasses 10, such as the processor 110 of the VR glasses 10. Fig. 10A As shown, the method comprises the following steps:
[0160] S1001: It is detected that the VR glasses 10 enter the calibration mode.
[0161] For example, when the VR glasses 10 detect that the user has put on the VR glasses 10, it can detect that the VR glasses have entered a calibration mode, thereby triggering the eye tracking method provided in the embodiment of the present application.
[0162] For another example, when the VR glasses 10 detect the user's operation through the button 150, they can enter the calibration mode and trigger the eye tracking method provided in the embodiment of the present application.
[0163] For another example, the VR glasses 10 may enter a calibration mode and trigger the eye tracking method provided in an embodiment of the present application when receiving a calibration instruction sent by other electronic devices, such as a calibration instruction sent by an electronic device connected to the VR glasses 10 through an application corresponding to the VR glasses 10.
[0164] S1002: Acquire the eye image of the current frame.
[0165] After entering the calibration mode, the VR glasses 10 collect the eye image of the current frame.
[0166] It can be understood that in some embodiments, if the current frame of eye image is the first frame of eye image collected during the calibration process of the VR glasses 10, when the VR glasses 10 collect the current frame of eye image, the light-emitting parameters of the light sources in the light source group 11L / light source group 11R can be the light-emitting parameters preset by the VR glasses, for example, all light sources emit light with the same preset brightness.
[0167] S1003: Determine the light emitting parameters of each light source in each light source group 11L / light source group 11R during the calibration process according to the eye image of the current frame.
[0168] After collecting the eye image of the current frame, the VR glasses 10 determines the luminous parameters of each light source in each light source group 11L / light source group 11R during the calibration process according to the eye image of the current frame. The specific determination method can refer to the aforementioned step S702, which will not be described in detail here.
[0169] For example, if there is an invalid projection in the current frame eye image, the VR glasses 10 can turn off the light source corresponding to the invalid projection during the calibration process.
[0170] For example, refer to Fig. 10B ,for Figure 2B As shown in the current frame eye image P3A, the VR glasses 10 can determine that the light source at the 225° position and the light source at the 315° position are turned off during the calibration process based on the presence of invalid projections NR1 and NR2 in the current frame eye image P3A, and the invalid projections NR1 and NR2 correspond to the light source at the 225° position and the light source at the 315° position, respectively.
[0171] S1004: Adjust the light emitting parameters of at least part of the light sources in the light source group 11L / the light source group 11R according to the determined light emitting parameters.
[0172] After determining the light emission parameters of each light source in the light source group 11L / the light source group 11R during the calibration process, the VR glasses 10 adjust the light emission parameters of the light sources in the light source group 11L / the light source group 11R to the determined light emission parameters.
[0173] For example, refer to Fig. 10B , the VR glasses 10 can turn off the light source at the 225° position and the light source at the 315° position.
[0174] S1005: Acquire multiple frames of calibration eye images, and control the turned-on light sources in the light source group 11L / the light source group 11R to flash during the acquisition process.
[0175] The VR glasses 10 collect multiple frames of calibration eye images, and control the turned-on light sources in the light source group 11L / the light source group 11R to flash during the collection process.
[0176] It can be understood that in some embodiments, the VR glasses 10 can control the flickering of the turned-on light sources in the light source group 11L / light source group 11R by periodically adjusting the voltage or current applied to the turned-on light sources through the light source driver 14. For example, in a flickering cycle, the voltage or current applied to the turned-on light sources is controlled to gradually increase first and then gradually decrease, or to gradually decrease first and then gradually increase, so as to achieve the flickering of the turned-on light sources in the light source group 11L / light source group 11R.
[0177] It can be understood that in some embodiments, the flashing frequency of the light sources turned on in light source group 11L / light source group 11R can be lower than the frequency of collecting calibration eye images. In this way, the collected multiple frames of calibration eye images can include eye images of the user under different brightness of the light sources, which improves the effectiveness of the collected eye images and helps to improve the accuracy of human eye calibration. For example, assuming that the sampling frequency of camera 13L and camera 13R is a, the flashing frequency of the light sources turned on in light source group 11L / light source group 11R is b (b <a)。
[0178] For example, reference Fig. 10B By controlling the light source flickering other than the light source at the 225° position and the light source at the 315° position, the VR glasses 10 can collect multiple frames of calibration eye images with different brightness and contrast. In this way, it can be avoided that the calibration accuracy is affected by the brightness or contrast corresponding to the collected calibration eye images not meeting the requirements of the calibration algorithm.
[0179] S1006: Determine the user's eye model based on the collected calibration eye image.
[0180] The VR glasses 10 determine the user's eye model based on the collected calibration eye images.
[0181] Through the method provided in the application embodiment, the VR glasses 10 can reduce the brightness of the light source and the number of turned-on light sources in some scenes, thereby reducing the power consumption of each light source group 11L / light source group 11R and improving the battery life of the VR glasses 10. In addition, since the VR glasses 10 collect multiple frames of calibration eye images and the turned-on light sources in the light source group 11L / light source group 11R flicker, the user's eye images under different brightness of the light source can be collected, which is conducive to improving the accuracy of the determined eye model.
[0182] It can be understood that in some embodiments, camera 13L and camera 13R periodically collect eye images. The following takes the example of camera 13R periodically collecting the eye image of the user's right eye to introduce the control timing of the eye tracking method provided by the embodiment of the present application.
[0183] Specifically, Fig.11A According to some embodiments of the present application, a schematic diagram of a process of an eye tracking method provided by the present application is shown; Fig. 11B According to some embodiments of the present application, a control timing diagram of the eye tracking method provided by the present application is shown.
[0184] like Fig.11A As shown, the process of the eye tracking method includes:
[0185] S11.1: The camera 13R performs exposure while the camera sampling signal is at a high level.
[0186] Exemplarily, during the high level period of each cycle of the camera sampling signal of the camera 13R, for example Fig. 11B During the period from time 0 to time T1 shown, exposure is performed to obtain image data corresponding to the eye image.
[0187] It can be understood that during the period when the camera sampling signal is at a high level, the driving signal corresponding to the light source to be turned on is at a high level. Fig. 11B ,against Figure 2B and Fig. 8A In the illustrated situation, during the period from 0 to T1, the driving signals of all the light sources in the light source group 11R are at a high level, that is, all the light sources are in the on state.
[0188] It is understood that after the exposure is completed, or when the camera sampling signal is converted from a high level to a low level, the camera 13R can convert the driving signals of all light sources from a high level to a low level to turn off all light sources. In this way, during the period when the camera 13R is not exposing, for example Fig. 11B During the period from time T1 to time T shown, all light sources are turned off to save energy.
[0189] S11.2: The camera 13R obtains the eye image of the current frame and transmits it to the processor 110 through the SDK.
[0190] After the camera 13R has completed the exposure, for example Fig. 11B At time T2 shown, the camera 13R may acquire the eye image of the current frame and transmit it to the processor 110 via a software development kit (SDK).
[0191] S11.3: The processor 110 determines the light emission parameters when acquiring the next frame of eye image.
[0192] After receiving the eye image of the current frame, the processor 110, for example Fig. 11B At the time T3 shown, the light parameters of the light sources in the light source groups 11L and 11R, such as light brightness and on / off, can be determined based on the current frame eye image when capturing the next frame eye image. The specific determination method refers to the aforementioned steps S702 and S803, which will not be described in detail here.
[0193] S11.4: The processor 110 configures the light emission parameters of the light source when collecting the next frame of eye image through the I2C bus.
[0194] After the processor 110 determines the light emission parameters of the light sources in the light source group 11L and the light source group 11R when capturing the next frame of eye image, for example, Fig. 11BAt time T4 shown, the determined light emission parameters are configured into the light source driver 14 via the I2C bus. For example, the processor 110 can write the light emission brightness and on / off parameters into the register of the light source driver 14 via the I2C bus.
[0195] It is understandable that in other embodiments, the processor 110 may also configure the light emission parameters of the light source when capturing the next frame of eye image through other buses or interfaces, such as the GPIO of the processor 110, which is not limited here.
[0196] S11.5: The light source driver 14 drives the light source to emit light according to the light emission parameters during the period when the camera sampling signal is at a high level.
[0197] Exemplarily, when the processor 110 detects that the camera sampling signal corresponding to the next frame of eye image is converted from a low level to a high level, for example Fig. 11B At time T, the driving signal of the light source to be turned on determined in step S11.4 is set to a high level. When the light source driver 14 detects the high level light source driving signal, it drives the corresponding light source to emit light according to the light emission parameters determined in step S11.4.
[0198] For example, corresponding to Figure 2B and Fig. 8A In the situation shown, the light source driver 14 can detect that the driving signals of the light sources at the 225° position and the 315° position are converted from low level to high level and turn on the other light sources in the light source group 11R except the light source at the 225° position and the light source at the 315° position according to the driving signals of the light sources at other positions.
[0199] S11.6: Camera 13R triggers an interrupt.
[0200] When the camera 13R is exposed or the camera sampling signal is converted from a high level to a low level, an interrupt can be triggered to convert the driving signal of each light source from a high level to a low level. In this way, after receiving the interrupt, the light source driver 14 can respond to the interrupt and turn off all the light sources to save power.
[0201] The process of adjusting the light emission parameters of the light sources in the light source group 11L based on the current frame eye image of the user's left eye captured by the camera 13L when the camera 13L captures the next frame eye image can refer to the aforementioned steps S11.1 to S11.6 and will not be repeated here.
[0202] Through the method provided by the present application, in some scenes, the brightness of the light source can be reduced and the number of turned-on light sources can be reduced, thereby reducing the power consumption of each light source group 11L / light source group 11R and improving the battery life of the VR glasses 10. In addition, since the VR glasses 10 collect multiple frames of calibration eye images and the turned-on light sources in the light source group 11L / light source group 11R flicker, the user's eye images under different brightness of the light source can be collected, which is conducive to improving the accuracy of the calibration of the human eye model.
[0203] The various embodiments of the mechanisms disclosed in this application may be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application may be implemented as a computer program product or program code executed on a programmable system or device, the programmable system comprising at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. When the computer program product or program code is executed on a programmable system or device, the programmable system or device may implement the eye tracking method provided in the various embodiments of the present application.
[0204] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit, or a microprocessor.
[0205] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.
[0206] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic, or other form of propagation signal. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).
[0207] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.
[0208] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.
[0209] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "includes one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.
Claims
1. An eye tracking method, applied to an electronic device, characterized in that: The electronic device includes a plurality of light sources, and the method includes: Acquire a first eye image of the user, where the first eye image includes a projection of at least one light source among the multiple light sources in the user's eye; adjusting light emission parameters of at least part of the plurality of light sources according to the first eye image; A second eye image is acquired, and eye tracking is performed based on the second eye image.
2. The method according to claim 1, characterized in that: The light-emitting parameters include at least one of the following parameters: light-emitting brightness, on / off.
3. The method according to claim 1, characterized in that The step of adjusting the light emission parameters of at least part of the plurality of light sources according to the first eye image comprises: Adjusting the light emission parameters of at least part of the plurality of light sources by at least one of the following methods: adjusting the light luminance of at least some of the multiple light sources, or turning on / off at least some of the light sources according to image parameters of the first eye image, wherein the image parameters include contrast; Controlling turning on or off at least some of the plurality of light sources according to projection parameters of each of the projections, wherein the projection parameters include at least one of distribution, area, brightness, and shape; Controlling turning on or off of at least some of the multiple light sources according to a movement state of the user's eyes, wherein the movement state is obtained based on the first eye image.
4. The method according to claim 3, characterized in that The adjusting the light luminance of at least part of the plurality of light sources according to the image parameters of the first eye image, or turning on / off at least part of the light sources, comprises: When the contrast of the first eye image is greater than a first contrast, reducing the light luminance of the at least one light source, or turning off part of the at least one light source; When the contrast of the first eye image is smaller than the second contrast, the light brightness of the at least one light source is increased, or the number of turned-on light sources is increased.
5. The method according to claim 3, characterized in that: The controlling the turning on or off of at least part of the plurality of light sources according to the projection parameters of each projection comprises: An invalid projection in the first eye image is determined, and a light source corresponding to the invalid projection is turned off.
6. The method according to claim 5, characterized in that The determining of the invalid projection in the first eye image comprises: When the first projection meets at least one of the following conditions, the first projection is determined as an invalid projection: In the first eye image, the distance between the first projection and the center of the user's eye is greater than a first distance; In the first eye image, an area of a connected domain formed by the first projection and a second projection corresponding to at least one second light source is larger than the first area, or a distance of the connected domain from a center of a user's eye is larger than a second distance; The area of the first projection is smaller than the second area; The brightness of the first projection is less than the first brightness; The similarity between the shape of the first projection and the first shape is less than the first similarity.
7. The method according to claim 3, characterized in that The step of adjusting the light emitting parameters of at least part of the plurality of light sources according to the movement state of the user's eyes comprises: When the eye movement mode of the user is gaze, adjusting the number of turned-on light sources among the plurality of light sources to a first number; When the eye movement of the user is saccade or tracking movement, the number of turned-on light sources among the plurality of light sources is adjusted to a second number, wherein the first number is smaller than the second number.
8. The method according to claim 3, characterized in that The method further comprises: When a distance between a first position of a center of a pupil of the user in the first eye image and a second position of a center of a pupil of the user in the third eye image is less than a third distance, adjusting the number of turned-on light sources among the plurality of light sources to a third number; When the distance between the first position and the second position is greater than or equal to the third distance, adjusting the number of turned-on light sources among the plurality of light sources to a fourth number; The third number is smaller than the fourth number, and the third eye image is an eye image captured before the first eye image.
9. The method according to claim 1, characterized in that: The second eye image includes a plurality of frames of eye images, and the method further includes: During the process of acquiring the multiple frames of eye images, the multiple light sources are controlled to flash, and a model of the user's eyes is determined based on the second eye images.
10. The method according to claim 9, characterized in that The flashing frequency of the multiple light sources is less than the acquisition frequency of the multiple frames of eye images.
11. A readable storage medium, characterized in that: The readable storage medium includes instructions, and when the instructions are executed by the electronic device, the electronic device implements the eye tracking method according to any one of claims 1 to 10.
12. An electronic device, characterized in that: The electronic device comprises: Multiple light sources; A memory for storing instructions; A processor, configured to execute the instructions so that the electronic device controls the multiple light sources to implement the eye tracking method according to any one of claims 1 to 10.
13. A program product, characterized in that When the program product is run on an electronic device, the electronic device implements the eye tracking method according to any one of claims 1 to 10.
Citation Information
Cited By
Eye movement tracking method, readable storage medium, program product and electronic device
EP4692993A1
Eye movement tracking method, readable storage medium, program product and electronic device
WO2024230227A1
