Method and apparatus for controlling eyeball tracking apparatus, electronic device, and storage medium

Abstract

The present application relates to the technical field of virtual reality devices, and discloses a method and apparatus for controlling an eyeball tracking apparatus, an electronic device, and a storage medium. The eyeball tracking apparatus comprises a tracking module and at least one light-emitting member; the at least one light-emitting member is used for emitting light to an eyeball; and the tracking module is used for generating a first image of the eyeball and tracking the eyeball on the basis of a light spot on the eyeball in the first image. The method for controlling the eyeball tracking apparatus comprises: acquiring an on-off state of a light-emitting member; acquiring a light spot state corresponding to the light-emitting member from a first image; and on the basis of the on-off state and the light spot state, controlling the light-emitting member to be turned on or turned off.

Description

Control method, device, electronic device and storage medium for eye tracking device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to a Chinese patent application filed with the Patent Office of China on September 18, 2023, with application number 202311202153.X and entitled “Control method, device, electronic device and storage medium for eye tracking device,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application belongs to the technical field of virtual reality equipment, and specifically relates to a control method, device, electronic device and storage medium of an eye tracking device. Background Art

[0004] In related technology, head-mounted virtual reality devices are equipped with an eye-tracking device. This device tracks the user's eye movements while wearing the device to ensure optimal viewing. The eye-tracking device uses multiple infrared lamps to emit light toward the user's eyes. A tracking module generates an image of the eyeball, which displays the light spot formed by the infrared lamps on the eyeball. The tracking module then determines the direction of eye movement based on the relative position of the light spot and the eyeball.

[0005] During actual use, in order to ensure that more light spots can be obtained, that is, to ensure the accuracy of eye movement direction tracking, the eye tracking device needs to turn on all infrared lights, which causes the virtual reality device to have a large power consumption, causing the virtual reality device to heat up, affecting the user experience of the virtual reality device.

[0006] Summary of the Invention

[0007] The purpose of the embodiments of the present application is to provide a control method, device, electronic device and readable storage medium for an eye tracking device, which can solve the problem of high power consumption during the use of virtual reality equipment.

[0008] In a first aspect, embodiments of the present application provide a method for controlling an eye-tracking device. The eye-tracking device includes a tracking module and at least one light-emitting element. The at least one light-emitting element is configured to emit light toward an eye. The tracking module is configured to generate a first image of the eye and track the eye based on a light spot on the eye in the first image. The control method includes:

[0009] Get the switch status of the light-emitting element;

[0010] Acquire a light spot state corresponding to the light-emitting element in the first image;

[0011] According to the switch state and the light spot state, the light emitting element is controlled to be turned on or off.

[0012] In a second aspect, embodiments of the present application provide a control device for an eye-tracking device. The eye-tracking device includes a tracking module and at least one light-emitting element. The at least one light-emitting element is configured to emit light toward the eye to form a light spot on the eye. The tracking module is configured to generate a first image of the eye and track the eye based on the light spot in the first image. The control device includes:

[0013] an acquisition device for acquiring the on / off state of the light-emitting element; and

[0014] Acquire a light spot state corresponding to the light-emitting element in the first image;

[0015] The control device is used to control the light-emitting element to turn on or off according to the switch state and the light spot state.

[0016] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method of the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, which stores a program or instruction. When the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the method of the first aspect.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a readable storage medium and is executed by at least one processor to implement the method of the first aspect.

[0020] In a seventh aspect, an embodiment of the present application provides a control device for an eye-tracking device, which is used to execute the method of the first aspect.

[0021] In an embodiment of the present application, an eye-tracking device includes at least one light-emitting element that can emit light toward the user's eyeball when the user wears the virtual reality device, thereby forming a light spot on the user's eyeball. Furthermore, the eye-tracking device also includes a tracking module that is configured to generate a first image of the eyeball, thereby tracking the eyeball's movement based on the position of the light spot in the first image.

[0022] Furthermore, after the user puts on the virtual reality device, the on / off state of at least one light-emitting element is first obtained, that is, whether each light-emitting element is on or off. Then, the state of the light spot corresponding to the light-emitting element is obtained in the first image. That is, when the light-emitting element is turned on, the state of the light spot corresponding to the light-emitting element is determined in the first image.

[0023] Furthermore, the light emitting element is controlled to be turned on or off according to the light emitting state of the light emitting element and the state of the light spot corresponding to the light emitting element.

[0024] It is understandable that during actual use, as the user's eyeballs rotate, the tracking module of the eye tracking device will obtain a first image at regular intervals. The state of the light spot in each first image may be different. Therefore, during the eye tracking process, it cannot be guaranteed that all light-emitting components can generate effective light spots in the first image. In addition, it is only necessary to ensure that two light spots can be obtained in the first image to ensure the accuracy of eye movement tracking, and it is not necessary to obtain the light spots of all light-emitting components. Therefore, by controlling the turning on and off of the light-emitting components based on the light state of the light-emitting components and the light spot state in the first image during actual use, it is achieved to achieve dynamic control of the switching of the light-emitting components as the user's eyeballs rotate during use, and then on the basis of ensuring that there are at least two light spots in the first image, that is, on the basis of ensuring the final effect of the eyeball, unnecessary light-emitting components are turned off to achieve the purpose of reducing power consumption in virtual reality.

[0025] During actual use of the virtual reality device, the embodiment of the present application obtains information based on the on / off state of the light-emitting element of the eye tracking device and the corresponding light spot state in the first image, thereby dynamically controlling the turning on or off of the light-emitting element as the user's eye moves. While ensuring the eye tracking effect, unnecessary light-emitting elements are turned off, thereby achieving the purpose of reducing the power consumption of the virtual reality device, reducing the heat generation of the virtual reality device, and improving the user experience of the virtual reality device. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a flow chart showing a method for controlling an eye tracking device according to an embodiment of the present application;

[0027] FIG2 shows one of the schematic diagrams of the first image according to an embodiment of the present application;

[0028] FIG3 shows a second schematic diagram of the first image according to an embodiment of the present application;

[0029] FIG4 is a schematic diagram showing an eyeball region image of the first image in FIG3 ;

[0030] FIG5 is a schematic diagram showing the light spot of the first image in FIG3 ;

[0031] FIG6 shows a third schematic diagram of the first image according to an embodiment of the present application;

[0032] FIG7 is a schematic diagram of a light-emitting element in an eye tracking device according to an embodiment of the present application;

[0033] FIG8 shows a structural block diagram of a control device of an eye tracking device according to an embodiment of the present application;

[0034] FIG9 shows a structural block diagram of an electronic device according to an embodiment of the present application;

[0035] FIG10 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application. Specific embodiments

[0036] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0037] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0038] The control method, device, electronic device, and readable storage medium of the eye tracking device provided in the embodiments of the present application are described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0039] In some embodiments of the present application, a method for controlling an eye-tracking device is provided. The eye-tracking device includes a tracking module and at least one light-emitting element. The at least one light-emitting element is configured to emit light toward an eye. The tracking module is configured to generate a first image of the eye and track the eye based on a light spot on the eye in the first image. FIG1 shows a flow chart of the method for controlling the eye-tracking device according to an embodiment of the present application. As shown in FIG1 , the method for controlling the eye-tracking device includes:

[0040] Step 102, obtaining the on / off state of the light-emitting element;

[0041] Step 104, obtaining a light spot state corresponding to the light-emitting element in the first image;

[0042] Step 106: Control the light-emitting element to turn on or off according to the switch state and the light spot state.

[0043] In an embodiment of the present application, the control method of the above-mentioned eye tracking device is executed by an electronic device, which may be a virtual reality device (VR or AR device), a mobile phone, a tablet computer, a laptop computer, etc.

[0044] Specifically, the eye-tracking device includes at least one light-emitting element that can emit light toward the user's eyeball when the user wears the virtual reality device. Furthermore, the eye-tracking device also includes a tracking module that generates a first image of the eyeball. The first image can display a light spot formed by the light-emitting element on the eyeball, thereby tracking the movement of the eyeball based on the position of the light spot in the first image.

[0045] Furthermore, after the user puts on the virtual reality device, the on / off state of at least one light-emitting element is first obtained, that is, whether each light-emitting element is on or off. Then, the state of the light spot corresponding to the light-emitting element is obtained in the first image. That is, when the light-emitting element is turned on, the state of the light spot corresponding to the light-emitting element is determined in the first image.

[0046] Furthermore, the light emitting element is controlled to be turned on or off according to the light emitting state of the light emitting element and the state of the light spot corresponding to the light emitting element.

[0047] FIG2 shows one of the schematic diagrams of the first image of an embodiment of the present application. As shown in FIG2 , during actual use, as the user's eyeball rotates, the tracking module of the eye tracking device will capture a first image at regular intervals. The state of the light spot in each first image may be different. Therefore, during the eye tracking process, it cannot be guaranteed that all light-emitting components can generate effective light spots in the first image. In addition, it is only necessary to ensure that two light spots can be captured in the first image to ensure the accuracy of eye rotation tracking, and it is not necessary to capture the light spots of all light-emitting components. Therefore, by controlling the on and off of the light-emitting components based on the light state of the light-emitting components and the light spot state in the first image during actual use, the light-emitting components are dynamically controlled as the user's eyeball rotates during use. Then, on the basis of ensuring that there are at least two light spots in the first image, that is, on the basis of ensuring the final effect of the eyeball, unnecessary light-emitting components are turned off to achieve the purpose of reducing power consumption in virtual reality.

[0048] During actual use of the virtual reality device, the embodiment of the present application obtains information based on the on / off state of the light-emitting element of the eye tracking device and the corresponding light spot state in the first image, thereby dynamically controlling the turning on or off of the light-emitting element as the user's eye moves. While ensuring the eye tracking effect, unnecessary light-emitting elements are turned off, thereby achieving the purpose of reducing the power consumption of the virtual reality device, reducing the heat generation of the virtual reality device, and improving the user experience of the virtual reality device.

[0049] In some embodiments of the present application, controlling the light-emitting element to turn on or off according to the switch state and the light spot state includes:

[0050] When the light emitting element is in the on state and the light spot state corresponding to the light emitting element is not present in N frames of first images generated continuously, the light emitting element is controlled to be turned off, where N is a positive integer.

[0051] In an embodiment of the present application, while the light-emitting element is in the on state, N consecutive first image frames are acquired, and then, in the N consecutive first image frames, the light spot state corresponding to the light-emitting element in the on state is detected. If the light spot state corresponding to the light-emitting element is not present in all the N consecutive first image frames, that is, although the light-emitting element is in the on state, no light spot corresponding to the light-emitting element is detected in the first image, then it means that the light-emitting element in the on state has not formed an effective light spot in the first image. In this case, the light-emitting element can be turned off to avoid the light-emitting element being left on for a long time, which would increase the power consumption of the virtual reality device.

[0052] Specifically, during actual operation, a table can be created to tally all light-emitting components and the presence of their corresponding light spots. This table can then be used to tally the on / off status of each light-emitting component and the corresponding light spot status in N consecutive first image frames. For example, if a light-emitting component is on, the table records the status as +1; if it is off, the status is -1. If the light spot corresponding to the light-emitting component exists, the status is recorded as 0; if it does not exist, the status is left blank. Based on the data tallied in the table, it is determined whether the light spot corresponding to the light-emitting component has not been detected in N consecutive first image frames while it is on, and the light-emitting component can then be controlled accordingly.

[0053] The embodiment of the present application determines the light spot state corresponding to the turned-on light-emitting element in N consecutive frames of first images, and when the light spot state in the N consecutive frames of first images is non-existent, it can be determined that although the light-emitting element has been turned on, it has not formed an effective light spot in the first image, and then the light-emitting element is turned off to reduce the power consumption of the virtual reality device.

[0054] In some embodiments of the present application, the light spot state further includes the number of pixels of the light spot. Controlling the light emitting element to turn on or off according to the switch state and the light spot state includes:

[0055] When the light-emitting element is in an on state and the number of pixels of the light spot corresponding to the light-emitting element is greater than a first value, controlling the light-emitting element corresponding to the light spot to be turned off;

[0056] When the light-emitting element is in an on state and the number of pixels of the light spot corresponding to the light-emitting element is less than a second value, the light-emitting element corresponding to the light spot is controlled to be turned off.

[0057] In an embodiment of the present application, in the first image, if the number of pixels of the light spot is detected to be greater than the first value, or the number of pixels of the light spot is less than the second value, it indicates that there is an abnormality in the light spot, which may affect the eye tracking effect. At this time, the light-emitting element corresponding to the light spot can be turned off to ensure the eye tracking effect.

[0058] It can be understood that FIG3 shows a second schematic diagram of the first image of the embodiment of the present application, FIG4 shows a schematic diagram of the eye area image of the first image in FIG3, and FIG5 shows a schematic diagram of the light spot of the first image in FIG3. As shown in FIG3, FIG4 and FIG5, by analyzing the first image, the number of pixels of the light spot in the first image can be determined. Specifically, first, the first image can be analyzed by an image analysis algorithm, such as a Histogram of Gradient algorithm or a Harr-like algorithm, so as to extract the eye area image from the first image. Further, the eye area image is processed to extract the light spot information in the eye area image, obtain the number of pixels of the light spot, and then determine whether the light-emitting element corresponding to the light spot is turned on or off according to the number of pixels of the light spot.

[0059] In the embodiment of the present application, the number of pixels of the light spot in the first image is determined to determine whether the light spot display in the first image is abnormal. When the number of pixels of the light spot is equal to the first value or less than the second value, it can be determined that the light spot display is abnormal. At this time, the light-emitting element corresponding to the light spot is turned off, thereby ensuring the eye tracking effect and reducing power consumption.

[0060] In some embodiments of the present application, the light spot state further includes a ratio between the length of the light spot and the width of the light spot. Controlling the light-emitting element to turn on or off according to the switch state and the light spot state includes:

[0061] When the light emitting element is in an on state and the ratio between the length of the light spot corresponding to the light emitting element and the width of the light spot is greater than a third value, the light emitting element is controlled to be turned off.

[0062] In this embodiment of the present application, if the ratio of the length to the width of a light spot detected in the first image is greater than a third value, it indicates that the light spot is abnormal and may affect the eye tracking effect. In this case, the light element corresponding to the light spot can be turned off to ensure the eye tracking effect. It is understood that the length of the light spot can be the maximum dimension of the light spot in the first direction, the width of the light spot can be the maximum dimension of the light spot in the second direction, and the first direction and the second direction are not parallel.

[0063] It is understood that FIG3 shows a second schematic diagram of the first image of the embodiment of the present application, FIG4 shows a schematic diagram of the eyeball area image of the first image in FIG3, and FIG5 shows a schematic diagram of the light spot of the first image in FIG3. As shown in FIG3, FIG4 and FIG5, by analyzing the first image, the number of pixels of the light spot in the first image can be determined. Specifically, first, the first image can be analyzed by an image analysis algorithm, such as a Histogram of Gradient algorithm or a Harr-like algorithm, so as to extract the eyeball area from the first image. Further, the eyeball area image is processed to extract the light spot information in the eyeball area image, obtain the length and width of the light spot, and then determine the ratio between the length and width of the light spot, and then determine whether the light-emitting element corresponding to the light spot is on or off.

[0064] The embodiment of the present application determines whether the light spot display in the first image is abnormal by obtaining the ratio between the length and the width of the light spot. When the ratio between the length and the width of the light spot is greater than a third value, it can be determined that the light spot display is abnormal. At this time, the light-emitting element corresponding to the light spot is turned off, thereby ensuring the eye tracking effect and reducing power consumption.

[0065] In some embodiments of the present application, controlling the light-emitting element to turn on or off according to the switch state and the light spot state further includes:

[0066] When the duration of the light emitting element being in the off state reaches a first duration, the light emitting element is controlled to be turned on.

[0067] In the embodiment of the present application, the timing of turning on the light-emitting element can also be determined based on the duration of the light-emitting element being in the off state. That is, when the duration of the light-emitting element being in the off state reaches a first duration, the light-emitting element can be controlled to turn on. Specifically, the first duration can be set to 33 milliseconds.

[0068] The embodiment of the present application turns on the light-emitting element when the duration of the light-emitting element being in the off state reaches a first duration, thereby avoiding the number of light spots in the first image being unable to meet the requirements of eye tracking and ensuring the effect of eye tracking.

[0069] In some embodiments of the present application, controlling the light-emitting element to turn on or off according to the switch state and the light spot state further includes:

[0070] When the number of light spots in the first image is less than a fourth value, all the light-emitting elements are controlled to be turned on.

[0071] In the embodiment of the present application, the timing of turning on the light emitting elements may also be determined based on the number of light spots in the first image, that is, when the number of light spots in the first image is less than the fourth value, all light emitting elements are turned on.

[0072] It is understood that in order to ensure that the light spots meet the requirements for eye tracking, the number of light spots in the first image must be greater than or equal to 2. Therefore, if the number of light spots in the first image is less than 2, all light-emitting elements are turned on to ensure that the number of light spots in the first image is greater than or equal to 2, thereby ensuring the eye tracking effect. In other words, the fourth value can be 2.

[0073] The embodiment of the present application determines the number of light spots in the first image, and when the number of light spots in the first image is less than a fourth value, all light-emitting components are turned on to ensure that the number of light spots in the first image is greater than or equal to the fourth value, thereby ensuring the accuracy of eye tracking.

[0074] In some embodiments of the present application, before obtaining the on / off state of the light-emitting element, the control method further includes:

[0075] Obtaining a distribution state of light spots in the first image of M frames, where M is an integer greater than or equal to 2;

[0076] The corresponding relationship between the light emitting element and the light spot is determined according to the on / off state of the light emitting element corresponding to the M frames of the first image and the distribution state of the light spot in the M frames of the first image.

[0077] In an embodiment of the present application, before controlling the on and off of the light-emitting element through the on / off state of the light-emitting element and the state of the light spot in the first image, it is first necessary to determine the correspondence between the light-emitting element and the light spot, so as to ensure accurate determination during the on and off control of the light-emitting element and avoid calculation errors during eye tracking.

[0078] Specifically, the distribution of light spots in M ​​frames of first images is first acquired, where M is an integer greater than or equal to 2. That is, at least two first images are acquired. Then, based on the on / off state of the light-emitting element corresponding to each first image and the distribution of light spots in each first image, the correspondence between the light-emitting element and the light spot is determined. The on / off state of the light-emitting element corresponding to each first image is different. In other words, the light spot distribution states of the M frames of first images are compared, and the light spot corresponding to each light-emitting element is determined based on the on / off state of the light-emitting element corresponding to each first image.

[0079] For example, FIG6 shows a third schematic diagram of the first image of an embodiment of the present application, and FIG7 shows a schematic diagram of the light-emitting element in the eye tracking device of an embodiment of the present application, as shown in FIG6 and FIG7. The number of light-emitting elements is eight. In the first image at the first moment, a light spot appears and it is difficult to define whether the light spot on the far right of the pupil corresponds to light-emitting element 2 or 3. At this time, at the second moment, light-emitting element No. 2 can be controlled to be turned off, and then the first image at the second moment is checked. If the light spot on the far right of the pupil disappears in the first image at the second moment, then it can be confirmed that the light spot corresponds to light No. 2. On the contrary, if the light spot still exists in the first image at the second moment, then it can be confirmed that the light spot corresponds to light No. 3. Generally, there is a situation where at most one light spot corresponds to three light-emitting elements that cannot be defined. The correct relationship between the light spot and the light-emitting element can be confirmed by controlling two of the light-emitting elements to turn on and off in sequence. Assume that in the first image at the first moment, the suspected light-emitting components corresponding to the light spot are 1, 2, and 3; in the first step, light-emitting component No. 1 is controlled to be turned off at the second moment, and the light spot in the first image at the second moment disappears. It can be determined that light-emitting component No. 1 corresponds to the light spot. Otherwise, light-emitting component No. 1 is excluded, and the next step is carried out. At the third moment, light-emitting component No. 2 is controlled to be turned off. The light spot in the first image at the second moment disappears. It can be determined that light-emitting component No. 2 corresponds to the light spot. Otherwise, it is determined that light-emitting component No. 3 corresponds to the light spot, and the process ends and exits.

[0080] The embodiment of the present application obtains the distribution state of the light spot in the first image of the M frame and the on-off state of the light-emitting component in the first image of the M frame, thereby determining the correspondence between the light-emitting component and the light spot according to the distribution state of the light spot in the first image of the M frame and the on-off state of the corresponding light-emitting component, thereby avoiding the inaccurate determination of the light spot position that affects the eye tracking algorithm process, and thus ensuring the robustness of the eye tracking algorithm.

[0081] In some embodiments of the present application, before obtaining the on / off state of the light-emitting element, the control method further includes:

[0082] At a first moment, respectively obtaining a distribution state of the light spot in the first image corresponding to the left eyeball and a distribution state of the light spot in the first image corresponding to the right eyeball;

[0083] The correspondence between the light emitting element and the light spot is determined according to the distribution state of the light spot in the first image corresponding to the left eyeball and the distribution state of the light spot in the first image corresponding to the right eyeball.

[0084] In the embodiment of the present application, the correspondence between the light emitting element and the light spot can also be determined by the distribution of the light spots in the first image corresponding to the left eye and the first image corresponding to the right eye at the first moment. In other words, the correspondence between the light emitting element and the light panel can be determined by utilizing the symmetry of the left and right eyes.

[0085] Specifically, the first images corresponding to the left eyeball and the right eyeball are obtained respectively, and then the distribution states of the light spots in the two first images are compared, that is, the relative position between each light spot and the eyeball is determined, and then the corresponding relationship between the light-emitting component and the light spot is determined based on the symmetrical relationship between the left and right eyeballs and the switching state of the light-emitting component.

[0086] The embodiment of the present application utilizes the symmetry of the left and right eyeballs to determine the correspondence between the light-emitting element and the light spot based on the distribution states of the light spots in the first image of the left eyeball and the first image of the right eyeball at the first moment. This can avoid the inaccurate determination of the light spot position affecting the eye tracking algorithm process, thereby ensuring the robustness of the eye tracking algorithm.

[0087] In some embodiments of the present application, a control device for an eye-tracking device is provided. The eye-tracking device includes a tracking module and at least one light-emitting element. The at least one light-emitting element is configured to emit light toward the eye. The tracking module is configured to generate a first image of the eye and track the eye based on a light spot on the eye in the first image. FIG8 shows a block diagram of a control device 800 for an eye-tracking device according to an embodiment of the present application. As shown in FIG8 , the control device 800 for the eye-tracking device includes:

[0088] An acquisition unit 802 is configured to acquire the on / off state of the light-emitting element; and

[0089] Acquire a light spot state corresponding to the light-emitting element in the first image;

[0090] The control unit 804 is used to control the light-emitting element to turn on or off according to the switch state and the light spot state.

[0091] During actual use of the virtual reality device, the embodiment of the present application obtains information based on the on / off state of the light-emitting element of the eye tracking device and the corresponding light spot state in the first image, thereby dynamically controlling the turning on or off of the light-emitting element as the user's eye moves. While ensuring the eye tracking effect, unnecessary light-emitting elements are turned off, thereby achieving the purpose of reducing the power consumption of the virtual reality device, reducing the heat generation of the virtual reality device, and improving the user experience of the virtual reality device.

[0092] In some embodiments of the present application, the control unit is specifically used to control the light-emitting element to turn off when the light-emitting element is in the on state and the light spot state corresponding to the light-emitting element is non-existent in N frames of first images generated continuously, where N is a positive integer.

[0093] The embodiment of the present application determines the light spot state corresponding to the turned-on light-emitting element in N consecutive frames of first images, and when the light spot state in the N consecutive frames of first images is non-existent, it can be determined that although the light-emitting element has been turned on, it has not formed an effective light spot in the first image, and then the light-emitting element is turned off to reduce the power consumption of the virtual reality device.

[0094] In some embodiments of the present application, the light spot state further includes the number of pixels of the light spot, and the control unit is specifically configured to: when the light-emitting element is in the on state and the number of pixels of the light spot corresponding to the light-emitting element is greater than a first value, control the light-emitting element corresponding to the light spot to turn off;

[0095] When the light-emitting element is in an on state and the number of pixels of the light spot corresponding to the light-emitting element is less than a second value, the light-emitting element corresponding to the light spot is controlled to be turned off.

[0096] In the embodiment of the present application, the number of pixels of the light spot in the first image is determined to determine whether the light spot display in the first image is abnormal. When the number of pixels of the light spot is equal to the first value or less than the second value, it can be determined that the light spot display is abnormal. At this time, the light-emitting element corresponding to the light spot is turned off, thereby ensuring the eye tracking effect and reducing power consumption.

[0097] In some embodiments of the present application, the light spot state also includes the ratio between the length of the light spot and the width of the light spot, and the control unit is specifically used to: when the light-emitting element is in the on state and the ratio between the length of the light spot corresponding to the light-emitting element and the width of the light spot is greater than a third value, control the light-emitting element to turn off.

[0098] The embodiment of the present application determines whether the light spot display in the first image is abnormal by obtaining the ratio between the length and the width of the light spot. When the ratio between the length and the width of the light spot is greater than a third value, it can be determined that the light spot display is abnormal. At this time, the light-emitting element corresponding to the light spot is turned off, thereby ensuring the eye tracking effect and reducing power consumption.

[0099] In some embodiments of the present application, the control unit is further configured to control the light emitting element to turn on when the duration of the light emitting element being in the off state reaches a first duration.

[0100] The embodiment of the present application turns on the light-emitting element when the duration of the light-emitting element being in the off state reaches a first duration, thereby avoiding the number of light spots in the first image being unable to meet the requirements of eye tracking and ensuring the effect of eye tracking.

[0101] In some embodiments of the present application, the control unit is further configured to: control all light-emitting elements to turn on when the number of light spots in the first image is less than a fourth value.

[0102] The embodiment of the present application determines the number of light spots in the first image, and when the number of light spots in the first image is less than a fourth value, all light-emitting components are turned on to ensure that the number of light spots in the first image is greater than or equal to the fourth value, thereby ensuring the accuracy of eye tracking.

[0103] In some embodiments of the present application, the acquisition unit is further configured to acquire a distribution state of light spots in M ​​frames of the first image, where M is an integer greater than or equal to 2;

[0104] The control device also includes a determination unit for determining the correspondence between the light-emitting element and the light spot based on the on state of the light-emitting element corresponding to the M-frame first image and the distribution state of the light spot in the M-frame first image, wherein the on state of the light-emitting element corresponding to the M-frame first image is different.

[0105] The embodiment of the present application obtains the distribution state of the light spot in the first image of the M frame and the on-off state of the light-emitting component in the first image of the M frame, thereby determining the correspondence between the light-emitting component and the light spot according to the distribution state of the light spot in the first image of the M frame and the on-off state of the corresponding light-emitting component, thereby avoiding the inaccurate determination of the light spot position that affects the eye tracking algorithm process, and thus ensuring the robustness of the eye tracking algorithm.

[0106] In some embodiments of the present application, the acquisition unit is further configured to respectively acquire, at a first moment, a distribution state of the light spot in the first image corresponding to the left eyeball and a distribution state of the light spot in the first image corresponding to the right eyeball;

[0107] The determining unit is further configured to determine a correspondence between the light emitting element and the light spot according to a distribution state of the light spot in the first image corresponding to the left eyeball and a distribution state of the light spot in the first image corresponding to the right eyeball.

[0108] The control device of the eye tracking device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0109] The control device of the eye tracking device in the embodiment of the present application can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0110] The control device of the eye tracking device provided in the embodiment of the present application can implement each process implemented in the above method embodiment. To avoid repetition, they will not be described here.

[0111] Optionally, an embodiment of the present application further provides an electronic device. Figure 9 shows a structural block diagram of the electronic device of an embodiment of the present application. As shown in Figure 9, the electronic device 900 includes a processor 902, a memory 904, and a program or instruction stored in the memory 904 and executable on the processor 902. When the program or instruction is executed by the processor 902, each process of the above-mentioned method embodiment is implemented and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0112] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0113] FIG10 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0114] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009 and a processor 1010.

[0115] Those skilled in the art will appreciate that the electronic device 1000 may further include a power source (such as a battery) to power various components. The power source may be logically connected to the processor 1010 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The electronic device structure shown in FIG10 does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components, or have different component arrangements, which will not be described in detail here.

[0116] The processor 1010 is configured to obtain a switch state of the light-emitting element; obtain a light spot state corresponding to the light-emitting element in the first image; and control the light-emitting element to be turned on or off according to the switch state and the light spot state.

[0117] During actual use of the virtual reality device, the embodiment of the present application obtains information based on the on / off state of the light-emitting element of the eye tracking device and the corresponding light spot state in the first image, thereby dynamically controlling the turning on or off of the light-emitting element as the user's eye moves. While ensuring the eye tracking effect, unnecessary light-emitting elements are turned off, thereby achieving the purpose of reducing the power consumption of the virtual reality device, reducing the heat generation of the virtual reality device, and improving the user experience of the virtual reality device.

[0118] Optionally, the processor 1010 is further configured to control the light emitting element to turn off when the light emitting element is in an on state and the light spot state corresponding to the light emitting element is not present in N consecutively generated first image frames, where N is a positive integer.

[0119] The embodiment of the present application determines the light spot state corresponding to the turned-on light-emitting element in N consecutive frames of first images, and when the light spot state in the N consecutive frames of first images is non-existent, it can be determined that although the light-emitting element has been turned on, it has not formed an effective light spot in the first image, and then the light-emitting element is turned off to reduce the power consumption of the virtual reality device.

[0120] Optionally, the light spot state also includes the number of pixels of the light spot, and the processor 1010 is also used to control the light emitting element corresponding to the light spot to turn off when the light emitting element is in the on state and the number of pixels of the light spot corresponding to the light emitting element is greater than a first value; and to control the light emitting element corresponding to the light spot to turn off when the light emitting element is in the on state and the number of pixels of the light spot corresponding to the light emitting element is less than a second value.

[0121] In the embodiment of the present application, the number of pixels of the light spot in the first image is determined to determine whether the light spot display in the first image is abnormal. When the number of pixels of the light spot is equal to the first value or less than the second value, it can be determined that the light spot display is abnormal. At this time, the light-emitting element corresponding to the light spot is turned off, thereby ensuring the eye tracking effect and reducing power consumption.

[0122] Optionally, the light spot state also includes the ratio between the length of the light spot and the width of the light spot. The processor 1010 is also used to control the light emitting element to turn off when the light emitting element is in the on state and the ratio between the length of the light spot corresponding to the light spot and the width of the light spot is greater than a third value.

[0123] The embodiment of the present application determines whether the light spot display in the first image is abnormal by obtaining the ratio between the length and the width of the light spot. When the ratio between the length and the width of the light spot is greater than a third value, it can be determined that the light spot display is abnormal. At this time, the light-emitting element corresponding to the light spot is turned off, thereby ensuring the eye tracking effect and reducing power consumption.

[0124] Optionally, the processor 1010 is further configured to control the light emitting element to turn on when the duration of the light emitting element being in the off state reaches a first duration.

[0125] The embodiment of the present application turns on the light-emitting element when the duration of the light-emitting element being in the off state reaches a first duration, thereby avoiding the number of light spots in the first image being unable to meet the requirements of eye tracking and ensuring the effect of eye tracking.

[0126] Optionally, the processor 1010 is further configured to control all the light-emitting elements to turn on when the number of light spots in the first image is less than a fourth value.

[0127] The embodiment of the present application determines the number of light spots in the first image, and when the number of light spots in the first image is less than a fourth value, all light-emitting components are turned on to ensure that the number of light spots in the first image is greater than or equal to the fourth value, thereby ensuring the accuracy of eye tracking.

[0128] Optionally, before obtaining the switching state of the light-emitting component, the processor 1010 is further used to obtain the distribution state of the light spot in the M-frame first image, where M is an integer greater than or equal to 2; according to the switching state of the light-emitting component corresponding to the M-frame first image and the distribution state of the light spot in the M-frame first image, the correspondence between the light-emitting component and the light spot is determined, wherein the on state of the light-emitting component corresponding to the M-frame first image is different.

[0129] The embodiment of the present application obtains the distribution state of the light spot in the first image of the M frame and the on-off state of the light-emitting component in the first image of the M frame, thereby determining the correspondence between the light-emitting component and the light spot according to the distribution state of the light spot in the first image of the M frame and the on-off state of the corresponding light-emitting component, thereby avoiding the inaccurate determination of the light spot position that affects the eye tracking algorithm process, and thus ensuring the robustness of the eye tracking algorithm.

[0130] Optionally, before obtaining the switching state of the light-emitting component, the processor 1010 is also used to obtain the distribution state of the light spot in the first image corresponding to the left eyeball and the distribution state of the light spot in the first image corresponding to the right eyeball at the first moment; and determine the correspondence between the light-emitting component and the light spot based on the distribution state of the light spot in the first image corresponding to the left eyeball and the distribution state of the light spot in the first image corresponding to the right eyeball.

[0131] The embodiment of the present application utilizes the symmetry of the left and right eyeballs to determine the correspondence between the light-emitting element and the light spot based on the distribution states of the light spots in the first image of the left eyeball and the first image of the right eyeball at the first moment. This can avoid the inaccurate determination of the light spot position affecting the eye tracking algorithm process, thereby ensuring the robustness of the eye tracking algorithm.

[0132] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of a static image or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0133] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0134] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0135] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0136] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0137] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0138] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0139] An embodiment of the present application provides a computer program product, which is stored in a readable storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0140] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a readable storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0142] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A control method for an eye tracking device, the eye tracking device comprising a tracking module and at least one light emitting element, the at least one light emitting element being used to emit light to an eye, the tracking module being used to generate a first image of the eye and track the eye according to a light spot on the eye in the first image, the control method comprising: Obtaining the switch state of the light-emitting element; Acquire a light spot state corresponding to the light emitting element in the first image; The light emitting element is controlled to be turned on or off according to the switch state and the light spot state.

2. The control method according to claim 1, wherein: The step of controlling the light emitting element to turn on or off according to the switch state and the light spot state includes: When the light-emitting element is in the on state and the light spot state corresponding to the light-emitting element in the N frames of first images generated continuously is non-existent, the light-emitting element is controlled to be turned off, wherein N is a positive integer.

3. The control method according to claim 1, wherein: The light spot state also includes the number of pixels of the light spot, and controlling the light emitting element to turn on or off according to the switch state and the light spot state includes: When the light-emitting element is in an on state and the number of pixels of the light spot corresponding to the light-emitting element is greater than a first value, controlling the light-emitting element corresponding to the light spot to be turned off; When the light-emitting element is in an on state and the number of pixels of the light spot corresponding to the light-emitting element is less than a second value, the light-emitting element corresponding to the light spot is controlled to be off.

4. The control method according to claim 1, wherein: The light spot state further includes a ratio between the length of the light spot and the width of the light spot. Controlling the light emitting element to be turned on or off according to the switch state and the light spot state includes: When the light emitting element is in an on state and the ratio between the length of the light spot corresponding to the light emitting element and the width of the light spot is greater than a third value, the light emitting element is controlled to be turned off.

5. The control method according to claim 1, wherein: The controlling the light emitting element to be turned on or off according to the switch state and the light spot state further includes: When the duration of the light emitting element being in the off state reaches a first duration, the light emitting element is controlled to be turned on.

6. The control method according to claim 1, wherein: The controlling the light emitting element to be turned on or off according to the switch state and the light spot state further includes: When the number of the light spots in the first image is less than a fourth value, all the light-emitting elements are controlled to turn on.

7. The control method according to any one of claims 1 to 6, wherein: Before acquiring the switch state of the light-emitting element, the control method further includes: Acquire the distribution state of the light spots in M ​​frames of the first images, where M is an integer greater than or equal to 2; The correspondence between the light emitting element and the light spot is determined according to the on state of the light emitting element corresponding to the first image of the M frames and the distribution state of the light spot in the first image of the M frames, wherein the on states of the light emitting element corresponding to the first image of the M frames are different.

8. The control method according to any one of claims 1 to 6, wherein: Before acquiring the switch state of the light-emitting element, the control method further includes: At a first moment, respectively acquiring a distribution state of the light spot in the first image corresponding to a left eyeball and a distribution state of the light spot in the first image corresponding to a right eyeball; The corresponding relationship between the light emitting element and the light spot is determined according to the distribution state of the light spot in the first image corresponding to the left eyeball and the distribution state of the light spot in the first image corresponding to the right eyeball.

9. A control device for an eye tracking device, the eye tracking device comprising a tracking module and at least one light emitting element, at least one of the light emitting elements being used to emit light to an eye, the tracking module being used to generate a first image of the eye and to track the eye according to a light spot on the eye in the first image, the control device comprising: An acquisition device, used for acquiring the switch state of the light-emitting element; as well as Acquire a light spot state corresponding to the light emitting element in the first image; A control device is used to control the light emitting element to turn on or off according to the switch state and the light spot state.

10. The control device according to claim 9, wherein: A control unit is used to control the light-emitting element to be turned off when the light-emitting element is in an on state and the light spot state corresponding to the light-emitting element in N frames of first images generated continuously is non-existent, wherein N is a positive integer.

11. The control device according to claim 9, wherein: The light spot state also includes the number of pixels of the light spot, and the control unit is used to: When the light-emitting element is in an on state and the number of pixels of the light spot corresponding to the light-emitting element is greater than a first value, controlling the light-emitting element corresponding to the light spot to be turned off; When the light-emitting element is in an on state and the number of pixels of the light spot corresponding to the light-emitting element is less than a second value, the light-emitting element corresponding to the light spot is controlled to be off.

12. The control device according to claim 9, wherein: The light spot state also includes a ratio between the length of the light spot and the width of the light spot. The control unit is used to control the light emitting component to be turned off when the light emitting component is in an on state and the ratio between the length of the light spot corresponding to the light spot and the width of the light spot is greater than a third value.

13. The control device according to claim 9, wherein: The controlling the light emitting element to be turned on or off according to the switch state and the light spot state further includes: When the duration of the light emitting element being in the off state reaches a first duration, the light emitting element is controlled to be turned on.

14. The control device according to claim 9, wherein: A control unit is used to control all the light-emitting elements to turn on when the number of the light spots in the first image is less than a fourth value.

15. The control device according to any one of claims 9 to 14, wherein: An acquisition unit, used for acquiring the distribution state of the light spot in M ​​frames of the first image, where M is an integer greater than or equal to 2; The control device also includes a determination unit, which is used to determine the correspondence between the light-emitting component and the light spot based on the turn-on state of the light-emitting component corresponding to the first image of the M frames and the distribution state of the light spot in the first image of the M frames, wherein the turn-on states of the light-emitting components corresponding to the first images of the M frames are different.

16. The control device according to any one of claims 9 to 14, wherein: an acquisition unit, configured to respectively acquire, at a first moment, a distribution state of the light spot in the first image corresponding to a left eyeball and a distribution state of the light spot in the first image corresponding to a right eyeball; The control device also includes a determination unit, which is used to determine the correspondence between the light-emitting element and the light spot based on the distribution state of the light spot in the first image corresponding to the left eyeball and the distribution state of the light spot in the first image corresponding to the right eyeball.

17. An electronic device comprising a processor and a memory, wherein the memory stores a program or input executable on the processor, wherein the program or input implements the steps of the method according to any one of claims 1 to 8 when executed by the processor.

18. A readable storage medium storing a program or input, wherein the program or input is executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

19. A chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method according to any one of claims 1 to 8.

20. A computer program product, wherein the computer program product is stored in a readable storage medium, and wherein the computer program product is executed by at least one processor to implement the method according to any one of claims 1 to 8.

Images