Eyeball detection method, device, electronic device and storage medium

Abstract

The present disclosure relates to an eyeball detection method, device, system, electronic device and storage medium. The method is applied to a head-mounted device, which is provided with an event camera and at least one light-emitting element, and comprises: controlling the light-emitting element to flash light toward an eyeball according to a preset frequency, and acquiring event information output by the event camera, wherein the event information is used to characterize the change of pixels on an imaging plane of the event camera; filtering the event information according to the preset frequency to obtain a target event, wherein the target event includes an event caused by the flashing light of the light-emitting element; determining the position of a flashing point of the light-emitting element on the imaging plane according to the target event, and determining the position of the eyeball according to the position of the flashing point.

Description

Technical Field

[0001] The present disclosure relates to the field of target detection technology, and in particular to an eye detection method, device, electronic device and storage medium. Background Art

[0002] In recent years, with the advancement of science and technology, rendering devices such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality) have gradually emerged. These rendering devices can render various virtual scenes, thereby bringing users an audio-visual experience different from the display content of traditional display screens. These rendering devices have various forms such as head-mounted, and each form can provide corresponding functions. For example, head-mounted rendering devices can use their fixed characteristics with the user's head to track the user's eyes so that the rendering effect adapts to the user's line of sight. However, in the related art, the accuracy of eye tracking of head-mounted devices needs to be improved. Summary of the invention

[0003] The present disclosure provides an eye detection method, an apparatus, an electronic device and a storage medium.

[0004] According to some embodiments of the present disclosure, there is provided an eye detection method, which is applied to a head mounted device, wherein the head mounted device is provided with an event camera and at least one light emitting element, and the method comprises:

[0005] Controlling the light-emitting element to flash light toward the eyeball according to a preset frequency, and acquiring event information output by the event camera, wherein the event information is used to characterize changes in pixels on an imaging plane of the event camera;

[0006] Filtering the event information according to the preset frequency to obtain a target event, wherein the target event includes an event caused by the flashing light of the light-emitting element;

[0007] The position of the flashing point of the light emitting element on the imaging plane is determined according to the target event, and the position of the eyeball is determined according to the position of the flashing point.

[0008] In one embodiment, the light emitting element comprises a lamp group, and the lamp group comprises a plurality of lamps;

[0009] The step of controlling the light emitting element to flash and emit light toward the eyeball according to a preset frequency comprises:

[0010] Each lamp in the lamp group is controlled to flash in sequence according to a preset frequency.

[0011] In one embodiment, controlling the light emitting element to flash light toward the eyeball at a preset frequency includes:

[0012] According to the pulse control sequence, the light-emitting duration of the light-emitting element is controlled in each flashing cycle corresponding to the preset frequency.

[0013] In one embodiment, the pulse control sequence includes a plurality of pulse control units, each of which is used to control the light emission duration of at least one flashing cycle.

[0014] In one embodiment, in the pulse control sequences of different light-emitting elements, the pulse control units at the same position are different.

[0015] In one embodiment, filtering the event information according to the preset frequency to obtain the target event includes:

[0016] According to each light emitting duration, the target event in each flashing cycle corresponding to the preset frequency is retained.

[0017] In one embodiment, filtering the event information according to the preset frequency to obtain the target event includes:

[0018] The event information is filtered according to the preset frequency in a window suppression manner to obtain a filtered image.

[0019] In one embodiment, determining the position of the flashing point of the light emitting element on the imaging plane according to the target event includes:

[0020] The position of the scintillation point of each of the light emitting elements on the imaging plane is determined according to the target event in each of the at least one scintillation cycle and the pulse control sequence of each of the light emitting elements.

[0021] In one embodiment, determining the position of the eyeball according to the position of the scintillation point includes:

[0022] The position of the eyeball is determined according to the flashing point position of each light emitting element and the reprojection position of each light emitting element on the imaging plane.

[0023] In one embodiment, it further includes:

[0024] The currently determined position of the eyeball is corrected according to the position of the eyeball determined at least once previously.

[0025] In one embodiment, the head mounted device further includes a rendering element for rendering a virtual scene; and the method further includes:

[0026] Determining the position of the gaze point in the virtual scene according to the position of the eyeball;

[0027] The focus of the rendering element is adjusted according to the gaze point position.

[0028] According to some embodiments of the present disclosure, an eye detection device is provided, which is applied to a head-mounted device, wherein the head-mounted device is provided with an event camera and at least one light-emitting element, and the device includes:

[0029] A blinking module, used to control the light emitting element to blink light toward the eyeball at a preset frequency, and obtain event information output by the event camera, wherein the event information is used to characterize the change of pixels on the imaging plane of the event camera;

[0030] A filtering module, used for filtering the event information according to the preset frequency to obtain a target event, wherein the target event includes an event caused by the flashing light of the light-emitting element;

[0031] A detection module is used to determine the position of the flashing point of the light emitting element on the imaging plane according to the target event, and determine the position of the eyeball according to the position of the flashing point.

[0032] In one embodiment, the light emitting element comprises a lamp group, and the lamp group comprises a plurality of lamps;

[0033] The flashing module is used to control the light emitting element to flash and emit light toward the eyeball according to a preset frequency, and is specifically used to:

[0034] Each lamp in the lamp group is controlled to flash in sequence according to a preset frequency.

[0035] In one embodiment, the flashing module is used to control the light emitting element to flash light toward the eyeball according to a preset frequency, specifically for:

[0036] According to the pulse control sequence, the light-emitting duration of the light-emitting element is controlled in each flashing cycle corresponding to the preset frequency.

[0037] In one embodiment, the pulse control sequence includes a plurality of pulse control units, each of which is used to control the light emission duration of at least one flashing cycle.

[0038] In one embodiment, in the pulse control sequences of different light-emitting elements, the pulse control units at the same position are different.

[0039] In one embodiment, the filtering module is specifically used for:

[0040] According to each light emitting duration, the target event in each flashing cycle corresponding to the preset frequency is retained.

[0041] In one embodiment, the filtering module is specifically used for:

[0042] The event information is filtered according to the preset frequency in a window suppression manner to obtain a filtered image.

[0043] In one embodiment, when the detection module is used to determine the position of the flashing point of the light emitting element on the imaging plane according to the target event, it is specifically used to:

[0044] The position of the scintillation point of each of the light emitting elements on the imaging plane is determined according to the target event in each of the at least one scintillation cycle and the pulse control sequence of each of the light emitting elements.

[0045] In one embodiment, when the detection module is used to determine the position of the eyeball according to the position of the scintillation point, it is specifically used to:

[0046] The position of the eyeball is determined according to the flashing point position of each light emitting element and the reprojection position of each light emitting element on the imaging plane.

[0047] In one embodiment, a correction module is further included, which is used to:

[0048] The currently determined position of the eyeball is corrected according to the position of the eyeball determined at least once previously.

[0049] In one embodiment, the head mounted device further comprises a rendering element for rendering a virtual scene; and the method further comprises a focusing module for:

[0050] Determining the position of the gaze point in the virtual scene according to the position of the eyeball;

[0051] The focus of the rendering element is adjusted according to the gaze point position.

[0052] According to some embodiments of the present disclosure, an electronic device is provided, the device comprising a memory and a processor, the memory being used to store computer instructions executable on the processor, the processor being used to implement the method described in the first aspect when executing the computer instructions.

[0053] According to some embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in the first aspect is implemented.

[0054] According to the above embodiments, the light-emitting element of the head-mounted device is first controlled to flash toward the eyeball according to a preset frequency, and the event information output by the event camera of the head-mounted device is obtained, so that the event information can be filtered according to the preset frequency to obtain the target event caused by the flashing light of the light-emitting element. Finally, the position of the flashing point of the light-emitting element on the imaging plane of the event camera can be determined according to the target event, and the position of the eyeball can be determined according to the position of the flashing point. Since the event camera outputs events in real time when the pixels on the imaging plane change, it can avoid the problems of delay, blur and information redundancy of ordinary cameras; and the light-emitting element flashes according to the preset frequency, so the signal-to-noise ratio of the target event obtained after filtering with the preset frequency is high, avoiding noise interference, and further improving the precision and accuracy of eyeball detection.

[0055] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0057] Figure 1 is a flow chart of an eye detection method shown in some embodiments of the present disclosure;

[0058] Figure 2 is a schematic diagram of the structure of an event camera and a light-emitting element shown in some embodiments of the present disclosure;

[0059] Figure 3 is a schematic diagram of the structure of an eye detection device shown in some embodiments of the present disclosure;

[0060] Figure 4 It is a schematic diagram of the structure of an electronic device shown in some embodiments of the present disclosure. DETAILED DESCRIPTION

[0061] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0062] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0063] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0064] In recent years, with the advancement of science and technology, rendering devices such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality) have gradually emerged. These rendering devices can render various virtual scenes, thereby bringing users an audio-visual experience different from the display content of traditional display screens. These rendering devices have various forms such as head-mounted, and each form can provide corresponding functions. For example, head-mounted rendering devices can use their fixed characteristics with the user's head to track the user's eyes so that the rendering effect adapts to the user's line of sight. However, in the related art, the accuracy of eye tracking of head-mounted devices needs to be improved.

[0065] Some embodiments of the present disclosure provide an eye detection method, please refer to Figure 1 , which shows the process of the authentication method, including steps S101 to S103.

[0066] The method can be applied to head-mounted devices, such as head-mounted VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality) and other rendering devices. The head-mounted device is provided with an event camera and at least one light-emitting element, and the positions of the event camera and each light-emitting element are fixedly installed on the head-mounted device, so the relative positions of the event camera and each light-emitting element are stable.

[0067] The event camera is a bio-inspired sensor with no concept of exposure and frame rate. The event camera captures events, that is, the output is the change of pixel brightness on the imaging plane. Each pixel in the imaging plane of the event camera independently detects the change of the logarithm of the light intensity, and when the change reaches a certain threshold, it outputs an event containing timestamp, position, and polarity information (for example, from dark to dark or from bright to dark). All pixels in the imaging plane asynchronously detect the change of the logarithm of the light intensity and output event information at the same time, forming an asynchronous event stream. The event camera has no frame rate limit, actively detects the location where the light intensity changes in the scene and outputs the change information. The amount of data in the event stream is much smaller than the data transmitted by the traditional camera, and the event stream has no minimum time unit, which means that the event camera does not have a fixed sampling rate, nor does it sample all pixels, but only samples when changes occur. Although the collected data is small, the accuracy and efficiency are good enough. The event camera has a high dynamic range (for example, 120dB), very little motion blur, and the delay can be as low as sub-millisecond level. The image is clear when the eyeball moves quickly, and the pupil edge position can be accurately extracted. Moreover, the power of the event camera is about 10mW, which is much lower than that of traditional cameras and can solve the problem of short battery life of head-mounted devices.

[0068] The light-emitting element may be an LED lamp, etc. In the case where the head mounted device has multiple light-emitting elements, the multiple light-emitting elements may be evenly arranged around the event camera. The light-emitting element may be a lamp group, which includes multiple lamps, such as multiple LED lamps. For example, please refer to the attached Figure 2 There are eight light emitting elements 202 evenly distributed around the event camera 201, and each light emitting element is a light group consisting of two LED lights.

[0069] In step S101, the light emitting element is controlled to flash light toward the eyeball at a preset frequency, and event information output by the event camera is obtained.

[0070] The event information is used to characterize the change of pixels on the imaging plane of the event camera. The preset frequency may be a frequency of kHz level.

[0071] The light emitting element can be controlled to flash once in each flashing cycle corresponding to a preset frequency. For example, the light emitting element is turned on from dark to bright at the beginning of the flashing cycle, and after a period of time (the time is shorter than the flashing cycle), the light emitting element is turned off to make it turn from bright to dark.

[0072] In a possible embodiment, the light-emitting duration of the light-emitting element can be controlled in each flashing cycle corresponding to the preset frequency according to the pulse control sequence. For example, multiple light-emitting durations can be pre-set to control the light-emitting duration of each flashing cycle to be one of the preset multiple light-emitting durations. The pulse control sequence includes multiple pulse control units, each of which is used to control the light-emitting duration of at least one flashing cycle. In the pulse control sequences of different light-emitting elements, the pulse control units at the same position can be different, so that the pulse control sequence can be used to make each light-emitting element present different flashes in the above-mentioned at least one flashing cycle, thereby making the event camera output different events (or event combinations).

[0073] Exemplarily, the pulse control sequence may be a binary sequence, each binary number in the sequence is a pulse control unit, and each bit of the binary number corresponds to a flashing cycle. For example, the pulse control sequence of a certain light-emitting element may be a binary sequence composed of 001, i.e. (001, 001, 001, 001...), wherein 0 and 1 represent different light-emitting durations, for example, 0 is one-third of the flashing cycle (i.e. short light), and 1 is two-thirds of the flashing cycle (i.e. long light); then the light-emitting element may emit light in the first third of the first flashing cycle (i.e. short light), in the first third of the second flashing cycle (i.e. short light), in the first two-thirds of the third flashing cycle (i.e. long light), in the first third of the fourth flashing cycle (i.e. short light), in the first third of the fifth flashing cycle (i.e. short light), in the first two-thirds of the sixth flashing cycle (i.e. long light), and the remaining cycles are not repeated again.

[0074] It is understandable that the number of flashing cycles controlled by each pulse control unit may be related to the number of light emitting elements, thereby achieving the distinction of events triggered by different light emitting elements. n light emitting elements, you can pre-set 2 n The order of binary numbers (for example, from small to large) is then used to make the pulse control sequence of each light emitting element the above 2 nThe pulse control sequence of the first light emitting element is a binary sequence composed of one of the binary numbers, and the binary numbers in the pulse control sequences of different light emitting elements are different. For example, the pulse control sequence of the first light emitting element is a binary sequence composed of 000, the pulse control sequence of the second light emitting element is a binary sequence composed of 001, the pulse control sequence of the third light emitting element is a binary sequence composed of 010, the pulse control sequence of the fourth light emitting element is a binary sequence composed of 011, the pulse control sequence of the fifth light emitting element is a binary sequence composed of 100, the pulse control sequence of the sixth light emitting element is a binary sequence composed of 101, the pulse control sequence of the seventh light emitting element is a binary sequence composed of 110, and the pulse control sequence of the eighth light emitting element is a binary sequence composed of 111.

[0075] When the light-emitting element includes a light group consisting of multiple lights, each light in the light group can be controlled to flash in sequence according to a preset frequency, thereby achieving differential illumination and enhancing the reflection concentration of the light-emitting element on the surface of the eyeball; in addition, the signal-to-noise ratio of the flashing on the surface of the eyeball can be improved by setting the distance between adjacent light-emitting elements. Exemplarily, when the light-emitting element includes a light group consisting of two lights, the two lights can be controlled to flash alternately according to a preset frequency.

[0076] In step S102, the event information is filtered according to the preset frequency to obtain a target event, wherein the target event includes an event caused by the flashing light of the light-emitting element.

[0077] There is certain noise in the event information of the event camera, such as background noise, hot spot noise, etc.

[0078] Optionally, each corresponding flickering cycle is first determined according to a preset frequency, and then corresponding flickering parameters (such as light emission duration, etc.) are used in each flickering cycle to filter event information to obtain a target event.

[0079] Exemplarily, the target event in each flickering cycle corresponding to the preset frequency can be retained according to each luminous duration. That is, among the multiple events in each flickering cycle, the events whose time matches any luminous duration are retained. The luminous duration will trigger an event from dark to bright at the start time, and will trigger an event from bright to dark at the end time. For example, the target event corresponding to each luminous duration can be obtained, and a frequency filtered image can be used to represent the target event corresponding to the luminous duration, such as the frequency filtered image corresponding to the first third of the flickering cycle, and the frequency filtered image corresponding to the first two thirds of the flickering cycle. The frequency filtered image can be the pixel coordinates of the target event corresponding to the luminous duration in the flickering cycle.

[0080] Among them, each light-emitting duration refers to each light-emitting duration when the light-emitting element is controlled to emit light in each flashing cycle in step S101. For example, the light-emitting duration may be the first third of the flashing cycle, or the first two thirds of the flashing cycle, and these two are each of the above light-emitting durations. This filtering method can retain only events at specific moments in each preset cycle, which is convenient and fast, and the filtering is accurate, thereby eliminating most of the noise in the event information.

[0081] Exemplarily, the event information may be filtered according to the preset frequency in a window suppression manner to obtain a filtered image, that is, when an event occurs, events within a short period of time within a spatial window of a preset size around the event are considered as noise, and these events are deleted.

[0082] In step S103, the position of the flashing point (formed) by the (flashing light of the) light-emitting element on the imaging plane is determined according to the target event, and the position of the eyeball is determined according to the position of the flashing point.

[0083] In a possible embodiment, the position of the flickering point of each light-emitting element on the imaging plane is determined according to the target event in each flickering cycle in the at least one flickering cycle and the pulse control sequence of each light-emitting element. Since the pulse control units at the same position are different in the pulse control sequences of different light-emitting elements, a binary flicker tracker can be used to determine the position of the flickering point of each light-emitting element using the pulse control sequence of each light-emitting element, and the position of each flicker can be obtained through binary flicker coding tracking. For example, if a pulse control unit in the pulse control sequence can control the flickering duration of three flickering cycles, the light-emitting element corresponding to each flickering point can be determined in combination with the flickering duration of the three flickering cycles. That is to say, in any three consecutive flickering cycles, the flickering duration combination of each light-emitting element is different, so the light-emitting element corresponding to the position can be determined based on the flickering duration corresponding to three events at the same or similar positions.

[0084] It can be understood that the binary flicker tracker can be divided into three stages: full initialization, semi-initialization and uninitialization; during the flicker point tracking process, if it is in an uninitialized state, that is, the tracking starts or the tracking is lost, the brightest point is searched in the two flicker images as the new position x to start tracking; when the tracker is in a fully initialized state, the pixel brightness peak is searched and updated within the λ×λ pixel range around the flicker image position x; when the tracker is in a semi-initialized state, the flicker position is determined as the weighted average pixel coordinate within the λ×λ range.

[0085] In another possible embodiment, the position of the eyeball is determined according to the position of the scintillation point of each light-emitting element and the reprojection position of each light-emitting element on the imaging plane. Exemplarily, the corneal scintillation can be simulated on a perfect sphere as an ideal mirror reflection. Under this assumption, there is a ray passing from each light-emitting element to the eyeball, and from the eyeball to the corresponding scintillation point on the imaging plane. The light from the light-emitting element to the eyeball in the camera coordinates is fixed in position, so the size and position of the corneal ball can be determined by optimizing the secondary projection error of the light on the imaging plane. * :

[0086]

[0087] Where θ represents the corneal sphere position (x c ,y c , z c ) of the vector [x c ,y c ,z c ,r c ] T ,n led Represents the number of light-emitting elements, X rn is the reprojected position of a light-emitting element on the imaging plane, X en is the flashing point position of the light emitting element.

[0088] In addition, the currently determined position of the eyeball can be corrected according to the position of the eyeball determined at least once before. Exemplarily, the currently determined position of the eyeball is smoothed according to the following formula:

[0089]

[0090] Among them, α and β are weight coefficients. The larger the coefficient α is, the closer it is to the previous position and the more stable the data is.

[0091] Through the above correction, the eyeball position change trajectory can be made more stable and realistic, further improving the eyeball tracking effect.

[0092] According to the above embodiments, the light-emitting element of the head-mounted device is first controlled to flash toward the eyeball according to a preset frequency, and the event information output by the event camera of the head-mounted device is obtained, so that the event information can be filtered according to the preset frequency to obtain the target event caused by the flashing light of the light-emitting element. Finally, the position of the flashing point of the light-emitting element on the imaging plane of the event camera can be determined according to the target event, and the position of the eyeball can be determined according to the position of the flashing point. Since the event camera outputs events in real time when the pixels on the imaging plane change, it can avoid the problems of delay, blur and information redundancy of ordinary cameras; and the light-emitting element flashes according to the preset frequency, so the signal-to-noise ratio of the target event obtained after filtering with the preset frequency is high, avoiding noise interference, and further improving the precision and accuracy of eyeball detection.

[0093] In some embodiments of the present disclosure, the head mounted device further comprises a rendering element for rendering a virtual scene, that is, an optical system for virtual reality or augmented reality. Therefore, after determining the position of the eyeball, the position of the gaze point in the virtual scene can be determined according to the position of the eyeball, and the focus of the rendering element can be further adjusted according to the position of the gaze point.

[0094] Current head-mounted devices focus at a fixed distance, and when users alternately observe close and distant objects, sensory conflicts will occur, such as eye fatigue, visual discomfort, and low image clarity. However, this embodiment can achieve real-time zoom display, thereby optimizing rendering efficiency, saving power, and alleviating visual fatigue and dizziness caused by eye pressure. Moreover, the combination of real-time dynamic zoom and eye tracking in this embodiment allows real objects at any distance to be focused, thereby improving visual clarity and image clarity.

[0095] According to some embodiments of the present disclosure, an eye detection device is provided, which is applied to a head-mounted device, wherein the head-mounted device is provided with an event camera and at least one light-emitting element. Figure 3 , the device comprises:

[0096] A blinking module 301 is used to control the light emitting element to blink light toward the eyeball according to a preset frequency, and obtain event information output by the event camera, wherein the event information is used to characterize the change of pixels on the imaging plane of the event camera;

[0097] A filtering module 302, configured to filter the event information according to the preset frequency to obtain a target event, wherein the target event includes an event caused by the flashing light of the light-emitting element;

[0098] The detection module 303 is used to determine the position of the flashing point of the light emitting element on the imaging plane according to the target event, and determine the position of the eyeball according to the position of the flashing point.

[0099] In some embodiments of the present disclosure, the light emitting element comprises a lamp group, and the lamp group comprises a plurality of lamps;

[0100] The flashing module is used to control the light emitting element to flash and emit light toward the eyeball according to a preset frequency, specifically for:

[0101] Each lamp in the lamp group is controlled to flash in sequence according to a preset frequency.

[0102] In some embodiments of the present disclosure, when the flashing module is used to control the light emitting element to flash light toward the eyeball according to a preset frequency, it is specifically used to:

[0103] According to the pulse control sequence, the light-emitting duration of the light-emitting element is controlled in each flashing cycle corresponding to the preset frequency.

[0104] In some embodiments of the present disclosure, the pulse control sequence includes a plurality of pulse control units, each of which is used to control the light emission duration of at least one flashing cycle.

[0105] In some embodiments of the present disclosure, in the pulse control sequences of different light-emitting elements, the pulse control units at the same position are different.

[0106] In some embodiments of the present disclosure, the filtering module is specifically used to:

[0107] According to each light emitting duration, the target event in each flashing cycle corresponding to the preset frequency is retained.

[0108] In some embodiments of the present disclosure, the filtering module is specifically used to:

[0109] The event information is filtered according to the preset frequency in a window suppression manner to obtain a filtered image.

[0110] In some embodiments of the present disclosure, when the detection module is used to determine the position of the flashing point of the light emitting element on the imaging plane according to the target event, it is specifically used to:

[0111] The position of the scintillation point of each of the light emitting elements on the imaging plane is determined according to the target event in each of the at least one scintillation cycle and the pulse control sequence of each of the light emitting elements.

[0112] In some embodiments of the present disclosure, when the detection module is used to determine the position of the eyeball according to the position of the scintillation point, it is specifically used to:

[0113] The position of the eyeball is determined according to the flashing point position of each light emitting element and the reprojection position of each light emitting element on the imaging plane.

[0114] In some embodiments of the present disclosure, a correction module is further included, which is used to:

[0115] The currently determined position of the eyeball is corrected according to the position of the eyeball determined at least once previously.

[0116] In some embodiments of the present disclosure, the head mounted device further includes a rendering element for rendering a virtual scene; and the method further includes a focusing module for:

[0117] Determining the position of the gaze point in the virtual scene according to the position of the eyeball;

[0118] The focus of the rendering element is adjusted according to the gaze point position.

[0119] Regarding the device in the above embodiment, the specific manner in which each module and the network perform operations has been described in detail in the embodiment of the first aspect related to the method, and will not be elaborated here.

[0120] Please refer to the attached Figure 4 Some embodiments of the present disclosure provide an electronic device, the device comprising a memory and a processor, the memory being used to store computer instructions executable on the processor, the processor being used to perform material processing based on the method described in the first aspect when executing the computer instructions.

[0121] Some embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in the first aspect is implemented.

[0122] The various component embodiments of the present disclosure may be implemented in hardware, or in a software module running on one or more processors, or in a combination thereof.

[0123] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0124] In the present disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise clearly defined.

[0125] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0126] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An eyeball detection method, characterized in that: Applied to a head mounted device, the head mounted device is provided with an event camera and at least one light emitting element, the method comprising: Controlling the light-emitting element to flash light toward the eyeball according to a preset frequency, and acquiring event information output by the event camera, wherein the event information is used to characterize changes in pixels on an imaging plane of the event camera; Filtering the event information according to the preset frequency to obtain a target event, wherein the target event includes an event caused by the flashing light of the light-emitting element; Determine the position of the flashing point of the light emitting element on the imaging plane according to the target event, and determine the position of the eyeball according to the position of the flashing point; The step of controlling the light emitting element to flash and emit light toward the eyeball according to a preset frequency comprises: According to the pulse control sequence, the light-emitting duration of the light-emitting element is controlled in each flashing cycle corresponding to the preset frequency, wherein the pulse control sequence includes a plurality of pulse control units, each pulse control unit is used to control the light-emitting duration of at least one flashing cycle, and in the pulse control sequences of different light-emitting elements, the pulse control units at the same position are different; The filtering the event information according to the preset frequency to obtain the target event includes: According to each light-emitting duration, retain the target event in each flashing cycle corresponding to the preset frequency; The step of determining the position of the flashing point of the light emitting element on the imaging plane according to the target event comprises: The position of the scintillation point of each of the light emitting elements on the imaging plane is determined according to the target event in each of the at least one scintillation cycle and the pulse control sequence of each of the light emitting elements.

2. The eyeball detection method according to claim 1, characterized in that: The light emitting element comprises a lamp group, and the lamp group comprises a plurality of lamps; The step of controlling the light emitting element to flash and emit light toward the eyeball according to a preset frequency comprises: Each lamp in the lamp group is controlled to flash in sequence according to a preset frequency.

3. The eyeball detection method according to claim 1, characterized in that: The filtering the event information according to the preset frequency to obtain the target event includes: The event information is filtered according to the preset frequency in a window suppression manner to obtain a filtered image.

4. The eyeball detection method according to claim 1, characterized in that: Determining the position of the eyeball according to the position of the scintillation point comprises: The position of the eyeball is determined according to the flashing point position of each light emitting element and the reprojection position of each light emitting element on the imaging plane.

5. The eyeball detection method according to claim 1 or 4, characterized in that: Also includes: The currently determined position of the eyeball is corrected according to the position of the eyeball determined at least once previously.

6. The eyeball detection method according to claim 1, characterized in that: The head mounted device further includes a rendering element for rendering a virtual scene; the method further includes: Determining the position of the gaze point in the virtual scene according to the position of the eyeball; The focus of the rendering element is adjusted according to the gaze point position.

7. An eyeball detection device, characterized in that: Applied to a head-mounted device, the head-mounted device is provided with an event camera and at least one light-emitting element, the device comprising: A blinking module, used to control the light emitting element to blink light toward the eyeball at a preset frequency, and obtain event information output by the event camera, wherein the event information is used to characterize the change of pixels on the imaging plane of the event camera; A filtering module, used for filtering the event information according to the preset frequency to obtain a target event, wherein the target event includes an event caused by the flashing light of the light-emitting element; A detection module, used to determine the position of the scintillation point of the light emitting element on the imaging plane according to the target event, and determine the position of the eyeball according to the position of the scintillation point; The flashing module is specifically used for: According to the pulse control sequence, the light-emitting duration of the light-emitting element is controlled in each flashing cycle corresponding to the preset frequency, wherein the pulse control sequence includes a plurality of pulse control units, each pulse control unit is used to control the light-emitting duration of at least one flashing cycle, and in the pulse control sequences of different light-emitting elements, the pulse control units at the same position are different; The filtering module is specifically used for: According to each light-emitting duration, retain the target event in each flashing cycle corresponding to the preset frequency; The detection module is specifically used for: The position of the scintillation point of each of the light emitting elements on the imaging plane is determined according to the target event in each of the at least one scintillation cycle and the pulse control sequence of each of the light emitting elements.

8. An electronic device, characterized in that: The device comprises a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to implement the method according to any one of claims 1 to 6 when executing the computer instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images