Eye tracking method, apparatus, and storage medium

Abstract

The application relates to the technical field of head-mounted smart devices, and discloses an eye movement tracking method and device and a storage medium. The method comprises the following steps: outputting an emission light signal to a target region of a wearer's eye in a detection time period in a detection period, and not outputting the emission light signal to the target region of the wearer's eye in a contrast time period; receiving a reflected light signal, and outputting a detection signal according to the reflected light signal, wherein the reflected light signal is formed by the wearer's eye reflecting the emission light signal; receiving a reference reflected light signal, and outputting a reference signal according to the reference reflected light signal, wherein the reference reflected light signal is formed by the wearer's eye reflecting ambient light; and acquiring an eye movement state of the wearer's eye in at least the detection time period according to the detection signal and the reference signal.

Description

[0001] This application claims priority to Chinese patent application No. 2025104043714, filed with the China National Intellectual Property Administration on April 1, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of head-mounted smart device technology, and more particularly to an eye-tracking method, device, and storage medium. Background Technology

[0003] The application of head-mounted smart devices is becoming more and more widespread. For example, AR (Augmented Reality) glasses can provide users with many convenient functions. When users interact with AR glasses, they can interact through eye movements. Users only need to look at virtual objects to select or control them, achieving a more natural and efficient interactive experience.

[0004] However, the eye-tracking detection function of existing head-mounted smart devices is greatly affected by the environment and is not suitable for different environments. In particular, the eye-tracking detection results may have some errors in strong light conditions. Summary of the Invention

[0005] This application provides an eye-tracking method, including: During the detection period, a light signal is emitted to the target area of ​​the wearer's eyes during the detection time period, and no light signal is emitted to the target area of ​​the wearer's eyes during the control time period. The detection time period and the control time period do not overlap. The device receives reflected light signals and outputs detection signals based on these signals. The reflected light signals are formed by the eye reflecting the emitted light signals. The system receives a reference reflected light signal and outputs a reference signal based on the reference reflected light signal, which is formed by the eye reflecting ambient light. The eye movement state of the wearer's eyes during at least the detection time period is obtained based on the detection signal and the reference signal.

[0006] Secondly, embodiments of this application also provide an eye-tracking device, including a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for implementing communication between the processor and the memory, wherein when the computer program is executed by the processor, it implements the steps of any eye-tracking method provided in embodiments of this application.

[0007] Thirdly, embodiments of this application also provide a storage medium storing a computer program, which, when executed by a processor, causes the processor to implement any of the eye-tracking methods provided in embodiments of this application.

[0008] Fourthly, embodiments of this application also provide a method for generating an eye-tracking algorithm model. The eye-tracking algorithm model is applied to a smart device worn on the wearer's head and includes an eye-tracking module. The eye-tracking module is configured to output emitted light signals to the wearer's eyes, receive reflected light signals formed by the reflected emitted light signals from the eyes, and generate detection signals. The method includes: A first initial model is obtained, which is used to generate eye movement states corresponding to the eyes based on the detection signals; The initial model is trained to obtain the eye-tracking algorithm model.

[0009] Fifthly, embodiments of this application also provide a method for generating a calibration algorithm model. The calibration algorithm model is applied to a smart device worn on the wearer's head and includes an eye-tracking module. The eye-tracking module is configured to output emitted light signals to the wearer's eyes, receive reflected light signals corresponding to at least one target area in the eyes, and generate a detection signal. The reflected light signals are formed by the reflected emitted light signals of the corresponding target area. The method includes: A second initial model is obtained, which is used to generate light intensity distribution information based on the detection signal. The light intensity distribution information includes the intensity of reflected light signal corresponding to at least one target area in the eye. When the wearer performs the target eye movement behavior, the eye movement module is used to acquire the detection signal sample corresponding to the target eye movement behavior; The second initial model is trained using training samples to obtain a calibration algorithm model corresponding to the wearer. The training samples include at least one detection signal sample corresponding to the target eye movement behavior and standard light intensity distribution information. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A modular structure diagram of a smart device provided for one or more embodiments of this application; Figure 2 A module structure diagram of an eye-tracking module provided for one or more embodiments of this application; Figure 3A schematic diagram illustrating the eye movement detection effect of an eye movement module provided in one or more embodiments of this application; Figure 4 A schematic diagram of the structure of an eye-tracking module provided for one or more embodiments of this application; Figure 5 A schematic diagram illustrating the signal intensity received by the light receiving module in an eye-tracking module, provided for one or more embodiments of this application; Figure 6 A schematic diagram of another eye-tracking module provided in one or more embodiments of this application; Figure 7 A schematic diagram illustrating the signal intensity received by the light receiving module in another eye-tracking module provided in one or more embodiments of this application; Figure 8 A schematic diagram illustrating the signal intensity received by the light receiving module in another eye-tracking module provided in one or more embodiments of this application; Figure 9 A schematic diagram illustrating the signal intensity received by the light receiving module in another eye-tracking module provided in one or more embodiments of this application; Figure 10 A schematic diagram illustrating the signal intensity received by the light receiving module in another eye-tracking module provided in one or more embodiments of this application; Figure 11 A schematic diagram illustrating the arrangement of time periods in an eye-tracking module, provided for one or more embodiments of this application; Figure 12 A schematic diagram illustrating the arrangement of light output components in an eye-tracking module, provided for one or more embodiments of this application; Figure 13 A schematic diagram of the target region arrangement of an eye-tracking module provided for one or more embodiments of this application; Figure 14 A modular structure for a smart device provided in one or more embodiments of this application; Figure 15 A module structure diagram of another smart device provided in one or more embodiments of this application; Figure 16 A schematic diagram of the structure of a smart device provided in one or more embodiments of this application; Figure 17 This is a schematic diagram of the target region arrangement of another eye-tracking module provided in one or more embodiments of this application; Figure 18 A schematic diagram of the signal intensity received by the optical receiving module in another eye-tracking module provided in one or more embodiments of this application; Figure 19 A schematic diagram of the signal intensity received by the optical receiving module in another eye-tracking module provided in one or more embodiments of this application; Figure 20 A flowchart illustrating the steps of an eye-tracking method provided in one or more embodiments of this application; Figure 21 A schematic diagram illustrating ambient light illuminating a wearer's eyes, provided for one or more embodiments of this application; Figure 22 A module structure diagram of an eye-tracking device provided in one or more embodiments of this application; Figure 23 A flowchart illustrating the steps of an eye-tracking calibration method provided in one or more embodiments of this application; Figure 24 A flowchart illustrating the steps for establishing and applying an eye-tracking algorithm model, provided for one or more embodiments of this application; Figure 25 A flowchart illustrating the steps of the eye-tracking algorithm preprocessing model provided in one or more embodiments of this application; Figure 26 A schematic diagram of the verification steps for an eye-tracking algorithm model provided in one or more embodiments of this application; Figure 27 A schematic flowchart illustrating the steps for establishing a pre-calibration model provided in one or more embodiments of this application; Figure 28 A schematic diagram illustrating the steps for establishing and optimizing the calibration algorithm model provided in one or more embodiments of this application; Figure 29 A schematic diagram illustrating the application process of the calibration algorithm model provided in one or more embodiments of this application; Figure 30 A flowchart illustrating the steps of an eye-tracking algorithm model generation method provided in one or more embodiments of this application; Figure 31 A schematic diagram of the model structure of a first initial model provided for one or more embodiments of this application; Figure 32 A schematic diagram of calibration points displayed by the display module in a smart device provided in one or more embodiments of this application; Figure 33 A schematic diagram illustrating the iterative training of the eye-tracking algorithm model generation method provided in one or more embodiments of this application; Figure 34 A schematic diagram of the eye movement error corresponding to each training sample in the eye movement tracking algorithm model generation method provided in one or more embodiments of this application; Figure 35 A schematic diagram illustrating the change of eye movement error during the training process in an eye-tracking algorithm model provided in one or more embodiments of this application; Figure 36A flowchart illustrating the steps of the calibration algorithm model generation method provided in one or more embodiments of this application; Figure 37 This is a flowchart illustrating the steps of an eye-tracking algorithm model generation method provided in one or more embodiments of this application.

[0012] Figure label: 1. Smart devices; 10. Eye-tracking module; 11. Light output module; 12. Light output component; 121. First light output component; 122. Second light output component; 123. Third light output component; 124. Light emitter; 125. First optical element; 13. Light receiving module; 131. Light sensor; 132. Second optical element; 133. First light sensor; 134. Second light sensor; 14. First eye-tracking module; 15. Second eye-tracking module; 20. Controller; 30. Display module; 40. Target area; 41. First target area; 42. Second target area; 43. Third target area; 50. Wearing bracket; 51. Eyeglass frame; 52. Front frame; 521. Display mounting part; 522. Temple connection part; 523. Receiving chamber; 53. Temple piece; 60. Projection module; T1, first testing cycle; T2, second testing cycle. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Unless otherwise specified, the embodiments of this application can be combined with each other. Other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, combined, or partially merged; therefore, the actual execution order may change according to the actual situation. The above descriptions are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0014] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0015] It should be noted that the eye-tracking method provided in this application embodiment can be applied to, but is not limited to, the eye-tracking module 10 and the smart device 1. The eye-tracking module 10 will be described in detail below: Please see Figures 1 to 4 , Figure 1 This is a module structure diagram of a smart device provided for one or more embodiments of this application. Figure 2 This application provides a module structure diagram of an eye-tracking module according to one or more embodiments. Figure 3 This is a schematic diagram illustrating the eye movement detection effect of an eye movement module provided in one or more embodiments of this application. Figure 4 This is a schematic diagram of the structure of an eye-tracking module provided for one or more embodiments of this application.

[0016] like Figures 1 to 4 As shown, this application embodiment provides an eye-tracking module 10 for a smart device 1. The smart device 1 is wearable on the wearer's head. The smart device 1 can be a head-mounted display device, such as a VR (Virtual Reality) display device, an AR (Augmented Reality) display device, an XR (Extender Reality) display device, or a MR (Mixed Reality) display device, or other wearable devices. By way of example and not limitation, in one embodiment, the head-mounted display device can be smart glasses, such as VR glasses, AR glasses, XR glasses, MR glasses, etc.

[0017] It should be noted that during use, the smart device 1 can monitor the wearer's eye movement state, namely the wearer's gaze direction and gaze target, through the eye-tracking module 10, and then perform corresponding operations based on the wearer's eye movement state to realize intelligent interaction between the smart device 1 and the wearer, thereby improving the wearer's user experience.

[0018] For example, the smart glasses are equipped with a display module 30 that corresponds to the wearer's settings. The display module 30 is configured to output an image screen. The wearer can select or manipulate the virtual object by looking at it in the image screen, thus achieving a more natural and efficient interactive experience. The above-mentioned interaction method has wide applications in scenarios such as games and education.

[0019] In some embodiments, in addition to detecting the wearer's eye movement status through the eye-tracking module 10, the smart device 1 can also monitor the wearer's eye health based on the eye movement status, helping to detect problems such as dry eye syndrome and eye fatigue.

[0020] The eye-tracking module 10 in this embodiment includes a light output module 11 and a light receiving module 13. The structure and function of the light output module 11 and the light receiving module 13 will be described in detail below.

[0021] Specifically, the light output module 11 includes a light output component 12, which can be used to output emitted light signals. The light output module 11 is configured to output emitted light signals to different target areas 40 of the same eye of the wearer through the light output component 12 during the detection period. After the emitted light signal is directed to the wearer's eye, the wearer's eye will emit the emitted light signal, thereby forming a reflected light signal.

[0022] The light receiving module 13 is configured to receive the reflected light signal formed by the light emitted by the eye and output a detection signal.

[0023] It should be noted that the detection signal output by the light receiving module 13 is matched with the reflected light signal. Based on the detection signal, optical features related to the reflected light signal can be identified, such as the signal intensity (light intensity) of the reflected light signal. Thus, the motion state of the wearer's eyes during the detection period can be further analyzed based on the reflected light signal.

[0024] To further explain, due to the differences in light reflectivity among different areas of the human eyeball (such as the pupil, iris, and sclera), the direction and state of eye movement can be inferred by measuring the dynamic changes in reflected light intensity. The pupil absorbs most light and has extremely low reflectivity (approximately 4%), thus appearing as a dark area under infrared light. The reflectivity of the iris and sclera increases sequentially (the iris is approximately 10%-20%, and the sclera is even higher), with the iris's reflectivity depending on its color (dark irises have even lower reflectivity). The sclera's reflectivity can reach 30-40%, appearing as a bright area under infrared light. Therefore, when the eyeball moves, the changes in the relative positions of the pupil, iris, and sclera within the eye socket lead to changes in the distribution of reflected light intensity.

[0025] Specifically, when the eyeball moves, the relative positions of the pupil, iris, and sclera within the eye socket change, causing dynamic changes in the intensity of reflected light received by the infrared sensor. On the other hand, changes in the light intensity received by the eyeball and the gaze target also cause the pupil to dilate and contract, leading to a relative decrease or increase in the size of the iris. Therefore, a combination of one or more of these factors results in changes in the relative area of ​​the pupil, iris, and sclera within the target area 40, thereby causing changes in the reflected light signal. During the detection period, the light output component 12 emits light signals to different target areas 40 of the same eye at different time intervals. These emitted light signals are reflected by the wearer's eye, forming corresponding reflected light signals that are collected by the light receiving module 13. The light receiving module 13 then outputs a corresponding detection signal, which can be used to determine the relative relationship between the pupil and different target areas 40. For example, the detection signal can be used to determine whether there is an overlap between the pupil and a certain target area 40, thus determining the pupil's position information. By tracking changes in the pupil's position, the wearer's eye movement state can be obtained.

[0026] like Figure 5 As shown, in one embodiment, during the detection period, the light output module 11 outputs a light signal to the right side of the wearer's eyeball in the first time period and outputs a light signal to the left side of the wearer's eyeball in the second time period. If the wearer's eyeball turns horizontally to the right during the detection period: the pupil, iris, and sclera move to the right, and the right half of the sclera is partially or completely covered by the eye socket. The iris, pupil, and sclera deflect to the right relative to the eye socket, and the right half or all of the sclera is hidden in the eye socket. The light intensity received by the light receiving module 13 in the first time period decreases. The left half of the iris and sclera is exposed, and the light intensity received by the light receiving module 13 in the second time period increases.

[0027] For example, during the detection period, the light output module 11 emits a light signal to the upper side of the wearer's eyeball in the first time period and to the lower side of the wearer's eyeball in the second time period. If the wearer's eyeball turns upward during the detection period, the iris, pupil, and sclera deflect upward relative to the eye socket, and part or all of the sclera is blocked by the eye socket. Therefore, the light intensity received by the light receiving module 13 decreases in the first time period and increases in the second time period. When the wearer's eyeball turns downward, the pupil, iris, and sclera deflect downward relative to the eye socket, and part or all of the lower sclera is blocked by the eye socket. Therefore, the light intensity received by the light receiving module 13 increases in the first time period and decreases in the second time period.

[0028] Compared to related technologies that use multiple light output components 12 to continuously output emitted light signals to different target areas 40 of the eye, the eye-tracking module 1 provided in this application provides emitted light signals to different target areas 40 of the same eye of the wearer in a time-division multiplexing manner. This effectively reduces power consumption and can identify the wearer's eye movement state based on the corresponding emitted light signals and detection signals, thereby achieving eye tracking of the wearer.

[0029] In some embodiments, the light output module 11 includes at least a first light output component 121 and a second light output component 122. The first light output component 121 is configured to output a first emitted light signal to a first target region 41, and the second light output component 122 is configured to output a second emitted light signal to a second target region 42. The first target region 41 and the second target region 42 are different positions of the same eye of the wearer, and the first target region 41 and the second target region 42 do not overlap at least partially.

[0030] Specifically, the first light output component 121 and the second light output component 122 have different light output angles and / or are located in different positions.

[0031] As one embodiment, the first light output component 121 and the second light output component 122 have different light output angles, and the first light output component 121 and the second light output component 122 are configured to emit light to the wearer's eyes from different directions.

[0032] For example, the application of eye-tracking module 10 to smart device 1 is described. Smart device 1 includes wearable bracket 50 and the aforementioned eye-tracking module 10. The first light output component 121 and the second light output component 122 of eye-tracking module 10 are disposed on wearable bracket 50. When the wearer wears smart device 1 through wearable bracket 50, the first light output component 121 and the second light output component 122 are located at different positions of the wearer's same eye and are configured to output emitted light signals to different target areas 40 of the eye.

[0033] like Figures 5 to 10 As shown, in some embodiments, within the same detection cycle, the first optical output component 121 is configured to output a first emitted optical signal in a first time period, and the second optical output component 122 is configured to output a second emitted optical signal in a second time period. The first and second time periods do not overlap. It should be noted that the lengths of the first and second time periods can be the same or different. Furthermore, the lengths of the individual first time periods can be the same or different; similarly, the lengths of the individual second time periods can be the same or different.

[0034] Specifically, the first optical output component 121 outputs a first emitted optical signal during a first time period within the detection cycle, and the second optical output component 122 outputs a second emitted optical signal during a second time period within the detection cycle.

[0035] The eye-tracking module 10 provided in this application provides time-division multiplexing to output emitted light signals to different target areas 40 of the same eye of the wearer. Therefore, when processing the detection signals output by the light receiving module, it can effectively distinguish the information corresponding to different target areas 40 in the eye based on the first time period and the second time period, thereby realizing eye tracking of the wearer. Moreover, compared with related technologies, this application embodiment outputs emitted light signals by different light output components 12 in a time-division multiplexing manner. When performing eye tracking, it is no longer necessary for multiple light output components 12 to continuously output emitted light signals. The time-division multiplexing method effectively reduces the overall power consumption of the light output module 11. In particular, for the smart device 1 where the eye-tracking module 10 is located, the reduction in power consumption of the eye-tracking module 10 also makes the smart device 1 have better battery life.

[0036] like Figures 6 to 10 As shown, in one embodiment, the smart device 1 includes two eye-tracking modules 10, which correspond to the wearer's left and right eyes respectively, to detect the eye movement state of the wearer's left and right eyes. Specifically, the two modules are respectively set to correspond to the wearer's first and second eyes, and the two modules can be modules with the same configuration or modules with different configurations.

[0037] Specifically, when the eyeball moves, the relative positions of the pupil, iris, and sclera within the eye socket change. Alternatively, changes in the intensity of light received by the eyeball and the target being gazed at cause the pupil to dilate and constrict, all of which lead to dynamic changes in the intensity of reflected light received by the infrared sensor. For example, when the wearer's eyeball turns horizontally to the right (to clarify, the turning direction of the left and right eyeballs is consistent for a normal wearer; the same applies to turning to the left, upward, and downward): the pupil, iris, and sclera move to the right, and the right half of the sclera is partially or completely obscured by the eye socket, resulting in a decrease in the light intensity received by the light receiving module 13 of the right eye-tracking module 10; the left half of the iris / sclera is exposed, resulting in an increase in the light intensity received by the light receiving module 13 of the left eye-tracking module 10.

[0038] For example, during the detection period, the light output module 11 emits a light signal to the upper side of the wearer's eyeball in the first time period and to the lower side of the wearer's eyeball in the second time period. If the wearer's eyeball turns upward during the detection period, since the eyeball rotates as a whole, the iris, pupil, and sclera deflect upward relative to the eye socket, and the upper part or all of the sclera is hidden inside the eye socket. The light intensity received by the light receiving modules 13 of both eye movement modules 10 will decrease in the first time period and increase in the second time period. When the wearer's eyeball turns downward, the iris, pupil, and sclera deflect downward relative to the eye socket, and the lower part or all of the sclera is hidden inside the eye socket. The light intensity received by the light receiving modules 13 of both eye movement modules 10 will increase in the first time period and decrease in the second time period.

[0039] For example, if the wearer's pupils dilate during the detection period, increasing the pupillary area, the light intensity received by the light-receiving modules 13 of both eye-tracking modules 10 will decrease in both the first and second time periods. Conversely, if the wearer's pupils constrict during the detection period, decreasing the pupillary area, the light intensity received by the light-receiving modules 13 of both eye-tracking modules 10 will increase in both the first and second time periods. It is understandable that the pupils constrict when the wearer actively looks at something close or when the light intensity received by the eyes increases, and the pupils dilate when the light intensity received by the wearer's eyes decreases.

[0040] like Figure 6 As shown, in one embodiment, the eye-tracking module 10 on the left is the first eye-tracking module 14, and the eye-tracking module 10 on the right is the second eye-tracking module 15.

[0041] In the first eye-tracking module 14, the first target area 41 corresponding to the first light output component 121 is located on the upper left side of the left eye, and the second target area 42 corresponding to the second light output component 122 is located on the lower left side of the left eye; in the second eye-tracking module 15, the first target area 41 corresponding to the first light output component 121 is located on the upper right side of the left eye, and the second target area 42 corresponding to the second light output component 122 is located on the lower right side of the left eye, and when the wearer looks straight ahead horizontally, the pupils on the left and right sides do not coincide with the first target area 41 and the second target area 42.

[0042] like Figure 7 As shown, during the detection period, the eyeball turns to the upper left (the turning of the left and right eyeballs is consistent). The light intensity received in the first eye movement module 14 during the first time period decreases, while the light intensity received during the second time period remains unchanged. In the second eye movement module 15, the light intensity received during the first time period remains unchanged, and the light intensity received during the second time period remains unchanged.

[0043] like Figure 8As shown, during the detection period, the eyeball turns to the lower left. The light intensity received by the first eye movement module 14 in the first time period remains unchanged, while the light intensity received in the second time period decreases. The light intensity received by the second eye movement module 15 in the first time period remains unchanged, while the light intensity received in the second time period remains unchanged.

[0044] like Figure 9 As shown, during the detection period, the eyeball turns to the upper right. The light intensity received by the second eye movement module 15 in the first time period decreases, while the light intensity received in the second time period remains unchanged. The light intensity received by the first eye movement module 14 in the first time period remains unchanged, while the light intensity received in the second time period remains unchanged.

[0045] like Figure 10 As shown, during the detection period, the eyeball turns to the right and downward. The light intensity received by the second eye movement module 15 in the first time period remains unchanged, while the light intensity received in the second time period decreases. The light intensity received by the first eye movement module 14 in the first time period remains unchanged, while the light intensity received in the second time period remains unchanged.

[0046] Therefore, based on the two eye-tracking modules 10 that are set up to correspond to the wearer's left and right eyes, the eye movement state of the wearer's eyeballs can be accurately and completely detected during the detection period.

[0047] like Figure 12 As shown, in one embodiment, the smart device 1 also includes a display module 30, which is set to correspond to the wearer's eyes and configured to display images to the wearer's eyes. The first light output component 121 and the second light output component 122 are located on different sides of the display module 30. When the wearer wears the smart device 1 through the wearable bracket 50, the first light output component 121 is located on the upper side of the display module 30, that is, above the wearer's eyeballs, and the second light output component 122 is located on the upper side of the display module 30, that is, below the wearer's eyeballs, so as to output emitted light signals to different target areas 40 of the eyes from different angles. The wearable bracket 50 is, for example, an eyeglass bracket 51, and the display module 30 is, for example, a display lens or a display screen.

[0048] As one embodiment, the light receiving module 13 is disposed between the first light output component 121 and the second light output component 122. For example, the light receiving module 13 includes one or more light sensors 131. Taking only one light sensor 131 as an example, the light sensor 131 is disposed between the first light output component 121 and the second light output component 122 and is configured to receive a first reflected light signal and a second reflected light signal, and output a first detection signal and a second detection signal.

[0049] In some embodiments, the light output module 11 further includes a greater number of light output components 12, and different light output components 12 have different light emission angles. These light output components 12 are configured to output emitted light signals to different target areas 40 of the same eye of the wearer in a time-division manner. The number of light output components 12 can be set according to actual needs.

[0050] like Figure 13 As shown, when the light output module 11 has three or more light output components 12, adjacent corresponding target areas may not overlap or may partially overlap.

[0051] Furthermore, the optical output module 11 also includes a greater number of optical output components 12, such as a third optical output component 123 that is different from the first optical output component 121 and the second optical output component 122.

[0052] like Figure 12 As shown, the light receiving module 13 includes at least a light sensor 131. For example, when the light output module 11 has only two light output components 12, these two light output components 12 can be symmetrically distributed about the light sensor 131 as an axis. Figure 12 The arrangement of the first optical output component 121 and the second optical output component 122 shown in (a).

[0053] When the light output module 11 is provided with multiple light output components 12, these multiple light output components 12 can be distributed circumferentially around the light sensor 131, for example... Figure 12 The first light output component 121, the second light output component 122 and the third light output component 112 shown in (b) are arranged in a triangle with the light sensor 131 as the center, so as to output corresponding emitted light signals to the same eye of the wearer from different angles, thereby obtaining a more comprehensive and accurate eye movement detection effect.

[0054] It should be noted that the symmetrical distribution, circumferential distribution, and triangular distribution mentioned in this article are all positional descriptions based on the wearer's head from a lateral perspective.

[0055] In some embodiments, depending on the process, structure, or light output requirements, the individual light output components 12 may also be asymmetrically distributed, which is not limited here.

[0056] In some embodiments, the light receiving module 13 is configured to receive a first reflected light signal and output a first detection signal, wherein the first reflected light signal is formed by the eye reflecting a first emitted light signal; The system receives the second reflected light signal and outputs the second detection signal. The second reflected light signal is formed by the eye reflecting the second emitted light signal.

[0057] It should be noted that in the eye-tracking module 10 provided in this application embodiment, the light receiving module 13 actually continuously receives light signals directed at it when it is working, which includes reflected light signals reflected back from each target area 40 in the glasses; the light receiving module 13 is also used to output corresponding detection signals based on the received reflected light signals, the detection signals include a first detection signal corresponding to the first reflected light signal and a second detection signal corresponding to the second reflected light signal.

[0058] like Figure 3 As shown in (a), in some embodiments, the optical receiving module 13 includes an optical sensor 131, and the first detection signal and the second detection signal are output by the same optical sensor 131.

[0059] Specifically, the optical sensor 131 is used to receive a first reflected light signal and output a first detection signal corresponding to the first reflected light signal, and is also used to receive a second reflected light signal and output a second detection signal corresponding to the second reflected light signal.

[0060] As one embodiment, the light output module 11 includes a first light output component 121 and a second light output component 122, and the light receiving module 13 may include only one light sensor 131. Both the first reflected light signal and the second reflected light signal are configured to be received by the light sensor 131. After receiving the first reflected light signal, the light sensor 131 outputs a first detection signal corresponding to the first reflected light signal, and after receiving the second reflected light signal, it outputs a second detection signal corresponding to the second reflected light signal.

[0061] For example, the eye-tracking module 10 is applied to a smart device 1 (smart glasses). The smart device 1 includes a glasses frame 51 and the aforementioned eye-tracking module 10. The first light output component 121 and the second light output component 122 are located at different positions of the same eye of the wearer and are configured to output emitted light signals to different target areas 40 of the eye. When the wearer wears the smart device 1 through the glasses frame 51, the first light output component 121 is located above the wearer's eye, the second light output component 122 is located below the wearer's eye, and the light sensor 131 is located between the first light output component 121 and the second light output component 122. The first reflected light signal and the second reflected light signal formed by the wearer's eye can both be received by the light sensor 131.

[0062] In some embodiments, the first light output component 121 and the second light output component 122 are symmetrically distributed about the light sensor 131 as the axis.

[0063] For example, when the light output module 11 has only two light output components 12, the two light output components 12 can be symmetrically distributed with respect to the light sensor 131 as the axis (it can be symmetrically distributed with respect to the horizontal axis passing through the light sensor 131, or with respect to the vertical axis passing through the light sensor 131, or with respect to the inclined axis passing through the light sensor 131). The two light output components 12 are symmetrically distributed about a horizontal axis passing through the light sensor 131, as can be seen from, for example... Figure 12 The arrangement of the first light output component 121 and the second light output component 122 is shown in (a). When the light output module 11 has multiple light output components 12, these multiple light output components 12 can be distributed circumferentially around the light sensor 131, for example... Figure 12 The first light output component 121, the second light output component 122 and the third light output component 112 shown in (b) are arranged in a triangle with the light sensor 131 as the center.

[0064] Thus, through clever positioning design, a single optical sensor 131 can receive at least two kinds of reflected light signals to output corresponding detection signals, and the first detection signal and the second detection signal can be distinguished from the detection signals output by the optical sensor 131 according to the first time period corresponding to the first emitted light signal and the second time period corresponding to the second emitted light signal.

[0065] Compared to the related technologies that set up a light sensor 131 for each light output component 12, this application only needs to set up one light sensor 131 in the light receiving module 13 to receive the reflected light signals corresponding to at least two different emitted light signals. This not only reduces the power consumption of the eye-tracking module 10, but also optimizes the overall size and weight of the eye-tracking module 10. In particular, for the smart device 1 where the eye-tracking module 10 is located, the reduction in power consumption of the eye-tracking module 10 also makes the smart device 1 have better battery life.

[0066] In other embodiments, the light receiving module 13 may include two or more light sensors 131, and each light sensor 131 is configured to receive a reflected light signal obtained from the emitted light signal reflected by the eye.

[0067] like Figure 3 As shown in (b) in some embodiments, the optical receiving module 13 includes at least two optical sensors 131, and the first detection signal and the second detection signal are output by the corresponding optical sensors 131 respectively.

[0068] Specifically, the light sensor 131 includes a first light sensor 133 and a second light sensor 134, wherein the first light sensor 133 and the second light sensor 134 are different light sensors 131, and the reflected light signals received by the different light sensors 131 are reflected light signals formed by the eye reflecting different emitted light signals. Moreover, the reflected light signal received by one light sensor 131 can be a reflected light signal formed by the eye reflecting one, two or more kinds of emitted light signals.

[0069] Specifically, the first optical sensor 133 is configured to receive a first reflected light signal and output a first detection signal, and the second optical sensor 134 is configured to receive a second reflected light signal and output a second detection signal.

[0070] In some other embodiments, the optical receiving module includes at least two optical sensors 131, both of which are capable of outputting a first detection signal and a second detection signal.

[0071] For example, the light sensor 131 includes at least two different light sensors, a first light sensor 133 and a second light sensor 134. Both the first light sensor 133 and the second light sensor 134 can output a first detection signal and a second detection signal. The controller 20 can receive and process the first and second detection signals output by the first light sensor 133 and the second light sensor 134. Specifically, the controller 20 can compare or average the detection signals output by the first light sensor 133 and the second light sensor 134, etc., without limitation. As one embodiment, the first light sensor 133 is used to output the first detection signal, and the second light sensor 134 is used to output the second detection signal. As one embodiment, the first light sensor 133 and the second light sensor 134 can simultaneously output the first detection signal and / or simultaneously output the second detection signal.

[0072] For example, both the first light sensor 133 and the second light sensor 134 can be infrared sensors.

[0073] In one embodiment, the first light sensor 133 is positioned corresponding to the first light output component 121, and the second light sensor 134 is positioned corresponding to the first light output component 121.

[0074] For example, along a horizontal axis passing through the center of the wearer's eye, the first light sensor 133 is disposed on the same side as the first light output component 121, and the second light sensor 134 is disposed on the same side as the second light output component 122.

[0075] like Figure 4As shown, in some embodiments, the light output component 12 includes a light emitter 124 and a first optical element 125. The light emitter 124 is used to output emitted light signals. The first optical element 125 is set to correspond to the light output direction of the light emitter 124 and is used to process the emitted light signal output by the light emitter 124 by any one or more of the following methods: convergence, reflection, divergence or refraction, and then direct it toward the target area 40, so that the light can uniformly illuminate the eyeball and avoid local high brightness reflection.

[0076] For example, the light emitter 124 can be a divergent light source or a parallel light source.

[0077] For example, the first optical element 125 includes one or a combination of a prism, a mirror, a lens, a diffuser, an optical coating, and wherein the lens is, for example, one or a combination of a biconvex lens, a plano-convex lens, a convex aspherical lens, and a convex freeform lens.

[0078] To further explain, the light output component 12 may include a combination of a diverging light source and a lens, or a combination of a parallel light source and a prism / reflector / diffuser.

[0079] For example, in some embodiments, the first optical element 125 is a lens. The lens focuses the emitted light signal output from the light emitter 124, making the emitted light signal more concentrated. Especially when the light emitter 124 is a divergent light source, the lens can better converge the divergent emitted light signal into a state closer to parallel light, and directionally output it to the wearer's eyes. Moreover, the directional light output does not require illuminating the entire eye to obtain a relatively clear and reliable reflected light signal, which is convenient for the light receiving module 13 to receive the reflected light signal. In addition, the directional light output allows the light emitter 124 to operate with lower power consumption, improving the overall working efficiency of the eye-tracking module 10.

[0080] In some embodiments, the light output module 11 may be provided with a first optical element 125 to achieve light distribution control, or the light output surface of the light output module 11 may be directly changed to achieve a similar effect as having a first optical element 125.

[0081] In some embodiments, the light output component 12 may be installed at any suitable position on the eye-tracking module 10, and / or the light output component 12 may output emitted light signals at any suitable angle, thereby enabling the emitted light signals to be directly directed to various target areas of the wearer's eyes. Preferably, the light output component 12 may be installed at an angle relative to the display module 30. By adjusting the installation position and angle of the light output component 12, the incident angle of the emitted light signal relative to the eyeball can be adjusted, further simplifying the structure of the eye-tracking module 10 and improving the lightweight nature of the eye-tracking module 10.

[0082] like Figure 4 As shown, in some embodiments, the light receiving module 13 further includes a second optical element 132, which is disposed in the direction corresponding to the light sensor 131 receiving the reflected light signal. The second optical element 132 is configured to process the reflected light signal formed by the wearer's eye in any one or more ways, such as convergence, reflection, divergence or refraction, and then direct it toward the light sensor 131.

[0083] As one embodiment, the second optical element 132 includes one or a combination of a prism, a mirror, a lens, a diffuser, and an optical coating.

[0084] For example, the second optical element 132 includes a convex lens for converging the reflected light signal, thereby concentrating the intensity of the processed reflected light signal and improving the detection effect of the light sensor 131 on the reflected light signal. Furthermore, the second optical element 132 may also be provided with a filter / optical coating to reduce the influence of other reflected light besides the reflected light signal on the light sensor 131, thereby improving the anti-interference capability of the eye-tracking module 1.

[0085] In some embodiments, the light output component 12 is an infrared light emitter, and the light receiving module 13 includes an infrared light sensor, detecting an electrical signal. Infrared light is invisible to the human eye, thus avoiding interference with visual tasks, and it can penetrate the cornea, reducing corneal reflection interference.

[0086] Specifically, an infrared light emitter is used as the light output component 12, and the emitted light signal it outputs is infrared light. The reflected light signal formed by the wearer's eyes reflecting the emitted light signal is also infrared light. The infrared light sensor in the light receiving module 13 can receive this reflected light signal and output a detection signal as an electrical signal.

[0087] It should be noted that the infrared light sensor in this embodiment may be the same as the light sensor 131 in the aforementioned embodiment.

[0088] In some embodiments, the light emitter 124 is a divergent light source, such as an infrared Vcsel (Vertical-Cavity Surface-Emitting Laser), and the first optical element 125 is a prism; the infrared light emitted by the infrared Vcsel is refracted to the target area after passing through the prism.

[0089] In some embodiments, the light receiving module 13 is configured to receive and sense the light intensity of the reflected light signal in order to output a detection signal.

[0090] In some implementations, the detection sampling frequency of the optical receiving module 13 is 25-50 Hz, 60-200 Hz, 250-500 Hz, or 600-1200 Hz.

[0091] Specifically, the intensity of the detection signal is matched with the intensity of the reflected light signal. When processing the detection signal, the intensity of the reflected light signal can be identified based on the detection signal, and the intensity of the reflected light signal can be used to determine whether the pupil coincides with the corresponding target area 40, so as to determine the position information of the pupil. Thus, the wearer's eye movement state can be obtained by tracking the position change of the pupil.

[0092] For example, the first detection signal and the second detection signal in the detection signal can be determined based on the time distribution of the first time period and the second time period within the detection cycle. Then, the light intensity of the first reflected light signal can be identified based on the first detection signal, and the light intensity of the second reflected light signal can be identified based on the second detection signal. Then, the wearer's pupil can be determined to have rotated to the first target area 41 within the detection cycle based on the light intensity of the first reflected light signal, and the wearer's pupil can be determined to have rotated to the first target area 41 within the detection cycle based on the light intensity of the second reflected light signal.

[0093] The following is a detailed description of smart device 1: Please see Figures 14 to 16 , Figure 14 This is a module structure diagram of another smart device provided in one or more embodiments of this application. Figure 15 A module structure diagram of yet another smart device provided in one or more embodiments of this application. Figure 16 This is a module structure diagram of another smart device provided in one or more embodiments of this application.

[0094] like Figures 1 to 16 As shown in the embodiments of this application, a smart device 1 is also provided, including any eye-tracking module 10 as provided in the embodiments of this application. The smart device 1 can be smart glasses, such as VR glasses, AR glasses, XR glasses, MR glasses, etc.

[0095] It should be noted that during use, the smart device 1 can monitor the wearer's eye movement state, namely the wearer's gaze direction and gaze target, through the eye-tracking module 10, and then perform corresponding operations based on the wearer's eye movement state to realize intelligent interaction between the smart device 1 and the wearer, thereby improving the wearer's user experience.

[0096] For example, the smart glasses are equipped with a display module 30 that corresponds to the wearer's settings. The display module 30 is configured to display images to the wearer's eyes. The wearer can select or manipulate virtual objects by looking at them in the images, achieving a more natural and efficient interactive experience. The above-mentioned interaction method has wide applications in games, education and other scenarios.

[0097] Among them, the display module 30 corresponding to VR glasses, XR glasses or MR glasses includes a display screen, while the display module 30 corresponding to AR glasses includes display lenses, such as waveguide lenses, freeform surface lenses, holographic lenses, etc.

[0098] In some embodiments, in addition to detecting the wearer's eye movement status through the eye-tracking module 10, the smart device 1 can also monitor the wearer's eye health based on the eye movement status, helping to detect problems such as dry eye syndrome and eye fatigue.

[0099] As one embodiment, the smart device 1 includes two eye-tracking modules 10, which correspond to the wearer's left and right eyes respectively, to detect the eye movement status of the wearer's left and right eyes.

[0100] Specifically, when the eyeball moves, the relative positions of the pupil, iris, and sclera in the eye socket change, causing dynamic changes in the intensity of reflected light received by the infrared sensor.

[0101] In some embodiments, the smart device 1 is a smart glasses, which also includes a glasses frame 51, a display module 30 and a projection module 60. The eye-tracking module 10, the display module 30 and the projection module 60 are all disposed on the glasses frame 51.

[0102] Specifically, the projection module 60 is configured to provide image light associated with an image or information to the display module 30, and the display module 30 receives the image light and displays the corresponding image or information. Further explanation: smart glasses may be, for example, AR glasses, VR glasses, XR glasses, or MR glasses. The display module 30 for VR glasses, XR glasses, or MR glasses includes a display screen, while the display module 30 for AR glasses includes display lenses, such as waveguide lenses, freeform lenses, holographic lenses, etc.

[0103] In some embodiments, the eyeglasses frame 51 includes a front frame 52 and temple members 53. The front frame 52 includes opposing temple connecting portions 522 (also called posts), which are located on the side of the front frame 52 near the temple members 53, and the temple members 53 are connected to the front frame 52 via the temple connecting portions 522. By way of example and not limitation, two temple connecting portions 522 are located on the left and right sides of the front frame 52 and are respectively connected to two different temple members 53. For example, when the smart device 1 is smart glasses, the wearable support 50 can be the eyeglasses frame 51.

[0104] The eye-tracking module 10 is disposed on the temple connection portion 522 of at least one side of the front frame 52.

[0105] In some embodiments, the eye-tracking module 10 includes a first eye-tracking module 14 and a second eye-tracking module 15, which are respectively disposed on different temple connection portions 522 on both sides of the front frame 52.

[0106] The first eye-tracking module 14 is configured to detect the eye movement state of the wearer's first eye and output a first eye detection signal, and the second eye-tracking module 15 is configured to detect the eye movement state of the wearer's second eye and output a second eye detection signal.

[0107] Specifically, the first eye and the second eye are the wearer's left and right eyes. The first eye movement module 14 and the second eye movement module 15 are different eye movement modules 10. They are set to correspond to the wearer's first eye and second eye respectively. The first eye detection signal is used to characterize the eye movement state of the first eye, and the second eye detection signal is used to characterize the eye movement state of the second eye.

[0108] As one embodiment, when the wearer wears the smart device, the front frame 52 is located in front of the wearer's head, the temples 53 are located on the left and right sides of the wearer's head, and the front frame 52 is adapted to the wearer's nose bridge, while the temples 53 are adapted to the wearer's ears.

[0109] As one embodiment, the front frame 52 further includes a display mounting portion 521, and temple connecting portions 522 are disposed on opposite sides of the display mounting portion 521. A display module 30, such as a display screen or a display lens, is connected to the display mounting portion 521, thereby positioning the display mounting portion 521 in front of the wearer's head when the wearer wears the smart device. By way of example and not limitation, the display module 30 is a display lens, the display mounting portion 521 is connected to at least one side of the display lens in the circumferential direction, and the display lens is snap-fitted to the display mounting portion 521. This snap-fitting method can be rigid or elastic.

[0110] Based on the eye-tracking module 10 disposed on the temple connection portion 522 on at least one side of the front frame 52, when the wearer wears the smart device 1 through the wear bracket 50, the eye-tracking module 10 is configured on the side of the front frame 52 to detect the wearer's eye movement state.

[0111] It should be noted that in the relevant technology, placing part or all of the structure of the eye-tracking module 10 in front of the wearer's eyes will obstruct the wearer's vision, increase the weight of the display mounting part 521, increase the burden on the bridge of the nose, and make it uncomfortable to wear. Moreover, improper operation when disassembling the eye-tracking module 10 during production line installation or maintenance may cause wear and tear on the display module 30, especially the display lens.

[0112] Compared to related technologies, the eye-tracking module 10 in the smart device 1 provided in this embodiment is located on the side, which reduces obstruction of the wearer's vision and burden on the bridge of the nose, thus improving the wearing experience. On the other hand, the side-mounted eye-tracking module 10 makes better use of the space in the smart device 1 and avoids wear and tear on the display lenses during installation and removal.

[0113] In some implementations, the eye-tracking module 10 is integrated with the projection module 60. As one embodiment, the eye-tracking module 10 and the projection module 60 are integrated on the same circuit board.

[0114] It should be understood that the above settings can effectively reduce the space occupied by the eye-tracking module 10 and the projection module 60, making the smart device 1 more portable.

[0115] In some embodiments, the temple connection 522 forms a receiving chamber 523, in which the eye-tracking module 10 and the projection module 60 are integrated and both housed within the receiving chamber 523. It should be understood that the above arrangement makes full use of the space of the smart device 1, making the smart device 1 more portable and reducing obstruction to the wearer's vision.

[0116] Furthermore, in some embodiments, the image light output by the projection module 60 is associated with the eye-tracking module 10. Therefore, integrating the eye-tracking module 10 with the projection module 60 also facilitates the arrangement of the electrical connection between the eye-tracking module 10 and the projection module 60.

[0117] As one embodiment, the eye-tracking module 10 and the projection module 60 are integrated on the same circuit board module. For example, the circuit board module includes at least one circuit board, and the eye-tracking module 10 and the projection module 60 are integrated on the same circuit board; or, for another example, the circuit board module includes at least two stacked circuit boards, and adjacent circuit boards are electrically connected, and the eye-tracking module 10 and the projection module 60 are disposed on different circuit boards on the same circuit board module.

[0118] In some implementations, the smart device 1 is configured to acquire the eye movement state of the wearer's eyes during the detection period based on the detection signal.

[0119] Specifically, based on the image displayed by the display module 30, the wearer can select or manipulate virtual objects by looking at them in the image. The wearer's eye movement state (i.e., gaze direction) is determined by the detection signal detected by the eye movement module 10.

[0120] like Figure 14 and Figure 15As shown, in some embodiments, the smart device 1 further includes a controller 20, which is used to control the operation of the eye-tracking module 10 in the smart device 1.

[0121] As one embodiment, the controller 20 may be housed in the receiving chamber 523, and further, the controller 20 may be integrated with the eye-tracking module 10 and the projection module 60.

[0122] It should be understood that integrating the eye-tracking module 10, projection module 60, and controller 20 within the accommodating chamber 523 also avoids exposing each module, thus providing effective protection for the eye-tracking module 10, projection module 60, and controller 20 through the temple connection portion 522. Furthermore, in some embodiments, the controller 20 is electrically connected to the projection module 60 and eye-tracking module 10 to transmit electrical signals; therefore, integrating the controller 20, eye-tracking module 10, and projection module 60 also facilitates the arrangement of electrical connections between the modules.

[0123] In another embodiment, the controller 20 is disposed on the temple 53. Preferably, the controller 20 is disposed on the side of the temple 53 near the front frame 54 so that the controller 20 can form an electrical connection with the projection module 60 and the eye-tracking module 10 to realize the transmission of electrical signals.

[0124] As one embodiment, the controller 20 is also used to control the operation of the display module 30 and the projection module 60.

[0125] In some implementations, the controller 20 is used to receive the detection signal output by the eye-tracking module 10 and adjust the image light output by the projection module 60 according to the detection signal.

[0126] Specifically, the controller 20 can identify the wearer's active eye movement behavior based on the detection signal and adjust the image light output by the projection module 60 accordingly, thereby enabling interaction between the wearer and the device; it can also identify changes in the wearer's pupil area based on the detection signal to adjust the image light output by the projection module 60 and the brightness of the screen in the display module 30 accordingly; it can also collect the wearer's current gaze direction based on the detection signal and adjust the image light output by the projection module 60 according to the current gaze direction to form corresponding guidance in the screen displayed on the display module 30, thereby instructing the wearer to gaze in the target direction.

[0127] For example, controller 20 includes one of the following: central processing unit (CPU), digital signal processor (DSP), application specific integrated circuit (ASIC), field-programmable gate array (FPGA), other general-purpose processor or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component.

[0128] In some implementations, the controller 20 is configured to: The first light output component 121 is controlled to output a first emitted light signal to the first target area 41, and the second light output component 122 is controlled to output a second emitted light signal to the second target area 42, wherein the first target area 41 and the second target area 42 are at least partially non-overlapping different areas of the same eye of the wearer.

[0129] Specifically, the first light output component 121 and the second light output component 122 have different light output angles and / or are located in different positions.

[0130] like Figures 7 to 10 As shown, and not limited to, for the left eye, the first target region 41 and the second target region 42 are the lower left and upper left sides, respectively, and the first target region 41 and the second target region 42 need to cover at least the left side and most of the middle area of ​​the left eye; for the right eye, the first target region 41 and the second target region 42 are the lower right and upper right sides, respectively, and the first target region 41 and the second target region 42 need to cover at least the right side and most of the middle area of ​​the right eye.

[0131] It is understandable that the first target area 41 and the second target area 42 are opposite sides of the eye, such as the upper side and the lower side, the left side and the right side, or the diagonal side.

[0132] It is understandable that, while ensuring detection accuracy, the first target region 41 and the second target region 42 may not overlap (e.g., Figure 3 (As shown); or, the first target area 41 and the second target area 42 partially overlap, thereby covering most of the eyeglasses, such as the sclera area (e.g. Figures 7 to 10 (As shown).

[0133] In some embodiments, the first target region 41 and the second target region 42 are respectively the upper and lower sides of the eye.

[0134] In some embodiments, a third target region may also be included. By way of example and not limitation, the first target region 41, the second target region 42, and the third target region 43 are the left, middle, and right sides of the eye, respectively.

[0135] As an example, and not a limitation, the sampling frequency of the optical receiving module is greater than 25 Hz. For example Figure 3 and Figure 6 As shown, in each eyeball, the first target area 41 is located above the pupil, and the second target area 42 is located below the pupil.

[0136] As one embodiment, the first light output component 121 and the second light output component 122 have different light output angles, and the first light output component 121 and the second light output component 122 are configured to emit light to the wearer's eyes from different directions.

[0137] As one embodiment, the smart device 1 also includes a display module 30, which is set to correspond to the wearer's eyes and configured to display images to the wearer's eyes. The first light output component 121 and the second light output component 122 are located on different sides of the display module 30. When the wearer wears the smart device 1 through the wear bracket 50, the first light output component 121 is located on the upper side of the display module 30, that is, above the wearer's eyeballs, and the second light output component 122 is located on the upper side of the display module 30, that is, below the wearer's eyeballs, so as to output emitted light signals to different target areas 40 of the eyes from different angles.

[0138] As one embodiment, the controller 20 is also used to control the smart device 1 or the eye-tracking module 10 to execute any of the eye-tracking methods provided in the embodiments of this application. The control method is described in detail below.

[0139] This application also provides an eye-tracking method. As one embodiment, the eye-tracking method can be applied to any eye-tracking module 10 or smart device 1 provided in the embodiments of this application, but is not limited to application to the eye-tracking module 10 and smart device 1 provided in the embodiments of this application.

[0140] As an example and not a limitation, the eye-tracking method provided in this application is applied to the eye-tracking module 10 provided in this application for illustration. The configuration of each component of the eye-tracking module 10 and the smart device 1 can be found in the specific descriptions of the eye-tracking module 10 and the smart device 1 provided in the foregoing embodiments, and will not be repeated here.

[0141] Please see Figure 20 , Figure 20 This is a flowchart illustrating the steps of an eye-tracking method provided in one or more embodiments of this application.

[0142] like Figure 20 As shown, specifically, the eye-tracking method includes steps S101 to S104: Step S101: During the detection period, a light signal is emitted to the target area of ​​the wearer's eyes during the detection time period, and no light signal is emitted to the target area of ​​the wearer's eyes during the control time period. The detection time period and the control time period do not overlap. Step S102: Receive the reflected light signal and output a detection signal based on the reflected light signal. The reflected light signal is formed by the light signal reflected by the eye. Step S103: Receive the reference reflected light signal and output a reference signal based on the reference reflected light signal. The reference reflected light signal is formed by the reflection of ambient light by the eye. Step S104: Obtain the eye movement state of the wearer's eyes at least during the detection period based on the detection signal and the reference signal.

[0143] It should be noted that the detection signal is matched with the reflected light signal. Based on the detection signal, optical features related to the reflected light signal can be identified, such as the signal intensity (light intensity) of the reflected light signal. This allows for further analysis of the wearer's eye movement state during the detection period based on the reflected light signal.

[0144] To further explain, due to the differences in light reflectivity among different areas of the human eye (such as the pupil, iris, and sclera), the direction and state of eye movement can be inferred by measuring the dynamic changes in reflected light intensity. The pupil absorbs most light and has extremely low reflectivity (approximately 4%), thus appearing as a dark area under infrared light. The reflectivity of the iris and sclera increases sequentially (the iris is approximately 10%-20%, and the sclera is even higher), with the iris's reflectivity depending on its color (dark irises have even lower reflectivity). The sclera's reflectivity can reach 30-40%, appearing as a bright area under infrared light. Therefore, when the eye moves, the change in the relative position of the pupil and the iris / sclera leads to a change in the distribution of reflected light intensity.

[0145] Specifically, when the eyeball moves, the relative positions of the pupil, iris, and sclera within the eye socket change, causing dynamic changes in the intensity of reflected light received by the infrared sensor. At different time points within the detection period, emitted light signals are output to different target areas 40 of the same eye. These emitted light signals are reflected by the wearer's eye, forming corresponding reflected light signals, which in turn generate corresponding detection signals. Based on these detection signals, the relative relationship between the pupil and different target areas 40 can be determined. For example, the detection signals can determine whether there is any overlap between the pupil and each target area 40, thus determining the pupil's positional information. Therefore, by tracking changes in the pupil's position, the wearer's eye movement state can be obtained.

[0146] It should be understood that the eye-tracking method provided in this application embodiment does not emit light during the control period in the detection cycle. Therefore, the reference signal generated by the reference reflected light signal in the control period can be used to optimize the detection signal generated by the reflected light signal in the detection period. At the algorithm level, the reference signal is used to cancel the influence of ambient light on the detection signal, so as to obtain the accurate eye movement state of the wearer's eyes in the detection period. Furthermore, the eye movement state in the detection period can be used as the eye movement state in the detection cycle.

[0147] Taking the eye-tracking method applied to the eye-tracking module 10 provided in this application embodiment as an example: all light output components 12 do not emit light during the control period. Thus, during the control period, only ambient light shines on the wearer's eyes. The ambient light is reflected by the wearer's eyes to form a reference reflected light signal. The light receiving module 13 receives the reference reflected light signal and outputs a reference signal based on the reference reflected light signal.

[0148] As an example and not a limitation, in the embodiments of this application, the control time period may be within a certain detection cycle, or it may exist within all detection cycles.

[0149] like Figure 11 As shown in (c) in the example, in one part of the detection period there is no control time period, only the detection time period (e.g., the first time period and the second time period); in another part of the detection period there are both control time period and detection time period.

[0150] It should be understood that the detection signal can also be obtained by comparing the detection signal with the reference signal within different detection cycles, especially when the reference signal is not present in some detection cycles. like Figure 11 As shown in (d) in the figure, as an example, the arrangement order of the detection time period and the control time period is the same in each detection cycle. For example, the arrangement order of the time periods in each detection cycle is: "Detection time period - Control time period - Detection time period".

[0151] In another embodiment, the order of the detection time period and the control time period is different in different detection cycles. For example, the order of the time periods in some detection cycles is "detection time period - control time period", while the order of the time periods in other detection cycles is "control time period - detection time period".

[0152] In some implementations, step S104 acquires the eye movement state of the wearer's eyes during the detection period based on the detection signal and the reference signal, including: The corresponding detection light intensity is determined based on the detection signal; Determine the reference light intensity corresponding to the ambient light based on the reference signal; The detected light intensity and the reference light intensity are differentially processed to obtain the target light intensity corresponding to the reflected light signal, and the eye movement state of the wearer's eyes during the detection time period is obtained based on the target light intensity.

[0153] The reference signal is matched with the ambient light, and optical features related to the ambient light can be identified based on the reference signal, such as the signal intensity (light intensity) of the ambient light. Optimizing the detection signal using the reference signal can counteract the influence of ambient light on the detection signal.

[0154] Specific combination Figure 21 To explain, the detected light intensity is the sum of the ambient light intensity (i.e., the reference light intensity) and the reflected light signal intensity. Therefore, the target light intensity obtained by differential processing of the detected light intensity and the reference light intensity can accurately reflect the relative relationship between the wearer's pupil and the target area 40 during the detection period.

[0155] In some implementations, the eye movement state of the wearer's eyes during the detection period is obtained based on the target light intensity, including: When the target light intensity does not exceed the preset intensity threshold, it is determined that the position of the wearer's pupil overlaps with the target area 40 during the detection period. When the target light intensity exceeds the preset intensity threshold, it is determined that the wearer's pupil position does not overlap with the target area 40 during the detection period.

[0156] Specifically, as described in the foregoing embodiments, due to the differences in light reflectivity of different areas of the human eyeball (such as the pupil, iris, and sclera), when the position of the wearer's pupil overlaps with the target area 40, the light intensity of the target light signal corresponding to the target area 40 will decrease significantly. In this embodiment, a preset intensity threshold can be used to quickly and accurately determine whether the position of the pupil overlaps with the target area 40.

[0157] It should be added that the preset intensity threshold value can be set according to the wearer's individual eye characteristics and eye habits.

[0158] In some implementations, the duration of the detection period is longer than the duration of the control period within the detection cycle.

[0159] In some embodiments, the duration of the data acquisition period is less than or equal to the duration of the detection period within the detection cycle.

[0160] Specifically, the purpose of setting the comparison time period is to detect the baseline value of the intensity of ambient light emitted by the wearer's eyes when no light signal is emitted towards the wearer's eyes. Increasing the proportion of the detection time period in the detection cycle increases the effective detection time, thereby improving the accuracy of eye movement detection.

[0161] In some implementations, step S101, which involves outputting an emitted light signal to the target area 40 of the wearer's eye during the detection period, includes: During the detection period, light signals are emitted to different target areas 40 of the same eye of the wearer at different times.

[0162] As explained above, when the eyeball moves, the relative positions of the pupil, iris, and sclera within the eye socket change, causing dynamic changes in the intensity of the received reflected light. During different time periods within the detection cycle, emitted light signals are output to different target areas 40 of the same eye. These emitted light signals are reflected by the wearer's eye, forming corresponding reflected light signals, which in turn generate corresponding detection signals. The relative relationship between the pupil and different target areas 40 can then be determined based on these detection signals. For example, the detection signals can determine whether there is an overlap between the pupil and a certain target area 40, thus determining the pupil's positional information. Therefore, by tracking changes in the pupil's position, the wearer's eye movement state can be obtained.

[0163] Compared to related technologies that use multiple light output components 12 to continuously output emitted light signals to different target areas 40 of the eye, this technology outputs emitted light signals to different target areas 40 of the same eye of the wearer in a time-division multiplexing manner, effectively reducing power consumption.

[0164] like Figure 5 As shown in the figure, in one embodiment, during the detection period, the light output module 11 outputs a light signal to the right side of the wearer's eyeball in the first time period and outputs a light signal to the left side of the wearer's eyeball in the second time period. If the wearer's eyeball turns horizontally to the right during the detection period: the pupil, iris, and sclera move to the right, and the right half of the sclera is partially or completely blocked by the eye socket, the light intensity received by the light receiving module 13 in the first time period decreases; the left half of the iris and sclera is exposed, and the light intensity received by the light receiving module 13 in the second time period increases.

[0165] For example, during the detection period, the light output module 11 emits a light signal to the upper side of the wearer's eyeball in the first time period and to the lower side of the wearer's eyeball in the second time period. If the wearer's eyeball turns upward during the detection period, due to the upward movement of the pupil, iris, and sclera, the upper half of the sclera is partially or completely blocked by the eye socket, resulting in a decrease in the light intensity received by the light receiving module 13 in the first time period and an increase in the light intensity detected in the second time period. When the wearer's eyeball turns downward, due to the downward movement of the pupil, iris, and sclera, the lower half of the sclera is partially or completely blocked by the eye socket, resulting in an increase in the light intensity received by the light receiving module 13 in the first time period and a decrease in the light intensity received in the second time period.

[0166] like Figures 6 to 10As shown, in some embodiments, the eye-tracking method further includes: determining the wearer's overall eye-tracking state based on the eye-tracking states of each of the wearer's eyes.

[0167] Specifically, when the eyeball moves, the relative positions of the pupil, iris, and sclera within the eye socket change, causing dynamic changes in the intensity of reflected light received by the infrared sensor. For example, when the wearer's eyeball turns horizontally to the right (to clarify, the turning direction of the left and right eyeballs is consistent in a normal wearer, and the same applies to turning to the left, upward, and downward): the pupil, iris, and sclera move to the right, and the right half of the sclera is partially or completely obscured by the eye socket, resulting in a decrease in the light intensity received by the light receiving module 13 of the right eye movement module 10; the left half of the iris / sclera is exposed, resulting in an increase in the light intensity received by the light receiving module 13 of the left eye movement module 10.

[0168] For example, during the detection period, the light output module 11 emits a light signal to the upper side of the wearer's eyeball in the first time period and to the lower side of the wearer's eyeball in the second time period. If the wearer's eyeball turns upward during the detection period, the upper sclera is blocked by the eye socket due to the upward deflection of the pupil, iris, and sclera. Therefore, the light intensity received by the light receiving modules 13 of both eye-tracking modules 10 will decrease in the first time period and increase in the second time period. When the wearer's eyeball turns downward, the lower sclera is partially or completely blocked by the eye socket due to the downward deflection of the pupil, iris, and sclera. Therefore, the light intensity received by the light receiving modules 13 of both eye-tracking modules 10 will increase in the first time period and decrease in the second time period.

[0169] In some implementations, emitted light signals are output to different target areas 40 of the same eye of the wearer at different times during the detection period, including: Within the same detection cycle, a first emitted light signal is output to the first target area 41 in the first time period, and a second emitted light signal is output to the second target area 42 in the second time period; Among them, the first target area 41 and the second target area 42 are different positions of the same eye of the wearer, and the first time period and the second time period do not overlap.

[0170] like Figures 7 to 10 As shown, and not limited to, for the left eye, the first target region 41 and the second target region 42 are the lower left and upper left sides, respectively, and the first target region 41 and the second target region 42 need to cover at least the left side and most of the middle area of ​​the left eye; for the right eye, the first target region 41 and the second target region 42 are the lower right and upper right sides, respectively, and the first target region 41 and the second target region 42 need to cover at least the right side and most of the middle area of ​​the right eye.

[0171] It is understandable that the first target area 41 and the second target area 42 are opposite sides of the eye, such as the upper side and the lower side, the left side and the right side, or the diagonal side.

[0172] It is understandable that, while ensuring detection accuracy, the first target region 41 and the second target region 42 may not overlap (e.g., Figure 3 (As shown); or, the first target area 41 and the second target area 42 partially overlap, thereby covering most of the eyeglasses, such as the sclera area (e.g. Figures 7 to 10 (As shown).

[0173] In some embodiments, the first target region 41 and the second target region 42 are respectively the upper and lower sides of the eye.

[0174] In some embodiments, a third target region may also be included. By way of example and not limitation, the first target region 41, the second target region 42, and the third target region 43 are the left, middle, and right sides of the eye, respectively.

[0175] For example Figure 3 and Figure 6 As shown, in each eyeball, the first target area 41 is located above the pupil, and the second target area 42 is located below the pupil.

[0176] This embodiment of the application outputs emitted light signals to different target areas 40 of the same eye of the wearer in a time-division manner. Therefore, when processing the detection signal output by the light receiving module, it is possible to effectively distinguish the information corresponding to different target areas 40 in the eye based on the first time period and the second time period, thereby realizing eye tracking of the wearer. Moreover, compared with related technologies, the time-division output of emitted light signals eliminates the need for multiple light output components 12 to continuously output emitted light signals during eye tracking. The time-division multiplexing method effectively reduces the power consumption of eye tracking and improves the battery life of the corresponding device.

[0177] like Figure 11 As shown in (a) of the figure, as an example, the first time period, the second time period and the control time period are arranged in the same order in each detection cycle. For example, the time period arrangement order in each detection cycle is: "first time period - second time period - control time period".

[0178] like Figure 11 As shown in (b) in another embodiment, the order of the first time period, the second time period and the control time period may be different in different detection cycles. For example, the order of the time periods in some detection cycles is: "first time period - second time period - control time period", while the order of the time periods in other detection cycles is: "second time period - control time period - first time period".

[0179] like Figure 3 As shown in (a), in some embodiments, step S102 receives the reflected light signal and outputs a detection signal based on the reflected light signal, including: The same optical sensor 131 is used to receive the first reflected light signal and the second reflected light signal, and outputs the detection signal corresponding to the first reflected light signal and the second reflected light signal.

[0180] Taking the application of eye-tracking method to the eye-tracking module 10 provided in this application embodiment as an example: when the light output module 11 has only two light output components 12, the two light output components 12 can be symmetrically distributed with the light sensor 131 as the axis, for example... Figure 12 The arrangement of the first light output component 121 and the second light output component 122 is shown in (a). When the light output module 11 has multiple light output components 12, these multiple light output components 12 can be distributed circumferentially around the light sensor 131, for example... Figure 12 The first light output component 121, the second light output component 122 and the third light output component 112 shown in (b) are arranged in a triangle with the light sensor 131 as the center.

[0181] Thus, through clever positioning design, a single optical sensor 131 can receive at least two kinds of reflected light signals to output corresponding detection signals, and the first detection signal and the second detection signal can be distinguished from the detection signals output by the optical sensor 131 according to the first time period corresponding to the first emitted light signal and the first time period corresponding to the second emitted light signal.

[0182] This application only requires one light sensor 131 in the light receiving module 13 to receive the reflected light signals corresponding to at least two different emitted light signals. This not only reduces the power consumption of the eye-tracking module 10, but also optimizes the overall size and weight of the eye-tracking module 10. In particular, for the smart device 1 where the eye-tracking module 10 is located, the reduction in power consumption of the eye-tracking module 10 also makes the smart device 1 have better battery life.

[0183] like Figure 3 As shown in (b), in some embodiments, step S102 receives the reflected light signal and outputs a detection signal based on the reflected light signal, including: The first reflected light signal is received using the first optical sensor 133, and a first detection signal is output. The second reflected light signal is received using the second optical sensor 134, and a second detection signal is output. Among them, the first light sensor 133 and the second light sensor 134 are different light sensors 131. The reflected light signals received by the different light sensors 131 are reflected light signals formed by different emitted light signals reflected by the eye. Moreover, the reflected light signal received by a light sensor 131 can be a reflected light signal formed by one, two or more emitted light signals reflected by the eye.

[0184] In some embodiments, during the detection period, an emitted light signal is output to the target area of ​​the wearer's eye during the detection time period, including: outputting a first emitted light signal and a second emitted light signal to the target area of ​​the wearer's eye to form a first reflected light signal and a second reflected light signal; receiving the first reflected light signal using a first light sensor 133 and outputting a first detection signal; and receiving the second reflected light signal using a second light sensor 134 and outputting a second detection signal. The first emitted light signal and the second emitted light signal can be emitted in a time-division multiplexing manner or simultaneously, without limitation.

[0185] Specifically, the first optical sensor 133 is configured to receive a first reflected light signal and output a first detection signal, and the second optical sensor 134 is configured to receive a second reflected light signal and output a second detection signal.

[0186] In one embodiment, the first light sensor 133 is positioned corresponding to the first light output component 121, and the second light sensor 134 is positioned corresponding to the first light output component 121.

[0187] like Figure 3 As shown in (b) above, for example, with the horizontal center of the wearer's eye as the axis, the first light sensor 133 is disposed on the same side as the first light output component 121, and the second light sensor 134 is disposed on the same side as the second light output component 122.

[0188] In some implementations, receiving the reflected light signal includes: The system receives a first reflected light signal and outputs a first detection signal based on the first reflected light signal, which is formed by the eye reflecting a first emitted light signal. The system receives the second reflected light signal and outputs a second detection signal based on the first reflected light signal. The second reflected light signal is formed by the eye reflecting the second emitted light signal.

[0189] Taking the application of eye-tracking method to the eye-tracking module 10 provided in this application embodiment as an example: In fact, when the light receiving module 13 is working, it continuously receives light signals directed at it, including reflected light signals reflected back from each target area 40 in the glasses, that is, including the first reflected light signal and the second reflected light signal; the light receiving module 13 is also used to output corresponding detection signals based on the received reflected light signals, the detection signals including the first detection signal corresponding to the first reflected light signal and the second detection signal corresponding to the second reflected light signal.

[0190] When processing the detection signal, the first detection signal and the second detection signal can be distinguished from different time periods based on the first time period corresponding to the first emitted light signal and the second time period corresponding to the second emitted light signal.

[0191] As an example, and not a limitation, there are multiple ways to determine eye movement status based on detection signals. For example, within the detection period, the static value of the detection signal can be compared with a preset threshold to determine the eye movement status; or, within the detection period, the eye movement status can be determined based on the dynamic numerical characteristics of different detection signals. Specifically, it can be any one or a combination of the following methods: 1) Compare the value of a single detection signal with a preset threshold, or combine multiple detection signals into a composite value and compare it with a preset threshold, or compare the difference between multiple detection signals with a preset threshold. 2) Analyze the dynamic numerical characteristics of multiple detected signals, such as the rate of change, trend of change, time derivative, frequency domain characteristics, and waveform pattern; 3) Combine the detection signal with other information (such as temperature, distance and / or position between the device and the eye, ambient light influence, head movement information, etc.) for analysis.

[0192] As an example rather than a limitation, eye movement states can include pupil size, pupil position information, pupil movement direction, pupil movement speed, and complex eye behaviors (blinking, saccadic eye movements, fixation, etc.).

[0193] In some implementations, a threshold judgment method can be used. Step S103 obtains the eye movement state of the wearer's eyes within the detection period based on the detection signal, including: within the detection period, comparing the static value of the detection signal with a preset threshold to determine the eye movement state.

[0194] For example, the specific process of threshold determination includes: determining the first position information of the wearer's pupil in the first time period based on the first detection signal; determining the second position information of the wearer's pupil in the second time period based on the second detection signal; and obtaining the wearer's eye movement state within the detection period based on the first position information and the second position information.

[0195] Specifically, the first location information includes at least whether the wearer's pupil position overlaps with the first target area 41, and the second location information includes at least whether the wearer's pupil position overlaps with the second target area 42.

[0196] As explained above, due to the differences in light reflectivity of different areas of the human eyeball (such as the pupil, iris, and sclera), the controller 20 can determine the light intensity received by the light receiving module 13 based on the detection signal corresponding to the intensity of reflected light. Thus, based on the dynamic changes in light intensity, it can infer whether the wearer's pupil coincides with each target area 40, thereby obtaining the position of the wearer's pupil during the detection period, i.e., the eye movement state.

[0197] For example, the first target area is located on the right side of the wearer's eyeball, and the second target area is located on the left side of the wearer's eyeball. During the detection period, the light output module 11 outputs a light signal to the right side of the wearer's eyeball in the first time period and outputs a light signal to the left side of the wearer's eyeball in the second time period. If the wearer's eyeball turns horizontally to the right during the detection period: the pupil, iris, and sclera move to the right, and the right half of the sclera is partially or completely blocked by the eye socket, and the light intensity received by the light receiving module 13 decreases in the first time period; the left half of the iris / sclera is exposed, and the light intensity received by the light receiving module 13 increases in the second time period. Thus, it can be seen that the first target area 41 coincides with the position of the wearer's pupil, and the second target area 42 does not coincide with the position of the wearer's pupil.

[0198] For example, the first target area is located above the wearer's eyeball, and the second target area is located below the wearer's eyeball. During the detection period, the light output module 11 outputs a light signal to the upper side of the wearer's eyeball in the first time period and outputs a light signal to the lower side of the wearer's eyeball in the second time period. If the wearer's eyeball turns upward during the detection period, the pupil, iris, and sclera deflect upward, and the upper sclera is blocked by the eye socket. The light intensity received by the light receiving module 13 decreases in the first time period and increases in the second time period. When the wearer's eyeball turns downward, the pupil, iris, and sclera deflect downward, and the lower sclera is partially or completely blocked by the eye socket. The light intensity received by the light receiving module 13 increases in the first time period and decreases in the second time period. Thus, it can be concluded that the first target area 41 coincides with the wearer's pupil position, while the second target area 42 does not coincide with the wearer's pupil position.

[0199] Therefore, the controller 20 can determine the wearer's pupil position within the detection period based on the first position information indicating whether the pupil position coincides with the first target area 41 and the second position information indicating whether the pupil position coincides with the second target area 42, thereby obtaining the wearer's eye movement state.

[0200] In some implementations, determining the first position information of the wearer's pupil during a first time period based on a first detection signal includes: The light intensity of the first reflected light signal is determined based on the first detection signal; When the light intensity of the first reflected light signal does not exceed the first intensity threshold, it is determined that the position of the wearer's pupil overlaps with the first target area during the first time period. When the light intensity of the first reflected signal exceeds the first intensity threshold, it is determined that the wearer's pupil position does not overlap with the first target area during the first time period. And / or, The second position information of the wearer's pupil during the second time period is determined based on the second detection signal, including: The light intensity of the second reflected light signal is determined based on the second detection signal; When the light intensity of the second reflected light signal does not exceed the second intensity threshold, it is determined that the wearer's pupil position and the second target area overlap during the second time period. When the light intensity of the second reflected light signal exceeds the second intensity threshold, it is determined that the wearer's pupil position does not overlap with the second target area during the second time period.

[0201] Specifically, as described in the foregoing embodiments, due to the differences in light reflectivity of different areas of the human eyeball (such as the pupil, iris, and sclera), when the position of the wearer's pupil coincides with the target area 40, the intensity of the reflected light signal corresponding to the target area 40 will decrease significantly. In this embodiment, the first intensity threshold / second intensity threshold can be used to quickly and accurately determine whether the pupil position coincides with each target area 40.

[0202] It should be added that the values ​​of the first intensity threshold and the second intensity threshold can be set according to the wearer's individual eye characteristics and eye habits.

[0203] In some implementations, threshold determination can also be based on the difference between different detection signals or the difference between a detection signal and a reference value. The difference between different detection signals can be the difference between the detection signals of the left and right eye sensors in the same time period, or the difference between the detection signals of the same sensor in different time periods; the reference value can be a baseline value recorded during the calibration of each wearer, or a reference signal for dynamic detection (such as the signal output by the light receiving module 13 when all the aforementioned light output components 12 are not emitting light).

[0204] In some implementations, step S103, which acquires the eye movement state of the wearer's eyes during the detection period based on the detection signal, includes: determining the eye movement state based on the dynamic numerical characteristics of different detection signals during the detection period.

[0205] Specifically, in addition to threshold judgment, analysis and judgment can also be made based on dynamic numerical characteristics such as the rate of change, trend of change, time derivative, frequency domain features, and waveform pattern of the detected signal, and / or the static values ​​of the detected signal: eye movement speed and direction can be determined by the first derivative (slope / rate of change) of the detected signal; or, based on machine learning, dynamic features (such as frequency domain features, zero crossing rate, trend of change, etc.) can be extracted from the waveform of the detected signal, and the eye movement state can be determined by a classification model; or, as needed, the eye movement state can be mapped by a polynomial fitting function based on the parameters of different detected signals; or, the position of the pupil in different time periods can be determined by the change of the detected signal (increase in intensity / decrease in intensity).

[0206] In some implementations, eye movement status can be determined by combining the above methods, thereby increasing accuracy.

[0207] In some implementations, the detection signal can be combined with other information (such as temperature, distance between the device and the eye, ambient light effects, head movement information, etc.) for analysis, and then the eye movement state can be determined through relevant methods. For example, the distance information between the device and the eye can be added, and then analyzed using deep neural networks (such as MLP), multinomial fitting, or Bayesian fitting in machine learning.

[0208] In some implementations, the method further includes: Eye-tracking information of the wearer's eyes is obtained based on the eye movement status corresponding to at least two detection cycles.

[0209] Specifically, the changes in the wearer's pupil position during the target time period can be obtained based on the eye movement states corresponding to at least two detection cycles, i.e., eye tracking information. The target time period is a time period including at least two detection cycles, which can be continuous or discontinuous in time.

[0210] The following combination Figures 17 to 19 This section explains how eye-tracking information is obtained, including... Figure 18 and Figure 19 In the example shown, both the first intensity threshold and the second intensity threshold are equal to M.

[0211] like Figure 17 As shown in Figure (a), in one embodiment, the first target area 41 is located on the upper left side of the eyeball, and the second target area 42 is located on the lower left side of the eyeball. When the wearer is looking straight ahead, neither the first target area 41 nor the second target area 42 coincides with the position of the wearer's pupil.

[0212] like Figure 17As shown in (b), in another embodiment, the first target area 41 is located on the upper left side of the eyeball, and the second target area 42 is located on the lower left side of the eyeball. When the wearer is looking straight ahead, both the first target area 41 and the second target area 42 coincide with the position of the wearer's pupil.

[0213] like Figure 18 As shown in (a), specifically, during the first detection period T1, the wearer looks straight ahead, and neither the first target area 41 nor the second target area 42 coincides with the wearer's pupil position. During the first time period of the first detection period T1, the intensity of the reflected light signal exceeds the first intensity threshold, and during the second time period of the first detection period T1, the intensity of the reflected light signal exceeds the second intensity threshold.

[0214] like Figure 18 As shown in (b), if the wearer's eyeballs turn horizontally to the left during the second detection period T2, and the intensity of the reflected light signal in the first time period of the second detection period T2 does not exceed the first intensity threshold, and the intensity of the reflected light signal in the second time period of the second detection period T2 does not exceed the second intensity threshold, the controller 20 can obtain the wearer's eye movement tracking information for the target time period as: eyeballs turn horizontally to the left based on the eye movement states of the first detection period T1 and the second detection period T2.

[0215] like Figure 18 As shown in (c), if the wearer's eyeball turns to the upper left during the second detection period T2, and the intensity of the reflected light signal in the first time period of the second detection period T2 does not exceed the first intensity threshold, and the intensity of the reflected light signal in the second time period of the second detection period T2 exceeds the second intensity threshold, the controller 20 can obtain the wearer's eye movement tracking information for the target time period as: eyeball turns to the upper left based on the eye movement state of the first detection period T1 and the second detection period T2.

[0216] like Figure 18 As shown in (d), if the wearer's eyeball turns to the lower left during the second detection period T2, and the intensity of the reflected light signal in the second time period of the second detection period T2 does not exceed the second intensity threshold, the controller 20 can obtain the wearer's eye movement tracking information for the target time period as: eyeball turns to the lower left based on the eye movement state of the first detection period T1 and the second detection period T2.

[0217] like Figure 19 As shown in (a), specifically, during the first detection period T1, the wearer looks straight ahead, and both the first target area 41 and the second target area 42 coincide with the wearer's pupil position. During the first time period of the first detection period T1, the intensity of the reflected light signal does not exceed the first intensity threshold, and during the second time period of the first detection period T1, the intensity of the reflected light signal does not exceed the second intensity threshold.

[0218] like Figure 19As shown in (b), if the wearer's eyeballs turn horizontally to the right during the second detection period T2, and the intensity of the reflected light signal exceeds the first intensity threshold during the first time period of the second detection period T2, and the intensity of the reflected light signal exceeds the second intensity threshold during the second time period of the second detection period T2, the controller 20 can obtain the wearer's eye movement tracking information for the target time period as: eyeballs turn horizontally to the right based on the eye movement states of the first detection period T1 and the second detection period T2.

[0219] like Figure 19 As shown in (c), if the wearer's eyeball turns to the right and downward in the second detection period T2, the intensity of the reflected light signal in the first time period of the second detection period T2 exceeds the first intensity threshold, and the intensity of the reflected light signal in the second time period of the second detection period T2 does not exceed the second intensity threshold, the controller 20 can obtain the wearer's eye movement tracking information for the target time period as: eyeball turns to the right and downward based on the eye movement state of the first detection period T1 and the second detection period T2.

[0220] like Figure 19 As shown in (d), if the wearer's eyeball turns to the upper right in the second detection period T2, and the intensity of the reflected light signal in the first time period of the second detection period T2 does not exceed the first intensity threshold, and the intensity of the reflected light signal in the second time period of the second detection period T2 exceeds the second intensity threshold, the controller 20 can obtain the wearer's eye movement tracking information for the target time period as: eyeball turns to the upper right based on the eye movement state of the first detection period T1 and the second detection period T2.

[0221] Therefore, the controller 20 can obtain the wearer's continuous pupil position changes within the target time period based on the eye movement state of the first detection period T1 and the second detection period T2, so as to obtain eye movement tracking information.

[0222] As one embodiment, the detection cycle includes at least two first time periods and at least two second time periods. The wearer's eye movement path, i.e., eye tracking information, can also be analyzed based on the intensity of the reflected light signals received in the first and second time periods within the same detection cycle.

[0223] like Figure 5 As shown in the example, the detection cycle includes at least two first time periods and at least two second time periods, which are arranged alternately. Furthermore, the first target area 41 is located on the upper left side of the eyeball, and the second target area 42 is located on the lower left side of the eyeball. When the wearer is looking straight ahead, neither the first target area 41 nor the second target area 42 coincides with the wearer's pupil position.

[0224] During the first time period and the first second time period, the wearer looks straight ahead, and neither the first target area 41 nor the second target area 42 coincides with the wearer's pupil position. During the first detection period T1, the intensity of the reflected light signal exceeds the first intensity threshold, and during the first detection period T1, the intensity of the reflected light signal exceeds the second intensity threshold.

[0225] If the intensity of the reflected light signal in the second first time period does not exceed the first intensity threshold, and the intensity of the reflected light signal in the second second time period does not exceed the second intensity threshold, the wearer's eye movement path during the detection period can be obtained as: the eyeball turns horizontally to the left.

[0226] If the intensity of the reflected light signal does not exceed the first intensity threshold in the second first time period, and the intensity of the reflected light signal exceeds the second intensity threshold in the second second time period, the wearer's eye movement path during the detection period can be obtained as: the eyeball turns to the upper left.

[0227] If the intensity of the reflected light signal exceeds the first intensity threshold in the second first time period, and the intensity of the reflected light signal does not exceed the second intensity threshold in the second second time period, the wearer's eye movement path during the detection period can be obtained as: the eyeball turns to the lower left.

[0228] Please see Figure 22 This application also provides an eye-tracking device 300, which is, for example, an eye-tracking module 10 or a smart device 1.

[0229] Specifically, the eye-tracking device 300 includes a processor 301 and a memory 302, which are connected via a bus 303, such as an I2C (Inter-integrated Circuit) bus.

[0230] Specifically, processor 301 provides computing and control capabilities to support the operation of the entire eye-tracking device. Processor 301 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0231] Specifically, the memory 302 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc.

[0232] Those skilled in the art will understand that Figure 22 The structure shown in the figure is merely a block diagram of a portion of the structure related to the embodiments of this application, and does not constitute a limitation on the eye-tracking device to which the embodiments of this application are applied. A specific eye-tracking device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0233] The processor is configured to run a computer program stored in a memory, and to implement any of the eye-tracking methods provided in the embodiments of this application when executing the computer program.

[0234] This application also provides a storage medium storing a computer program that can be executed by one or more processors to implement the steps of any of the conflict detection map construction methods provided in the specification of this application.

[0235] The storage medium can be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0236] like Figure 15 As shown, this application also provides a smart device 1, comprising: At least one display module 30 is configured to display a computer image; At least one eye-tracking module 10 is configured to detect and output a detection signal associated with at least one eye of the wearer. The eye-tracking module 10 includes a light output module 11 and a light receiving module 13. The light output module 11 is configured to output emitted light signals to different target areas 40 of the same eye of the wearer in a time-division manner during the detection period. The light receiving module 13 is configured to detect reflected light signals formed by the reflected emitted light signals from the eye and output a detection signal. Controller 20, which is communicatively connected to display module 30 and eye-tracking module 10, is configured to: Within the preset calibration period, the control display module 30 displays the calibration image; The detection signal output by the light receiving module 13 is acquired when the wearer's eyes are focused on a preset calibration point or perform a preset eye movement; The intensity threshold of the reflected light signal corresponding to the target area 40 is determined based on the detection signal.

[0237] It should be understood that the control behavior of the controller 20 is intended to calibrate the intensity threshold of the reflected light signal corresponding to the target area 40 of the smart device 1 or the eye-tracking module 10.

[0238] As an example and not a limitation, the preset eye movement is when the wearer's eyes are looking straight ahead. As an example and not a limitation, when the wearer gazes at the preset calibration point, the wearer's eyes are looking straight ahead.

[0239] As one embodiment, the smart device 1 includes two eye-tracking modules 10, which correspond to the wearer's left and right eyes respectively, to detect the eye movement state of the wearer's left and right eyes. Specifically, the two modules are respectively set to correspond to the wearer's first and second eyes, and the two modules can be modules with the same configuration or modules with different configurations.

[0240] In some embodiments, the light output module 11 includes at least a first light output component 121 and a second light output component 122. The first light output component 121 is configured to output a first emitted light signal to a first target region 41, and the second light output component 122 is configured to output a second emitted light signal to a second target region 42. The first target region 41 and the second target region 42 are at least partially non-overlapping different regions of the same eye of the wearer.

[0241] Accordingly, controller 20 is configured as follows: Control the first optical output component 121 to output a first calibration emission optical signal to the first target area 41; The optical receiving module 13 is controlled to receive the first calibration reflected light signal and output the first calibration detection signal. The first calibration reflected light signal is formed by the eye reflecting the first calibration emitted light signal. The first intensity threshold corresponding to the first reflected light signal is determined based on the first calibration detection signal.

[0242] Controller 20 is also configured to: Control the second optical output component 122 to output a second calibration emission optical signal to the second target area 42; The optical receiving module 13 is controlled to receive the second calibration reflected light signal and output the second calibration detection signal. The second calibration reflected light signal is formed by the reflection of the second emitted light signal by the eye. The second intensity threshold corresponding to the second reflected light signal is determined based on the second calibration detection signal.

[0243] Therefore, by using the above method, suitable first and second intensity thresholds can be set according to the wearer's individual eye characteristics and eye habits, thereby improving the accuracy of judging the wearer's pupil position and obtaining eye movement status.

[0244] Please see Figure 23 This application embodiment also provides an eye-tracking calibration method, applied to any of the smart devices 1 or eye-tracking modules 10 provided in this application, for calibrating the intensity threshold of the reflected light signal corresponding to the target area 40 of the smart device 1 or eye-tracking module 10. The eye-tracking calibration method includes steps S201 to S202: Step S201: Acquire the detection signal output by the light receiving module 13 when the wearer's eyes are focused on a preset calibration point or perform a preset eye movement; Step S202: Determine the intensity threshold of the reflected light signal corresponding to the target area 40 based on the detection signal.

[0245] The specific descriptions of the smart device 1 and the eye-tracking module 10 can be found in the description of any embodiment of this application.

[0246] Specifically, the eye-tracking calibration method provided in this application embodiment is performed after the wearer has put on the smart glasses. As one embodiment, before acquiring the detection signal, this method further includes detecting the wearer's wearing status of the smart device 1. Furthermore, steps S201 and S202 are only executed after it is determined that the wearer is wearing the smart device 1.

[0247] As an example and not a limitation, the preset eye movement is when the wearer's eyes are looking straight ahead. As an example and not a limitation, when the wearer gazes at the preset calibration point, the wearer's eyes are looking straight ahead.

[0248] It should be understood that since each wearer may have different eye height and interpupillary distance, the relative position of the eyes of different wearers with respect to the eye movement module 10 is also different. Therefore, before using the smart device 1, the wearer needs to perform an eye position calibration operation, that is, the wearer gazes at a preset calibration point or performs a preset eye movement light intensity calibration of the receiving module 13.

[0249] Based on the intensity threshold obtained by the eye-tracking calibration method, during the subsequent use of the smart device 1 or the eye-tracking module 10, the light intensity received by the light receiving module 13 can be compared with the intensity threshold to determine the wearer's eye movement state. For the specific method of obtaining the eye movement state, please refer to the detailed description of the eye-tracking module 10, the smart device 1, and the eye-tracking method in the foregoing embodiments.

[0250] In some embodiments, the light output module 11 includes at least a first light output component 121 and a second light output component 122. The first light output component 121 is configured to output a first emitted light signal to a first target region 41, and the second light output component 122 is configured to output a second emitted light signal to a second target region 42. The first target region 41 and the second target region 42 are at least partially non-overlapping different regions of the same eye of the wearer. Acquire the detection signal output by the light receiving module when the wearer's eyes are focused on a preset calibration point or performing a preset eye movement, including: Control the first optical output component 121 to output a first calibration emission optical signal to the first target area 41; When the wearer's eyes are focused on a preset calibration point or when a preset eye movement is performed, the light receiving module 13 is controlled to receive the first calibration reflected light signal and output the first calibration detection signal. The first calibration reflected light signal is formed by the reflection of the first calibration emitted light signal by the eye. A first intensity threshold corresponding to the first reflected light signal is determined based on the first calibration detection signal; And / or, Control the second optical output component 122 to output a second calibration emission optical signal to the second target area 42; When the wearer's eyes are focused on a preset calibration point or when a preset eye movement is performed, the light receiving module 13 is controlled to receive the second calibration reflected light signal and output the second calibration detection signal. The second calibration reflected light signal is formed by the reflection of the second emitted light signal by the eyes. The second intensity threshold corresponding to the second reflected light signal is determined based on the second calibration detection signal.

[0251] It should be noted that the first calibration emitted light signal has the same signal attributes as the first emitted light signal, and the second calibration emitted light signal has the same signal attributes as the second emitted light signal, in order to ensure the accuracy of the first intensity threshold and the second intensity threshold. The signal attributes include, but are not limited to, the wavelength and intensity of the light signal.

[0252] Specifically, the intensity of the first calibration detection signal is matched with the intensity of the first calibration emitted light signal. Based on the first calibration detection signal, the intensity of the first calibration emitted light signal when the wearer's eyes are fixed on a preset calibration point or when performing a preset eye movement can be obtained, and this intensity is used as a first intensity threshold. Similarly, the intensity of the second calibration detection signal is matched with the intensity of the second calibration emitted light signal. Based on the second calibration detection signal, the intensity of the second calibration emitted light signal when the wearer's eyes are fixed on a preset calibration point or when performing a preset eye movement can be obtained, and this intensity is used as a second intensity threshold.

[0253] It should be understood that the calibration of the first intensity threshold and the calibration of the second intensity threshold can be performed simultaneously or sequentially.

[0254] like Figure 17 As shown in (a), as an embodiment and not a limitation, the first target region 41 is located on the upper left side of the eyeball, and the second target region 42 is located on the lower left side of the eyeball. When the wearer gazes at a preset calibration point or performs a preset eye movement, neither the first target region 41 nor the second target region 42 coincides with the wearer's pupil position. The first intensity threshold obtained using the eye-tracking calibration method is the light signal intensity obtained from the first calibration reflected light signal of the iris. Similarly, the second intensity threshold obtained using the eye-tracking calibration method is the light signal intensity obtained from the second calibration reflected light signal of the iris.

[0255] In summary, embodiments of this application provide one or more eye-tracking methods, devices, and storage media. The method includes: during a detection period, outputting emitted light signals to different target areas of the same eye of the wearer in a time-division manner, receiving reflected light signals, and outputting detection signals based on the reflected light signals. The reflected light signals are formed by the eye reflecting the emitted light signals. The eye movement state of the wearer's eye during the detection period is obtained based on the detection signals. Thus, the wearer's eye movement state can be identified based on the corresponding emitted light signals and detection signals. Furthermore, the power consumption of related devices (such as eye-tracking modules and smart devices) is reduced, and their size and weight are reduced.

[0256] This application also provides an eye-tracking algorithm model, a training method for the eye-tracking algorithm model, and a verification method for the eye-tracking algorithm model.

[0257] In some embodiments, during the detection of eye movement states, such as Figure 24 As shown, a preprocessing model is first established by simulating the input detection signal. Then, the preprocessing model is verified and calibrated using a real device to obtain the actual processing model. Finally, the actual processing model is applied to the smart device 1. The actual processing model is the eye-tracking algorithm model.

[0258] In some embodiments, when the preprocessing model preprocesses the detection signal, and / or the eye-tracking algorithm model processes the detection signal, the eye-tracking state can be obtained by analyzing the rule engine such as first derivative and function fitting, and / or by analyzing the eye-tracking state through machine learning algorithms such as random forest and convolutional neural network.

[0259] Similarly, in some embodiments, the preprocessing model can be corrected during the real device verification process using rule engines such as function fitting and / or machine learning algorithms such as convolutional neural networks. It is understood that the preprocessing model and the input parameters for real device verification, such as the detection signals of each optical receiving module, can be obtained using the eye-tracking method described in this application to determine the eye movement state.

[0260] Taking the reflected light signal as an example, such as light intensity Figure 25 As shown, in some embodiments, when the preprocessing model determines the eye movement state, firstly, during a first time period, the first light output component 121 outputs a first emitted light signal to the first target area 41; during a second time period, the second light output component 122 outputs a second emitted light signal to the second target area 42. Each emitted light signal is reflected by the pupil, iris, and sclera to form a reflected light signal. Then, the light receiving module 13 receives the first reflected light signal to form a first detection signal, and analyzes the first detection signal through a rule engine and / or machine learning algorithm to identify the first light intensity distribution information; the light receiving module 13 receives the second reflected light signal to form a second detection signal, and analyzes the second detection signal through a rule engine and / or machine learning algorithm to identify the second light intensity distribution information. Further, through a rule engine and / or machine learning algorithm, the first light intensity distribution information and the second light intensity distribution information are comprehensively analyzed to obtain the eye movement state.

[0261] Since the reflected light signals of the first target area 41 and the second target area 42 change with eye movement at different times during the detection period, in some embodiments, analyzing the first detection signal in the first time period and / or the second detection signal in the second time period can yield eye movement states such as pupil position. Furthermore, in some embodiments, since the display screen of the display module 30 may have a corresponding Cartesian coordinate system, analyzing the first light intensity distribution information in the first time period and / or the second light intensity distribution information in the second time period based on the coordinate system of the display screen can yield eye movement states such as the pupil's fixation position on the screen.

[0262] It is understood that in some embodiments, after acquiring the detection signal, the analysis of the first light intensity distribution information and the second light intensity distribution information can be performed separately in chronological order, or simultaneously after obtaining the first and second detection signals; no limitation is made here. When the light output module 11 includes a third light output component 123, during the third time period, the third light output component 123 outputs a third emitted light signal to the third target region 43, and the third light intensity distribution information can be identified based on the third detection signal.

[0263] Furthermore, the emitted light signal in the preprocessing model is the light signal from the simulation system, while the emitted light signal for real-device verification is the light signal generated by the actual light output component 12. The training of the preprocessing model should be performed using at least two light output modules 11 on the smart device 1. In some embodiments, each light output module 11 includes at least two light output components.

[0264] During the real device verification process, such as Figure 26As shown, firstly, detection signals from the real-world environment are acquired. A preprocessing model processes these signals to obtain a preprocessed eye-tracking state. Then, the eye-tracking state of the preprocessing model is verified based on the actual eye-tracking state and detection requirements. If the verification result meets the detection requirements, positive feedback is input to the preprocessing model to strengthen it. If the verification result does not meet the detection requirements, negative feedback is input to the preprocessing model to correct it, until the verification result meets the detection requirements. The detection signal processing, verification, strengthening, and correction mentioned in this application can all be performed by any suitable rule engine, machine learning algorithm, or other suitable algorithm; no restrictions are imposed here.

[0265] In some embodiments, the detection signal can be combined with other information (such as temperature, relative position of the device and the eye, ambient light influence, head movement information, etc.) for analysis, and the preprocessing model training, real machine verification and actual processing model application can be performed in sequence with reference to the above methods.

[0266] In some embodiments, temperature information can be detected by a temperature sensor, the relative position between the device and the eyes can be detected by one or more devices such as an infrared sensor, a camera, an accelerometer, or a gyroscope, ambient light can be detected by one or more devices such as an ambient light sensor, a proximity sensor, a camera, a spectral sensor, or spatial lighting modeling, and head movement information can be detected by one or more devices such as an accelerometer, a gyroscope, a magnetometer, an inertial measurement unit, a vision sensor, or an ultrasonic sensor, without limitation.

[0267] This application also provides a calibration algorithm model, a training method for the calibration algorithm model, and a verification method for the calibration algorithm model.

[0268] In the establishment and application of the calibration algorithm model, a pre-calibration model needs to be established first through a rule engine and / or machine learning algorithm; then the pre-calibration model is verified to obtain the calibration algorithm model, and a calibration database can be established to further optimize the calibration algorithm model; finally, the smart device 1 is calibrated during user use.

[0269] In some embodiments, such as Figure 27 As shown, in the training method of the calibration algorithm model, firstly, initial parameters are set, and after the user wears the smart device 1, they gaze at the calibration point, obtaining input calibration information through the eye-tracking module 10. Then, a pre-calibrated model is obtained through a rule engine and / or machine learning algorithms, etc. In some embodiments, the input calibration information may be gaze point position information measured by an infrared sensor, etc., and sensor information, where the sensor information includes at least the detection signal; furthermore, the sensor information may also include temperature and the relative position of the glasses.

[0270] In some embodiments, such as Figure 28 As shown, in the calibration algorithm model verification process, firstly, the pre-calibrated model generates output parameters, compares the initial parameters with the output parameters, and performs error analysis using a rule engine and / or machine learning algorithm. Then, it is determined whether the error meets the calibration requirements; if the requirements are met, optimization stops, and the calibrated algorithm model is obtained.

[0271] Furthermore, in some embodiments, different calibration samples can be formed by collecting any suitable number of user data. A calibration database can be established through a calibration algorithm model, and the calibration algorithm model can be optimized. Virtual sampling points can be supplemented with less actual sampling point data to reduce the number of actual sampling points. For example, a small number of input calibration points can be set, specifically 4, 9, 16, or any other suitable number of n. The calibration algorithm model uses a rule engine and / or machine learning algorithm to fit and generate multiple virtual output calibration points, specifically 81, 91, or any other suitable number of kn, where k is an integer not less than 1. Establishing a calibration database reduces the number of samples, reduces the amount of calibration computation, expands the calibration area, and improves calibration accuracy.

[0272] In some embodiments, such as Figure 29 As shown, during user calibration, firstly, the user wears the smart device 1 and gazes at the calibration point to obtain user calibration information; then, the calibration algorithm model determines the actual parameters based on the detected user calibration information, so that the eye-tracking module 10 outputs according to the actual parameters. The application of the calibration algorithm model reduces the error of eye-tracking detection and improves the adaptability of the eye-tracking module 10 to different users.

[0273] In some embodiments, the calibration of smart device 1 may include static calibration and dynamic calibration. Static calibration refers to calibration performed when wearing smart device 1, while dynamic calibration refers to calibration performed during the use of smart device 1. The parameter processing, verification, and optimization mentioned in this application can all be handled by any suitable rule engine, machine learning algorithm, or other suitable method, and no limitation is made here.

[0274] Please see Figure 30 This application embodiment also provides an eye-tracking algorithm model generation method. The eye-tracking algorithm model is applied to a smart device 1, which is worn on the wearer's head and includes an eye-tracking module 10. The eye-tracking module 10 is configured to output emitted light signals to the wearer's eyes, receive reflected light signals formed by the reflected emitted light signals from the eyes, and generate detection signals.

[0275] like Figure 30 As shown, the eye-tracking algorithm model generation method specifically includes the following steps: S911: Obtain the first initial model, which is used to generate an eye movement state corresponding to the eye based on the detection signal; S912: Train the first initial model to obtain the eye-tracking algorithm model.

[0276] Specifically, the model input of the first initial model is a detection signal. This detection signal is generated by the eye-tracking module 10 (e.g., the light receiving module 13 in the eye-tracking module 10) receiving a reflected light signal formed by the reflection of light emitted by the eye. The detection signal has been specifically described in the preceding embodiments and can be referred to therein; therefore, it will not be repeated here. The model output of the first initial model is the eye movement state corresponding to the eyes of the wearer of the smart device 1. This eye movement state includes, but is not limited to, information such as the direction of eye movement and pupil position. The eye movement state has been specifically described in the preceding embodiments and can be referred to therein; therefore, it will not be repeated here.

[0277] Specifically, the input to the eye-tracking algorithm model is also the detection signal, and the output is also the eye movement state. However, based on the training of the first initial model, with the same detection signal as the model input, the eye movement state output by the eye-tracking algorithm model is more accurate.

[0278] It should be noted that model training greatly improves the accuracy of the trained model (i.e., the eye-tracking algorithm model) in determining the eye movement state based on the detection signal, making the eye movement state more consistent with the wearer's actual eye movement behavior. For example, the smart device 1, eye-tracking module 10, or other devices / modules that can acquire detection signals provided in the aforementioned embodiments of this application can all use the trained eye-tracking algorithm model to obtain the wearer's accurate eye movement state based on the detection signal.

[0279] It should also be noted that the first initial model can be obtained either by directly creating the first initial model or by calling an existing first initial model. As one example, the first initial model is created by inputting the simulation's detection signals.

[0280] In some embodiments, the process of generating eye-tracking states based on detection signals using the first initial model includes: analyzing eye-tracking states using rule engines such as first derivatives and function fitting, and / or analyzing eye-tracking states using machine learning algorithms such as random forests and convolutional neural networks. Based on the model backbone of the first initial model, the process of building the first initial model can also be performed using rule engines and / or machine learning algorithms.

[0281] In some embodiments, the eye-tracking module 10 is configured to receive a first reflected light signal formed by an emitted light signal reflected from a first target region and a second reflected light signal formed by an emitted light signal reflected from a second target region, and to generate a detection signal based at least on the first reflected light signal and the second reflected light signal, wherein the first target region and the second target region are at least partially non-overlapping different regions of the same eye of the wearer.

[0282] The first initial model is used to generate light intensity distribution data of the eye based on the detection signal, and to generate eye movement state based on the light intensity distribution data. The light intensity distribution data includes at least the first light intensity data of the first reflected light signal and the second light intensity data of the second reflected light signal.

[0283] Specifically, the light intensity distribution data includes the light intensity of the reflected light signal corresponding to different areas of the eye. It should be understood that the reflected light signal corresponding to each area is the reflected signal generated by the reflected emitted light signal of that area.

[0284] As one embodiment, the eye-tracking module 10 includes at least a first light output component 12112, a second light output component 12212, and a light receiving module 13. The first light output component 12112 is configured to output a first emitted light signal to a first target area, the second light output component 12212 is configured to output a second emitted light signal to a second target area, and the light receiving module 13 is configured to receive the first emitted light signal and the second emitted light signal and generate a reflected light signal.

[0285] Furthermore, the eye-tracking module 10 is also configured to receive a third reflected light signal formed by the reflected light signal from the third target region, and to generate a detection signal based on the first reflected light signal, the second reflected light signal, and the third reflected light signal. The third target region, the first target region, and the second target region are at least partially non-overlapping different regions of the same eye of the wearer; the light intensity distribution data also includes third light intensity data of the third reflected light signal.

[0286] In practice, depending on the needs of detection and training, the eye-tracking module 10 can be configured to receive reflected light signals from more target areas. It should be understood that the detection signal generated from the reflected light signals corresponding to more areas can generate more accurate and complete light intensity distribution data. Based on this light intensity distribution data, a more accurate eye-tracking state can be generated. However, for the sake of balancing detection accuracy with the cost, weight, and volume of the light output component 12, the number of light output components 12 is preferably controlled to be two or three.

[0287] like Figure 31As shown, in some embodiments, the first initial model includes at least a first model layer and a second model layer. The first model layer is used to generate light intensity distribution data based on the detection signal, and the second model layer is used to generate eye movement state based on the light intensity distribution data.

[0288] As one embodiment, based on the hierarchical structure of the first initial model, the process of establishing the first initial model includes establishing / calling a first model layer; establishing / calling a second model layer; and generating the first initial model based on the first and second model layers. For example, the first and second model layers can also be established using a rule engine and / or machine learning algorithms.

[0289] It should be noted that when training the first initial model, training can be performed on at least one of the first model layer and the second model layer individually, or the entire first initial model can be trained. For example, in some embodiments, training the second model layer can enable it to generate more accurate eye movement states based on light intensity distribution data. In other embodiments, the entire first initial model is trained.

[0290] In some embodiments, in the display module 30 of the smart device 1, the display screen may have a corresponding Cartesian coordinate system. Based on the coordinate system of the display screen, eye movement states such as the gaze position of the pupil on the screen can be obtained.

[0291] In some implementations, the process of training the first initial model includes: The eye-tracking module 10 outputs eye-tracking commands to enable the wearer to perform corresponding eye-tracking behaviors based on the eye-tracking commands, and uses the eye-tracking module 10 to collect detection signal samples under the condition that the wearer performs eye-tracking behaviors. The first initial model is trained using training samples, which include at least one eye movement behavior and actual eye movement state.

[0292] like Figure 32 As shown, the training of the first initial model using smart device 1 will be illustrated as an example: The display module 30 (e.g., display screen) of smart device 1 displays... Figure 32 The calibration image shown has multiple calibration points (each E represents a calibration point). It outputs eye-tracking commands to instruct the wearer to actively look at the corresponding calibration point to perform the corresponding eye-tracking behavior. It controls the eye-tracking module 10 to output emitted light signals to the wearer's eyes and output emitted light signal samples. It receives reflected light signal samples formed by the reflected emitted light signals from the eyes and generates detection signal samples.

[0293] For example, in conjunction with the calibration image, eye movement behavior can refer to looking at any calibration point in the calibration image, such as looking at the calibration point with coordinates (1,1), looking at the calibration point with coordinates (1,2), or looking at the calibration point with coordinates (1,3), and so on.

[0294] After generating detection signal samples, eye movement behaviors and the corresponding detection signal samples are associated as a set of training samples, and then the model to be trained is trained based on at least one set of training samples.

[0295] like Figure 33 As shown in the illustration, in one embodiment, training the first initial model using training samples specifically involves iteratively training the model to be trained using the training samples for multiple iteration cycles. It should be noted that the model to be trained refers to either the first initial model or an optimized model that appears midway through the iterative training. The optimized model is the model that has been trained on the first initial model for any number of iteration cycles but has not yet reached the condition of training completion. The model that has been trained successfully is the eye-tracking algorithm model.

[0296] Specifically, in each iteration cycle, the model to be trained first processes the detection signal samples to obtain the predicted eye movement state, and then verifies the predicted eye movement state based on the actual eye movement state. If the verification result meets the detection requirements, positive feedback is input to the model to be trained to strengthen the model. If the verification result does not meet the detection requirements, negative feedback is input to the model to be trained to correct the model, until the training objective of iterative training is reached and the parameter optimization is completed.

[0297] It should be noted that the model verification, enhancement, and correction in the embodiments of this application can all be completed by any suitable rule engine, machine learning algorithm, or other suitable algorithm, and there are no restrictions here.

[0298] Here's an example illustrating any iteration in the training process: In some implementations, training a first initial model using training samples includes: inputting detection signal samples into the model to be trained in the same iteration cycle, so that the model to be trained generates a predicted eye movement state based on the detection signal samples, determining the eye movement error between the predicted eye movement state and the actual eye movement state corresponding to the same eye movement behavior, and optimizing the model parameters of the model to be trained based on the eye movement error.

[0299] In some implementations, determining the eye movement error between the predicted eye movement state and the actual eye movement state corresponding to the same target eye movement behavior includes: determining the actual gaze position corresponding to the actual eye movement state and the predicted gaze position corresponding to the predicted eye movement state; calculating the coordinate distance between the actual gaze position and the predicted gaze position in the same iteration cycle, so as to determine the eye movement error based on the coordinate distance.

[0300] For example, it is possible to utilize, such as Figure 33 The calibration image shown is used to calculate eye movement error. Specifically, the calibration points on the calibration image are used to represent the actual gaze position, that is, the position that the wearer is actually looking at. The model to be trained generates prediction points based on the detection signal samples. These prediction points are used to represent the predicted gaze position, that is, the position that the model to be trained predicts the position that the wearer is looking at based on the detection signal samples. The eye movement error can be represented by the coordinate distance (e.g., Euclidean distance) between the calibration points and prediction points corresponding to the same set of training samples on the calibration image.

[0301] like Figure 34 As shown, for example, with the coordinates of the calibration point as the reference and the predicted point as the eye movement point obtained in the model optimization, the predicted eye movement state is considered to meet the detection requirements when the distance information between the predicted point and the calibration point is less than a preset threshold, and the predicted eye movement state is considered to not meet the detection requirements when the distance information is greater than or equal to the preset threshold.

[0302] like Figure 33 As shown, in some implementations, the model parameters of the model to be trained are optimized based on the eye movement error, including: adjusting the model parameters of the model to be trained to obtain the optimized model in the current iteration cycle, so that the eye movement error obtained by inputting the detection signal samples in the current iteration cycle into the optimized model reaches the minimum value.

[0303] It should be noted that the model to be trained generates corresponding model outputs based on its model parameters and inputs, thereby producing predicted eye movement states based on the detected signal samples. Therefore, under different model parameters, the predicted eye movement states and corresponding eye movement errors obtained by inputting the same set of training samples into the model to be trained will also differ. Based on this, in the process of optimizing the model parameters of the model to be trained, the specific goal is to adjust the model parameters of the model to be trained to minimize the eye movement error corresponding to the current detected signal sample.

[0304] In some implementations, the training samples include detection signal samples corresponding to at least two different eye movement behaviors and actual eye movement states. Specifically, at least two different sets of training samples can be used to train the model to be trained, and one set of training samples can be used for model training in each iteration cycle.

[0305] As one embodiment, training the model using training samples further includes: The model parameters of the model to be trained are iteratively optimized over multiple iteration cycles; When the model to be trained meets the preset training objective in any iteration cycle, the model training is completed and the eye-tracking algorithm model is obtained.

[0306] It should be noted that, under the condition that the training objective has not been achieved, the intelligent device 1 optimizes the parameters in the model to be trained and enters the next iteration cycle to continue to optimize the model parameters of the model to be trained; after the training objective is achieved, the iterative training of the model to be trained can be stopped and it can be output as an eye-tracking algorithm model.

[0307] For example Figure 35 In the diagram, the horizontal axis of the Error Value represents the eye-tracking error. As one example, the training objective of the model to be trained can be: the eye-tracking error corresponding to at least the last two consecutive iterations in the sequence reaches the convergence condition, or the eye-tracking error corresponding to a certain iteration reaches a preset error threshold.

[0308] In other embodiments, training the first initial model may also involve training the model to be trained using a rule engine and / or machine learning algorithms. In still other embodiments, training the first initial model may involve performing a portion of the training using a rule engine and / or machine learning algorithms, and another portion of the training using training samples. As one example, training the first initial model includes: The first initial model is trained to obtain a preprocessed model, and the preprocessed model is trained to obtain an eye-tracking algorithm model; or, the first initial model is trained to obtain an eye-tracking algorithm model; or, the first initial model is trained to obtain an eye-tracking algorithm model. The first model training involves training a first initial model using a rule engine and / or machine learning algorithms, while the second model training involves training the first initial model or a preprocessed model using training samples.

[0309] In some embodiments, the detection signal can be combined with other information (such as temperature, relative position of the device and the eye, ambient light influence, head movement state, etc.) for analysis, and the acquisition of the first initial model and the training of the first initial model can be performed in sequence with reference to the above method.

[0310] In some implementations, before inputting the detection signal samples into the model to be trained, the method further includes: obtaining environmental information from the eye-tracking module 10 when the detection signal samples are obtained, and correcting the detection signal samples based on the environmental information. After correction, the corrected detection signal samples are input into the model to be trained to obtain the corresponding predicted eye-tracking state, thereby training the first initial model based on the predicted eye-tracking state and eye-tracking behavior. Thus, correcting the detection signal samples based on environmental information can eliminate / reduce the interference of environmental information on the training of the first initial model.

[0311] In some implementations, the environmental information includes at least one of the following: the ambient temperature corresponding to the eye-tracking module 10, the ambient light intensity corresponding to the eye-tracking module 10, the position information of the eye-tracking module 10 relative to the wearer's eyes, and the wearer's head movement state.

[0312] In some embodiments, temperature information can be detected by a temperature sensor on the smart device 1, the position information of the eye-tracking module 10 relative to the wearer's eyes can be detected by one or more devices such as an infrared sensor, camera, accelerometer or gyroscope on the smart device 1, the ambient light intensity can be detected by one or more devices such as an ambient light sensor, proximity sensor, camera, spectral sensor or spatial lighting modeling, and the head movement state can be detected by one or more devices such as an accelerometer, gyroscope, magnetometer, inertial measurement unit, vision sensor or ultrasonic sensor, without limitation.

[0313] This application embodiment also provides an eye-tracking method applied to a smart device 1, which is worn on the head of a wearer and includes an eye-tracking module 10. The eye-tracking module 10 is configured to output emitted light signals to the wearer's eyes, receive reflected light signals formed by the reflected emitted light signals from the wearer's eyes, and generate detection signals. The method includes: The eye-tracking algorithm model is invoked, and the eye-tracking algorithm model is generated using the eye-tracking algorithm model generation method provided in any embodiment of this application; The detection signal is input into the eye-tracking algorithm model so that the eye-tracking algorithm model can generate the eye movement state of the wearer's eyes.

[0314] This application also provides a storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps of the eye-tracking algorithm model generation method provided in any embodiment of this application and / or the eye-tracking method provided in any embodiment of this application.

[0315] Please see Figure 36This application also provides a method for generating a calibration algorithm model. The calibration algorithm model is applied to a smart device 1, which is worn on the wearer's head and includes an eye-tracking module 10. The eye-tracking module 10 is configured to output emitted light signals to the wearer's eyes, receive reflected light signals corresponding to at least one target area in the eyes, and generate a detection signal. The reflected light signals are formed by the reflected emitted light signals of the corresponding target area. The calibration algorithm model generation method specifically includes the following steps: S921: Obtain a second initial model, which is used to generate light intensity distribution information based on the detection signal. The light intensity distribution information includes the intensity of reflected light signal corresponding to at least one target area in the eye. S922: When the wearer performs the target eye movement behavior, the eye movement module 10 is used to acquire the detection signal sample corresponding to the target eye movement behavior; S923: Use training samples to train the second initial model to obtain a calibration algorithm model corresponding to the wearer. The training samples include at least one detection signal sample corresponding to the target eye movement behavior and standard light intensity distribution information.

[0316] Specifically, the model input of the second initial model is a detection signal. This detection signal is generated by the eye-tracking module 10 (e.g., the light receiving module 13 in the eye-tracking module 10) receiving a reflected light signal formed by the reflected light emitted by the eye. The detection signal has been specifically described in the preceding embodiments and will not be repeated here. The model output of the second initial model is the intensity of the reflected light signal corresponding to at least one target area in the eye of the wearer of the smart device 1. The intensity of the reflected light signal has been specifically described in the preceding embodiments and will not be repeated here. The standard light intensity distribution information can be the light intensity distribution information of a pre-detected or pre-set calibration point.

[0317] Specifically, acquiring detection signal samples corresponding to the target eye movement behavior using the eye-tracking module 10 includes: when the wearer performs the target eye movement behavior, the eye-tracking module 10 outputs a emitted light signal to the wearer's eyes, causing the eyes to reflect the emitted light signal to form a reflected light signal; the eye-tracking module 10 receives the reflected light signal and generates a detection signal. The output of the emitted light signal and the reception of the reflected light signal by the eye-tracking module 10 have been specifically described in the aforementioned embodiments and can be referred to the descriptions therein, and will not be repeated here.

[0318] As one embodiment, in S922, the acquisition of detection signal samples corresponding to the target eye movement behavior by the eye-tracking module 10 includes: the wearer looking at the calibration point displayed on the smart device 1 while wearing the smart device 1 (i.e. performing the target eye movement behavior), the eye-tracking module 10 outputting an emitted light signal to the wearer's eyes so that the eyes reflect the emitted light signal to form a reflected light signal, and then the eye-tracking module 10 receiving the reflected light signal and generating a detection signal sample, so as to use the detection signal sample as calibration information to train the second initial model.

[0319] Furthermore, after generating the detection signal, light intensity distribution information corresponding to the reflected light signal can be generated based on the detection signal. For example, the light intensity distribution information can be generated by the second initial model or the calibration algorithm model obtained after training based on the detection signal.

[0320] Specifically, the calibration algorithm model also uses the detection signal as its input and outputs light intensity distribution information. However, based on the training of the second initial model, with the same detection signal as the model input, the calibration algorithm model outputs more accurate light intensity distribution information.

[0321] It should be noted that model training greatly improves the accuracy of the trained model (i.e., the calibration algorithm model) in determining light intensity distribution information based on the detection signal, making the light intensity distribution information more closely match the wearer's actual eye movement behavior. For example, the smart device 1, eye-tracking module 10, or other devices / modules capable of acquiring detection signals provided in the aforementioned embodiments of this application can all use the trained calibration algorithm model to determine the corresponding calibration light intensity distribution information based on the detection signal to be calibrated, thereby determining the eye feature information corresponding to the wearer based on the calibration light intensity distribution information. The detection signal to be calibrated is the detection signal obtained by the eye-tracking module 10 when the wearer performs the eye movement behavior to be calibrated.

[0322] It should also be noted that the second initial model can be obtained either by directly creating a second initial model or by calling an existing second initial model. As one example, the second initial model is created by inputting the simulation's detection signals.

[0323] In some embodiments, the process of generating eye-tracking states based on the detection signals in the second initial model includes: analyzing the eye-tracking states using rule engines such as first derivatives and function fitting, and / or analyzing the eye-tracking states using machine learning algorithms such as random forests and convolutional neural networks. The process of building the first initial model based on the model backbone of the second initial model can also be performed using rule engines and / or machine learning algorithms.

[0324] In some implementations, training the second initial model using training samples includes: In the same iteration cycle, the detection signal samples are input into the second initial model so that the second initial model generates light intensity distribution information samples based on the detection signal samples; the light intensity distribution error between the light intensity distribution information samples corresponding to the same target eye movement behavior and the standard light intensity distribution information is determined, and the model parameters of the second initial model are optimized based on the light intensity distribution error.

[0325] The light intensity distribution information sample includes standard light intensities corresponding to at least two different areas on the wearer's eye. It should be noted that these different areas can be completely different areas or partially overlapping areas.

[0326] For example, the light intensity distribution error can be referred to as follows: Figure 37 The error map shown corresponds to a specific location on the wearer's eye. The pixel value of each pixel represents the error between the standard light intensity and the sample light intensity at that location, i.e., the error distribution. It should be noted that the optimization goal of optimizing the model parameters of the second initial model based on the light intensity distribution error is to make the error represented by each pixel on the error map approach 0.

[0327] In other embodiments, S923 specifically includes training the second initial model using training samples: in the same iteration cycle, inputting the detection signal samples into the second initial model so that the second initial model generates light intensity distribution information samples based on the detection signal samples, then comparing the standard light intensity distribution information with the light intensity distribution information samples to obtain the light intensity distribution error between the standard light intensity distribution information and the light intensity distribution information samples, then analyzing the light intensity distribution error through a rule engine and / or machine learning algorithm, and then determining whether the light intensity distribution error meets the calibration requirements. If it does not meet the calibration requirements, then the model parameters of the second initial model are optimized based on the analysis results, and the next iteration cycle is entered. If it meets the calibration requirements, then the optimization is stopped, and the calibration algorithm model is obtained.

[0328] In some implementations, the model parameters of the second initial model are optimized based on the light intensity distribution error, including: Establish an objective function relating model parameters to light intensity distribution error; Adjust the model parameters to minimize the light intensity distribution error obtained by inputting the adjusted model parameters into the objective function.

[0329] It should be noted that the second initial model generates corresponding model outputs based on its model parameters and inputs, thereby generating light intensity distribution information samples from the detected signal samples. Therefore, under different model parameters, the light intensity distribution information samples and corresponding light intensity distribution errors obtained by inputting the same set of training samples into the model to be trained will also differ. Based on this, in the process of optimizing the model parameters of the second initial model, a target function is specifically established between the model parameters and the light intensity distribution error. The input of the target function is the model parameters, and the output of the target function is the light intensity distribution error. Then, the model parameters of the model to be trained are adjusted with the goal of minimizing the eye-tracking error corresponding to the current detected signal sample.

[0330] In some implementations, the training samples include detection signal samples corresponding to at least two different target eye movements and standard light intensity distribution information. Specifically, at least two different sets of training samples can be used to train the model to be trained, and one set of training samples can be used for model training in each iteration cycle.

[0331] As one embodiment, training the second initial model using training samples further includes: The model parameters of the second initial model are iteratively optimized for at least two iteration cycles.

[0332] Specifically, it should be noted that, under the condition that the training objective has not been achieved, the intelligent device 1 optimizes the parameters in the second initial model and enters the next iteration cycle to continue to optimize the parameters of the second initial model; after the training objective is achieved, the parameter optimization of the second initial model can be stopped and it can be output as an eye-tracking algorithm model.

[0333] In some implementations, the model parameters of the second initial model are iteratively optimized for at least two iteration cycles, including: When the light intensity distribution error meets the preset training target in any iteration cycle, the model training is completed and the calibration algorithm model is obtained.

[0334] As one embodiment, the preset training objective of the second initial model may be: the light intensity distribution error corresponding to at least the last two consecutive iteration cycles of the sequence reaches the convergence condition, or the eye movement error light intensity distribution error corresponding to a certain iteration cycle reaches a preset error threshold.

[0335] In some embodiments, the smart device 1 further includes a display module 30, and the display module 30 forms a display interface, the target eye movement behavior including: gazing at a target calibration point on the display interface; As one embodiment, the calibration algorithm model generation method further includes: It outputs eye-tracking commands so that the wearer can fixate on the target calibration point based on the eye-tracking commands.

[0336] like Figure 32 As shown, the training of the second initial model using smart device 1 will be illustrated as an example: The display module 30 (e.g., display screen) of smart device 1 displays... Figure 32 The calibration image shown has multiple calibration points (each E represents a calibration point). It outputs eye-tracking commands to instruct the wearer to actively look at the corresponding calibration point to perform the corresponding target eye-tracking behavior. It controls the eye-tracking module 10 to output emitted light signals to the wearer's eyes and output emitted light signal samples. It receives reflected light signal samples formed by the reflected emitted light signals from the eyes and generates detection signal samples.

[0337] For example, in conjunction with the calibration image, the target eye movement behavior can refer to looking at any calibration point in the calibration image, such as looking at the calibration point with coordinates (1,1), looking at the calibration point with coordinates (1,2), or looking at the calibration point with coordinates (1,3), and so on.

[0338] After generating the detection signal samples, the target eye movement behavior and the corresponding detection signal samples are associated as a set of training samples, and then the second initial model is trained based on at least one set of training samples.

[0339] In some implementations, eye-tracking commands are output, including: Highlight the target calibration point on the display interface; and / or output the voice command corresponding to the target calibration point.

[0340] In some implementations, the training samples include real training samples and virtual training samples. Real training samples include detection signal samples corresponding to at least one target calibration point and standard light intensity distribution information. Virtual training samples include detection signal samples corresponding to at least one virtual calibration point and standard light intensity distribution information, where the virtual calibration point is a point on the display interface that differs from the target calibration point. Before training the second initial model using the training samples, the process further includes: Virtual training samples are generated based on real training samples.

[0341] It should be noted that the real training samples are those obtained by the wearer performing actual target eye movements. Specifically, the real training samples include detection signal samples corresponding to at least one target calibration point and standard light intensity distribution information. The detection signal samples are obtained as follows: the wearer looks at the target calibration point, the eye-tracking module 10 emits a light signal to the wearer's eye, receives the reflected light signal formed by the reflected light signal from the eye, and generates a detection signal. To increase the number of training samples and reduce the cost of obtaining them, different virtual training samples can be formed by collecting any appropriate number of real training samples. This allows the virtual calibration point data to be supplemented with less target calibration point data, thus reducing the number of actual sampling points.

[0342] For example, a virtual sampling point can be a point between two adjacent target sampling points, and there can be one, two or more virtual sampling points between two adjacent target sampling points.

[0343] Specifically, generating virtual training samples from real training samples can be achieved by using a rule engine and / or machine learning algorithms to fit and generate multiple virtual calibration points, along with the corresponding detection signal samples and standard light intensity distribution information for each virtual calibration point. For example, a small number of target calibration points can be sampled to obtain real training samples; specifically, the number of target calibration points can be 4, 9, 16, or any other suitable n. 2 The calibration algorithm model generates multiple virtual output calibration points through rule engine and / or machine learning algorithms. The specific number of output calibration points can be 81, 100, or any other suitable number. Generating virtual training samples based on real training samples reduces the number of sampling points required for calibration, expands the calibration area, and improves calibration accuracy.

[0344] Furthermore, after generating virtual training samples based on real training samples, a calibration database (including real training samples and virtual training samples) can be established based on the training samples, and the calibration database can be used to calibrate the wearer's eye features.

[0345] This application embodiment also provides a calibration method. The eye-tracking calibration method is applied to a smart device 1, which is worn on the wearer's head and includes an eye-tracking module 10. The eye-tracking module 10 is configured to output emitted light signals to the wearer's eyes, receive reflected light signals corresponding to at least one target area in the eye, and generate a detection signal. The reflected light signals are formed by the reflected emitted light signals of the corresponding target area. The method includes: The calibration algorithm model is invoked, and the calibration algorithm model is generated using the calibration algorithm model generation method provided in any embodiment of this application; When the wearer performs the eye movement behavior to be calibrated, the eye movement module 10 is used to acquire the calibration detection signal corresponding to the eye movement behavior to be calibrated. The detection signal to be calibrated is input into the calibration algorithm model so that the eye-tracking algorithm model generates calibration light intensity distribution information corresponding to the eye movement behavior to be calibrated based on the detection signal to be calibrated. The calibration light intensity distribution information includes the intensity threshold of the reflected light signal corresponding to at least one target area. The eye feature information corresponding to the wearer is determined based on the calibrated light intensity distribution information; The location of the target area is matched with eye feature information.

[0346] For example, eye feature information includes interpupillary distance and pupillary height.

[0347] In some embodiments, during calibration, the wearer first wears the smart device 11 and gazes at the calibration point. The calibration algorithm model then acquires calibration light intensity distribution information and the wearer's eye feature information. The eye-tracking module 10 then operates based on the actual eye feature information (e.g., outputting emitted light signals, receiving emitted light signals, and outputting detection signals). The application of the calibration algorithm model reduces eye-tracking errors and improves the adaptability of the eye-tracking module 10 and the smart device 1 to different wearers.

[0348] In some embodiments, the calibration of smart device 1 includes static calibration and dynamic calibration. Static calibration refers to calibration performed before the wearer wears smart device 1 and before using smart device 1, while dynamic calibration refers to calibration performed during the wearer's use of smart device 1. The parameter processing, verification, and optimization mentioned in this application can all be handled by any suitable rule engine, machine learning algorithm, or other suitable method, and no limitation is imposed here.

[0349] This application also provides a storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the calibration algorithm model generation method and / or the calibration method provided in any embodiment of this application.

[0350] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware embodiments, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as setup circuitry, such as dedicated setup circuitry. Such software can be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0351] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, referring to a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection via an intermediate medium; or a connection within two elements. Those skilled in the art will understand the specific meaning of the above terms in this application based on the specific circumstances. As used in this specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a," "an," and "the" are intended to include the plural forms.

[0352] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. The character " / " generally indicates that the preceding and following related objects are in an "and / or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, "at least one of A, B and C" includes A, B, C, AB, AC, BC, or ABC. And, unless otherwise specified, the ordinal numbers such as "first," "second," etc., mentioned in the embodiments of this application are used to distinguish multiple objects and are not used to limit the order, sequence, priority, or importance of multiple objects. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or system.

Claims

1. An eye-tracking method, characterized in that, The method includes: During the detection period, a light signal is emitted to the target area of ​​the wearer's eyes during the detection time period, and no light signal is emitted to the target area of ​​the wearer's eyes during the control time period. The detection time period and the control time period do not overlap. It receives reflected light signals and outputs detection signals based on the reflected light signals, which are formed by the eye reflecting the emitted light signals; The system receives a reference reflected light signal and outputs a reference signal based on the reference reflected light signal, which is formed by the eye reflecting ambient light. The eye movement state of the wearer's eyes during at least the detection time period is obtained based on the detection signal and the reference signal.

2. The eye-tracking method as described in claim 1, characterized in that, The step of obtaining the wearer's eye movement state during the detection period based on the detection signal and the reference signal includes: The corresponding detection light intensity is determined based on the detection signal; The reference light intensity corresponding to the ambient light is determined based on the reference signal; The detected light intensity and the reference light intensity are differentially processed to obtain the target light intensity corresponding to the reflected light signal, and the eye movement state of the wearer's eyes during the detection time period is obtained based on the target light intensity.

3. The eye-tracking method as described in claim 2, characterized in that, The step of obtaining the wearer's eye movement state during the detection time period based on the target light intensity includes: When the target light intensity does not exceed a preset intensity threshold, it is determined that the position of the wearer's pupil overlaps with the target area during the detection time period; When the target light intensity exceeds the preset intensity threshold, it is determined that the wearer's pupil position does not overlap with the target area during the detection time period.

4. The eye-tracking method as described in claim 1, characterized in that, Within the detection period, the duration of the data acquisition time period is less than or equal to the duration of the detection time period.

5. The eye-tracking method as described in claim 1, characterized in that, The step of outputting an emitted light signal to the target area of ​​the wearer's eye during the detection period includes: During the detection period, emitted light signals are output to different target areas of the same eye of the wearer at different times.

6. The eye-tracking method as described in claim 5, characterized in that, The step of outputting emitted light signals to different target areas of the same eye of the wearer at different times during the detection period includes: Within the same detection cycle, a first emitted light signal is output to a first target area in a first time period, and a second emitted light signal is output to a second target area in a second time period; The first target area and the second target area are different positions of the same eye of the wearer, and the first time period and the second time period do not overlap.

7. The eye-tracking method as described in claim 5, characterized in that, The receiving of the reflected light signal and the output of a detection signal based on the reflected light signal include: The target area generates a first reflected light signal and a second reflected light signal. The first reflected light signal and the second reflected light signal are received by the same optical sensor, and the detection signal corresponding to the first reflected light signal and the second reflected light signal is output.

8. The eye-tracking method as described in claim 1, characterized in that, The step of outputting emitted light signals to the target area of ​​the wearer's eyes during the detection period includes: A first emitted light signal and a second emitted light signal are output to the target area of ​​the wearer's eye to form a first reflected light signal and a second reflected light signal; The receiving of the reflected light signal and the output of a detection signal based on the reflected light signal include: The first reflected light signal is received using a first optical sensor, and a first detection signal is output. The first reflected light signal is received using a second optical sensor, and a second detection signal is output. The first optical sensor and the second optical sensor are different optical sensors.

9. An eye-tracking device, characterized in that, The system includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for establishing communication between the processor and the memory, wherein when the computer program is executed by the processor, it implements the steps of the eye-tracking method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to implement the eye-tracking method as described in any one of claims 1 to 8.

Images