Eye movement calibration method, device and storage medium

Abstract

The application relates to the technical field of head-mounted smart devices, and discloses an eye movement calibration method and device and a storage medium. The method comprises the following steps: when a wearer performs a to-be-calibrated eye movement behavior, real calibration information of at least one real calibration point corresponding to the to-be-calibrated eye movement behavior is collected; based on the real calibration information, a calibration algorithm model is used to generate virtual calibration information of at least one virtual calibration point; and actual parameters corresponding to the wearer are determined according to the real calibration information and the virtual calibration information.

Description

Technical Field

[0001] This application relates to the field of head-mounted smart device technology, and in particular to an eye-tracking calibration method, device and storage medium. Background Technology

[0002] The application of head-mounted smart devices is becoming more and more widespread. For example, AR (Augmented Reality) glasses can provide wearers with many convenient functions. When wearers interact with AR glasses, they can interact through eye movements. Users only need to look at virtual objects to select or control them, achieving a more natural and efficient interactive experience.

[0003] In some head-mounted smart devices, tracking the wearer's eye movement mainly relies on identifying and locating the wearer's pupil position (i.e., eye tracking calibration). During the eye tracking calibration process, the wearer needs to perform the target eye movement behavior when collecting calibration information, which makes it impossible to efficiently collect a large amount of calibration information. This makes it difficult for smart devices to be compatible with identifying and locating the pupil position of different wearers, affecting the accuracy of subsequent eye tracking. Summary of the Invention

[0004] This application provides an eye-tracking calibration method, including:

[0005] When the wearer performs the eye movement behavior to be calibrated, real calibration information of at least one real calibration point corresponding to the eye movement behavior to be calibrated is collected;

[0006] Based on the real calibration information, a calibration algorithm model is used to generate virtual calibration information for at least one virtual calibration point.

[0007] Based on the real calibration information and the virtual calibration information, the actual parameters corresponding to the wearer are determined.

[0008] Secondly, embodiments of this application also provide an eye-tracking calibration device, including a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for implementing communication between the processor and the memory, wherein when the computer program is executed by the processor, it implements the steps of any of the eye-tracking calibration methods provided in embodiments of this application.

[0009] Thirdly, embodiments of this application also provide a storage medium storing a computer program, which, when executed by a processor, causes the processor to implement any of the eye-tracking calibration methods provided in embodiments of this application. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a modular structure diagram of a smart device provided in one or more embodiments of this application;

[0012] Figure 2 This is a module structure diagram of an eye-tracking module provided in one or more embodiments of this application;

[0013] Figure 3 This is a schematic diagram illustrating the eye movement detection effect of an eye movement module provided in one or more embodiments of this application;

[0014] Figure 4 This is a schematic diagram of the structure of an eye-tracking module provided in one or more embodiments of this application;

[0015] Figure 5 This is a schematic diagram of the signal strength received by the light receiving module in an eye-tracking module according to one or more embodiments of this application;

[0016] Figure 6 This is a schematic diagram of the structure of another eye-tracking module provided in one or more embodiments of this application;

[0017] Figure 7 This is a schematic diagram of the signal strength received by the light receiving module in another eye-tracking module provided in one or more embodiments of this application;

[0018] Figure 8 This is a schematic diagram of the signal strength received by the light receiving module in another eye-tracking module provided in one or more embodiments of this application;

[0019] Figure 9 This is a schematic diagram of the signal strength received by the light receiving module in another eye-tracking module provided in one or more embodiments of this application;

[0020] Figure 10 This is a schematic diagram of the signal strength received by the light receiving module in another eye-tracking module provided in one or more embodiments of this application;

[0021] Figure 11 This is a schematic diagram showing the arrangement of different time periods in an eye-tracking module provided in one or more embodiments of this application;

[0022] Figure 12 This is a schematic diagram of the arrangement of light output components in an eye-tracking module according to one or more embodiments of this application;

[0023] Figure 13 This is a schematic diagram of the target area arrangement of an eye-tracking module provided in one or more embodiments of this application;

[0024] Figure 14 This application provides a modular structure for a smart device according to one or more embodiments;

[0025] Figure 15 This is a module structure diagram of another smart device provided in one or more embodiments of this application;

[0026] Figure 16 This is a schematic diagram of the structure of a smart device provided in one or more embodiments of this application;

[0027] Figure 17 This is a schematic diagram of the target area arrangement of another eye-tracking module provided in one or more embodiments of this application;

[0028] Figure 18 This is a schematic diagram of the signal strength received by the light receiving module in another eye-tracking module provided in one or more embodiments of this application;

[0029] Figure 19 This is a schematic diagram of the signal strength received by the light receiving module in another eye-tracking module provided in one or more embodiments of this application;

[0030] Figure 20 This is a schematic flowchart of the eye-tracking calibration method provided in one or more embodiments of this application;

[0031] Figure 21 This is a schematic diagram of the steps for training the calibration algorithm model provided in one or more embodiments of this application;

[0032] Figure 22 This is a schematic diagram of the steps for eye calibration and eye tracking provided in one or more embodiments of this application;

[0033] Figure 23 This is a modular structure diagram of an eye-tracking calibration device provided in one or more embodiments of this application.

[0034] Figure label:

[0035] 1. Smart devices;

[0036] 10. Eye-tracking module; 11. Light output module; 12. Light output component; 121. First light output component; 122. Second light output component; 123. Third light output component; 124. Light emitter; 125. First optical element; 13. Light receiving module; 131. Light sensor; 132. Second optical element; 133. First light sensor; 134. Second light sensor; 14. First eye-tracking module; 15. Second eye-tracking module;

[0037] 20. Controller;

[0038] 30. Display module;

[0039] 40. Target area; 41. First target area; 42. Second target area; 43. Third target area;

[0040] 50. Wearing bracket; 51. Eyeglass frame; 52. Front frame; 521. Display mounting part; 522. Temple connection part; 523. Receiving chamber; 53. Temple piece;

[0041] 60. Projection module;

[0042] T1, first testing cycle; T2, second testing cycle. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Unless otherwise specified, the embodiments of this application can be combined with each other. Other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, combined, or partially merged; therefore, the actual execution order may change according to the actual situation. The above descriptions are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0044] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0045] It should be noted that the eye-tracking method provided in this application embodiment can be applied to, but is not limited to, the eye-tracking module 10 and the smart device 1. The eye-tracking module 10 will be described in detail below:

[0046] Please see Figures 1 to 4 , Figure 1 This is a module structure diagram of a smart device provided for one or more embodiments of this application. Figure 2 This application provides a module structure diagram of an eye-tracking module according to one or more embodiments. Figure 3 This is a schematic diagram illustrating the eye movement detection effect of an eye movement module provided in one or more embodiments of this application. Figure 4 This is a schematic diagram of the structure of an eye-tracking module provided for one or more embodiments of this application.

[0047] like Figures 1 to 4 As shown, this application embodiment provides an eye-tracking module 10 for a smart device 1. The smart device 1 is wearable on the wearer's head. The smart device 1 can be a head-mounted display device, such as a VR (Virtual Reality) display device, an AR (Augmented Reality) display device, an XR (Extender Reality) display device, or an MR (Mixed Reality) display device, or other wearable devices. By way of example and not limitation, in one embodiment, the head-mounted display device can be smart glasses, such as VR glasses, AR glasses, XR glasses, MR glasses, etc.

[0048] It should be noted that during use, the smart device 1 can monitor the wearer's eye movement state, namely the wearer's gaze direction and gaze target, through the eye-tracking module 10, and then perform corresponding operations based on the wearer's eye movement state to realize intelligent interaction between the smart device 1 and the wearer, thereby improving the wearer's user experience.

[0049] For example, the smart glasses are equipped with a display module 30 that corresponds to the wearer's settings. The display module 30 is configured to output an image screen. The wearer can select or manipulate the virtual object by looking at it in the image screen, thus achieving a more natural and efficient interactive experience. The above-mentioned interaction method has wide applications in scenarios such as games and education.

[0050] In some embodiments, in addition to detecting the wearer's eye movement status through the eye-tracking module 10, the smart device 1 can also monitor the wearer's eye health based on the eye movement status, helping to detect problems such as dry eye syndrome and eye fatigue.

[0051] The eye-tracking module 10 in this embodiment includes a light output module 11 and a light receiving module 13. The structure and function of the light output module 11 and the light receiving module 13 will be described in detail below.

[0052] Specifically, the light output module 11 includes a light output component 12, which can be used to output emitted light signals. The light output module 11 is configured to output emitted light signals to different target areas 40 of the same eye of the wearer through the light output component 12 during the detection period. After the emitted light signal is directed to the wearer's eye, the wearer's eye will emit the emitted light signal, thereby forming a reflected light signal.

[0053] The light receiving module 13 is configured to receive the reflected light signal formed by the light emitted by the eye and output a detection signal.

[0054] It should be noted that the detection signal output by the light receiving module 13 is matched with the reflected light signal. Based on the detection signal, optical features related to the reflected light signal can be identified, such as the signal intensity (light intensity) of the reflected light signal. Thus, the motion state of the wearer's eyes during the detection period can be further analyzed based on the reflected light signal.

[0055] To further explain, due to the differences in light reflectivity among different areas of the human eye (such as the pupil, iris, and sclera), the direction and state of eye movement can be inferred by measuring the dynamic changes in reflected light intensity. The pupil absorbs most light and has extremely low reflectivity (approximately 4%), thus appearing as a dark area under infrared light. The reflectivity of the iris and sclera increases sequentially (the iris is approximately 10%-20%, and the sclera is even higher), with the iris's reflectivity depending on its color (dark irises have even lower reflectivity). The sclera's reflectivity can reach 30-40%, appearing as a bright area under infrared light. Therefore, when the eye moves, the changes in the relative positions of the pupil, iris, and sclera within the eye socket lead to changes in the distribution of reflected light intensity.

[0056] Specifically, when the eyeball moves, the relative positions of the pupil, iris, and sclera within the eye socket change, causing dynamic changes in the intensity of reflected light received by the infrared sensor. On the other hand, changes in the light intensity received by the eyeball and the gaze target also cause the pupil to dilate and contract, leading to a relative decrease or increase in the size of the iris. Therefore, a combination of one or more of these factors results in changes in the relative area of ​​the pupil, iris, and sclera within the target area 40, thereby causing changes in the reflected light signal. During the detection period, the light output component 12 emits light signals to different target areas 40 of the same eye at different time intervals. These emitted light signals are reflected by the wearer's eye, forming corresponding reflected light signals that are collected by the light receiving module 13. The light receiving module 13 then outputs a corresponding detection signal, which can be used to determine the relative relationship between the pupil and different target areas 40. For example, the detection signal can be used to determine whether there is an overlap between the pupil and a certain target area 40, thus determining the pupil's position information. By tracking changes in the pupil's position, the wearer's eye movement state can be obtained.

[0057] like Figure 5 As shown, in one embodiment, during the detection period, the light output module 11 outputs a light signal to the right side of the wearer's eyeball in the first time period and outputs a light signal to the left side of the wearer's eyeball in the second time period. If the wearer's eyeball turns horizontally to the right during the detection period: the pupil, iris, and sclera move to the right, and the right half of the sclera is partially or completely covered by the eye socket. The iris, pupil, and sclera deflect to the right relative to the eye socket, and the right half or all of the sclera is hidden in the eye socket. The light intensity received by the light receiving module 13 in the first time period decreases. The left half of the iris and sclera is exposed, and the light intensity received by the light receiving module 13 in the second time period increases.

[0058] For example, during the detection period, the light output module 11 emits a light signal to the upper side of the wearer's eyeball in the first time period and to the lower side of the wearer's eyeball in the second time period. If the wearer's eyeball turns upward during the detection period, the iris, pupil, and sclera deflect upward relative to the eye socket, and part or all of the sclera is blocked by the eye socket. Therefore, the light intensity received by the light receiving module 13 decreases in the first time period and increases in the second time period. When the wearer's eyeball turns downward, the pupil, iris, and sclera deflect downward relative to the eye socket, and part or all of the lower sclera is blocked by the eye socket. Therefore, the light intensity received by the light receiving module 13 increases in the first time period and decreases in the second time period.

[0059] Compared to related technologies that use multiple light output components 12 to continuously output emitted light signals to different target areas 40 of the eye, the eye-tracking module 1 provided in this application provides emitted light signals to different target areas 40 of the same eye of the wearer in a time-division multiplexing manner. This effectively reduces power consumption and can identify the wearer's eye movement state based on the corresponding emitted light signals and detection signals, thereby achieving eye tracking of the wearer.

[0060] In some embodiments, the light output module 11 includes at least a first light output component 121 and a second light output component 122. The first light output component 121 is configured to output a first emitted light signal to a first target region 41, and the second light output component 122 is configured to output a second emitted light signal to a second target region 42. The first target region 41 and the second target region 42 are different positions of the same eye of the wearer, and the first target region 41 and the second target region 42 do not overlap at least partially.

[0061] Specifically, the first light output component 121 and the second light output component 122 have different light output angles and / or are located in different positions.

[0062] As one embodiment, the first light output component 121 and the second light output component 122 have different light output angles, and the first light output component 121 and the second light output component 122 are configured to emit light to the wearer's eyes from different directions.

[0063] For example, the application of eye-tracking module 10 to smart device 1 is described. Smart device 1 includes wearable bracket 50 and the aforementioned eye-tracking module 10. The first light output component 121 and the second light output component 122 of eye-tracking module 10 are disposed on wearable bracket 50. When the wearer wears smart device 1 through wearable bracket 50, the first light output component 121 and the second light output component 122 are located at different positions of the wearer's same eye and are configured to output emitted light signals to different target areas 40 of the eye.

[0064] like Figures 5 to 10 As shown, in some embodiments, within the same detection cycle, the first optical output component 121 is configured to output a first emitted optical signal in a first time period, and the second optical output component 122 is configured to output a second emitted optical signal in a second time period.

[0065] The first and second time periods do not overlap. It should be noted that the lengths of the first and second time periods can be the same or different. Furthermore, the lengths of the individual first time periods can be the same or different; similarly, the lengths of the individual second time periods can be the same or different.

[0066] Specifically, the first optical output component 121 outputs a first emitted optical signal during a first time period within the detection cycle, and the second optical output component 122 outputs a second emitted optical signal during a second time period within the detection cycle.

[0067] The eye-tracking module 10 provided in this application provides time-division multiplexing to output emitted light signals to different target areas 40 of the same eye of the wearer. Therefore, when processing the detection signals output by the light receiving module, it can effectively distinguish the information corresponding to different target areas 40 in the eye based on the first time period and the second time period, thereby realizing eye tracking of the wearer. Moreover, compared with related technologies, this application embodiment outputs emitted light signals by different light output components 12 in a time-division multiplexing manner. When performing eye tracking, it is no longer necessary for multiple light output components 12 to continuously output emitted light signals. The time-division multiplexing method effectively reduces the overall power consumption of the light output module 11. In particular, for the smart device 1 where the eye-tracking module 10 is located, the reduction in power consumption of the eye-tracking module 10 also makes the smart device 1 have better battery life.

[0068] like Figures 6 to 10 As shown, in one embodiment, the smart device 1 includes two eye-tracking modules 10, which correspond to the wearer's left and right eyes respectively, to detect the eye movement state of the wearer's left and right eyes. Specifically, the two modules are respectively set to correspond to the wearer's first and second eyes, and the two modules can be modules with the same configuration or modules with different configurations.

[0069] Specifically, when the eyeball moves, the relative positions of the pupil, iris, and sclera within the eye socket change. Alternatively, changes in the intensity of light received by the eyeball and the target being gazed upon cause the pupil to dilate and constrict, all of which dynamically change the intensity of reflected light received by the infrared sensor. For example, when the wearer's eyeball turns horizontally to the right (to clarify, in a normal wearer, the turning of the left and right eyeballs is consistent; the same applies to turning to the left, upward, and downward): the pupil, iris, and sclera move to the right, partially or completely obscuring the right half of the sclera within the eye socket, thus reducing the light intensity received by the light receiving module 13 of the right eye-tracking module 10; the left half of the iris / sclera is exposed, increasing the light intensity received by the light receiving module 13 of the left eye-tracking module 10.

[0070] For example, during the detection period, the light output module 11 emits a light signal to the upper side of the wearer's eyeball in the first time period and to the lower side of the wearer's eyeball in the second time period. If the wearer's eyeball turns upward during the detection period, since the eyeball rotates as a whole, the iris, pupil, and sclera deflect upward relative to the eye socket, and the upper part or all of the sclera is hidden inside the eye socket. The light intensity received by the light receiving modules 13 of both eye movement modules 10 will decrease in the first time period and increase in the second time period. When the wearer's eyeball turns downward, the iris, pupil, and sclera deflect downward relative to the eye socket, and the lower part or all of the sclera is hidden inside the eye socket. The light intensity received by the light receiving modules 13 of both eye movement modules 10 will increase in the first time period and decrease in the second time period.

[0071] For example, if the wearer's pupils dilate during the detection period, increasing the pupillary area, the light intensity received by the light-receiving modules 13 of both eye-tracking modules 10 will decrease in both the first and second time periods. Conversely, if the wearer's pupils constrict during the detection period, decreasing the pupillary area, the light intensity received by the light-receiving modules 13 of both eye-tracking modules 10 will increase in both the first and second time periods. It is understandable that the pupils constrict when the wearer actively looks at something close or when the light intensity received by the eyes increases, and the pupils dilate when the light intensity received by the wearer's eyes decreases.

[0072] like Figure 6 As shown, in one embodiment, the eye-tracking module 10 on the left is the first eye-tracking module 14, and the eye-tracking module 10 on the right is the second eye-tracking module 15.

[0073] In the first eye-tracking module 14, the first target area 41 corresponding to the first light output component 121 is located on the upper left side of the left eye, and the second target area 42 corresponding to the second light output component 122 is located on the lower left side of the left eye; in the second eye-tracking module 15, the first target area 41 corresponding to the first light output component 121 is located on the upper right side of the left eye, and the second target area 42 corresponding to the second light output component 122 is located on the lower right side of the left eye, and when the wearer looks straight ahead horizontally, the pupils on the left and right sides do not coincide with the first target area 41 and the second target area 42.

[0074] like Figure 7 As shown, during the detection period, the eyeball turns to the upper left (the turning of the left and right eyeballs is consistent). The light intensity received in the first eye movement module 14 during the first time period decreases, while the light intensity received during the second time period remains unchanged. In the second eye movement module 15, the light intensity received during the first time period remains unchanged, and the light intensity received during the second time period remains unchanged.

[0075] like Figure 8As shown, during the detection period, the eyeball turns to the lower left. The light intensity received by the first eye movement module 14 in the first time period remains unchanged, while the light intensity received in the second time period decreases. The light intensity received by the second eye movement module 15 in the first time period remains unchanged, while the light intensity received in the second time period remains unchanged.

[0076] like Figure 9 As shown, during the detection period, the eyeball turns to the upper right. The light intensity received by the second eye movement module 15 in the first time period decreases, while the light intensity received in the second time period remains unchanged. The light intensity received by the first eye movement module 14 in the first time period remains unchanged, while the light intensity received in the second time period remains unchanged.

[0077] like Figure 10 As shown, during the detection period, the eyeball turns to the right and downward. The light intensity received by the second eye movement module 15 in the first time period remains unchanged, while the light intensity received in the second time period decreases. The light intensity received by the first eye movement module 14 in the first time period remains unchanged, while the light intensity received in the second time period remains unchanged.

[0078] Therefore, based on the two eye-tracking modules 10 that are set up to correspond to the wearer's left and right eyes, the eye movement state of the wearer's eyeballs can be accurately and completely detected during the detection period.

[0079] like Figure 12 As shown, in one embodiment, the smart device 1 also includes a display module 30, which is set to correspond to the wearer's eyes and configured to display images to the wearer's eyes. The first light output component 121 and the second light output component 122 are located on different sides of the display module 30. When the wearer wears the smart device 1 through the wearable bracket 50, the first light output component 121 is located on the upper side of the display module 30, that is, above the wearer's eyeballs, and the second light output component 122 is located on the upper side of the display module 30, that is, below the wearer's eyeballs, so as to output emitted light signals to different target areas 40 of the eyes from different angles. The wearable bracket 50 is, for example, an eyeglass bracket 51, and the display module 30 is, for example, a display lens or a display screen.

[0080] As one embodiment, the light receiving module 13 is disposed between the first light output component 121 and the second light output component 122. For example, the light receiving module 13 includes one or more light sensors 131. Taking only one light sensor 131 as an example, the light sensor 131 is disposed between the first light output component 121 and the second light output component 122 and is configured to receive a first reflected light signal and a second reflected light signal, and output a first detection signal and a second detection signal.

[0081] In some embodiments, the light output module 11 further includes a greater number of light output components 12, and different light output components 12 have different light emission angles. These light output components 12 are configured to output emitted light signals to different target areas 40 of the same eye of the wearer in a time-division manner. The number of light output components 12 can be set according to actual needs.

[0082] like Figure 13 As shown, when the light output module 11 has three or more light output components 12, adjacent corresponding target areas may not overlap or may partially overlap.

[0083] Furthermore, the optical output module 11 also includes a greater number of optical output components 12, such as a third optical output component 123 that is different from the first optical output component 121 and the second optical output component 122.

[0084] like Figure 12 As shown, the light receiving module 13 includes at least a light sensor 131. For example, when the light output module 11 has only two light output components 12, these two light output components 12 can be symmetrically distributed about the light sensor 131 as an axis. Figure 12 The arrangement of the first optical output component 121 and the second optical output component 122 shown in (a).

[0085] When the light output module 11 is provided with multiple light output components 12, these multiple light output components 12 can be distributed circumferentially around the light sensor 131, for example... Figure 12 The first light output component 121, the second light output component 122 and the third light output component 112 shown in (b) are arranged in a triangle with the light sensor 131 as the center, so as to output corresponding emitted light signals to the same eye of the wearer from different angles, thereby obtaining a more comprehensive and accurate eye movement detection effect.

[0086] It should be noted that the symmetrical distribution, circumferential distribution, and triangular distribution mentioned in this article are all positional descriptions based on the wearer's head from a lateral perspective.

[0087] In some embodiments, depending on the process, structure, or light output requirements, the individual light output components 12 may also be asymmetrically distributed, which is not limited here.

[0088] In some embodiments, the light receiving module 13 is configured to receive a first reflected light signal and output a first detection signal, wherein the first reflected light signal is formed by the eye reflecting a first emitted light signal;

[0089] The system receives the second reflected light signal and outputs the second detection signal. The second reflected light signal is formed by the eye reflecting the second emitted light signal.

[0090] It should be noted that in the eye-tracking module 10 provided in this application embodiment, the light receiving module 13 actually continuously receives light signals directed at it when it is working, which includes reflected light signals reflected back from each target area 40 in the glasses; the light receiving module 13 is also used to output corresponding detection signals based on the received reflected light signals, the detection signals include a first detection signal corresponding to the first reflected light signal and a second detection signal corresponding to the second reflected light signal.

[0091] like Figure 3 As shown in (a), in some embodiments, the optical receiving module 13 includes an optical sensor 131, and the first detection signal and the second detection signal are output by the same optical sensor 131.

[0092] Specifically, the optical sensor 131 is used to receive a first reflected light signal and output a first detection signal corresponding to the first reflected light signal, and is also used to receive a second reflected light signal and output a second detection signal corresponding to the second reflected light signal.

[0093] As one embodiment, the light output module 11 includes a first light output component 121 and a second light output component 122, and the light receiving module 13 may include only one light sensor 131. Both the first reflected light signal and the second reflected light signal are configured to be received by the light sensor 131. After receiving the first reflected light signal, the light sensor 131 outputs a first detection signal corresponding to the first reflected light signal, and after receiving the second reflected light signal, it outputs a second detection signal corresponding to the second reflected light signal.

[0094] For example, the eye-tracking module 10 is applied to a smart device 1 (smart glasses). The smart device 1 includes a glasses frame 51 and the aforementioned eye-tracking module 10. The first light output component 121 and the second light output component 122 are located at different positions of the same eye of the wearer and are configured to output emitted light signals to different target areas 40 of the eye. When the wearer wears the smart device 1 through the glasses frame 51, the first light output component 121 is located above the wearer's eye, the second light output component 122 is located below the wearer's eye, and the light sensor 131 is located between the first light output component 121 and the second light output component 122. The first reflected light signal and the second reflected light signal formed by the wearer's eye can both be received by the light sensor 131.

[0095] In some embodiments, the first light output component 121 and the second light output component 122 are symmetrically distributed about the light sensor 131 as the axis.

[0096] For example, when the light output module 11 has only two light output components 12, the two light output components 12 can be symmetrically distributed with respect to the light sensor 131 as the axis (it can be symmetrically distributed with respect to the horizontal axis passing through the light sensor 131, or with respect to the vertical axis passing through the light sensor 131, or with respect to the inclined axis passing through the light sensor 131).

[0097] The two light output components 12 are symmetrically distributed about a horizontal axis passing through the light sensor 131, as can be seen from, for example... Figure 12 The arrangement of the first light output component 121 and the second light output component 122 is shown in (a). When the light output module 11 has multiple light output components 12, these multiple light output components 12 can be distributed circumferentially around the light sensor 131, for example... Figure 12 The first light output component 121, the second light output component 122 and the third light output component 112 shown in (b) are arranged in a triangle with the light sensor 131 as the center.

[0098] Thus, through clever positioning design, a single optical sensor 131 can receive at least two kinds of reflected light signals to output corresponding detection signals, and the first detection signal and the second detection signal can be distinguished from the detection signals output by the optical sensor 131 according to the first time period corresponding to the first emitted light signal and the second time period corresponding to the second emitted light signal.

[0099] Compared to the related technologies that set up a light sensor 131 for each light output component 12, this application only needs to set up one light sensor 131 in the light receiving module 13 to receive the reflected light signals corresponding to at least two different emitted light signals. This not only reduces the power consumption of the eye-tracking module 10, but also optimizes the overall size and weight of the eye-tracking module 10. In particular, for the smart device 1 where the eye-tracking module 10 is located, the reduction in power consumption of the eye-tracking module 10 also makes the smart device 1 have better battery life.

[0100] In other embodiments, the light receiving module 13 may include two or more light sensors 131, and each light sensor 131 is configured to receive a reflected light signal obtained from the emitted light signal reflected by the eye.

[0101] like Figure 3 As shown in (b) in some embodiments, the optical receiving module 13 includes at least two optical sensors 131, and the first detection signal and the second detection signal are output by the corresponding optical sensors 131 respectively.

[0102] Specifically, the light sensor 131 includes a first light sensor 133 and a second light sensor 134, wherein the first light sensor 133 and the second light sensor 134 are different light sensors 131, and the reflected light signals received by the different light sensors 131 are reflected light signals formed by the eye reflecting different emitted light signals. Moreover, the reflected light signal received by one light sensor 131 can be a reflected light signal formed by the eye reflecting one, two or more kinds of emitted light signals.

[0103] Specifically, the first optical sensor 133 is configured to receive a first reflected light signal and output a first detection signal, and the second optical sensor 134 is configured to receive a second reflected light signal and output a second detection signal.

[0104] In some other embodiments, the optical receiving module includes at least two optical sensors 131, both of which are capable of outputting a first detection signal and a second detection signal.

[0105] For example, the light sensor 131 includes at least two different light sensors, a first light sensor 133 and a second light sensor 134. Both the first light sensor 133 and the second light sensor 134 can output a first detection signal and a second detection signal. The controller 20 can receive and process the first and second detection signals output by the first light sensor 133 and the second light sensor 134. Specifically, the controller 20 can compare or average the detection signals output by the first light sensor 133 and the second light sensor 134, etc., without limitation. As one embodiment, the first light sensor 133 is used to output the first detection signal, and the second light sensor 134 is used to output the second detection signal. As one embodiment, the first light sensor 133 and the second light sensor 134 can simultaneously output the first detection signal and / or simultaneously output the second detection signal.

[0106] For example, both the first light sensor 133 and the second light sensor 134 can be infrared sensors.

[0107] In one embodiment, the first light sensor 133 is positioned corresponding to the first light output component 121, and the second light sensor 134 is positioned corresponding to the first light output component 121.

[0108] For example, along a horizontal axis passing through the center of the wearer's eye, the first light sensor 133 is disposed on the same side as the first light output component 121, and the second light sensor 134 is disposed on the same side as the second light output component 122.

[0109] like Figure 4As shown, in some embodiments, the light output component 12 includes a light emitter 124 and a first optical element 125. The light emitter 124 is used to output emitted light signals. The first optical element 125 is set to correspond to the light output direction of the light emitter 124 and is used to process the emitted light signal output by the light emitter 124 by any one or more of the following methods: convergence, reflection, divergence or refraction, and then direct it toward the target area 40, so that the light can uniformly illuminate the eyeball and avoid local high brightness reflection.

[0110] For example, the light emitter 124 can be a divergent light source or a parallel light source.

[0111] For example, the first optical element 125 includes one or a combination of a prism, a mirror, a lens, a diffuser, an optical coating, and wherein the lens is, for example, one or a combination of a biconvex lens, a plano-convex lens, a convex aspherical lens, and a convex freeform lens.

[0112] To further explain, the light output component 12 may include a combination of a diverging light source and a lens, or a combination of a parallel light source and a prism / reflector / diffuser.

[0113] For example, in some embodiments, the first optical element 125 is a lens. The lens focuses the emitted light signal output from the light emitter 124, making the emitted light signal more concentrated. Especially when the light emitter 124 is a divergent light source, the lens can better converge the divergent emitted light signal into a state closer to parallel light, and directionally output it to the wearer's eyes. Moreover, the directional light output does not require illuminating the entire eye to obtain a relatively clear and reliable reflected light signal, which is convenient for the light receiving module 13 to receive the reflected light signal. In addition, the directional light output allows the light emitter 124 to operate with lower power consumption, improving the overall working efficiency of the eye-tracking module 10.

[0114] In some embodiments, the light output module 11 may be provided with a first optical element 125 to achieve light distribution control, or the light output surface of the light output module 11 may be directly changed to achieve a similar effect as having a first optical element 125.

[0115] In some embodiments, the light output component 12 may be installed at any suitable position on the eye-tracking module 10, and / or the light output component 12 may output emitted light signals at any suitable angle, thereby enabling the emitted light signals to be directly directed to various target areas of the wearer's eyes. Preferably, the light output component 12 may be installed at an angle relative to the display module 30. By adjusting the installation position and angle of the light output component 12, the incident angle of the emitted light signal relative to the eyeball can be adjusted, further simplifying the structure of the eye-tracking module 10 and improving the lightweight nature of the eye-tracking module 10.

[0116] like Figure 4 As shown, in some embodiments, the light receiving module 13 further includes a second optical element 132, which is disposed in the direction corresponding to the light sensor 131 receiving the reflected light signal. The second optical element 132 is configured to process the reflected light signal formed by the wearer's eye in any one or more ways, such as convergence, reflection, divergence or refraction, and then direct it toward the light sensor 131.

[0117] As one embodiment, the second optical element 132 includes one or a combination of a prism, a mirror, a lens, a diffuser, and an optical coating.

[0118] For example, the second optical element 132 includes a convex lens for converging the reflected light signal, thereby concentrating the intensity of the processed reflected light signal and improving the detection effect of the light sensor 131 on the reflected light signal. Furthermore, the second optical element 132 may also be provided with a filter / optical coating to reduce the influence of other reflected light besides the reflected light signal on the light sensor 131, thereby improving the anti-interference capability of the eye-tracking module 1.

[0119] In some embodiments, the light output component 12 is an infrared light emitter, and the light receiving module 13 includes an infrared light sensor, detecting an electrical signal. Infrared light is invisible to the human eye, thus avoiding interference with visual tasks, and it can penetrate the cornea, reducing corneal reflection interference.

[0120] Specifically, an infrared light emitter is used as the light output component 12, and the emitted light signal it outputs is infrared light. The reflected light signal formed by the wearer's eyes reflecting the emitted light signal is also infrared light. The infrared light sensor in the light receiving module 13 can receive this reflected light signal and output a detection signal as an electrical signal.

[0121] It should be noted that the infrared light sensor in this embodiment may be the same as the light sensor 131 in the aforementioned embodiment.

[0122] In some embodiments, the light emitter 124 is a divergent light source, such as an infrared Vcsel (Vertical-Cavity Surface-Emitting Laser), and the first optical element 125 is a prism; the infrared light emitted by the infrared Vcsel is refracted to the target area after passing through the prism.

[0123] In some embodiments, the light receiving module 13 is configured to receive and sense the light intensity of the reflected light signal in order to output a detection signal.

[0124] In some implementations, the detection sampling frequency of the optical receiving module 13 is 25-50Hz, 60-200Hz, 250-500Hz, or 600-1200Hz.

[0125] Specifically, the intensity of the detection signal is matched with the intensity of the reflected light signal. When processing the detection signal, the intensity of the reflected light signal can be identified based on the detection signal, and the intensity of the reflected light signal can be used to determine whether the pupil coincides with the corresponding target area 40, so as to determine the position information of the pupil. Thus, the wearer's eye movement state can be obtained by tracking the position change of the pupil.

[0126] For example, the first detection signal and the second detection signal in the detection signal can be determined based on the time distribution of the first time period and the second time period within the detection cycle. Then, the light intensity of the first reflected light signal can be identified based on the first detection signal, and the light intensity of the second reflected light signal can be identified based on the second detection signal. Then, the wearer's pupil can be determined to have rotated to the first target area 41 within the detection cycle based on the light intensity of the first reflected light signal, and the wearer's pupil can be determined to have rotated to the first target area 41 within the detection cycle based on the light intensity of the second reflected light signal.

[0127] The following is a detailed description of smart device 1:

[0128] Please see Figures 14 to 16 , Figure 14 This is a module structure diagram of another smart device provided in one or more embodiments of this application. Figure 15 A module structure diagram of yet another smart device provided in one or more embodiments of this application. Figure 16 This is a module structure diagram of another smart device provided in one or more embodiments of this application.

[0129] like Figures 1 to 16 As shown in the embodiments of this application, a smart device 1 is also provided, including any eye-tracking module 10 as provided in the embodiments of this application. The smart device 1 can be smart glasses, such as VR glasses, AR glasses, XR glasses, MR glasses, etc.

[0130] It should be noted that during use, the smart device 1 can monitor the wearer's eye movement state, namely the wearer's gaze direction and gaze target, through the eye-tracking module 10, and then perform corresponding operations based on the wearer's eye movement state to realize intelligent interaction between the smart device 1 and the wearer, thereby improving the wearer's user experience.

[0131] For example, the smart glasses are equipped with a display module 30 that corresponds to the wearer's settings. The display module 30 is configured to display images to the wearer's eyes. The wearer can select or manipulate virtual objects by looking at them in the images, achieving a more natural and efficient interactive experience. The above-mentioned interaction method has wide applications in games, education and other scenarios.

[0132] Among them, the display module 30 corresponding to VR glasses, XR glasses or MR glasses includes a display screen, while the display module 30 corresponding to AR glasses includes display lenses, such as waveguide lenses, freeform surface lenses, holographic lenses, etc.

[0133] In some embodiments, in addition to detecting the wearer's eye movement status through the eye-tracking module 10, the smart device 1 can also monitor the wearer's eye health based on the eye movement status, helping to detect problems such as dry eye syndrome and eye fatigue.

[0134] As one embodiment, the smart device 1 includes two eye-tracking modules 10, which correspond to the wearer's left and right eyes respectively, to detect the eye movement status of the wearer's left and right eyes.

[0135] Specifically, when the eyeball moves, the relative positions of the pupil, iris, and sclera in the eye socket change, causing dynamic changes in the intensity of reflected light received by the infrared sensor.

[0136] In some embodiments, the smart device 1 is a smart glasses, which also includes a glasses frame 51, a display module 30 and a projection module 60. The eye-tracking module 10, the display module 30 and the projection module 60 are all disposed on the glasses frame 51.

[0137] Specifically, the projection module 60 is configured to provide image light associated with an image or information to the display module 30, and the display module 30 receives the image light and displays the corresponding image or information. Further explanation: smart glasses may be, for example, AR glasses, VR glasses, XR glasses, or MR glasses. The display module 30 for VR glasses, XR glasses, or MR glasses includes a display screen, while the display module 30 for AR glasses includes display lenses, such as waveguide lenses, freeform lenses, holographic lenses, etc.

[0138] In some embodiments, the eyeglasses frame 51 includes a front frame 52 and temple members 53. The front frame 52 includes opposing temple connecting portions 522 (also called posts), which are located on the side of the front frame 52 near the temple members 53, and the temple members 53 are connected to the front frame 52 via the temple connecting portions 522. By way of example and not limitation, two temple connecting portions 522 are located on the left and right sides of the front frame 52 and are respectively connected to two different temple members 53. For example, when the smart device 1 is smart glasses, the wearable support 50 can be the eyeglasses frame 51.

[0139] The eye-tracking module 10 is disposed on the temple connection portion 522 of at least one side of the front frame 52.

[0140] In some embodiments, the eye-tracking module 10 includes a first eye-tracking module 14 and a second eye-tracking module 15, which are respectively disposed on different temple connection portions 522 on both sides of the front frame 52.

[0141] The first eye-tracking module 14 is configured to detect the eye movement state of the wearer's first eye and output a first eye detection signal, and the second eye-tracking module 15 is configured to detect the eye movement state of the wearer's second eye and output a second eye detection signal.

[0142] Specifically, the first eye and the second eye are the wearer's left and right eyes. The first eye movement module 14 and the second eye movement module 15 are different eye movement modules 10. They are set to correspond to the wearer's first eye and second eye respectively. The first eye detection signal is used to characterize the eye movement state of the first eye, and the second eye detection signal is used to characterize the eye movement state of the second eye.

[0143] As one embodiment, when the wearer wears the smart device, the front frame 52 is located in front of the wearer's head, the temples 53 are located on the left and right sides of the wearer's head, and the front frame 52 is adapted to the wearer's nose bridge, while the temples 53 are adapted to the wearer's ears.

[0144] As one embodiment, the front frame 52 further includes a display mounting portion 521, and temple connecting portions 522 are disposed on opposite sides of the display mounting portion 521. A display module 30, such as a display screen or a display lens, is connected to the display mounting portion 521, thereby positioning the display mounting portion 521 in front of the wearer's head when the wearer wears the smart device. By way of example and not limitation, the display module 30 is a display lens, the display mounting portion 521 is connected to at least one side of the display lens in the circumferential direction, and the display lens is snap-fitted to the display mounting portion 521; this snap-fitting method can be rigid snap-fitting or elastic snap-fitting.

[0145] Based on the eye-tracking module 10 disposed on the temple connection portion 522 on at least one side of the front frame 52, when the wearer wears the smart device 1 through the wear bracket 50, the eye-tracking module 10 is configured on the side of the front frame 52 to detect the wearer's eye movement state.

[0146] It should be noted that in the relevant technology, placing part or all of the structure of the eye-tracking module 10 in front of the wearer's eyes will obstruct the wearer's vision, increase the weight of the display mounting part 521, increase the burden on the bridge of the nose, and make it uncomfortable to wear. Moreover, improper operation when disassembling the eye-tracking module 10 during production line installation or maintenance may cause wear and tear on the display module 30, especially the display lens.

[0147] Compared to related technologies, the eye-tracking module 10 in the smart device 1 provided in this embodiment is located on the side, which reduces obstruction of the wearer's vision and burden on the bridge of the nose, thus improving the wearing experience. On the other hand, the side-mounted eye-tracking module 10 makes better use of the space in the smart device 1 and avoids wear and tear on the display lenses during installation and removal.

[0148] In some implementations, the eye-tracking module 10 is integrated with the projection module 60. As one embodiment, the eye-tracking module 10 and the projection module 60 are integrated on the same circuit board.

[0149] It should be understood that the above settings can effectively reduce the space occupied by the eye-tracking module 10 and the projection module 60, making the smart device 1 more portable.

[0150] In some embodiments, the temple connection 522 forms a receiving chamber 523, in which the eye-tracking module 10 and the projection module 60 are integrated and both housed within the receiving chamber 523. It should be understood that the above arrangement makes full use of the space of the smart device 1, making the smart device 1 more portable and reducing obstruction to the wearer's vision.

[0151] Furthermore, in some embodiments, the image light output by the projection module 60 is associated with the eye-tracking module 10. Therefore, integrating the eye-tracking module 10 with the projection module 60 also facilitates the arrangement of the electrical connection between the eye-tracking module 10 and the projection module 60.

[0152] As one embodiment, the eye-tracking module 10 and the projection module 60 are integrated on the same circuit board module. For example, the circuit board module includes at least one circuit board, and the eye-tracking module 10 and the projection module 60 are integrated on the same circuit board; or, for another example, the circuit board module includes at least two stacked circuit boards, and adjacent circuit boards are electrically connected, and the eye-tracking module 10 and the projection module 60 are disposed on different circuit boards on the same circuit board module.

[0153] In some implementations, the smart device 1 is configured to acquire the eye movement state of the wearer's eyes during the detection period based on the detection signal.

[0154] Specifically, based on the image displayed by the display module 30, the wearer can select or manipulate virtual objects by looking at them in the image. The wearer's eye movement state (i.e., gaze direction) is determined by the detection signal detected by the eye movement module 10.

[0155] like Figure 14 and Figure 15As shown, in some embodiments, the smart device 1 further includes a controller 20, which is used to control the operation of the eye-tracking module 10 in the smart device 1.

[0156] As one embodiment, the controller 20 may be housed in the receiving chamber 523, and further, the controller 20 may be integrated with the eye-tracking module 10 and the projection module 60.

[0157] It should be understood that integrating the eye-tracking module 10, projection module 60, and controller 20 within the accommodating chamber 523 also avoids exposing each module, thus providing effective protection for the eye-tracking module 10, projection module 60, and controller 20 through the temple connection portion 522. Furthermore, in some embodiments, the controller 20 is electrically connected to the projection module 60 and eye-tracking module 10 to transmit electrical signals; therefore, integrating the controller 20, eye-tracking module 10, and projection module 60 also facilitates the arrangement of electrical connections between the modules.

[0158] In another embodiment, the controller 20 is disposed on the temple 53. Preferably, the controller 20 is disposed on the side of the temple 53 near the front frame 54 so that the controller 20 can form an electrical connection with the projection module 60 and the eye-tracking module 10 to realize the transmission of electrical signals.

[0159] As one embodiment, the controller 20 is also used to control the operation of the display module 30 and the projection module 60.

[0160] In some implementations, the controller 20 is used to receive the detection signal output by the eye-tracking module 10 and adjust the image light output by the projection module 60 according to the detection signal.

[0161] Specifically, the controller 20 can identify the wearer's active eye movement behavior based on the detection signal and adjust the image light output by the projection module 60 accordingly, thereby enabling interaction between the wearer and the device; it can also identify changes in the wearer's pupil area based on the detection signal to adjust the image light output by the projection module 60 and the brightness of the screen in the display module 30 accordingly; it can also collect the wearer's current gaze direction based on the detection signal and adjust the image light output by the projection module 60 according to the current gaze direction to form corresponding guidance in the screen displayed on the display module 30, thereby instructing the wearer to gaze in the target direction.

[0162] For example, controller 20 includes one of the following: central processing unit (CPU), digital signal processor (DSP), application specific integrated circuit (ASIC), field-programmable gate array (FPGA), other general-purpose processor or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component.

[0163] In some implementations, the controller 20 is configured to:

[0164] The first light output component 121 is controlled to output a first emitted light signal to the first target area 41, and the second light output component 122 is controlled to output a second emitted light signal to the second target area 42, wherein the first target area 41 and the second target area 42 are at least partially non-overlapping different areas of the same eye of the wearer.

[0165] Specifically, the first light output component 121 and the second light output component 122 have different light output angles and / or are located in different positions.

[0166] like Figures 7 to 10 As shown, and not limited to, for the left eye, the first target region 41 and the second target region 42 are the lower left and upper left sides, respectively, and the first target region 41 and the second target region 42 need to cover at least the left side and most of the middle area of ​​the left eye; for the right eye, the first target region 41 and the second target region 42 are the lower right and upper right sides, respectively, and the first target region 41 and the second target region 42 need to cover at least the right side and most of the middle area of ​​the right eye.

[0167] It is understandable that the first target area 41 and the second target area 42 are opposite sides of the eye, such as the upper side and the lower side, the left side and the right side, or the diagonal side.

[0168] It is understandable that, while ensuring detection accuracy, the first target region 41 and the second target region 42 may not overlap (e.g., Figure 3 (as shown); or, the first target area 41 and the second target area 42 partially overlap, thereby covering most of the eyeglasses, such as the sclera area, etc. (e.g.) Figures 7 to 10 (As shown).

[0169] In some embodiments, the first target region 41 and the second target region 42 are respectively the upper and lower sides of the eye.

[0170] In some embodiments, a third target region may also be included. By way of example and not limitation, the first target region 41, the second target region 42, and the third target region 43 are the left, middle, and right sides of the eye, respectively.

[0171] As an example, and not a limitation, the sampling frequency of the optical receiving module is greater than 25Hz. For example... Figure 3 and Figure 6 As shown, in each eyeball, the first target area 41 is located above the pupil, and the second target area 42 is located below the pupil.

[0172] As one embodiment, the first light output component 121 and the second light output component 122 have different light output angles, and the first light output component 121 and the second light output component 122 are configured to emit light to the wearer's eyes from different directions.

[0173] As one embodiment, the smart device 1 also includes a display module 30, which is set to correspond to the wearer's eyes and configured to display images to the wearer's eyes. The first light output component 121 and the second light output component 122 are located on different sides of the display module 30. When the wearer wears the smart device 1 through the wear bracket 50, the first light output component 121 is located on the upper side of the display module 30, that is, above the wearer's eyeballs, and the second light output component 122 is located on the upper side of the display module 30, that is, below the wearer's eyeballs, so as to output emitted light signals to different target areas 40 of the eyes from different angles.

[0174] As one embodiment, the controller 20 is also used to control the smart device 1 or the eye-tracking module 10 to execute any of the eye-tracking calibration methods provided in the embodiments of this application. The control method is described in detail below.

[0175] This application also provides an eye-tracking calibration method. As one embodiment, the eye-tracking calibration method can be applied to any eye-tracking module 10 or smart device 1 provided in the embodiments of this application, but is not limited to application to the eye-tracking module 10 and smart device 1 provided in the embodiments of this application.

[0176] As an example and not a limitation, the eye-tracking calibration method provided in this application is applied to the eye-tracking module 10 provided in this application for illustration. The configuration of each component of the eye-tracking module 10 and the smart device 1 can be referred to the specific description of the eye-tracking module 10 and the smart device 1 provided in the foregoing embodiments, and will not be repeated here.

[0177] Please see Figure 20 , Figure 20 This is a flowchart illustrating the steps of an eye-tracking calibration method provided in one or more embodiments of this application.

[0178] like Figure 20 As shown, the specific process of eye-tracking calibration can be as follows:

[0179] 210. When the wearer performs the eye movement to be calibrated, collect the real calibration information of at least one real calibration point corresponding to the eye movement to be calibrated.

[0180] The wearer can be understood as the person wearing the eye-tracking module 10 or the smart device 1. The wearer can wear the eye-tracking module 10 or the smart device 1 on their eyes.

[0181] The eye movement behavior to be calibrated may include fixation on a target calibration point, which can be understood as the wearer fixating on at least one target calibration point. A target calibration point can be understood as a fixation point set in the calibration area, and the wearer's eyes fixate on this fixation point, thereby enabling the wearer to perform the eye movement behavior to be calibrated.

[0182] The true calibration information can be understood as the calibration information collected when the wearer is looking at the true calibration point. This true calibration information may include reflected light signals. The reflected light signals may include light intensity or other information, etc. The true calibration point can be understood as the target calibration point that the wearer is actually looking at. In contrast, the virtual calibration point is the target calibration point that the wearer is not actually looking at.

[0183] There are several ways to collect real calibration information of at least one real calibration point corresponding to the eye movement to be calibrated when the wearer performs the eye movement to be calibrated. Specifically, they can be as follows:

[0184] For example, when the wearer performs the eye movement behavior to be calibrated, a first emitted light signal can be output to a first target area of ​​the wearer's eye during the same detection cycle, so that the first emitted light signal is reflected on the eye to form a first reflected light signal. During the second time period, a second emitted light signal can be output to a second target area of ​​the wearer's eye, so that the second emitted light signal is reflected on the eye to form a second reflected light signal. The first reflected light signal and the second reflected light signal are received using the same light sensor, and a first detection signal corresponding to the first emitted light signal and a second detection signal corresponding to the second reflected light signal are output. The actual calibration information includes at least the first detection signal and the second detection signal.

[0185] The first target area and the second target area can be different areas of the wearer's eyes. It should be noted that the first target area and the second target area may or may not overlap.

[0186] The first time period may or may not have a time interval. The time within the first time period can be any value, such as 0.1s, 0.2s, 0.5s, 1s, or any other value. The time within the second time period can be the same as or different from the time within the first time period, depending on the specific application.

[0187] Optionally, in some embodiments, after outputting the second emitted light signal to the second target area of ​​the wearer's eye during the second time period, a third emitted light signal may also be output to the third target area. Specifically, this may include: outputting the third emitted light signal to the third target area of ​​the wearer's eye during the third time period, such that the third emitted light signal is reflected on the eye to form a third reflected light signal, receiving the third reflected light signal using a light sensor, and outputting a third detection signal corresponding to the third reflected light signal, wherein the actual calibration information also includes the third detection signal.

[0188] In this scheme, reflected light signals can be collected by turning the light emitter on and off in a time-sharing manner. This allows a single light emitter to collect reflected light signals from multiple different regions, thereby simulating the effect of multiple light reflectors and improving the efficiency of eye-tracking calibration.

[0189] Optionally, in some embodiments, before collecting the actual calibration information of at least one actual calibration point corresponding to the eye-tracking behavior to be calibrated when the wearer performs the eye-tracking behavior to be calibrated, an eye-tracking command can be output to cause the wearer to perform the eye-tracking behavior to be calibrated. For example, object information of the wearer can be collected, and the wearer's current eye-tracking calibration state can be determined based on the object information. When the eye-tracking calibration state indicates that the wearer has not performed eye-tracking calibration before the current moment, an eye-tracking behavior command to be calibrated can be output so that the wearer fixates on the target calibration point based on the command.

[0190] The calibration algorithm model can be understood as the model used to generate virtual calibration information. This model can have various network structures, such as CNN, MLP, LSTM, or other models capable of generating calibration information. Deploying the calibration algorithm model locally can be understood as deploying it within a smart device or eye-tracking calibration apparatus.

[0191] The "object information" can be understood as information indicating the wearer's identity. For example, it can include the wearer's image, fingerprint information, iris information, voiceprint information, or other information that can identify them. There are various ways to collect the wearer's object information. For instance, an image acquisition device can be used to collect the wearer's facial or body image to obtain the object information. Alternatively, a biometric information acquisition device can be used to collect at least one of the wearer's fingerprint, iris, or voiceprint information to obtain the object information, and so on.

[0192] After collecting the wearer's object information, the wearer's current eye-tracking calibration state can be determined based on this information. The current eye-tracking calibration state indicates whether the wearer has performed eye-tracking calibration on the smart device or eye-tracking calibration device. There are several ways to determine the wearer's current eye-tracking calibration state. For example, based on the object information, the wearer's object identifier can be determined, and the identifier can be searched in the set of calibrated object identifiers. If the identifier exists, the wearer's current eye-tracking calibration state is determined to be that eye-tracking calibration was performed before the current moment; if the identifier does not exist, the wearer's current eye-tracking calibration state is determined to be that eye-tracking calibration was not performed before the current moment. Alternatively, feature extraction can be performed on the object information to obtain the wearer's object features. The object features can then be searched in the set of object features corresponding to calibrated objects. If the object features exist, the wearer's current eye-tracking calibration state is determined to be that eye-tracking calibration was performed before the current moment; if the object features do not exist, the wearer's current eye-tracking calibration state is determined to be that eye-tracking calibration was not performed before the current moment, and so on.

[0193] After determining the wearer's current eye-tracking calibration state based on object information, when the calibration state indicates that the wearer has not performed eye-tracking calibration before the current moment, a calibration instruction for eye-tracking behavior can be output, causing the wearer to fixate on the target calibration point based on the instruction. The calibration instruction for eye-tracking behavior can be understood as an instruction to instruct the wearer to fixate on the target calibration point. There are several ways to output the calibration instruction for eye-tracking behavior. For example, the target calibration point can be highlighted on the display interface or display area of ​​the smart device or eye-tracking calibration device, causing the wearer to fixate on the target calibration point based on the calibration instruction. Alternatively, a voice instruction corresponding to the target calibration point can be output (e.g., "Please fixate on the fixation point in the XXX direction"), causing the wearer to fixate on the target calibration point based on the calibration instruction, and so on.

[0194] In this solution, eye-tracking calibration is performed on the wearer using a static calibration method. Static calibration refers to the eye-tracking calibration performed on the wearer the first time they use the smart device or eye-tracking module 10.

[0195] Optionally, after eye-tracking calibration of the wearer, dynamic calibration can also be performed. That is, after determining the actual parameters corresponding to the wearer based on real and virtual calibration information, dynamic calibration can be performed on the wearer. Dynamic calibration methods can include, for example, dynamically outputting eye-tracking behavior commands to be calibrated based on preset acquisition conditions, so that the wearer fixates on the target calibration point according to the commands.

[0196] The preset collection conditions can be understood as the pre-defined conditions for collecting the wearer's actual calibration information. There are various types of preset collection conditions. For example, collection can be based on a time period (e.g., once every time period), or collection can be dynamically triggered by events (e.g., detecting a change in the wearer's wearing status of the smart device or eye-tracking module 10 from never wearing to wearing), or other pre-defined collection conditions, etc.

[0197] It should be noted that dynamic calibration of the wearer allows for dynamic adjustment of the actual calibration parameters, thereby improving the accuracy of eye-tracking calibration.

[0198] 220. Based on real calibration information, use a calibration algorithm model to generate virtual calibration information for at least one virtual calibration point.

[0199] Virtual calibration points can be understood as calibration points that the wearer does not actually gaze at. It should be noted that both real and virtual calibration points can be pre-set calibration points (gaze points), while real calibration points can be understood as the target calibration points that the wearer actually gazes at.

[0200] The virtual calibration information can be understood as the calibration information of virtual calibration points, such as the virtual reflected light signals of the virtual calibration points. The types of reflected light signals can be various, including light intensity or other signals, etc.

[0201] There are several ways to generate virtual calibration information for at least one virtual calibration point based on real calibration information using a calibration algorithm model. These methods include:

[0202] For example, a calibration algorithm model can be used to extract features from real calibration information to obtain real calibration features. Based on the real calibration features, reference calibration points that are related to real calibration points can be selected from a preset calibration database. Based on the location information of virtual calibration points and the calibration algorithm model, virtual calibration information of virtual calibration points can be generated.

[0203] The preset calibration database can be understood as a pre-defined calibration database, which may include calibration information of at least one reference calibration point. The calibration information may include light intensity data or light intensity distribution information, etc. There are several ways to select reference calibration points that are related to real calibration points from a preset calibration database based on real calibration features. For example, based on real calibration features, at least one reference calibration point that has a mapping relationship with the real calibration point can be selected from the reference calibration points to obtain at least one virtual calibration point. Alternatively, based on real calibration features, the current position information of the real calibration point can be determined, and based on the current position information, at least one reference calibration point whose distance from the real calibration point is within a preset distance range can be selected from the reference calibration points to obtain at least one virtual calibration point. Or, a target reference calibration point corresponding to the real calibration point can be identified from the reference calibration points, and the target reference calibration point can be deleted from the reference calibration points to obtain at least one candidate reference calibration point. From the candidate reference calibration points, a preset number of reference calibration points can be randomly selected or selected according to a preset position to obtain at least one virtual calibration point, and so on.

[0204] After selecting reference calibration points that are related to the real calibration points from a pre-set calibration database based on the real calibration features, virtual calibration information of the virtual calibration points can be generated according to the point information of the virtual calibration points and the calibration algorithm model. There are several ways to generate virtual calibration points using the calibration algorithm model. For example, the point information of the virtual calibration points can be identified from the pre-set calibration database, and the calibration algorithm model can be used to fit the calibration information of the virtual calibration points based on the point information to obtain the virtual calibration information. Alternatively, the calibration information of the virtual calibration points can be selected from the pre-set calibration database to obtain reference calibration features. These reference calibration features can then be adjusted based on the real calibration features to obtain adjusted calibration features. Based on these adjusted calibration features, the calibration algorithm model can be used to fit the calibration information of the virtual calibration points to obtain the virtual calibration information. Another method is to select the calibration information of the virtual calibration points from the pre-set calibration database to obtain the virtual calibration information, and so on.

[0205] Optionally, in some embodiments, a preset calibration database may be constructed before collecting real calibration information of at least one real calibration point corresponding to the eye movement to be calibrated when the wearer performs the eye movement to be calibrated. The process of constructing the preset calibration database may include: collecting light intensity information of at least one reference calibration point in the calibration area when at least one sample wearer performs the target eye movement, and constructing the preset calibration database based on the light intensity information.

[0206] The process of constructing a preset calibration database based on light intensity information may include: normalizing the light intensity information, identifying the position of the reference calibration point in the calibration area to obtain the current point position information, identifying the position of the center point in the calibration area to obtain the center point position, calculating the distance between the reference calibration point and the center point based on the current point position information and the center point position to obtain the current point position distance, and modeling the light intensity distribution of the reference calibration point in the calibration area based on the normalized light intensity information and the current point position distance to obtain the calibration database.

[0207] There are several ways to model the light intensity distribution of the reference calibration point in the calibration area based on the normalized light intensity information and the current distance. For example, radial basis functions (RBFs) can be used to model the light intensity distribution of the reference calibration point in the calibration area based on the normalized light intensity information and the current distance, thus obtaining a preset calibration database. Alternatively, kernel functions can be used to model the light intensity distribution of the reference calibration point in the calibration area based on the normalized light intensity information and the current distance, thus obtaining a preset calibration database. Or, wavelet functions can be used to model the light intensity distribution of the reference calibration point in the calibration area based on the normalized light intensity information and the current distance, thus obtaining a calibration database. Alternatively, compactly supported radial basis functions (RBFs) can be used to model the light intensity distribution of the reference calibration point in the calibration area based on the normalized light intensity information and the current distance, thus obtaining a preset calibration database, and so on.

[0208] It should be noted that during the construction of the pre-defined calibration database, different calibration samples can be formed by collecting any suitable number of user data. By establishing the calibration database, the data of virtual sampling points can be supplemented with a smaller number of actual sampling points, thereby reducing the number of actual sampling points. For example, a small number of input calibration points can be set, specifically 4, 9, 16, or any other suitable number of n. The calibration algorithm model uses a rule engine and / or machine learning algorithm to fit and generate multiple virtual output calibration points, specifically 81, 91, or any other suitable number of kn, where k is an integer not less than 1. Establishing a calibration database reduces the number of samples, reduces the amount of calibration computation, expands the calibration area, and improves calibration accuracy.

[0209] Optionally, in some embodiments, before generating virtual calibration information for at least one virtual calibration point using a calibration algorithm model based on real calibration information, the calibration algorithm model can be trained. The process of training the calibration algorithm model may include: acquiring a first detection signal sample corresponding to the target eye movement behavior, and using the first detection signal sample to train a preset calibration algorithm model to obtain an initial calibration algorithm model; verifying the initial calibration algorithm model based on reference calibration information to obtain a verified calibration algorithm model; and optimizing the verified calibration algorithm model according to a preset calibration database to obtain the final calibration algorithm model. Figure 21 As shown, the specific details are as follows:

[0210] S1. Obtain the first detection signal sample corresponding to the target eye movement behavior, and use the first detection signal sample to train the preset calibration algorithm model to obtain the initial calibration algorithm model.

[0211] The first detection signal sample can be understood as the calibration information of the wearer sample when performing the target eye movement behavior. It should be noted that the target eye movement behavior can be understood as the eye movement behavior of the sample wearer gazing at at least one sample calibration point (gaze point) when training the preset calibration algorithm model.

[0212] The method of obtaining the first detection signal sample corresponding to the target eye movement behavior can be similar to the method of collecting real calibration information of at least one real calibration point corresponding to the eye movement behavior to be calibrated, as described above, and will not be repeated here.

[0213] After obtaining the first detection signal sample corresponding to the target eye movement behavior, the first detection signal sample can be used to train the preset calibration algorithm model, thereby obtaining the initial calibration algorithm model. The process of training the preset calibration algorithm model using the first detection signal sample can be as follows:

[0214] For example, a preset calibration algorithm model can be used to predict the light intensity distribution information corresponding to the first detection signal sample, obtain the predicted light intensity distribution information, obtain the standard light intensity distribution information corresponding to the target eye movement behavior, and determine the light intensity distribution loss corresponding to the first detection signal sample based on the predicted light intensity distribution information and the standard light intensity distribution information. According to the light intensity distribution loss, the preset calibration algorithm model is converged to obtain the initial calibration algorithm model.

[0215] Taking light intensity as an example, the light intensity distribution information can be understood as the distribution information of light intensity in the eye region when the target eye movement is performed. The standard light intensity distribution information can be understood as the actual light intensity distribution information collected when the target eye movement is performed.

[0216] The light intensity distribution loss can be understood as the difference between the light intensity distribution information predicted by the preset calibration algorithm model and the standard light intensity distribution information. There are several ways to determine the light intensity distribution loss corresponding to the first detection signal sample based on the predicted and standard light intensity distribution information. For example, at least one region's light intensity distribution can be identified in the predicted light intensity distribution information to obtain the predicted light intensity distribution for that region. The standard light intensity distribution corresponding to that region can be extracted from the standard light intensity distribution information, and the difference between the predicted and standard light intensity distributions can be calculated to obtain the initial light intensity distribution loss for that region. The initial light intensity distribution losses can then be fused to obtain the light intensity distribution loss corresponding to the first detection signal sample. Alternatively, feature extraction can be performed on the predicted light intensity distribution information to obtain the predicted light intensity distribution features. Feature extraction can also be performed on the standard light intensity distribution information to obtain the reference light intensity distribution information. The similarity between the predicted and reference light intensity distribution features can then be calculated to obtain the light intensity distribution loss for the first detection signal sample, and so on.

[0217] There are several ways to calculate the difference between the predicted light intensity distribution and the standard light intensity distribution. For example, the KL divergence between the predicted light intensity distribution and the standard light intensity distribution can be calculated to obtain the initial light intensity distribution loss of the region. Alternatively, the JS divergence between the predicted light intensity distribution and the standard light intensity distribution can be calculated to obtain the initial light intensity distribution loss of the region. Or, the cross-entropy between the predicted light intensity distribution and the standard light intensity distribution can be calculated to obtain the initial light intensity distribution loss of the region, and so on.

[0218] After calculating the difference between the predicted light intensity distribution and the standard light intensity distribution, the calculated initial light intensity distribution loss can be fused to obtain the light intensity distribution loss corresponding to the first detected signal sample. There are several ways to fuse the initial light intensity distribution. For example, the initial light intensity distributions can be accumulated to obtain the light intensity distribution loss corresponding to the first detected signal sample; alternatively, fusion weights for different regions can be obtained, and the initial light intensity distribution information can be weighted based on these weights, then the weighted initial light intensity distributions can be accumulated to obtain the light intensity distribution loss corresponding to the first detected signal sample, and so on.

[0219] There are several ways to calculate the similarity between the predicted light intensity distribution features and the reference light intensity distribution features. For example, the cosine similarity between the predicted light intensity distribution features and the reference light intensity distribution features can be calculated to obtain the light intensity distribution loss corresponding to the first detection signal sample. Alternatively, the feature distance between the predicted light intensity distribution features and the reference light intensity distribution features can be calculated to obtain the feature similarity, which can then be used as the light intensity distribution loss corresponding to the first detection signal sample, and so on.

[0220] After determining the light intensity distribution loss corresponding to the first detection signal sample based on the predicted light intensity distribution loss and the standard light intensity distribution loss, the preset calibration algorithm model can be converged according to the light intensity distribution loss to obtain the initial calibration algorithm model. The process of converging the preset calibration algorithm model according to the light intensity distribution loss can include: updating the model parameters of the preset calibration algorithm model based on the light intensity distribution loss to obtain the updated calibration algorithm model, using the updated model as the preset calibration algorithm model, and returning to execute the step of obtaining the first detection signal sample corresponding to the target eye movement behavior until the preset calibration algorithm model converges, thus obtaining the initial calibration algorithm model.

[0221] One method for updating the model parameters of a preset calibration algorithm model based on light intensity distribution loss can be: constructing an objective function relating the model parameters to the light intensity distribution loss; adjusting the model parameters using the objective function to minimize the light intensity distribution loss; and using the preset calibration algorithm model with adjusted model parameters as the updated calibration algorithm model.

[0222] The objective function can be the function corresponding to the least squares method. The process of adjusting the model parameters through the objective function can be understood as finding a set of model parameters through the least squares method that minimizes the sum of squared residuals between the predicted light intensity distribution information and the standard light intensity distribution information.

[0223] Optionally, in some embodiments, updating the model parameters of the preset calibration algorithm model based on the light intensity distribution loss may further include: updating the model parameters of the preset calibration algorithm model using a gradient descent algorithm based on the light intensity distribution loss, thereby obtaining the updated calibration algorithm model.

[0224] After updating the model parameters of the preset calibration algorithm model, the updated calibration algorithm model can be used as the preset calibration algorithm model. Then, the step of obtaining the first detection signal sample corresponding to the target eye movement behavior is returned until the preset calibration algorithm model converges, thus obtaining the initial calibration algorithm model.

[0225] It should be noted that during the training process of the preset calibration algorithm model, the model parameters of the preset calibration algorithm model will be iteratively optimized or updated through the iteration cycle until the preset calibration algorithm model converges, thereby obtaining the initial calibration algorithm model.

[0226] The overall training process for the preset calibration algorithm model can include: setting initial parameters (initial model parameters of the preset calibration algorithm model); the wearer wearing the smart device or eye-tracking module 10 fixating on the calibration point; obtaining input calibration information (first detection signal sample) through the eye-tracking module 10; and then training the preset calibration algorithm model through a rule engine and / or machine learning algorithms to obtain the initial calibration algorithm model. During the training process, the input calibration information can be obtained through the positional information of the fixation point measured by infrared sensors, etc., and sensor information, which may include detection signals, temperature, the relative position of the smart device, etc.

[0227] S2. Verify the initial calibration algorithm model to obtain the verified calibration algorithm model.

[0228] For example, a third detection signal sample corresponding to the target eye movement behavior can be used, and based on the third detection signal sample, an initial calibration algorithm model can be used to perform eye movement calibration to obtain the current light intensity distribution information. Based on the current light intensity distribution information, at least one eye movement detection point corresponding to the target eye movement behavior can be determined. Based on the eye movement detection point, the initial calibration algorithm model can be verified to obtain the verified calibration algorithm model.

[0229] Here, the eye-tracking detection point can be understood as a calibration point identified by the current light intensity distribution information when the wearer sample is performing the target eye movement behavior. The process of verifying the initial calibration algorithm model based on the eye-tracking detection point can include: acquiring at least one preset calibration point corresponding to the target eye movement behavior; selecting candidate calibration points corresponding to the eye-tracking detection point from the preset calibration points; calculating the distance between the eye-tracking detection point and the candidate calibration points to obtain the current error; updating the initial calibration algorithm model based on the current error to obtain the verified calibration algorithm model when the current error meets the calibration requirements; and using the initial calibration algorithm model as the verified calibration algorithm model when the current error meets the calibration requirements.

[0230] The calibration requirements can include a reasonable error threshold or other reasonable error condition information. Using the calibration requirements as a reasonable error threshold means that when the current error does not meet the calibration requirements, it indicates that the current error exceeds the threshold. In this case, the initial calibration algorithm model can be updated based on the current error to obtain a validated calibration algorithm model. There are several ways to update the initial calibration algorithm model based on the current error. For example, the current error can be used as the target loss, and the model parameters of the initial calibration algorithm model can be updated using the least squares method based on the target loss to obtain the validated calibration algorithm model. Alternatively, the current error can be used as the target loss, and the model parameters of the initial calibration algorithm model can be updated using the gradient descent algorithm based on the target loss to obtain the validated calibration algorithm model. Another option is to calculate the difference between the current error and the target error threshold to obtain the target loss, and update the model parameters of the initial calibration algorithm model based on the target loss to obtain the validated calibration algorithm model.

[0231] When the current error meets the calibration requirements, it can be determined that the initial calibration algorithm model has passed the verification. At this time, the initial calibration algorithm model can be directly used as the calibration algorithm model after verification.

[0232] The verification process for the initial calibration algorithm model can essentially include: generating output parameters from the initial calibration algorithm model, comparing the initial parameters with the output parameters, and performing error analysis using a rule engine and / or machine learning algorithms. Then, it is determined whether the error meets the calibration requirements; if the requirements are met, optimization stops, resulting in the verified calibration algorithm model.

[0233] S3. Based on the preset calibration database, optimize the calibration algorithm model after verification to obtain the calibration algorithm model.

[0234] The preset calibration database can be understood as a pre-defined database containing calibration information for multiple calibration points. The preset calibration database may include reference calibration information for at least one reference calibration point.

[0235] The process of optimizing the calibration algorithm model after verification based on the preset calibration database can be as follows:

[0236] For example, a second detection signal sample corresponding to the target's eye movement behavior is collected. The second detection signal sample includes at least one real calibration information sample of the target's real calibration point. Based on the real calibration information sample and a preset calibration database, a virtual calibration information sample of at least one virtual calibration point of the target is generated using a calibration algorithm model after verification. The calibration algorithm model is optimized based on the real calibration information sample and the virtual calibration information sample to obtain the calibration algorithm model.

[0237] The process of generating virtual calibration information samples of at least one target virtual calibration point using a post-calibration algorithm model based on real calibration information samples and a preset calibration database may include: extracting features from real calibration information samples using a post-calibration algorithm model to obtain real sample calibration features; selecting reference calibration points that are related to the target real calibration points from the reference calibration points based on the real sample calibration features to obtain at least one target virtual calibration point; and generating calibration information of the target virtual calibration point using a post-calibration algorithm model based on the point location information of the target virtual calibration point to obtain at least one virtual calibration information sample.

[0238] There are several ways to select reference calibration points that are related to the target real calibration point based on the calibration features of real samples. For example, at least one reference calibration point that has a mapping relationship with the target real calibration point can be selected from the reference calibration points to obtain at least one target virtual calibration point. Alternatively, at least one reference calibration point whose distance from the target real calibration point is within a preset distance range can be selected from the reference calibration points to obtain at least one target virtual calibration point. Or, the target reference calibration point corresponding to the target real calibration point can be identified from the reference calibration points, and the target reference calibration point can be deleted from the reference calibration points to obtain at least one candidate reference calibration point. A preset number of reference calibration points can then be randomly selected or selected according to a preset position from the candidate reference calibration points to obtain at least one target virtual calibration point, and so on.

[0239] The process of optimizing the calibration algorithm model after verification based on real calibration information samples and virtual calibration information samples may include: selecting the target reference calibration information corresponding to the target virtual calibration point from the reference calibration information; comparing the target reference calibration information with the virtual calibration information samples to obtain the virtual calibration point error; and updating the calibration algorithm model after verification using real calibration information samples and virtual calibration information samples when the virtual calibration point error meets the preset error conditions to obtain the calibration algorithm model.

[0240] When the error of the virtual calibration point meets the preset error conditions, the calibration algorithm model can be updated using real calibration information samples and virtual calibration information samples to obtain the calibration algorithm model.

[0241] Optionally, in some embodiments, when the virtual calibration point error does not meet the preset error condition, the model parameters of the verified algorithm model can be updated based on the virtual calibration point error to obtain a candidate calibration algorithm model. The candidate calibration algorithm model is then used as the verified calibration algorithm model, and the step of using the verified calibration algorithm model to extract features from the real calibration information sample is returned to be executed until the virtual calibration point error meets the preset error condition, so as to obtain the calibration algorithm model.

[0242] It should also be noted that the parameter processing, verification and optimization mentioned in this application can be handled by any suitable rule engine, machine learning algorithm or other suitable method, and there are no restrictions here.

[0243] 230. Based on the real calibration information and the virtual calibration information, determine the actual parameters corresponding to the wearer.

[0244] The actual parameters can be understood as the parameters output after eye-tracking calibration of the wearer. These actual parameters indicate the mapping relationship between the wearer's eye movements and the fixation point. Additionally, the actual parameters may include at least one update parameter. The update parameter is used to update or adjust the eye-tracking algorithm model. The eye-tracking algorithm model can be understood as the algorithm model that tracks the wearer's eyes. The eye-tracking algorithm model can detect the wearer's eye movement state during the detection period.

[0245] The process of determining the actual parameters corresponding to the wearer based on real calibration information and virtual calibration information can be as follows:

[0246] For example, an update coefficient can be determined based on at least one set of real calibration information and virtual calibration information.

[0247] The update coefficients are used to adjust the eye-tracking algorithm model. The actual parameters include at least one update coefficient.

[0248] The actual calibration data may include first light intensity data, and the virtual calibration data may include second light intensity data. A method for determining an update coefficient based on at least one set of actual calibration information and virtual calibration information may include: determining the current light intensity distribution information corresponding to the wearer based on at least one set of first and second light intensity data, obtaining the standard light intensity distribution information corresponding to the behavior to be calibrated, and comparing the current light intensity distribution information with the standard light intensity distribution information to obtain an update coefficient.

[0249] There are several ways to determine the current light intensity distribution information corresponding to the wearer based on at least one set of first light intensity data and second light intensity data. For example, the first light intensity distribution information of the real calibration point can be identified in the first light intensity data, and the second light intensity distribution information of the virtual calibration point can be identified in the second light intensity data. The first light intensity distribution information and the second light intensity distribution information are then fused to obtain the current light intensity distribution information corresponding to the wearer. Alternatively, the first light intensity distribution information of the real calibration point can be identified in the first light intensity data, and the second light intensity distribution information of the virtual calibration point can be identified in the second light intensity data. Based on the first light intensity distribution information and the second light intensity distribution information, the preset light intensity distribution information is adjusted to obtain the current light intensity distribution information corresponding to the wearer, and so on.

[0250] After determining the current light intensity distribution information corresponding to the wearer based on at least one set of first and second light intensity data, the standard light intensity distribution information corresponding to the behavior to be calibrated is obtained. The current light intensity distribution information is then compared with the standard light intensity distribution information to obtain an update coefficient. There are several ways to compare the current light intensity distribution information with the standard light intensity distribution information. For example, the comparison can be performed to obtain a comparison result, and then an update coefficient corresponding to the comparison result can be selected from a preset set of update coefficients. Alternatively, the current light intensity distribution information can be compared with the standard light intensity distribution information, and the comparison result can be used as an update coefficient, and so on.

[0251] Optionally, and it should be noted, an update coefficient can be determined based on at least one set of real calibration information and virtual calibration information. This set of real and virtual calibration information can include the real calibration information of one real calibration point and the virtual calibration information of one virtual calibration point. That is, an update coefficient can be determined using one set of real and virtual calibration information, or it can be determined using multiple sets of real and virtual calibration information.

[0252] Optionally, in some embodiments, actual parameters can be determined based on multiple sets of real and virtual calibration information. These actual parameters are used to adjust the eye-tracking algorithm model. That is, the eye-tracking algorithm model is updated by determining the actual parameters using multiple sets of real and virtual calibration information. These multiple sets can include at least two or all sets of real and virtual calibration information. When the multiple sets consist of all real and virtual calibration information, it can be understood that the eye-tracking algorithm model can be updated using the real calibration information of all real calibration points and the virtual calibration information of all virtual calibration points.

[0253] Optionally, in some embodiments, after determining the actual parameters corresponding to the wearer based on real calibration information and virtual calibration information, eye tracking of the wearer can also be performed using the actual parameters. The process of eye tracking of the wearer using actual parameters may include: updating the eye tracking algorithm model based on the actual parameters, collecting the reflected light signal of the wearer during the detection period to output a detection signal and a reference signal, and obtaining the eye movement state of the wearer's eyes at least during the detection period based on the detection signal, the reference signal, and the updated eye tracking algorithm model.

[0254] There are several ways to update the eye-tracking algorithm model based on actual parameters. For example, the eye-tracking algorithm model can be used to predict the gaze position of the wearer's eyes during the detection period based on actual parameters, and the predicted gaze position can be compared with the labeled gaze position of the wearer's eyes to obtain the position loss. Based on the position loss, the eye-tracking algorithm model can be converged to obtain the updated eye-tracking algorithm model. Alternatively, the preset parameters in the preset eye-tracking model can be replaced with the wearer's actual parameters to obtain the updated eye-tracking algorithm model, and so on.

[0255] After updating the preset eye-tracking algorithm model based on actual parameters, the reflected light signals of the wearer during the detection period can be collected to output detection signals and reference signals. The process of collecting the reflected light signals of the wearer during the detection period can include: outputting emitted light signals to the target area of ​​the wearer's eyes during the detection period, and not outputting emitted light signals to the target area of ​​the wearer's eyes during the control period, with the detection period and control period not overlapping; receiving reflected light signals and outputting detection signals based on the reflected light signals, which are formed by the reflected emitted light signals from the eyes; receiving reference reflected light signals and outputting reference signals based on the reference reflected light signals, which are formed by the reflected ambient light from the eyes.

[0256] After collecting the reflected light signal from the wearer during the detection period to output the detection signal and the reference signal, the wearer's eye movement state during at least the detection period can be obtained based on the detection signal, the reference signal, and the updated eye-tracking algorithm. The process of obtaining the wearer's eye movement state during at least the detection period, based on the detection signal, the reference signal, and the eye-tracking algorithm model, can include: determining the corresponding detection light based on the detection signal; determining the reference light intensity corresponding to the ambient light based on the reference signal; performing differential processing on the detection light intensity and the reference light intensity to obtain the target light intensity corresponding to the reflected signal; obtaining the wearer's gaze position during the detection period based on the target light intensity using the updated eye-tracking algorithm model; and determining the wearer's eye movement state based on continuous changes in the gaze position.

[0257] This actual parameter can be used to execute instructions in various application scenarios, such as selection or control, etc.

[0258] Optionally, in some embodiments, after determining the actual parameters corresponding to the wearer based on the real calibration information and the virtual calibration information, various in-application instructions can also be executed. For example, when receiving an operation instruction for the wearer, the operation corresponding to the operation instruction is executed based on the actual parameters. When the operation instruction is a selection operation instruction, the selection operation can be executed based on the actual parameters, and so on.

[0259] It should be noted that in the eye-tracking calibration process for the wearer, the wearer first gazes at the calibration point after wearing the smart device 1, and user calibration information is obtained. Then, the calibration algorithm model determines the actual parameters based on the detected user calibration information, so that the eye-tracking module 10 outputs according to the actual parameters. The application of the calibration algorithm model reduces the error of eye-tracking detection and improves the adaptability of the eye-tracking module 10 to different users.

[0260] This solution involves eye-tracking calibration of the wearer, followed by eye-tracking using the calibrated parameters. The eye-tracking calibration and tracking process can be as follows: Figure 22 As shown. In Figure 22In the eye-tracking calibration process, when the wearer performs the action to be calibrated, real calibration information of at least one real calibration point corresponding to the eye-tracking action is collected. Based on the real calibration information, a calibration algorithm model is used to generate virtual calibration information of at least one virtual calibration point. The wearer's actual parameters are obtained based on the real and virtual calibration information. The eye-tracking algorithm model is then updated using the actual parameters to obtain the eye-tracking algorithm model corresponding to the wearer. After eye-tracking calibration, eye-tracking can then be performed. During eye tracking, the eye tracking algorithm model parameters are calibrated and adjusted. Within the detection period, a emitted light signal is output to the target area of ​​the wearer's eyes during the detection time period, and no emitted light signal is output to the target area of ​​the wearer's eyes during the control time period. The detection time period and the control time period do not overlap. The reflected light signal is received, and a detection signal is output based on the reflected light signal. The reflected light signal is formed by the eye reflecting the emitted light signal. The corresponding detection light intensity is determined based on the detection signal. A reference reflected light signal is received, and a reference signal is output based on the reference reflected light signal. The reference reflected light signal is formed by the eye reflecting the ambient light signal. The reference light intensity corresponding to the ambient light is determined based on the reference signal. The detection light intensity and the reference light intensity are differentially processed to obtain the target light intensity corresponding to the reflected light signal. The eye tracking algorithm model obtains the wearer's gaze position (pupil coordinates) within the detection time period based on the target light intensity. The wearer's eye movement state is obtained based on the continuous changes in the gaze position. As can be seen from the above, in this embodiment of the application, when the wearer performs the eye-tracking behavior to be calibrated, the real calibration information of at least one real calibration point corresponding to the eye-tracking behavior to be calibrated is collected. Based on the real calibration information, the virtual calibration information of at least one virtual calibration point is generated using a calibration algorithm model. Based on the real calibration information and the virtual calibration information, the actual parameters corresponding to the wearer are determined. Since this scheme can collect a small amount of real calibration information of real calibration points, and fit the virtual calibration information of at least one virtual calibration point based on the real calibration information using a calibration algorithm model, the number of real calibration points sampled is reduced, the amount of calibration computation is reduced, and the calibration area is expanded. Therefore, the accuracy of eye-tracking calibration and subsequent eye-tracking can be improved.

[0261] Please see Figure 23 This application also provides an eye-tracking calibration device 300, wherein the eye-tracking device 300 is, for example, an eye-tracking module 10 or a smart device 1.

[0262] Specifically, the eye-tracking device 300 includes a processor 301 and a memory 302, which are connected via a bus 303, such as an I2C (Inter-integrated Circuit) bus.

[0263] Specifically, processor 301 provides computing and control capabilities to support the operation of the entire eye-tracking device. Processor 301 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0264] Specifically, the memory 302 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc.

[0265] Those skilled in the art will understand that Figure 23 The structure shown in the figure is merely a block diagram of a portion of the structure related to the embodiments of this application, and does not constitute a limitation on the eye-tracking calibration device to which the embodiments of this application are applied. A specific eye-tracking calibration device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0266] The processor is used to run a computer program stored in a memory, and to implement any of the eye-tracking calibration methods provided in the embodiments of this application when executing the computer program.

[0267] This application also provides a storage medium storing a computer program that can be executed by one or more processors to implement the steps of any of the eye-tracking calibration methods provided in the specification of this application.

[0268] The storage medium can be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0269] like Figure 15 As shown, this application also provides a smart device 1, comprising:

[0270] At least one display module 30 is configured to display a computer image;

[0271] At least one eye-tracking module 10 is configured to detect and output a detection signal associated with at least one eye of the wearer. The eye-tracking module 10 includes a light output module 11 and a light receiving module 13. The light output module 11 is configured to output emitted light signals to different target areas 40 of the same eye of the wearer in a time-division manner during the detection period. The light receiving module 13 is configured to detect reflected light signals formed by the reflected emitted light signals from the eye and output a detection signal.

[0272] Controller 20, which is communicatively connected to display module 30 and eye-tracking module 10, is configured to:

[0273] When the wearer performs the eye movement behavior to be calibrated, the real calibration information of at least one real calibration point corresponding to the eye movement behavior to be calibrated is collected by controlling the light output module 11 and the light receiving module 12.

[0274] Based on the real calibration information, a calibration algorithm model is used to generate virtual calibration information for at least one virtual calibration point.

[0275] Based on the real calibration information and the virtual calibration information, the actual parameters corresponding to the wearer are determined.

[0276] It should be understood that the control behavior of the controller 20 is intended to calibrate the intensity threshold of the reflected light signal corresponding to the target area 40 of the smart device 1 or the eye-tracking module 10.

[0277] As an example and not a limitation, the eye movement is the wearer's eyes looking straight ahead. As an example and not a limitation, the wearer's eyes are looking straight ahead when gazing at a preset calibration point.

[0278] As one embodiment, the smart device 1 includes two eye-tracking modules 10, which correspond to the wearer's left and right eyes respectively, to detect the eye movement state of the wearer's left and right eyes. Specifically, the two modules are respectively set to correspond to the wearer's first and second eyes, and the two modules can be modules with the same configuration or modules with different configurations.

[0279] In some embodiments, the light output module 11 includes at least a first light output component 121 and a second light output component 122. The first light output component 121 is configured to output a first emitted light signal to a first target region 41, and the second light output component 122 is configured to output a second emitted light signal to a second target region 42. The first target region 41 and the second target region 42 are at least partially non-overlapping different regions of the same eye of the wearer.

[0280] Accordingly, controller 20 is configured as follows:

[0281] When the wearer performs the eye movement behavior to be calibrated, the first light output module 121 and the light receiving module 12 are controlled to collect the real calibration information of at least one real calibration point corresponding to the eye movement behavior to be calibrated.

[0282] Based on the real calibration information, a calibration algorithm model is used to generate virtual calibration information for at least one virtual calibration point.

[0283] Based on the real calibration information and the virtual calibration information, the actual parameters corresponding to the wearer are determined.

[0284] Controller 20 is also configured to:

[0285] When the wearer performs the eye movement behavior to be calibrated, the second light output module 122 and the light receiving module 12 are controlled to collect the real calibration information of at least one real calibration point corresponding to the eye movement behavior to be calibrated.

[0286] Based on the real calibration information, a calibration algorithm model is used to generate virtual calibration information for at least one virtual calibration point.

[0287] Based on the real calibration information and the virtual calibration information, the actual parameters corresponding to the wearer are determined.

[0288] Therefore, the above methods can reduce the number of samples for real calibration points, reduce the amount of calibration computation, expand the calibration area, and thus improve the accuracy of eye-tracking calibration and subsequent eye-tracking.

[0289] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware embodiments, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as setup circuitry, such as dedicated setup circuitry. Such software can be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0290] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, referring to a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection via an intermediate medium; or a connection within two elements. Those skilled in the art will understand the specific meaning of the above terms in this application based on the specific circumstances. As used in this specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a," "an," and "the" are intended to include the plural forms.

[0291] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. The character " / " generally indicates that the preceding and following related objects are in an "and / or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, "at least one of A, B and C" includes A, B, C, AB, AC, BC, or ABC. And, unless otherwise specified, the ordinal numbers such as "first," "second," etc., mentioned in the embodiments of this application are used to distinguish multiple objects and are not used to limit the order, sequence, priority, or importance of multiple objects. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or system.

Claims

1. An eye-tracking calibration method, characterized in that, The method includes: When the wearer performs the eye movement behavior to be calibrated, the real calibration information of at least one real calibration point corresponding to the eye movement behavior to be calibrated is collected; Based on the real calibration information, a calibration algorithm model is used to generate virtual calibration information for at least one virtual calibration point. Based on the real calibration information and the virtual calibration information, the actual parameters corresponding to the wearer are determined.

2. The eye-tracking calibration method according to claim 1, characterized in that, The step of determining the actual parameters corresponding to the wearer based on the real calibration information and the virtual calibration information includes: An update coefficient is determined based on at least one set of real calibration information and virtual calibration information, and the update coefficient is used to adjust the eye-tracking algorithm model; wherein, the actual parameters include at least one of the update coefficients.

3. The eye-tracking calibration method according to claim 2, characterized in that, The real calibration data includes first light intensity data, and the virtual calibration data includes second light intensity data. Determining an update coefficient based on at least one set of real calibration information and virtual calibration information includes: Based on the first light intensity data and the second light intensity data, the current light intensity distribution information corresponding to the wearer is determined; Obtain the standard light intensity distribution information corresponding to the behavior to be calibrated, and compare the current light intensity distribution information with the standard light intensity distribution information to obtain an update coefficient.

4. The eye-tracking calibration method according to claim 1, characterized in that, The step of determining the actual parameters corresponding to the wearer based on the real calibration information and the virtual calibration information includes: Based on multiple sets of real calibration information and virtual calibration information, the actual parameters are determined, and the actual parameters are used to adjust the eye-tracking algorithm model.

5. The eye-tracking calibration method according to claim 1, characterized in that, The acquisition of real calibration information for at least one real calibration point corresponding to the eye movement behavior to be calibrated includes: Within the same detection cycle, a first emitted light signal is output to a first target area of ​​the wearer's eye within a first time period, so that the first emitted light signal is reflected on the eye to form a first reflected light signal; During the second time period, a second emitted light signal is output to the second target area of ​​the wearer's eye, so that the second emitted light signal is reflected on the eye to form a second reflected light signal; The first reflected light signal and the second reflected light signal are received using the same optical sensor, and a first detection signal corresponding to the first reflected light signal and a second detection signal corresponding to the second reflected light signal are output; wherein, the true calibration information includes at least the first detection signal and the second detection signal.

6. The eye-tracking calibration method according to claim 5, characterized in that, After outputting the second emitted light signal to the second target area of ​​the wearer's eye during the second time period, the method further includes: During the third time period, a third emitted light signal is output to the third target area of ​​the wearer's eye, so that the third emitted light signal is emitted on the eye to form a third reflected light signal; The optical sensor receives the third reflected light signal and outputs a third detection signal corresponding to the third reflected light signal; wherein, the true calibration information also includes the third detection signal.

7. The eye-tracking calibration method according to claim 1, characterized in that, The step of generating virtual calibration information for at least one virtual calibration point based on the real calibration information using a calibration algorithm model includes: The calibration algorithm model is used to extract features from the real calibration information to obtain real calibration features; Based on the real calibration features, reference calibration points that are related to the real calibration points are selected from the preset calibration database to obtain at least one virtual calibration point. Based on the virtual calibration point and the calibration algorithm model, virtual calibration information of the virtual calibration point is generated.

8. The eye-tracking calibration method according to claim 1 or 7, characterized in that, Before collecting the real calibration information of at least one real calibration point corresponding to the eye movement to be calibrated when the wearer performs the eye movement to be calibrated, the process also includes: Collect light intensity information of at least one reference calibration point within the calibration area when at least one sample wearer performs the target eye movement behavior; Based on the light intensity information, a preset calibration database is constructed.

9. The eye-tracking calibration method according to claim 8, characterized in that, The step of constructing a preset calibration database based on the light intensity information includes: The light intensity information is normalized, and the position of the reference calibration point is identified in the calibration area to obtain the current position information; The location of the center point is identified in the calibration area to obtain the center point location information; Based on the current point location information and the center point location information, the distance between the reference calibration point and the center point is calculated to obtain the current point location distance; Based on the normalized light intensity information and the current point distance, the light intensity distribution of the reference calibration point in the calibration area is modeled to obtain the preset calibration database.

10. The eye-tracking calibration method according to claim 1, characterized in that, After determining the actual parameters corresponding to the wearer based on the real calibration information and the virtual calibration information, the method further includes: Based on the actual parameters, the eye-tracking algorithm model is updated; The reflected light signal from the wearer during the detection period is collected to output a detection signal and a reference signal; Based on the detection signal, the reference signal, and the updated eye-tracking algorithm model, the eye movement state of the wearer's eyes is obtained at least during the detection time period.

11. The eye-tracking calibration method according to claim 1, characterized in that, The eye movement behavior to be calibrated includes fixation on a target calibration point. Before collecting the real calibration information of at least one real calibration point corresponding to the eye movement behavior to be calibrated when the wearer performs the eye movement behavior to be calibrated, the method further includes: Collect the wearer's object information; Based on the object information, the wearer's current eye-tracking calibration state is determined; When the eye-tracking calibration state indicates that the wearer has not performed eye-tracking calibration before the current moment, the eye-tracking behavior instruction to be calibrated is output so that the wearer gazes at the target calibration point based on the instruction.

12. The eye-tracking calibration method according to claim 1, characterized in that, The eye movement behavior to be calibrated includes fixation on a target calibration point. After determining the actual parameters corresponding to the wearer based on the real calibration information and the virtual calibration information, the method further includes: Based on preset acquisition conditions, the eye movement behavior command to be calibrated is dynamically output so that the wearer gazes at the target calibration point based on the command.

13. An eye-tracking calibration device, characterized in that, It includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for implementing communication between the processor and the memory; wherein, when the computer program is executed by the processor, it implements the steps of the eye-tracking calibration method as described in any one of claims 1 to 12.

14. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to implement the eye-tracking calibration method as described in any one of claims 1 to 12.

Images