Smart glasses and multi-point light source device and eye movement tracking method thereof

Abstract

This application relates to the field of eye-tracking technology, and more particularly to a smart glasses device and its multi-point light source device and eye-tracking method. Addressing the technical problems of existing infrared light sources, such as large divergence angles, low effective light energy utilization, excessive stray light, and insufficient stability in the shape and spacing of multiple light spots during formation, which affect the accuracy and robustness of eye tracking, this application proposes an integrated multi-point light source structure. The device sets multiple independently driveable auxiliary light sources on a substrate within the package, and each auxiliary light source is equipped with a corresponding microlens. The microlenses directionally refract and control the divergence angle of the emitted infrared light, enabling each beam to form a controllable light spot on a preset reflective surface. This solution can improve the effective utilization of infrared light within a limited package space, reduce stray light interference, and enhance the consistency, boundary clarity, and positional stability of multiple light spots, thereby improving eye-tracking positioning accuracy and reducing image processing complexity.

Description

Technical Field

[0001] This application relates to the field of eye-tracking technology, and more particularly to a smart pair of glasses, a multi-point light source device thereof, and an eye-tracking method. Background Technology

[0002] In eye-tracking systems, existing solutions typically use an infrared light source in conjunction with an eye-tracking camera to detect gaze based on the positional relationship between the center of the pupil and the corneal reflective spot.

[0003] However, existing infrared light sources often have large divergence angles, which can easily disperse light energy to non-target areas, reducing the imaging signal-to-noise ratio. At the same time, the shape and spacing of the reflected light spots are not stable enough, which can affect the accuracy of spot positioning and increase the complexity of multi-spot recognition and line-of-sight calculation.

[0004] Therefore, how to improve the light energy utilization and spot positioning accuracy in eye tracking under the condition of miniaturization of head-mounted devices, so as to balance tracking accuracy and system complexity, has become a technical problem that needs to be solved. Summary of the Invention

[0005] This application provides a smart pair of glasses, a multi-point light source device, and an eye-tracking method to solve the aforementioned technical problems. The multi-point light source device addresses the application requirements of multi-point infrared illumination in eye-tracking scenarios. By integrating multiple light-emitting units and coordinating their output light paths, the infrared light output from each light-emitting point meets miniaturization packaging requirements while possessing more suitable emission characteristics for target area illumination and imaging recognition. This improves upon existing solutions that suffer from light energy dispersion, insufficient stability of reflected light spots, and limited accuracy in multi-spot positioning and recognition.

[0006] In a first aspect, embodiments of this application provide a multi-point light source device, including: a package, multiple auxiliary light sources, multiple sets of electrode pins, and multiple microlenses;

[0007] The package includes a package substrate and a light-transmitting surface, and the package is used to encapsulate multiple auxiliary light sources;

[0008] The auxiliary light source corresponds one-to-one with the electrode pin. Inside the package, each auxiliary light source is set at a first preset position on the package substrate through a corresponding electrode pin. The electrode pin is used to input a driving current to the corresponding auxiliary light source. The auxiliary light source is used to emit infrared light through the light-transmitting surface.

[0009] Each microlens corresponds to one of the auxiliary light sources. Each microlens is disposed on the light-emitting side of the corresponding auxiliary light source. Each microlens is located inside the package or on the light-transmitting surface. Each microlens is used to refract the diverging light beam emitted by the corresponding auxiliary light source into a preset divergence angle range to form a light spot on a preset reflective surface.

[0010] In one possible implementation, each of the microlenses is disposed on the light-emitting side of the corresponding auxiliary light source, including:

[0011] Each of the microlenses is disposed on the light-emitting side of the corresponding auxiliary light source, and the optical axis of each microlens is offset by a preset distance relative to the geometric center of the corresponding auxiliary light source, so that the light spots formed by the infrared light emitted by each auxiliary light source on the preset reflective surface are separated from each other.

[0012] In one possible implementation, the device further includes a main light source and electrode pins corresponding to the main light source; wherein,

[0013] Inside the package, the main light source is positioned at a second preset location on the package substrate via an electrode pin corresponding to the main light source.

[0014] In one possible implementation, the device further includes a microlens corresponding to the main light source, the microlens corresponding to the main light source being disposed on the light-emitting side of the main light source, and the microlens corresponding to the main light source being located inside the package or on the light-transmitting surface.

[0015] In one possible implementation, the microlens corresponding to the auxiliary light source is configured to refract the diverging beam emitted by the corresponding auxiliary light source into a preset first divergence angle range.

[0016] The microlens corresponding to the main light source is configured to refract the diverging beam emitted by the main light source into a preset second divergence angle range; wherein the second divergence angle range is greater than the first divergence angle range.

[0017] In one possible implementation, the microlens is an aspherical or freeform surface.

[0018] In one possible implementation, when each of the microlenses is located on the light-transmitting surface, each of the microlenses is integrally formed with the light-transmitting surface.

[0019] Secondly, embodiments of this application provide a smart glasses, the smart glasses including a camera and a multi-point light source device as described in the first aspect;

[0020] The multi-point light source device is used to emit infrared light into the user's eyeball and form multiple light spots on the corneal surface of the user's eye;

[0021] The camera is used to capture images of the user's eyes after the multi-point light source device emits infrared light towards the user's eyeballs.

[0022] Thirdly, embodiments of this application provide an eye-tracking method applied to smart glasses as described in the second aspect, the method comprising:

[0023] If a multi-point light source device emits infrared light into the user's eyeball, multiple light spots are formed on the surface of the user's cornea, thus acquiring the first image of the user's eye;

[0024] Identify the pupil and each of the light spots from the first image, and extract the first position information of the pupil and the second position information of each of the light spots;

[0025] Based on the first location information and each of the second location information, the user's eye gaze direction is determined.

[0026] In one possible implementation, if the multi-point light source device includes multiple auxiliary light sources and a main light source, the method further includes:

[0027] Within a preset first time period, the main light source is turned on to acquire a second image of the user's eyes;

[0028] During the preset second time period, all auxiliary light sources are lit to form multiple light spots on the corneal surface of the user's eye, thereby acquiring a third image of the user's eye.

[0029] Based on the second and third images, the user's eye gaze direction is determined.

[0030] In one possible implementation, determining the user's eye gaze direction based on the second and third images includes:

[0031] Identify the pupil from the second image and extract the third position information of the pupil;

[0032] Identify each light spot from the third image and extract the fourth position information of each light spot;

[0033] Based on the third location information and each of the fourth location information, the user's eye gaze direction is determined.

[0034] The smart glasses, multi-point light source device, and eye-tracking method provided in this application embodiment, by setting multiple auxiliary light sources corresponding one-to-one with electrode pins in the package, and setting microlenses on the light-emitting side of each auxiliary light source, so that the diverging light beam emitted by the corresponding auxiliary light source falls into a preset divergence angle range after refraction and forms a light spot on a preset reflective surface, can improve the directional utilization efficiency of infrared light under miniaturized integration conditions, and improve the stability of the shape and spacing of the formed light spot, thereby improving the light spot positioning accuracy in eye tracking and reducing the complexity of multi-spot recognition and gaze calculation. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0036] Figure 1 A side view of the multi-point light source device provided in this application;

[0037] Figure 2 Left view of the multi-point light source device provided in this application;

[0038] Figure 3 Top view of the multi-point light source device provided in this application Figure 1 ;

[0039] Figure 4 The optical path diagram provided in this application when the microlens is not set with an eccentric distance;

[0040] Figure 5 Optical path diagram when setting the eccentricity distance for the microlens provided in this application;

[0041] Figure 6 Top view of the multi-point light source device provided in this application Figure 2 ;

[0042] Figure 7 This is a schematic diagram of the structure of the smart glasses provided in this application;

[0043] Figure 8 A flowchart illustrating the eye-tracking method provided in this application;

[0044] Figure 9 A schematic diagram of the eye-tracking device provided in this application;

[0045] Figure 10 This is a schematic diagram of the eye-tracking device provided in this application.

[0046] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0048] Eye-tracking technology is primarily used in smart glasses, virtual reality devices, augmented reality devices, and other head-mounted human-computer interaction terminals. It typically combines infrared illumination with eye-tracking imaging to collect information on the user's pupil position and corneal reflection, and then estimates the direction of gaze. Such systems generally require integrating light source devices, imaging devices, and related electrical connections within a limited space in the frame, temples, or near the eyes. Therefore, this places high demands on the miniaturization of the devices, the stability of optical output, and assembly adaptability.

[0049] Current eye-tracking solutions typically employ one or more infrared light-emitting devices to project infrared light onto the eye, utilizing the positional relationship between the reflected light spot on the corneal surface and the center of the pupil to detect gaze. These infrared light sources are usually directly emitted from a packaged light-emitting chip, resulting in a large beam divergence angle. The proportion of light energy entering the effective area of ​​the eye is limited, with some light energy scattering to the eyelids, eyelashes, or periorbital region, thus reducing the signal-to-noise ratio when the eye-tracking camera acquires images.

[0050] In multi-spot detection scenarios, if multiple auxiliary light sources are used to form multiple corneal reflective spots, the morphology, brightness distribution, and spatial spacing of the resulting spots are often unstable due to the difficulty in uniformly controlling the divergence characteristics of each light source. Especially under the small-size packaging conditions of head-mounted devices, the arrangement space between multiple light sources is limited, which can easily lead to excessively close spacing between reflected spots, unclear boundaries, or even local adhesion, making the spot recognition, matching, and subsequent gaze calculation processes more complex.

[0051] The aforementioned defects directly affect the accuracy and robustness of eye-tracking results. On the one hand, increased stray light interferes with the extraction of effective reflection information; on the other hand, unstable spot shape and position increase positioning errors, requiring the system to adopt more complex image processing and matching strategies, which is not conducive to achieving high-precision eye tracking in lightweight, low-power head-mounted devices.

[0052] In view of this, how to improve the effective utilization rate of infrared light within a limited packaging space, and to make multiple reflected light spots have a more controllable divergence range and a more stable formation effect, has become an urgent technical problem to be solved.

[0053] To address the aforementioned technical problems, a smart glasses device and its multi-point light source device and eye-tracking method are provided. The device includes a package, multiple auxiliary light sources, multiple sets of electrode pins, and multiple microlenses corresponding to the auxiliary light sources. The multiple auxiliary light sources are disposed at a first preset position on the package substrate and are driven by inputting a driving current through the corresponding electrode pins. Each microlens is disposed on the light-emitting side of the corresponding auxiliary light source and is located inside the package or on the light-transmitting surface, for refracting the divergent light beam emitted by the corresponding auxiliary light source into a preset divergence angle range to form a light spot on a preset reflective surface.

[0054] The above structure enables integrated control of multi-point infrared light emission within a miniaturized system architecture for head-mounted devices, providing a foundation for improving spot positioning accuracy and reducing system processing complexity.

[0055] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0056] Figure 1 This is a side view of the multi-point light source device provided in this application. Figure 2 This is a left view of the multi-point light source device provided in this application. Figure 3 Top view of the multi-point light source device provided in this application Figure 1 ,like Figures 1 to 3 As shown, the device includes: a package 101, multiple auxiliary light sources 102, multiple sets of electrode pins 103, and multiple microlenses 104. The package 101 includes a package substrate 105 and a light-transmitting emission surface 106. The package 101 is used to integrate and package the multiple auxiliary light sources 102 to form an optical emission channel. The multiple auxiliary light sources 102 correspond one-to-one with the multiple sets of electrode pins 103 and are disposed at a first preset position on the package substrate 105. The electrode pins 103 are used to input driving current to the corresponding auxiliary light source 102 to make it emit infrared light. The multiple microlenses 104 correspond one-to-one with the multiple auxiliary light sources 102 and are disposed on the light-emitting side of the corresponding auxiliary light source 102. Each microlens 104 is located inside the package 101 or on the light-transmitting emission surface 106, and is used to refract the divergent beam emitted by the corresponding auxiliary light source 102 into a preset divergence angle range to form a light spot on a preset reflective surface.

[0057] The package refers to the structure used to integrate, position, and encapsulate multiple auxiliary light sources, electrode pins, and microlenses. Its function is to provide unified mechanical support and optical path boundaries for each light-emitting unit, and to output the infrared light generated inside to the external space through the light-transmitting surface.

[0058] The package includes at least a package substrate and a light-transmitting exit surface in its structure. The package substrate is used to support the auxiliary light source and its electrical connection structure, and the light-transmitting exit surface is used to release the shaped light beam outward. The two together define the light path propagation path and form a closed or quasi-closed optical cavity.

[0059] For example, the package can be a circular package, a square package, or a rectangular package to adapt to the arrangement space of different head-mounted devices; the package substrate can be a ceramic substrate, a glass substrate, a silicon substrate, or a metal-based composite substrate to take into account thermal stability, electrical connection wiring capability, and mechanical strength; the light-transmitting surface can be formed by sapphire sheet, glass cover plate, transparent resin layer, or high light transmittance organic silicon material to meet the transmission requirements of the near-infrared band.

[0060] The package is typically a millimeter-scale microstructure. Its planar dimensions and thickness can be matched according to the chip height, microlens focal length, and light emission angle requirements. The thickness relationship between the light-transmitting surface and the package substrate can be set to a thinner cover plate structure, an equal-thickness structure, or a relatively thick support structure according to the light path refraction conditions.

[0061] It should be understood that the above structural forms are only for demonstration purposes and are not limited. The package itself usually does not move mechanically during operation, but rather acts as a static load-bearing structure and works with other components to complete optical output.

[0062] Auxiliary light source refers to a miniature light-emitting device set inside the package to receive external driving current and convert it into infrared light output. Its function is to provide multiple independent or time-controlled illumination points for scenarios such as eye tracking, and to form a light spot for reflective imaging in conjunction with microlenses.

[0063] The auxiliary light sources are respectively set at the first preset position on the packaging substrate and form a one-to-one correspondence with the corresponding electrode pins. The auxiliary light sources can be arranged at a predetermined interval to match the optical axis position of the corresponding microlens.

[0064] In one possible implementation, the auxiliary light sources can be Micro-LED (Light Emitting Diode), infrared LED, VCSEL (Vertical Cavity Surface Emitting Laser), or other miniature near-infrared light-emitting chips. The chip materials can be GaAs (Gallium Arsenide), AlGaAs (Aluminum Gallium Arsenide), InGaAs (Indium Gallium Arsenide), or semiconductor materials compatible with near-infrared light emission.

[0065] The emitting surface of the auxiliary light source can be square, rectangular, or nearly circular to accommodate different lens apertures and light-emitting windows. The size of the auxiliary light source is typically in the range of tens to hundreds of micrometers. Its emitting surface size, chip thickness, and installation tolerances must be coordinated with the aperture and focal length of the microlens to ensure that the diverging beam enters the preset divergence angle range under the action of the microlens.

[0066] When the auxiliary light source is working, it outputs infrared light in the form of electroluminescence and propagates outward through the light-transmitting exit surface. Its output state can be controlled by independent driving, common cathode driving, common anode driving or array driving. If necessary, time-division switching can also be used to achieve sequential excitation of multiple light spots.

[0067] Multiple sets of electrode pins refer to conductive structures used to input driving current to the corresponding auxiliary light source and complete electrical connection. Their function is to realize independent power supply, timing control and electrical isolation of each auxiliary light source, and to connect the external control circuit with the light-emitting unit inside the package.

[0068] Multiple sets of electrode pins correspond one-to-one with the auxiliary light source and are arranged between the external and internal electrical connection paths of the package, forming a current path with the metal wiring layer, pads or bonding points on the package substrate.

[0069] In one possible implementation, the electrode pins can be metal leads, pad pins, bump solder balls, or board-level solder feet, and their materials can be gold, silver, copper, nickel, gold-plated copper alloys, or composite conductive materials to meet the requirements of conductivity, solderability, and reliability.

[0070] The linewidth, line spacing, or diameter of the electrode pins are typically matched to the chip pad size. Typical pin diameters range from 20 micrometers to 500 micrometers, and the spacing can be set according to multi-chip layout and packaging process.

[0071] The electrode pins do not require large mechanical displacement during operation. Instead, they transfer electrical energy to the corresponding auxiliary light source through continuous conduction, pulse drive, or timing switching. In alternative implementations, flexible circuit leads can be used to replace traditional rigid pins to adapt to different installation spaces.

[0072] A microlens is a miniature optical element set on the light-emitting side of a corresponding auxiliary light source. Its function is to refract, collimate, or moderately converge the diverging light beam emitted by the auxiliary light source, so that the light beam enters a preset divergence angle range and forms a light spot with clear boundaries and stable position on a preset reflective surface.

[0073] Each microlens corresponds to an auxiliary light source and is located on the light-emitting side of the auxiliary light source. It can be placed inside the package or directly on the light-transmitting surface, forming an optical path relationship with the auxiliary light source, the light-transmitting surface, and the encapsulating colloid.

[0074] In one possible implementation, the microlens can be a single lens or an array of lenses, and the material can be optical resin, silicone, glass, polymer or high refractive index transparent material to accommodate different refractive indices and processing methods.

[0075] The aperture of a microlens is typically tens to hundreds of micrometers, and the focal length can be 0.05 mm to 1 mm. The lens aperture usually covers the light-emitting surface of the corresponding auxiliary light source. The positional relationship between the microlens and the corresponding auxiliary light source can be set according to the assembly requirements to ensure the consistency of the light output direction.

[0076] Microlenses do not undergo macroscopic displacement during operation, but rely on static refraction to change the propagation path of the light beam. In alternative implementations, microlenses can also be replaced by diffractive optical elements, microprisms, metasurface optical structures, or combined lens structures.

[0077] It should be noted that, Figures 1 to 3 The example uses four auxiliary light sources and a circular package. In practice, the number of auxiliary light sources can be four or more than one. The package can also be square or rectangular. This application does not limit the specific number of auxiliary light sources.

[0078] Based on the above analysis, it can be seen that the multi-point light source device provided in this application, during operation, involves an external driving circuit applying driving current to each auxiliary light source inside the package through multiple sets of electrode pins. This causes multiple auxiliary light sources to enter the infrared emission state according to a predetermined synchronous or time-division method. The emitted divergent beams are first shaped by corresponding microlenses. The microlenses refract and correct the incident light according to their own surface parameters and installation position, limiting the originally divergent beams to a preset divergence angle range. The beams are then output outward through the light-emitting surface of the package and projected onto a preset reflective surface. Since each auxiliary light source corresponds one-to-one with each microlens and their positions on the package substrate are fixed, the direction, angle, and emission area of ​​each emitted beam can be structurally constrained, resulting in multiple light spots with a clear spatial distribution and stable morphological boundaries. By integrating the light-emitting unit, conductive unit, and microlens into the same package, external optical alignment components and discrete assembly steps can be reduced, enabling synchronous multi-point infrared illumination within a limited package space. This allows more effective light energy to enter the target reflection area rather than being scattered to the eyelids, eyelashes, or periorbital area. Therefore, subsequent eye-tracking imaging devices can obtain multiple light spots with relatively clear boundaries and relatively stable positions when acquiring corneal reflection images, thereby reducing the complexity of light spot recognition and matching, and making the gaze estimation process more suitable for implementation in miniaturized, low-power head-mounted devices.

[0079] It should be understood that the above examples are for illustrative purposes only and are not limiting. Without departing from the spirit of this application, the package material, auxiliary light source type, electrode pin form and microlens structure can be replaced or modified accordingly.

[0080] Based on the aforementioned embodiments, each microlens is further disposed on the light-emitting side of the corresponding auxiliary light source, including:

[0081] Each microlens is positioned on the light-emitting side of its corresponding auxiliary light source, and the optical axis of each microlens is offset by a preset distance relative to the geometric center of its corresponding auxiliary light source, so that the infrared light emitted by each auxiliary light source forms light spots on a preset reflective surface that are separated from each other.

[0082] Figure 4 This is the optical path diagram of the microlens provided in this application when no eccentricity distance is set. Figure 5 The optical path diagram when setting the eccentricity distance for the microlens provided in this application. For example... Figure 4 and Figure 5As shown, assuming that multiple auxiliary light sources are distributed in a circle within the package, when the microlenses are not set with an eccentric distance, the position of the light spot formed by each microlens on the preset reflective surface does not shift, and there may be edge adhesion between the light spots; when the microlenses are set with an eccentric distance, the position of the light spot formed by each microlens on the preset reflective surface shifts, increasing the distance between the light spots, thereby reducing the phenomenon of edge adhesion between the light spots.

[0083] Multiple auxiliary light sources are distributed in a circular pattern on the package substrate. The microlenses are set with an off-center distance of a preset distance. If the distance between the geometric center of each auxiliary light source and the center of the package is... On a plane at a distance L from the light source, the equivalent radius of the light spot formed by the light beams emitted by each auxiliary light source. for:

[0084] (1);

[0085] In the formula, It represents the angle of deflection of the principal ray of the emitted light beam relative to the optical axis after refraction by the lens.

[0086] Based on this, the magnification factor can be defined. for:

[0087] (2);

[0088] In the formula, M is the amplification factor.

[0089] Eccentricity corresponding to microlens The relationship between the effective focal length of the microlens and the effective focal length can be expressed as the following formula (3):

[0090] (3);

[0091] In the formula, This indicates the effective focal length of the microlens.

[0092] Based on the above relationships, it can be seen that the equivalent magnification of the light source spacing can be achieved through optical means without changing the package size and the placement of the auxiliary light source. The typical range of the magnification factor M is 1.1 to 2.0, meaning that the equivalent light source spacing can be magnified by 10% to 100%. This resolves the contradiction between miniaturized packaging and light spot separation.

[0093] It should be noted that the off-center distances of auxiliary light sources located at different positions can be equal or unequal. For example, a camera is included in a head-mounted device to take pictures of the eyes. Since the camera is usually positioned on one side of the head-mounted device, its angle of observation of the cornea is not directly opposite, resulting in perspective distortion. The auxiliary light source on the side closer to the camera can have a smaller off-center distance, while the auxiliary light source on the side farther from the camera can have a relatively larger off-center distance. This differentiated off-center design can compensate for the perspective distortion of the light spot caused by the camera's perspective, resulting in a more regular and uniform distribution of the light spot array ultimately captured by the camera in the image.

[0094] For example, when the system is started, multiple auxiliary light sources in the package emit light simultaneously or in a preset sequence after the corresponding electrode pins are driven by the input of driving current. The infrared light first enters the corresponding microlens from the light-emitting side of each auxiliary light source. Since the optical axis of each microlens is offset from the geometric center of the auxiliary light source by a preset offset, the main light rays after passing through the lens will be emitted in a direction that is not completely coincident with the geometric center of the device, and after passing through the light-transmitting emission surface, they form spatially staggered divergent beams.

[0095] As these diverging beams are directed toward a preset reflective surface, such as the corneal surface of the user's eye, each beam of infrared light will form multiple separate light spots under different incident angle conditions, thereby preventing multiple light spots from getting close, overlapping, or sticking together at the edges on the reflective surface.

[0096] Based on the above settings, it can be seen that this eccentric setting can improve the spatial differentiation between multiple corneal reflective spots without increasing the number of auxiliary light sources or significantly expanding the packaging area, reduce image recognition ambiguity caused by spot overlap, and thus facilitate the stable extraction and matching of spot positions by the eye-tracking camera, reduce the dependence of subsequent algorithms on complex separation processing, and improve the detection accuracy and robustness of the head-mounted eye-tracking system under miniaturized packaging conditions.

[0097] It should be understood that the above examples are for demonstration purposes only and are not limiting. Without departing from the technical concept of this application, the eccentric direction, the magnitude of the offset, and the specific surface shape and material of the microlens can all be adjusted according to actual application requirements.

[0098] Figure 6 Top view of the multi-point light source device provided in this application Figure 2 ,like Figure 6 As shown, the above-mentioned device also includes a main light source 107 and corresponding electrode pins for the main light source; wherein,

[0099] Inside the package, the 107 main light source is positioned at the second preset position on the package substrate via the electrode pins corresponding to the main light source.

[0100] The main light source refers to a near-infrared emitting component located inside the package to provide a larger illumination coverage area compared to the auxiliary light source. It and the auxiliary light source form a main-auxiliary synergistic light emission system. A microlens is provided at the light-emitting side of the auxiliary light source to refract the emitted light beam within a preset divergence angle range, while no microlens is provided at the light-emitting side of the main light source. Therefore, the emitted light beam of the main light source will cover a larger area.

[0101] The main light source functions to provide flood illumination to the target area of ​​the eye in eye-tracking or similar head-mounted human-computer interaction scenarios. This complements the overall brightness distribution beyond the directional light spot formed by the microlens shaping of the auxiliary light source, thus providing a more uniform imaging basis for pupil contours, iris texture, and corneal reflection characteristics. The main light source is electrically connected to the external driving circuit through corresponding electrode pins. These electrode pins are used both to introduce driving current into the main light source and to achieve its mechanical fixation and electrical lead-out on the package substrate. Therefore, the main light source can be stably positioned at the second preset position within the package, and its connection method is similar to that of the auxiliary light source.

[0102] The second preset position can be arranged according to the package size, heat distribution and optical relationship with the auxiliary light source. It can be arranged at intervals on the plane with the first preset position where the auxiliary light source is located, or it can be arranged in a center and edge partition, a ring arrangement or an array arrangement, so that the main light source undertakes background lighting and the auxiliary light source undertakes local spot lighting, and the two complement each other.

[0103] In one possible implementation, the main light source can be a single infrared Micro-LED (Micro Light Emitting Diode), a VCSEL chip, or a large-area near-infrared light-emitting chip. Infrared Micro-LEDs are suitable for achieving low-power, fast-response planar light emission, VCSEL chips are suitable for achieving high electro-optical conversion efficiency and easy array integration with wide-angle light emission, while large-area near-infrared light-emitting chips are suitable for obtaining a wider illumination coverage with fewer devices.

[0104] The semiconductor material used in the main light source can be GaAs-based, AlGaAs-based, InGaAs-based, or other III-V group material systems that match the near-infrared band. The specific selection can be made in combination with the target emission wavelength, external quantum efficiency, and thermal stability.

[0105] The packaging methods can include flip-chip bonding, bonding, or wafer-level packaging. Flip-chip bonding helps to shorten the current path and reduce parasitic inductance, bonding facilitates a reliable connection with the packaging substrate, and wafer-level packaging helps to further reduce the overall size and improve device consistency.

[0106] The main light source can be a square, rectangular, or nearly circular light-emitting unit, or it can be a spliced ​​surface composed of multiple small light-emitting units; its light-emitting surface can be a plane, a micro-arc surface, or a flat surface that matches the light-emitting window.

[0107] The electrode pins corresponding to the main light source can be metal pads, lead frame terminals, bump arrays, or flexible leads. The materials can be copper, silver, gold, nickel-gold composite layers, aluminum, or tin-plated metal layers to meet the requirements of conductivity, weldability, and corrosion resistance.

[0108] The main light source can typically be similar in size to or slightly larger than the auxiliary light source. Its individual side length, diameter, or effective light-emitting area can be set from hundreds of micrometers to several millimeters depending on the target illumination range. The distance between the second preset position and the edge of the package substrate, the adjacent auxiliary light source, and the light-transmitting surface is usually matched according to the wiring density and heat dissipation channels. It can typically be in the range of hundreds of micrometers to several millimeters to balance optical output, thermal management, and package manufacturability.

[0109] When the system starts, the external driving circuit applies a predetermined driving current to the electrode pins corresponding to the main light source and the electrode pins corresponding to each auxiliary light source. The main light source first generates near-infrared radiation at a second preset position inside the package and outputs a wide-angle floodlight illumination outward through the light-emitting surface of the package to form a basic lighting background for the target area. At the same time, multiple auxiliary light sources located at the first preset position emit light synchronously or sequentially under the power supply of their respective electrode pins. After the diverging beam is refracted and shaped by the corresponding microlenses, it is projected onto the preset reflective surface at a controlled divergence angle to form multiple mutually separated light spots. Since the main light source provides overall brightness compensation, the light spots output by the auxiliary light sources can obtain clearer boundaries and more stable contrast on the reflective surface. Furthermore, the spatial partitioning of the main and auxiliary light sources can avoid significant optical path interference between the two types of light outputs inside the package.

[0110] Based on the above working process, it can be seen that, on the one hand, the structure improves the overall imaging uniformity of the pupil and eye area by using the flood illumination of the main light source, and on the other hand, it works in conjunction with the multi-point light spots formed by the auxiliary light source to improve the recognition stability of corneal reflection features and reduce the image processing complexity caused by insufficient local illumination or uneven distribution of stray light, thereby improving the positioning accuracy and robustness of the eye tracking system.

[0111] It should be understood that the above examples are for illustrative purposes only and are not limiting. The specific number, arrangement, emission wavelength, and electrode pin structure of the main light source in this application can be equivalently replaced according to the actual package space and driving requirements.

[0112] In some specific implementations, the above-mentioned device also includes a microlens corresponding to the main light source, which is disposed on the light-emitting side of the main light source, and is located inside the package or on the light-transmitting surface.

[0113] Among them, the microlens corresponding to the main light source is a miniature optical element that is set on the light-emitting side of the main light source and performs optical control on the diverging beam of the main light source. It is used to shape the direction, control the diffusion or collimate the infrared light output by the main light source so that the main light source forms a beam divergence characteristic different from multiple auxiliary light sources, thereby meeting the usage requirements of main illumination, ambient lighting or reference illumination in the system, and forming a differentiated light-emitting configuration with the microlenses corresponding to the auxiliary light sources.

[0114] In this embodiment, the microlens corresponding to the main light source is disposed on the light-emitting side of the main light source, and can be located inside the package or on the light-transmitting surface, so that it maintains an optical path alignment relationship with the main light source. The light beam emitted by the main light source passes through the microlens and the light-transmitting surface in sequence and is emitted outward, or when the microlens is located on the light-transmitting surface, it is directly refracted and output through the emission surface.

[0115] Since the microlens, together with the encapsulation substrate, encapsulation colloid, and light-transmitting exit surface, defines the propagation path of the main light source, it can effectively control the light emission range of the main light source within a limited encapsulation space, reducing the probability of light energy escaping to non-target areas.

[0116] For example, in one possible implementation, the microlens corresponding to the main light source can be a spherical microlens, an aspherical microlens, a freeform microlens, a composite lens, or a microlens with a beam-expanding structure. Spherical microlenses are easy to process and shape, aspherical microlenses can compensate for spherical aberration, freeform microlenses are suitable for customization according to the installation space and target light pattern, and composite lenses can achieve a more stable beam shaping effect through a multi-layer structure. The material can be transparent resin, glass, silicone, polymer, or high refractive index optical material to take into account light transmittance, heat resistance, and packaging compatibility.

[0117] Furthermore, the aperture and focal length of the microlens corresponding to the main light source are usually designed to match the luminous area, chip size and target divergence angle of the main light source. In one possible implementation, the aperture can be tens to hundreds of micrometers and the focal length can be 0.1 mm to several millimeters. The microlens corresponding to the main light source and the microlens corresponding to the auxiliary light source can be configured differently in terms of focal length, aperture or curvature parameters so that the main light source forms a larger coverage area or stronger uniform illumination, while the auxiliary light source forms a controlled light spot that is more suitable for corneal reflection imaging.

[0118] It should be understood that the above examples are for demonstration purposes only and are not limiting. The microlens corresponding to the main light source can also be replaced with a light-transmitting mask, a Fresnel structure, a diffraction diffuser, or a composite micro-optical array without departing from the technical concept of this application, in order to achieve similar optical control functions.

[0119] When the system starts, the main light source, powered by its corresponding electrode pins, generates infrared light output. The beam first propagates along a predetermined optical path to the microlens corresponding to the main light source on its output side. The microlens, using its refractive surface or composite optical structure, redistributes the angle of the incident light, shaping the beam with a large divergence angle into outgoing light within the target divergence angle range. It can also diffuse, collimate, or locally expand the outgoing light as needed by the system, thus forming a more uniform and stable illumination area outside the package. If the microlens corresponding to the main light source is located inside the package, it forms an internal optical path modulation interface with the encapsulating colloid, enabling primary refraction control before the light reaches the light-transmitting exit surface. If it is located on the light-transmitting exit surface, the exit surface can be directly used as the final optical interface, reducing internal optical path loss and simplifying the package structure. Simultaneously, multiple auxiliary light sources and their corresponding microlenses form multiple separate reflective spots according to a preset method. The main light source provides background illumination or compensation illumination through its corresponding microlenses, resulting in a more stable light intensity distribution in the eye imaging area.

[0120] Based on the above working process, it can be seen that by setting a dedicated microlens on the light-emitting side of the main light source, this application can achieve independent control of the light pattern of the main light source without significantly increasing the packaging space occupied. This enables the main light source and the auxiliary light source to cooperate in the illumination function, thereby improving the effective utilization rate of infrared light, reducing stray light interference, and enhancing the imaging stability and recognition accuracy of the eye-tracking system in miniaturized head-mounted devices.

[0121] Based on the aforementioned embodiments, the microlens corresponding to the auxiliary light source is further configured to refract the diverging beam emitted by the corresponding auxiliary light source into a preset first divergence angle range; the microlens corresponding to the main light source is configured to refract the diverging beam emitted by the corresponding main light source into a preset second divergence angle range; wherein the second divergence angle range is greater than the first divergence angle range.

[0122] Among them, the microlens corresponding to the auxiliary light source can be defined as a miniature refractive element installed directly in front of the auxiliary light source. Its function is to perform secondary refraction and angle compression on the initial large-angle divergent infrared light emitted by the auxiliary light source, so that the emitted light can form a small, bright and relatively stable reflective spot while meeting the illumination distance requirements. The microlens corresponding to the main light source can be defined as a miniature beam shaping element installed directly in front of the main light source. Its function is to appropriately expand the infrared light output by the main light source so that it covers a larger eye area or camera field of view, thereby taking into account both local spot formation and overall illumination compensation in a multi-point light source device.

[0123] Both are typically positioned and aligned by a base positioning structure within the package. The microlens corresponding to the auxiliary light source is arranged coaxially or nearly coaxially with the light-emitting center of the auxiliary light source, and the microlens corresponding to the main light source is arranged coaxially or nearly coaxially with the light-emitting center of the main light source. The bottom surface of the lens can be fixed to the package substrate by bonding, embedding, molding, or secondary injection molding, while the top surface of the lens faces the light-transmitting exit surface so that the light beam can be emitted directly after refraction.

[0124] The microlenses corresponding to the auxiliary light source and the main light source can be designed with different radii of curvature, center thickness, refractive index materials, or surface microstructure parameters to achieve different divergence angle control effects. For example, the microlens corresponding to the auxiliary light source can use a smaller radius of curvature to enhance the converging effect, while the microlens corresponding to the main light source can use a larger radius of curvature or local diffusion texture to weaken the converging intensity. Correspondingly, the first divergence angle range can be set to a smaller range, and the second divergence angle range can be set to a larger range. The difference between the two can be matched according to the spot spacing, illumination radius, and imaging field of view.

[0125] Based on the above analysis, it can be seen that differentiated divergence angle control makes the reflected light spots of the auxiliary light source easier for the image sensor to recognize stably, while the main light source can supplement the illumination information of a wider area, thereby reducing the impact of local shadows and dark edge areas on imaging and improving the adaptability and robustness of the multi-point light source system.

[0126] When the system starts up, the drive circuit inputs a set current to each light source through the electrode pins corresponding to the auxiliary light source and the main light source, so that each light source emits infrared diverging light synchronously or sequentially at a preset position on the package substrate. The beam emitted by the auxiliary light source is first refracted by its corresponding microlens and compressed into a preset first divergence angle range, and then projected onto the eye or other preset reflective surfaces at a narrower exit angle, thereby forming a small and stable light spot on the reflective surface. The beam emitted by the main light source is refracted by its corresponding microlens and adjusted into a preset second divergence angle range. Since the second divergence angle range is larger than the first divergence angle range, the main light source beam can cover a larger area and provide a wider range of background illumination or supplementary lighting support. In this way, during image acquisition, the concentrated light spot formed by the auxiliary light source helps improve the distinguishability and positioning accuracy of the light spot boundary, while the wider illumination provided by the main light source helps improve the overall brightness uniformity and reduce local underexposure. The combined effect of these two sources enables layered utilization and differentiated control of infrared light energy within a limited encapsulation space, thereby reducing the interference of stray light on effective reflection information and lowering the complexity of subsequent image processing and light spot matching. This improves the recognition stability and measurement accuracy in eye-tracking or other near-field optical detection scenarios. It should be understood that the above example is merely illustrative and not limiting.

[0127] In some possible implementations, the microlens is an aspherical or freeform surface.

[0128] When the system is started, each auxiliary light source or main light source inside the package emits infrared light under the drive of the corresponding electrode pin. The original beam is refracted and shaped after passing through the microlens on its respective light-emitting side. If the microlens adopts an aspherical or free-form surface, its continuously changing curvature can differentially modulate the light at different incident angles, so that the diverging beam forms a light field distribution that is closer to the preset divergence angle range when it passes through the package.

[0129] For the microlens corresponding to the auxiliary light source, the beam can be shaped to form multiple light spots with clearer boundaries and higher separation on the preset reflective surface; for the microlens corresponding to the main light source, it can form different divergence characteristics from the auxiliary light source to meet the differentiated needs of the main and auxiliary light sources in terms of illumination coverage and light spot shape.

[0130] Because aspherical or freeform surfaces allow for more precise control over the angular distribution and energy concentration of the emitted light beam, ineffective scattering of infrared light towards the eyelids, eyelashes, and periorbital area can be reduced, improving the utilization rate of effective incoming light energy and reducing background interference in corneal reflection images. Consequently, the light spot on the pre-defined reflective surface exhibits greater consistency and repeatability in terms of position, size, and brightness distribution, thereby reducing the difficulty of feature extraction in subsequent eye-tracking camera recognition and matching processes, improving the accuracy and robustness of gaze estimation, and facilitating a more stable multi-point infrared emission effect under miniaturized packaging conditions.

[0131] In some possible implementations, when each microlens is located on the light-transmitting exit surface, each microlens is integrally formed with the light-transmitting exit surface.

[0132] Among them, the integrally formed microlens refers to an optical refractive structure that is directly formed on the surface of the light-transmitting surface and forms a continuous whole with the light-transmitting surface. Its function is to shape and collimate the infrared diverging beam emitted by the corresponding auxiliary light source or the main light source while ensuring the compactness of the package, so that the emitted beam forms a light spot distribution that meets the requirements of eye tracking on the predetermined propagation path.

[0133] Since the microlenses are not independently mounted on the emission surface, but are formed synchronously with the light-transmitting emission surface during the manufacturing stage, positional deviations during assembly processes can be reduced, reflection losses caused by interface gaps can be decreased, and the consistency of the light emission direction of each light source and the reliability of the packaging can be improved. In one possible embodiment, the light-transmitting emission surface can be made of transparent resin, glass, polycarbonate, epoxy material, or siloxane material, while the microlenses can be directly formed on the outer or inner surface layer of the light-transmitting emission surface through injection molding, hot stamping, molding replication, wafer-level replication, or UV curing replication, forming a convex lens area aligned with the light emission port of each auxiliary light source; in other exemplary implementations, if the light-transmitting emission surface is made of hard transparent materials such as glass or sapphire, an integrated microlens surface can also be constructed through precision etching, laser micromachining, or surface polishing.

[0134] This one-piece molded microlens typically has an aperture and radius of curvature that are adapted to the light-emitting port of the corresponding light source. Its characteristic height can range from several micrometers to hundreds of micrometers. The light-transmitting aperture of the lens can be matched and set according to the chip's light-emitting area and the target divergence angle to keep the overall thickness of the light-transmitting surface, mechanical strength and optical focal length parameters coordinated.

[0135] Structurally, after each microlens is integrally formed with the light-transmitting exit surface, the optical axis of the microlens is usually aligned with the light-emitting center line of the corresponding auxiliary or main light source. The convex surface of the lens faces directly towards the predetermined emission direction outside the package, thereby refracting and shaping the infrared light emitted by the internal light source and guiding it through the light-transmitting exit surface for outward output. Since there are no additional adhesive layers, bonding interfaces, or assembly gaps between the microlens and the exit surface, light can reduce refraction discontinuities and stray reflections when passing through this interface, making it particularly suitable for the requirements of thinness, miniaturization, and low power consumption in head-mounted eye-tracking devices.

[0136] For multiple microlenses corresponding to multiple auxiliary light sources, the integrated molding structure also facilitates the batch replication of multiple lens units on the same light-transmitting exit surface. By controlling the height difference, curvature difference, and center distance between each lens through unified mold parameters, the light spots formed by each infrared light source on the preset reflective surface have better stability and distinguishability. For the microlenses corresponding to the main light source, this integrated molding method can also ensure the shaping accuracy of the main beam and, together with the auxiliary light source microlenses, form an integrated optical system with the same packaged light-emitting interface.

[0137] In terms of size and proportion, the characteristic dimensions of a one-piece molded microlens are typically set based on the light-emitting area of ​​the light source chip, the size of the light-emitting window of the package, and the target divergence angle range. The radius of curvature of the lens can range from tens of micrometers to several millimeters. The lens height is preferably controlled within a range sufficient to form effective refraction without significantly increasing the package thickness. The spacing between the lens arrays can be matched with the center distance of each auxiliary or main light source to avoid mutual interference of lens apertures or beam crosstalk. If a thin cover plate structure is used for the light-transmitting exit surface, the protrusion height of the one-piece molded microlens should generally be coordinated with the thickness of the cover plate to ensure that the lens still has sufficient mechanical strength and anti-warping ability after molding. If a thicker transparent package layer is used, a smoother beam shaping effect can be achieved through a larger radius of curvature. Based on the above analysis, it can be seen that the one-piece molded structure can achieve high-precision integration of the microlens and the exit surface without adding independent assembly parts, and provides stable optical boundary conditions for subsequent spot control.

[0138] When the system starts, each auxiliary or main light source emits light inside the package under the drive of its corresponding electrode pin. The infrared beam first enters the corresponding integrated microlens area. The microlens refracts and shapes the diverging beam before outputting it to the outside of the package through the light-transmitting exit surface and propagating towards the preset reflective surface. Because the microlens and the light-transmitting exit surface are a single continuous structure, the reflection loss and assembly error at the interface are further suppressed, resulting in a more stable exit angle, more consistent spot shape, and easier separation and recognition of the reflected spots formed by multiple auxiliary light sources. For eye-tracking applications, this integrated molding method can improve the utilization efficiency of infrared energy to the effective area of ​​the eye, reduce the amount of light ineffectively scattered to the eyelids, eyelashes, or surrounding orbital areas, and reduce stray light interference in the images acquired by the camera. This makes the extraction of corneal reflection information and pupil position more stable and helps reduce the complexity of subsequent image processing and spot matching.

[0139] It should be understood that the above examples are for illustrative purposes only and are not limiting. Without departing from the overall technical concept of this application, the specific materials, manufacturing methods, surface parameters, and integration forms with the package of the integrally molded microlens can be adjusted according to the actual optical indicators and packaging process.

[0140] Figure 7 A schematic diagram of the structure of the smart glasses provided in this application is shown below. Figure 7 As shown, this application also provides smart glasses, which include a camera and the multi-point light source device provided above;

[0141] The multi-point light source device is used to emit infrared light into the user's eyeball and form multiple light spots on the surface of the user's cornea;

[0142] The camera is used to capture images of the user's eyes after a multi-point light source device emits infrared light towards the user's eyeballs.

[0143] By integrating a multi-point light source device with a camera into smart glasses, the multi-point light source device can directionally emit infrared light towards the eyeball in a limited space close to the eye, forming multiple identifiable light spots on the corneal surface. This allows the eye image to simultaneously include pupil information and multiple corneal reflection features. The camera captures images of the user's eyes after infrared light illumination, thereby obtaining raw image data for eye tracking. Combined with the control of the light emission direction and divergence range by the multi-point light source device, the brightness, spacing, and boundaries of the multiple light spots are more stable, thereby reducing stray light interference on the image, improving the image signal-to-noise ratio and the accuracy of light spot localization. Therefore, it is beneficial to improve the accuracy of gaze estimation, computational robustness, and the miniaturization and integration adaptability of smart glasses.

[0144] Figure 8 A flowchart illustrating the eye-tracking method provided in this application is shown below. Figure 8 As shown, the method is applied to the aforementioned smart glasses, and the method includes:

[0145] S801: If the multi-point light source device emits infrared light towards the user's eyeball, forming multiple light spots on the corneal surface of the user's eye, a first image of the user's eye is acquired.

[0146] Among them, the eye-tracking method is applied to smart glasses. The main execution body is a control processing unit set in the frame or temple of the glasses. The control processing unit is electrically connected to a multi-point light source device and an eye imaging device, respectively.

[0147] The multi-point light source device serves as the light source component for implementing the eye-tracking method. It emits infrared light to the user's eyeball and forms multiple light spots on the corneal surface for line-of-sight calculation. The infrared light is the invisible light illumination used in the eye-tracking process. The first image is an image of the user's eye captured by the eye imaging device, which simultaneously contains information on multiple light spots in the pupil area and on the corneal surface.

[0148] In practice, the multi-point light source device is installed in the inner area of ​​the frame near the user's eye, with its light emission direction facing the front surface of the eyeball. The eye imaging device is set in a position that can cover the eye imaging area and captures images when the multi-point light source device emits light.

[0149] In some implementations, the infrared light emitted by the multi-point light source device can be configured to form multiple spatially distinct reflective spots on the user's corneal surface to facilitate subsequent identification and gaze calculation. Based on the aforementioned optical illumination relationship, the infrared light illuminating the anterior surface of the cornea undergoes specular reflection, thereby forming multiple reflective bright spots with higher brightness than the background in the first image. When acquiring the first image, the eye imaging device can set corresponding acquisition parameters according to imaging needs to maintain a recognizable grayscale level between the pupil area and the multiple light spots in the first image.

[0150] In the specific processing flow, the control processing unit first outputs a drive signal to the multi-point light source device, causing the multi-point light source device to emit infrared light towards the user's eyeball. Subsequently, after the infrared light illuminates the eyeball and forms multiple light spots on the corneal surface, the control processing unit sends a acquisition trigger signal to the eye imaging device, and the eye imaging device outputs the corresponding first image to the image buffer area.

[0151] Based on the above analysis, it can be seen that by combining the light output of the multi-point light source device with imaging acquisition, the first image can stably contain pupil information and multiple corneal reflection information for line of sight estimation, thereby providing input data for subsequent position extraction.

[0152] S802: Identify the pupil and each light spot from the first image, and extract the first position information of the pupil and the second position information of each light spot.

[0153] Among them, the pupil is one of the eye targets identified in the first image, used to provide the reference position for line of sight calculation; the light spot is a bright spot formed by infrared light on the surface of the user's cornea, used to provide auxiliary geometric information for line of sight direction calculation; the first position information represents the position data of the pupil in the first image; the second position information represents the position data of each light spot in the first image.

[0154] In practice, the control processing unit reads the first image from the image buffer, performs preprocessing, target segmentation and position calculation on the image, and outputs the pupil center coordinates and the center coordinates of each light spot.

[0155] In one possible implementation, for pupil recognition, grayscale normalization and noise suppression processing are first performed on the first image. Noise suppression can be achieved by median filtering or Gaussian filtering to reduce the impact of eyelash edges, local bright reflections, and image sensor noise on subsequent segmentation.

[0156] Next, edge detection is performed in the eye area to extract candidate pupil boundaries. Then, morphological operations such as closing and opening operations are combined to remove scattered false edges. Finally, pupil contours are selected based on area thresholds, roundness thresholds, or ellipse fitting residuals.

[0157] Since the pupil typically appears as a relatively dark area in infrared images, the control processing unit can further perform ellipse fitting within the candidate region to obtain pupil boundary parameters. Based on the coordinates of the center of the fitted ellipse or the centroid coordinates of the target region, the first position information of the pupil can be extracted. This first position information can be represented using two-dimensional image coordinates, for example, using the horizontal and vertical coordinates in a pixel coordinate system to characterize the pupil center position.

[0158] For each light spot recognition, the control processing unit searches for locally bright areas in the first image whose brightness is significantly higher than the background. Specifically, the bright areas can be separated from the background first through threshold segmentation. The threshold can be a fixed threshold or adaptively determined based on the current image's average brightness, local contrast, or historical frame brightness statistics.

[0159] After obtaining the binary highlighted region, the area, peak brightness, aspect ratio and boundary sharpness of each connected region are calculated. False targets that do not meet the preset shape constraints are eliminated, and multiple connected regions are retained as candidate regions for light spots.

[0160] For each identified light spot, the control processing unit extracts its second location information based on the brightness distribution within the spot area. When the spot energy distribution approximates a Gaussian distribution, a gray-level weighted centroid method can be used, employing pixel gray values ​​as weights to calculate the center coordinates. When the spot edge exhibits slight distortion due to local reflection, two-dimensional Gaussian fitting or ellipse fitting can be performed within the candidate region to obtain a more stable center position. The second location information can also be represented using two-dimensional image coordinates and stored separately according to the spot number.

[0161] When there are many light spots or the spatial spacing is small, the control processing unit can determine the region affiliation of the light spots based on the preset number of light spots or relative position distribution, and then extract the second position information of each light spot.

[0162] If the first image contains both a pupil and multiple light spots, the control processing unit can first locate the pupil, and then search for light spots within a preset corneal reflection area around the pupil to reduce false bright spots in the background. For potential issues such as wearing misalignment, eyelid occlusion, or enhanced local reflection during real-time acquisition, the control processing unit dynamically adjusts the threshold, search window, and fitting parameters based on changes in light spot area, peak brightness, and relative position in consecutive frames, ensuring that the extracted first and second position information remain stable across consecutive frames.

[0163] Based on the above analysis, this step involves locating the pupil target and multiple reflective light spot targets in the first image, and outputting the first position information and each second position information in the form of center coordinates, thus converting the original image into geometric input data required for subsequent gaze distance calculation. The pupil center serves as the representation point of eye orientation, and the centers of the multiple light spots serve as corneal reflection constraint points; together, they constitute the basic data structure for multi-point PCCR (Pupil Center Corneal Reflection) calculation. By constraining and recognizing the shape, brightness, and spatial distribution of the light spots, the correspondence between the multiple light spots in the image becomes clearer, thereby reducing the transmission of position extraction errors to the gaze distance calculation process.

[0164] S803: Determine the user's eye gaze direction based on the first position information and each of the second position information.

[0165] The gaze direction is determined by the user's gaze direction based on the pupil position and the light spot position. In practice, the control processing unit reads the first position information and each of the second position information within the same sampling period, performs gaze calculation based on a preset eye geometry model, camera calibration parameters, and light source layout parameters, and outputs the user's current gaze direction data. This gaze direction can be represented as a direction vector relative to the eye-tracking camera coordinate system, the smart glasses' own coordinate system, or the display interface coordinate system, or it can be further converted into the coordinates of the gaze point.

[0166] In one possible implementation, the control processing unit employs a multi-point PCCR algorithm to determine the gaze direction. The multi-point PCCR algorithm is a gaze estimation method based on the relationship between the pupil center and the corneal reflection point, and it uses multiple spot centers as joint constraints.

[0167] In another exemplary implementation, the control processing unit pre-establishes intrinsic parameters of the eye-tracking camera, extrinsic parameters of the camera's installation relative to the eyeball, spatial layout parameters of the multi-point light source device, and individual user calibration parameters.

[0168] Before the system is used, the smart glasses guide the user to gaze at several known calibration points, record the first position information and the second position information at each calibration point, and fit an individualized mapping relationship from image features to gaze direction.

[0169] During the formal operation phase, the control processing unit directly substitutes the real-time extracted position data into the calibrated model to obtain the gaze direction. The output gaze direction can then be further transmitted to the interactive control module for menu selection, cursor movement, or command triggering.

[0170] In one possible implementation, the interactive control module only receives gaze direction sequences that have already been filtered and smoothed. The control processing unit performs jitter removal on the continuous frame gaze data using Kalman filtering, exponential smoothing, or sliding window averaging, and then outputs gaze results usable by the terminal.

[0171] Based on the above analysis, this step uses the first position information of the pupil and the second position information of multiple light spots as geometric constraint inputs. Compared with using only a single reflection point, multiple light spots provide more comprehensive corneal reflection distribution information, making the gaze direction calculation more tolerant to local noise, single-point reflection anomalies, and minor wear-related offsets. The higher the stability of the position data entering this step, the more consistent the input obtained by the gaze direction calculation model, thus resulting in better accuracy and continuity of the final output gaze direction.

[0172] Based on the above analysis, this application provides an eye-tracking method, comprising: emitting infrared light from a multi-point light source device towards a user's eyeball, forming multiple light spots on the corneal surface of the user's eye, and acquiring a first image of the user's eye; identifying the pupil and each light spot from the first image, and extracting first position information of the pupil and second position information of each light spot; and determining the user's gaze direction based on the first position information and each second position information. In this application, by linking multi-point infrared illumination, eye image acquisition, pupil and multi-light spot position extraction, and multi-point PCCR algorithm gaze calculation along the same processing link, effective utilization of eye reflection information within the limited encapsulation space of smart glasses is achieved. The infrared light emitted by the multi-point light source device forms multiple light spots with high spatial separation on the corneal surface. Combined with pupil center extraction and light spot center extraction in the image recognition process, stable geometric constraint data can be provided to the gaze calculation model, thereby determining the current gaze direction of the user's eye.

[0173] It should be understood that the above examples are merely illustrative and not limiting. Any implementation that can determine the user's eye gaze direction based on the first location information and each of the second location information can be applied to this application.

[0174] In one possible implementation, if the multi-point light source device includes multiple auxiliary light sources and a main light source, the above method further includes: lighting up the main light source within a preset first time period to acquire a second image of the user's eye; lighting up all auxiliary light sources within a preset second time period to form multiple light spots on the corneal surface of the user's eye to acquire a third image of the user's eye; and determining the direction of the user's gaze based on the second and third images.

[0175] For example, the control processing unit can use time-division multiplexing to control the lighting sequence of the main light source and the auxiliary light source, so that the main light source and the auxiliary light source work separately in different time periods.

[0176] When the main light source is turned on, the eye-tracking camera is aimed at the user's eye to acquire an image, obtaining a second image used to identify the pupil position. Subsequently, during a second time period, the main light source is turned off and all auxiliary light sources are turned on, so that the beams emitted by each auxiliary light source are incident on the corneal surface of the user's eye and form multiple distinguishable light spots. The eye-tracking camera simultaneously acquires a third image.

[0177] As an example, the main light source can be an infrared light-emitting diode, and the auxiliary light source can be multiple infrared light-emitting chips. The two can be housed in the same package and controlled by independent electrode pins. In practical applications, other models of this component can also be selected, and this application does not limit them.

[0178] When determining the direction of gaze based on the second and third images, the pupil center position can be extracted from the second image first, and the spatial position information of multiple light spots can be extracted from the third image. The gaze direction vector can then be calculated by combining the preset geometric mapping relationship.

[0179] Since the second image provides a pupil reference and the third image provides corneal reflection constraints, the user's gaze direction can be determined jointly by these two types of images. This method separates the main light source imaging from the auxiliary light source imaging, facilitating pupil localization and spot localization within the same device and outputting stable gaze direction results.

[0180] By adopting the above method, the main light source and the auxiliary light source work in a time-sharing manner, and the imaging information of the second and third images can correspond to the pupil features and light spot features respectively, thereby making the calculation basis of the gaze direction clearer, the image recognition process easier to implement, and suitable for integration in space-constrained head-mounted devices.

[0181] In one possible implementation, determining the user's eye gaze direction based on the second and third images includes: identifying the pupil from the second image and extracting the third position information of the pupil; identifying each light spot from the third image and extracting the fourth position information of each light spot; and determining the user's eye gaze direction based on the third position information and each of the fourth position information.

[0182] For example, the second image can be first converted to grayscale, and the pupil contour can be extracted by edge detection combined with morphological closing operation. Then, the contour area can be filled and connected component filtered to obtain the pupil area. Subsequently, the centroid of the region can be calculated as the third position information of the pupil.

[0183] For the third image, the light spots can be separated from the background based on brightness threshold segmentation, and connected component analysis can be performed on the separated bright regions to identify each light spot. When the light spots have an approximately Gaussian bright spot distribution, gray-level weighted centroid calculation can be performed on each bright region to obtain the fourth position information of each light spot. If the light spots have edge distortion or local saturation, dynamic threshold adjustment and contour smoothing can be combined to correct their position extraction results. In practical applications, image acquisition can be completed by an eye-tracking camera, and the position data of the pupil and light spots are output in the form of image coordinates. This application does not limit this.

[0184] After obtaining the third position information of the pupil and the fourth position information of each light spot, the geometric relationship between the center of the pupil and the center of each light spot can be established, and the mapping relationship between the direction of eye rotation and the point of gaze can be calculated accordingly.

[0185] Specifically, the offset vector of the pupil center relative to the centers of each light spot can be input into the multi-point PCCR calculation model. The single-reflection error is constrained by the positional distribution of multiple light spots, thereby outputting the user's gaze direction. When multiple light spots exist, their positions participate in the solution, making the calculation results more stable. The output of the gaze direction can be represented as two-dimensional gaze coordinates, a three-dimensional gaze direction vector, or a landing point region; this application does not limit this representation.

[0186] By adopting the above method, the pupil position and multiple light spot positions can be calculated in the same coordinate frame. The process of determining the line of sight has clear image basis and geometric constraints, which can improve the stability and consistency of the line of sight solution and reduce the fluctuation of direction judgment caused by the deviation of a single light spot.

[0187] This application is applied to eye-tracking in smart glasses. A multi-point light source emits infrared light towards the user's eyeball. The infrared light illuminates the corneal surface, forming multiple light spots. Simultaneously, an eye image acquisition component acquires a first image of the user's eye. Image processing is performed on the first image to identify the pupil region and each corneal light spot. First position information of the pupil and second position information of each light spot are extracted. The first position information represents the pupil's coordinates within the image, and each second position information represents the corresponding light spot's coordinates within the image. Subsequently, the user's gaze direction is calculated based on the spatial relationship between the first and second position information.

[0188] In one possible implementation, the multi-point light source device includes multiple auxiliary light sources and a main light source. The smart glasses acquire eye images according to a time-division lighting method. During a preset first time period, the main light source is illuminated to acquire a second image of the user's eye. During a preset second time period, all auxiliary light sources are illuminated, forming multiple light spots on the corneal surface of the user's eye, and a third image of the user's eye is acquired. The second image is identified to obtain the pupil and extract its third position information. The third image is identified to obtain each light spot and extract its fourth position information. The user's gaze direction is then determined based on the third and fourth position information.

[0189] Figure 9 This is a schematic diagram of the eye-tracking device provided in this application, as shown below. Figure 9 As shown, the eye-tracking device 90 provided in this embodiment is applied to the aforementioned smart glasses. The device includes:

[0190] The acquisition module 901 is used to: acquire a first image of the user's eye if the multi-point light source device emits infrared light to the user's eyeball and forms multiple light spots on the surface of the user's cornea;

[0191] The extraction module 902 is used to: identify the pupil and each light spot from the first image, and extract the first position information of the pupil and the second position information of each light spot;

[0192] The determination module 903 is used to: determine the direction of the user's eye gaze based on the first position information and each of the second position information.

[0193] In one possible implementation, if the multi-point light source device includes multiple auxiliary light sources and a main light source, the acquisition module 901 is further configured to:

[0194] Within a preset first time period, the main light source is turned on to obtain a second image of the user's eyes;

[0195] During the preset second time period, all auxiliary light sources are lit to form multiple light spots on the corneal surface of the user's eye, thereby acquiring a third image of the user's eye.

[0196] Based on the second and third images, determine the direction of the user's gaze.

[0197] In one possible implementation, the extraction module 902 is also used for:

[0198] Identify the pupil from the second image and extract the third position information of the pupil;

[0199] Identify each light spot from the third image and extract the fourth position information of each light spot;

[0200] The determination module 903 is also used to: determine the direction of the user's eye gaze based on the third position information and each of the fourth position information.

[0201] The eye-tracking device provided in this embodiment can execute the method provided in the above-described method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0202] Figure 10 This is a schematic diagram of the eye-tracking device provided in this application. Figure 10 As shown, the eye-tracking device 100 provided in this embodiment includes at least one processor 1001 and a memory 1002. Optionally, the eye-tracking device 100 further includes a communication component 1003. The processor 1001, memory 1002, and communication component 1003 are connected via a bus.

[0203] In the specific implementation process, at least one processor 1001 executes computer execution instructions stored in memory 1002, causing at least one processor 1001 to execute the above-described eye-tracking method.

[0204] The specific implementation process of processor 1001 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0205] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0206] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0207] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0208] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described eye-tracking method.

[0209] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described eye-tracking method.

[0210] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0211] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0212] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0213] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0214] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0215] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0216] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0217] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A multi-spot light source device, characterized by, include: The package includes multiple auxiliary light sources, multiple sets of electrode pins, and multiple microlenses. The package includes a package substrate and a light-transmitting surface, and the package is used to encapsulate multiple auxiliary light sources; The auxiliary light source corresponds one-to-one with the electrode pin. Inside the package, each auxiliary light source is set at a first preset position on the package substrate through a corresponding electrode pin. The electrode pin is used to input a driving current to the corresponding auxiliary light source. The auxiliary light source is used to emit infrared light through the light-transmitting surface. Each microlens corresponds to one of the auxiliary light sources. Each microlens is disposed on the light-emitting side of the corresponding auxiliary light source. Each microlens is located inside the package or on the light-transmitting surface. Each microlens is used to refract the diverging light beam emitted by the corresponding auxiliary light source into a preset divergence angle range to form a light spot on a preset reflective surface.

2. The apparatus of claim 1, wherein, Each of the microlenses is disposed on the light-emitting side of the corresponding auxiliary light source, including: Each of the microlenses is disposed on the light-emitting side of the corresponding auxiliary light source, and the optical axis of each microlens is offset by a preset distance relative to the geometric center of the corresponding auxiliary light source, so that the light spots formed by the infrared light emitted by each auxiliary light source on the preset reflective surface are separated from each other.

3. The apparatus of claim 1, wherein, The device further includes a main light source and electrode pins corresponding to the main light source; wherein... Inside the package, the main light source is positioned at a second preset location on the package substrate via an electrode pin corresponding to the main light source.

4. The apparatus according to claim 3, characterized in that, The device also includes a microlens corresponding to the main light source, which is disposed on the light-emitting side of the main light source. The microlens corresponding to the main light source is located inside the package or on the light-transmitting surface.

5. The apparatus according to claim 4, characterized in that, The microlens corresponding to the auxiliary light source is configured to refract the diverging beam emitted by the corresponding auxiliary light source into a preset first divergence angle range. The microlens corresponding to the main light source is configured to refract the diverging beam emitted by the main light source into a preset second divergence angle range; wherein the second divergence angle range is greater than the first divergence angle range.

6. The apparatus according to any one of claims 1-5, characterized in that, The microlens is an aspherical or freeform surface.

7. The apparatus according to any one of claims 1-5, characterized in that, When each of the microlenses is located on the light-transmitting surface, each of the microlenses and the light-transmitting surface are integrally formed.

8. A type of smart glasses, characterized in that, The smart glasses include a camera and a multi-point light source device as described in any one of claims 1-7; The multi-point light source device is used to emit infrared light into the user's eyeball and form multiple light spots on the corneal surface of the user's eye; The camera is used to capture images of the user's eyes after the multi-point light source device emits infrared light towards the user's eyeballs.

9. An eye-tracking method, characterized in that, Applied to the smart glasses as described in claim 8, the method includes: If a multi-point light source device emits infrared light into the user's eyeball, multiple light spots are formed on the surface of the user's cornea, thus acquiring the first image of the user's eye; Identify the pupil and each of the light spots from the first image, and extract the first position information of the pupil and the second position information of each of the light spots; Based on the first location information and each of the second location information, the user's eye gaze direction is determined.

10. The method according to claim 9, characterized in that, If the multi-point light source device includes multiple auxiliary light sources and a main light source, the method further includes: Within a preset first time period, the main light source is turned on to acquire a second image of the user's eyes; During the preset second time period, all auxiliary light sources are lit to form multiple light spots on the corneal surface of the user's eye, thereby acquiring a third image of the user's eye. Based on the second and third images, the user's eye gaze direction is determined.

11. The method according to claim 10, characterized in that, Based on the second image and the third image, the direction of the user's gaze is determined, including: Identify the pupil from the second image and extract the third position information of the pupil; Identify each light spot from the third image and extract the fourth position information of each light spot; Based on the third location information and each of the fourth location information, the user's eye gaze direction is determined.

Images