Multi-light source and light spot matching method of head-mounted sight tracking system

Abstract

The invention provides a multi-light source and light spot matching method of a head-mounted sight tracking system. The method is applied to a VR head-mounted display device comprising a camera and N light sources, and comprises the steps of obtaining a light spot image which is shot by the camera and comprises M light spots, constructing a to-be-matched convex hull graph based on the M light spots, obtaining a reference convex hull graph set which is obtained by selecting the M light sources from the N light sources and constructing the corresponding light spots, and in the set, determining a reference convex hull graph with the highest similarity with the convex hull graph to be matched as a target convex hull graph, and further determining the corresponding relationship between the light spot and the light source. Through the method and the device, matching of the light spots and the light sources can be converted into similarity matching of convex hull patterns, the corresponding relation can still be determined under the condition that the number of the light spots is smaller than that of the light sources, dependence on additional coding or complex driving is reduced, matching stability and applicability are improved, and reliable input is provided for follow-up eye movement tracking.

Description

[0001] This application claims priority to Chinese invention patent application No. 202610135912.2, filed with the State Intellectual Property Office on January 30, 2026, entitled "VR Head-Mounted Display Device and Method and Apparatus for Matching Light Source and Light Spot". Technical Field

[0002] This invention relates to the field of data processing technology, and in particular to a method for matching multiple light sources and light spots in a head-mounted eye-tracking system. Background Technology

[0003] With the rapid development of virtual reality (VR) head-mounted display devices, eye-tracking technology based on eye movements has extremely high application value in fields such as interactive operation, foveated rendering, fatigue detection, and user behavior analysis. To achieve high-precision eye tracking, the industry currently generally adopts near-eye infrared imaging, which involves arranging several infrared LED light sources around an infrared camera. By capturing the bright spot formed by the reflection from the cornea on the surface of the eyeball and the pupil edge information, the three-dimensional pose of the eyeball and the direction of gaze are determined by combining the imaging geometry.

[0004] In typical VR headsets, infrared cameras are mounted near the lenses, usually surrounded by a series of infrared light sources arranged regularly or in a ring, such as 4, 6, 8, or more. When the light from these infrared sources shines on the user's eye, it undergoes specular reflection on the outer surface of the cornea. The reflected light then enters the camera, forming a light spot. Theoretically, each LED light source should form a corresponding light spot. The correspondence between the light spot and the light source is crucial for determining the corneal center and is a key component in the entire eye-tracking process.

[0005] The inventors discovered that in existing eye-tracking technologies, a common approach to establishing the correspondence between infrared light sources and corneal reflective spots is light source encoding. This approach applies differentiated emission patterns to multiple light sources, creating distinguishable spot characteristics in the image, which are then matched during subsequent image processing. Existing light source encoding methods include, but are not limited to: brightness difference encoding, flicker frequency encoding, timing pulse encoding, duty cycle encoding, and light source structure shape encoding. These methods can be used individually or combined arbitrarily depending on hardware conditions. For example, some systems distinguish light sources solely through brightness differences, others identify light source numbers through flicker or timing patterns, and still others combine brightness differences with timing pulses to improve recognition reliability. Because light source encoding is directly embedded in the grayscale value, shape, or time sequence of the spot, it eliminates the need for complex geometric model solutions, enabling relatively direct spot matching assuming the light source is functioning correctly and the spot is intact.

[0006] However, the inventors further research and find that, firstly, the technical solutions of this type of light source encoding rely on the correct operation of the encoding hardware. Whether using brightness difference, stroboscopic frequency, timing pulse, or a combination of multiple encodings, additional light source driving circuits and synchronization control logic are required, resulting in a complex system structure, increased power consumption, and limited volume and heat dissipation design of the head-mounted display; secondly, the encoding features are jointly affected by factors such as eyeball rotation, reflection angle change, camera exposure, spot occlusion, lens reflection, and environmental infrared noise. Whether using single encoding or combined encoding, the stability of the spot features cannot be fully guaranteed; thirdly, once a certain encoding signal is damaged, it may lead to the overall failure of the spot matching link. In addition, most encoding schemes need to count the bright and dark or intensity changes of spots in multiple frames of images. Therefore, the stability significantly decreases during rapid eye movement, low exposure time, or when the number of spots decreases, making it difficult to meet the requirements of real-time and robustness in VR scenarios.

[0007] Therefore, no matter which encoding method is used, or the encoding combination strategy is used, it is difficult to maintain reliable spot matching performance in scenarios with unfixed spot numbers, strong noise interference, and fast eye movement speed. How to better achieve the matching of spots and light sources has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] The object of the present invention is to provide a method for matching multiple light sources and spots in a head-mounted eye tracking system to solve the above technical problems in the prior art.

[0009] On the one hand, to achieve the above object, the present invention provides a method for matching multiple light sources and spots in a head-mounted eye tracking system.

[0010] The method for matching multiple light sources and spots in the head-mounted eye tracking system is applied to a VR head-mounted display device, which includes a camera and N light sources. After the light sources irradiate the user's eyeball, the reflected light formed on the corneal surface enters the camera to form spots. The matching method includes: obtaining a spot image captured by the camera, where the spot image includes M spots, and both M and N are positive integers, 1 < M ≤ N; constructing a convex hull graph based on the M spots to obtain a to-be-matched convex hull graph; obtaining a set of reference convex hull graphs corresponding to the M spots, where M light sources are selected from the N light sources, and a convex hull graph is constructed using the spots corresponding to the M light sources to obtain a reference convex hull graph; determining the reference convex hull graph with the highest similarity to the to-be-matched convex hull graph in the set of reference convex hull graphs to obtain a target convex hull graph; and determining the correspondence between the spots in the spot image and the light sources according to the target convex hull graph.

[0011] Furthermore, after the step of acquiring the light spot image and before the step of obtaining the convex hull pattern to be matched, the matching method also includes: determining the pupil center of the user's eyeball and the optical center of the camera; connecting the pupil center and the optical center to form a central optical axis; and performing coordinate transformation on the coordinates of the light spot in the light spot image so that the imaging plane where the coordinates of the transformed light spot are located is perpendicular to the central optical axis.

[0012] Further, the step of determining the benchmark convex hull graph with the highest similarity to the convex hull graph to be matched in the benchmark convex hull graph set, and obtaining the target convex hull graph, includes: calculating the feature vector of each benchmark convex hull graph in the benchmark convex hull graph set and the feature vector of the convex hull graph to be matched; calculating the similarity between the benchmark convex hull graph and the convex hull graph to be matched based on the feature vectors of the benchmark convex hull graph and the convex hull graph to be matched; and determining the benchmark convex hull graph with the highest similarity in the benchmark convex hull graph set as the target convex hull graph.

[0013] Furthermore, when M>2, and all M light spots are located on the convex hull boundary and serve as convex hull vertices, the convex hull graphic is a convex polygon formed by the M light spots as vertices. The steps for calculating the feature vector of the convex hull graphic include: sorting each vertex of the convex hull graphic according to a preset sorting rule to obtain a vertex sequence. Calculate the interior angle vectors (a1, a2, a3, ..., a4) of the convex hull according to the order of the vertices in the vertex sequence. M ) and the side length vectors of the convex hull (d1,d2,d3,......,d M ); Calculate the eigenvectors of the interior angle ratios (a1 / a2, a2 / a3, ..., a) based on the interior angle vectors of the convex hull figure. M-1 / a M ); Calculate the eigenvector of the side length ratio (d1 / d2, d2 / d3, ..., d) based on the side length vector of the convex hull figure. M-1 / d M ), where the eigenvectors of the interior angle ratio and the eigenvectors of the side length ratio are the eigenvectors of the convex hull.

[0014] Further, the steps for calculating the similarity between the reference convex hull and the convex hull to be matched include: performing a dot product between the feature vector of the interior angle ratio of the convex hull to be matched and the feature vector of the interior angle ratio of the reference convex hull to obtain a first dot product value; performing a dot product between the feature vector of the side length ratio of the convex hull to be matched and the feature vector of the side length ratio of the reference convex hull to obtain a second dot product value; and calculating the similarity based on the first dot product value and the second dot product value, wherein the larger the first dot product value and the second dot product value, the greater the similarity.

[0015] Furthermore, when M=2, the convex hull pattern is the line segment connecting two light spots; the reference convex hull pattern is constructed from the light spots corresponding to two adjacent light sources; the set of reference convex hull patterns includes a first set, a second set, and a third set. In the horizontal direction, the light source corresponding to the first set is located on the first side of the camera, the light source corresponding to the second set is located on the second side of the camera, and the light source corresponding to the third set is located on both sides of the camera; if the two light spots are located on the same side of the pupil center of the user's eyeball in the horizontal direction, the slope of the line segment is the feature vector of the convex hull pattern. In the first set or the second set, the reference convex hull pattern corresponding to the maximum similarity is determined as the target convex hull pattern; if the two light spots are located on both sides of the pupil center in the horizontal direction, the positional relationship between the midpoint of the line segment and the pupil center in the vertical direction is the feature vector of the convex hull pattern. In the third set, the reference convex hull pattern corresponding to the maximum similarity is determined as the target convex hull pattern.

[0016] Further, the step of obtaining the set of reference convex hull graphics corresponding to M light spots includes: selecting M light sources from N light sources; obtaining reference light spot images of different types corresponding to the M light sources, wherein the reference light spot image includes M light spots corresponding to the M light sources, and the reference light spot image is divided into different regions along the horizontal direction, and the pupil center of the user's eyeball is located in different regions in the different types of reference light spot images; constructing a convex hull graphic using the M light spots in each type of reference light spot image to obtain a type of reference convex hull graphic; and forming a set of reference convex hull graphics by selecting all combinations of M light sources from N light sources.

[0017] Furthermore, the reference spot image is divided into three different regions along the horizontal direction, resulting in three types of reference spot images corresponding to M light sources. The reference convex hull pattern set includes three subsets of reference convex hull patterns corresponding to the three types of reference spot images. The step of determining the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched in the reference convex hull pattern set includes: dividing the spot image into three regions along the horizontal direction; selecting a subset of reference convex hull patterns from the reference convex hull pattern set based on the region where the pupil center of the user's eyeball is located in the spot image; and determining the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched from the subset of reference convex hull patterns.

[0018] Further, the N light sources include a first light source group and a second light source group located on both sides of the camera. The steps of determining the reference convex hull graph with the highest similarity to the to-be-matched convex hull graph in the set of reference convex hull graphs to obtain the target convex hull graph include: sorting the M light spots in ascending order of the X coordinate to obtain a light spot sequence, where N > 3 and M ≥ 3; calculating the differences between the X coordinates of two adjacent light spots in the light spot sequence in turn to obtain a difference sequence; when the maximum difference in the difference sequence is greater than twice of at least one of its adjacent differences, determining that the position between the two light spots corresponding to the maximum difference is the light spot dividing line, where the light spot dividing line divides the light spot sequence into a first light spot group and a second light spot group, the light sources corresponding to the first light spot group are located in the first light source group, and the light sources corresponding to the second light spot group are located in the second light source group; screening the reference convex hull graphs in the set of reference convex hull graphs whose corresponding light source grouping matches the light spot grouping divided by the light spot dividing line; determining the reference convex hull graph with the highest similarity to the to-be-matched convex hull graph among the screened reference convex hull graphs to obtain the target convex hull graph.

[0019] On the other hand, to achieve the above object, the present invention provides a matching device for multiple light sources and light spots of a head-mounted eye tracking system.

[0020] The matching device for multiple light sources and light spots of the head-mounted eye tracking system is applied to a VR head-mounted display device. The device includes a camera and N light sources. After the light sources irradiate the user's eyeball, the reflected light formed on the corneal surface enters the camera to form light spots. The matching device includes: a first acquisition module for acquiring a light spot image captured by the camera, where the light spot image includes M light spots, and both M and N are positive integers, 1 < M ≤ N; a construction module for constructing a convex hull graph based on the M light spots to obtain a to-be-matched convex hull graph; a second acquisition module for acquiring a set of reference convex hull graphs corresponding to the M light spots, where M light sources are selected from the N light sources, and convex hull graphs are constructed using the light spots corresponding to the M light sources to obtain reference convex hull graphs; a matching module for determining the reference convex hull graph with the highest similarity to the to-be-matched convex hull graph in the set of reference convex hull graphs to obtain the target convex hull graph; a determination module for determining the correspondence between the light spots in the light spot image and the light sources according to the target convex hull graph.

[0021] On the other hand, to achieve the above object, the present invention provides a VR head-mounted display device.

[0022] The VR head-mounted display device includes: a camera and N light sources. After the light sources irradiate the user's eyeball, the reflected light formed on the corneal surface enters the camera to form light spots; a controller for executing any one of the light source and light spot matching methods provided in this application.

[0023] On the other hand, to achieve the above objectives, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0024] On the other hand, to achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above method.

[0025] The present invention provides a method for matching multiple light sources and light spots in a head-mounted gaze tracking system. First, it acquires a light spot image containing M light spots captured by a camera. Then, it constructs a convex hull pattern to be matched based on the M light spots. Next, it acquires a set of reference convex hull patterns constructed from M light sources selected from N light sources, using the light spots corresponding to these M light sources. Then, it identifies the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched from the set of reference convex hull patterns as the target convex hull pattern. Finally, it determines the correspondence between light spots and light sources in the light spot image based on the target convex hull pattern. Through this invention, the spatial distribution of light spots is abstracted into a convex hull pattern, and the similarity between the convex hull pattern to be matched in the current frame and the set of reference convex hull patterns pre-constructed based on light source combinations is compared. This enables the selection of the most matching light source combination in multi-light source, multi-light spot scenes, and the determination of the correspondence between light spots and light sources, thereby achieving reliable determination of the correspondence between light sources and light spots. Without requiring light source encoding and multi-frame statistics, adapting to scenarios with varying light spot numbers and stably determining the correspondence between light spots and light sources based on geometric distribution can reduce system hardware complexity and power consumption / heat generation risks, and improve matching robustness and real-time performance under noise interference, exposure changes, light spot loss, and rapid eye movement conditions. Attached Figure Description

[0026] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart of the method for matching a light source and a light spot according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the light source arrangement of a VR head-mounted display device provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of light spot formation provided in an embodiment of the present invention; Figure 4 This is a flowchart of the method for matching a light source and a light spot according to Embodiment 2 of the present invention; Figure 5A schematic diagram of a light spot image provided in an embodiment of the present invention; Figure 6 This is a block diagram of the light source and light spot matching device provided in Embodiment 3 of the present invention; Figure 7 This is a hardware structure diagram of a computer device provided in Embodiment 5 of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0028] Example 1 Embodiment 1 of this invention provides a method for matching multiple light sources with light spots in a head-mounted eye-tracking system. This method eliminates the need for light source coding and correctly maps light spots detected in camera images to multiple light sources within the VR head-mounted display device during eye-tracking imaging, thus providing a reliable basis for subsequent eye-tracking parameter calculations. Specifically, Figure 1 The flowchart of the light source and light spot matching method provided in Embodiment 1 of the present invention is as follows: Figure 1 As shown, the matching method between the light source and the light spot provided in this embodiment includes the following steps S101 to S105.

[0029] Step S101: Acquire the light spot image captured by the camera.

[0030] In this embodiment, the VR head-mounted display device includes an eye-tracking camera and multiple light sources. The light sources can be infrared light-emitting devices, such as infrared LEDs, used to emit light towards the user's eyes. After the light shines on the user's eyes, it forms reflected light spots on the corneal surface. These reflected light spots enter the camera and are imaged as light spots, thus the light spot image captured by the camera includes several light spots. N represents the number of light sources set in the device, and M represents the number of light spots actually detected in the light spot image. Therefore, the light spot image includes M light spots. Since the light sources may be affected by factors such as eyelid obstruction, line-of-sight deflection, lens reflection, and local overexposure or underexposure, some light spots corresponding to certain light sources may not be captured by the camera. Therefore, M and N are both positive integers and satisfy 1 / N = 1 / 2. <M≤N。

[0031] Optionally, in one implementation, the spot image can be a grayscale image or a color image. After acquiring the spot image, the spot image can be processed by grayscale conversion, threshold segmentation, connected component filtering, etc., to locate the spot region. The centroid coordinates of each spot connected component in the image coordinate system are taken as the spot coordinates to obtain a set of coordinates of M spots for marking and calculation.

[0032] Step S102: Construct a convex hull pattern based on the M light spots to obtain the convex hull pattern to be matched.

[0033] In this embodiment, based on the M light spots obtained in step S101, a convex hull is constructed on the M light spots to obtain a convex hull pattern to be matched. This convex hull pattern can characterize the overall geometric shape of the light spots in the current frame light spot image in terms of spatial distribution.

[0034] Optionally, when constructing a convex hull pattern based on M light spots, geometric operations are performed on representative points of the M light spots to obtain the smallest convex set that can enclose these representative points. The geometric shape corresponding to this smallest convex set is then used as the convex hull pattern to be matched. When the number of light spots in the light spot image is different, the convex hull pattern to be matched can represent different geometric forms. For example, when M is 2, the convex hull pattern is a line segment formed by connecting two light spots; when M is 3 or greater than 3, the convex hull pattern presents a polygonal shape. By abstracting the light spot distribution into a convex hull pattern, the subsequent matching process becomes more robust to local noise and individual point position shifts.

[0035] Step S103: Obtain the set of reference convex hull graphics corresponding to the M light spots.

[0036] In this embodiment, a pre-established reference is used to determine the correspondence between light spots and light sources. Specifically, a set of reference convex hull patterns corresponding to M light spots is pre-defined. M light sources are selected from N light sources, and convex hull patterns are constructed using the light spots corresponding to these M light sources to obtain the reference convex hull patterns. Furthermore, based on the set of available light sources in the device (i.e., from N light sources), a group of M light sources is selected as candidate light source combinations. For each candidate light source combination, M light spots corresponding to that combination are obtained, and convex hull patterns are constructed based on these M light spots to form a reference convex hull pattern. By constructing convex hull patterns for different light source combinations, multiple reference convex hull patterns can be obtained, which together constitute a set of reference convex hull patterns.

[0037] Alternatively, in one implementation, the reference convex hull graphic set can be stored in the device as an offline template set, or it can be stored after being generated during device initialization or calibration; during actual matching, the set is directly read for comparison.

[0038] Step S104: Determine the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched from the reference convex hull pattern set, and obtain the target convex hull pattern.

[0039] In this embodiment, based on the convex hull pattern to be matched obtained in step S102, a similarity comparison is performed with the set of reference convex hull patterns obtained in step S103 to determine the reference convex hull pattern that is most similar to the convex hull pattern to be matched, and the reference convex hull pattern is used as the target convex hull pattern.

[0040] The similarity test measures the geometric similarity between the convex hull shape to be matched and each reference convex hull shape. Then, the reference convex hull shape with the highest similarity is selected from the set of reference convex hull shapes; this is the target convex hull shape. The light source combination corresponding to this target convex hull shape is most likely to correspond to the M light spots in the current frame's light spot image, thus providing a basis for establishing the correspondence between light spots and light sources.

[0041] Step S105: Determine the correspondence between the light spot and the light source in the light spot image based on the target convex hull shape.

[0042] In this embodiment, the target convex hull pattern originates from a reference convex hull pattern in the set of reference convex hull patterns. This reference convex hull pattern is constructed from the light spots corresponding to M light sources selected from N light sources. Therefore, after determining the target convex hull pattern, the combination of light sources corresponding to the target convex hull pattern can be used as the candidate corresponding light source set for the light spots in the current frame. Furthermore, based on the matching result between the target convex hull pattern and the convex hull pattern to be matched, a correspondence can be established between the M light spots in the light spot image and the M light sources in the candidate light source set, thereby completing the matching of light sources and light spots.

[0043] In the head-mounted gaze tracking system multi-source and light spot matching method provided in this embodiment, a light spot image containing M light spots captured by a camera is first acquired. Then, a convex hull pattern to be matched is constructed based on the M light spots. Next, a set of reference convex hull patterns is acquired, which is constructed by selecting M light sources from N light sources and using the light spots corresponding to these M light sources. Subsequently, the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched in the reference convex hull pattern set is determined as the target convex hull pattern. Finally, the correspondence between the light spots and light sources in the light spot image is determined based on the target convex hull pattern. By using the head-mounted gaze tracking system multi-source and light spot matching method provided in this embodiment, the spatial distribution of light spots is abstracted into a convex hull pattern, and the similarity between the convex hull pattern to be matched in the current frame and the set of reference convex hull patterns pre-constructed based on light source combinations is compared. This allows the selection of the most matching light source combination in multi-source and multi-light spot scenes, and the determination of the correspondence between light spots and light sources, thereby achieving reliable determination of the correspondence between light sources and light spots. Without requiring light source encoding and multi-frame statistics, adapting to scenarios with varying light spot numbers and stably determining the correspondence between light spots and light sources based on geometric distribution can reduce system hardware complexity and power consumption / heat generation risks, and improve matching robustness and real-time performance under noise interference, exposure changes, light spot loss, and rapid eye movement conditions.

[0044] Optionally, in one embodiment, after the step of acquiring the spot image and before the step of obtaining the convex hull pattern to be matched, the matching method further includes: determining the pupil center of the user's eyeball and the optical center of the camera; connecting the pupil center and the optical center to form a central optical axis; and performing coordinate transformation on the coordinates of the spot in the spot image so that the imaging plane where the coordinates of the transformed spot are located is perpendicular to the central optical axis.

[0045] Specifically, the pupil center represents the central position of the user's eyeball at the current moment, while the camera's optical center represents the projection center in the camera's imaging model. The pupil center can be the geometric center of the pupil contour or the center of a fitted circle, while the camera's optical center is the imaging geometric parameter corresponding to the principal point position determined by the camera's intrinsic parameters and the optical center. Optionally, the pupil edge can be detected or fitted in the spot image to obtain the position of the pupil center in the image coordinate system; the camera's optical center can be obtained through device calibration and stored in the device, or directly determined by the camera's intrinsic parameters.

[0046] A central optical axis is formed by connecting the center of the pupil to the optical center of the camera. This central optical axis characterizes the principal directional relationship between the eye center and the camera imaging center under the current imaging conditions. It should be noted that in this embodiment, the central optical axis is a geometric reference for coordinate correction, which can be used to define the rotation or projection reference for subsequent transformations, thereby unifying the spot coordinates under different viewing postures within the same reference frame. After obtaining the central optical axis, the coordinates of each spot in the spot image are subjected to rotation or projection transformation. The imaging plane containing the transformed spot coordinates is perpendicular to the central optical axis, thus reducing the impact of geometric distortion caused by eye rotation or relative camera posture changes on the spot distribution morphology. After completing the coordinate transformation, the transformed spot coordinates are used to replace the original spot coordinates for subsequent steps.

[0047] The head-mounted gaze tracking system multi-source and spot matching method described in this embodiment first determines the central optical axis based on the pupil center and the camera optical center before constructing the convex hull pattern to be matched. The spot coordinates are then transformed to a reference imaging plane perpendicular to the central optical axis, ensuring that the spot coordinates used to construct the convex hull pattern are within a unified geometric reference frame. Therefore, when comparing the similarity between the convex hull pattern to be matched and the reference convex hull pattern, the differences in spot distribution caused by eye movement and changes in the principal imaging direction can be reduced. This allows the similarity calculation to better reflect the spatial layout characteristics of the light source combination itself, thereby improving the matching consistency and stability under different gaze posture conditions and further enhancing the reliability of determining the correspondence between the spot and the light source.

[0048] Optionally, in one embodiment, the step of determining the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched in the reference convex hull pattern set, and obtaining the target convex hull pattern, includes: calculating the feature vector of each reference convex hull pattern in the reference convex hull pattern set and the feature vector of the convex hull pattern to be matched; calculating the similarity between the reference convex hull pattern and the convex hull pattern to be matched based on the feature vectors of the reference convex hull pattern and the feature vector of the convex hull pattern to be matched; and determining the reference convex hull pattern with the highest similarity in the reference convex hull pattern set as the target convex hull pattern.

[0049] Specifically, feature vectors are used to structurally represent the geometric shape of the convex hull, enabling unified similarity calculations in the subsequent process. For each baseline convex hull in the set of baseline convex hulls, a corresponding vectorized result is obtained according to a unified feature extraction rule. For the convex hull to be matched constructed from the current spot image, the same feature extraction rule as the baseline convex hull is used to obtain a vectorized result. By using the same rule to extract features from both the baseline and the convex hull to be matched convex hulls, the feature vectors of different shapes are comparable in dimension and meaning, thus providing a consistent input format for subsequent similarity calculations.

[0050] The similarity between the reference convex hull and the target convex hull is calculated based on the feature vectors. This similarity measure quantifies the geometrical closeness between the two convex hulls. Optionally, the similarity is a scalar value representing the degree of similarity between the two in the feature space; a higher similarity indicates a closer geometrical representation. Among all calculated similarity results, the reference convex hull corresponding to the highest value is selected as the target convex hull.

[0051] The head-mounted gaze tracking system multi-source and spot matching method described in this embodiment introduces a unified expression of feature vectors between the set of reference convex hull patterns and the convex hull patterns to be matched. This allows different convex hull patterns to be compared and calculated in the same feature space. The similarity is used to quantify the closeness between each reference convex hull pattern and the convex hull pattern to be matched. Then, the target convex hull pattern is determined by the maximum similarity rule. This reduces the risk of uncertain matching results when there are many candidate light source combinations, thereby further improving the reliability and stability of the determination of the correspondence between spot and light source.

[0052] Optionally, when M>2, and all M light spots are located on the boundary of the convex hull and serve as vertices of the convex hull, the convex hull shape is a convex polygon formed by the M light spots as vertices. The steps for calculating the feature vector of the convex hull shape include: sorting each vertex of the convex hull shape according to a preset sorting rule to obtain a vertex sequence; and calculating the interior angle vectors (a1, a2, a3, ..., a...) of the convex hull shape according to the order of the vertices in the vertex sequence. M ) and the side length vectors of the convex hull (d1,d2,d3,......,d M ); Calculate the eigenvectors of the interior angle ratios (a1 / a2, a2 / a3, ..., a) based on the interior angle vectors of the convex hull figure. M-1 / a M ); Calculate the eigenvector of the side length ratio (d1 / d2, d2 / d3, ..., d) based on the side length vector of the convex hull figure. M-1 / d M ), where the eigenvectors of the interior angle ratio and the eigenvectors of the side length ratio are the eigenvectors of the convex hull.

[0053] Specifically, when the number of detected light spots in the current frame light spot image satisfies M>2, and all M light spots are located on the convex hull boundary and serve as vertices of the convex hull, the convex hull graphic can be represented as a convex polygon formed by the M light spots as vertices. When calculating the feature vectors of the reference convex hull graphic or the convex hull graphic to be matched, the calculation can be performed in the manner described in this embodiment.

[0054] First, the vertices of the convex hull are sorted according to a preset sorting rule to obtain a vertex sequence. This vertex sequence provides a unified traversal order for subsequent calculations of interior angles and side lengths, ensuring a consistent definition method for feature vector extraction across different convex hulls. The preset sorting rule refers to the rule of arranging vertices sequentially along the boundary of the convex polygon in a fixed direction, such as clockwise or counterclockwise. Furthermore, to ensure consistency of starting points across different shapes, a rule for selecting the starting vertex can be set, such as selecting the vertex with the smallest X-coordinate in the image coordinate system, or selecting the vertex with the smallest Y-coordinate if the X-coordinates are the same. The vertex sequence obtained through this sorting method ensures that the subsequently calculated angle vectors and side length vectors are comparable in both dimensionality and positional meaning.

[0055] Next, following the order of the vertices in the vertex sequence, calculate the interior angle vector and side length vector of the polygon. The interior angle vector represents the angle at each vertex of the convex polygon, and the side length vector represents the side length between adjacent vertices of the convex polygon. Using the vertex sequence obtained from the above steps, connect adjacent vertices sequentially to form the polygon's edges. For the i-th vertex in the vertex sequence, calculate the interior angle formed by the two adjacent edges at that vertex, and then sequentially form the interior angle vector (a1, a2, a3, ..., a...). M Simultaneously, the Euclidean distance between two adjacent vertices can be calculated as the side length, and the side lengths can be sequentially arranged into a side length vector (d1, d2, d3, ..., d...). M By defining interior angles and side lengths based on the vertex sequence, it can be ensured that the angle and side length features of the same convex hull shape correspond consistently in the sequence dimension.

[0056] After obtaining the interior angle vectors (a1, a2, a3, ..., a...), M After that, calculate the eigenvectors of the interior angle ratios (a1 / a2, a2 / a3, ..., a) based on the interior angle vectors. M-1 / a M This represents the relative proportional relationship between adjacent interior angles, thus reducing the influence of overall scale or slight local distortion on the absolute value of the angle. After obtaining the side length vector (d1, d2, d3, ..., d...), M After that, calculate the eigenvector of the side length ratio (d1 / d2, d2 / d3, ..., d2 / d3) based on the side length vector. M-1 / d M ( ), representing the relative proportional relationship between adjacent side lengths, in order to reduce the impact of overall size changes on the absolute value of side lengths.

[0057] The aforementioned interior angle ratio feature vector and side length ratio feature vector are used together as the feature vector of the convex hull figure to calculate the similarity between the reference convex hull figure and the convex hull figure to be matched.

[0058] The head-mounted gaze tracking system multi-source and spot matching method described in this embodiment allows for the representation of both the convex hull to be matched and the reference convex hull as convex polygons composed of M spots when there are three or more spots. By uniformly sorting the polygon vertices, a consistent sequence definition is ensured for feature extraction of different convex hulls. Furthermore, by calculating the interior angle vectors and side length vectors, and constructing feature vectors for the ratios of adjacent interior angles and adjacent side lengths, the geometric shape of the convex hull can be expressed in terms of proportional features. This reduces the impact of absolute scale changes or local imaging differences on matching during similarity calculation, improving the comparability and stability of similarity comparisons between convex hulls. Therefore, the target convex hull shape that most closely resembles the shape of the convex hull to be matched can be selected more reliably from the reference convex hull set, further improving the accuracy and consistency of spot-source correspondence determination.

[0059] Optionally, in one embodiment, the step of calculating the similarity between the reference convex hull and the convex hull to be matched includes: performing a dot product calculation on the feature vector of the interior angle ratio of the convex hull to be matched and the feature vector of the interior angle ratio of the reference convex hull to obtain a first dot product value; performing a dot product calculation on the feature vector of the side length ratio of the convex hull to be matched and the feature vector of the side length ratio of the reference convex hull to obtain a second dot product value; and calculating the similarity based on the first dot product value and the second dot product value, wherein the larger the first dot product value and the second dot product value, the greater the similarity.

[0060] Specifically, when calculating similarity, the first step is to perform a dot product calculation on the eigenvectors of the interior angle ratios of the convex hull to be matched and the eigenvectors of the interior angle ratios of the reference convex hull, respectively, to obtain the first dot product value. This measure the consistency of the two interior angle ratio eigenvectors in terms of direction and component values, reflecting the similarity between the two convex hulls in terms of their interior angle ratio structure. The first dot product value is relatively larger when the corresponding components are closer. Secondly, the second dot product is calculated on the eigenvectors of the side length ratios of the convex hull to be matched and the eigenvectors of the side length ratios of the reference convex hull, reflecting the similarity between the two convex hulls in terms of their side length ratio structure. The second dot product value is relatively larger when the components of their side length ratios are more consistent.

[0061] The similarity score is calculated based on the first and second dot product values ​​to comprehensively reflect the degree of closeness between the convex hull of the object to be matched and the reference convex hull in terms of both angular and side-length proportions. Optionally, the similarity score can be obtained by adding the first and second dot product values.

[0062] The head-mounted gaze tracking system multi-source and spot matching method described in this embodiment uses dot product calculations on the interior angle ratio feature vector and the side length ratio feature vector to numerically measure the consistency between the convex hull shape to be matched and the reference convex hull shape in terms of angular and side length ratio structures. Furthermore, the similarity is obtained by combining the first and second dot product values, allowing the similarity evaluation to consider two complementary geometric feature dimensions: angle and side length. Therefore, when selecting a target convex hull shape, the risk of misjudgment due to relying solely on a single geometric feature can be reduced, improving the discriminative power and stability of convex hull shape matching, thereby further enhancing the reliability and consistency of determining the correspondence between the spot and the light source.

[0063] Optionally, in one embodiment, when M=2, the convex hull pattern is a line segment connecting two light spots; the reference convex hull pattern is constructed from the light spots corresponding to two adjacent light sources; the set of reference convex hull patterns includes a first set, a second set, and a third set. In the horizontal direction, the light source corresponding to the first set is located on the first side of the camera, the light source corresponding to the second set is located on the second side of the camera, and the light source corresponding to the third set is located on both sides of the camera; if the two light spots are located on the same side of the pupil center of the user's eyeball in the horizontal direction, the slope of the line segment is the feature vector of the convex hull pattern, and the reference convex hull pattern with the maximum similarity in the first set or the second set is determined as the target convex hull pattern; if the two light spots are located on both sides of the pupil center in the horizontal direction, the positional relationship between the midpoint of the line segment and the pupil center in the vertical direction is the feature vector of the convex hull pattern, and the reference convex hull pattern with the maximum similarity in the third set is determined as the target convex hull pattern.

[0064] Specifically, when the camera detects only two light spots in the current frame of the light spot image, i.e., when M=2, the convex hull pattern is the line segment connecting the two light spots. Furthermore, the inventors discovered that in actual working scenarios, based on the characteristics of light source settings, when only two light spots are detected in the light spot image, the light sources corresponding to these two light spots are usually not adjacent light sources. Therefore, the reference convex hull patterns are all constructed from the light spots corresponding to two adjacent light sources. Adjacent light sources refer to two light sources that are arranged adjacent to each other in the device's geometric layout. In this embodiment, in the horizontal direction, the light sources are distributed on both sides of the camera. Therefore, the reference convex hull patterns formed by adjacent light sources located on the first side of the camera are defined as the first set, the reference convex hull patterns formed by adjacent light sources located on the second side of the camera are defined as the second set, and the reference convex hull patterns formed by adjacent light sources located on both sides of the camera are defined as the third set. That is, the reference convex hull pattern set includes the first set, the second set, and the third set. In the horizontal direction, the light sources corresponding to the first set and the second set are located on the second side of the camera, and the light sources corresponding to the third set are located on both sides of the camera.

[0065] First, the positional relationship between the two light spots and the pupil center is determined. When the two light spots are located on the same side of the pupil center, the slope of the convex hull shape (i.e., the line segment) is used as the feature vector, and the target convex hull shape is obtained by matching within the first or second set. That is, if the two light spots are located on the same side of the pupil center in the horizontal direction, it can be determined that the light sources corresponding to the two light spots are located in the light source region on the same side of the camera. At this time, the slope of the line segment is used as the feature vector of the convex hull shape to represent the relative geometric orientation of the two light spots in this region, realizing the vectorized expression of the convex hull shape in the M=2 scenario. Further, in this case, it is only necessary to calculate the similarity between the reference convex hull shape and the convex hull shape to be matched in the first or second set (corresponding to the light spots located to the left or right of the pupil center), and determine the reference convex hull shape corresponding to the maximum similarity as the target convex hull shape.

[0066] When two light spots are located on either side of the pupil center, the vertical positional relationship between the midpoint of the convex hull (i.e., the midpoint of the line segment) and the pupil center is used as the feature vector, and the target convex hull is obtained by matching within the third set. That is, if the two light spots are located on either side of the pupil center in the horizontal direction, it can be determined that the light sources corresponding to the two light spots are located in the light source regions on either side of the camera. In this case, the vertical positional relationship between the midpoint of the line segment and the pupil center is used as the feature vector of the convex hull. The midpoint of the line segment can be obtained from the coordinates of the two light spots, and the positional relationship refers to the relative relationship of the midpoint with respect to the pupil center in the vertical direction, such as the midpoint being above or below the pupil center. Further, in this case, the similarity between the reference convex hull and the convex hull to be matched is calculated only within the third set, and the reference convex hull corresponding to the maximum similarity is determined as the target convex hull, thus completing the determination of the target convex hull in the M=2 scene.

[0067] After completing the above matching, the adjacent light source pairs corresponding to the determined target convex hull pattern can be used as the corresponding light sources for the two light spots.

[0068] The head-mounted gaze tracking system multi-source and spot matching method described in this embodiment, when only two spots are actually detected, uses the convex hull shape as the line segment connecting the two spots. Furthermore, the slope of the line segment or the vertical positional relationship between the midpoint of the line segment and the pupil center is used as a calculable feature vector. This allows the feature vector and similarity matching framework, originally applicable to general convex hull shapes, to still be executed in scenarios with few spots. Simultaneously, by limiting the benchmark convex hull shape to line segments corresponding to adjacent light sources and further dividing it into a first set, a second set, and a third set, the matching candidate range can be constrained based on the position of the spot relative to the pupil center, thereby reducing unnecessary candidate comparisons and lowering the probability of false matches. Therefore, even when the number of spots is reduced, or insufficient occlusion or reflection results in only two spots being obtained, the corresponding light source combination can still be stably determined and spot-light matching completed, significantly enhancing the applicability and robustness in scenarios with an variable number of spots.

[0069] Optionally, in one embodiment, the step of obtaining a set of reference convex hull patterns corresponding to M light spots includes: selecting M light sources from N light sources; obtaining reference light spot images of different types corresponding to the M light sources, wherein the reference light spot image includes M light spots corresponding to the M light sources, and the reference light spot image is divided into different regions along the horizontal direction, and the pupil center of the user's eyeball is located in different regions in the different types of reference light spot images; constructing a convex hull pattern using the M light spots in each type of reference light spot image to obtain a type of reference convex hull pattern; and forming a set of reference convex hull patterns by selecting all combinations of M light sources from N light sources.

[0070] Specifically, to ensure that the set of reference convex hull patterns can cover the spot distribution under different viewing postures or imaging positions, thereby improving the reliability of subsequent convex hull pattern matching, the set of reference convex hull patterns corresponding to M spots can be obtained as follows. Further, optionally, the set of reference convex hull patterns corresponding to the M spots can be pre-constructed and stored, and the set of reference convex hull patterns can be obtained by reading the stored data.

[0071] When constructing a set of reference convex hull graphics corresponding to M light spots, one approach is to first select a group of M light sources from N light sources as a candidate combination, and then obtain the reference light spot image corresponding to the candidate combination by controlling the emission of the candidate combination.

[0072] Alternatively, another approach is to ensure that all light sources emit light, obtain a master spot image containing N light spots, and then select a group of M light sources from the N light sources as a candidate combination. Delete the light spots corresponding to the light sources other than the candidate combination from the master spot image to obtain a reference spot image corresponding to the candidate combination.

[0073] Regardless of the method used to obtain the baseline spot image, each candidate combination corresponds to multiple types of baseline spot images. The difference between the types lies in the fact that horizontal segmentation of the baseline spot image yields different regions, and the pupil center of each type of baseline spot image is located in a different region. For example, horizontal segmentation of the baseline spot image yields three regions: left, center, and right, with the pupil center of each type of baseline spot image located in one of these three different regions. In this way, the set of baseline convex hull graphics can cover the morphological changes that may occur when the pupil center is located in different horizontal regions, thus providing a more comprehensive reference template for subsequent matching.

[0074] After obtaining multiple reference spot images corresponding to M light spots, a convex hull pattern is constructed using the M light spots in each reference spot image to obtain a reference convex hull pattern. This process is repeated for each candidate combination to obtain multiple reference convex hull patterns. Finally, all reference convex hull patterns obtained from all candidate combinations are summarized to form a reference convex hull pattern set. This set covers convex hull morphologies under different light source combinations and different pupil center horizontal region conditions, providing a more complete set of candidate references for determining the reference convex hull pattern with the highest similarity.

[0075] The head-mounted gaze tracking system described in this embodiment employs a multi-source light source and light spot matching method. This method constructs reference convex hull patterns for each candidate combination of M light sources selected from N light sources. Furthermore, it introduces different types of reference light spot images based on the horizontal region of the pupil center. This ensures that the set of reference convex hull patterns simultaneously covers the convex hull distribution patterns under different light source combinations and different pupil center positions. Therefore, when comparing the similarity between the convex hull pattern to be matched and the set of reference convex hull patterns, it can find candidate reference convex hull patterns with more similar shapes to the current eye position, reducing the impact of convex hull shape differences caused by changes in pupil center position on the matching results. This improves the accuracy and stability of target convex hull pattern selection and further enhances the reliability of determining the correspondence between light spots and light sources.

[0076] Optionally, before determining the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched from the reference convex hull pattern set, the same segmentation method as segmenting the reference spot image is used. First, the spot image is divided into multiple regions along the horizontal direction. Then, based on the region where the pupil center of the user's eyeball is located in the spot image, a reference convex hull pattern with the same pupil center distribution is selected from the reference convex hull pattern set and matched with the convex hull pattern to be matched. For example, if the region where the pupil center of the user's eyeball is located in the spot image is region A, then a reference convex hull pattern whose pupil center is also located in region A is selected from the reference convex hull pattern set for matching. This reduces the interference of convex hull shape differences caused by changes in the horizontal position of the pupil center on the matching results. Furthermore, since the matching search space is expanded from the entire reference convex hull pattern set, the number of candidate reference convex hull patterns is reduced, which helps to reduce the probability of incorrect matching and improve the efficiency of the matching process.

[0077] Optionally, in one embodiment, the reference spot image is divided into three different regions along the horizontal direction, and the reference spot images correspond to three types of M light sources. The reference convex hull pattern set includes three subsets of reference convex hull patterns corresponding to the three types of reference spot images. The step of determining the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched in the reference convex hull pattern set includes: dividing the spot image into three regions along the horizontal direction; selecting a subset of reference convex hull patterns from the reference convex hull pattern set according to the region where the pupil center of the user's eyeball is located in the spot image; and determining the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched from the subset of reference convex hull patterns.

[0078] Specifically, when determining the target convex hull pattern, the reference spot image is divided into three different regions along the horizontal direction, such as the left, middle, and right regions. For the same group of M light sources, reference spot images with the pupil center located in the above three different regions are acquired or selected respectively, constructing three corresponding subsets of reference convex hull patterns. During matching, the subsets are selected first, and then the target convex hull pattern is determined. Thus, the set of reference convex hull patterns can contain three subsets of reference convex hull patterns corresponding to the three types of reference spot images, which are used to characterize the set of reference convex hull shapes when the pupil center is located in the left, middle, or right region.

[0079] On the other hand, the light spot image is also divided into three equal regions horizontally, and the pupil center position is determined within the light spot image, i.e., determining which region the pupil center is located in. Specifically, when the pupil center is located in the left region, a subset of the reference convex hull corresponding to the left region is selected; when the pupil center is located in the middle region, a subset of the reference convex hull corresponding to the middle region is selected; and when the pupil center is located in the right region, a subset of the reference convex hull corresponding to the right region is selected. In this way, subsequent similarity comparisons are limited to a candidate set that better matches the current horizontal position of the pupil center.

[0080] Finally, within the selected subset of reference convex hull figures, the similarity between each reference convex hull figure and the convex hull figure to be matched is calculated, and the reference convex hull figure with the highest similarity is determined as the target convex hull figure. Since the candidate range is limited to the subset of reference convex hull figures that coincides with the current pupil center region, the selection of the most similar convex hull figure can be achieved within a smaller candidate space.

[0081] The head-mounted gaze tracking system multi-source and spot matching method described in this embodiment divides the reference spot image and the spot image to be matched into three equal regions. Based on the region where the pupil center is located in the current spot image, a corresponding subset of reference convex hull patterns is selected. This ensures that the similarity comparison is performed within a reference candidate range that is more consistent with the current eye position, thereby reducing the interference of convex hull morphology differences caused by changes in the horizontal position of the pupil center on the matching results. Simultaneously, since the matching search space is reduced from the entire set of reference convex hull patterns to a single subset, the number of candidate reference convex hull patterns is reduced, which helps to reduce the probability of false matching and improve the efficiency of the matching process. Therefore, the target convex hull pattern can be selected more stably from a candidate set that is closer to the current eye position, further improving the accuracy and stability of determining the correspondence between the spot and the light source. Based on this, the region is divided into three equal parts along the horizontal direction, and three reference convex hull graphic subsets are constructed accordingly. This method ensures that the determination of the pupil center region has consistent sensitivity in each region, reducing the misselection of subsets caused by pupil center detection errors or slight eye movements leading to cross-regional jumps, thereby improving the stability of subsequent similarity comparisons. In addition, while ensuring effective differentiation of typical left, middle, and right eye positions, the three-part division avoids the problem of cross-regional jitter amplification and sparse subset samples caused by excessively narrow regions, making the reference template coverage more balanced. On the other hand, it avoids the complexity of reference set classification, storage, and management caused by too many classifications.

[0082] Optionally, in one embodiment, the N light sources include a first light source group and a second light source group located on both sides of the camera. The step of determining the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched in the reference convex hull pattern set to obtain the target convex hull pattern includes: sorting the M light spots according to their X coordinates from smallest to largest to obtain a light spot sequence, where N>3 and M≥3; calculating the difference in the X coordinates of two adjacent light spots in the light spot sequence in turn to obtain a difference sequence; when the maximum difference in the difference sequence is greater than twice the difference between at least one of its adjacent differences, determining the two light spots corresponding to the maximum difference as a light spot boundary line, wherein the light spot boundary line divides the light spot sequence into a first light spot group and a second light spot group, the light source corresponding to the first light spot group is located in the first light source group, and the light source corresponding to the second light spot group is located in the second light source group; in the reference convex hull pattern set, filtering the reference convex hull patterns whose corresponding light source groups match the light spot groups divided by the light spot boundary line; and determining the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched in the filtered reference convex hull patterns to obtain the target convex hull pattern.

[0083] Specifically, the N light sources of the VR head-mounted display device include a first light source group and a second light source group located on both sides of the camera. To further improve the efficiency and stability of determining the target convex hull pattern from the set of reference convex hull patterns, under the condition that N>3 and M≥3, the light spots can be first grouped into left and right groups, then reference convex hull patterns that match the group can be selected from the set of reference convex hull patterns, and finally, similarity maximization matching can be performed on the selection results.

[0084] When determining the target convex hull shape, the M light spots are first sorted in ascending order of their X-coordinates to obtain a light spot sequence, where the X-coordinate is the horizontal coordinate of the center point of the light spot in the image coordinate system. By sorting the M light spots by their X-coordinates, a light spot sequence from left to right of the image can be obtained, providing a basis for subsequently calculating the horizontal interval between adjacent light spots. Then, the difference in X-coordinates between two adjacent light spots in the light spot sequence is calculated to obtain a difference sequence, which reflects the size of the horizontal interval between adjacent light spots.

[0085] When the maximum difference in the difference sequence is greater than twice at least one of its adjacent differences, the two spots corresponding to the maximum difference are defined as a spot boundary line. This boundary line is a perpendicular horizontal dividing line between the two spots, used to divide the spot sequence into left and right groups. Further, this boundary line divides the spot sequence into a first spot group and a second spot group. The spots to the left of the boundary line (the side with smaller X) constitute the first spot group, and the spots to the right of the boundary line (the side with larger X) constitute the second spot group. Therefore, the light source corresponding to the first spot group is located in the first light source group, and the light source corresponding to the second spot group is located in the second light source group, thus establishing a consistency constraint between the left and right grouping of the spots and the left and right grouping of the light sources.

[0086] In this embodiment, each reference convex hull pattern in the reference convex hull pattern set corresponds to a candidate light source combination formed by M light sources selected from N light sources. Therefore, the light sources in this candidate light source combination can be divided horizontally into light sources belonging to the first light source group and light sources belonging to the second light source group. Based on this, according to the first spot group and the second spot group, reference convex hull patterns whose left and right groupings of the candidate light source combinations are consistent with the left and right groupings of the spots are selected, that is, reference convex hull patterns whose corresponding light source groupings match the spot groupings divided by the spot boundary line are selected. In other words, among the M light sources corresponding to this reference convex hull pattern, the number of light sources belonging to the first light source group matches the number of spots in the first spot group, and the number of light sources belonging to the second light source group matches the number of spots in the second spot group, so that the candidate reference convex hull pattern is consistent with the currently observed spot grouping structure in terms of left and right grouping. Finally, in the candidate reference convex hull pattern set obtained after the above screening steps, the similarity between each reference convex hull pattern and the convex hull pattern to be matched is calculated, and the reference convex hull pattern with the highest similarity is determined as the target convex hull pattern.

[0087] The head-mounted gaze tracking system described in this embodiment employs a multi-source light spot matching method. With N light sources distributed on both sides of the camera and M≥3, the light spots are sorted by X-coordinate and adjacent differences are calculated. The boundary line of the light spots is determined using the multiple relationship between the maximum difference and adjacent differences, thus naturally dividing the light spots into the first and second spot groups corresponding to both sides of the camera. Based on this, reference convex hull graphics that match the left and right groups of the light sources and the left and right groups of the light spots are first selected from the reference convex hull graphic set. This ensures that subsequent similarity calculations are performed only within the candidate set that satisfies the left-right structural consistency. This reduces the number of candidate combinations that do not meet the left-right geometric layout constraints from entering the similarity comparison, lowering the probability of false matching. Furthermore, by reducing the size of the candidate set, the search space for similarity calculation is reduced, improving matching efficiency. Ultimately, under the combined effect of left-right grouping constraints and morphological similarity constraints, the target convex hull graphic can be determined more stably, thereby improving the reliability and consistency of determining the correspondence between light spots and light sources.

[0088] Example 2 Embodiment 2 of the present invention provides a method for matching multiple light sources and light spots in a head-mounted gaze tracking system, applied to a VR head-mounted display device. The device includes a camera and eight light sources, which are LED lights. The eight light sources are symmetrically distributed on both sides of the camera. The four light sources on the left side of the camera form a first light source group, and the four light sources on the right side of the camera form a second light source group. The arrangement of the camera and the eight LED lights is as follows: Figure 2 As shown, the four light sources on the left and the four on the right are approximately symmetrical about the camera. The light sources are represented by the symbol Li. Figure 3 As shown, each light source, after being reflected by the cornea of ​​the human eye, forms a light spot on the camera's imaging surface. Specifically, Figure 4 The flowchart of the matching method for the light source and the light spot provided in the second embodiment is as follows. Figure 4 As shown, the method includes steps S201 to S208.

[0089] Step S201: Obtain the light spot image captured by the camera.

[0090] Among them, the light spot image is as Figure 5 shown. Detect the light spot and the pupil contour. The symbol of the light spot is defined as gi, which corresponds to the light source Li one by one. The pupil center of the user's eyeball is p, and the optical center of the camera is denoted as o. The light spot image includes M light spots, and 1 < M ≤ 8. Among them, Figure 5 the shown light spot image includes 8 light spots g1 to g8.

[0091] Step S202: Perform coordinate transformation preprocessing on the light spot coordinates in the light spot image.

[0092] Specifically, through coordinate transformation, that is, rotating the imaging plane, so that the central optical axis op is perpendicular to the new imaging plane, and the light source is projected onto the new imaging plane. That is, the imaging plane where the light spot is located after coordinate transformation is perpendicular to the central optical axis op. By constructing the central optical axis and performing coordinate transformation, the subsequent geometric shape comparison can be carried out under a unified imaging geometric reference, thereby improving the comparability of the geometric features of the convex hull graphics under different eye positions and line-of-sight postures.

[0093] Step S203: Construct the convex hull graphics to be matched.

[0094] Perform a convex hull operation on the light spot set composed of M light spots to construct a convex hull graphic and obtain the convex hull graphic to be matched. The convex hull graphic represents the boundary representation of the smallest convex set containing the M light spot subsets. When M = 2, the convex hull graphic is a line segment connecting two light spots; when M > 2, that is, M is greater than or equal to 3 and less than or equal to 8, the convex hull graphic is a convex polygon composed of the convex hull boundary vertices. In order to compare the shapes of the convex hull graphics of different light spot sets, it is necessary to select a starting point with consistent geometric meaning. For example, find the point with the largest X coordinate in the convex hull graphic as the starting point, and use this point as the first position to obtain a light spot sequence arranged counterclockwise in the convex hull graphic from this starting point, so that each convex hull graphic corresponds to a light spot sequence. Subsequently, when calculating the feature vector of this convex hull graphic, it is calculated based on this light spot sequence.

[0095] Step S204: Obtain the set of reference convex hull graphics.

[0096] Among them, the acquisition of the set of reference convex hull graphics can be realized by pre-constructing and reading during matching. The construction process includes the following steps: From N=8 light sources, select M light sources to form a candidate light source combination. Obtain three types of reference spot images corresponding to the M light sources, that is, the three types of reference spot images corresponding to the pupil center being located in the left, middle, and right regions, respectively. For each type of reference spot image, construct a convex hull pattern using the M spots to obtain a reference convex hull pattern, which is taken as a subset of reference convex hull patterns. Repeat the above process for all combinations of selecting M light sources from N light sources. Finally, the reference convex hull patterns obtained from each combination together form a set of reference convex hull patterns, which includes three subsets of reference convex hull patterns.

[0097] Step S205: Filter the set of reference convex hull graphics.

[0098] In this filtering step, a subset of reference convex hull graphics can be selected from the reference convex hull graphics set based on the region where the pupil center is located in the spot image. For example, if the pupil center is located in the left region of the spot image, a subset of reference convex hull graphics with the pupil center also located in the left region can be selected.

[0099] Furthermore, when M≥3, the reference convex hull graphics can be further filtered from the selected subset of reference convex hull graphics based on the spot boundary line for similarity calculation.

[0100] In this embodiment, eight light sources are symmetrically distributed on both sides of the camera, which typically causes the light spots formed on the left and right sides to exhibit a certain degree of separation in the horizontal direction. When there is a significant horizontal gap among the detected M light spots, the gap corresponding to the maximum difference can be used as the light spot boundary line to divide the light spots into left and right groups.

[0101] For example, if we sort the 8 light spots according to their pixel x-coordinates from smallest to largest, we can see that the difference in pixel values ​​between the four left and four right light spots increases dramatically at the boundary, meaning the slope of the straight line from the fourth to the fifth light source is very steep. After sorting, we calculate the difference between the left and right light spots. If the difference is the largest at a certain point and is more than twice the difference between the previous or next light spot, then this point is the boundary between the left and right light spots, dividing the light spots into two parts.

[0102] In summary, when M≥3, when filtering the set of reference convex hull patterns, we can first select a subset of reference convex hull patterns based on the pupil center, and then filter within the subset of reference convex hull patterns based on the light spot boundary structure, which can further narrow down the candidate space and reduce false matches.

[0103] Step S206: Calculate the feature vectors of the convex hull to be matched and the selected baseline convex hull.

[0104] Optionally, the feature vectors of all reference convex hull figures can be calculated and stored separately, so that the corresponding feature vectors can be read in this step. Specifically, when calculating the feature vectors of the reference convex hull figures, the same spot sorting method as in step S203 above is used, so that each reference convex hull figure also corresponds to a spot sequence. The feature vectors of this reference convex hull figure are also calculated based on the corresponding spot sequence.

[0105] Step S207: Calculate the similarity based on the feature vectors, and determine the target convex hull shape based on the maximum similarity.

[0106] Step S208: Determine the correspondence between the light spot and the light source based on the target convex hull pattern.

[0107] The target convex hull pattern originates from a reference convex hull pattern in the set of reference convex hull patterns. This reference convex hull pattern is constructed from the light spots corresponding to M light sources selected from N light sources. Therefore, the target convex hull pattern naturally carries the identity information of these M light sources. By sorting the light spot coordinates of the target convex hull pattern and the convex hull pattern to be matched according to the same rule, a one-to-one correspondence between the light spots in the target convex hull pattern and the convex hull pattern to be matched can be established, thereby obtaining the correspondence between the light spots and the light sources.

[0108] As mentioned above, when M>2, the convex hull shape is a convex polygon; when M=2, the convex hull shape is a line segment. Based on the differences in the number of light spots resulting from these differences, different calculation methods are used when calculating the feature vector and similarity of the convex hull shapes for light spots greater than or equal to 2. It should be noted that when the number of light spots is 1, this application does not perform light source and light spot matching.

[0109] Specifically, when M>2, all M light spots are located on the boundary of the convex hull and serve as vertices of the convex hull. At this time, the convex hull pattern is a convex polygon formed with the M light spots as vertices.

[0110] The calculation of eigenvectors includes: calculating the polygon interior angle vectors (a1, a2, a3, ...) and side length vectors (d1, d2, d3, ...) according to the light spot sequence; the interior angle ratio eigenvector Fa is (a1 / a2, a2 / a3, ...); and the side length ratio eigenvector Fb is (d1 / d2, d2 / d3, ...). These features eliminate the effects of scale changes and rotation, retaining only the light spot distribution structure information.

[0111] The similarity calculation includes: taking the dot product of Fa calculated using the convex polygon to be matched and Fa of the selected reference convex polygon, taking the dot product of Fb calculated using the convex polygon to be matched and Fb of the selected reference convex polygon, and then adding the two dot product results together, which is the similarity. The reference convex polygon with the highest similarity is the target convex hull.

[0112] When M=2, the convex hull pattern is the line segment connecting the two light spots. Based on the reference convex hull pattern set, a first set, a second set, and a third set are constructed. The light source corresponding to the first set is located on the first side of the camera, that is, the first set corresponds to light sources L1 to L4, and the line segments included in the first set are g1g2, g2g3, and g3g4. The light source corresponding to the second set is located on the second side of the camera, that is, the second set corresponds to light sources L5 to L8, and the line segments included in the second set are g5g6, g6g7, and g7g8. The light source corresponding to the third set is located on both sides of the camera, that is, the third set corresponds to light sources L1, L4, L5, and L8, and the line segments included in the third set are g1g5 and g4g8.

[0113] Specifically, if the x-coordinates of both light spots are simultaneously smaller than the x-coordinate of the pupil center p, then the two light spots are located on the first side of the pupil center p in the horizontal direction. In this case, the slope of the line segment is calculated as a feature vector. The line segment with the closest slope, i.e., the line segment with the highest similarity, is found in the first set. Based on this line segment, the light source corresponding to the two light spots is further determined. For example, if the slope of the line segment is closest to the slope of line segment g1g2, then the light spot with the smaller y-coordinate is matched with light source L1, and the light spot with the larger y-coordinate is matched with light source L2.

[0114] If the x-coordinates of both light spots are simultaneously greater than the x-coordinate of the pupil center p, then the two light spots are located on the second side of the pupil center p in the horizontal direction. In this case, the slope of the line segment is calculated as the feature vector. The line segment with the closest slope, i.e., the line segment with the highest similarity, is found in the second set. Based on this line segment, the light source corresponding to the two light spots is further determined. For example, if the slope of the line segment is closest to the slope of line segment g6g7, then the light spot with the smaller y-coordinate is matched with the L6 light source, and the light spot with the larger y-coordinate is matched with the L7 light source.

[0115] If the x-coordinates of two light spots are such that one is greater than the x-coordinate of the pupil center p and the other is less than the x-coordinate of p, then the positional relationship between the midpoint of the line segment and the pupil center in the vertical direction is used as a feature vector. A line segment whose midpoint in the third set has the same positional relationship with the pupil center in the vertical direction is found. Based on this line segment, the light source corresponding to the two light spots is further determined. For example, if the midpoint of the line segment is located above the pupil center in the vertical direction, and this positional relationship is consistent with the midpoint of line segment g1g5 in the third set, then the light spot with the smaller x-coordinate is matched with light source L1, and the light spot with the larger x-coordinate is matched with light source L5.

[0116] Optionally, in one embodiment, the slopes of line segments g1g2, g2g3, and g3g4 are respectively less than 0, greater than 0.5, and between 0 and 0.5. In this case, if the x-coordinates of both light spots are simultaneously less than the x-coordinate of the pupil center p, the correspondence between the light spots and the light source can also be determined by comparing the relationship between the slopes of the line segments and 0 and 0.5. If the slope is less than 0, the light spot with the smaller y-coordinate is matched with the L1 light source, and the larger one is matched with the L2 light source; if the slope is greater than 0.5, the light spot with the smaller y-coordinate is matched with the L2 light source, and the larger one is matched with the L3 light source; if the slope is greater than 0 and less than 0.5, the light spot with the smaller y-coordinate is matched with the L3 light source, and the larger one is matched with the L4 light source.

[0117] The slopes of line segments g5g6, g6g7, and g7g8 are respectively greater than 0, less than -0.5, and between -0.5 and 0. In this case, if the x-coordinates of both light spots are simultaneously greater than the x-coordinate of the pupil center p, the correspondence between the light spots and the light sources can be determined by comparing the slopes of the line segments with 0 and -0.5. If the slope is greater than 0, the light spot with the smaller y-coordinate is matched with light source L5, and the larger one with light source L6; if the slope is less than 0 but greater than -0.5, the light spot with the smaller y-coordinate is matched with light source L6, and the larger one with light source L7; if the slope is less than -0.5, the light spot with the smaller y-coordinate is matched with light source L7, and the larger one with light source L8.

[0118] If the x-coordinate of one light spot is greater than the x-coordinate of the pupil center p, and the x-coordinate of the other is less than the x-coordinate of the pupil center p, the two light spots are added together and averaged to obtain the coordinates of the midpoint of the line connecting them. If the y-coordinate of the midpoint is less than the y-coordinate of the pupil center p, the light spot with the smaller x-coordinate is matched with the L1 light source, and the one with the larger x-coordinate is matched with the L5 light source; if the y-coordinate of the midpoint is greater than the y-coordinate of the pupil center p, the light spot with the smaller x-coordinate is matched with the L4 light source, and the one with the larger x-coordinate is matched with the L8 light source.

[0119] In special cases, if the number of light spots is 8, and the pupil center position is fixed, there is only one reference convex polygon. In this case, the target convex polygon can be determined without matching. When determining the correspondence between light spots and light sources based on the target convex hull shape, the light spots in the first light spot group are sorted by pixel y-coordinate from smallest to largest, corresponding to light sources L1 to L4 in sequence. The light spots in the second light spot group are sorted by pixel y-coordinate from smallest to largest, corresponding to light sources L5 to L8 in sequence.

[0120] Adopting the method for matching multiple light sources and light spots of the head-mounted line-of-sight tracking system provided by this embodiment, a method for matching light sources and light spots for a head-mounted V display device is provided. Through image processing, imaging plane transformation, left and right partitioning, coordinate sorting, and a matching strategy based on the morphological characteristics of convex hull graphics, high-robustness matching of light sources and light spots is achieved, which is applicable to complex scenarios such as light spot loss and interference from impurity light spots. At the same time, it does not rely on light source brightness coding, shape coding, or stroboscopic coding, and performs recognition completely based on the geometric structure characteristics of the light spot distribution, so it has higher device compatibility and lower hardware requirements. It has high robustness to unstable light spot quantity (can stably process 2 to 8 light spots); does not rely on light source coding, and does not require special light source driving or hardware modification; can effectively filter out pseudo light spots and is applicable to scenarios of glasses reflection and external light interference; has low complexity and strong real-time performance, meeting the high frame rate requirements of head-mounted VR devices; has a high matching accuracy, especially can still ensure stable recognition when light spots are missing; is highly adapted to the 8-light-ring layout method and is compatible with the mainstream VR hardware layout.

[0121] Embodiment III Corresponding to the above Embodiment I, Embodiment III of the present invention provides a device for matching multiple light sources and light spots of a head-mounted line-of-sight tracking system. The corresponding technical feature details and corresponding technical effects can be referred to the above Embodiment I and will not be elaborated herein. Figure 6 It is a block diagram of the device for matching light sources and light spots provided by Embodiment III of the present invention. As Figure 6 shown, this matching device is applied to a VR head-mounted display device, which includes a camera and N light sources. After the light sources irradiate the user's eyeball, the reflected light formed on the corneal surface enters the camera to form light spots. This matching device includes: a first acquisition module 301, a construction module 302, a second acquisition module 303, a matching module 304, and a determination module 305.

[0122] The first acquisition module 301 is used to acquire the light spot image captured by the camera, where the light spot image includes M light spots, both M and N are positive integers, and 1 < M ≤ N; the construction module 302 is used to construct a convex hull graph based on the M light spots to obtain a to-be-matched convex hull graph; the second acquisition module 303 is used to acquire a set of reference convex hull graphs corresponding to the M light spots, where M light sources are selected from the N light sources, and convex hull graphs are constructed using the light spots corresponding to the M light sources to obtain reference convex hull graphs; the matching module 304 is used to determine the reference convex hull graph with the highest similarity to the to-be-matched convex hull graph in the set of reference convex hull graphs to obtain a target convex hull graph; the determination module 305 is used to determine the correspondence between the light spots in the light spot image and the light sources according to the target convex hull graph.

[0123] Optionally, in one embodiment, after the first acquisition module 301 acquires the spot image and before the construction module 302 obtains the convex hull pattern to be matched, the matching device further includes a conversion module. The conversion module is used to determine the pupil center of the user's eyeball and the optical center of the camera, connect the pupil center and the optical center to form a central optical axis, and perform coordinate transformation on the coordinates of the spot in the spot image so that the imaging plane where the coordinates of the converted spot are located is perpendicular to the central optical axis.

[0124] Optionally, in one embodiment, the matching module includes: a first calculation unit, configured to calculate the feature vector of each reference convex hull shape in the reference convex hull shape set and the feature vector of the convex hull shape to be matched; a second calculation unit, configured to calculate the similarity between the reference convex hull shape and the convex hull shape to be matched based on the feature vector of the reference convex hull shape and the feature vector of the convex hull shape to be matched; and a first determination unit, configured to determine the reference convex hull shape with the highest similarity in the reference convex hull shape set as the target convex hull shape.

[0125] Optionally, in one embodiment, when M>2, and all M light spots are located on the convex hull boundary and serve as convex hull vertices, the convex hull shape is a convex polygon formed by the M light spots as vertices. When the first computing unit calculates the feature vector of the convex hull shape, the specific steps include: sorting each vertex of the convex hull shape according to a preset sorting rule to obtain a vertex sequence; and calculating the interior angle vectors (a1, a2, a3, ..., a...) of the convex hull shape according to the order of the vertices in the vertex sequence. M ) and the side length vectors of the convex hull (d1,d2,d3,......,d M ); Calculate the interior angle ratio characteristic vector (a1 / a2, a2 / a3, ..., a) based on the interior angle vectors of the convex hull figure. M-1 / a M ); Calculate the side length ratio eigenvector (d1 / d2, d2 / d3, ..., d) based on the side length vector of the convex hull figure. M-1 / d M ), wherein the interior angle ratio feature vector and the side length ratio feature vector are the feature vectors of the convex hull figure.

[0126] Optionally, in one embodiment, when the second calculation unit calculates the similarity between the reference convex hull graphic and the convex hull graphic to be matched, the specific steps include: performing a dot product calculation on the feature vector of the interior angle ratio of the convex hull graphic to be matched and the feature vector of the interior angle ratio of the reference convex hull graphic to obtain a first dot product value; performing a dot product calculation on the feature vector of the side length ratio of the convex hull graphic to be matched and the feature vector of the side length ratio of the reference convex hull graphic to obtain a second dot product value; and calculating the similarity based on the first dot product value and the second dot product value, wherein the larger the first dot product value and the second dot product value, the greater the similarity.

[0127] Optionally, in one embodiment, when M=2, the convex hull pattern is a line segment connecting two light spots; the reference convex hull pattern is constructed from the light spots corresponding to two adjacent light sources; the reference convex hull pattern set includes a first set, a second set, and a third set. In the horizontal direction, the light source corresponding to the first set is located on the first side of the camera, the light source corresponding to the second set is located on the second side of the camera, and the light source corresponding to the third set is located on both sides of the camera; if the two light spots are located on the same side of the pupil center of the user's eyeball in the horizontal direction, the slope of the line segment is the feature vector of the convex hull pattern, and the reference convex hull pattern with the maximum similarity in the first set or the second set is determined as the target convex hull pattern; if the two light spots are located on both sides of the pupil center in the horizontal direction, the positional relationship between the midpoint of the line segment and the pupil center in the vertical direction is the feature vector of the convex hull pattern, and the reference convex hull pattern with the maximum similarity in the third set is determined as the target convex hull pattern.

[0128] Optionally, in one embodiment, the set of reference convex hull patterns corresponding to the M light spots acquired by the second acquisition module is formed by the following steps: selecting M light sources from the N light sources; acquiring different types of reference light spot images corresponding to the M light sources, wherein the reference light spot image includes M light spots corresponding to the M light sources, and the reference light spot image is divided into different regions along the horizontal direction, and the pupil center of the user's eyeball is located in the different regions in the different types of reference light spot images; constructing a convex hull pattern using the M light spots in each type of reference light spot image to obtain a type of reference convex hull pattern; and forming the set of reference convex hull patterns by selecting all combinations of M light sources from the N light sources.

[0129] Optionally, in one embodiment, the reference spot image is divided into three different regions along the horizontal direction, the M light sources correspond to three types of reference spot images, and the reference convex hull pattern set includes three reference convex hull pattern subsets corresponding to the three types of reference spot images. The matching module includes: a segmentation unit, used to divide the spot image into three regions along the horizontal direction; a selection unit, used to select a reference convex hull pattern subset from the reference convex hull pattern set according to the region where the pupil center of the user's eyeball is located in the spot image; and a second determination unit, used to determine the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched from the reference convex hull pattern subset.

[0130] Optionally, in one embodiment, the N light sources include a first light source group and a second light source group located on both sides of the camera. The matching module includes: a sorting unit, used to sort the M light spots according to their X coordinates from smallest to largest to obtain a light spot sequence, wherein N>3 and M≥3; a third calculation unit, used to calculate the difference in X coordinates between two adjacent light spots in the light spot sequence to obtain a difference sequence; a third determination unit, used to determine that when the maximum difference in the difference sequence is greater than twice the difference between at least one of its adjacent differences, the two light spots corresponding to the maximum difference are a light spot boundary line, wherein the light spot boundary line divides the light spot sequence into a first light spot group and a second light spot group, the light source corresponding to the first light spot group is located in the first light source group, and the light source corresponding to the second light spot group is located in the second light source group; a filtering unit, used to filter reference convex hull graphics in the reference convex hull graphics set that correspond to the light spot grouping divided by the light spot boundary line; and a fourth determination unit, used to determine the reference convex hull graphics with the highest similarity to the convex hull graphics to be matched among the filtered reference convex hull graphics to obtain a target convex hull graphics.

[0131] Example 4 Embodiment 4 of the present invention provides a VR head-mounted display device, including a camera, N light sources and a controller. When the light source shines on the user's eyeball, the reflected light formed on the corneal surface enters the camera to form a light spot. The controller is used to execute any of the matching methods of light source and light spot provided by the present invention, and has its technical features and corresponding technical effects, which will not be described in detail here.

[0132] Example 5 This embodiment also provides a computer device, such as a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including a standalone server or a server cluster composed of multiple servers), etc., capable of executing programs. Figure 7As shown, the computer device 01 in this embodiment includes, but is not limited to, a memory 012 and a processor 011 that can be interconnected via a system bus, such as... Figure 7 As shown. It should be noted that, Figure 7 Only a computer device 01 with component memory 012 and processor 011 is shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0133] In this embodiment, the memory 012 (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 012 may be an internal storage unit of the computer device 01, such as the hard disk or memory of the computer device 01. In other embodiments, the memory 012 may also be an external storage device of the computer device 01, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 01. Of course, the memory 012 may include both the internal storage unit and its external storage device of the computer device 01. In this embodiment, the memory 012 is typically used to store the operating system and various reference software installed on the computer device 01, such as the program code of the multi-source light source and light spot matching method and device of the head-mounted eye tracking system in Embodiment 3. In addition, memory 012 can also be used to temporarily store various types of data that have been output or will be output.

[0134] In some embodiments, processor 011 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 011 is typically used to control the overall operation of computer device 01. In this embodiment, processor 011 is used to run program code stored in memory 012 or process data, such as a method for matching multiple light sources and light spots in a head-mounted eye-tracking system.

[0135] Example 6 This embodiment also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores a computer program. When the program is executed by a processor, it implements the corresponding function. The computer-readable storage medium of this embodiment is used to store a matching device for multiple light sources and light spots in a head-mounted eye-tracking system. When executed by a processor, it implements the matching method for multiple light sources and light spots in the head-mounted eye-tracking system of Embodiment 1 or Embodiment 2.

[0136] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0137] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0139] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for matching multiple light sources and light spots in a head-mounted gaze tracking system, characterized in that, The matching method is applied to a VR head-mounted display device, which includes a camera and N light sources. After the light sources illuminate the user's eyeball, the reflected light formed on the corneal surface enters the camera, forming a light spot. The matching method includes: Acquire a bokeh image captured by the camera, wherein the bokeh image includes M bokeh spots, where M and N are both positive integers, 1 <M≤N; Construct a convex hull pattern based on the M light spots to obtain the convex hull pattern to be matched; Obtain a set of reference convex hull patterns corresponding to the M light spots, wherein M light sources are selected from the N light sources, and convex hull patterns are constructed using the light spots corresponding to the M light sources to obtain the reference convex hull patterns; In the set of reference convex hull patterns, the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched is determined, and the target convex hull pattern is obtained; The correspondence between the light spot in the light spot image and the light source is determined based on the target convex hull pattern.

2. The method for matching multiple light sources and light spots in a head-mounted gaze tracking system according to claim 1, characterized in that, After acquiring the spot image and before obtaining the convex hull shape to be matched, the matching method further includes: Determine the center of the user's pupil and the optical center of the camera; Connecting the center of the pupil and the optical center forms a central optical axis; The coordinates of the light spot in the light spot image are transformed so that the imaging plane containing the transformed light spot coordinates is perpendicular to the central optical axis.

3. The method for matching multiple light sources and light spots in a head-mounted gaze tracking system according to claim 1, characterized in that, The step of determining the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched from the set of reference convex hull patterns, and obtaining the target convex hull pattern, includes: Calculate the feature vector of each reference convex hull shape in the reference convex hull shape set and the feature vector of the convex hull shape to be matched; The similarity between the reference convex hull and the convex hull to be matched is calculated based on the feature vectors of the reference convex hull and the feature vectors of the convex hull to be matched. In the set of reference convex hull graphics, the reference convex hull graphic with the highest similarity is determined as the target convex hull graphic.

4. The method for matching multiple light sources and light spots in a head-mounted gaze tracking system according to claim 3, characterized in that, When M>2, and all M light spots are located on the convex hull boundary and serve as vertices of the convex hull, the convex hull shape is a convex polygon formed with the M light spots as vertices. The steps for calculating the feature vector of the convex hull shape include: The vertices of the convex hull are sorted according to a preset sorting rule to obtain a vertex sequence; Calculate the interior angle vectors (a1, a2, a3, ..., a4) of the convex hull according to the order of the vertices in the vertex sequence. M ) and the side length vectors of the convex hull (d1,d2,d3,......,d M ); Calculate the interior angle ratio characteristic vector (a1 / a2, a2 / a3, ..., a3) based on the interior angle vector of the convex hull figure. M-1 / a M ); Calculate the characteristic vector of the side length ratio (d1 / d2, d2 / d3, ..., d2) based on the side length vector of the convex hull figure. M-1 / d M ), wherein the interior angle ratio feature vector and the side length ratio feature vector are the feature vectors of the convex hull figure.

5. The method for matching multiple light sources and light spots in a head-mounted gaze tracking system according to claim 4, characterized in that, The steps for calculating the similarity between the reference convex hull pattern and the convex hull pattern to be matched include: The first dot product value is obtained by performing a dot product calculation on the feature vector of the interior angle ratio of the convex hull shape to be matched and the feature vector of the interior angle ratio of the reference convex hull shape. The second dot product value is obtained by performing a dot product calculation on the feature vector of the side length ratio of the convex hull to be matched and the feature vector of the side length ratio of the reference convex hull. The similarity is calculated based on the first dot product and the second dot product, wherein the larger the first dot product and the second dot product, the greater the similarity.

6. The method for matching multiple light sources and light spots in a head-mounted gaze tracking system according to claim 3, characterized in that, When M=2, the convex hull pattern is a line segment connecting two light spots; The reference convex hull pattern is constructed from the light spots corresponding to two adjacent light sources; The reference convex hull graphic set includes a first set, a second set, and a third set. In the horizontal direction, the light source corresponding to the first set is located on the first side of the camera, the light source corresponding to the second set is located on the second side of the camera, and the light source corresponding to the third set is located on both sides of the camera. If the two light spots are located on the same side of the center of the pupil of the user's eyeball in the horizontal direction, the slope of the line segment is the feature vector of the convex hull pattern. In the first set or the second set, the reference convex hull pattern corresponding to the maximum similarity is determined as the target convex hull pattern. If the two light spots are located on either side of the center of the pupil in the horizontal direction, the positional relationship between the midpoint of the line segment and the center of the pupil in the vertical direction is the feature vector of the convex hull pattern. In the third set, the reference convex hull pattern corresponding to the maximum similarity is determined as the target convex hull pattern.

7. The method for matching multiple light sources and light spots in a head-mounted gaze tracking system according to claim 1, characterized in that, The steps to obtain the set of reference convex hull graphics corresponding to M light spots include: Select M light sources from the N light sources; Acquire reference spot images of different types corresponding to the M light sources, wherein the reference spot image includes M light spots corresponding to the M light sources, and the reference spot image is divided into different regions along the horizontal direction, and the pupil center of the user's eyeball is located in the different regions in the different types of reference spot images; For each type of reference spot image, a convex hull pattern is constructed using the M spots therein to obtain a type of reference convex hull pattern; and The reference convex hull pattern set is composed of various types of reference convex hull patterns obtained by selecting all combinations of M light sources from the N light sources.

8. The method for matching multiple light sources and light spots in a head-mounted gaze tracking system according to claim 7, characterized in that, The reference spot image is divided into three different regions along the horizontal direction. The M light sources correspond to three types of reference spot images. The reference convex hull pattern set includes three subsets of reference convex hull patterns corresponding to the three types of reference spot images. The step of determining the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched in the reference convex hull pattern set includes: The light spot image is divided into three equal regions along the horizontal direction; Based on the region where the pupil center of the user's eyeball is located in the light spot image, a subset of reference convex hull graphics is selected from the set of reference convex hull graphics; The reference convex hull pattern with the highest similarity to the convex hull pattern to be matched is determined from the subset of reference convex hull patterns.

9. The method for matching multiple light sources and light spots in a head-mounted gaze tracking system according to claim 1, characterized in that, The N light sources include a first light source group and a second light source group located on both sides of the camera. The step of determining the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched from the reference convex hull pattern set, and obtaining the target convex hull pattern, includes: The M light spots are sorted in ascending order of their X coordinates to obtain a light spot sequence, where N>3 and M≥3; The difference between the X coordinates of two adjacent light spots in the light spot sequence is calculated sequentially to obtain the difference sequence; When the maximum difference in the difference sequence is greater than twice the difference of at least one of its adjacent differences, the two light spots corresponding to the maximum difference are determined to be the light spot boundary line. The light spot boundary line divides the light spot sequence into a first light spot group and a second light spot group. The light source corresponding to the first light spot group is located in the first light source group, and the light source corresponding to the second light spot group is located in the second light source group. In the set of reference convex hull patterns, the reference convex hull patterns that correspond to the light source group and the light spot group divided by the light spot boundary line are selected; Among the selected reference convex hull graphics, the reference convex hull graphic with the highest similarity to the convex hull graphic to be matched is determined, and the target convex hull graphic is obtained.

10. A matching device for multiple light sources and light spots in a head-mounted gaze tracking system, characterized in that, The matching device is applied to a VR head-mounted display device. The device includes a camera and N light sources. After the light sources illuminate the user's eyeball, the reflected light formed on the corneal surface enters the camera, forming a light spot. The matching device includes: The first acquisition module is used to acquire a spot image captured by the camera, wherein the spot image includes M spots, where M and N are both positive integers, 1 <M≤N; The construction module is used to construct a convex hull pattern based on the M light spots to obtain the convex hull pattern to be matched; The second acquisition module is used to acquire a set of reference convex hull patterns corresponding to the M light spots, wherein M light sources are selected from the N light sources, and convex hull patterns are constructed using the light spots corresponding to the M light sources to obtain the reference convex hull patterns; The matching module is used to determine the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched in the reference convex hull pattern set, so as to obtain the target convex hull pattern; The determination module is used to determine the correspondence between the light spot in the light spot image and the light source based on the target convex hull pattern.

11. A VR head-mounted display device, characterized in that, include: A camera and N light sources; after the light sources illuminate the user's eyeball, the reflected light formed on the corneal surface enters the camera and forms a light spot. A controller for performing the matching method of light source and light spot as described in any one of claims 1 to 9.

Images