VR head-mounted display device and light source and light spot matching method and device

Abstract

The invention provides VR head-mounted display equipment and a matching method and device of a light source and a light spot. The method is applied to a VR head-mounted display device comprising a camera and N light sources, and comprises the steps of obtaining a light spot image which is shot by the camera and comprises M light spots, constructing a to-be-matched convex hull graph based on the M light spots, obtaining a reference convex hull graph set which is obtained by selecting the M light sources from the N light sources and constructing the corresponding light spots, and in the set, determining a reference convex hull graph with the highest similarity with the convex hull graph to be matched as a target convex hull graph, and further determining the corresponding relationship between the light spot and the light source. Through the method and the device, matching of the light spots and the light sources can be converted into similarity matching of convex hull patterns, the corresponding relation can still be determined under the condition that the number of the light spots is smaller than that of the light sources, dependence on additional coding or complex driving is reduced, matching stability and applicability are improved, and reliable input is provided for follow-up eye movement tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a VR head-mounted display device and a light source and light spot matching method and device. BACKGROUND

[0002] With the rapid development of virtual reality (VR) head-mounted display devices, the line-of-sight tracking technology based on eye movement has a high application value in the fields of interactive operation, gaze point rendering, fatigue detection and user behavior analysis. In order to realize high-precision eye movement tracking, the industry generally adopts a near-eye infrared imaging method at present, that is, a plurality of infrared LED light sources are arranged around an infrared camera, the highlight light spot formed by the corneal reflection of the eyeball surface and the pupil edge information are captured, and the three-dimensional posture and line-of-sight direction of the eyeball are solved by combining the imaging geometric relationship.

[0003] In a typical VR head-mounted display device, the infrared camera is installed near the lens, and a plurality of infrared light sources are regularly or circularly arranged around it, for example, 4, 6, 8 or more. The light generated by the infrared light source irradiates the user's eyeball, forms a mirror reflection on the outer surface of the cornea, and the reflected light forms a light spot after entering the camera. In theory, each LED light source should form a corresponding light spot, and the corresponding relationship between the light spot and the light source plays a decisive role in solving the center of the cornea and is a key link in the entire eye movement tracking chain.

[0004] The inventors have found that in the existing eye movement tracking technology, in order to realize the corresponding relationship between the infrared light source and the corneal reflection light spot, a technical route of light source coding is usually adopted. This kind of scheme distinguishes the light spot features of different light sources in imaging by applying different light emitting modes to multiple light sources, so as to complete the matching in subsequent image processing. The existing light source coding modes mainly include but are not limited to: brightness difference coding, flashing frequency coding, time sequence pulse coding, duty cycle coding and light source structure shape coding, and these coding modes can be used alone or combined according to hardware conditions. For example, some systems only distinguish light sources by brightness difference, some systems identify light source numbers by frequency flash or time sequence mode, and some schemes use brightness difference and pulse time sequence combination to improve the recognition reliability. Since the light source coding is directly embedded into the gray value, shape or time sequence of the light spot, it does not need to rely on complex geometric model solving, and can realize relatively direct light spot matching under the premise that the light source is working normally and the light spot is complete.

[0005] However, the inventors further found that, first, the light source coding technology relies on the correct operation of the coding hardware, and no matter whether the brightness difference, the frequency flash frequency, the timing pulse, or the combination of multiple encodings is used, additional light source driving circuit and synchronization control logic are needed, resulting in complex system structure, increased power consumption, and limitation of the volume and heat design of the head-mounted display; second, the coding features are jointly affected by factors such as eye rotation, change of reflection angle, camera exposure, light spot shielding, lens reflection, and environmental infrared noise, and no matter whether a single encoding or a combined encoding is used, the stability of the light spot features cannot be completely guaranteed; third, once a certain encoding signal is destroyed, the whole light spot matching link may fail. In addition, most encoding schemes need to count the on-off or intensity change of the light spot in multiple images, so the stability significantly decreases when the saccadic eye movement, low exposure time, or the number of light spots is reduced, and it is difficult to meet the requirements of real-time performance and robustness in the VR scene.

[0006] Therefore, no matter which encoding method is used or which encoding combination strategy is used, it is difficult to maintain reliable light spot matching performance in a scene where the number of light spots is not fixed, the noise interference is strong, and the eye movement speed is fast. How to better realize the matching of the light spot and the light source has become a technical problem that needs to be solved in the field. SUMMARY

[0007] The purpose of the present application is to provide a VR head-mounted display device and a light source and light spot matching method and device to solve the above technical problems in the prior art.

[0008] In one aspect, to achieve the above-mentioned purpose, the present application provides a light source and light spot matching method.

[0009] The light source and light spot matching method is applied to a VR head-mounted display device, which includes a camera and N light sources. The reflected light formed on the corneal surface after the light sources irradiate the user's eyeball enters the camera to form a light spot. The matching method includes: acquiring a light spot image photographed by the camera, wherein the light spot image includes M light spots, M and N are positive integers, and 1 < M ≤ N; constructing a convex hull graph according to the M light spots to obtain a to-be-matched convex hull graph; acquiring a reference convex hull graph set corresponding to the M light spots, wherein M light sources are selected from the N light sources, and a convex hull graph is constructed using the light spots corresponding to the M light sources to obtain a reference convex hull graph; determining a reference convex hull graph with the highest similarity to the to-be-matched convex hull graph in the reference convex hull graph set to obtain a target convex hull graph; and determining the correspondence between the light spots in the light spot image and the light sources according to the target convex hull graph.

[0010] Further, before the step of obtaining the to-be-matched convex hull pattern, the matching method further comprises: determining the pupil center of the user's eyeball and the optical center of the camera; connecting the pupil center and the optical center to form a central optical axis; and performing coordinate conversion on the coordinates of the light spots in the light spot image, so that the imaging plane of the converted light spots is perpendicular to the central optical axis.

[0011] Further, the step of determining the reference convex hull pattern with the highest similarity to the to-be-matched convex hull pattern in the reference convex hull pattern set to obtain the target convex hull pattern comprises: respectively calculating the feature vector of each reference convex hull pattern in the reference convex hull pattern set and the feature vector of the to-be-matched convex hull pattern; calculating the similarity between the reference convex hull pattern and the to-be-matched convex hull pattern according to the feature vector of the reference convex hull pattern and the feature vector of the to-be-matched convex hull pattern; and determining the reference convex hull pattern corresponding to the maximum similarity in the reference convex hull pattern set as the target convex hull pattern.

[0012] Further, when M>2 and the M light spots are all located on the convex hull boundary and serve as the convex hull vertices, the convex hull pattern is a convex polygon formed by the M light spots as vertices, and the step of calculating the feature vector of the convex hull pattern comprises: sorting the vertices of the convex hull pattern according to a preset sorting rule to obtain a vertex sequence; calculating the interior angle vector (a1, a2, a3,..., a M ) and the side length vector (d1, d2, d3,..., d M ) of the convex hull pattern according to the order of the vertices in the vertex sequence; calculating the interior angle ratio feature vector (a1 / a2, a2 / a3,..., a M-1 / a M ) according to the interior angle vector of the convex hull pattern; and calculating the side length ratio feature vector (d1 / d2, d2 / d3,..., d M-1 / d M ) according to the side length vector of the convex hull pattern, wherein the interior angle ratio feature vector and the side length ratio feature vector are the feature vector of the convex hull pattern.

[0013] Further, the step of calculating the similarity between the reference convex hull pattern and the to-be-matched convex hull pattern comprises: performing dot product calculation on the interior angle ratio feature vector of the to-be-matched convex hull pattern and the interior angle ratio feature vector of the reference convex hull pattern to obtain a first dot product value; performing dot product calculation on the side length ratio feature vector of the to-be-matched convex hull pattern and the side length ratio feature vector of the reference convex hull pattern to obtain a second dot product value; and calculating the similarity according to the first dot product value and the second dot product value, wherein the greater the first dot product value and the second dot product value, the greater the similarity.

[0014] Further, when M=2, the convex hull figure is a line segment connecting two light spots; the reference convex hull figure is constructed by two light spots corresponding to two adjacent light sources; the reference convex hull figure set includes a first set, a second set and a third set, in the horizontal direction, the light sources corresponding to the first set are located on the first side of the camera, the light sources corresponding to the second set are located on the second side of the camera, and the light sources corresponding to the third set are located on both sides of the camera; if the two light spots are located on the same side of the pupil center of the user's eyeball in the horizontal direction, the slope of the line segment is the feature vector of the convex hull figure, and in the first set or the second set, the reference convex hull figure corresponding to the maximum similarity is determined as the target convex hull figure; if the two light spots are located on both sides of the pupil center in the horizontal direction, the position relationship between the midpoint of the line segment and the pupil center in the vertical direction is the feature vector of the convex hull figure, and in the third set, the reference convex hull figure corresponding to the maximum similarity is determined as the target convex hull figure.

[0015] Further, the step of obtaining the reference convex hull figure set corresponding to the M light spots includes: selecting M light sources from the N light sources; obtaining different types of reference light spot images corresponding to the M light sources, wherein the reference light spot image includes M light spots corresponding to the M light sources, and the reference light spot image is divided into different regions along the horizontal direction, and in different types of reference light spot images, the pupil center of the user's eyeball is located in different regions; for each type of reference light spot image, a convex hull figure is constructed using the M light spots therein to obtain a type of reference convex hull figure; and each type of reference convex hull figure obtained by selecting M light sources from all combinations of the N light sources forms the reference convex hull figure set.

[0016] Further, the reference light spot image is equally divided into three different regions along the horizontal direction, three types of reference light spot images corresponding to the M light sources, and the reference convex hull figure set includes three reference convex hull figure subsets corresponding to the three types of reference light spot images, the step of determining the reference convex hull figure with the highest similarity to the to-be-matched convex hull figure in the reference convex hull figure set includes: equally dividing the light spot image into three regions along the horizontal direction; selecting a reference convex hull figure subset in the reference convex hull figure set according to the region where the pupil center of the user's eyeball is located in the light spot image; and determining the reference convex hull figure with the highest similarity to the to-be-matched convex hull figure in the reference convex hull figure subset.

[0017] Further, the N light sources include a first light source group and a second light source group located on two sides of the camera, the step of determining the reference convex hull graph with the highest similarity to the to-be-matched convex hull graph in the reference convex hull graph set includes: sorting the M light spots according to the X coordinates in ascending order to obtain a light spot sequence, wherein N>3 and M≥3; calculating the difference between the X coordinates of two adjacent light spots in the light spot sequence in sequence to obtain a difference sequence; when the maximum difference in the difference sequence is greater than 2 times at least one adjacent difference, it is determined that the two light spots corresponding to the maximum difference are a light spot boundary, wherein the light spot boundary divides the light spot sequence into a first light spot group and a second light spot group, the first light spot group corresponds to the first light source group, and the second light spot group corresponds to the second light source group; in the reference convex hull graph set, the reference convex hull graph corresponding to the light source grouping consistent with the light spot grouping divided by the light spot boundary is screened; and the reference convex hull graph with the highest similarity to the to-be-matched convex hull graph is determined in the screened reference convex hull graph to obtain the target convex hull graph.

[0018] In another aspect, to achieve the above object, the application provides a light source and light spot matching device.

[0019] The light source and light spot matching device is applied to a VR head-mounted display device, the device includes a camera and N light sources, the light sources irradiate to the eyeball of a user, and the reflected light formed on the corneal surface enters the camera to form light spots, the matching device includes: a first acquisition module configured to acquire a light spot image captured by the camera, wherein the light spot image includes M light spots, M and N are positive integers, and 1<M≤N; a construction module configured to construct a convex hull graph according to the M light spots to obtain a to-be-matched convex hull graph; a second acquisition module configured to acquire a reference convex hull graph set corresponding to the M light spots, wherein M light sources are selected from the N light sources, and a convex hull graph is constructed by using the light spots corresponding to the M light sources to obtain the reference convex hull graph; a matching module configured to determine a reference convex hull graph with the highest similarity to the to-be-matched convex hull graph in the reference convex hull graph set to obtain a target convex hull graph; and a determination module configured to determine the corresponding relationship between the light spots in the light spot image and the light sources according to the target convex hull graph.

[0020] In another aspect, to achieve the above object, the application provides a VR head-mounted display device.

[0021] The VR head-mounted display device includes: a camera and N light sources, the light sources irradiate to the eyeball of a user, and the reflected light formed on the corneal surface enters the camera to form light spots; and a controller configured to execute any one of the light source and light spot matching methods provided by the application.

[0022] In another aspect, to achieve the above object, the present application also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0023] In another aspect, to achieve the above object, the present application also provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable on a processor to implement the steps of the above method.

[0024] The VR head-mounted display device and the light spot matching method and device provided by the present application first acquire a light spot image containing M light spots captured by a camera, then construct a to-be-matched convex hull graph based on the M light spots, then acquire a reference convex hull graph set constructed by selecting M light sources from N light sources and using the light spots corresponding to the M light sources, then determine a reference convex hull graph with the highest similarity to the to-be-matched convex hull graph in the reference convex hull graph set as a target convex hull graph, and finally determine the correspondence between the light spots in the light spot image and the light sources according to the target convex hull graph. By the present application, the spatial distribution form of the light spots is abstracted into a convex hull graph, and the to-be-matched convex hull graph of the current frame is compared with the reference convex hull graph set constructed based on the light source combination in terms of similarity, so that the most matched light source combination can be selected in the multi-light-source and multi-light-spot scene, and the correspondence between the light spots and the light sources is determined accordingly, thereby realizing reliable determination of the correspondence between the light sources and the light spots. Without light source coding and multi-frame statistics, the present application is suitable for scenes with varying number of light spots and can stably determine the correspondence between the light spots and the light sources based on geometric distribution, thereby reducing the system hardware complexity and the risk of power consumption and heat generation and improving the matching robustness and real-time performance under the conditions of noise interference, exposure change, light spot loss, and rapid eye movement. BRIEF DESCRIPTION OF DRAWINGS

[0025] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The detailed description is made with reference to the accompanying drawings. Figure 1 A flowchart of the light source and light spot matching method provided for the first embodiment of the present application; Figure 2 A light source arrangement schematic diagram of the VR head-mounted display device provided for the embodiment of the present application; Figure 3 A schematic diagram of light spot formation provided for the embodiment of the present application; Figure 4 A flowchart of the light source and light spot matching method provided for the second embodiment of the present application; Figure 5A schematic diagram of a light spot image provided by an embodiment of the present application; Figure 6 A block diagram of a light source and light spot matching device provided by Embodiment Three of the present application; Figure 7 A hardware structure diagram of a computer device provided by Embodiment Five of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0027] Embodiment One Embodiment One of the present application provides a light source and light spot matching method. Through this method, without encoding the light source, the light spots detected in the camera image can be correctly corresponded to the multiple light sources in the VR head-mounted display device during the eye movement imaging process of the device, thereby providing a reliable light spot identification basis for subsequent eye movement parameter calculation. Specifically, Figure 1 A flowchart of the light source and light spot matching method provided by Embodiment One of the present application is shown in Figure 1 The light source and light spot matching method provided by this embodiment includes the following steps S101 to S105.

[0028] Step S101: Obtain a light spot image captured by a camera.

[0029] In this embodiment, the VR head-mounted display device includes an eye movement camera and multiple light sources. The light source can be an infrared light emitting device, such as an infrared LED, for emitting light to the user's eyeball. After the light is irradiated to the user's eyeball, a reflected light spot is formed on the corneal surface, and the reflected light spot enters the camera and is imaged as a light spot, so that the light spot image captured by the camera includes several light spots. Define N to represent the number of light sources provided in the device, and M to represent the number of light spots actually detected in the light spot image, so the light spot image includes M light spots. Due to factors such as occlusion of the eyelid, deflection of the line of sight, mirror reflection, local overexposure or underexposure, etc., some light sources may not correspond to light spots that are not collected by the camera, so M and N are both positive integers, and satisfy 1<M≤N.

[0030] Optionally, in an implementation, the light spot image can be a grayscale image or a color image, and after the light spot image is acquired, the light spot image can be processed by grayscale processing, threshold segmentation, connected domain screening, etc. to locate the light spot region, and the centroid coordinates of each light spot connected domain in the image coordinate system are taken as the light spot coordinates to obtain the coordinate set of the M light spots for marking and calculating the light spots.

[0031] Step S102: constructing a convex hull figure according to the M light spots to obtain a to-be-matched convex hull figure.

[0032] In this embodiment, the M light spots obtained in step S101 are used to construct a convex hull to obtain a to-be-matched convex hull figure, which can represent the overall geometric shape of the light spots in the spatial distribution of the current frame light spot image.

[0033] Optionally, when the convex hull figure is constructed according to the M light spots, geometric operations are performed on the representative points of the M light spots to obtain the smallest convex set that can enclose these representative points, and the geometric figure corresponding to the smallest convex set is taken as the to-be-matched convex hull figure. When the number of light spots in the light spot image is different, the to-be-matched convex hull figure can have different geometric forms, for example, when M is 2, the convex hull figure is a line segment formed by connecting two light spots; when M is 3 or greater than 3, the convex hull figure has a polygonal shape. By abstracting the light spot distribution into a convex hull figure, the subsequent matching process can be more robust to local noise and individual point position offset.

[0034] Step S103: obtaining a set of reference convex hull figures corresponding to the M light spots.

[0035] In this embodiment, the correspondence between the light spots and the light sources is determined by pre-establishing a reference. Specifically, a set of reference convex hull figures corresponding to the M light spots is pre-set, wherein M light sources are selected from N light sources, and a convex hull figure is constructed using the light spots corresponding to the M light sources to obtain the reference convex hull figure. Based on the set of available light sources in the device, that is, a group of M light sources are selected from the N light sources as a candidate light source combination, for each candidate light source combination, M light spots corresponding to the combination can be obtained, and a convex hull figure is constructed based on the M light spots to form a reference convex hull figure. By constructing convex hull figures for different light source combinations, a plurality of reference convex hull figures can be obtained, which together form the set of reference convex hull figures.

[0036] Optionally, in an implementation, the set of reference convex hull figures can be stored in the device as an offline constructed template set, or can be generated and stored during the initialization or calibration phase of the device; during actual matching, the set is directly read for comparison.

[0037] Step S104: determining a reference convex hull graph with the highest similarity to the to-be-matched convex hull graph from the reference convex hull graph set, to obtain a target convex hull graph.

[0038] In this embodiment, the to-be-matched convex hull graph obtained in step S102 is compared with the reference convex hull graph set obtained in step S103 in terms of similarity, to determine the reference convex hull graph most similar to the to-be-matched convex hull graph, and the reference convex hull graph is taken as the target convex hull graph.

[0039] The similarity is used to measure the closeness of the to-be-matched convex hull graph and each reference convex hull graph in terms of geometric shape. Then, the reference convex hull graph with the highest similarity is selected from the reference convex hull graph set, i.e., the target convex hull graph. The light source combination corresponding to the target convex hull graph is most likely to correspond to the M spots in the current frame spot image, thereby providing a basis for establishing the correspondence between the spots and the light sources in the subsequent process.

[0040] Step S105: determining the correspondence between the spots in the spot image and the light sources according to the target convex hull graph.

[0041] In this embodiment, the target convex hull graph is derived from a reference convex hull graph in the reference convex hull graph set, and the reference convex hull graph is constructed by the spots corresponding to the M light sources selected from the N light sources. Therefore, after the target convex hull graph is determined, the light source combination corresponding to the target convex hull graph can be taken as the candidate corresponding light source set of the current frame spots. Further, based on the matching result between the target convex hull graph and the to-be-matched convex hull graph, the correspondence between the M spots in the spot image and the M light sources in the candidate light source set can be established, thereby completing the matching between the light sources and the spots.

[0042] In the matching method of the light source and the light spot provided in this embodiment, first, a light spot image containing M light spots photographed by a camera is obtained, then a to-be-matched convex hull graph is constructed based on the M light spots, next, a reference convex hull graph set is obtained by selecting M light sources from N light sources and constructing a light spot corresponding to the M light sources, then a reference convex hull graph with the highest similarity to the to-be-matched convex hull graph is determined as a target convex hull graph in the reference convex hull graph set, and finally, the correspondence between the light spot in the light spot image and the light source is determined according to the target convex hull graph. By adopting the matching method of the light source and the light spot provided in this embodiment, the spatial distribution form of the light spot is abstracted into a convex hull graph, and the to-be-matched convex hull graph of the current frame is compared with the reference convex hull graph set constructed based on the light source combination in terms of similarity, so that the most matched light source combination can be selected in the multi-light-source and multi-light-spot scene, and the correspondence between the light spot and the light source is determined accordingly, thereby realizing reliable determination of the correspondence between the light source and the light spot. Without light source coding and multi-frame statistics, the correspondence between the light spot and the light source can be determined based on the geometric distribution in the scene with varying number of light spots, the system hardware complexity and the power consumption and heating risk can be reduced, and the matching robustness and real-time performance in the conditions of noise interference, exposure change, light spot loss and rapid eye movement can be improved.

[0043] Optionally, in an embodiment, after the step of obtaining the light spot image and before the step of obtaining the to-be-matched convex hull graph, the matching method further comprises: determining the pupil center of the user's eyeball and the optical center of the camera; connecting the pupil center and the optical center to form a central optical axis; and performing coordinate transformation on the coordinates of the light spots in the light spot image, so that the imaging plane where the coordinates of the transformed light spots are located is perpendicular to the central optical axis.

[0044] Specifically, the pupil center is used to represent the central position of the user's eyeball at the current moment, and the optical center of the camera is used to represent the projection center in the camera imaging model. The pupil center can be the geometric center or the fitting center of the pupil contour, and the optical center of the camera is the imaging geometric parameter corresponding to the principal point position determined by the camera internal parameter and the optical center. Optionally, the pupil edge can be detected or fitted in the light spot image to obtain the position of the pupil center in the image coordinate system; the optical center of the camera can be obtained by device calibration and stored in the device, or directly determined by the camera internal parameter.

[0045] The center optical axis is used to represent the main direction relationship between the center direction of the eyeball and the center direction of the camera under the current imaging condition. It should be noted that the center optical axis in the embodiment is a geometric reference for coordinate correction, which can be used to define the rotation reference or projection reference of subsequent conversion, so as to unify the spot coordinates under different visual line postures into the same reference frame. After obtaining the center optical axis, the coordinates of each spot in the spot image are subjected to rotation transformation or projection transformation, and the imaging plane of the converted spot coordinates is perpendicular to the center optical axis, so that the influence of the geometric distortion caused by the eyeball rotation or the relative attitude change of the camera on the spot distribution form can be reduced. After completing the coordinate conversion, the subsequent steps are performed by using the converted spot coordinates instead of the original spot coordinates.

[0046] The matching method of the light source and the spot adopts the method, before constructing the to-be-matched convex hull graph, the center optical axis is determined based on the pupil center and the camera optical center, and the spot coordinates are converted to a reference imaging plane perpendicular to the center optical axis, so that the spot coordinates used for constructing the convex hull graph are in a unified geometric reference frame. Therefore, when the similarity between the to-be-matched convex hull graph and the reference convex hull graph is compared, the difference in the spot distribution form caused by the eyeball rotation and the change in the imaging main direction can be reduced, the similarity calculation can better reflect the spatial layout characteristics corresponding to the light source combination, and the matching consistency and stability under different visual line postures can be improved, and the reliability of the determination of the corresponding relationship between the spot and the light source is further improved.

[0047] Optionally, in an embodiment, the step of determining the reference convex hull graph with the highest similarity to the to-be-matched convex hull graph in the reference convex hull graph set comprises: calculating the feature vector of each reference convex hull graph in the reference convex hull graph set and the feature vector of the to-be-matched convex hull graph respectively; calculating the similarity between the reference convex hull graph and the to-be-matched convex hull graph according to the feature vector of the reference convex hull graph and the feature vector of the to-be-matched convex hull graph; and determining the reference convex hull graph corresponding to the maximum similarity in the reference convex hull graph set as the target convex hull graph.

[0048] Specifically, the feature vector is used to structurally represent the geometric form of the convex hull graph, so as to perform unified similarity calculation subsequently. For each reference convex hull graph in the reference convex hull graph set, the corresponding vectorization result is obtained according to the unified feature extraction rule; and for the to-be-matched convex hull graph constructed according to the current spot image, the same feature extraction rule as that of the reference convex hull graph is used to obtain the vectorization result. By using the same rule to extract the features of the reference convex hull graph and the to-be-matched convex hull graph, the feature vectors of different graphs can be compared in terms of dimension and meaning, so that the subsequent similarity calculation has a consistent input form.

[0049] The similarity between the reference convex hull graph and the to-be-matched convex hull graph is calculated according to the feature vector, and the similarity is used to quantify the closeness between the reference convex hull graph and the to-be-matched convex hull graph in the geometric shape. Optionally, the similarity is a scalar value, which is used to represent the closeness between the two in the feature space; the greater the similarity, the closer the two are in the geometric feature expression; and among all the calculated similarity results, the reference convex hull graph corresponding to the maximum value is selected as the target convex hull graph.

[0050] The light source and light spot matching method described in this embodiment introduces a unified expression of the feature vector between the reference convex hull graph set and the to-be-matched convex hull graph, so that different convex hull graphs can be compared and calculated in the same feature space, and the closeness between each reference convex hull graph and the to-be-matched convex hull graph is quantified by the similarity, and then the target convex hull graph is determined by the maximum similarity rule, which reduces the risk of uncertain matching results when there are many candidate light source combinations, thereby further improving the reliability and stability of the determination of the corresponding relationship between the light spot and the light source.

[0051] Optionally, when M>2, and the M light spots are located on the convex hull boundary and serve as the convex hull vertices, the convex hull graph is a convex polygon formed by the M light spots as the vertices, and the step of calculating the feature vector of the convex hull graph comprises: sorting the vertices of the convex hull graph according to a preset sorting rule to obtain a vertex sequence; calculating the convex hull graph interior angle vector (a1, a2, a3, …, a M ) and the convex hull graph side length vector (d1, d2, d3, …, d M ) according to the order of the vertices in the vertex sequence; calculating the interior angle ratio feature vector (a1 / a2, a2 / a3, …, a M-1 / a M ) according to the convex hull graph interior angle vector; and calculating the side length ratio feature vector (d1 / d2, d2 / d3, …, d M-1 / d M ) according to the convex hull graph side length vector, wherein the interior angle ratio feature vector and the side length ratio feature vector are the feature vector of the convex hull graph.

[0052] Specifically, when the number of light spots detected in the current frame light spot image satisfies M>2, and the M light spots are located on the convex hull boundary and serve as the convex hull vertices, the convex hull graph can be represented as a convex polygon formed by the M light spots as the vertices. When calculating the feature vector of the reference convex hull graph or the to-be-matched convex hull graph, the calculation can be performed in the manner described in this embodiment.

[0053] First, the vertices of the convex hull graph are sorted according to a preset sorting rule to obtain a vertex sequence. The vertex sequence provides a uniform traversal order for subsequent inner angle and edge length calculation, so as to ensure that different convex hull graphs have consistent definitions when extracting feature vectors. The preset sorting rule refers to a rule of sequentially arranging the vertices along the boundary of the convex polygon in a fixed direction, such as sequentially arranging in a clockwise direction or a counterclockwise direction. In order to ensure the consistency of the starting point between different graphs, a selection rule for the starting vertex can also be set, such as selecting the vertex with the smallest X coordinate in the image coordinate system as the starting vertex, or selecting the vertex with the smallest Y coordinate as the starting vertex in the case of the same X coordinate, etc. The vertex sequence obtained by this sorting method can be used to ensure that the angle vectors and edge length vectors calculated subsequently are comparable in dimension and position meaning.

[0054] Secondly, according to the order of the vertices in the vertex sequence, the inner angle vector and the edge length vector of the polygon are calculated. The inner angle vector represents the turning angle at each vertex of the convex polygon, and the edge length vector represents the length of the edge between adjacent vertices of the convex polygon. The vertex sequence obtained according to the above steps sequentially connects the adjacent vertices to form the edges of the polygon. For the i-th vertex in the vertex sequence, the inner angle formed by the adjacent two edges at the vertex can be calculated, and the inner angles are sequentially combined to form an inner angle vector (a1, a2, a3,..., a M ). At the same time, the Euclidean distance between the adjacent two vertices can be calculated as the edge length, and the edge lengths are sequentially combined to form an edge length vector (d1, d2, d3,..., d M ). By defining the inner angle and the edge length based on the vertex sequence, the angle and edge length features of the same convex hull graph are ensured to correspond consistently in the sequence dimension.

[0055] After obtaining the inner angle vector (a1, a2, a3,..., a M ), the inner angle ratio feature vector (a1 / a2, a2 / a3,..., a M-1 / a M ) is calculated according to the inner angle vector, which represents the relative proportional relationship between adjacent inner angles, so as to weaken the influence of overall scale or local slight distortion on the absolute value of the angle. After obtaining the edge length vector (d1, d2, d3,..., d M ), the edge length ratio feature vector (d1 / d2, d2 / d3,..., d M-1 / d M ) is calculated according to the edge length vector, which represents the relative proportional relationship between adjacent edge lengths, so as to weaken the influence of overall size change on the absolute value of the edge length.

[0056] The aforementioned interior angle ratio feature vector and side length ratio feature vector are used together as the feature vector of the convex hull figure to calculate the similarity between the reference convex hull figure and the convex hull figure to be matched.

[0057] Using the light source and light spot matching method described in this embodiment, when there are three or more light spots, both the convex hull shape to be matched and the reference convex hull shape can be represented as a convex polygon composed of M light spots. In this case, by uniformly sorting the vertices of the polygon, a consistent sequence definition is ensured for different convex hull shapes during feature extraction. Furthermore, by calculating the interior angle vectors and side length vectors, and constructing feature vectors for the ratios of adjacent interior angles and adjacent side lengths, the geometric shape of the convex hull shape can be expressed in the form of proportional features. This reduces the impact of absolute scale changes or local imaging differences on matching during similarity calculation, improving the comparability and stability of similarity comparisons between convex hull shapes. Therefore, the target convex hull shape that is closest in shape to the convex hull shape to be matched can be selected more reliably from the set of reference convex hull shapes, further improving the accuracy and consistency of determining the correspondence between light spots and light sources.

[0058] Optionally, in one embodiment, the step of calculating the similarity between the reference convex hull and the convex hull to be matched includes: performing a dot product calculation on the feature vector of the interior angle ratio of the convex hull to be matched and the feature vector of the interior angle ratio of the reference convex hull to obtain a first dot product value; performing a dot product calculation on the feature vector of the side length ratio of the convex hull to be matched and the feature vector of the side length ratio of the reference convex hull to obtain a second dot product value; and calculating the similarity based on the first dot product value and the second dot product value, wherein the larger the first dot product value and the second dot product value, the greater the similarity.

[0059] Specifically, when calculating similarity, the first step is to perform a dot product calculation on the eigenvectors of the interior angle ratios of the convex hull to be matched and the eigenvectors of the interior angle ratios of the reference convex hull, respectively, to obtain the first dot product value. This measure the consistency of the two interior angle ratio eigenvectors in terms of direction and component values, reflecting the similarity between the two convex hulls in terms of their interior angle ratio structure. The first dot product value is relatively larger when the corresponding components are closer. Secondly, the second dot product is calculated on the eigenvectors of the side length ratios of the convex hull to be matched and the eigenvectors of the side length ratios of the reference convex hull, reflecting the similarity between the two convex hulls in terms of their side length ratio structure. The second dot product value is relatively larger when the components of their side length ratios are more consistent.

[0060] The similarity score is calculated based on the first and second dot product values ​​to comprehensively reflect the degree of closeness between the convex hull of the object to be matched and the reference convex hull in terms of both angular and side-length proportions. Optionally, the similarity score can be obtained by adding the first and second dot product values.

[0061] By using the matching method of the light spot and the light source, the consistency degree of the angle proportion structure and the edge length proportion structure of the convex hull graph to be matched and the reference convex hull graph can be measured in a numerical way by respectively performing dot product calculation on the inner angle ratio feature vector and the edge length ratio feature vector, and the similarity is further obtained by synthesizing the first dot product value and the second dot product value, so that the similarity evaluation considers the angle and the edge length as two complementary geometric feature dimensions. Therefore, when selecting the target convex hull graph, the risk of misjudgment caused by only relying on a single geometric feature can be reduced, the discrimination ability and stability of the convex hull graph matching are improved, and the reliability and consistency of the determination of the corresponding relationship between the light spot and the light source are further improved.

[0062] Optionally, in an embodiment, when M=2, the convex hull graph is a line segment formed by connecting two light spots; the reference convex hull graph is constructed by the light spots corresponding to two adjacent light sources; the reference convex hull graph set includes a first set, a second set and a third set, in the horizontal direction, the light sources corresponding to the first set are located on the first side of the camera, the light sources corresponding to the second set are located on the second side of the camera, and the light sources corresponding to the third set are located on both sides of the camera; if the two light spots are located on the same side of the pupil center of the user's eyeball in the horizontal direction, the slope of the line segment is the feature vector of the convex hull graph, and in the first set or the second set, the reference convex hull graph corresponding to the maximum similarity is determined as the target convex hull graph; if the two light spots are located on both sides of the pupil center in the horizontal direction, the position relationship between the midpoint of the line segment and the pupil center in the vertical direction is the feature vector of the convex hull graph, and in the third set, the reference convex hull graph corresponding to the maximum similarity is determined as the target convex hull graph.

[0063] Specifically, when the camera detects only two light spots in the current frame light spot image, i.e. M=2, the convex hull graph is a line segment formed by connecting the two light spots. Meanwhile, the inventors have found that in actual working scenarios, based on the characteristics of light source arrangement, when only two light spots are detected in the light spot image, the light sources corresponding to the two light spots are usually not adjacent to each other, based on which, the reference convex hull graph is constructed by the light spots corresponding to two adjacent light sources. The adjacent light sources refer to two light sources arranged adjacent to each other in the device geometric layout. In this embodiment, in the horizontal direction, the light sources are distributed on both sides of the camera, and thus, the reference convex hull graphs formed by the adjacent light sources located on the first side of the camera are defined as the first set, the reference convex hull graphs formed by the adjacent light sources located on the second side of the camera are defined as the second set, and the reference convex hull graphs formed by the adjacent light sources located on both sides of the camera are defined as the third set, i.e. the reference convex hull graph set includes the first set, the second set and the third set, in the horizontal direction, the light sources corresponding to the first set, the light sources corresponding to the second set are located on the second side of the camera, and the light sources corresponding to the third set are located on both sides of the camera.

[0064] Firstly, the positional relationship between the two spots and the pupil center is determined. When the two spots are located on the same side of the pupil center, the slope of the convex hull figure, i.e. the line segment, is taken as the feature vector, and the target convex hull figure is obtained by matching in the first set or the second set. That is, if the two spots are located on the same side of the pupil center of the user's eyeball in the horizontal direction, it can be determined that the light sources corresponding to the two spots are located in the light source region on the same side of the camera. At this time, the slope of the line segment is taken as the feature vector of the convex hull figure, representing the relative geometric trend of the two spots in the region on the same side, so as to realize the vectorization expression of the convex hull figure form in the M=2 scenario. Further, in this case, only the similarity between the reference convex hull figure and the to-be-matched convex hull figure needs to be calculated in the first set or the second set (corresponding to the light spots being located on the left side or the right side of the pupil center), and the reference convex hull figure corresponding to the maximum similarity is determined as the target convex hull figure.

[0065] When the two spots are located on the two sides of the pupil center respectively, the vertical positional relationship between the midpoint of the convex hull figure, i.e. the line segment, and the pupil center is taken as the feature vector, and the target convex hull figure is obtained by matching in the third set. That is, if the two spots are located on the two sides of the pupil center in the horizontal direction, it can be determined that the light sources corresponding to the two spots are located in the light source region on the two sides of the camera. At this time, the positional relationship between the midpoint of the line segment and the pupil center in the vertical direction is taken as the feature vector of the convex hull figure. The midpoint of the line segment can be obtained from the coordinates of the two spots, and the positional relationship refers to the relative relationship of the midpoint with respect to the pupil center in the vertical direction, for example, the midpoint is located above or below the pupil center. Further, in this case, only the similarity between the reference convex hull figure and the to-be-matched convex hull figure needs to be calculated in the third set, and the reference convex hull figure corresponding to the maximum similarity is determined as the target convex hull figure, so as to complete the determination of the target convex hull figure in the M=2 scenario.

[0066] After the above matching is completed, the adjacent light source pair corresponding to the determined target convex hull figure can be taken as the corresponding light sources of the two spots.

[0067] The matching method of the light source and the light spot according to the embodiment can be used when the actual detected number of light spots is only two. The convex hull graph is a line segment connecting the two light spots. Further, the slope of the line segment or the position relationship between the midpoint of the line segment and the pupil center in the vertical direction is used as a calculable feature vector, so that the feature vector and the similarity matching framework originally applicable to the general convex hull graph can still be executed in the few-light-spot scene. Meanwhile, by limiting the reference convex hull graph to the line segment corresponding to the adjacent light source and further dividing the line segment into a first set, a second set and a third set, the matching candidate range can be constrained according to the position of the light spot relative to the pupil center, thereby reducing unnecessary candidate comparison and reducing the probability of false matching. Thus, in the case where the number of light spots is reduced, occluded or insufficiently reflected, only two light spots can be obtained. The corresponding light source combination can still be stably determined and the light spot and light source matching can be completed, thereby significantly enhancing the applicability and robustness in the scene with a variable number of light spots.

[0068] Optionally, in an embodiment, the step of obtaining the reference convex hull graph set corresponding to the M light spots comprises: selecting M light sources from the N light sources; obtaining different-class reference light spot images corresponding to the M light sources, wherein the reference light spot image comprises M light spots corresponding to the M light sources, and the reference light spot image is divided into different regions along the horizontal direction, and the pupil centers of the user's eyeballs are located in the different regions in the different-class reference light spot images; for each class of reference light spot image, constructing a convex hull graph using the M light spots in the reference light spot image to obtain a class of reference convex hull graph; and the reference convex hull graphs obtained from all combinations of selecting M light sources from the N light sources form the reference convex hull graph set.

[0069] Specifically, in order to enable the reference convex hull graph set to cover the light spot distribution under different visual poses or different imaging positions, thereby improving the reliability of subsequent convex hull graph matching, the reference convex hull graph set corresponding to the M light spots can be obtained in the following manner. Further optionally, the reference convex hull graph set corresponding to the M light spots is constructed and stored in advance, and the reference convex hull graph set is obtained by reading the stored data.

[0070] When constructing the reference convex hull graph set corresponding to the M light spots, one way is to first take a group of light sources with a number of M from the N light sources as a candidate combination, and obtain the reference light spot image corresponding to the candidate combination by controlling the light emission of the candidate combination.

[0071] Alternatively, another way is to ensure that all light sources emit light, obtain a mother light spot image including N light spots, and then select a group of light sources with a number of M from the N light sources as a candidate combination, and delete the light spots corresponding to the light sources outside the candidate combination from the mother light spot image to obtain the reference light spot image corresponding to the candidate combination.

[0072] Regardless of the way of obtaining the reference spot image, each candidate combination corresponds to multiple categories of reference spot images. The difference between the categories is that the reference spot image is divided into different regions in the horizontal direction, and the pupil centers of different categories of reference spot images are located in different regions. For example, the reference spot image is divided into three left, middle and right regions in the horizontal direction, and the pupil centers of different categories of reference spot images are located in the three different regions. In this way, the reference convex hull pattern set can cover the morphological changes of the spot distribution when the pupil center is in different horizontal regions, thereby providing more comprehensive reference templates for subsequent matching.

[0073] After obtaining the multiple categories of reference spot images corresponding to the M spots, for each category of reference spot image, a convex hull pattern is constructed using the M spots therein, to obtain a category of reference convex hull pattern, so that multiple categories of reference convex hull patterns are obtained for each candidate combination. Finally, the reference convex hull patterns of all categories obtained by the candidate combinations are summarized to form a reference convex hull pattern set, wherein the reference convex hull pattern set covers the convex hull shapes under different light source combinations and different pupil center horizontal region conditions, thereby providing a more complete candidate reference set for determining the reference convex hull pattern with the highest similarity.

[0074] The light source and spot matching method described in the embodiment constructs a reference convex hull pattern for each candidate combination of selecting M light sources from N light sources, and further introduces different categories of reference spot images based on the pupil center horizontal region, so that the reference convex hull pattern set covers the convex hull distribution shapes under different light source combinations and different pupil center positions. Therefore, when comparing the to-be-matched convex hull pattern with the reference convex hull pattern set, a candidate reference convex hull pattern with a closer shape can be found in the reference closer to the current eye position condition, thereby reducing the influence of the convex hull shape difference caused by the change of the pupil center position on the matching result, improving the accuracy and stability of the target convex hull pattern selection, and further improving the reliability of the determination of the corresponding relationship between the spot and the light source.

[0075] Further optionally, before the step of determining the reference convex hull pattern with the highest similarity to the to-be-matched convex hull pattern in the reference convex hull pattern set, the pupil image is segmented into a plurality of regions along the horizontal direction using the same segmentation manner as that of the reference spot image, and then, according to the region in which the pupil center of the user's eyeball is located in the pupil image, a reference convex hull pattern with consistent pupil center distribution is selected from the reference convex hull pattern set, and matched with the to-be-matched convex hull pattern. For example, if the region in which the pupil center of the user's eyeball is located in the pupil image is region A, then a reference convex hull pattern with the pupil center located in region A is selected from the reference convex hull pattern set for matching. In this way, the influence of the difference in convex hull shape caused by the change in horizontal position of the pupil center on the matching result is reduced, and since the matching search space is from the entire reference convex hull pattern set, the number of candidate reference convex hull patterns is reduced, which is conducive to reducing the probability of false matching and improving the efficiency of the matching process.

[0076] Optionally, in an embodiment, the reference spot image is equally divided into three different regions along the horizontal direction, three types of reference spot images corresponding to the M light sources, and the reference convex hull pattern set includes three reference convex hull pattern subsets corresponding to the three types of reference spot images. The step of determining the reference convex hull pattern with the highest similarity to the to-be-matched convex hull pattern in the reference convex hull pattern set includes: equally dividing the pupil image into three regions along the horizontal direction; selecting a reference convex hull pattern subset from the reference convex hull pattern set according to the region in which the pupil center of the user's eyeball is located in the pupil image; and determining the reference convex hull pattern with the highest similarity to the to-be-matched convex hull pattern in the reference convex hull pattern subset.

[0077] Specifically, in determining the target convex hull pattern, the reference spot image is equally divided into three different regions along the horizontal direction, for example, which can be understood as a left region, a middle region, and a right region. For the same group of M light sources, reference spot images with pupil centers located in the three different regions are obtained or selected, three reference convex hull pattern subsets are constructed, and the target convex hull pattern is determined by selecting a subset first during matching. In this way, the reference convex hull pattern set can include three reference convex hull pattern subsets corresponding to three types of reference spot images, which are used to represent the reference convex hull shape set when the pupil center is located in the left region, the middle region, or the right region, respectively.

[0078] On the other hand, the spot image is also equally divided into three regions in the horizontal direction, and the pupil center position is determined in the spot image, that is, it is judged which region the pupil center is located in. Among them, when the pupil center is located in the left region, the reference convex hull pattern sub-set corresponding to the left region is selected; when the pupil center is located in the middle region, the reference convex hull pattern sub-set corresponding to the middle region is selected; when the pupil center is located in the right region, the reference convex hull pattern sub-set corresponding to the right region is selected. In this way, the subsequent similarity comparison can be limited in the candidate set that is more consistent with the current pupil center horizontal position condition.

[0079] Finally, in the selected reference convex hull pattern sub-set, the similarity of each reference convex hull pattern and the to-be-matched convex hull pattern is calculated, and the reference convex hull pattern with the highest similarity is determined as the target convex hull pattern. Since the candidate range has been limited in the reference convex hull pattern sub-set consistent with the current pupil center region, the similarity maximization selection can be completed in a smaller candidate space.

[0080] The light source and spot matching method described in the embodiment divides the reference spot image and the to-be-matched spot image using consistent three-equal-division regions, and selects the corresponding reference convex hull pattern sub-set according to the region where the pupil center in the current spot image is located, so that the similarity comparison is performed in the reference candidate range that is more consistent with the current eye position condition, thereby reducing the interference of the convex hull shape difference caused by the change of the horizontal position of the pupil center on the matching result. At the same time, since the matching search space is reduced from the entire reference convex hull pattern set to a single sub-set, the number of candidate reference convex hull patterns is reduced, which is conducive to reducing the probability of false matching and improving the efficiency of the matching process. Therefore, the target convex hull pattern can be more stably selected in the candidate set that is closer to the current eye position condition, further improving the accuracy and stability of the determination of the corresponding relationship between the spot and the light source. On this basis, the three-equal-division segmentation method is used for region division, and three reference convex hull pattern sub-sets are constructed according to the segmentation method, which can make the determination of the region where the pupil center is located have consistent sensitivity in each region, reduce the misselection of the sub-set caused by the cross-region jump due to the detection error of the pupil center or slight eye movement, and thus improve the stability of the subsequent similarity comparison. In addition, the three-equal-division can effectively distinguish the typical eye position states of left, middle and right, on the one hand, avoid the amplification of cross-region jitter caused by too narrow regions and the problem of sparse samples in the sub-set, so that the reference template covers more evenly, and on the other hand, avoid the complexity of reference set classification storage and management caused by too many classifications.

[0081] Optionally, in an embodiment, the N light sources include a first light source group and a second light source group located on two sides of the camera, and the step of determining the target convex hull graph from the reference convex hull graph set with the highest similarity to the to-be-matched convex hull graph includes: sorting the M light spots in ascending order of X coordinates to obtain a light spot sequence, where N > 3 and M ≥ 3; sequentially calculating the difference between the X coordinates of two adjacent light spots in the light spot sequence to obtain a difference sequence; when the maximum difference in the difference sequence is greater than twice at least one of its adjacent differences, determining that the two light spots corresponding to the maximum difference are a light spot boundary, where the light spot boundary divides the light spot sequence into a first light spot group and a second light spot group, the first light spot group corresponds to the first light source group, and the second light spot group corresponds to the second light source group; screening, from the reference convex hull graph set, a reference convex hull graph corresponding to the light source grouping consistent with the light spot grouping divided by the light spot boundary; and determining, from the screened reference convex hull graphs, the target convex hull graph with the highest similarity to the to-be-matched convex hull graph.

[0082] Specifically, the N light sources of the VR head-mounted display device include a first light source group and a second light source group located on two sides of the camera. To further improve the efficiency and stability of determining the target convex hull graph from the reference convex hull graph set, under the condition that N > 3 and M ≥ 3, the light spots can be first grouped into left and right groups, then the reference convex hull graphs consistent with the grouping are screened from the reference convex hull graph set, and finally the similarity maximization matching is performed in the screening results.

[0083] In determining the target convex hull graph, the M light spots are first sorted in ascending order of X coordinates to obtain a light spot sequence, where the X coordinate is the horizontal coordinate of the light spot center point in the image coordinate system. By sorting the M light spots according to the X coordinates, a light spot sequence from the left to the right of the image can be obtained, providing a basis for subsequent calculation of the horizontal interval of adjacent light spots. Then, the difference between the X coordinates of two adjacent light spots in the light spot sequence is calculated to obtain a difference sequence, which reflects the interval size of adjacent light spots in the horizontal direction.

[0084] When the maximum difference in the difference sequence is greater than twice at least one of its adjacent differences, it is determined that the two light spots corresponding to the maximum difference are a light spot boundary, where the light spot boundary is a vertical-to-horizontal dividing position located between the two light spots, used to divide the light spot sequence into left and right groups. Further, the light spot boundary divides the light spot sequence into a first light spot group and a second light spot group, the light spots on the left side (X smaller side) of the boundary constitute the first light spot group, and the light spots on the right side (X larger side) of the boundary constitute the second light spot group. Therefore, the first light spot group corresponds to the first light source group, and the second light spot group corresponds to the second light source group, thereby establishing a consistency constraint between the left and right grouping of light spots and the left and right grouping of light sources.

[0085] In this embodiment, each reference convex hull pattern in the reference convex hull pattern set corresponds to a candidate light source combination formed by selecting M light sources from N light sources, so the light sources in the candidate light source combination can be divided into light sources belonging to the first light source group and light sources belonging to the second light source group in the horizontal direction. Based on this, the left and right groups of those candidate light source combinations are filtered out which are consistent with the left and right groups of the light spots, that is, the reference convex hull patterns corresponding to the light source group division consistent with the light spot group division are filtered out. In other words, among the M light sources corresponding to the reference convex hull pattern, the number of light sources belonging to the first light source group matches the number of light spots in the first light spot group, and the number of light sources belonging to the second light source group matches the number of light spots in the second light spot group, so that the candidate reference convex hull pattern is consistent with the currently observed light spot group structure in the left and right group structure. Finally, in the candidate reference convex hull pattern set obtained after the above filtering step, the similarity of each reference convex hull pattern and the convex hull pattern to be matched is calculated, and the reference convex hull pattern with the highest similarity is determined as the target convex hull pattern.

[0086] The light source and light spot matching method described in this embodiment can be used in the case where N light sources are distributed on both sides of the camera and M is greater than or equal to 3. By sorting the light spots according to the X coordinate and calculating the adjacent difference value, the light spot boundary is determined using the relationship between the maximum difference value and the multiple of the adjacent difference value, so that the light spots are naturally divided into the first light spot group and the second light spot group corresponding to the two sides of the camera. On this basis, the reference convex hull patterns whose light source left and right groups are consistent with the light spot left and right groups are first filtered out in the reference convex hull pattern set, so that the subsequent similarity calculation is only performed in the candidate set that meets the left and right structure consistency. In this way, on the one hand, the candidate combinations that do not meet the left and right geometric layout constraints can be reduced to enter the similarity comparison, thereby reducing the probability of false matching. On the other hand, by reducing the size of the candidate set, the search space of the similarity calculation is reduced, and the matching efficiency is improved. Finally, the target convex hull pattern can be more stably determined under the joint action of the left and right group constraints and the shape similarity constraints, thereby improving the reliability and consistency of the determination of the corresponding relationship between the light spots and the light sources.

[0087] Embodiment two The embodiment two of the present application provides a light source and light spot matching method applied to a VR head-mounted display device. The device includes a camera and 8 light sources, which are LED lamps. The 8 light sources are symmetrically distributed on both sides of the camera. The 4 light sources on the left side of the camera form a first light source group, and the 4 light sources on the right side of the camera form a second light source group. The arrangement of the camera and the 8 LED lamps is shown in Figure 2 As shown in Figure 3 Each light source will form a light spot on the imaging surface of the camera after being reflected by the cornea of the human eye. Specifically, Figure 4A flow chart of the matching method of the light source and the light spot provided for the second embodiment is shown in Figure 4 The method includes steps S201 to S208.

[0088] Step S201: Obtain the light spot image captured by the camera.

[0089] As shown in Figure 5 , the light spot and the pupil contour are detected, the symbol of the light spot is gi, which corresponds to the light source Li one by one, the pupil center of the user's eyeball is p, and the optical center of the camera is o. The light spot image includes M light spots, and 1 < M ≤ 8, wherein, Figure 5 The light spot image shown in

[0090] Step S202: Coordinate conversion preprocessing is performed on the light spot coordinates in the light spot image.

[0091] Specifically, by coordinate conversion, that is, rotating the imaging surface, the central optical axis op is made perpendicular to the new imaging surface, and the light source is projected onto the new imaging surface, that is, the imaging surface after coordinate conversion is perpendicular to the central optical axis op. By constructing the central optical axis and performing coordinate conversion, the subsequent geometric shape comparison is performed under a unified imaging geometric reference, thereby improving the comparability of the convex hull graphic geometric features under different eye positions and line of sight attitudes.

[0092] Step S203: Construct the convex hull graphic to be matched.

[0093] The convex hull operation is performed on the light spot set composed of M light spots to construct the convex hull graphic, and the convex hull graphic to be matched is obtained. The convex hull graphic represents the boundary representation of the minimum convex set containing the M light spot set. When M = 2, the convex hull graphic is a line segment formed by connecting two light spots; when M > 2, that is, M is greater than or equal to 3 and less than or equal to 8, the convex hull graphic is a convex polygon composed of convex hull boundary vertices. In order to compare the shapes of the convex hull graphics of different light spot sets, a starting point with consistent geometric meaning needs to be selected, for example, the point with the maximum X coordinate in the convex hull graphic is found as the starting point, which is taken as the first, and a light spot sequence in counterclockwise order starting from the starting point is obtained, so that each convex hull graphic corresponds to a light spot sequence, and the feature vector of the convex hull graphic is calculated based on the light spot sequence in the subsequent calculation.

[0094] Step S204: Obtain the reference convex hull graphic set.

[0095] The acquisition of the reference convex hull graphic set can be achieved by pre-construction and reading during matching. The construction process includes the following steps: M light sources are selected from N=8 light sources to form a candidate light source combination, and three types of reference light spot images corresponding to the M light sources are obtained, i.e., three types of reference light spot images corresponding to the pupil center located in the left region, the middle region and the right region respectively. For each type of reference light spot image, a convex hull graph is constructed using the M light spots therein to obtain a reference convex hull graph as a reference convex hull graph subset. For all combinations of selecting M light sources from N light sources, the above process is repeated, and finally the reference convex hull graphs of each type obtained from each combination together form a reference convex hull graph set, which includes three reference convex hull graph subsets.

[0096] Step S205: screening the reference convex hull graph set.

[0097] In this screening step, a reference convex hull graph subset in the reference convex hull graph set can be selected according to the region where the pupil center is located in the light spot image, for example, the pupil center is located in the left region in the light spot image, and the reference convex hull graph subset in which the pupil center is also located in the left region is selected.

[0098] Further, when M≥3, the reference convex hull graph can also be further screened based on the light spot boundary line in the selected reference convex hull graph subset for similarity calculation.

[0099] In this embodiment, 8 light sources are symmetrically distributed on both sides of the camera, which usually causes the light spots formed on the left and right sides to have a certain separation trend in the horizontal direction. When there is a significant horizontal gap in the detected M light spots, the gap corresponding to the maximum difference can be used as the light spot boundary line to divide the light spots into two groups.

[0100] For example, 8 light spots are sorted according to the pixel x coordinate from small to large, and it can be found that the difference value of the boundary part of the left 4 light spots and the right 4 light spots will sharply increase, that is, the straight line slope from the 4th light source to the 5th light source is very large. After sorting, the difference value is calculated, and if the difference value at a certain position is the largest and greater than twice the difference value of the previous or next position, then this position is the boundary line of the left and right light spots, and the light spots are divided into two parts from this position.

[0101] In summary, when M≥3, when screening the reference convex hull graph set, the reference convex hull graph subset can be selected according to the pupil center first, and then the reference convex hull graph subset is screened according to the light spot boundary line structure, which further reduces the candidate space and reduces the false matching.

[0102] Step S206: calculating the feature vectors of the to-be-matched convex hull graph and the screened reference convex hull graph.

[0103] Optionally, the feature vectors of all the reference convex hull patterns can be calculated and stored respectively, so that the corresponding feature vector can be read in this step. In the calculation of the feature vector of the reference convex hull pattern, the same spot sorting mode as step S203 is adopted, so that the reference convex hull pattern also corresponds to a spot sequence, and the feature vector of the reference convex hull pattern is calculated based on the corresponding spot sequence.

[0104] Step S207: Calculate the similarity according to the feature vector, and determine the target convex hull pattern according to the maximum similarity.

[0105] Step S208: Determine the correspondence between the spot and the light source according to the target convex hull pattern.

[0106] The target convex hull pattern is derived from a reference convex hull pattern in the reference convex hull pattern set, and the reference convex hull pattern is constructed by the spots corresponding to the M light sources selected from the N light sources, so the target convex hull pattern naturally carries the identity information of the M light sources. By sorting the spot coordinates of the target convex hull pattern and the to-be-matched convex hull pattern in the same rule, a one-to-one correspondence between the spots in the target convex hull pattern and the to-be-matched convex hull pattern can be established, and then the correspondence between the spot and the light source can be obtained.

[0107] As described above, when M>2, the convex hull pattern is a convex polygon, and when M=2, the convex hull pattern is a line segment. The difference between the convex hull patterns caused by the difference in the number of spots is taken into account in the calculation of the feature vector and the similarity of the convex hull pattern, and different calculation methods are used when the number of spots is greater than 2 and equal to 2. It should be noted that when the number of spots is 1, the present application does not perform the matching of the light source and the spot.

[0108] Specifically, when M>2, the M spots are located on the convex hull boundary and serve as the convex hull vertices, and at this time the convex hull pattern is a convex polygon formed by the M spots as vertices.

[0109] The calculation of the feature vector includes: calculating the polygon interior angle vector (a1, a2, a3,...) according to the spot sequence, the edge length vector (d1, d2, d3,...), the interior angle ratio feature vector Fa (a1 / a2, a2 / a3,...), and the edge length ratio feature vector Fb (d1 / d2, d2 / d3,...). These features eliminate the effects of scale change and rotation, and only retain the spot distribution structure information.

[0110] The similarity calculation includes: dot product of Fa calculated by the to-be-matched convex polygon and Fa of the screened reference convex polygon, dot product of Fb calculated by the to-be-matched convex polygon and Fb of the screened reference convex polygon, and then adding the two dot product results, which is the similarity, and the reference convex polygon with the largest similarity is the target convex hull graph.

[0111] When M=2, the convex hull graph is a line segment connecting two light spots, a first set, a second set and a third set are constructed based on the reference convex hull graph set, the first set corresponds to the light sources located on the first side of the camera, that is, the first set corresponds to light sources L1 to L4, the first set includes line segments g1g2, g2g3 and g3g4; the second set corresponds to the light sources located on the second side of the camera, that is, the second set corresponds to light sources L5 to L8, the second set includes line segments g5g6, g6g7 and g7g8; the third set corresponds to the light sources located on both sides of the camera, that is, the third set corresponds to light sources L1, L4, L5 and L8, and the third set includes line segments g1g5 and g4g8.

[0112] Specifically, if the x coordinates of the two light spots are both less than the x coordinate of the pupil center p, the two light spots are located on the first side of the pupil center p in the horizontal direction, at this time the line segment slope is taken as the feature vector, the line segment with the closest slope in the first set is found, that is, the line segment with the largest similarity, and the light sources corresponding to the two light spots are further determined according to the line segment. For example, the line segment slope is closest to the slope of the line segment g1g2, then the light spot with the smaller y coordinate in the two light spots is matched with the light source L1, and the light spot with the larger y coordinate is matched with the light source L2.

[0113] If the x coordinates of the two light spots are both greater than the x coordinate of the pupil center p, the two light spots are located on the second side of the pupil center p in the horizontal direction, at this time the line segment slope is taken as the feature vector, the line segment with the closest slope in the second set is found, that is, the line segment with the largest similarity, and the light sources corresponding to the two light spots are further determined according to the line segment. For example, the line segment slope is closest to the slope of the line segment g6g7, then the light spot with the smaller y coordinate in the two light spots is matched with the light source L6, and the light spot with the larger y coordinate is matched with the light source L7.

[0114] If the x coordinates of the two light spots are one greater than the x coordinate of the pupil center p and the other less than the x coordinate of the pupil center p, the position relationship of the line segment midpoint with the pupil center in the vertical direction is taken as the feature vector, the line segment with the same position relationship of the midpoint with the pupil center in the vertical direction in the third set is found, and the light sources corresponding to the two light spots are further determined according to the line segment. For example, the line segment midpoint is located above the pupil center in the vertical direction, and the position relationship of the midpoint of the line segment g1g5 with the pupil center in the third set is consistent, then the light spot with the smaller x coordinate in the two light spots is matched with the light source L1, and the light spot with the larger x coordinate is matched with the light source L5.

[0115] Optionally, in one embodiment, the slopes of the line segments g1g2, g2g3 and g3g4 are less than 0, greater than 0.5 and between 0 and 0.5, respectively. In this case, if the x coordinates of the two spots are both less than the x coordinate of the pupil center p, the corresponding relationship between the spots and the light sources can also be determined by comparing the slopes of the line segments with 0 and 0.5. If the slope is less than 0, the spot with the smaller y coordinate is matched with the L1 light source and the larger one is matched with the L2 light source. If the slope is greater than 0.5, the spot with the smaller y coordinate is matched with the L2 light source and the larger one is matched with the L3 light source. If the slope is greater than 0 and less than 0.5, the spot with the smaller y coordinate is matched with the L3 light source and the larger one is matched with the L4 light source.

[0116] The slopes of the line segments g5g6, g6g7 and g7g8 are greater than 0, less than -0.5 and between -0.5 and 0, respectively. In this case, if the x coordinates of the two spots are both greater than the x coordinate of the pupil center p, the corresponding relationship between the spots and the light sources can also be determined by comparing the slopes of the line segments with 0 and -0.5. If the slope is greater than 0, the spot with the smaller y coordinate is matched with the L5 light source and the larger one is matched with the L6 light source. If the slope is less than 0 and greater than -0.5, the spot with the smaller y coordinate is matched with the L6 light source and the larger one is matched with the L7 light source. If the slope is less than -0.5, the spot with the smaller y coordinate is matched with the L7 light source and the larger one is matched with the L8 light source.

[0117] If the x coordinate of one spot is greater than the x coordinate of the pupil center p and the x coordinate of the other spot is less than the x coordinate of the pupil center p, the two spots are added and averaged to obtain the coordinates of the midpoint of the line segment. If the y coordinate of the midpoint is less than the y coordinate of the pupil center p, the spot with the smaller x coordinate is matched with the L1 light source and the larger one is matched with the L5 light source. If the y coordinate of the midpoint is greater than the y coordinate of the pupil center p, the spot with the smaller x coordinate is matched with the L4 light source and the larger one is matched with the L8 light source.

[0118] In a special case, if the number of spots is 8 and the position of the pupil center is certain, the reference convex polygon is only one, and the target convex polygon can be determined without matching. When the corresponding relationship between the spots and the light sources is determined according to the target convex polygon, the spots in the first spot group are sorted in ascending order of the pixel y coordinates and are sequentially matched with the light sources L1 to L4. The spots in the second spot group are sorted in ascending order of the pixel y coordinates and are sequentially matched with the light sources L5 to L8.

[0119] The matching method for the light source and the light spot provided by the embodiment provides a matching method for a light source and a light spot of a head-mounted VR display device. Through image processing, imaging plane transformation, left-right partitioning, coordinate sorting, and a matching strategy based on convex hull graphic morphological features, high-robustness matching of the light source and the light spot is achieved, and the matching method is suitable for complex scenes such as light spot loss and impurity light spot interference. At the same time, the matching method does not depend on light source brightness coding, shape coding, or stroboscopic coding, and is completely based on the geometric structure characteristics of the light spot distribution for identification, and therefore has higher device compatibility and lower hardware requirements. The matching method has high robustness to unstable light spot quantity (2-8 light spots can be stably processed); the matching method does not depend on light source coding, and does not require special light source driving or hardware modification; the matching method can effectively filter out false light spots, and is suitable for scenarios of eyeglass reflection and external light interference; the matching method has low complexity, high real-time performance, and is suitable for the high frame rate requirement of a head-mounted VR device; the matching method has high matching accuracy, and can ensure stable identification even when light spots are lost; and the matching method is highly adaptive to an 8-lamp ring arrangement, and is compatible with mainstream VR hardware layouts.

[0120] Embodiment three Corresponding to the above-described embodiment one, the embodiment three of the present application provides a matching device for a light source and a light spot. Correspondingly, the technical feature details and the corresponding technical effects can be referred to the above-described embodiment one, which will not be described herein again. Figure 6 A block diagram of the matching device for a light source and a light spot provided by the embodiment three of the present application is shown in Figure 6 The matching device is applied to a VR head-mounted display device, and the device includes a camera and N light sources. The light source irradiates to the eyeball of a user, and the reflected light formed on the corneal surface enters the camera to form a light spot. The matching device includes a first acquisition module 301, a construction module 302, a second acquisition module 303, a matching module 304, and a determination module 305.

[0121] The first acquisition module 301 is configured to acquire a light spot image captured by the camera. The light spot image includes M light spots, and M and N are positive integers, 1 The construction module 302 is configured to construct a convex hull graphic according to the M light spots to obtain a to-be-matched convex hull graphic. The second acquisition module 303 is configured to acquire a reference convex hull graphic set corresponding to the M light spots. The M light spots are selected from the N light sources, and a convex hull graphic is constructed by using the light spots corresponding to the M light sources to obtain a reference convex hull graphic. The matching module 304 is configured to determine, in the reference convex hull graphic set, a reference convex hull graphic having the highest similarity with the to-be-matched convex hull graphic to obtain a target convex hull graphic. The determination module 305 is configured to determine a corresponding relationship between the light spots in the light spot image and the light sources according to the target convex hull graphic.

[0122] Optionally, in an embodiment, before the step of obtaining the to-be-matched convex hull pattern by the constructing module 302, after the step of obtaining the spot image by the first obtaining module 301, the matching device further comprises a converting module, the converting module is configured to determine a pupil center of the user's eyeball and an optical center of the camera, connect the pupil center and the optical center to form a central optical center axis, and perform coordinate conversion on coordinates of the spot in the spot image, so that an imaging plane where the coordinates of the converted spot are located is perpendicular to the central optical center axis.

[0123] Optionally, in an embodiment, the matching module comprises: a first calculating unit configured to calculate a feature vector of each reference convex hull pattern in the set of reference convex hull patterns and a feature vector of the to-be-matched convex hull pattern respectively; a second calculating unit configured to calculate a similarity between the reference convex hull pattern and the to-be-matched convex hull pattern according to the feature vector of the reference convex hull pattern and the feature vector of the to-be-matched convex hull pattern; and a first determining unit configured to determine, in the set of reference convex hull patterns, a reference convex hull pattern corresponding to the maximum similarity as the target convex hull pattern.

[0124] Optionally, in an embodiment, when M>2 and the M spots are all located on the convex hull boundary and serve as convex hull vertices, the convex hull pattern is a convex polygon formed by taking the M spots as vertices, and the first calculating unit, when calculating the feature vector of the convex hull pattern, specifically performs the following steps: sorts the vertices of the convex hull pattern according to a preset sorting rule to obtain a vertex sequence; calculates a convex hull internal angle vector (a1, a2, a3,..., a M ) and a convex hull side length vector (d1, d2, d3,..., d M ) according to the order of the vertices in the vertex sequence; calculates an internal angle ratio feature vector (a1 / a2, a2 / a3,..., a M-1 / a M ) according to the convex hull internal angle vector; and calculates a side length ratio feature vector (d1 / d2, d2 / d3,..., d M-1 / d M ) according to the convex hull side length vector, wherein the internal angle ratio feature vector and the side length ratio feature vector are the feature vector of the convex hull pattern.

[0125] Optionally, in an embodiment, when the second computing unit calculates the similarity between the reference convex hull pattern and the to-be-matched convex hull pattern, the steps specifically performed include: performing dot product calculation between the internal angle ratio feature vector of the to-be-matched convex hull pattern and the internal angle ratio feature vector of the reference convex hull pattern to obtain a first dot product value; performing dot product calculation between the edge length ratio feature vector of the to-be-matched convex hull pattern and the edge length ratio feature vector of the reference convex hull pattern to obtain a second dot product value; and calculating the similarity according to the first dot product value and the second dot product value, wherein the greater the first dot product value and the second dot product value, the greater the similarity.

[0126] Optionally, in an embodiment, when M=2, the convex hull pattern is a line segment formed by connecting two light spots; the reference convex hull pattern is constructed by light spots corresponding to two adjacent light sources; the reference convex hull pattern set includes a first set, a second set and a third set, in the horizontal direction, the light sources corresponding to the first set are located on the first side of the camera, the light sources corresponding to the second set are located on the second side of the camera, and the light sources corresponding to the third set are located on both sides of the camera; if the two light spots are located on the same side of the pupil center of the user's eyeball in the horizontal direction, the slope of the line segment is the feature vector of the convex hull pattern, and in the first set or the second set, the reference convex hull pattern corresponding to the maximum similarity is determined as the target convex hull pattern; if the two light spots are located on both sides of the pupil center in the horizontal direction, the positional relationship between the midpoint of the line segment and the pupil center in the vertical direction is the feature vector of the convex hull pattern, and in the third set, the reference convex hull pattern corresponding to the maximum similarity is determined as the target convex hull pattern.

[0127] Optionally, in an embodiment, the second obtaining module obtains the reference convex hull pattern set corresponding to the M light spots by the following steps: selecting M light sources from the N light sources; obtaining different types of reference light spot images corresponding to the M light sources, wherein the reference light spot image includes M light spots corresponding to the M light sources, and the reference light spot image is divided into different regions along the horizontal direction, and the pupil center of the user's eyeball is located in the different regions in the different types of reference light spot images; for each type of reference light spot image, constructing a convex hull pattern by using the M light spots in the reference light spot image to obtain a type of reference convex hull pattern; and each type of reference convex hull pattern obtained from all combinations of selecting M light sources from the N light sources constitutes the reference convex hull pattern set.

[0128] Optionally, in an embodiment, the reference light spot image is equally divided into three different regions along the horizontal direction, the M light sources correspond to three types of reference light spot images, the reference convex hull pattern set comprises three reference convex hull pattern sub-sets corresponding to the three types of reference light spot images, and the matching module comprises: a segmentation unit configured to equally divide the light spot image into three regions along the horizontal direction; a selection unit configured to select a reference convex hull pattern sub-set from the reference convex hull pattern set according to a region in which the pupil center of the user's eyeball is located in the light spot image; and a second determination unit configured to determine, from the reference convex hull pattern sub-set, a reference convex hull pattern having the highest similarity to the to-be-matched convex hull pattern.

[0129] Optionally, in an embodiment, the N light sources comprise a first light source group and a second light source group located on two sides of the camera, the matching module comprises: an ordering unit configured to order the M light spots according to X coordinates from small to large to obtain a light spot sequence, wherein N>3 and M≥3; a third calculation unit configured to sequentially calculate differences between X coordinates of adjacent two light spots in the light spot sequence to obtain a difference sequence; a third determination unit configured to determine, when a maximum difference in the difference sequence is greater than 2 times at least one adjacent difference, that a light spot boundary exists between two light spots corresponding to the maximum difference, wherein the light spot boundary divides the light spot sequence into a first light spot group and a second light spot group, the first light spot group corresponds to light sources located in the first light source group, and the second light spot group corresponds to light sources located in the second light source group; a screening unit configured to screen, from the reference convex hull pattern set, a reference convex hull pattern corresponding to a light source group consistent with a light spot group divided by the light spot boundary; and a fourth determination unit configured to determine, from the screened reference convex hull pattern, a reference convex hull pattern having the highest similarity to the to-be-matched convex hull pattern to obtain a target convex hull pattern.

[0130] Embodiment Four Embodiment Four of the present application provides a VR head-mounted display device, comprising a camera, N light sources, and a controller, wherein the light sources irradiate the eyeball of a user, and reflected light formed on the corneal surface enters the camera to form a light spot; the controller is configured to execute any one of the light source and light spot matching methods provided by the present application, has the technical features and corresponding technical effects thereof, which will not be described here.

[0131] Embodiment Five The present embodiment also provides a computer device, such as a smartphone, a tablet computer, a notebook computer, a desktop computer, a rack-mounted server, a blade server, a tower server, or a cabinet server (including a stand-alone server or a server cluster composed of multiple servers), etc. that can execute programs. Figure 7As shown, the computer device 01 of the present embodiment at least includes, but is not limited to, a memory 012 and a processor 011 which are communicatively connected to each other through a system bus. Figure 7 It should be noted that, Figure 7 Only the computer device 01 with the components of the memory 012 and the processor 011 is shown, but it should be understood that all the components shown are not required to be implemented, and more or less components can be alternatively implemented.

[0132] In the present embodiment, the memory 012 (i.e. a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g. an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 012 can be an internal storage unit of the computer device 01, such as a hard disk or a memory of the computer device 01. In other embodiments, the memory 012 can also be an external storage device of the computer device 01, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 01. Of course, the memory 012 can also include both the internal storage unit and the external storage device of the computer device 01. In the present embodiment, the memory 012 is generally used to store an operating system and various types of reference software installed on the computer device 01, such as program codes of the light source and light spot matching method and device of the embodiment three, etc. In addition, the memory 012 can also be used to temporarily store various types of data that have been output or will be output.

[0133] The processor 011 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 011 is generally used to control the overall operation of the computer device 01. In the present embodiment, the processor 011 is used to run program codes or process data stored in the memory 012, such as the light source and light spot matching method, etc.

[0134] Embodiment six The embodiment also provides a computer readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a programmable read only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App reference mall, etc., which stores a computer program, and the program is executed by a processor to realize corresponding functions. The computer readable storage medium of the embodiment is used to store the matching device of the light source and the light spot, and the matching method of the light source and the light spot of the embodiment one or the embodiment two is realized when the computer readable storage medium is executed by the processor.

[0135] It should be noted that in this paper, the term "including", "containing" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0136] The above-mentioned embodiment numbers of the application are only for description, not representing the advantages and disadvantages of the embodiments.

[0137] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platform, of course, they can also be realized by hardware, but in many cases, the former is a better embodiment.

[0138] The above is only the preferred embodiment of the application, and does not limit the patent scope of the application, and any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the application.

Claims

1. A method of matching a light source to a light spot, characterized in that, The matching method is applied to a VR head-mounted display device, the device comprising a camera and N light sources, the light sources irradiating to the user's eyeball, the reflected light formed on the corneal surface entering the camera to form a light spot, the matching method comprising: obtaining a light spot image captured by the camera, wherein the light spot image comprises M light spots, M and N are positive integers, 1 constructing a convex hull graph according to the M light spots to obtain a to-be-matched convex hull graph; obtaining a reference convex hull graph set corresponding to the M light spots, wherein M light sources are selected from the N light sources, and a convex hull graph is constructed by using the light spots corresponding to the M light sources to obtain a reference convex hull graph; determining a reference convex hull graph with the highest similarity to the to-be-matched convex hull graph in the reference convex hull graph set to obtain a target convex hull graph; determining the correspondence between the light spots in the light spot image and the light sources according to the target convex hull graph.

2. The light source and light spot matching method according to claim 1, characterized in that, After the step of obtaining the light spot image and before the step of obtaining the to-be-matched convex hull graph, the matching method further comprises: determining the pupil center of the user's eyeball and the optical center of the camera; connecting the pupil center and the optical center to form a central optical axis; performing coordinate transformation on the coordinates of the light spots in the light spot image to make the imaging plane where the coordinates of the light spots are located after the transformation perpendicular to the central optical axis.

3. The light source and light spot matching method according to claim 1, wherein, The step of determining a reference convex hull graph with the highest similarity to the to-be-matched convex hull graph in the reference convex hull graph set to obtain a target convex hull graph comprises: respectively calculating the feature vector of each reference convex hull graph in the reference convex hull graph set and the feature vector of the to-be-matched convex hull graph; calculating the similarity between the reference convex hull graph and the to-be-matched convex hull graph according to the feature vector of the reference convex hull graph and the feature vector of the to-be-matched convex hull graph; determining the reference convex hull graph corresponding to the maximum similarity in the reference convex hull graph set as the target convex hull graph.

4. The light source and light spot matching method according to claim 3, characterized in that, When M>2, and the M light spots are all located on the convex hull boundary and serve as convex hull vertices, the convex hull graph is a convex polygon formed by the M light spots as vertices, and the step of calculating the feature vector of the convex hull graph comprises: sequencing the vertices of the convex hull graph according to a preset sequencing rule to obtain a vertex sequence; According to the order of each vertex in the vertex sequence, calculate the convex hull figure inner angle vector (a1, a2, a3,......,a M ) and the convex hull figure side length vector (d1, d2, d3,......,d M ); According to the inner angle vector in the convex hull graph, an inner angle ratio feature vector (a1 / a2, a2 / a3,..., an-1 / an) is calculated. M-1 / a M ) According to the convex hull figure side length vector, a side length ratio feature vector (d1 / d2, d2 / d3,..., d M-1 / d M ) is calculated, wherein the internal angle ratio feature vector and the side length ratio feature vector are feature vectors of the convex hull figure.

5. The light source and light spot matching method according to claim 4, characterized in that, The step of calculating the similarity between the reference convex hull graph and the to-be-matched convex hull graph comprises: performing dot product calculation on the internal angle ratio feature vector of the to-be-matched convex hull graph and the internal angle ratio feature vector of the reference convex hull graph to obtain a first dot product value; performing dot product calculation on the side length ratio feature vector of the to-be-matched convex hull graph and the side length ratio feature vector of the reference convex hull graph to obtain a second dot product value; calculating the similarity according to the first dot product value and the second dot product value, wherein the greater the first dot product value and the second dot product value, the greater the similarity.

6. The light source and light spot matching method according to claim 3, wherein when M=2, the convex hull graph is a line segment connecting two light spots; the reference convex hull graph is constructed by the light spots corresponding to two adjacent light sources; The reference convex hull pattern set includes a first set, a second set and a third set, in the horizontal direction, the first set corresponds to the light source located on the first side of the camera, the second set corresponds to the light source located on the second side of the camera, and the third set corresponds to the light source located on both sides of the camera; If the two light spots are located on the same side of the pupil center of the user's eyeball in the horizontal direction, the slope of the line segment is the feature vector of the convex hull pattern, and in the first set or the second set, the reference convex hull pattern corresponding to the maximum similarity is determined as the target convex hull pattern; If the two light spots are located on both sides of the pupil center of the user's eyeball in the horizontal direction, the position relationship between the midpoint of the line segment and the pupil center in the vertical direction is the feature vector of the convex hull pattern, and in the third set, the reference convex hull pattern corresponding to the maximum similarity is determined as the target convex hull pattern.

7. The light source and light spot matching method according to claim 1, wherein, The step of obtaining the reference convex hull pattern set corresponding to the M light spots includes: selecting M light sources from the N light sources; obtaining different reference light spot images corresponding to the M light sources, wherein the reference light spot image includes M light spots corresponding to the M light sources, and the reference light spot image is divided into different regions along the horizontal direction, and in the different reference light spot images, the pupil center of the user's eyeball is located in the different regions respectively; for each type of reference light spot image, constructing a convex hull pattern using the M light spots in the reference light spot image to obtain a reference convex hull pattern of the type; and the reference convex hull patterns of all types obtained by selecting M light sources from all combinations of the N light sources form the reference convex hull pattern set.

8. The light source and light spot matching method according to claim 7, characterized in that, The reference light spot image is equally divided into three different regions along the horizontal direction, the M light sources correspond to three types of reference light spot images, and the reference convex hull pattern set includes three reference convex hull pattern sub-sets corresponding to the three types of reference light spot images. The light spot image is equally divided into three regions along the horizontal direction; selecting a reference convex hull pattern sub-set from the reference convex hull pattern set according to the region where the pupil center of the user's eyeball is located in the light spot image; determining the reference convex hull pattern with the highest similarity to the to-be-matched convex hull pattern in the reference convex hull pattern sub-set.

9. The light source and light spot matching method according to claim 1, wherein, The N light sources include a first light source group and a second light source group located on both sides of the camera, and the step of determining the reference convex hull pattern with the highest similarity to the to-be-matched convex hull pattern in the reference convex hull pattern set to obtain a target convex hull pattern includes: sorting the M light spots according to the X coordinates from small to large to obtain a light spot sequence, wherein N>3 and M≥3; calculating the difference between the X coordinates of two adjacent light spots in the light spot sequence in sequence to obtain a difference sequence; and calculating the slope of the line segment connecting the two light spots in the difference sequence to obtain the target convex hull pattern. When a maximum difference in the difference sequence is greater than 2 times at least one adjacent difference, it is determined that a light spot boundary is between two light spots corresponding to the maximum difference, wherein the light spot boundary divides the light spot sequence into a first light spot group and a second light spot group, the first light spot group corresponds to the first light source group, and the second light spot group corresponds to the second light source group; In the reference convex hull pattern set, a reference convex hull pattern corresponding to a light source group consistent with the light spot group divided by the light spot boundary is screened; A reference convex hull pattern with the highest similarity to the to-be-matched convex hull pattern is determined in the screened reference convex hull patterns, to obtain a target convex hull pattern.

10. A light source and light spot matching device, characterized in that, The matching device is applied to a VR head-mounted display device, the device includes a camera and N light sources, the light sources irradiate to eyeballs of a user, reflected light formed on a corneal surface enters the camera, and forms light spots, and the matching device includes: A first acquisition module is configured to acquire a light spot image captured by the camera, wherein the light spot image includes M light spots, M and N are positive integers, and 1 < M ≤ N; A construction module is configured to construct a convex hull pattern according to the M light spots, to obtain a to-be-matched convex hull pattern; A second acquisition module is configured to acquire a reference convex hull pattern set corresponding to the M light spots, wherein M light sources are selected from the N light sources, and a convex hull pattern is constructed by using light spots corresponding to the M light sources, to obtain a reference convex hull pattern; A matching module is configured to determine a reference convex hull pattern with the highest similarity to the to-be-matched convex hull pattern in the reference convex hull pattern set, to obtain a target convex hull pattern; A determination module is configured to determine a corresponding relationship between the light spots in the light spot image and the light sources according to the target convex hull pattern.

11. A VR head-mounted display device, comprising: The device includes: A camera and N light sources, the light sources irradiate to eyeballs of a user, reflected light formed on a corneal surface enters the camera, and forms light spots; A controller is configured to perform the light source and light spot matching method in any one of claims 1 to 9.

Images