Eye-tracking methods, head-mounted display devices, and computer-readable storage media

Abstract

This invention discloses a gaze tracking method, a head-mounted display device, and a computer-readable storage medium. The gaze tracking method comprises the following steps: controlling the alternating operation of a left-eye bright pupil tracking component and a left-eye dark pupil tracking component to alternately acquire left-eye bright pupil images and left-eye dark pupil images; tracking and locating the left-eye gaze based on the alternately acquired left-eye bright pupil images and left-eye dark pupil images; controlling the alternating operation of a right-eye bright pupil tracking component and a right-eye dark pupil tracking component to alternately acquire right-eye bright pupil images and right-eye dark pupil images; and tracking and locating the right-eye gaze based on the alternately acquired right-eye bright pupil images and right-eye dark pupil images. This invention can minimize interference between the light emitted by the light sources of the corresponding eye-tracking modules of the left and right eyes in the head-mounted display device, thereby improving the recognition accuracy of gaze tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wearable devices, in particular to a line-of-sight tracking method, a head-mounted display device and a computer readable storage medium. BACKGROUND

[0002] The head-mounted display device such as a VR (Virtual Reality) device or an AR (Augmented Reality) device is a virtual reality and augmented reality product that is rapidly developing and popularizing at present.

[0003] At present, the mainstream technology for line-of-sight tracking of the head-mounted display device is an eye tracking and line-of-sight detection technology based on image processing, which calculates and records the position looked at by the eyes in real time. The head-mounted display device mainly uses dark pupil technology and takes the corneal reflection point as a reference point to calculate the pupil-corneal reflection point vector to track the line-of-sight of the human eye. However, there are the following problems: since the left eye and the right eye correspond to one eye tracking module respectively, and the same light source is used in both eye tracking modules, the light emitted by the light sources in the two eye tracking modules will interfere with each other when calibrating or using, so that the error of the calculation result is increased, thereby causing the accuracy of tracking and identifying the line-of-sight of the user to be low. SUMMARY

[0004] The main purpose of the present application is to provide a line-of-sight tracking method, a head-mounted display device and a computer readable storage medium, which aims to solve the technical problem that the light emitted by the light sources of the two eye tracking modules in the head-mounted display device is easy to interfere with each other, affecting the line-of-sight tracking accuracy.

[0005] To achieve the above-mentioned purpose, the present application provides a line-of-sight tracking method, which comprises:

[0006] The line-of-sight tracking method is applied to a head-mounted display device, the head-mounted display device comprises a left eye bright pupil tracking assembly, a left eye dark pupil tracking assembly, a right eye bright pupil tracking assembly and a right eye dark pupil tracking assembly, and the method comprises:

[0007] controlling the left eye bright pupil tracking assembly and the left eye dark pupil tracking assembly to alternately run so as to alternately collect left eye bright pupil images and left eye dark pupil images; and tracking and positioning the left eye line-of-sight according to the alternately collected left eye bright pupil images and left eye dark pupil images;

[0008] controlling the right eye bright pupil tracking assembly and the right eye dark pupil tracking assembly to alternately run so as to alternately collect right eye bright pupil images and right eye dark pupil images; and tracking and positioning the right eye line-of-sight according to the alternately collected right eye bright pupil images and right eye dark pupil images;

[0009] The left eye bright pupil image and the right eye dark pupil image are collected in the same collection cycle, and the left eye dark pupil image and the right eye bright pupil image are collected in the same collection cycle.

[0010] Optionally, before the step of tracking and positioning the left eye visual line according to the alternately collected left eye bright pupil image and left eye dark pupil image, the method further comprises:

[0011] identifying the eyeball features of the left eye bright pupil image and the left eye dark pupil image collected in adjacent collection cycles to obtain left eye bright pupil eyeball features and left eye dark pupil eyeball features, wherein the eyeball features include at least one of a pupil center position, a pupil shape, an iris position, an iris shape and a glint position;

[0012] performing feature matching on the left eye bright pupil eyeball features and the left eye dark pupil eyeball features to obtain a feature matching degree;

[0013] If the feature matching degree is greater than a preset degree threshold, then the step of tracking and positioning the left eye visual line according to the alternately collected left eye bright pupil image and left eye dark pupil image is performed.

[0014] Optionally, after the step of performing feature matching on the left eye bright pupil eyeball features and the left eye dark pupil eyeball features to obtain a feature matching degree, the method further comprises:

[0015] If the feature matching degree is less than or equal to a preset degree threshold, then performing image recognition on the left eye bright pupil image to determine a bright pupil collection interference degree corresponding to the left eye bright pupil image, and performing image recognition on the left eye dark pupil image to determine a dark pupil collection interference degree corresponding to the left eye dark pupil image;

[0016] If the bright pupil collection interference degree is greater than the dark pupil collection interference degree by a first preset degree value, then correcting the left eye bright pupil image by using the left eye dark pupil image; and tracking and positioning the left eye visual line according to the left eye dark pupil image and the corrected left eye bright pupil image.

[0017] If the bright pupil collection interference degree is less than the dark pupil collection interference degree by a second preset degree value, then correcting the left eye dark pupil image by using the left eye bright pupil image; and tracking and positioning the left eye visual line according to the left eye bright pupil image and the corrected left eye dark pupil image.

[0018] Optionally, the step of correcting the left eye bright pupil image by using the left eye dark pupil image comprises:

[0019] obtaining left eye dark pupil eyeball features corresponding to the left eye dark pupil image and left eye bright pupil eyeball features corresponding to the left eye bright pupil image;

[0020] performing a sum value calculation on the left eye dark pupil eyeball feature of the first preset weight and the left eye bright pupil eyeball feature of the second preset weight to obtain a fusion eyeball feature, wherein the first preset weight is greater than the second preset weight, and a sum of the first preset weight and the second preset weight is equal to one;

[0021] associating the bright pupil image corresponding to the fusion eyeball feature as a corrected left eye bright pupil image.

[0022] Optionally, the step of performing image recognition on the left eye bright pupil image to determine the bright pupil collection interference degree corresponding to the left eye bright pupil image comprises:

[0023] performing image recognition on the left eye bright pupil image to determine the size of the pupil and / or the depth of the iris color in the left eye bright pupil image;

[0024] determining the bright pupil collection interference degree corresponding to the left eye bright pupil image according to the size of the pupil and / or the depth of the iris color in the left eye bright pupil image.

[0025] Optionally, the step of performing image recognition on the left eye dark pupil image to determine the dark pupil collection interference degree corresponding to the left eye dark pupil image comprises:

[0026] performing image recognition on the left eye dark pupil image to determine the covering degree of the eyelashes and / or the depth of the iris color in the left eye dark pupil image;

[0027] determining the dark pupil collection interference degree corresponding to the left eye dark pupil image according to the covering degree of the eyelashes and / or the depth of the iris color in the left eye dark pupil image.

[0028] Optionally, the step of tracking and positioning the left eye visual line according to the alternately collected left eye bright pupil image and left eye dark pupil image comprises:

[0029] dynamically identifying the left eye pupil center position and the left eye glint position according to the alternately collected left eye bright pupil image and left eye dark pupil image;

[0030] determining the relative position vector between the pupil center and the glint corresponding to the left eye according to the left eye pupil center position and the left eye glint position;

[0031] querying the left eye visual line direction mapped by the relative position vector from a preset left eye data mapping library to track and position the left eye visual line.

[0032] Optionally, the method further comprises:

[0033] displaying a preset calibration mark at a preset display position;

[0034] Control the working state of the left eye bright pupil tracking component and the left eye dark pupil tracking component, identify the left eye calibration vector between the left eye corresponding pupil center and the light spot when the user gazes at the preset calibration mark;

[0035] According to the left eye visual line direction corresponding to the preset display position gazed by the user, determine the left eye calibration visual line direction;

[0036] Establish the mapping relationship of the left eye calibration vector and the left eye calibration visual line direction, and calibrate the pre-calibrated left eye vector mapping library according to the mapping relationship of the left eye calibration vector and the left eye calibration visual line direction;

[0037] The calibrated left eye vector mapping library is used as the preset left eye vector mapping library.

[0038] To achieve the above-mentioned purposes, the application provides a head-mounted display device, which is applied to the visual line tracking method as described above, and the head-mounted display device comprises:

[0039] The left eye visual line tracking device comprises a left eye bright pupil tracking component and a left eye dark pupil tracking component, the left eye bright pupil tracking component comprises a first light source and a first image acquisition device belonging to the same optical path, the left eye dark pupil tracking component comprises a second light source and a second image acquisition device belonging to different optical paths, and the left eye dark pupil tracking component and the left eye bright pupil tracking component are set to be opened and operated at different times;

[0040] The right eye visual line tracking device comprises a right eye bright pupil tracking component and a right eye dark pupil tracking component, the right eye bright pupil tracking component comprises a third light source and a third image acquisition device belonging to the same optical path, the right eye dark pupil tracking component comprises a fourth light source and a fourth image acquisition device belonging to different optical paths, and the right eye dark pupil tracking component and the right eye bright pupil tracking component are set to be opened and operated at different times;

[0041] Among them, the left eye dark pupil tracking component and the right eye bright pupil tracking component are set to be opened and operated at the same acquisition cycle, and the left eye bright pupil tracking component and the right eye dark pupil tracking component are set to be opened and operated at the same acquisition cycle.

[0042] In addition, the application also provides a computer readable storage medium, the computer readable storage medium stores a visual line tracking program, the computer readable storage medium stores a visual line tracking program, and the visual line tracking program is executed by a processor to realize the steps of the visual line tracking method as described above.

[0043] The technical solution of the present application is to control the alternate operation between the left eye bright pupil tracking assembly and the left eye dark pupil tracking assembly to alternately collect left eye bright pupil images and left eye dark pupil images, track and locate the left eye visual line according to the alternately collected left eye bright pupil images and left eye dark pupil images, control the alternate operation between the right eye bright pupil tracking assembly and the right eye dark pupil tracking assembly to alternately collect right eye bright pupil images and right eye dark pupil images, and then track and locate the right eye visual line according to the alternately collected right eye bright pupil images and right eye dark pupil images, wherein the left eye bright pupil image and the right eye dark pupil image are collected in the same collection cycle, and the left eye dark pupil image and the right eye bright pupil image are collected in the same collection cycle. Thus, through bright pupil tracking and dark pupil tracking, the visual line tracking is realized in a separate and time-sharing manner, and the mutual interference factors of the left and right eye light sources in the tracking process are reduced.

[0044] At the same time, since there are many defects in the single tracking method, for example, in dark pupil tracking, the accuracy of recognition is easily affected by obstacles such as eyelashes, eye spots, and glasses frames, and in bright pupil tracking, the camera needs to recognize the bright pupil, and the accuracy of recognition is easily affected by factors such as pupil size, and the present application is based on the different tracking methods (bright pupil tracking and dark pupil tracking) of the alternate frames, and the visual line directions of the two eyes are tracked and located by comparing the front and back two frames of images. In this way, each eye includes a bright pupil image and a dark pupil image, so that the advantages of tracking and locating the visual line direction by the bright pupil image and the dark pupil image can be combined, the defects of the bright pupil image and the dark pupil image can be eliminated as much as possible, and the recognition accuracy is increased. At the same time, since the visual line tracking methods of the left and right eyes are different in a certain frame, the situation that the light sources of the two eyes interfere with each other can be greatly reduced, the accuracy of visual line tracking is further improved, and the problem of accuracy and easy interference of visual line tracking is solved. The present application proposes a solution from the aspects of hardware and software through dark pupil tracking and bright pupil tracking, further reduces the demand for chip computing power, and improves the recognition accuracy.

[0045] Compared with the dark pupil technology in the prior art, a set of light sources is respectively arranged in the left eye and right eye corresponding eye tracking modules (i.e. left eye dark pupil tracking assembly and right eye dark pupil tracking assembly), which leads to the problem that the light emitted by the two sets of light sources easily interferes with each other (especially for users wearing myopia glasses, which increases the error of the calculation result and affects the position accuracy of eye tracking), the left eye dark pupil tracking assembly and the right eye dark pupil tracking assembly can only light up one light source corresponding to each other in each specific frame, and the left eye bright pupil tracking assembly and the right eye bright pupil tracking assembly can also only light up one light source corresponding to each other in each specific frame, which reduces the mutual interference of the light sources of the two eyes, thereby reducing the line-of-sight positioning error and improving the tracking accuracy, providing a more perfect virtual reality immersion experience for the user, and further solving the technical problem that the light emitted by the two eye tracking modules of the head-mounted display device easily interferes with each other and affects the line-of-sight tracking accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained according to the structures shown in the drawings without creative labor.

[0047] Figure 1 Flowchart of the first embodiment of the line-of-sight tracking method of the present application;

[0048] Figure 2 Flowchart of the second embodiment of the line-of-sight tracking method of the present application;

[0049] Figure 3 Schematic diagram of a monocular bright pupil image in an embodiment of the present application;

[0050] Figure 4 Schematic diagram of a monocular dark pupil image in an embodiment of the present application;

[0051] Figure 5 Schematic diagram of a scene for collecting a monocular dark pupil image in an embodiment of the present application;

[0052] Figure 6 Schematic diagram of a scene for collecting a monocular bright pupil image in an embodiment of the present application;

[0053] Figure 7 Terminal structure schematic diagram of the hardware running environment involved in the embodiment of the present application.

[0054] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments in combination with the drawings. The above drawings have shown the specific embodiments of the present application, and will be described in more detail hereinafter. These drawings and the written description are not intended to limit the scope of the present application concept in any way, but to illustrate the present application concept for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0056] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly.

[0057] In addition, if the embodiments of the present application involve descriptions of “first”, “second”, etc., the descriptions of “first”, “second”, etc. are only for description purposes, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by “first”, “second” can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that a person skilled in the art can realize it, and when the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist, and is also not within the protection scope of the present application.

[0058] In the present embodiment, the head-mounted display device of the present application can be, for example, a Mixed Reality (MR) device (such as MR glasses or an MR helmet), an Augmented Reality (AR) device (such as AR glasses or an AR helmet), a Virtual Reality (VR) device (such as VR glasses or a VR helmet), an Extended Reality (XR) device (such as XR glasses or an XR helmet), or some combination thereof.

[0059] Currently, the mainstream technology for eye tracking in head-mounted displays is eye-tracking and gaze detection technology based on image processing. This technology calculates and records the position of the eyes in real time. Head-mounted displays mainly rely on dark pupil technology and use the corneal reflection point as a reference point to calculate the pupil-corneal reflection point vector to track the human eye's gaze. This has the following problems: Since each eye corresponds to an eye-tracking module, and the same light source is used in both eye-tracking modules, the light emitted by the light sources in the two eye-tracking modules will interfere with each other during calibration or use, which will increase the error of the calculation results and result in a low accuracy rate of tracking and recognizing the user's gaze.

[0060] Based on this, the present invention provides a gaze tracking method, please refer to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the gaze tracking method of the present invention. In this embodiment, the gaze tracking method is applied to a head-mounted display device, which includes a left eye bright pupil tracking component, a left eye dark pupil tracking component, a right eye bright pupil tracking component, and a right eye dark pupil tracking component. The method includes:

[0061] Step S10: Control the left eye bright pupil tracking component and the left eye dark pupil tracking component to operate alternately to alternately acquire left eye bright pupil images and left eye dark pupil images; track and locate the left eye gaze based on the alternately acquired left eye bright pupil images and left eye dark pupil images;

[0062] In this embodiment, the left-eye bright pupil tracking component is used to track and acquire the user's left-eye bright pupil image, and the left-eye dark pupil tracking component is used to track and acquire the user's left-eye dark pupil image. As those skilled in the art will know, the left-eye bright pupil image is the left-eye image formed when the optical axis of the light source is coaxial with the left-eye's line of sight. For example... Figure 3 As shown, Figure 3 This is a monocular bright pupil image according to an embodiment of this application, wherein the pupil 31 is bright, while the iris 32 is relatively dark. Figure 3 When the monocular bright pupil image is the left eye bright pupil image, 4 represents the light spot formed by the light source in the left eye bright pupil tracking component illuminating the left eye. Correspondingly, the left eye dark pupil image is the left eye image formed when the optical axis of the light source is not on the same axis as the left eye's line of sight. For example... Figure 4 As shown, Figure 4 This is a monocular dark pupil image according to an embodiment of this application. The pupil 31 is dark, while the iris 32 is relatively bright. Figure 4 When the monocular dark pupil image is the left eye dark pupil image, 4 is the light spot formed by the light source in the left eye dark pupil tracking component shining on the left eye.

[0063] like Figure 5 and Figure 6 As shown, Figure 5 This is a schematic diagram of a scene for acquiring monocular occult pupil images in one embodiment of this application.Figure 6 The schematic diagram of the scene for collecting the monocular bright pupil image in an embodiment of the present application is shown in FIG. 1. When the monocular bright pupil image of the user is the left eye bright pupil image, Figure 5 The monocular dark pupil image of the user is the left eye dark pupil image, Figure 6 When the monocular bright pupil image of the user is the left eye bright pupil image, "12" represents the left eye dark pupil tracking component, and "11" represents the left eye bright pupil tracking component. Specifically, the left eye bright pupil tracking component 11 includes a first light source 111 and a first image collecting device 112 belonging to the same optical path, so that the optical axis of the light source can be coaxial with the line of sight of the left eye when the left eye image is captured, to obtain the left eye bright pupil image. The left eye dark pupil tracking component 12 includes a second light source 121 and a second image collecting device 122 belonging to different optical paths, so that the optical axis of the light source can be in different axes with the line of sight of the left eye when the left eye image is captured, to obtain the left eye dark pupil image. The left eye dark pupil tracking component 11 and the left eye bright pupil tracking component 12 are set to operate in time division. In this embodiment, the left eye dark pupil tracking component 11 and the left eye bright pupil tracking component 12 can be controlled to operate in time division, so as to alternately collect the left eye bright pupil image and the left eye dark pupil image. For example, the first light source 111 is first controlled to start (the second light source 121 is in the off state), and then the first image collecting device 112 is controlled to start shooting, so as to obtain the left eye bright pupil image. Then, the first light source 111 is controlled to enter the off state, the second light source 121 is controlled to start, and then the second image collecting device 122 is controlled to start shooting, so as to obtain the left eye dark pupil image. It should be noted that the order of obtaining the left eye bright pupil image and the left eye dark pupil image is not limited in this embodiment, as long as they can be obtained at different times.

[0064] In an embodiment, according to the alternately collected left eye bright pupil image and left eye dark pupil image, the step of tracking and positioning the left eye line of sight can include: identifying the left eye pupil edge information of the user according to the alternately collected left eye bright pupil image and left eye dark pupil image, determining the left eye pupil center point (i.e., the left eye pupil center position) of the user according to the left eye pupil edge information, and determining the target point of the left eye observation according to the left eye pupil center point to obtain the left eye line of sight direction of the user. Specifically, the left eye line of sight direction of the user can be determined according to the preset matching relationship between the left eye pupil center point and the target point of the user observation. In this embodiment, the preset matching relationship can be determined by big data, without the need for training and learning or modeling of the user in advance. In addition, specifically, the left eye pupil edge information can be obtained by performing grayscale processing on the alternately collected left eye bright pupil image and left eye dark pupil image, obtaining the gradient value of the grayscale in the specified direction of the left eye bright pupil image and the left eye dark pupil image, and determining the position where the gradient value of the grayscale reaches the maximum value as the position of the left eye pupil edge.

[0065] In another embodiment, the line-of-sight estimation can be performed according to the pupil-corneal reflection method to track and locate the left eye line-of-sight, wherein the principle of the pupil-corneal reflection method is that a light source is illuminated to the eye, and the eye is photographed by the image acquisition device, and at the same time, the reflection point of the light source on the cornea, i.e., the light spot, is photographed, thereby obtaining the eye image with the light spot, and as the eyeball rotates, the relative positional relationship between the pupil center and the light spot changes. The plurality of eye images with the light spot acquired can reflect the positional change relationship, and the line-of-sight / gaze point is estimated according to the positional change relationship.

[0066] Step S20: alternately operating between the right eye bright pupil tracking assembly and the right eye dark pupil tracking assembly to alternately acquire the right eye bright pupil image and the right eye dark pupil image; and tracking and locating the right eye line-of-sight according to the alternately acquired right eye bright pupil image and right eye dark pupil image.

[0067] In the embodiment, the left eye bright pupil image and the right eye dark pupil image are acquired in the same acquisition cycle, and the left eye dark pupil image and the right eye bright pupil image are acquired in the same acquisition cycle.

[0068] In the embodiment, the right eye bright pupil tracking assembly is used to track and acquire the right eye bright pupil image of the user, and the right eye dark pupil tracking assembly is used to track and acquire the right eye dark pupil image of the user. As known by those skilled in the art, the right eye bright pupil image is the right eye image formed when the optical axis of the light source is coaxial with the right eye line-of-sight. Figure 3 As shown in FIG. 4, Figure 3 which is the single eye bright pupil image in an embodiment of the present application, wherein the pupil 31 is bright, and the iris 32 is relatively dark, and when Figure 3 the single eye bright pupil image is the right eye bright pupil image, 4 is the light spot formed by the light source in the right eye bright pupil tracking assembly. Figure 4 As shown in FIG. 5, Figure 4 which is the single eye dark pupil image in an embodiment of the present application, wherein the pupil 31 is dark, and the iris 32 is relatively bright, and when Figure 4 the single eye dark pupil image is the right eye dark pupil image, 4 is the light spot formed by the light source in the right eye dark pupil tracking assembly.

[0069] As shown in FIG. 6, Figure 5 and Figure 6 As shown in FIG. 7, Figure 5 which is a schematic diagram of the scene for acquiring the single eye dark pupil image in an embodiment of the present application, Figure 6 which is a schematic diagram of the scene for acquiring the single eye bright pupil image in an embodiment of the present application. Correspondingly, when Figure 5 the single eye dark pupil image is the right eye dark pupil image, Figure 6When the monocular bright-pupil image of the right eye is the right eye bright-pupil image, "12" represents the right eye dark-pupil tracking component, and "11" represents the right eye bright-pupil tracking component. Specifically, the right eye bright-pupil tracking component 11 includes a third light source 111 and a third image acquisition device 112 that belong to the same optical path, so that the optical axis of the light source can be coaxial with the line of sight of the right eye when the right eye image is captured, to capture the right eye bright-pupil image. The right eye dark-pupil tracking component 12 includes a fourth light source 121 and a fourth image acquisition device 122 that belong to different optical paths, so that the optical axis of the light source can be different from the line of sight of the left eye when the left eye image is captured, to capture the right eye dark-pupil image. The right eye dark-pupil tracking component 11 and the right eye bright-pupil tracking component 12 are set to operate in time division. In this embodiment, the right eye dark-pupil tracking component 11 and the right eye bright-pupil tracking component 12 can be controlled to operate in time division, so as to alternately acquire the right eye bright-pupil image and the right eye dark-pupil image. For example, the third light source 111 is first controlled to start (the fourth light source 121 is in a closed state), and then the third image acquisition device 112 is controlled to start photographing, so as to obtain the right eye bright-pupil image. Then, the third light source 111 is controlled to enter a closed state, the fourth light source 121 is controlled to start, and then the fourth image acquisition device 122 is controlled to start photographing, so as to obtain the right eye dark-pupil image. It should be noted that the order of obtaining the right eye bright-pupil image and the right eye dark-pupil image is not limited in this embodiment, as long as they can be obtained at different times.

[0070] In an embodiment, according to the alternately acquired right eye bright-pupil image and right eye dark-pupil image, the step of tracking and positioning the line of sight of the right eye can include: identifying the right eye pupil edge information of the user according to the alternately acquired right eye bright-pupil image and right eye dark-pupil image, determining the right eye pupil center point (i.e., the right eye pupil center position) of the user according to the right eye pupil edge information, and determining the target point of the right eye observation according to the right eye pupil center point to obtain the direction of the line of sight of the right eye of the user. Specifically, the direction of the line of sight of the right eye of the user can be determined according to the right eye pupil center point and a preset matching relationship between the user and the target point of observation. In this embodiment, the preset matching relationship can be determined by big data, without the need for training and learning or modeling of the user in advance. In addition, specifically, the alternately acquired right eye bright-pupil image and right eye dark-pupil image can be subjected to grayscale processing to obtain the gradient value of the grayscale in a specified direction of the right eye bright-pupil image and the right eye dark-pupil image, and the position where the gradient value of the grayscale reaches a maximum value is determined as the position of the right eye pupil edge to obtain the right eye pupil edge information.

[0071] In another embodiment, the line-of-sight estimation can be performed according to the pupil-corneal reflection method, to track and locate the right eye line-of-sight, wherein the principle of the pupil-corneal reflection method is that a light source is directed to the eye, and the eye is photographed by the image acquisition device, and at the same time, the reflection point of the light source on the cornea, i.e. the light spot, is photographed, thereby obtaining an eye image with the light spot, and as the eyeball rotates, the relative positional relationship between the pupil center and the light spot changes. Multiple eye images with light spots can reflect the positional change relationship, and the line-of-sight / gaze point is estimated according to the positional change relationship.

[0072] It should be noted that in the present embodiment, the left eye bright pupil image and the right eye dark pupil image are acquired in the same acquisition cycle, and the left eye dark pupil image and the right eye bright pupil image are acquired in the same acquisition cycle. That is, in the process of alternately running the left eye bright pupil tracking assembly and the left eye dark pupil tracking assembly, and alternately running the right eye bright pupil tracking assembly and the right eye dark pupil tracking assembly, the left eye bright pupil tracking assembly and the right eye dark pupil tracking assembly run in the same running cycle, and the left eye dark pupil tracking assembly and the right eye bright pupil tracking assembly run in the same running cycle. For example, in the 6 running cycles T, 2T, 3T, 4T, 5T and 6T running alternately in turn, the running cycles of the left eye bright pupil tracking assembly are T, 3T and 5T respectively, and the running cycles of the left eye dark pupil tracking assembly are 2T, 4T and 6T respectively. At this time, the running cycles of the right eye dark pupil tracking assembly are T, 3T and 5T respectively, and the running cycles of the right eye bright pupil tracking assembly are 2T, 4T and 6T respectively.

[0073] To facilitate understanding, a table example is used for illustration. In an example, the images of the eyes of the user are continuously acquired by controlling the working states of the left eye bright pupil tracking assembly, the left eye dark pupil tracking assembly, the right eye bright pupil tracking assembly and the right eye dark pupil tracking assembly, wherein in the odd frames, the left eye is tracked for bright pupil and the right eye is tracked for dark pupil; in the even frames, the left eye is tracked for dark pupil and the right eye is tracked for bright pupil. The left eye bright pupil image and the right eye dark pupil image are obtained in the odd frames; the left eye dark pupil image and the right eye bright pupil image are obtained in the even frames, so that the bright pupil image and the dark pupil image of each eye are obtained in the two frames. The line-of-sight directions of the eyes are determined by comparing the two frames of images.

[0074] Of course, in another example, the right eye can also be tracked with a bright pupil in odd frames, and the left eye can be tracked with a dark pupil in odd frames; the right eye can be tracked with a dark pupil in even frames, and the left eye can be tracked with a bright pupil in even frames. In odd frames, a right eye bright pupil image and a left eye dark pupil image are obtained; in even frames, a right eye dark pupil image and a left eye bright pupil image are obtained, so that each eye in the two frames obtains a bright pupil image and a dark pupil image. By comparing the two frames of images, the gaze direction of the two eyes is determined, so that each eye includes a bright pupil image and a dark pupil image, thereby the advantages of fusing the bright pupil image and the dark pupil image to track and position the gaze direction can be achieved, and the defects of tracking and positioning the gaze direction by the bright pupil image and the dark pupil image respectively can be eliminated (for example, for the bright pupil image, the shielding of the eyelashes and the eye spot has less effect on the gaze tracking effect, and the pupil size has greater effect on the gaze tracking effect, and the bright pupil image has better tracking effect on blue or light-colored irises. For the dark pupil image, the shielding of the eyelashes and the eye spot has greater effect on the gaze tracking effect, and the pupil size has less effect on the gaze tracking effect, and the dark pupil image has better tracking effect on dark-colored irises), excluding the effects of some interference factors, including eyelashes, black spots, too small pupil, and the lightness and darkness of the iris color, and completing pupil recognition through an algorithm, thereby increasing the accuracy of recognition. At the same time, since the gaze tracking methods of the left and right eyes are different in a frame, the situation of mutual interference of light sources of the two eyes can be greatly reduced, and the accuracy of gaze tracking is further improved.

[0075] Further, the embodiment can determine the gaze focus of the user's two eyes by tracking and positioning the left eye gaze and the right eye gaze, thereby realizing gaze positioning.

[0076] At present, the gaze tracking module in the head-mounted display device such as virtual reality products selects a single tracking method, bright pupil tracking or dark pupil tracking. The two tracking methods have advantages and disadvantages, are affected by different interference factors, and result in low accuracy of gaze recognition, which affects the judgment of the subsequent gaze point estimation position. Moreover, the currently commonly used dark pupil tracking method has the situation of mutual interference of light of the left and right eyes.

[0077] Based on this, the technical scheme of the embodiment is to control the left eye bright pupil tracking component and the left eye dark pupil tracking component to run alternately to alternately collect left eye bright pupil images and left eye dark pupil images, track and locate the left eye visual line according to the alternately collected left eye bright pupil images and left eye dark pupil images, control the right eye bright pupil tracking component and the right eye dark pupil tracking component to run alternately to alternately collect right eye bright pupil images and right eye dark pupil images, and then track and locate the right eye visual line according to the alternately collected right eye bright pupil images and right eye dark pupil images, wherein the left eye bright pupil image and the right eye dark pupil image are collected in the same collection cycle, and the left eye dark pupil image and the right eye bright pupil image are collected in the same collection cycle, so as to realize visual line tracking in a divided method and at different times through bright pupil tracking and dark pupil tracking, and reduce the factors of mutual interference of left and right eye light sources in the tracking process.

[0078] At the same time, since there are many defects in the single tracking method, for example, in dark pupil tracking, the accuracy of recognition is easily affected by obstacles such as eyelashes, eye spots, and glasses frames, and in bright pupil tracking, the camera needs to recognize the bright pupil, and the accuracy of recognition is easily affected by factors such as pupil size, and the embodiment is based on the different tracking methods (bright pupil tracking and dark pupil tracking) of the alternating frames, and the visual line directions of the two eyes are tracked and located by comparing the front and back two frames of images, so that each eye includes a bright pupil image and a dark pupil image, so that the advantages of tracking and locating the visual line direction by the bright pupil image and the dark pupil image can be fused, the defects of the bright pupil image and the dark pupil image can be eliminated as much as possible, and the recognition accuracy is increased. At the same time, since the visual line tracking methods of the left and right eyes are different in a certain frame, the situation that the light sources of the two eyes interfere with each other can be greatly reduced, the accuracy of visual line tracking is further improved, and the problem of accuracy and easy interference of visual line tracking is solved. The embodiment proposes a solution from the aspects of hardware and software through dark pupil tracking and bright pupil tracking, further reduces the demand for chip computing power, and improves the recognition accuracy.

[0079] Compared with the dark pupil technology in the prior art, each of the left eye and the right eye corresponds to an eye tracking module (i.e., a left eye dark pupil tracking component and a right eye dark pupil tracking component), and each of the left eye and the right eye has a set of light sources, which leads to the problem that the light emitted by the two sets of light sources easily interferes with each other (especially for users wearing myopia glasses, which increases the error of the calculation result and affects the position accuracy of eye tracking). In the embodiment, in each specific frame, only one light source corresponding to the left eye dark pupil tracking component and the right eye dark pupil tracking component can be lit, and in each specific frame, only one light source corresponding to the left eye bright pupil tracking component and the right eye bright pupil tracking component can be lit, thereby reducing the mutual interference of the light sources of the two eyes, reducing the line-of-sight positioning error, improving the tracking accuracy, providing a more perfect virtual reality immersion experience for the user, and solving the technical problem that the light emitted by the two eye tracking modules of the head-mounted display device easily interferes with each other and affects the line-of-sight tracking accuracy.

[0080] In a possible implementation, the step of tracking and positioning the left eye line of sight according to the alternately acquired left eye bright pupil image and left eye dark pupil image comprises the following steps:

[0081] Step A10, identifying the eye features of the left eye bright pupil image and the left eye dark pupil image acquired in the adjacent acquisition period to obtain the left eye bright pupil eye features and the left eye dark pupil eye features, wherein the eye features include at least one of the pupil center position, the pupil shape, the iris position, the iris shape and the light spot position.

[0082] As known by those skilled in the art, the light spot position refers to the position of the light spot formed by the light source of the left eye bright pupil tracking component or the left eye dark pupil tracking component on the left eye.

[0083] Step A20, performing feature matching on the left eye bright pupil eye features and the left eye dark pupil eye features to obtain a feature matching degree.

[0084] As easily understood, the left eye bright pupil eye features refer to the eye features identified by image recognition of the left eye bright pupil image, and the left eye dark pupil eye features refer to the eye features identified by image recognition of the left eye dark pupil image.

[0085] In the embodiment, the left eye bright-pupil eyeball features and the left eye dark-pupil eyeball features are matched, specifically, the eyeball features recognized by image recognition on the left eye bright-pupil image are matched with the eyeball features recognized by image recognition on the left eye dark-pupil image, to obtain a feature matching degree. The eyeball features include at least one of a pupil center position, a pupil shape, an iris position, an iris shape and a glint position. For example, in one example, the eyeball features are the pupil center position, the pupil shape and the iris position. In another example, the eyeball features are the pupil center position, the iris position and the iris shape. In yet another example, the eyeball features are the pupil center position and the glint position. The embodiment is not specifically limited here.

[0086] To facilitate understanding of the embodiment, an example is cited in which the eyeball features are the pupil center position and the glint position. At this time, the step of matching the left eye bright-pupil eyeball features and the left eye dark-pupil eyeball features to obtain a feature matching degree can specifically be: constructing a first relative position vector between the first pupil center position and the first glint position according to the first pupil center position and the first glint position recognized by image recognition on the left eye bright-pupil image; constructing a second relative position vector between the second pupil center position and the second glint position according to the second pupil center position and the second glint position recognized by image recognition on the left eye dark-pupil image, determining a consistency degree of the first relative position vector and the second relative position vector, and taking the consistency degree as the feature matching degree. It is easy to understand that according to the knowledge of eyeball optics, the pupil of the human eye is consistent with the line of sight direction, and if the line of sight direction of the human eye is directly opposite to the light source (for example, infrared light), then the position of the glint in the human eye image collected by the human eye should be closer to the position of the pupil. Specifically, the vector lengths of the first relative position vector and the second relative position vector can be compared respectively, and if the vector lengths are closer, it means that the feature matching degree is higher. It should be understood that the extraction of the position information of the glint and the pupil from the eye region image and the calculation of the vector lengths of two points can be realized by common technical means in the technical field, and the embodiment is not specifically limited.

[0087] In step A30, if the feature matching degree is greater than a preset degree threshold, the step of tracking and positioning the left eye line of sight according to the alternately collected left eye bright-pupil image and left eye dark-pupil image is performed.

[0088] In one possible implementation, after the step of matching the left eye bright-pupil eyeball features and the left eye dark-pupil eyeball features to obtain a feature matching degree, the method further includes:

[0089] Step B10, if the feature matching degree is less than or equal to a preset degree threshold, performing image recognition on the left eye bright pupil image to determine a bright pupil collection interference degree corresponding to the left eye bright pupil image, and performing image recognition on the left eye dark pupil image to determine a dark pupil collection interference degree corresponding to the left eye dark pupil image;

[0090] Step B20, if the bright pupil collection interference degree is greater than the dark pupil collection interference degree first preset degree value, correcting the left eye bright pupil image through the left eye dark pupil image; tracking and positioning the left eye visual line according to the left eye dark pupil image and the corrected left eye bright pupil image.

[0091] Step B30, if the bright pupil collection interference degree is less than the dark pupil collection interference degree second preset degree value, correcting the left eye dark pupil image through the left eye bright pupil image; tracking and positioning the left eye visual line according to the left eye bright pupil image and the corrected left eye dark pupil image.

[0092] In the embodiment, the preset degree threshold can be set by those skilled in the art according to actual conditions to better determine whether the collection difficulty of the left eye bright pupil image or the left eye dark pupil image is high, resulting in large deviation of the left eye bright pupil image or the left eye dark pupil image.

[0093] In the embodiment, the first preset degree value and the second preset degree value can be the same or different, and the embodiment is not specifically limited.

[0094] It should be noted that the feature matching degree refers to the matching degree of the eye features corresponding to the left eye bright pupil image and the left eye dark pupil image collected in adjacent collection periods, that is, the left eye bright pupil image and the left eye dark pupil image collected in two consecutive frames. Since the time interval between the two consecutive frames is extremely short and can be ignored, the left eye remains unchanged or changes slightly, so the eye features of the left eye bright pupil image and the left eye dark pupil image collected in two consecutive frames are almost the same. However, in some cases, for example, during dark pupil tracking, the left eye dark pupil image is affected by objects such as eyelashes, eye spots, and glasses frames, resulting in inaccurate collection of the left eye dark pupil image. Alternatively, during bright pupil tracking, the left eye bright pupil image is affected by factors such as pupil size, resulting in inaccurate collection of the left eye bright pupil image, and thus the feature matching degree of the left eye bright pupil image and the left eye dark pupil image is less than or equal to the preset degree threshold.

[0095] Therefore, the embodiment determines the bright pupil image interference degree corresponding to the left eye bright pupil image by performing image recognition on the left eye bright pupil image after determining that the feature matching degree is less than or equal to the preset degree threshold, and determines the dark pupil image interference degree corresponding to the left eye dark pupil image by performing image recognition on the left eye dark pupil image, and if the bright pupil image interference degree is greater than the dark pupil image interference degree first preset degree value, the left eye dark pupil image is corrected by the left eye bright pupil image; the left eye visual line is tracked and positioned according to the left eye dark pupil image and the corrected left eye bright pupil image; if the bright pupil image interference degree is less than the dark pupil image interference degree second preset degree value, the left eye bright pupil image is corrected by the left eye dark pupil image; the left eye visual line is tracked and positioned according to the left eye bright pupil image and the corrected left eye dark pupil image, so that when the left eye dark pupil image is affected by the object such as eyelashes, eye spots, glasses frames and the like, the collected left eye dark pupil image is not accurate, the left eye dark pupil image is calibrated by the left eye bright pupil image, and when the left eye bright pupil image is affected by factors such as pupil size or iris color depth, the collected left eye bright pupil image is not accurate, the left eye bright pupil image is calibrated by the left eye dark pupil image, so that the advantages of tracking and positioning the visual line direction by the bright pupil image and the dark pupil image are fused, the defects of the bright pupil image and the dark pupil image are eliminated as much as possible, and the accuracy of tracking the visual line of the user's eyes is further increased.

[0096] Correspondingly, in a possible implementation, the step of tracking and positioning the right eye visual line according to the alternately collected right eye bright pupil image and right eye dark pupil image comprises the following steps:

[0097] Step C10, identifying the eyeball features of the right eye bright pupil image and the right eye dark pupil image collected in the adjacent collection period to obtain the right eye bright pupil eyeball features and the right eye dark pupil eyeball features, wherein the eyeball features include at least one of the pupil center position, the pupil shape, the iris position, the iris shape and the light spot position;

[0098] Step C20, performing feature matching on the right eye bright pupil eyeball features and the right eye dark pupil eyeball features to obtain the feature similarity degree.

[0099] Step C30, if the feature similarity degree is greater than the preset degree threshold, performing the step of tracking and positioning the right eye visual line according to the alternately collected right eye bright pupil image and right eye dark pupil image.

[0100] In a possible implementation, after the step of performing feature matching on the right eye bright pupil eyeball features and the right eye dark pupil eyeball features to obtain the feature similarity degree, the following steps are further included:

[0101] Step D10, if the feature similarity degree is less than or equal to a preset degree threshold, performing image recognition on the right eye bright pupil image to determine a bright pupil shooting interference degree corresponding to the right eye bright pupil image, and performing image recognition on the right eye dark pupil image to determine a dark pupil shooting interference degree corresponding to the right eye dark pupil image;

[0102] Step D20, if the bright pupil shooting interference degree is greater than the dark pupil shooting interference degree by a third preset degree value, correcting the right eye bright pupil image by the right eye dark pupil image; and tracking and positioning the right eye visual line according to the right eye dark pupil image and the corrected right eye bright pupil image.

[0103] Step D30, if the bright pupil shooting interference degree is less than the dark pupil shooting interference degree by a fourth preset degree value, correcting the right eye dark pupil image by the right eye bright pupil image; and tracking and positioning the right eye visual line according to the right eye bright pupil image and the corrected right eye dark pupil image.

[0104] In the embodiment, the third preset degree value and the fourth preset degree value can be the same or different, and the embodiment is not limited specifically.

[0105] The technical principle or technical effect of the embodiment corresponds to the previous embodiment, and details are not repeated here. The difference between the two embodiments is that the previous embodiment corresponds to the left eye of the user, and the embodiment corresponds to the right eye of the user.

[0106] In a possible implementation, the step of correcting the left eye bright pupil image by the left eye dark pupil image in the embodiment includes:

[0107] Step E10, obtaining a left eye dark pupil eye feature corresponding to the left eye dark pupil image and a left eye bright pupil eye feature corresponding to the left eye bright pupil image;

[0108] Step E20, calculating a sum value of the left eye dark pupil eye feature with a first preset weight and the left eye bright pupil eye feature with a second preset weight to obtain a fused eye feature, wherein the first preset weight is greater than the second preset weight.

[0109] The first preset weight is greater than the second preset weight, and the sum of the first preset weight and the second preset weight is equal to one.

[0110] In an example, the first preset weight is 0.7, and the second preset weight is 0.3. In another example, the first preset weight is 0.8, and the second preset weight is 0.2. In another example, the first preset weight is 0.9, and the second preset weight is 0.1. In yet another example, the first preset weight is 1, and the second preset weight is 0.

[0111] Step E30: The fused pupil image associated with the eyeball features is used as the corrected left eye pupil.

[0112] image.

[0113] This embodiment obtains the left eye's dark pupil features corresponding to the left eye's dark pupil image and the left eye's bright pupil features corresponding to the left eye's bright pupil image, and then combines the left eye's dark pupil features with the first preset weighted left eye's dark pupil features with the second preset weighted left eye's bright pupil features.

[0114] The left eye's bright pupil features with two preset weights are summed to obtain fused eye features. Here, the first preset weight is greater than the second preset weight, and the sum of the first preset weight and the second preset weight is equal to one. This allows the eye features picked from the less disturbed frame to correct the eye features from the more disturbed frame, thereby eliminating the defect of the bright pupil image and the dark pupil image tracking and positioning the gaze direction separately, and improving the accuracy of binocular gaze positioning.

[0115] 0 In one possible implementation, in this embodiment, the step of correcting the right eye bright pupil image using the right eye dark pupil image includes:

[0116] Step F10: Obtain the right eye pupil features corresponding to the right eye pupil image and the right eye bright pupil features corresponding to the right eye bright pupil image;

[0117] Step F20: The right eye dark pupil feature with the first preset weight and the right eye bright pupil feature with the second preset weight are summed to obtain the combined eye feature, wherein the first preset weight is greater than the second preset weight, and the sum of the first preset weight and the second preset weight is equal to one.

[0118] Step F30: The bright pupil image associated with the eyeball features is used as the corrected bright pupil image of the right eye.

[0119] The technical principles or effects of this embodiment correspond to those of the previous embodiment, and will not be repeated here. The difference between the two is that the previous embodiment corresponds to the user's left eye, while this embodiment corresponds to the user's right eye.

[0120] In one possible implementation, in this embodiment, the step of correcting the left eye dark pupil image using the left eye bright pupil image includes:

[0121] Step G10: Obtain the left eye pupil features corresponding to the left eye dark pupil image and the left eye bright pupil features corresponding to the left eye bright pupil image;

[0122] Step G20, performing sum value calculation on the left eye bright pupil eye ball feature of the third preset weight and the left eye dark pupil eye ball feature of the fourth preset weight to obtain a comprehensive eye ball feature, wherein the third preset weight is greater than the fourth preset weight, and the sum of the third preset weight and the fourth preset weight is equal to one;

[0123] In an example, the third preset weight is 0.7, and the fourth preset weight is 0.3. In another example, the third preset weight is 0.8, and the fourth preset weight is 0.2. In another example, the third preset weight is 0.9, and the fourth preset weight is 0.1. In yet another example, the third preset weight is 1, and the fourth preset weight is 0.

[0124] Step G30, taking the bright pupil image associated with the comprehensive eye ball feature as the corrected left eye dark pupil image.

[0125] The embodiment obtains the left eye dark pupil eye ball feature corresponding to the left eye dark pupil image and the left eye bright pupil eye ball feature corresponding to the left eye bright pupil image, performs sum value calculation on the left eye bright pupil eye ball feature of the third preset weight and the left eye dark pupil eye ball feature of the fourth preset weight to obtain a comprehensive eye ball feature, wherein the third preset weight is greater than the fourth preset weight, and the sum of the third preset weight and the fourth preset weight is equal to one, so as to realize the purpose of correcting the eye ball feature of the image frame with less interference by taking the eye ball feature of the image frame with less interference, eliminating the defects of the bright pupil image and the dark pupil image in tracking and positioning the line of sight direction, and improving the accuracy of the line of sight positioning of the two eyes.

[0126] In a possible implementation, in the embodiment, the step of correcting the right eye dark pupil image by using the right eye bright pupil image comprises:

[0127] Step H10, obtaining the right eye dark pupil eye ball feature corresponding to the right eye dark pupil image and the right eye bright pupil eye ball feature corresponding to the right eye bright pupil image;

[0128] Step H20, performing sum value calculation on the right eye bright pupil eye ball feature of the third preset weight and the right eye dark pupil eye ball feature of the fourth preset weight to obtain a joint eye ball feature, wherein the third preset weight is greater than the fourth preset weight, and the sum of the third preset weight and the fourth preset weight is equal to one;

[0129] Step H30, taking the bright pupil image associated with the joint eye ball feature as the corrected right eye dark pupil image.

[0130] The technical principle or technical effect of the embodiment corresponds to the previous embodiment, and will not be described here. The difference between the two is that the previous embodiment corresponds to the left eye of the user, while the present embodiment corresponds to the right eye of the user.

[0131] In a possible implementation, the step of performing image recognition on the left-eye bright-pupil image to determine the interference degree of bright-pupil acquisition corresponding to the left-eye bright-pupil image comprises:

[0132] Step I10, performing image recognition on the left-eye bright-pupil image to determine the size of the pupil and / or the depth of the iris color in the left-eye bright-pupil image;

[0133] Step I20, determining the interference degree of bright-pupil acquisition corresponding to the left-eye bright-pupil image according to the size of the pupil and / or the depth of the iris color in the left-eye bright-pupil image.

[0134] Wherein, the smaller the pupil and / or the deeper the iris color, the greater the interference degree of bright-pupil acquisition.

[0135] The embodiment determines the interference degree of bright-pupil acquisition corresponding to the left-eye bright-pupil image according to the size of the pupil and / or the depth of the iris color in the left-eye bright-pupil image, wherein the smaller the pupil and / or the deeper the iris color, the greater the interference degree of bright-pupil acquisition, thereby more accurately identifying the interference degree of bright-pupil acquisition corresponding to the left-eye bright-pupil image.

[0136] Correspondingly, in a possible implementation, the step of performing image recognition on the right-eye bright-pupil image to determine the interference degree of bright-pupil shooting corresponding to the right-eye bright-pupil image comprises:

[0137] Step J10, performing image recognition on the right-eye bright-pupil image to determine the size of the pupil and / or the depth of the iris color in the right-eye bright-pupil image;

[0138] Step J20, determining the interference degree of bright-pupil shooting corresponding to the right-eye bright-pupil image according to the size of the pupil and / or the depth of the iris color in the right-eye bright-pupil image.

[0139] Wherein, the smaller the pupil and / or the deeper the iris color, the greater the interference degree of bright-pupil shooting.

[0140] In a possible implementation, the step of performing image recognition on the left-eye dark-pupil image to determine the interference degree of dark-pupil acquisition corresponding to the left-eye dark-pupil image comprises:

[0141] Step K10, performing image recognition on the left-eye dark-pupil image to determine the covering degree of eyelashes and / or the depth of the iris color in the left-eye dark-pupil image;

[0142] Step K20, determining the interference degree of dark-pupil acquisition corresponding to the left-eye dark-pupil image according to the covering degree of eyelashes and / or the depth of the iris color in the left-eye dark-pupil image.

[0143] Wherein, the greater the covering degree of eyelashes and / or the shallower the iris color, the greater the interference degree of dark-pupil acquisition.

[0144] The embodiment determines the interference degree of dark pupil acquisition corresponding to the left eye dark pupil image according to the covering degree of the eyelashes and / or the depth of the iris color in the left eye dark pupil image, wherein the greater the covering degree of the eyelashes and / or the lighter the iris color, the greater the interference degree of dark pupil acquisition, so that the interference degree of dark pupil acquisition corresponding to the left eye dark pupil image is more accurately identified.

[0145] Correspondingly, in a possible implementation, the step of performing image recognition on the right eye dark pupil image to determine the interference degree of dark pupil acquisition corresponding to the right eye dark pupil image comprises:

[0146] Step L10, performing image recognition on the right eye dark pupil image to determine the covering degree of the eyelashes and / or the depth of the iris color in the right eye dark pupil image;

[0147] Step L20, determining the interference degree of dark pupil acquisition corresponding to the right eye dark pupil image according to the covering degree of the eyelashes and / or the depth of the iris color in the right eye dark pupil image.

[0148] Wherein the greater the covering degree of the eyelashes and / or the lighter the iris color, the greater the interference degree of dark pupil acquisition.

[0149] Based on the first embodiment of the present application, the present application provides a second embodiment of a line-of-sight tracking method, please refer to Figure 2 In the embodiment, the step of tracking and positioning the left eye line-of-sight according to the alternately collected left eye bright pupil image and left eye dark pupil image comprises:

[0150] Step S11, dynamically identifying the left eye pupil center position and the left eye glint position according to the alternately collected left eye bright pupil image and left eye dark pupil image;

[0151] Step S12, determining the relative position vector between the pupil center and the glint corresponding to the left eye according to the left eye pupil center position and the left eye glint position;

[0152] Step S13, querying the left eye line-of-sight direction mapped by the relative position vector from a preset left eye data mapping library to track and position the left eye line-of-sight.

[0153] In the embodiment, the left eye data mapping library stores a plurality of vector parameters and a mapping relationship that each vector parameter is one-to-one mapped with a left eye line-of-sight direction, so that the left eye line-of-sight direction mapped by the corresponding vector parameter of the relative position vector can be queried from the left eye data mapping library. It should be noted that the left eye data mapping library can be obtained by experimental calibration by those skilled in the art before the head-mounted display device is shipped, and is pre-stored in the system of the head-mounted display device.

[0154] The embodiment dynamically identifies the left eye pupil center position and the left eye glint position according to the alternately collected left eye bright pupil image and left eye dark pupil image, determines the relative position vector between the corresponding pupil center and glint of the left eye according to the left eye pupil center position and the left eye glint position

[0155] The embodiment dynamically identifies the left eye pupil center position and the left eye glint position according to the alternately collected left eye bright pupil image and left eye dark pupil image, determines the relative position vector between the corresponding pupil center and glint of the left eye according to the left eye pupil center position and the left eye glint position

[0156] The embodiment dynamically identifies the left eye pupil center position and the left eye glint position according to the alternately collected left eye bright pupil image and left eye dark pupil image, determines the relative position vector between the corresponding pupil center and glint of the left eye according to the left eye pupil center position and the left eye glint position

[0157] The embodiment dynamically identifies the left eye pupil center position and the left eye glint position according to the alternately collected left eye bright pupil image and left eye dark pupil image, determines the relative position vector between the corresponding pupil center and glint of the left eye according to the left eye pupil center position and the left eye glint position

[0158] The embodiment dynamically identifies the left eye pupil center position and the left eye glint position according to the alternately collected left eye bright pupil image and left eye dark pupil image, determines the relative position vector between the corresponding pupil center and glint of the left eye according to the left eye pupil center position and the left eye glint position

[0159] The embodiment dynamically identifies the left eye pupil center position and the left eye glint position according to the alternately collected left eye bright pupil image and left eye dark pupil image, determines the relative position vector between the corresponding pupil center and glint of the left eye according to the left eye pupil center position and the left eye glint position

[0160] In the embodiment, the right eye data mapping library stores a plurality of vector parameters and a mapping relationship in which each vector parameter is mapped with a right eye line direction, so that the right eye line direction mapped by the corresponding vector parameter of the relative spatial vector can be obtained by querying the right eye data mapping library.

[0161] It should be noted that the right eye data mapping library can be obtained by experimental calibration by those skilled in the art before the head-mounted display device is manufactured and pre-stored in the system of the head-mounted display device.

[0162] The technical principle or technical effect of the embodiment corresponds to that of the previous embodiment, and will not be described here. The difference between the two embodiments is that the previous embodiment corresponds to the left eye of the user, while the embodiment corresponds to the right eye of the user.

[0163] In a possible implementation, the method further includes:

[0164] Step M10: displaying a preset calibration mark at a preset display position;

[0165] Step M20: controlling the working state of the left eye bright pupil tracking component and the left eye dark pupil tracking component to identify the left eye relative position vector between the corresponding pupil center and glint when the user gazes at the preset calibration mark.

[0166] eye calibration vector;

[0167] 0In this embodiment, the working state includes an open running state and a closed state. The working state of the left eye bright pupil tracking component and the left eye dark pupil tracking component is controlled, specifically, the left eye bright pupil tracking component and the left eye dark pupil tracking component can be controlled to run alternately.

[0168] Step M30, determining a left eye calibration line-of-sight direction according to the left eye line-of-sight direction corresponding to the preset display position at which the user gazes;

[0169] Step M40, establishing a mapping relationship between the left eye calibration vector and the left eye calibration line-of-sight direction, and calibrating a pre-calibrated left eye vector mapping library according to the mapping relationship between the left eye calibration vector and the left eye calibration line-of-sight direction;

[0170] Step M50, taking the calibrated left eye vector mapping library as a preset left eye vector mapping library.

[0171] Due to physiological differences of individual eyeballs, before using the left eye data mapping library, the embodiment can first calibrate the left eye data mapping library for a user. User calibration, also known as user calibration, in the line-of-sight / gaze point estimation process, in order to calibrate the parameters (i.e., the mapping relationship between each vector parameter and the left eye line-of-sight direction) in the left eye data mapping library for line-of-sight / gaze point estimation, a commonly used method is: let the user gaze at one or more preset calibration marks, assuming that the information of the preset calibration mark is a known line-of-sight (because the display position of the preset calibration mark is pre-set), from which the above calibration parameters can be inversely solved (i.e., test the mapping relationship between the vector parameter that matches the user's eyeball better and the left eye line-of-sight direction, and correct the mapping relationship between the vector parameter and the left eye line-of-sight direction).

[0172] The display position, number and shape of the preset calibration mark are not limited, and a person skilled in the art can set them according to the actual situation, such as uniformly distributing the positions of the preset calibration marks on the display, distributing the preset calibration marks at the center and boundary positions of the display, or displaying the calibration points in a nine-square grid form on the display. The shape of the preset calibration mark can be circular. By acquiring the image of the user gazing at the preset calibration mark and the position information of the preset calibration mark, the mapping relationship between the left eye calibration vector and the left eye calibration line-of-sight direction is established, and the pre-calibrated left eye vector mapping library is calibrated according to the mapping relationship between the left eye calibration vector and the left eye calibration line-of-sight direction, thereby obtaining a left eye data mapping library that matches the real situation of the user's left eye better.

[0173] Correspondingly, in one possible implementation, the method further includes:

[0174] Step O10, displaying a preset calibration mark at a preset display position;

[0175] Step O20, controlling the working states of the right eye bright pupil tracking component and the right eye dark pupil tracking component, and identifying a right eye calibration vector between the corresponding pupil center and the light spot of the right eye when the user gazes at the preset calibration mark;

[0176] In this embodiment, the working states of the right eye bright pupil tracking component and the right eye dark pupil tracking component can be controlled, and specifically, the right eye bright pupil tracking component and the right eye dark pupil tracking component can be controlled to alternately operate.

[0177] Step O30, determining a right eye calibration line-of-sight direction according to a right eye line-of-sight direction corresponding to the preset display position when the user gazes.

[0178] Step O40, establishing a mapping relationship between the right eye calibration vector and the right eye calibration line-of-sight direction, and calibrating a pre-calibrated right eye vector mapping library according to the mapping relationship between the right eye calibration vector and the right eye calibration line-of-sight direction.

[0179] Step O50, taking the calibrated right eye vector mapping library as a preset right eye vector mapping library.

[0180] The technical principle or technical effect of this embodiment corresponds to that of the previous embodiment, and will not be described here again. The difference between the two embodiments is that the previous embodiment corresponds to the left eye of the user, while this embodiment corresponds to the right eye of the user.

[0181] In addition, the present application also provides a head-mounted display device, which is applied to the line-of-sight tracking method according to any one of the above, and the head-mounted display device comprises:

[0182] The left eye line-of-sight tracking device comprises a left eye bright pupil tracking component and a left eye dark pupil tracking component. The left eye bright pupil tracking component comprises a first light source and a first image acquisition device belonging to the same optical path. The left eye dark pupil tracking component comprises a second light source and a second image acquisition device belonging to different optical paths. The left eye dark pupil tracking component and the left eye bright pupil tracking component are set to operate in time division.

[0183] The right eye line-of-sight tracking device comprises a right eye bright pupil tracking component and a right eye dark pupil tracking component. The right eye bright pupil tracking component comprises a third light source and a third image acquisition device belonging to the same optical path. The right eye dark pupil tracking component comprises a fourth light source and a fourth image acquisition device belonging to different optical paths. The right eye dark pupil tracking component and the right eye bright pupil tracking component are set to operate in time division.

[0184] The left eye dark pupil tracking component and the right eye bright pupil tracking component are configured to start running in the same acquisition cycle, and the left eye bright pupil tracking component and the right eye dark pupil tracking component are configured to start running in the same acquisition cycle.

[0185] The head-mounted display device provided in this invention, employing the gaze tracking method described above, can solve the technical problem that the light emitted by the light sources of the two eye-tracking modules in current head-mounted display devices easily interferes with each other, affecting the recognition accuracy of gaze tracking. Compared with the prior art, the beneficial effects of the head-mounted display device provided in this invention are the same as those of the gaze tracking method provided in the above embodiments, and other technical features in the head-mounted display device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0186] like Figure 7 As shown, Figure 7 This is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiments of the present invention.

[0187] In this embodiment of the invention, the terminal is a head-mounted display device. This head-mounted display device can be, for example, a Mixed Reality (MR) device (e.g., MR glasses or MR helmet), an Augmented Reality (AR) device (e.g., AR glasses or AR helmet), a Virtual Reality (VR) device (e.g., VR glasses or VR helmet), an Extended Reality (XR) device (e.g., XR glasses or XR helmet), or some combination thereof.

[0188] like Figure 7 As shown, the terminal may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0189] Optionally, the terminal can also include a camera, RF (Radio Frequency) circuit, sensors, audio circuit, WiFi module, etc. Among them, the sensors are, for example, light sensors, motion sensors, and other sensors. Specifically, the light sensors can include ambient light sensors and proximity sensors, wherein the ambient light sensors can adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensors can turn off the display screen and / or backlight when the terminal device is moved to the ear. Of course, the terminal device can also be configured with a gyroscope, barometer, hygrometer, thermometer, infrared sensor, and other sensors, which will not be described here.

[0190] Those skilled in the art can understand that, Figure 7 The terminal structure shown in the above embodiments does not constitute a limitation on the terminal, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0191] As shown in Figure 7 The memory 1005 as a computer storage medium can include an operating system, a network communication module, a user interface module, and a line-of-sight tracking program.

[0192] In the terminal shown in Figure 7 In the terminal shown in

[0193] controlling the left eye bright pupil tracking component and the left eye dark pupil tracking component to alternately run to alternately collect left eye bright pupil images and left eye dark pupil images; tracking and positioning the left eye line of sight according to the alternately collected left eye bright pupil images and left eye dark pupil images;

[0194] controlling the right eye bright pupil tracking component and the right eye dark pupil tracking component to alternately run to alternately collect right eye bright pupil images and right eye dark pupil images; tracking and positioning the right eye line of sight according to the alternately collected right eye bright pupil images and right eye dark pupil images;

[0195] Among them, the left eye bright pupil image and the right eye dark pupil image are collected in the same collection cycle, and the left eye dark pupil image and the right eye bright pupil image are collected in the same collection cycle.

[0196] The application also provides a computer readable storage medium, the computer readable storage medium has a line-of-sight tracking program stored thereon, and the line-of-sight tracking program is executed by a processor to realize the steps of the line-of-sight tracking method described above.

[0197] The computer readable storage medium of the present application has the same implementation as the above-mentioned line-of-sight tracking method, and will not be described here.

[0198] It should be noted that in this paper, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or system. Without more limitations, the element defined by the sentence "includes a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0199] The above-mentioned embodiment number of the present application is only for description, not representing the advantages and disadvantages of the embodiments.

[0200] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, including a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the present application.

[0201] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, any equivalent structural transformation made under the inventive concept of the present application, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A line of sight tracking method, characterized by, The line-of-sight tracking method is applied to a head-mounted display device, the head-mounted display device comprising a left-eye bright pupil tracking component, a left-eye dark pupil tracking component, a right-eye bright pupil tracking component and a right-eye dark pupil tracking component, and the method comprises: controlling the left-eye bright pupil tracking component and the left-eye dark pupil tracking component to alternately operate to alternately acquire left-eye bright pupil images and left-eye dark pupil images; tracking and positioning the left-eye line-of-sight according to the alternately acquired left-eye bright pupil images and left-eye dark pupil images; controlling the right-eye bright pupil tracking component and the right-eye dark pupil tracking component to alternately operate to alternately acquire right-eye bright pupil images and right-eye dark pupil images; tracking and positioning the right-eye line-of-sight according to the alternately acquired right-eye bright pupil images and right-eye dark pupil images; wherein the left-eye bright pupil images and the right-eye dark pupil images are acquired in the same acquisition cycle, and the left-eye dark pupil images and the right-eye bright pupil images are acquired in the same acquisition cycle.

2. The line of sight tracking method of claim 1, wherein, The step of tracking and positioning the left-eye line-of-sight according to the alternately acquired left-eye bright pupil images and left-eye dark pupil images comprises: identifying the eye features of the left-eye bright pupil images and the left-eye dark pupil images acquired in adjacent acquisition cycles to obtain left-eye bright pupil eye features and left-eye dark pupil eye features, wherein the eye features comprise at least one of a pupil center position, a pupil shape, an iris position, an iris shape and a glint position; performing feature matching on the left-eye bright pupil eye features and the left-eye dark pupil eye features to obtain a feature matching degree; if the feature matching degree is greater than a preset degree threshold, performing the step of tracking and positioning the left-eye line-of-sight according to the alternately acquired left-eye bright pupil images and left-eye dark pupil images.

3. The line of sight tracking method of claim 2, wherein, The step of performing feature matching on the left-eye bright pupil eye features and the left-eye dark pupil eye features to obtain a feature matching degree is followed by: if the feature matching degree is less than or equal to a preset degree threshold, performing image recognition on the left-eye bright pupil images to determine a bright pupil acquisition interference degree corresponding to the left-eye bright pupil images, and performing image recognition on the left-eye dark pupil images to determine a dark pupil acquisition interference degree corresponding to the left-eye dark pupil images; if the bright pupil acquisition interference degree is greater than the dark pupil acquisition interference degree by a first preset degree value, correcting the left-eye bright pupil images by the left-eye dark pupil images; tracking and positioning the left-eye line-of-sight according to the left-eye dark pupil images and the corrected left-eye bright pupil images; if the bright pupil acquisition interference degree is less than the dark pupil acquisition interference degree by a second preset degree value, correcting the left-eye dark pupil images by the left-eye bright pupil images; tracking and positioning the left-eye line-of-sight according to the left-eye bright pupil images and the corrected left-eye dark pupil images.

4. The line of sight tracking method of claim 3, wherein, The step of correcting the left-eye bright pupil images by the left-eye dark pupil images comprises: acquiring left-eye dark pupil eye features corresponding to the left-eye dark pupil images and left-eye bright pupil eye features corresponding to the left-eye bright pupil images; performing a sum value calculation on the left eye dark pupil eyeball feature of the first preset weight and the left eye bright pupil eyeball feature of the second preset weight to obtain a fusion eyeball feature, wherein the first preset weight is greater than the second preset weight, and a sum of the first preset weight and the second preset weight is equal to one; associating a bright pupil image corresponding to the fusion eyeball feature as a corrected left eye bright pupil image.

5. The line of sight tracking method of claim 3, wherein, The step of performing image recognition on the left eye bright pupil image to determine the bright pupil collection interference degree corresponding to the left eye bright pupil image comprises: performing image recognition on the left eye bright pupil image to determine the size of the pupil and / or the depth of the iris color in the left eye bright pupil image; determining the bright pupil collection interference degree corresponding to the left eye bright pupil image according to the size of the pupil and / or the depth of the iris color in the left eye bright pupil image.

6. The line of sight tracking method of claim 3, wherein, The step of performing image recognition on the left eye dark pupil image to determine the dark pupil collection interference degree corresponding to the left eye dark pupil image comprises: performing image recognition on the left eye dark pupil image to determine the covering degree of the eyelashes and / or the depth of the iris color in the left eye dark pupil image; determining the dark pupil collection interference degree corresponding to the left eye dark pupil image according to the covering degree of the eyelashes and / or the depth of the iris color in the left eye dark pupil image.

7. The line of sight tracking method of claim 1, wherein, The step of tracking and positioning the left eye visual line according to the alternately collected left eye bright pupil image and left eye dark pupil image comprises: dynamically identifying the left eye pupil center position and the left eye glint position according to the alternately collected left eye bright pupil image and left eye dark pupil image; determining the relative position vector between the pupil center and the glint corresponding to the left eye according to the left eye pupil center position and the left eye glint position; querying the left eye visual line direction mapped by the relative position vector from a preset left eye data mapping library to track and position the left eye visual line.

8. The line of sight tracking method of claim 7, wherein, The method further comprises: displaying a preset calibration mark at a preset display position; controlling the working states of the left eye bright pupil tracking component and the left eye dark pupil tracking component to identify the left eye calibration vector between the pupil center and the glint corresponding to the left eye when the user gazes at the preset calibration mark; determining the left eye calibration visual line direction according to the left eye visual line direction corresponding to the preset display position when the user gazes at the preset display position; establishing a mapping relationship between the left eye calibration vector and the left eye calibration visual line direction, and calibrating the pre-calibrated left eye vector mapping library according to the mapping relationship between the left eye calibration vector and the left eye calibration visual line direction; using the calibrated left eye vector mapping library as the preset left eye vector mapping library.

9. A head-mounted display device, comprising: The head-mounted display device is applied to the visual line tracking method according to any one of claims 1 to 8, and the head-mounted display device comprises: a left eye visual line tracking device comprising a left eye bright pupil tracking component and a left eye dark pupil tracking component, the left eye bright pupil tracking component comprising a first light source and a first image collection device belonging to the same optical path, and the left eye dark pupil tracking component comprising a second light source and a second image collection device belonging to different optical paths, the left eye dark pupil tracking component and the left eye bright pupil tracking component being set to operate in a time-sharing manner. The right eye line-of-sight tracking device comprises a right eye bright pupil tracking assembly and a right eye dark pupil tracking assembly, the right eye bright pupil tracking assembly comprises a third light source and a third image acquisition device belonging to the same optical path, the right eye dark pupil tracking assembly comprises a fourth light source and a fourth image acquisition device belonging to different optical paths, and the right eye dark pupil tracking assembly and the right eye bright pupil tracking assembly are arranged to be opened and operated at different times. The left eye dark pupil tracking assembly and the right eye bright pupil tracking assembly are arranged to be opened and operated at the same acquisition cycle, and the left eye bright pupil tracking assembly and the right eye dark pupil tracking assembly are arranged to be opened and operated at the same acquisition cycle.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a line-of-sight tracking program, and the line-of-sight tracking program is executed by the processor to realize the steps of the line-of-sight tracking method in any one of claims 1 to 8.

Images