Head-mounted display auxiliary wearing additional module and method

Abstract

The present invention discloses an auxiliary wearing additional module and method for a head-mounted display. After the additional module is added to the HMD, the module and algorithm can be used to calculate and assist in locating the eye position of the user currently wearing the HMD, so that an outsider can help the user adjust the HMD. The additional module is: two helmet lens through holes are opened on the support plate corresponding to the positions of the eyes; two infrared cameras and one infrared light source are set above each helmet lens through hole. The infrared camera collects human eye images under the auxiliary illumination of the infrared light source and sends them to the processor. The processor is used to calculate the current human eye position parameters based on the collected human eye image and display them on the display. The auxiliary wearing method provided by the present invention realizes the determination of the eye pupil relative to the module space coordinates, realizes the graphical representation of the user's current eye position, realizes the user's auxiliary wearing process, and prompts the user's eye working status.

Description

Technical Field

[0001] The present invention relates to the technical field, and in particular to an auxiliary wearing add-on module and method for a head-mounted display. Background Art

[0002] A head-mounted display (HMD) is a device that users must wear when experiencing virtual reality (VR), augmented reality (AR), mixed reality (MR), and other systems. As a near-eye display system, the user observes the image on the screen through an optical module (lens system).

[0003] When a user wears an HMD, the human eye must be within the exit pupil range of the optical module to see a clear image through the module. Otherwise, only blurred or highly distorted images can be seen, affecting the user's use. After wearing a virtual reality HMD, the user's eyes are blocked, making it difficult for other people to determine whether the user is wearing it properly. In specific HMD usage scenarios, such as the use of HMDs for non-drug interventions on patients, only when the user can observe clear images can the user be immersed in the virtual environment. Therefore, it is very necessary to be able to determine from the outside whether the depth of the user's eyes and the pupil distance are appropriate after wearing the HMD. It is also of great significance for scenarios other than the above, such as when the elderly, children and other users need auxiliary wearing.

[0004] There are technical solutions for eye movement detection, such as patents CN201010503531.4 and CN201710065321.3. When a user observes a picture with their eyes, this solution uses a camera to capture the reflected image of the eye and the cornea of ​​the eye, and analyzes the image to obtain the direction of the eye.

[0005] Eye tracker-based technical solutions primarily focus on the gaze direction of the eyes. However, these solutions fail to determine the spatial coordinates of the pupils, making them ineffective in assisting with the wearer's experience. Summary of the Invention

[0006] In view of this, the present invention provides an auxiliary wearing add-on module and method for a head-mounted display. After the additional module is added to the HMD, the module and algorithm can be used to calculate and assist in locating the eye position of the user currently wearing the HMD, so that an outsider can help the user adjust the HMD.

[0007] To achieve the above objectives, the technical solution of the present invention is: a head-mounted display auxiliary wearing additional module, which is added to the HMD optical module close to the human eye side. The additional module includes a support plate, an infrared camera, an infrared light source, a processor and a display.

[0008] Two helmet lens through holes are opened on the support plate corresponding to the positions of the eyes; two infrared cameras and one infrared light source are set above each helmet lens through hole.

[0009] The infrared camera collects images of the human eye under the auxiliary illumination of the infrared light source and sends them to the processor.

[0010] The processor is used to calculate the current human eye position parameters based on the collected human eye images, including the pupil coordinates of both eyes and the blinking frequency.

[0011] The display is used to display the current human eye position parameters and the relative status of the current human eye position and the recommended position.

[0012] Preferably, the infrared camera collects the human eye image under the auxiliary illumination of the infrared light source and sends it to the processor. Specifically, there are 4 infrared cameras on the through holes of the two helmet lenses. The infrared cameras collect and obtain the user's eyes and surrounding images in real time, and obtain the Image t,L1 , Image t,L2 , Image t,R1 , Image t,R2 ;Image t,L1 , Image t,L2 Image is the human eye image captured by the two infrared cameras on the left through hole at time t. t,R1 , Image t,R2 This is the human eye image captured by the two infrared cameras on the right through-hole at time t.

[0013] At each moment, four images are acquired and sent to the processor.

[0014] Preferably, the processor includes a pre-processing unit, a pupil range detection unit, a blink detection unit, and a pupil coordinate detection unit;

[0015] Preprocessing unit, for each image Image at time t t ∈{Image t,L1 , Image t,L2 , Image t,R1 , Image t,R2 The following preprocessing is performed: Considering that the pupil is a low-brightness area due to its low reflection, the image pixel brightness is convolved with a preset convolution kernel to obtain the preprocessed image.

[0016] The pupil range detection unit is used to perform sliding window detection on the image and obtain the area with the lowest brightness as the selected area, where: the sliding window starting point is the latest stored pupil center, and ellipse fitting is performed on the edge pixels of the selected area. If it is successful and the fitting confidence exceeds the threshold, the image is sent to the pupil coordinate detection unit; otherwise, the eye state at time t is judged to be closed, and the eye state time series data is sent to the blink detection unit.

[0017] The pupil coordinate detection unit takes the center of the ellipse as the pupil center position; stores the latest pupil center position pixel coordinates and the eye state at time t is open, and the eye state time series data is sent to the blink detection unit; the pupil coordinate detection unit calculates the coordinates of the current eye pupil in the specified coordinate system based on the calibration information of the two infrared cameras corresponding to each human eye and the latest detected pupil center position pixel coordinates, and outputs the pupil coordinates.

[0018] The blink detection unit receives the time series data of the eye state and performs sliding window detection. When the window finds a process of open-close-open eyes, it is detected as a blink and the blink frequency is output.

[0019] Preferably, the pupil coordinates are three-dimensional coordinates.

[0020] Another embodiment of the present invention further provides a head-mounted display auxiliary wearing method, comprising the following steps:

[0021] Step 1: The infrared light source provides infrared light for the infrared camera to collect;

[0022] Step 2: After the user wears the HMD, the infrared camera collects time-series images of the user's eyes and surroundings;

[0023] Step 3: The processing module obtains the data collected by the camera, performs image operations on the two images of each eye, and calculates the three-dimensional coordinates of the pupil and the blink frequency. Step 3 includes the following specific steps:

[0024] Step 301: receiving an input image captured by a camera at time t.

[0025] Step 302: Preprocess the input image: perform a convolution operation on the image based on the selected convolution kernel to extract low-brightness pixels.

[0026] Step 303: Perform sliding window detection on the image, and obtain the brightness and the lowest area as the selected area.

[0027] Step 304: Determine whether it is the pupil area: perform ellipse fitting on the edge pixels of the selected area. If successful and the fitting confidence exceeds the threshold, proceed to step 305.

[0028] Otherwise, store the state at time t as eyes closed; and execute step 308 .

[0029] Step 305: Take the center of the ellipse as the pupil center pixel.

[0030] Step 306: Store the latest pupil center pixel coordinates and the eye-opening state at time t.

[0031] Step 307: Based on the calibration information of the two infrared cameras corresponding to each human eye and the latest detected pupil center position pixel coordinates, calculate the coordinates of the current eye pupil in the specified coordinate system and output the pupil three-dimensional coordinates.

[0032] Step 308: Receive the time series data of the eye state and perform sliding window detection. When the window finds a process of eyes open-eye closed-eye open, it is detected as a blink and the blink frequency is output.

[0033] Step 4: Based on the output of step 3, a display is used to provide a visual presentation to the auxiliary personnel.

[0034] Furthermore, step 303: perform sliding window detection on the image, obtain the brightness and lowest area as the selected area, wherein the sliding window starting point is selected in the following manner: when the pupil position has been detected before, use this detected pupil position as the starting point to search around, otherwise use the upper left corner of the image as the default starting point to search.

[0035] Furthermore, step 307: based on the calibration information of the two infrared cameras corresponding to each human eye and the latest detected pupil center position pixel coordinates, the coordinates of the current eye pupil in the specified coordinate system are calculated and the pupil three-dimensional coordinates are output, specifically:

[0036] The pupil center points under the four infrared cameras at time t are obtained. Since the four infrared cameras have been calibrated in advance, the four sets of corresponding relationships at time t are obtained: {K i ,RT i ,C t,i}, K i is the internal parameter of infrared camera i, RT i is the external parameter of infrared camera i, C t,i is the pupil coordinate of infrared camera i, where i∈{L1,L2,R1,R2}.

[0037] Based on the calibration data of the calibrated infrared camera and the pixel coordinates of the center point, the back-projection line of the pixel point under the current infrared camera through the optical center of the camera is obtained. The two rays theoretically intersect at the pupil position. The three-dimensional coordinates of the theoretical intersection are the three-dimensional coordinates of the pupil; that is, based on the corresponding relationship {K i ,RT i ,C t,iSolve the two back-projection ray equations for each eye to obtain the reconstructed three-dimensional coordinates. Calculate the corresponding relationship for the left and right eyes respectively to obtain the three-dimensional coordinates P of the corresponding points in the infrared camera calibration coordinate system. t,j , where j∈{L,R}, P t,j Output to the display module, the third dimension in the three-dimensional coordinate is the pupil depth.

[0038] Beneficial effects:

[0039] 1. The present invention provides an auxiliary wearing add-on module for a head-mounted display, which aims to solve the problem that the wearing of the HMD can only be determined subjectively by the user, and cannot be objectively evaluated or assisted in positioning by other personnel, and the user's current eye status cannot be known. The key to the present invention is to achieve the determination of the spatial coordinates of the eye pupil relative to the module, to achieve a graphical representation of the user's current eye position, to assist the user in the wearing process, and to prompt the user's eye working status. The system can solve the problem that the user is not familiar with the VR / AR system and cannot wear the HMD correctly. When used in medical processes, it is even more necessary to ensure the user's wearing condition. Through this solution, it is possible to prompt the user's pupil position and the relative position relative to the helmet system, help the auxiliary wearer to complete the debugging of the HMD wearing position, and help external personnel observe the user's blinking status during use, so that external personnel can assist the user in using the HMD in healthy eye use.

[0040] 2. The present invention also provides a head-mounted display auxiliary wearing method, which is an auxiliary wearing process for the above-mentioned head-mounted display auxiliary wearing additional module. The method realizes the determination of the eye pupil relative to the module space coordinates, realizes the graphical representation of the user's current eye position, realizes the user's auxiliary wearing process, and prompts the user's eye working status. The system can solve the problem that the user is not familiar with the VR / AR system and cannot wear the HMD correctly. When used in medical processes, it is even more necessary to ensure the user's wearing condition. Through this solution, it is possible to prompt the user's pupil position and the relative position relative to the helmet system, help the auxiliary wearing personnel complete the debugging of the HMD wearing position, and can help external personnel observe the user's blinking status during use, so that external personnel can assist the user in using the HMD in healthy eye use. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the auxiliary module;

[0042] Figure 2 This is the workflow diagram of the auxiliary module;

[0043] Figure 3 This is a diagram of the triangulation principle in computer vision. DETAILED DESCRIPTION

[0044] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0045] The system structure of the present invention is as follows Figure 1 A head-mounted display auxiliary wearing add-on module is characterized in that the add-on module is added to the side of the HMD optical module close to the human eye, and the add-on module includes a support plate, an infrared camera, an infrared light source, a processor, and a display.

[0046] Two helmet lens through holes are opened on the support plate corresponding to the positions of the eyes; two infrared cameras and one infrared light source are set above each helmet lens through hole.

[0047] The infrared camera collects images of the human eye under the auxiliary illumination of the infrared light source and sends them to the processor.

[0048] The processor is used to calculate the current human eye position parameters based on the collected human eye images, including the pupil coordinates of both eyes and the blinking frequency.

[0049] The display is used to display the current human eye position parameters and the relative status of the current human eye position and the recommended position.

[0050] Among them, the camera uses the center of the auxiliary module support structure as the coordinate center, the long side and short side correspond to the x-axis and y-axis, and the coordinate system is defined with the positive direction of the z-axis facing the eye. The four cameras are calibrated in advance.

[0051] The specific technical solution is described as follows:

[0052] After the auxiliary module is started, the infrared light source emits infrared light to illuminate the surrounding environment and eyes without affecting the user's normal observation.

[0053] The infrared camera collects and obtains the user's eyes and surrounding images in real time, and obtains the Image t,L1 , Image t,L2 , Image t,R1 , Image t,R2 The infrared cameras have been calibrated before use, that is, the camera calibration obtains the internal and external parameters {K i ,RT i}, where i∈{L1,L2,R1,R2}. Since the infrared camera is fixed to the support structure, the internal and external parameters are constants that do not change over time if the camera components are not replaced.

[0054] The four images are acquired and processed by a processor comprising a pre-processing unit, a pupil range detection unit, a blink detection unit, and a pupil coordinate detection unit.

[0055] Preprocessing unit, for each image Image at time t t ∈{Image t,L1 , Image t,L2 , Image t,R1 , Image t,R2}Perform the following preprocessing operations:

[0056] Considering that the pupil position has less reflection, it will appear as a low-brightness (black) area in the image, while other parts have more reflection and will appear as a high-brightness area (leaning towards white) in the image. For the convenience of subsequent processing, this step uses convolution operation to enhance the image data before detection to improve the image contrast. For each pixel point, the more white pixels in the convolution area, the higher its own brightness; the more black pixels, the lower its own brightness. This operation will obtain the preprocessed image Image t ′.

[0057] Pupil range detection unit, for the processed image Image t ′, take 1 / 2 of the length and width of the original image obtained by the infrared camera as the selected window size, perform sliding window detection on the image, and the area with the lowest brightness is the area with the highest probability of pupil appearance. When multiple areas have the same value, the area at the center of multiple areas is taken as the target area S t .

[0058] To optimize the detection speed, if the pupil position has been detected before, the search will start from this position and move to the surrounding area. Otherwise, the search will start from the upper left corner of the image as the default starting point.

[0059] The pupil is an ellipse in the camera projection space. t The lowest brightness pixel closest to the center is used as the reference to find S t The pixel edge of the largest connected domain in the image is found and an ellipse is fitted. When the ellipse is fitted successfully and the confidence of the ellipse exceeds the set threshold, it is considered that the pupil is found and the center point of the fitted ellipse is used as the ellipse center C. t The pupil coordinates are used as the pixel coordinates of the pupil within the image, and the user's current eye state is assumed to be open. The image is then fed into the pupil coordinate detection unit. If the fit fails or the confidence level is too low, the pupil is considered undetectable and the user's eye state is considered closed. The eye state time series data is fed into the blink detection unit.

[0060] The pupil coordinate detection unit performs the following processing: When the detection is successful, the pupil center points under the four infrared cameras at time t are obtained. Since the four infrared cameras have been calibrated in advance, we have obtained four sets of correspondences at time t: {K i ,RT i ,Ct,i}, where i∈{L1,L2,R1,R2}. Based on the triangulation principle in computer vision, the two images obtained by each eye are reconstructed to obtain the three-dimensional coordinates of the point. The specific process is as follows: Based on the calibration data of the calibrated infrared camera and the pixel coordinates of the center point, the back projection line of the pixel point under the current infrared camera through the optical center of the camera can be obtained (as shown in Figure L l With L r ), the two rays intersect at the pupil theoretically, and the three-dimensional coordinates of the theoretical intersection are the three-dimensional coordinates of the pupil. In the actual solution process, based on the corresponding relationship {K i ,RT i ,C t,i The two back-projection ray equations for each eye can be solved to obtain the reconstructed three-dimensional coordinates. By calculating the corresponding relationship for the left and right eyes respectively, the three-dimensional coordinates P of the corresponding point in the infrared camera calibration coordinate system can be obtained. t,j , where j∈{L,R}. P t,j Output to the display.

[0061] The blink detection unit receives the time series data of the eye state and performs sliding window detection. When the window finds the process of open-closed-open eyes once, it is detected as a blink and the blink frequency is output. In the embodiment of the present invention, the blinking situation in the last minute is detected every 10 seconds. When the storage state sliding window finds the process of open-closed-open eyes once, a blink is recorded. The number of blinks within 1 minute is counted and output to the display as the blink frequency.

[0062] The display compares the acquired eye position information with the ideal data preset for the HMD model and displays the position marker on the screen. When blink data is received, it is compared with the preset normal blink frequency range to display the current status.

[0063] This system can address the issue of users not being able to properly wear the HMD due to unfamiliarity with VR / AR systems. When used in medical procedures, it's even more crucial to ensure proper fit. This solution provides prompts for the user's pupil position and relative position to the headset system, helping the wearer adjust the HMD's fit. It also allows external personnel to observe the user's blinking during use, enabling them to assist with healthy eye use.

[0064] The key of the present invention is to realize the determination of the spatial coordinates of the eye pupil relative to the module, realize the graphical representation of the user's current eye position, realize the auxiliary wearing process for the user, and prompt the user's eye working status.

[0065] The technical process of the present invention for processing human eye positioning is as follows Figure 2 A head-mounted display auxiliary wearing method includes the following steps:

[0066] Step 1: The infrared light source provides infrared light for the infrared camera to collect;

[0067] Step 2: After the user wears the HMD, the infrared camera collects time-series images of the user's eyes and surroundings;

[0068] Step 3: The processing module obtains the data collected by the camera, performs image operations on the two images of each eye, and calculates the three-dimensional coordinates of the pupil and the blink frequency. Step 3 includes the following specific steps:

[0069] Step 301: receiving an input image captured by a camera at time t;

[0070] Step 302: Preprocess the input image: perform a convolution operation on the image based on the selected convolution kernel to extract low-brightness pixels;

[0071] Step 303: Perform sliding window detection on the image and obtain the brightness and lowest area as the selected area; the sliding window starting point is selected as follows: if the pupil position has been detected before, use this detected pupil position as the starting point and search around it; otherwise, use the upper left corner of the image as the default starting point for searching.

[0072] Step 304: Determine whether it is the pupil area: perform ellipse fitting on the edge pixels of the selected area. If successful and the fitting confidence exceeds the threshold, proceed to step 305;

[0073] Otherwise, store the state at time t as closed eyes; and execute step 308;

[0074] Step 305: taking the center of the ellipse as the pupil center pixel;

[0075] Step 306: storing the latest pupil center pixel coordinates and the eye state at time t as open;

[0076] Step 307: Based on the calibration information of the two infrared cameras corresponding to each eye and the latest detected pupil center pixel coordinates, the coordinates of the current eye pupil in the specified coordinate system are calculated and the pupil three-dimensional coordinates are output. Specifically,

[0077] The pupil center points under the four infrared cameras at time t are obtained. Since the four infrared cameras have been calibrated in advance, the four sets of corresponding relationships at time t are obtained: {K i ,RT i ,C t,i}, K i is the intrinsic parameter of infrared camera i, RT i is the external parameter of infrared camera i, Ct,i is the pupil coordinate of infrared camera i, where i∈{L1,L2,R1,R2};

[0078] Based on the calibration data of the calibrated infrared camera and the pixel coordinates of the center point, the back-projection line of the pixel point under the current infrared camera through the optical center of the camera is obtained. The two rays theoretically intersect at the pupil position. The three-dimensional coordinates of the theoretical intersection are the three-dimensional coordinates of the pupil; that is, based on the corresponding relationship {K i ,RT i ,C t,i Solve the two back-projection ray equations for each eye to obtain the reconstructed three-dimensional coordinates. Calculate the corresponding relationship for the left and right eyes respectively to obtain the three-dimensional coordinates P of the corresponding points in the infrared camera calibration coordinate system. t,j , where j∈{L,R}, P t,j Output to the display module, the third dimension in the three-dimensional coordinate is the pupil depth.

[0079] Step 308: Receive the time series data of the eye state and perform sliding window detection. When the window finds a process of eyes open-closed-open, it is detected as a blink and the blink frequency is output.

[0080] Step 4: Based on the output of step 3, a display is used to provide a visual presentation to the auxiliary personnel.

[0081] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A head-mounted display auxiliary wearing additional module, characterized in that: The additional module is added to the side of the HMD optical module close to the human eye. The additional module includes a support plate, an infrared camera, an infrared light source, a processor, and a display; Two helmet lens through holes are provided on the support plate corresponding to the positions of the eyes; two infrared cameras and one infrared light source are provided above each helmet lens through hole; The infrared camera collects human eye images under the auxiliary illumination of the infrared light source and sends the images to the processor; The processor is used to calculate the current human eye position parameters based on the collected human eye images, including pupil coordinates of both eyes and blink frequency; The display is used to provide visual presentation for assistants, display current eye position parameters, and display the relative status of the current eye position and the recommended position.

2. The head-mounted display auxiliary wearing additional module according to claim 1, characterized in that: The infrared camera collects human eye images under the auxiliary illumination of the infrared light source and sends the images to the processor, specifically: There are 4 infrared cameras on the two helmet lens holes. The infrared cameras collect and obtain the user's eyes and surrounding images in real time, and obtain the time t respectively. , , , ; , is the human eye image captured by the two infrared cameras on the left through-hole at time t, , The image of the human eye at time t captured by the two infrared cameras on the right through-hole; At each moment, four images are acquired and sent to the processor.

3. The head-mounted display auxiliary wearing additional module according to claim 2, characterized in that: The processor includes a pre-processing unit, a pupil range detection unit, a blink detection unit, and a pupil coordinate detection unit; The pre-processing unit is for For each image at any time , The following preprocessing operations are performed: considering that the pupil position is a low-brightness area due to less reflection; therefore, the brightness of the image pixels is convolved with a preset convolution kernel to obtain the preprocessed image; The pupil range detection unit is used to perform sliding window detection on the image, and obtain the area with the lowest brightness as the selected area, wherein: the sliding window starting point is the latest stored pupil center, and ellipse fitting is performed on the edge pixels of the selected area. If the fitting is successful and the confidence level of the fitting exceeds a threshold, the image is sent to the pupil coordinate detection unit; otherwise, the eye state at time t is determined to be closed, and the eye state time series data is sent to the blink detection unit; The pupil coordinate detection unit takes the center of the ellipse as the pupil center position; The latest pupil center pixel coordinates and the eye state at time t, indicating that the eye is open, are stored and the eye state time series data is fed into a blink detection unit; the pupil coordinate detection unit calculates the coordinates of the current pupil in a specified coordinate system based on the calibration information of the two infrared cameras corresponding to each eye and the latest pupil center pixel coordinates detected, and outputs the pupil coordinates; The blink detection unit receives the time series data of the eye state and performs sliding window detection. When the window finds a process of eyes open-eye closed-eye open, it is detected as a blink and the blink frequency is output.

4. The head-mounted display auxiliary wearing additional module according to any one of claims 1 to 3, characterized in that: The pupil coordinates are three-dimensional coordinates.

5. A head-mounted display auxiliary wearing method, characterized in that: The steps include: Step 1: The infrared light source provides infrared light for the infrared camera to collect; Step 2: After the user wears the HMD, the infrared camera collects time-series images of the user's eyes and surroundings; Step 3: The processing module obtains the data collected by the camera, performs image operations on the two images of each eye, and calculates the three-dimensional coordinates of the pupil and the blink frequency. Step 3 includes the following specific steps: Step 301: receiving an input image captured by a camera at time t; Step 302: Preprocess the input image: perform a convolution operation on the image based on the selected convolution kernel to extract low-brightness pixels; Step 303: Perform sliding window detection on the image to obtain the brightness and the lowest area as the selected area; Step 304: Determine whether it is the pupil area: perform ellipse fitting on the edge pixels of the selected area. If successful and the fitting confidence exceeds the threshold, proceed to step 305; Otherwise, store the state at time t as closed eyes; and execute step 308; Step 305: taking the center of the ellipse as the pupil center pixel; Step 306: storing the latest pupil center pixel coordinates and the eye state at time t as open; Step 307: Based on the calibration information of the two infrared cameras corresponding to each eye and the latest detected pupil center position pixel coordinates, the coordinates of the current eye pupil in the specified coordinate system are calculated and the pupil three-dimensional coordinates are output; Step 308: Receive the time series data of the eye state and perform sliding window detection. When the window finds a process of eyes open-closed-open, it is detected as a blink and the blink frequency is output. Step 4: Based on the output of step 3, a display is used to provide a visual presentation to the auxiliary personnel.

6. The head mounted display auxiliary wearing method according to claim 5, characterized in that: Step 303: Perform sliding window detection on the image to obtain the brightness and lowest area as the selected area, wherein the sliding window starting point is selected in the following manner: if the pupil position has been detected before, the detected pupil position is used as the starting point to search around; otherwise, the upper left corner of the image is used as the default starting point for searching.

7. The head mounted display assisting wearing method according to claim 5, wherein: Step 307: Based on the calibration information of the two infrared cameras corresponding to each human eye and the latest detected pupil center position pixel coordinates, the coordinates of the current eye pupil in the specified coordinate system are calculated and the pupil three-dimensional coordinates are output, specifically: Obtained The pupil center points under the four infrared cameras at time t. Since the four infrared cameras have been calibrated in advance, the four sets of correspondences at time t are obtained: , is the intrinsic parameter of infrared camera i, is the external parameter of infrared camera i, is the pupil coordinate of infrared camera i, where ; Based on the calibration data of the calibrated infrared camera and the pixel coordinates of the center point, the back-projection line of the pixel point under the current infrared camera through the optical center of the camera is obtained. The two rays theoretically intersect at the pupil position. The three-dimensional coordinates of the theoretical intersection are the three-dimensional coordinates of the pupil; that is, based on the corresponding relationship Solve the two back-projection ray equations for each eye to obtain the reconstructed three-dimensional coordinates. Calculate the left and right eyes based on the above correspondence to obtain the three-dimensional coordinates of the corresponding points in the infrared camera calibration coordinate system. ,in , Output to the display module, the third dimension in the three-dimensional coordinate is the pupil depth.

Images