A system calibration method for a VR / AR head-mounted display

Abstract

The embodiment of the application provides a system calibration method of a VR / AR head-mounted display, the position relationship between a camera and an infrared lamp of the VR / AR head-mounted display is obtained through single target calibration, light spot detection, light spot serial number corresponding processing, binocular stereo matching to obtain light spot 3D coordinates, and then coordinate system conversion, and finally accurate position relationship between the camera and the infrared lamp of the VR / AR head-mounted display is obtained. According to different infrared lamp RGB three channel pixel values captured by an auxiliary camera, false light spots are filtered, and according to the RGB three channel mean value proportion, the light spot center position is obtained after weighting, false light spots are effectively filtered, and the light spot center position precision is improved.

Description

Technical Field

[0001] This application belongs to the field of calibration technology, and in particular relates to a system calibration method for VR / AR head-mounted displays. Background Technology

[0002] VR and AR are developing rapidly and represent a new growth point for future technology. Major companies are actively developing their technologies. Since 80% of the information humans receive from the outside world comes from their eyes, eye-tracking technology is indispensable in VR and AR. The most common solution for eye-tracking technology is a camera and infrared lights. Infrared lights create a Purkinje spot on the cornea, the camera captures images of the eye, and then the real-time gaze position is obtained through eye model analysis. VR and AR systems are often complex, and there are design and assembly errors between the infrared lights and the camera. Currently, most technologies only calibrate the camera and correct distortion; the specific positions of the camera and infrared lights are determined by structural engineers, without effective positional calibration between them. Summary of the Invention

[0003] To solve or alleviate the aforementioned technical problems, this application proposes to use an auxiliary camera to perform monocular and stereo calibration between cameras and to detect and sort infrared lights, thereby achieving the purpose of calibrating the positions of VR / AR infrared lights and cameras, thus reducing design and assembly errors and improving eye-tracking accuracy.

[0004] This application provides a system calibration method for a VR / AR head-mounted display, including:

[0005] The calibration datasets of the auxiliary camera and the infrared camera are processed to obtain the translation and rotation matrices between the auxiliary camera and the infrared camera, and between the auxiliary cameras. The infrared camera is set on the VR / AR head-mounted display.

[0006] The infrared light images captured by the auxiliary camera are subjected to preliminary connected component extraction, and the preliminary connected components are filtered to obtain filtered connected components that meet preset conditions.

[0007] Calculate the infrared light image corresponding to the filtered connected component, and calculate the ratio of the sum of pixel values ​​in the R channel to the sum of pixel values ​​in the G channel of the infrared light image. Filter out the filtered connected components whose ratio is less than a first threshold to obtain the selected connected components.

[0008] Calculate the pixel mean at each coordinate in the infrared light image based on the selected connected components, and determine the center of all light spots in the infrared light image based on the pixel mean:

[0009] Starting from the spot center with the largest y-value among all spot centers in any two sets of infrared light images, sort all spot centers in any two sets of infrared light images, and process the spot numbers of all spot centers captured by all auxiliary cameras accordingly to finally obtain two new spot center sets.

[0010] Calculate the 3D coordinates of each new spot center set in the coordinate system of the auxiliary camera;

[0011] The 3D coordinates of the new light spot center set are transformed into the coordinate system corresponding to the infrared camera to complete the system calibration of the VR / AR head-mounted display.

[0012] Compared with existing technologies, this application provides a system calibration method for VR / AR head-mounted displays. The method involves single-target calibration, spot detection, spot number correspondence processing, and binocular stereo matching to obtain the 3D coordinates of the spot. After coordinate system transformation, the accurate positional relationship between the VR / AR head-mounted display's camera and its infrared LED is obtained. Furthermore, based on the different RGB three-channel pixel values ​​captured by the auxiliary camera, false spots are filtered out. The center position of the spot is obtained by weighting the values ​​based on the average proportion of the RGB three channels, effectively filtering false spots and improving the accuracy of the spot center position. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. Some specific embodiments of this application will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings designate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0014] Figure 1 This is a flowchart of a system calibration method for a VR / AR head-mounted display provided in an embodiment of this application. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort should fall within the scope of protection of the present application.

[0016] like Figure 1As shown, this application embodiment provides a system calibration method for a VR / AR head-mounted display, including:

[0017] Step S1: Process the calibration datasets of the auxiliary camera and the infrared camera to obtain the translation and rotation matrices between the auxiliary camera and the infrared camera, and between the auxiliary cameras. The infrared camera is set on the VR / AR head-mounted display.

[0018] It should be noted that the calibration platform needs to be set up first. The calibration platform consists of two parts: an adjustable fixture for fixing the VR / AR head-mounted display and a fixture for fixing two auxiliary cameras. Note: Camera 1 and Camera 2 are color auxiliary cameras, while Camera 3 and Camera 4 are the built-in infrared cameras of the VR / AR head-mounted display. The fixture for fixing the VR / AR head-mounted display can rotate, and the auxiliary cameras can also rotate to allow for image acquisition from different angles.

[0019] Each auxiliary camera and infrared camera acquired 20 frames of checkerboard images from different angles. Each auxiliary camera and infrared camera was calibrated individually, and the calibration results were written to files calibrationResult_1.yml, calibrationResult_2.yml, calibrationResult_3.yml, and calibrationResult_4.yml. The files contain the image size, intrinsic parameter matrix, and distortion coefficients acquired by each auxiliary camera and infrared camera. This completes the calibration of the auxiliary cameras and infrared cameras.

[0020] Subsequently, the positions of Camera 1, Camera 2, and the two VR / AR head-mounted displays were adjusted. First, it was ensured that the relative positions of Camera 1, Camera 2, and Camera 3 remained unchanged, and that Camera 1 and Camera 2 could simultaneously capture chessboard images from different angles, and that Camera 2 and Camera 3 could also simultaneously capture chessboard images from different angles. In addition, Camera 1 and Camera 2 could simultaneously capture infrared lights around Camera 3 on the head-mounted displays.

[0021] The stereo calibration datasets for cameras 1 and 2 (dataset12) were collected (at a checkerboard location, if cameras 1 and 2 collect data simultaneously, there are a total of 20 different locations on the checkerboard, so cameras 1 and 2 collect a total of 40 frames), the stereo calibration datasets for cameras 2 and 3 (dataset23) were collected in the same way as dataset12, and the infrared light images (image1 and image2) of the surrounding area of ​​camera 3 were collected by cameras 1 and 2.

[0022] Similarly, adjust the position of the head-mounted display fixture, collect dataset dataset24, and use camera one and camera two to collect data from the infrared lights around camera four to obtain images image3 and image4.

[0023] Taking camera one and camera two as examples, a stereo calibration is performed between the cameras:

[0024] The corner coordinates of the chessboard in dataset12 are extracted using the OpenCV function findChessboardCorners, and the accuracy of the detected corner coordinates is improved using the function cv::cornerSubPix.

[0025] Using the camera calibration intrinsic parameter matrix, distortion coefficients, and the detected corner point results in the files calibrationResult_1.yml and calibrationResult_2.yml, the camera stereo calibration is performed using the cv::stereoCalibrate function in OpenCV to obtain the rotation and translation matrices from camera one to camera two.

[0026] Similarly, obtain the rotation and translation matrices from camera 2 to camera 3, and from camera 2 to camera 4.

[0027] Step S2: Perform preliminary connected component extraction on the infrared light image captured by the auxiliary camera, and filter the extracted preliminary connected components to obtain filtered connected components that meet preset conditions.

[0028] It should be noted that, taking the infrared light image image1 captured by camera 1 from the surrounding area of ​​camera 3 as an example, the threshold function in OpenCV is used for binarization to convert the color (RGB) image1 into a grayscale image gray1. Then, the threshold function in OpenCV is used to binarize the infrared light image image to obtain the binary image binnary1. Then, the connected components of the binary image binnary1 are extracted by the cv::connectedComponentsWithStats function. The area range (minArea~maxArea) and aspect ratio range (minRatio~maxRatio) of the connected components are set, and connected components with a center x-direction less than the threshold minx and a y-direction less than the threshold miny are filtered out. Connected component regions that meet the conditions are retained (the pixel values ​​of unwanted connected components are set to 0, and the pixel values ​​of desired connected components are 255).

[0029] Step S3: Calculate the infrared light image corresponding to the filtered connected component, and calculate the ratio of the sum of pixel values ​​in the R channel to the sum of pixel values ​​in the G channel of the infrared light image. Filter out the filtered connected components whose ratio is less than a first threshold to obtain the selected connected components.

[0030] It should be noted that the connected components selected in step S2 are further filtered, and the sum of the pixel values ​​in the R channel of the infrared lamp image1 corresponding to the connected component region is compared with the sum of the pixel values ​​in the G channel. If the ratio is less than the threshold thr2 (the threshold is set to 2 in this scheme), the connected component is filtered again. The calculation formula is as follows:

[0031]

[0032] in, Let w represent the position of the top-left corner of the circumscribed matrix of the connected component, and let h represent the width and height of the connected component, respectively. These represent the infrared light images in... The pixel values ​​of the R channel and the G channel at the location.

[0033] Step S4: Calculate the pixel mean at each coordinate in the infrared light image, and determine the center of all light spots in the infrared light image based on the pixel mean.

[0034] It should be noted that to calculate the center position of each infrared lamp, first calculate the average pixel value at each position in the infrared lamp image1, and then calculate the center coordinates:

[0035]

[0036] Where, averarg1(x,y) represents the x,y values ​​in image1. The average value of the RGB three channels at the location. This represents the scores of the three RGB channels in the 0th connected component of image1; P1(x,y) represents the center position of all the light spots in the image image1. P1(x,y) represents the pixel value of the filtered connected component that meets the preset conditions. The subscript 1 represents the image image1, and the superscript 0 represents the 0th preliminary connected component. There are a total of n preliminary connected components (preferably n=8 in this embodiment). x and y represent the horizontal and vertical coordinates of the image image1 at the position, respectively.

[0037] Similarly, the center positions of all light spots in images image2, image3, and image4 are calculated as follows: .

[0038] Step S5: Starting from the spot center with the largest y-value among all spot centers in any two sets of infrared light images, sort all spot centers in any two sets of infrared light images, and process the spot number of all spot centers captured by all auxiliary cameras accordingly to finally obtain two new spot center sets.

[0039] It should be noted that the light spot with the maximum y-axis value of any two sets of light spot centers is selected as the first light spot, and then sorted counterclockwise or clockwise to match the light spot numbers captured by the left and right auxiliary cameras, thus obtaining a new set of light spot centers. Similarly, the new light spot center set is obtained. (Note: image1 and image2 are a group, and image3 and image4 are a group.)

[0040] Step S6: Calculate the 3D coordinates of each new light spot center set in the coordinate system of the auxiliary camera;

[0041] It should be noted that, based on the above, the new light spot center set is obtained... Distortion is corrected by setting parameters for a single target, and then the 3D coordinates of the light spots around the camera in the camera's two-coordinate system are calculated using a stereo binocular algorithm. Similarly, the 3D coordinates of the light spots around the camera in the camera's two-coordinate system can be obtained. .

[0042] Step S7: Transform the 3D coordinates of the new light spot center set to the coordinate system corresponding to the infrared camera.

[0043] Will Transfer the 3D coordinates to the camera's three-coordinate system. The formula for transferring 3D coordinates to camera four is as follows:

[0044]

[0045] in, This refers to the rotation and translation matrices obtained in step S1. Where R... 23 This refers to the rotation matrix from camera two to camera three, R 24 This refers to the rotation matrix from camera two to camera four, T 23 This refers to the translation matrix from camera two to camera three, T 24 This refers to the translation matrix from camera two to camera four.

[0046] 3D coordinates are coordinates relative to their respective camera coordinate systems. This completes the calibration of the position of the light spot and the camera. The calibrated position can be applied to the actual 3D line-of-sight estimation to improve the accuracy of line-of-sight estimation.

[0047] This application provides a system calibration method for a VR / AR head-mounted display. The method involves single-target calibration, spot detection, spot number correspondence processing, and binocular stereo matching to obtain the 3D coordinates of the spot. After coordinate system transformation, the accurate positional relationship between the VR / AR head-mounted display's camera and its infrared LED is obtained. Furthermore, based on the different RGB three-channel pixel values ​​captured by the auxiliary camera, false spots are filtered out. The center position of the spot is obtained by weighting the values ​​based on the average proportion of the RGB three channels, effectively filtering false spots and improving the accuracy of the spot center position.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A system calibration method for a VR / AR head-mounted display, characterized in that, include: The calibration datasets of the auxiliary camera and the infrared camera are processed to obtain the translation and rotation matrices between the auxiliary camera and the infrared camera, and between the auxiliary cameras. The infrared camera is set on the VR / AR head-mounted display. The infrared light images captured by the auxiliary camera are subjected to preliminary connected component extraction, and the preliminary connected components are filtered to obtain filtered connected components that meet preset conditions. Calculate the infrared light image corresponding to the filtered connected component, and calculate the ratio of the sum of pixel values ​​in the R channel to the sum of pixel values ​​in the G channel of the infrared light image. Filter out the filtered connected components whose ratio is less than a first threshold to obtain the selected connected components. Calculate the pixel mean at each coordinate in the infrared light image based on the selected connected components, and determine the center of all light spots in the infrared light image based on the pixel mean: Starting from the spot center with the largest y-value among all spot centers in any two sets of infrared light images, sort all spot centers in any two sets of infrared light images, and process the spot numbers of all spot centers captured by all auxiliary cameras accordingly to finally obtain two new spot center sets. Calculate the 3D coordinates of each new spot center set in the coordinate system of the auxiliary camera; The 3D coordinates of the new light spot center set are transformed into the coordinate system corresponding to the infrared camera to complete the system calibration of the VR / AR head-mounted display.

2. The system calibration method for a VR / AR head-mounted display as described in claim 1, characterized in that, The auxiliary camera and the infrared camera each consist of two.

3. The system calibration method for a VR / AR head-mounted display as described in claim 1, characterized in that, The calibration datasets of the auxiliary camera and the infrared camera are processed to obtain translation and rotation matrices between the auxiliary camera and the infrared camera, and between the auxiliary cameras. The infrared camera is mounted on the VR / AR head-mounted display. The coordinates of the checkerboard corner points in the calibration dataset acquired by the auxiliary camera are extracted using the first function in OpenCV. Using the calibration intrinsic parameter matrix, distortion coefficient, and checkerboard corner coordinates in the calibration dataset of the auxiliary camera and the infrared camera, stereo calibration of the auxiliary camera and the infrared camera is performed through the second function in OpenCV, thereby obtaining the translation and rotation matrices between the auxiliary camera and the infrared camera, and between the auxiliary cameras.

4. The system calibration method for a VR / AR head-mounted display as described in claim 1, characterized in that, The preliminary connected component extraction of the infrared light image captured by the auxiliary camera, and the filtering of the preliminary connected components to obtain filtered connected components that meet preset conditions, includes: The infrared light image is converted into a grayscale image, and the grayscale image is binarized using the third function in OpenCV to obtain a binary image. The fourth function extracts connected components from the binary image, and the extracted connected components are then filtered to obtain filtered connected components that meet preset conditions.

5. The system calibration method for a VR / AR head-mounted display as described in claim 4, characterized in that, Filtered connected component pixel values ​​that do not meet the preset conditions are 0, while filtered connected component pixel values ​​that meet the preset conditions are 255.

6. The system calibration method for a VR / AR head-mounted display as described in claim 1, characterized in that, The ratio is calculated using the following formula: in, Let w represent the position of the top-left corner of the circumscribed rectangle of the connected region, and let h represent the width and height of the connected region, respectively. These represent the infrared light images in... The pixel values ​​of the R channel and the G channel at the location.

7. The system calibration method for a VR / AR head-mounted display as described in claim 1, characterized in that, The pixel mean of each coordinate in the infrared light image is calculated based on the selected connected components, and the centers of all light spots in the infrared light image are determined based on the pixel mean using the following formula: in, Indicates the infrared light image in The average pixel value of the RGB three channels at the location, the subscript 1 indicates the infrared light image number, and the superscript 0 indicates the 0th selected connected component; This represents the scores of the three RGB channels of the 0th selected connected component in the infrared lamp image. The subscript 1 indicates the infrared lamp image number, and the superscript 0 indicates the 0th selected connected component. Indicates the center of all light spots in the infrared light image; This represents the pixel values ​​of the connected components that meet the preset conditions. The subscript 1 indicates the infrared light image number, and the superscript 0 indicates the 0th selected connected component. There are a total of n selected connected components. x and y represent the infrared light image in the image. The x and y coordinates of the location.

8. The system calibration method for a VR / AR head-mounted display as described in claim 1, characterized in that, The calculation of the 3D coordinates of each new light spot center set in the coordinate system of the auxiliary camera includes: The new light spot center set is distorted using single-target positioning parameters, and then the 3D coordinates of the new light spot center set in the coordinate system of the auxiliary camera are calculated using a binocular stereo algorithm.

9. A system calibration method for a VR / AR head-mounted display as described in claim 1, characterized in that, Transforming the 3D coordinates of the new light spot center set to the coordinate system corresponding to the infrared camera includes: Based on the 3D coordinates of the new light spot center set in the coordinate system of the auxiliary camera, and the translation and rotation matrices between the auxiliary camera and the infrared camera, the 3D coordinates of the new light spot center set are transformed into the coordinate system of the corresponding infrared camera.

Images