Functioning target and calibration machine system

Abstract

The application provides a functional calibration plate and a calibration machine system, which can avoid the influence of pancake shape and color change on the positioning accuracy of the pancake center, and is beneficial to improve the calibration accuracy of interpupillary distance. The functional calibration plate comprises: a calibration plate having a mark point; and a calibration tool fixedly installed with the calibration plate and used for detachably connecting to a pancake eye movement module of a device to be calibrated, so as to enhance the pancake center of the pancake eye movement module through the mark point of the calibration plate.

Description

Technical Field

[0001] This application relates to the field of eye-tracking technology, and in particular to a functional target and calibration system. Background Technology

[0002] In recent years, in order to enhance immersion and improve interactive experience, eye tracking is becoming a new standard feature of XR (Extended Reality) head-mounted displays (such as AR glasses or VR glasses). This is because eye tracking, as an interaction method that extracts the characteristics of human eye movement and estimates the user's gaze (focus), changes the gaze from "see first, then choose" to "see and choose immediately," optimizing interaction efficiency and greatly improving the user experience.

[0003] Currently, eye tracking based on the pupil-corneal reflection (PCR) method has become a common and efficient algorithm. By analyzing the characteristics of reflected light from the eye's surface and combining this with the capture of pupil and corneal reflection points by an ET (Eye-Tracking) camera, it can accurately measure the eye's movement trajectory. In this eye-tracking system, the ET camera intrinsic parameters provide the conversion between 2D image feature information (such as pupil outline or LED reflective spot) and 3D feature information in the physical coordinate system (such as the 3D center of the pupil or cornea). The position of the LED (light-emitting diode) light source in the physical coordinate system serves as a known input to the eye-tracking system, participating together with the ET camera intrinsic parameters in calculating the 3D model information of the human eye (such as pupil center coordinates or corneal center coordinates), ultimately determining the accuracy of the 3D gaze tracking calculation.

[0004] In common interpupillary distance (IPD) calibration schemes, binocular auxiliary cameras are typically used to directly capture images of the left and right pancake (folded optical path) eye-tracking modules of an XR headset. The center of the pancake is then located by extracting its outer contour, thus calibrating the IPD adjustment mechanism of the XR headset. However, the accuracy of extracting the pancake's outer contour can fluctuate due to changes in the pancake's shape and color, affecting the accuracy of the pancake's center location. This makes this method of calibrating IPD by extracting the pancake's outer contour prone to instability. Utility Model Content

[0005] One advantage of this application is that it provides a functional standard plate and calibration system that can avoid the impact of changes in the shape and color of the pancake on the positioning accuracy of the pancake center, which is beneficial to improving the accuracy of interpupillary distance calibration.

[0006] Another advantage of this application is that it provides a functional calibration plate and calibration machine system. In one embodiment of this application, the functional calibration plate can ensure that the interpupillary distance calibration method can extract the center of the pancake more stably, which increases the universality of the interpupillary distance calibration method for application scenarios and facilitates its promotion and application.

[0007] Another advantage of this application is that it provides a functional calibration plate and calibration system that does not require a complex structure to achieve the above objectives. Therefore, this application successfully and effectively provides a solution that not only provides a simple functional calibration plate and calibration system, but also increases the practicality and reliability of the system.

[0008] To achieve at least one of the above advantages or other benefits and objectives of this application, this application provides a functional label, comprising:

[0009] Calibration plate, with marked points; and

[0010] A calibration fixture is used to fix the calibration plate and to detachably connect it to the pancake eye-tracking module of the device to be calibrated, so as to enhance the pancake center of the pancake eye-tracking module through the marking points of the calibration plate.

[0011] In one embodiment of this application, the calibration fixture is fixed to the pancake eye-tracking module by magnetic attraction or clamping.

[0012] In one embodiment of this application, the calibration fixture is a magnetically attached calibration plate frame located around the calibration plate; the shape of the magnetically attached calibration plate frame matches the pancake lens frame of the pancake eye-tracking module, and is used to magnetically fix it to the pancake lens frame so that the center of the calibration plate corresponds to the center of the pancake of the pancake eye-tracking module.

[0013] In one embodiment of this application, the magnetic label frame is made of magnetic material.

[0014] In one embodiment of this application, the magnetic calibration plate frame includes a non-magnetic frame fixedly connected to the calibration plate and a magnetic component fixedly disposed in the non-magnetic frame.

[0015] In one embodiment of this application, a plurality of the magnetic elements are spaced apart around the calibration plate.

[0016] In one embodiment of this application, the calibration board is a checkerboard calibration board with dot markings, a general checkerboard calibration board, a dot calibration board, a QR code calibration board, or a Charuco calibration board.

[0017] According to another aspect of this application, a further provision of this application is a calibration system, comprising:

[0018] The hardware structural unit includes a support platform, a motion platform mounted on the support platform, a calibration plate clamp disposed on the motion platform, an equipment clamp disposed on the support platform for holding the device to be calibrated, a functional calibration plate for being held by the calibration plate clamp, and a binocular auxiliary camera disposed on the support platform; and

[0019] The software algorithm unit includes a motion control module communicatively connected to the motion platform, an image acquisition control module communicatively connected to the binocular auxiliary camera, and a calibration algorithm module communicatively connected to the binocular auxiliary camera.

[0020] In one embodiment of this application, the hardware structure unit further includes a supplementary light source disposed on the support platform; the software algorithm includes a supplementary light control module communicatively connected to the supplementary light source.

[0021] In one embodiment of this application, the supplementary light source is a ring-shaped light strip located between the binocular auxiliary camera and the device fixture. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the structure of a functional label according to an embodiment of this application;

[0023] Figure 2 A schematic diagram of the structure of the reflective sphere in the functional label according to the above embodiments of this application is shown;

[0024] Figure 3 A schematic diagram of the structure of the planar label in the functional label panel according to the above embodiments of this application is shown;

[0025] Figure 4 This is a flowchart illustrating an external parameter calibration method according to an embodiment of this application;

[0026] Figure 5 A flowchart illustrating the external parameter calibration steps in the external parameter calibration method according to the above embodiments of this application is shown;

[0027] Figure 6 A flowchart illustrating the extrinsic parameter verification step in the extrinsic parameter calibration method according to the above embodiments of this application is shown.

[0028] Figure 7 This is a block diagram of a calibration system according to an embodiment of this application;

[0029] Figure 8 This is an example of a functional label according to a modified embodiment of this application;

[0030] Figure 9 Another example of a functional panel according to the above-described modified embodiments of this application is shown;

[0031] Figure 10 This is a flowchart illustrating an embodiment of the pupil distance calibration method according to this application;

[0032] Figure 11 A schematic flowchart of the pupil distance calibration steps in the pupil distance calibration method according to the above embodiments of this application is shown;

[0033] Figure 12 An example of the pupil distance adjustment step in the pupil distance calibration method according to the above embodiments of this application is shown;

[0034] Figure 13 Another example of the pupil distance adjustment step in the pupil distance calibration method according to the above embodiments of this application is shown;

[0035] Figure 14 This is a block diagram of a calibration system according to a modified embodiment of the present application.

[0036] Explanation of key component symbols:

[0037] 1. Calibration system; 10. Functional calibration plate; 11. Calibration fixture; 110. Magnetic calibration plate frame; 111. Non-magnetic frame; 112. Magnetic component; 12. Reflective ball; 13. Flat calibration plate; 14. Calibration plate; 20. Support platform; 30. Motion platform; 40. Calibration plate fixture; 50. Equipment fixture; 60. Software algorithm unit; 61. Motion control module; 62. Image acquisition control module; 63. Calibration algorithm module; 64. Verification algorithm module; 65. Supplemental lighting control module; 70. Supplemental lighting source; 80. Binocular auxiliary camera.

[0038] The above description of the main component symbols, together with the accompanying drawings and specific embodiments, provides a further detailed explanation of this application. Detailed Implementation

[0039] The following description is intended to disclose this application and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of this application defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of this application.

[0040] In the description of this application, it should be understood that terms such as "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this application, it should be noted that, unless otherwise expressly specified and limited, terms such as "connected" or "linked" should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral connection; it can refer to a mechanical connection or an electrical connection; it can refer to a direct connection or an indirect connection through a medium. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0041] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0042] Considering that the ET camera and / or ET light source (such as LED beads) of the pancake eye-tracking module in XR near-eye devices are usually embedded in the pancake lens structure, the LED light emission is easily blocked by the structure and cannot be emitted to areas beyond the module's suitable eye distance range; and existing extrinsic parameter calibration methods cannot simultaneously directly photograph / co-examine all built-in LED beads using binocular auxiliary cameras, causing such LED extrinsic parameter calibration schemes to fail. Based on this, this application creatively proposes a functional calibration plate, an extrinsic parameter calibration method, and a calibration machine system, which can improve the accuracy of extrinsic parameter calibration while reducing the complexity of calibration operations and improving the requirements for the calibration environment.

[0043] Specifically, refer to the accompanying drawings in the specification of this application. Figures 1 to 3According to one embodiment of this application, a functional calibration plate 10 is provided, which may include a calibration fixture 11 and one or more reflective spheres 12, wherein the reflective spheres 12 are fixed to the calibration fixture 11, and at least a hemispherical portion of the reflective spheres 12 protrudes from the surface of the calibration fixture 11; the reflectivity of the reflective spheres 12 is greater than or equal to 60%. In this way, the reflective spheres 12 on the functional calibration plate 10 of this application can simulate the human eye, and use the ET camera of the device to be calibrated to photograph the light spot formed by the ET light source of the device to be calibrated on the reflective spheres 12 to calibrate the position of the ET light source relative to the ET camera (i.e., the extrinsic parameters of the light source of the device to be calibrated). This not only makes the calibration environment as close as possible to the actual use environment, but also solves the problem that it is impossible to directly photograph after the light source is built in. It is understood that the calibration device mentioned in this application may be a single pancake eye-tracking module or an XR device including two pancake eye-tracking modules (left and right); in addition, the ET light source mentioned in this application may, but is not limited to, be implemented as LED beads built into or externally placed in the pancake eye-tracking module.

[0044] More specifically, the radius of the reflective sphere 12 is between 6 mm and 9 mm to better mimic the human eye. Preferably, the radius of the reflective sphere 12 is equal to 7.8 mm to more closely approximate the radius of curvature of the human cornea, thereby better simulating the human eye. It is understood that in other examples of this application, the radius of the reflective sphere 12 may be greater than or less than 7.8 mm, as long as it ensures that clear and well-shaped reflected light spots of all ET light sources are obtained on the reflective sphere 12. Furthermore, the reflective sphere 12 of this application may be made of bright steel, ceramic, or other known materials capable of forming significant LED reflections.

[0045] It is worth noting that for all eye-tracking modules using ET light sources, the functional label 10 of this application can be placed within the eye-tracking module's suitable distance range (i.e., near the eyebox range), and the reflected light spot image of the ET light source can be acquired through the ET camera of the eye-tracking module. For example... Figure 2 As shown, since the same ET light source generates multiple light paths reflected into the ET camera at different positions of the reflector 12 to form light spots, the intersection of the incident light rays (i.e., the position of the ET light source) can be found by tracing the multiple reflected light rays R1, R2, R3, etc., obtained by the ET camera in reverse, thereby achieving extrinsic parameter calibration. In this way, during the extrinsic parameter calibration process, the reflector 12 can simulate the human eye receiving the reflection of the ET light source within the appropriate eye distance range, which can avoid the disadvantage of common external auxiliary camera direct shooting of ET light source being easily blocked.

[0046] Furthermore, during the extrinsic parameter calibration process, converting the coordinates of the reflected light spot extracted from the light spot image into the reflected ray R requires knowledge of the intrinsic parameters of the ET camera. Therefore, extrinsic parameter calibration of the eye-tracking module is typically performed after the intrinsic parameter calibration of the ET camera. The extrinsic parameter calibration method based on the reflector sphere in this application can be combined with different ET camera intrinsic parameter calibration methods, including but not limited to traditional calibration methods based on different camera models such as pinhole models, fisheye models, spherical projection models, and asymmetric models; or it can be combined with a light path table method that directly measures the direction of incident light rays in front of the camera lens without optimizing the fitted mathematical model.

[0047] It is important to note that, as is readily apparent from the principle of spherical reflection, obtaining the reflected light spot for calibrating extrinsic parameters requires the radius and position of the reflective sphere 12 as input for the calculation. Therefore, this application designs different system schemes based on the method of obtaining the position of the reflective sphere 12 on the functional marker 10 in the ET camera coordinate system. These include systems that determine the position of the reflective sphere 12 based on high-precision structural alignment and motion control, or systems that estimate the position of the reflective sphere 12 based on binocular stereo vision. The required features of the functional marker 10 differ for each system scheme.

[0048] For example, such as Figure 2 As shown, the functional calibration plate 10 contains three reflective spheres 12, which are arranged at intervals on the same side of the calibration fixture 11 to ensure that the ET light source can form a reflected light spot on each reflective sphere 12, which can then be captured by the ET camera to obtain the corresponding light spot image. It is understood that the number of reflective spheres 12 in the functional calibration plate 10 can also be other, specifically selected based on a comprehensive consideration of the ET camera's field of view, shooting distance, and calibration efficiency index (UPH); theoretically, the more reflective spheres 12 there are, the higher the calibration efficiency index. Furthermore, when the number of reflective spheres in the functional calibration plate 10 is greater than or equal to two, the functional calibration plate 10 can also be reused in the verification stage of external parameter calibration results.

[0049] Preferably, such as Figure 2 As shown, the three reflective balls 12 are arranged in a triangle on the side of the calibration fixture 11 in order to reduce the calibration plate area of ​​the functional calibration plate 10 and ensure that the functional calibration plate 10 can fall completely within the field of view of the ET camera.

[0050] Optionally, the reflective ball 12 is magnetically fixed to the calibration fixture 11 for replacement with a new reflective ball 12. It is understood that in other examples of this application, the reflective ball 12 can also be glued or clamped to the calibration fixture 11, such as by applying adhesive to a groove in the calibration fixture 11, by applying adhesive to the end of a connecting rod in the calibration fixture 11, or by using grippers to fix the reflective ball 12 to the calibration fixture 11, as long as at least a hemispherical portion of the reflective ball 12 is exposed outside the calibration fixture 11.

[0051] It is worth noting that, such as Figure 1 and Figure 3 As shown, the functional calibration plate 10 may further include a planar calibration plate 13 fixed to the calibration fixture 11, and the correspondence between the feature point positions on the planar calibration plate 13 and the center position of the reflective sphere 12 is known. Thus, the functional calibration plate 10 is implemented as a composite calibration plate, which, during external parameter calibration, only requires the use of a binocular auxiliary camera to photograph the planar calibration plate 13 of the functional calibration plate 10 to visually estimate the position of the reflective sphere 12, without needing to ensure high-precision alignment and leveling of the functional calibration plate 10 with the pancake of the device to be calibrated.

[0052] Optionally, such as Figure 1 As shown, the planar target plate 13 and the reflective ball 12 are located on opposite sides of the calibration fixture 11, with the feature pattern of the planar target plate 13 facing away from the reflective ball 12. Thus, during extrinsic parameter calibration, the binocular auxiliary camera and the device to be calibrated are spaced apart, and the functional target plate 10 is moved between the binocular auxiliary camera and the device to be calibrated, such that the reflective ball 12 of the functional target plate 10 faces the device to be calibrated, and the planar target plate 13 of the functional target plate 10 faces the binocular auxiliary camera. It is not necessary to ensure that the target plate is aligned with the pancake or leveled, thereby further reducing the complexity of the calibration operation. It is understood that in other examples of this application, the planar target plate 13 and the reflective ball 12 may also be located on the same side of the calibration fixture 11, which will not be elaborated further in this application.

[0053] Optionally, the feature pattern of the planar calibration plate 13 is implemented as a checkerboard pattern with dot markings, that is, black and white dots are added to the central area of ​​the general checkerboard pattern. This can solve the problem of poor eye image quality caused by a certain degree of occlusion around the field of view when the ET camera is built in, so as to improve the disadvantage of the general checkerboard pattern requiring all corner points to be detected. It is understood that in other examples of this application, the feature pattern of the planar calibration plate 13 can also be implemented as a general checkerboard pattern, a dotted calibration plate pattern, a QR code pattern, or a Charuco calibration plate pattern, etc.

[0054] It is worth mentioning that, according to another aspect of this application, such as Figure 4 As shown, one embodiment of this application further provides an external parameter calibration method, which may include the steps of:

[0055] S110: Control the motion platform to move the functional target plate to the initial calibration point, so that the reflective ball in the functional target plate is within the field of view of the ET camera of the device to be calibrated, and the planar target plate in the functional target plate is within the field of view of the binocular auxiliary camera.

[0056] S120: Control the ET light source of the device to be calibrated to be lit, and control the ET camera of the device to be calibrated to take pictures of the reflective spheres at different calibration points to obtain calibration spot images at each calibration point, wherein the calibration spot images include the spot images formed by the ET light source on the surface of the reflective spheres by the ET camera.

[0057] S130: Control the ET light source of the device to be calibrated to turn off, and control the binocular auxiliary camera to take pictures of the planar target plate at different calibration points to obtain characteristic pattern images at each calibration point.

[0058] S140: When the ET camera and the binocular auxiliary camera have completed image acquisition at the initial calibration point, the motion platform is further controlled to move the functional target plate to the next calibration point, ensuring that the reflective ball in the functional target plate remains within the field of view of the ET camera of the device to be calibrated, and that the planar target plate in the functional target plate remains within the field of view of the binocular auxiliary camera; and

[0059] S150: Call the extrinsic calibration algorithm to process multiple calibration spot images and multiple feature pattern images to obtain the ET light source extrinsic parameters of the device to be calibrated.

[0060] It is worth noting that, such as Figure 5 As shown, step S150 of the external parameter calibration method of this application may include the following steps:

[0061] S151: Based on the feature pattern image at each calibration point and the extrinsic parameters of the binocular auxiliary camera, solve the extrinsic parameters of the functional calibration plate relative to the binocular auxiliary camera to obtain the center coordinates of the reflective ball in the coordinate system of the device to be calibrated.

[0062] S152: Based on the calibration spot image at each calibration point and the intrinsic parameters of the ET camera, extract the pixel coordinates of the spot to obtain the reflection vector of each spot;

[0063] S153: Based on the center coordinates and radius of the reflective sphere, the coordinates of the reflection point on the reflective sphere corresponding to each reflection vector are solved according to the principle of intersection of a straight line and a sphere, so as to locate the reflection point of each ET light source on the reflective sphere;

[0064] S154: Based on the coordinates of the center of the reflective sphere in the coordinate system of the device to be calibrated, the coordinates of the reflection point of the ET light source on the reflective sphere, and the corresponding reflection vector, the incident vector of the ET light source on the reflective sphere is calculated according to the law of reflection; and

[0065] S155: Based on the incident vector of the ET light source at different calibration points, the coordinates of the ET light source in the coordinate system of the device to be calibrated are obtained by fitting and optimization, so as to obtain the extrinsic parameters of the ET light source of the device to be calibrated.

[0066] It is understood that the coordinate system of the device to be calibrated mentioned in this application may refer to the ET camera coordinate system, such as a coordinate system constructed with the center of the ET camera lens as the origin, the lens end face as the xOy plane, and the direction of its optical axis away from the photosensitive element in the ET camera as the positive z-axis. Alternatively, the device coordinate system may also be constructed based on the display screen, pancake lens group, or LED light ring of the eye-tracking module in the XR head-mounted display device, as long as it can effectively calculate the 3D position of each feature point on the characterization plate. This application will not elaborate further on this.

[0067] Furthermore, in step S130 of this application, the supplementary light source of the binocular auxiliary camera is controlled to be lit, so that the feature pattern area of ​​the planar target plate captured by the binocular auxiliary camera has uniform brightness and the feature points are clear and distinct. In addition, the supplementary light source mentioned in this application can be the ET light source of the binocular auxiliary camera, or it can be an independent light source arranged outside the binocular auxiliary camera, which will not be elaborated further in this application.

[0068] It is worth mentioning that after completing the extrinsic parameter calibration process based on the reflective sphere, the extrinsic parameter calibration method of this application can also realize a fully automated verification process to complete the closed loop of the system calibration process and ensure the accuracy of the calibration results. Specifically, such as Figure 6 As shown, the external parameter calibration method of this application further includes the following steps:

[0069] S210: Move the functional marker to the verification point so that all the reflective balls of the functional marker are within the eye-friendly distance range of the device to be calibrated;

[0070] S220: Control the ET light source of the device to be calibrated to light up, and control the ET camera of the device to be calibrated to take a picture of the reflective ball at the verification point to obtain the verification spot image of the reflective ball at the verification point.

[0071] S230: Based on the intrinsic parameters of the ET camera and the calibrated extrinsic parameters of the ET light source, the extrinsic parameter verification algorithm is called to process the verification spot image to obtain the center-to-center distance between multiple reflective spheres; and

[0072] S240: Compare the error between the center distance of the verification ball and the corresponding actual center distance of the ball to verify the accuracy of the external parameter calibration results.

[0073] It is worth noting that the verification process in the extrinsic parameter calibration method of this application is similar to the principle of solving the corneal center in the eye-tracking algorithm, which can fully explain the impact of the extrinsic parameter calibration results on the algorithm, so as to accurately evaluate and judge the accuracy level of the extrinsic parameter calibration results.

[0074] It is worth mentioning that, according to another aspect of this application, such as Figure 7 As shown, one embodiment of this application further provides a calibration system 1, which may include a hardware structure unit and a software algorithm unit 60; the hardware structure unit includes a support platform 20, a motion platform 30 mounted on the support platform 20, a target plate clamp 40 disposed on the motion platform 30, a device clamp 50 disposed on the support platform 20 for clamping the device to be calibrated, a functional target plate 10 for being clamped by the target plate clamp 40, and a binocular auxiliary camera 80 disposed on the support platform 20; the software algorithm unit 60 includes a motion control module 61 communicatively connected to the motion platform 30, and a device clamp 50 for communicating with the device to be calibrated and the binocular auxiliary camera 80. The device 80 is communicatively connected to an image acquisition control module 62 and a calibration algorithm module 63; the motion control module 61 is used to control the motion platform 30 to move the functional target plate 10, so that the functional target plate 10 moves to multiple calibration points; the image acquisition control module 62 is used to control the device to be calibrated and the binocular auxiliary camera 80 to acquire the reflective sphere image and the planar target plate image of the functional target plate 10 respectively, so as to obtain multiple calibration spot images and multiple feature pattern images at different calibration points; the calibration algorithm module 63 is used to perform image processing on the multiple calibration spot images and multiple feature pattern images at different points to obtain the ET light source extrinsic parameters of the device to be calibrated.

[0075] In particular, such as Figure 1 and Figure 7 As shown, the functional target plate 10 includes a calibration fixture 11, one or more reflective spheres 12, and a flat target plate 13 fixed to the calibration fixture 11. The reflective spheres 12 are fixed to the calibration fixture 11, and at least a hemispherical portion of the reflective sphere 12 protrudes from the surface of the calibration fixture 11. The correspondence between the feature point positions on the flat target plate 13 and the center position of the reflective sphere 12 is known.

[0076] More specifically, such as Figure 7As shown, the motion control module 61 controls the motion platform 30 to move the functional target plate 10 to the verification point; the image acquisition control module 62 controls the device to be calibrated to acquire the image of the reflective sphere of the functional target plate 10 to obtain the verification spot image of the reflective sphere at the verification point. The software algorithm unit 60 also includes a verification algorithm module 64, which performs image processing on the verification spot image based on the ET camera intrinsic parameters of the device to be calibrated and the calibrated ET light source extrinsic parameters to obtain the verification sphere center distance between multiple reflective spheres; and when the error between the verification sphere center distance and the actual sphere center distance meets the index requirements, the calibrated ET light source extrinsic parameters are written into the device to be calibrated as the extrinsic parameter calibration result.

[0077] It is worth noting that, since the calibration system of this application only needs to move the functional target plate 10 along the optical axis during calibration and verification, unlike the traditional Zhang's calibration algorithm which requires rotating the target plate, this reduces the complexity of the calibration operation and improves the requirements for the calibration environment. Therefore, although the calibration system of this application adopts a whole-machine calibration mode, the placement of the target plate is not interfered with by the temple structure of the XR head-mounted display, ensuring that the camera's field of view can completely cover the target plate, thus improving the calibration accuracy. At the same time, in application scenarios with large distortion, the calibration system of this application can achieve higher and more stable calibration accuracy than the Zhang's calibration method, and can better solve the problem of extrinsic parameter calibration when the distortion model is unknown. In other words, the calibration system of this application can well support the calibration of ET cameras and / or ET light sources in scenarios where the optical path passes through pancake lenses / diopter lenses, without significantly affecting the calibration accuracy.

[0078] In addition, such as Figure 7 As shown, in order to further ensure that the feature points of the planar target 13 on the functional target 10 are clear and distinct, the hardware structure unit of this application may also include a supplementary light source 70 disposed on the support platform 20; correspondingly, the software algorithm unit 60 includes a supplementary light control module 65 communicatively connected to the supplementary light source 70, which is used to configure supplementary light parameters according to the sampling point information of the current target, so as to make the brightness of the image acquired by the binocular auxiliary camera 80 uniform.

[0079] For example, the calibration system 1 of this application controls the hardware structure unit through the software algorithm unit 60, which can realize the fully automated calibration process. The specific operation steps are as follows:

[0080] 1) Fix the XR near-eye device / single pancake eye-tracking module onto the device fixture 50, connect the power cord and data cable, and turn on the ET light source LED;

[0081] 2) The calibration software remotely starts the ET camera image acquisition service on the device side and sends out the settings for the ET camera exposure parameters;

[0082] 3) The calibration software remotely starts the image acquisition service of the binocular auxiliary camera on the device side and sends out the settings for the exposure parameters of the binocular auxiliary camera;

[0083] 4) The motion control module 61 controls the motion platform 30 to move the functional target to the preset initial calibration point within the suitable eye distance range of the XR near eye device / single pancake eye-tracking module, so that the reflective ball and the planar target in the functional target are respectively within the field of view of the ET camera and the binocular auxiliary camera.

[0084] 5) The calibration software control device's image acquisition APP acquires calibration spot images through the ET camera to ensure that the number of corresponding LED spots on the reflective sphere captured by the ET camera is complete, the brightness is uniform, and the shape is good, and then uploads them to the designated storage directory on the PC.

[0085] 6) The calibration software configures the supplementary lighting parameters of the supplementary light source 70 according to the sampling point information of the current target plate, so that the feature pattern on the planar target plate captured by the binocular auxiliary camera has uniform brightness and clear feature points. The image is then uploaded to the specified storage directory on the PC.

[0086] 7) The motion control module 61 controls the motion platform 30 to move the functional target plate according to the preset points. The point setting only requires the target plate to move and change within the viewing distance range of the ET camera, without ensuring horizontal alignment; repeat steps 5) and 6) above to obtain calibration spot images at multiple calibration points.

[0087] 8) Based on multiple calibration spot images, the calibration software executes the calibration algorithm to obtain the LED extrinsic parameters;

[0088] 9) If the device to be calibrated is divided into left and right eyes, the motion control module 61 controls the motion platform 30 to move the functional target plate to the preset initial calibration point in front of the pancake eye-tracking module on the other side, and repeat steps 4) to 7 above.

[0089] As described above regarding the sampling process, this system does not require high-precision structural motion assurance. The calibration pose is calculated by using a binocular auxiliary camera to capture images of the planar calibration plate. Combined with the calibration plate's processing parameters, the 3D physical coordinates of the center of each reflective sphere in each calibration spot image within the binocular camera coordinate system are easily determined. Then, by extracting the pixel positions of each reflected spot in the calibration spot image using image algorithms, the LED extrinsic parameter calculation process can be constructed. Compared to calibration methods based on spherical calibration plates, the entire solution process is equivalent to using a binocular coordinate system. After solving all LED extrinsic parameters, fitting the lamp ring plane allows the LED extrinsic parameters to be converted back to the lamp ring plane, thus reverting the calibration results back to the coordinate system of the device being calibrated. This facilitates subsequent use by the eye-tracking algorithm. The specific details are as follows:

[0090] 1) Input the target structure information, the intrinsic and extrinsic parameters of the binocular auxiliary camera, and 2N feature pattern images acquired by the binocular auxiliary camera. Perform feature extraction and feature corner point triangulation in sequence. First, calculate the pose of the target. Then, calculate the physical coordinates of the center of multiple reflective spheres in the binocular coordinate system point by point. Finally, based on the extrinsic parameter information between the binocular auxiliary camera and the coordinate system of the device to be calibrated, obtain the physical coordinates of the center of multiple reflective spheres in the coordinate system of the device to be calibrated.

[0091] 2) Input N calibration spot images acquired by the ET camera and the intrinsic parameters of the ET camera. First, extract features from the calibration spot images to obtain the pixel coordinates of the spot; then calculate the reflection vector of the LED light source based on the intrinsic parameters of the ET camera.

[0092] 3) Based on the light reflection vector, the position of the sphere's center, and the sphere's radius, the LED incident intersection point is solved using the line-sphere intersection equation;

[0093] 4) Based on the light reflection vector, the LED incident intersection point, the sphere radius, and the coordinates of the sphere center, solve for the incident light vector according to the law of reflection;

[0094] 5) Fit the incident light vectors obtained from solving the same LED at multiple reflector spheres and multiple calibration points, and use the intersection of the fitted points as the LED position;

[0095] 6) The optimizer uses a multi-round iterative optimization method to output the optimal LED position as the external parameter calibration result.

[0096] It is understood that the binocular coordinate system mentioned in this application refers to a coordinate system constructed based on the binocular auxiliary camera. Furthermore, the extrinsic parameter information between the binocular coordinate system and the coordinate system of the device to be calibrated mentioned in this application may refer to the RT information between the binocular auxiliary camera and the pancake lens of the XR headset.

[0097] It is worth noting that, for an XR head-mounted display device that includes two pancake-style eye-tracking modules as the calibration device, after calibrating the ET camera intrinsic parameters and the ET light source extrinsic parameters, it is also necessary to calibrate the interpupillary distance (IPD) of the device to be calibrated (i.e., the distance between the center points of the two pancake-style eye-tracking modules); that is, it is necessary to calibrate the correspondence between the IPD setting value of the IPD adjustment device in the device to be calibrated and the actual IPD value. It is understood that the IPD adjustment device mentioned in this application is typically implemented as a Hall motor; therefore, the IPD setting value mentioned in this application refers to the Hall value output by the Hall motor.

[0098] Common interpupillary distance (IPD) calibration methods typically locate the pancake center of the left and right pancake eye-tracking modules by directly extracting the pancake's outer contour to calibrate the IPD adjustment device of the XR head-mounted display. However, the accuracy of extracting the pancake's outer contour fluctuates due to changes in the pancake's shape and color, thus affecting the positioning accuracy of the pancake center. This leads to instability in this method of calibrating IPD by extracting the pancake's outer contour. To address this issue, one aspect of this application further provides a functional calibration plate and calibration system that avoids the impact of changes in the pancake's shape and color on the positioning accuracy of the pancake center, thereby improving the accuracy of IPD calibration.

[0099] Specifically, refer to the accompanying drawings in the specification of this application. Figure 8 and Figure 9 According to a modified embodiment of this application, a functional calibration plate 10 is provided, which may include a calibration fixture 11 and a calibration plate 14 with marking points. The calibration fixture 11 is fixedly mounted on the calibration plate 14 and is used for detachably connecting to the pancake eye-tracking module of the device to be calibrated, so as to enhance the pancake center of the pancake eye-tracking module through the marking points of the calibration plate 14.

[0100] It is worth noting that, since the functional calibration plate 10 of this application can be detachably connected to the pancake eye-tracking module of the device to be calibrated through the calibration fixture 11, so as to enhance the pancake center of the pancake eye-tracking module through the marker points of the calibration plate 14, during the process of calibrating the interpupillary distance of the device to be calibrated, it is only necessary to take a picture of the calibration plate 14 with a binocular auxiliary camera to obtain the calibration plate image, and calculate the calibration plate center of the calibration plate 14 by extracting the marker point features in the calibration plate image, so as to more stably locate the pancake center of the pancake eye-tracking module; and unlike the phenomenon of unstable calibration accuracy due to differences in pancake shape and color, which occurs when the outer contour of the pancake is extracted to locate the pancake center of the pancake eye-tracking module.

[0101] More specifically, the calibration fixture 11 is fixed to the pancake eye-tracking module using magnetic attraction. Thus, this application only needs to move the functional calibration plate 10 near the pancake eye-tracking module of the device to be calibrated, and the calibration fixture 11 of the functional calibration plate 10 and the pancake eye-tracking module can be automatically associated and fixed under magnetic attraction. It is understood that in other examples of this application, the calibration fixture 11 and the pancake eye-tracking module can also be detachably associated and fixed using methods such as clamping, which will not be elaborated upon here.

[0102] Optionally, such as Figure 8 and Figure 9 As shown, the calibration fixture 11 is implemented as a magnetically attached calibration frame 110 located around the calibration plate 14. The shape of the magnetically attached calibration frame 110 matches the pancake lens frame of the pancake eye-tracking module, and is used to magnetically fix it to the pancake lens frame so that the center of the calibration plate 14 corresponds to the center of the pancake of the pancake eye-tracking module. In this way, the distance between the centers of the two calibration plates 14 will always be consistent with the distance between the centers of the two pancakes of the pancake eye-tracking modules. This application can use the marking points of the calibration plate 14 to better enhance the center of the pancake of the pancake eye-tracking module, and completely eliminate the adverse effects on calibration accuracy caused by differences in pancake shape and color. It is understood that since the pancake eye-tracking module in the device to be calibrated mentioned in this application usually has its own magnetic lens interface for magnetically fixing such as diopter lenses or sunglasses lenses, the magnetically attached calibration frame 110 of this application can be magnetically fixed to the pancake lens frame of the pancake eye-tracking module without changing the structure of the device to be calibrated itself.

[0103] Exemplarily, in one example of this application, such as Figure 8 As shown, the magnetic label frame 110 can be made of a magnetic material. It is understood that the magnetic material mentioned in this application can be, but is not limited to, electrical steel, nickel-based alloys, rare earth alloys, or ferrite materials, as long as they can generate a magnetic attraction with the magnetic material. This application will not elaborate further on this.

[0104] In another example of this application, such as Figure 9As shown, the magnetic calibration plate frame 110 may also include a non-magnetic frame 111 fixedly connected to the calibration plate 14 and a magnetic element 112 fixed to the non-magnetic frame 111, so as to achieve magnetic fixation by the magnetic attraction force generated between the magnetic element 112 and the pancake frame of the pancake eye-tracking module. It is understood that the magnetic element 112 mentioned in this application may be implemented as a magnet or electromagnet, and the non-magnetic frame 111 mentioned in this application may be implemented as a plastic frame or a metal frame, but is not limited to that used in this application.

[0105] Optionally, such as Figure 9 As shown, multiple magnetic elements 112 are spaced apart around the calibration plate 14 to enhance the fixation strength between the magnetic calibration plate frame 110 and the pancake frame.

[0106] According to the above embodiments of this application, the calibration plate 14 can be implemented as a checkerboard calibration plate with dot markings, that is, black and white dots are added to the central area of ​​the general checkerboard pattern. This can solve the problem of incomplete calibration plate image caused by a certain degree of occlusion around the calibration plate 14, so as to improve the disadvantage that the general checkerboard pattern must detect all corner points. It is understood that in other examples of this application, the calibration plate 14 can also be implemented as a general checkerboard calibration plate, a dot calibration plate, a QR code calibration plate, or a Charuco calibration plate, etc.

[0107] It is worth mentioning that, according to another aspect of this application, such as Figure 10 As shown, one embodiment of this application further provides a pupil distance calibration method, which may include the following steps:

[0108] S310: Move a pair of functional markers to a pair of pancake eye-tracking modules of the device to be calibrated, so that the two functional markers are detachably connected to the two pancake eye-tracking modules respectively, in order to correspondingly enhance the pancake center of the two pancake eye-tracking modules;

[0109] S320: Sequentially adjust the interpupillary distance adjustment device of the device to be calibrated to different interpupillary distance settings so that the functional target plate can be moved synchronously through the pancake eye-tracking module;

[0110] S330: Controls the binocular auxiliary camera to capture images of two functional calibration boards when the interpupillary distance adjustment device is at various interpupillary distance settings, thereby obtaining multiple calibration board images corresponding to multiple interpupillary distance settings; and

[0111] S340: Call the interpupillary distance calibration algorithm to process multiple calibration board images in order to calibrate the interpupillary distance of the device to be calibrated.

[0112] It is worth noting that, such as Figure 11As shown, step S340 in the pupil distance calibration method of this application may include the following steps:

[0113] S341: Extract the marker point features of two functional calibration plates based on multiple calibration plate images;

[0114] S342: Based on the intrinsic and extrinsic parameters of the binocular auxiliary camera, the extracted marker point features are triangulated to calculate the pancake center coordinates of the two pancake eye-tracking modules under various interpupillary distance settings.

[0115] S343: Based on the pancake center coordinates of the two pancake eye-tracking modules at various interpupillary distance settings, calculate the Euclidean distance between the pancake centers of the two pancake eye-tracking modules at various interpupillary distance settings, and use this distance as multiple interpupillary distance calculation values ​​corresponding to each interpupillary distance setting; and

[0116] S344: Perform curve fitting on the corresponding pupil distance setting value and pupil distance calculation value to obtain the pupil distance coefficient of the device to be calibrated.

[0117] For example, in step S344 of the pupil distance calibration method of this application: with the pupil distance setting value (i.e., the Hall value of the Hall motor) as x and the pupil distance calculation value (i.e., the actual pupil distance value) as y, curve fitting is performed based on the corresponding pupil distance setting value and pupil distance calculation value to obtain the curve fitting equation y = ax + bx. 2 + cx 3 + dx 4 + C, where a, b, c, and d are the pupil distance coefficients of the device to be calibrated.

[0118] In addition, such as Figure 11 As shown, step S340 in the pupil distance calibration method of this application may further include the following steps:

[0119] S345: Substitute each pupil distance setting value into the curve equation constructed from the calibrated pupil distance coefficients to solve for multiple estimated pupil distance values ​​corresponding to each pupil distance setting value; and

[0120] S346: Calculate the root mean square error between the estimated pupil distance and the calculated pupil distance, and output the pupil distance coefficient of the device to be calibrated as the pupil distance calibration result when the error meets the requirements.

[0121] It is worth noting that, since the pupil distance adjustment device (such as a Hall motor) of the device to be calibrated in this application can adjust the pupil distance from the outside to the inside (i.e., adjust the pupil distance to a smaller value) or from the inside to the outside (i.e., adjust the pupil distance to a larger value), the pupil distance calibration method of this application can collect data for one direction, that is, calibrate only a single direction; correspondingly, when the calibrated device to be calibrated uses the pupil distance adjustment function, the pupil distance needs to be adjusted according to the calibration direction.

[0122] Exemplarily, in one example of this application, such as Figure 12 As shown, step S320 in the pupil distance calibration method of this application may include the following steps:

[0123] S321: Adjust the pupil distance setting value of the pupil distance adjustment device to the maximum pupil distance of the device to be calibrated; and

[0124] S322: When the binocular auxiliary camera completes image acquisition at the current interpupillary distance setting value, the interpupillary distance setting value of the interpupillary distance adjustment device is automatically reduced according to the specified step size until the interpupillary distance setting value of the interpupillary distance adjustment device is the minimum interpupillary distance of the device to be calibrated.

[0125] Thus, the pupillary distance calibration method of this application collects data by adjusting the pupillary distance from the outside to the inside, and then performs parameter fitting on the pupillary distance adjustment relationship in the direction from large to small, so that the pupillary distance calibration method of this application only calibrates the pupillary distance from large to small for the device to be calibrated; thereafter, when the user uses the pupillary distance adjustment function of the device to be calibrated, the pupillary distance of the device to be calibrated needs to be adjusted to the maximum first, and then the pupillary distance is adjusted according to the pupillary distance adjustment algorithm, that is, the pupillary distance of the device to be calibrated can only be adjusted from the outside to the inside.

[0126] In another example of this application, such as Figure 13 As shown, step S320 in the pupil distance calibration method of this application may also include the following steps:

[0127] S323: Adjust the pupil distance setting value of the pupil distance adjustment device to the minimum pupil distance value of the device to be calibrated; and

[0128] S324: When the binocular auxiliary camera completes image acquisition at the current interpupillary distance setting value, the interpupillary distance setting value of the interpupillary distance adjustment device is automatically increased according to the specified step size until the interpupillary distance setting value of the interpupillary distance adjustment device is the maximum interpupillary distance of the device to be calibrated.

[0129] Thus, the pupillary distance calibration method of this application collects data by adjusting the pupillary distance from the inside out, and then performs parameter fitting on the pupillary distance adjustment relationship in the direction from small to large, so that the pupillary distance calibration method of this application only calibrates the pupillary distance from small to large for the device to be calibrated; thereafter, when the user uses the pupillary distance adjustment function of the device to be calibrated, the pupillary distance of the device to be calibrated needs to be adjusted to the minimum first, and then the pupillary distance is adjusted according to the pupillary distance adjustment algorithm, that is, the device to be calibrated can only adjust the pupillary distance from the inside out.

[0130] It is worth noting that, since the pupil distance adjustment device of the device to be calibrated in this application has different trends in the two directions of changing from the outside to the inside and from the inside to the outside, the pupil distance calibration method of this application can also collect data according to the two methods of adjusting the pupil distance from the inside to the outside and from the outside to the inside, and then perform parameter fitting on the pupil distance adjustment relationship for the two directions of changing from small to large and from large to small, so that the pupil distance calibration method of this application calibrates the device to be calibrated in the two directions of changing from small to large and from large to small. This application will not elaborate further on this.

[0131] It is worth mentioning that, according to another aspect of this application, such as Figure 14 As shown, a modified embodiment of this application further provides a calibration machine system 1, which may include a hardware structure unit and a software algorithm unit 60. The hardware structure unit includes a support platform 20, a motion platform 30 mounted on the support platform 20, a target plate clamp 40 disposed on the motion platform 30, a device clamp 50 disposed on the support platform 20 for holding the device to be calibrated, a functional target plate 10 for being held by the target plate clamp 40, and a binocular auxiliary camera 80 disposed on the support platform 20. The software algorithm unit 60 includes a motion control module 61 communicatively connected to the motion platform 30, an image acquisition control module 62 communicatively connected to the binocular auxiliary camera 80, and a calibration algorithm module 63. The motion control module 61 controls the motion platform 30 to move the functional target plate 10, so that the two functional target plates 10 are detachably connected to the two pancake eye-tracking modules of the device to be calibrated. The image acquisition control module 62 sequentially adjusts the interpupillary distance adjustment device of the device to be calibrated to different interpupillary distance settings, so that the functional target plates 10 are moved synchronously through the pancake eye-tracking modules. It also controls the binocular auxiliary camera 80 to capture images of the two functional target plates 10 when the interpupillary distance adjustment device is at each interpupillary distance setting value, so as to obtain multiple calibration plate images corresponding to multiple interpupillary distance settings. The calibration algorithm module 63 calls the interpupillary distance calibration algorithm to process the multiple calibration plate images to calibrate the interpupillary distance of the device to be calibrated.

[0132] In particular, such as Figure 8 and Figure 14As shown, the functional calibration plate 10 includes a calibration fixture 11 and a calibration plate 14 with marking points. The calibration fixture 11 is fixedly mounted on the calibration plate 14 and is used to detachably connect to the pancake eye-tracking module of the device to be calibrated, so as to enhance the pancake center of the pancake eye-tracking module through the marking points of the calibration plate 14.

[0133] It is worth noting that the calibration system 1 of this application can take into account both external parameter calibration and pupil distance calibration without making any changes to the hardware structure unit. It only needs to select a matching functional calibration plate during calibration, thus realizing a closed loop of calibration effect and one-machine-station multi-functionality.

[0134] In addition, such as Figure 14 As shown, in order to further ensure that the marking points of the calibration board 14 on the functional calibration board 10 are clear and distinct, the hardware structure unit of this application may also include a supplementary light source 70 disposed on the support platform 20; correspondingly, the software algorithm unit 60 includes a supplementary light control module 65 communicatively connected to the supplementary light source 70, for configuring supplementary light parameters according to the sampling point information of the current calibration board, so as to make the brightness of the image acquired by the binocular auxiliary camera 80 uniform.

[0135] Optionally, the supplementary light source 70 is implemented as a ring-shaped light strip located between the binocular auxiliary camera 80 and the device fixture 50, so as to uniformly illuminate the calibration plate 14 of the functional calibration plate 10 while avoiding obstructing the field of view of the binocular auxiliary camera 80, ensuring that the binocular auxiliary camera 80 can acquire a uniform and complete calibration plate image.

[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are quite specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A functional label, characterized in that, include: The calibration plate has marking points; and A calibration fixture is used to fix the calibration plate and to detachably connect it to the pancake eye-tracking module of the device to be calibrated, so as to enhance the pancake center of the pancake eye-tracking module through the marking points of the calibration plate.

2. The functional label panel according to claim 1, characterized in that, The calibration fixture is fixed to the pancake eye-tracking module by magnetic attraction or clamping.

3. The functional label plate according to claim 1, characterized in that, The calibration fixture is a magnetic calibration plate frame located around the calibration plate; the shape of the magnetic calibration plate frame matches the pancake lens frame of the pancake eye-tracking module, and is used to magnetically fix it to the pancake lens frame so that the center of the calibration plate corresponds to the center of the pancake of the pancake eye-tracking module.

4. The functional label plate according to claim 3, characterized in that, The magnetic label frame is made of magnetic material.

5. The functional label plate according to claim 3, characterized in that, The magnetic calibration plate frame includes a non-magnetic frame fixedly connected to the calibration plate and a magnetic component fixed to the non-magnetic frame.

6. The functional label plate according to claim 5, characterized in that, Multiple magnetic elements are spaced apart around the calibration plate.

7. The functional label panel according to any one of claims 1 to 6, characterized in that, The calibration board can be a checkerboard calibration board with dots, a general checkerboard calibration board, a dot calibration board, a QR code calibration board, or a Charuco calibration board.

8. A calibration system, characterized in that, include: The hardware structure unit includes a support platform, a motion platform mounted on the support platform, a target plate clamp set on the motion platform, an equipment clamp set on the support platform for holding the device to be calibrated, a functional target plate for being held by the target plate clamp, and a binocular auxiliary camera set on the support platform. and The software algorithm unit includes a motion control module communicatively connected to the motion platform, an image acquisition control module communicatively connected to the binocular auxiliary camera, and a calibration algorithm module communicatively connected to the binocular auxiliary camera.

9. The calibration system according to claim 8, characterized in that, The hardware structure unit also includes a supplementary light source disposed on the support platform; the software algorithm includes a supplementary light control module communicatively connected to the supplementary light source.

10. The calibration system according to claim 9, characterized in that, The supplementary light source is a ring-shaped light strip located between the binocular auxiliary camera and the device fixture.

Images