Calibration method, calibration device, display equipment and storage medium

Abstract

The invention discloses a calibration method, a calibration device, a display device and a storage medium, the display device comprises a plane mirror, a calibration camera, an eye movement camera and a display module, and the calibration method adjusts a relative position relationship between an imaging assembly and the calibration camera so that the imaging assembly is aligned with a lens cone and an image plane of the calibration camera; establishing a coordinate system on the lens cone; the position of the display module is adjusted according to the pose of the imaging assembly, so that the virtual image plane of the display module is aligned with the image plane; collecting a plurality of images of the lens cone at different positions through the eye movement camera; and calculating the relative pose between the display module and the eye movement camera according to the image, the internal reference of the eye movement camera and the coordinate system. According to the calibration method, in the calibration process, the user does not depend on the control point on the display module subjectively, the interference of the user on the precision is greatly reduced, the precision consistency of different users is higher, and the user experience is effectively improved.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and more particularly, to a calibration method, a calibration device, a display device, and a storage medium. Background Art

[0002] In related technologies, for an eye movement device applied to near-eye head-mounted display devices such as virtual reality and mixed reality, the calibration is generally completed by having a human eye fixate on the positions of a certain number of points on the screen, that is, calibrating the coordinate system conversion relationship between the eye movement camera and the display module, including the calibration of the physiological angle of declination between the optical axis and the visual axis.

[0003] The accuracy of the above calibration method depends to a certain extent on the subjectivity of the user. That is to say, whether the human eye can stably fixate on the calibration points that appear on the screen determines the accuracy of a calibration, resulting in relatively low accuracy of this method, and even lower calibration accuracy for some users with visual impairments. In addition, this method requires the user to fixate on a relatively large number of calibration points, which places a high demand on the user's concentration and is likely to reduce the user experience. Summary of the Invention

[0004] Embodiments of this application provide a calibration method, a calibration device, a display device, and a storage medium.

[0005] The calibration method according to the embodiments of this application, where the display device includes a calibration camera, an eye movement camera, and a display module, and the calibration method includes:

[0006] Adjust the relative position relationship between the imaging component and the calibration camera so that the image plane of the imaging component is aligned with the lens barrel and the image plane of the calibration camera;

[0007] Establish a coordinate system on the lens barrel;

[0008] Adjust the position of the display module according to the pose of the imaging component so that the virtual image plane of the display module is aligned with the image plane;

[0009] Collect multiple images of the lens barrel at different positions through the eye movement camera;

[0010] Calculate the relative pose between the display module and the eye movement camera according to the images, the internal parameters of the eye movement camera, and the coordinate system.

[0011] In some embodiments, the imaging component includes a first calibration mark surface, the lens barrel of the calibration camera includes a second calibration mark surface, and the image plane of the calibration camera includes a third calibration mark surface;

[0012] The adjustment of the relative position relationship between the imaging component and the calibration camera so that the imaging component is aligned with the lens barrel and the image plane of the calibration camera includes:

[0013] Adjust the position and orientation of the imaging component so that the first calibration mark surface, the second calibration mark surface, and the third calibration mark surface coincide.

[0014] In some embodiments, the display module includes a fourth calibration mark surface. Adjusting the position of the display module so that the virtual image surface of the display module is parallel to the image surface includes:

[0015] Adjust the display module so that the third calibration mark surface and the fourth calibration mark surface coincide.

[0016] In some embodiments, the display device further includes a six-axis adjustment platform. The calibration method further includes:

[0017] Adjust the position and orientation of the imaging component and / or the display module by adjusting the six-axis adjustment platform.

[0018] In some embodiments, collecting multiple images of the lens barrel by the eye movement camera at different positions includes:

[0019] The eye movement camera collects multiple images of the lens barrel while the virtual image surface and the image surface are kept parallel.

[0020] In some embodiments, the calibration method further includes:

[0021] Use the Zhang calibration method to solve the internal parameters, where the internal parameters include at least one of focal length, principal point coordinates, and distortion coefficients.

[0022] In some embodiments, using the Zhang calibration method to solve the internal parameters includes:

[0023] Use the eye movement camera to collect multiple images of a checkerboard calibration board placed at different positions;

[0024] Extract the coordinates of the corner points in the checkerboard calibration board;

[0025] Use the coordinates of the corner points to solve the internal parameters.

[0026] In some embodiments, calculating the relative position and orientation between the display module and the eye movement camera according to the image, the internal parameters of the eye movement camera, and the coordinate system includes:

[0027] Use the image and the internal parameters to calibrate the external parameters of the lens barrel relative to the eye movement camera;

[0028] Use the external parameters to calculate the relative position and orientation between the display module and the eye movement camera.

[0029] In some embodiments, calibrating the external parameters of the lens barrel relative to the eye movement camera by using the image and the internal parameters includes:

[0030] Extracting key feature points of the image;

[0031] Obtaining the three-dimensional spatial coordinates of the key feature points in the coordinate system of the calibration camera and the two-dimensional coordinates on the eye movement camera image;

[0032] Calculating the external parameters of the lens barrel surface relative to the eye movement camera according to the three-dimensional spatial coordinates in the coordinate system of the calibration camera and the two-dimensional coordinates on the eye movement camera image.

[0033] In some embodiments, extracting the key feature points of the image includes:

[0034] Performing edge extraction on the image by using a multi-level edge detection algorithm to obtain edge lines;

[0035] Using an eight-connected domain to divide the edge lines and form connected domains, and simultaneously removing interfering connected domains with an area smaller than a preset threshold;

[0036] Extracting the straight lines existing in the image by using a Hough straight line detection algorithm, and finding four connected domains containing the most straight line points;

[0037] Respectively finding the points inside the four connected domains, and calculating the average value to obtain the key feature points;

[0038] Repeating to obtain the key feature points until four key feature points are obtained;

[0039] Forming a straight line with the two key feature points with the farthest distance among the four key feature points, connecting the remaining two key feature points as another straight line, and calculating the intersection point of the two straight lines as the fifth key feature point.

[0040] The calibration device of the display device proposed in the second aspect embodiment of the present invention, the display device includes a calibration camera, an eye movement camera and a display module, and the calibration device includes:

[0041] An adjustment module, configured to adjust the relative position relationship between the imaging component and the calibration camera so that the image plane of the imaging component is aligned with the lens barrel and the image plane of the calibration camera;

[0042] A establishment module, configured to establish a coordinate system on the lens barrel;

[0043] An adjustment module, adjusting the position of the display module according to the pose of the imaging component so that the virtual image plane of the display module is aligned with the image plane;

[0044] An acquisition module, configured to acquire multiple images of the lens barrel at different positions through the eye movement camera;

[0045] A calculation module, configured to calculate the relative pose between the display module and the eye movement camera according to the images, the internal parameters of the eye movement camera, and the coordinate system.

[0046] The display device provided by the third aspect embodiment of the present invention includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the processor implements the calibration method according to any one of the above claims.

[0047] The non-volatile computer-readable storage medium provided by the fourth aspect embodiment of the present invention stores a computer program. When the computer program is executed by a processor, the calibration method of the eye movement device according to any one of the above claims is implemented.

[0048] The calibration method, calibration device, display device, and storage medium provided by the embodiments of the present application adjust the positional relationship between the imaging component and the calibration camera, so that the image plane of the imaging component is parallel to the image plane of the calibration camera, and a coordinate system is established. Since the coordinate system is used for the subsequent calculation of the relative pose between the display module and the eye movement camera, at this time, the coordinate system established on the lens barrel is parallel to the coordinate system of the image plane of the calibration camera, which can effectively reduce the calculation difficulty. Then, the display module is adjusted so that the virtual image plane of the display module is parallel to the image plane of the calibration camera, thereby correcting the physiological squint angle so that the coordinate system of the display module is parallel to the coordinate system of the calibration camera, thereby reducing the calculation difficulty of the relative position between the virtual image plane of the display module and the eye movement camera. Then, using the images of the lens barrel surface collected by the eye movement camera and the internal parameters of the eye movement camera, the relative position between the virtual image plane of the display module and the eye movement camera is calculated to realize the calibration of the display device. Since the calibration process does not depend on the user's subjective fixation on the control points on the display module, it can greatly reduce the interference brought by the user himself to the accuracy, improve the calibration accuracy, and the accuracy consistency of different users is higher. In addition, the user only needs to fixate once to correct the physiological squint angle to complete the entire eye movement calibration work, effectively improving the user experience.

[0049] Additional aspects and advantages of the embodiments of the present application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the embodiments of the present application. Description of the Drawings

[0050] The above and / or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, where:

[0051] Figure 1It is a schematic flow chart of the calibration method according to the embodiments of the present application;

[0052] Figure 2 It is a schematic block diagram of the calibration device of the display device according to the embodiments of the present application;

[0053] Figure 3 It is a schematic block diagram of the display device according to the embodiments of the present application;

[0054] Figure 4 It is a schematic structural diagram of the display device when adjusting the relative position relationship between the imaging component and the calibration camera so that the lens barrel and the image plane of the imaging component and the calibration camera are aligned;

[0055] Figure 5 It is a schematic diagram of the adjustment process of the first calibration mark surface, the second calibration mark surface and the third calibration mark surface according to the embodiments of the present application;

[0056] Figure 6 It is a schematic structural diagram of the display device when adjusting the position of the display module according to the pose of the imaging component so that the virtual image plane of the display module is parallel to the image plane according to the embodiments of the present application;

[0057] Figure 7 It is a schematic flow chart of the calibration method according to some embodiments of the present application;

[0058] Figure 8 It is a schematic flow chart of solving the internal parameters by using the Zhang's calibration method according to some embodiments of the present application;

[0059] Figure 9 It is a schematic flow chart of calculating the relative position between the virtual image plane and the eye movement camera according to the image, the internal parameters of the eye movement camera and the coordinate system according to the embodiments of the present application;

[0060] Figure 10 It is a schematic flow chart of calibrating the external parameters of the lens barrel relative to the eye movement camera by using the image and the external parameters according to the embodiments of the present application;

[0061] Figure 11 It is a schematic flow chart of extracting the key feature points of the image according to the embodiments of the present application;

[0062] Figure 12 It is a schematic diagram of the acquisition method of five key feature points according to the embodiments of the present application.

[0063] Main element symbol description: display device 100, calibration device 10, adjustment module 11, establishment module 12, adjustment module 13, acquisition module 14, calculation module 15, processor 20, memory 30, imaging component 40, calibration camera 50, lens barrel 51, eye movement camera 60, display module 70. Detailed implementation manners

[0064] The embodiments of the present application will be further described below in conjunction with the accompanying drawings. The same or similar reference numerals in the drawings denote the same or similar elements or elements having the same or similar functions throughout.

[0065] In addition, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and should not be construed as a limitation of the present application.

[0066] In the present application, unless otherwise clearly specified and defined, the first feature being "on" or "under" the second feature may be that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. Moreover, the first feature being "above", "over" and "on" the second feature may be that the first feature is directly above or obliquely above the second feature, or simply means that the first feature has a higher horizontal height than the second feature. The first feature being "under", "below" and "beneath" the second feature may be that the first feature is directly below or obliquely below the second feature, or simply means that the first feature has a lower horizontal height than the second feature.

[0067] Please refer to Figure 1 , a calibration method for a display device according to an embodiment of the present application. The display device includes a calibration camera, an eye movement camera, and a display module. The calibration method includes:

[0068] Step 01: Adjust the relative position relationship between the imaging component and the calibration camera so that the imaging component is aligned with the lens barrel and the image plane of the calibration camera;

[0069] Step 02: Establish a coordinate system on the lens barrel;

[0070] Step 03: Adjust the position of the display module according to the pose of the imaging component so that the virtual image plane of the display module is aligned with the image plane;

[0071] Step 04: Collect multiple images of the lens barrel at different positions through the eye movement camera;

[0072] Step 05: Calculate the relative pose between the display module and the eye movement camera according to the images, the internal parameters of the eye movement camera, and the coordinate system.

[0073] Please refer to Figure 2, an embodiment of the present application also proposes a calibration device 10 for a display device 100. The display device 100 includes a calibration camera, an eye movement camera, and a display module. The calibration device 10 of the display device 100 includes an adjustment module 11, a coordinate system establishment module 12, an adjustment module 13, an acquisition module 14, and a calculation module 15. The adjustment module 11 is configured to adjust the relative position relationship between the imaging component and the calibration camera so that the imaging component is aligned with the lens barrel and the image plane of the calibration camera. The coordinate system establishment module 12 is configured to establish a coordinate system on the lens barrel. The adjustment module 13 is configured to adjust the position of the display module according to the pose of the imaging component so that the virtual image plane of the display module is aligned with the image plane. The acquisition module 14 is configured to acquire multiple images of the lens barrel at different positions through the eye movement camera. The calculation module 15 is configured to calculate the relative pose between the display module and the eye movement camera according to the images, the internal parameters of the eye movement camera, and the coordinate system.

[0074] Please refer to Figure 3 , an embodiment of the present application also proposes a display device 100, including a processor 20 and a memory 30. The memory 30 stores a computer program. When the computer program is executed by the processor 20, the processor 20 implements the calibration method of the display device 100 as described in any one of the above claims. Or, the processor 20 can be used to adjust the relative position relationship between the imaging component and the calibration camera so that the imaging component is aligned with the lens barrel and the image plane of the calibration camera, and establish a coordinate system on the lens barrel, and adjust the position of the display module according to the pose of the imaging component so that the virtual image plane of the display module is aligned with the image plane. The processor 20 can also be used to acquire multiple images of the lens barrel at different positions through the eye movement camera, and calculate the relative position between the virtual image plane and the eye movement camera according to the images, the internal parameters of the eye movement camera, and the coordinate system.

[0075] In the calibration method of the display device 100, the calibration device 10, and the display device 100 proposed in the embodiments of the present application, by adjusting the positional relationship between the imaging component and the calibration camera, the image plane of the imaging component is made parallel to the image plane of the calibration camera, and a coordinate system is established. Since the coordinate system is used for the calculation of the relative pose between the subsequent display module and the eye movement camera, at this time, the coordinate system established in the lens barrel is parallel to the coordinate system of the image plane of the calibration camera, which can effectively reduce the calculation difficulty. Then, the display module is adjusted so that the virtual image plane of the display module is parallel to the image plane of the calibration camera, thereby correcting the physiological squint angle so that the coordinate system of the display module is parallel to the coordinate system of the calibration camera, thereby reducing the calculation difficulty of the relative position between the virtual image plane of the display module and the eye movement camera. Then, using the image of the lens barrel surface collected by the eye movement camera and the internal parameters of the eye movement camera, the relative position between the virtual image plane of the display module and the eye movement camera is calculated to achieve the calibration of the display device 100. Since the calibration process does not rely on the user's subjective fixation on the control points on the display module, the interference caused by the user himself to the accuracy can be greatly reduced, the calibration accuracy is improved, and the accuracy consistency of different users is higher. In addition, the user only needs to fixate once to correct the physiological squint angle to complete the entire eye movement calibration work, effectively improving the user experience.

[0076] Please refer to Figures 1 to 6 , the display device 100 of this embodiment may be a head-mounted display or a non-head-mounted display, etc. Hereinafter, only the head-mounted display will be taken as an example for illustration.

[0077] The imaging component 40 is used to form an image of the lens barrel 51 and align the image of the lens barrel 51 with the image plane in the calibration camera 50. The imaging component 40 can select any device or equipment that can image, and can be specifically selected according to actual needs. In this embodiment, the imaging component 40 selects a plane mirror. The plane mirror is easy to obtain and has a low price, which is beneficial to simplifying the calibration method.

[0078] The alignment of the imaging component 40 with the lens barrel 51 and the image plane of the calibration camera 50 means that the angle between any two of the imaging plane of the imaging component 40, the plane where the image of the lens barrel 51 of the calibration camera 50 is located, and the image plane in the calibration camera 50 should be as small as possible. Among them, the larger the formed angle, the greater the calibration error. Preferably, the imaging plane of the imaging component 40 should be parallel to the plane where the image of the lens barrel 51 of the calibration camera 50 is located and the image plane in the calibration camera 50.

[0079] The calibration camera 50 is a camera used to determine the position and orientation of the eye movement camera 60, and is used to calculate parameters such as the internal and external parameters of the eye movement camera 60. These parameters are very important for the normal operation of the eye movement camera 60, so accurate calibration is required. Among them, the barrel 51 of the calibration camera 50 refers to the diaphragm surface of the camera lens. In a camera, the diaphragm and the imaging surface are usually parallel to each other. The diaphragm is a component of the camera lens system, and its function is to control the amount of light entering the lens by adjusting the size of the aperture, thereby affecting exposure and depth of field. The diaphragm is composed of a set of adjustable blades, which can be expanded and contracted to form a circular or polygonal aperture. The image plane of the calibration camera 50 refers to the plane where the image information obtained by the calibration camera 50 is located, and is usually also called the focal plane or image plane. It is the specific position where the lens system in the photography process focuses the light information of the scene to form an image. On this plane, the real image generated by the object through the optical system is captured or recorded.

[0080] The eye movement camera 60 is a special camera that can automatically take pictures when the user is looking at a certain target by tracking the movement of the user's eyes. It combines the devices of an eye tracker and a camera, and controls taking pictures by tracking the movement of the user's eyes. This camera can be used in various situations that require capturing the user's line of sight, such as human-computer interaction, virtual reality, advertising effect evaluation, etc.

[0081] The internal parameters of the eye movement camera 60 are the parameters of the internal attributes of the eye movement camera 60, such as the focal length of the lens, the position of the principal point coordinates (optical center), lens distortion, etc. These parameters have a crucial impact on the accuracy and precision of the eye movement camera 60, so precise calibration is required.

[0082] The display module 70 usually consists of a micro display and an optical system. The micro display is used to display images, and the optical system is used to project the images onto the user's retina to present clear virtual images. This device is usually used in virtual reality and augmented reality applications. Among them, the virtual image plane of the display module 70 refers to that the image is projected onto the user's retina through the optical system, and what the user sees is a virtual image rather than a real object. The alignment of the virtual image plane of the display module 70 with the image plane means that the angle between the virtual image plane and the image plane of the display module 70 should be as small as possible. Among them, the larger the formed angle, the greater the calibration error. Preferably, the virtual image plane and the image plane of the display module 70 should be parallel.

[0083] Please refer to Figure 4 and Figure 5 , in some embodiments, the imaging component 40 includes a first calibration marking surface, the barrel 51 of the calibration camera 50 includes a second calibration marking surface, and the image plane of the calibration camera 50 includes a third calibration marking surface. Step 01 includes:

[0084] 011: Adjust the pose of the imaging component so that the first calibration mark surface, the second calibration mark surface, and the third calibration mark surface coincide.

[0085] In some embodiments, sub-step 011 is implemented by the adjustment module 11. Or rather, the adjustment module 11 can be used to adjust the pose of the imaging component 40 so that the first calibration mark surface, the second calibration mark surface, and the third calibration mark surface coincide.

[0086] In some embodiments, the processor 20 can be used to adjust the pose of the imaging component 40 so that the first calibration mark surface, the second calibration mark surface, and the third calibration mark surface coincide.

[0087] Specifically, set a plane mirror as the imaging component 40, and set the first calibration mark surface on the plane mirror. Start the calibration camera 50 and set the viewfinder grid on the display image window to set the third calibration mark surface. Adjust the position of the plane mirror so that the first calibration mark surface coincides with the third calibration mark surface, thereby ensuring that the plane mirror is parallel to the camera image plane. Mark the second calibration mark surface on the lens barrel 51, and confirm that the marked second calibration mark surface overlaps with the first calibration mark surface and the third calibration mark surface. At this time, mark the coordinate axes on the lens barrel 51. Further, the first calibration mark surface and the second calibration mark surface are cross-shaped marking lines, and the third calibration mark surface is a "field"-shaped grid. After the second calibration mark surface overlaps with the first calibration mark surface and the third calibration mark surface, use the cross-shaped marking lines of the second calibration mark surface as the XY axes, and use the direction perpendicular to the outside of the lens barrel 51 as the Z axis to establish a coordinate system. With such a setting, a coordinate system can be set at the lens barrel 51 to make it parallel to the coordinate system at the image plane, which is convenient for subsequent calculations, reduces the calculation difficulty, and improves the calculation accuracy.

[0088] Please refer to Figure 6 , in some embodiments, the display module 70 includes a fourth calibration mark surface, and step 03 includes:

[0089] 031: Adjust the display module so that the third calibration mark surface and the fourth calibration mark surface coincide.

[0090] In some embodiments, sub-step 031 is implemented by the adjustment module 13. Or rather, the adjustment module 13 can be used to adjust the display module 70 so that the third calibration mark surface and the fourth calibration mark surface coincide.

[0091] In some embodiments, the processor 20 can be used to adjust the display module 70 so that the third calibration mark surface and the fourth calibration mark surface coincide.

[0092] Specifically, the imaging component 40 is removed and the display module 70 is set in its original position. The display module 70 is lit and the fourth calibration mark surface is projected and displayed. The calibration camera 50 is aligned with the lens center of the display module 70 to obtain the virtual image of the display module 70 in real time. The fourth calibration mark surface on the virtual image surface in the display module 70 is adjusted to overlap with the third calibration mark surface on the image surface of the calibration camera 50, so as to determine that the image surface of the calibration camera 50 is parallel to the virtual image surface of the display module 70. The fourth calibration mark surface is a grid in the shape of a Chinese character "tian". The overlap of the fourth calibration mark surface and the third calibration mark surface means that the crosses in the middle of the two grids in the shape of a Chinese character "tian" coincide. When the third calibration mark surface and the fourth calibration mark surface coincide, the image surface of the calibration camera 50 is parallel to the virtual image surface of the display module 70. That is to say, at this time, the coordinate system of the calibration camera 50 is parallel to the coordinate system of the display module 70. Such a setting is beneficial to simplify the subsequent calculation process and improve the calibration accuracy.

[0093] In some embodiments, the display device 100 further includes a six-axis adjustment platform. The calibration method of the display device 100 further includes:

[0094] 012: Adjust the pose of the imaging component and / or the display module by adjusting the six-axis adjustment platform.

[0095] In some embodiments, the processor 20 can be used to adjust the pose of the imaging component and / or the display module 70 by adjusting the six-axis adjustment platform.

[0096] Specifically, the six-axis adjustment platform is a device for high-precision positioning and adjustment. It has six degrees of freedom and can achieve complex three-dimensional space adjustment. It is commonly used in fields such as precision manufacturing, detection, and optical positioning. The advantages of the six-axis adjustment platform include high precision, high stability, and high flexibility. Using the six-axis adjustment platform to adjust the imaging component 40 and / or the display module 70 can achieve adjustment accuracy at the micron or even nanometer level, and can also achieve adjustment of any position and posture according to needs. In addition, the control algorithms and software of the six-axis adjustment platform are relatively mature and can meet the requirements of various complex applications.

[0097] In some embodiments, step 04 includes:

[0098] 041: The eye movement camera captures multiple images of the lens barrel in a state where the virtual image surface and the image surface are parallel.

[0099] In some embodiments, the sub-step 041 is implemented by the acquisition module 14. Or rather, the acquisition module 14 can be used for the eye movement camera 60 to capture multiple images of the lens barrel 51 in a state where the virtual image surface of the display module 70 is parallel to the image surface of the calibration camera 50.

[0100] In some embodiments, the processor 20 can be used to collect multiple images of the lens barrel 51 by the eye movement camera 60 when the virtual image plane of the display module 70 is parallel to the image plane of the calibration camera 50.

[0101] Specifically, when the virtual image plane of the display module 70 is parallel to the image plane of the calibration camera 50, the multiple images of the lens barrel 51 collected by the eye movement camera 60 can be regarded as the coordinate systems of the display module 70 and the calibration camera 50 being parallel when calculating the external parameters. Since the two are parallel, the positional relationship of the eye movement camera 60 relative to the image plane of the calibration camera 50 can be converted into the positional relationship of the eye movement camera 60 relative to the virtual image plane of the display module 70 through simple calculations, which is beneficial to reducing the calculation difficulty of the external parameters and improving the calculation accuracy.

[0102] Please refer to Figure 7 , in some embodiments, the calibration method of the display device 100 further includes:

[0103] Step 06: Use the Zhang calibration method to solve the internal parameters, and the internal parameters include at least one of the focal length, principal point coordinates, and distortion coefficients.

[0104] In some embodiments, the processor 20 can be used to solve the internal parameters using the Zhang calibration method, and the internal parameters include at least one of the focal length, principal point coordinates, and distortion coefficients.

[0105] Specifically, the Zhang calibration method is a method for calibrating a camera using a planar checkerboard. Using the Zhang calibration method to calculate the internal parameters of the eye movement camera 60 has high accuracy, and only one calibration board and one camera are required to complete the calibration. The method is simpler, and at the same time, compared with the self-calibration method, the Zhang calibration method has better robustness.

[0106] Please refer to Figure 8 , in some embodiments, Step 06 includes:

[0107] 061: Use the eye movement camera to collect multiple images of the checkerboard calibration board placed at different positions;

[0108] 062: Extract the coordinates of the corner points in the checkerboard calibration board;

[0109] 063: Use the coordinates of the corner points to solve the internal parameters.

[0110] In some embodiments, the processor 20 can be used to collect multiple images of the checkerboard calibration board placed at different positions by the eye movement camera 60; extract the coordinates of the corner points in the checkerboard calibration board; use the coordinates of the corner points to solve the internal parameters.

[0111] Specifically, the checkerboard calibration board is a special board used for camera calibration. It has a regular checkerboard pattern with black and white alternating colors, where the corner points in the checkerboard calibration board refer to the intersections of the checkerboard. The checkerboard calibration board plays an important role in the process of solving the internal parameters, which is conducive to improving the accuracy and precision of detection.

[0112] Please refer to Figure 9 , in some embodiments, step 05 includes:

[0113] 051: Calibrate the external parameters of the lens barrel relative to the eye movement camera using the image and internal parameters;

[0114] 052: Calculate the relative pose between the display module and the eye movement camera using the external parameters.

[0115] In some embodiments, sub-steps 051 and 052 are implemented by the calculation module 15. Or rather, the calculation module 15 can be used to calibrate the external parameters of the lens barrel 51 relative to the eye movement camera 60 using the image and internal parameters; calculate the relative pose between the display module 70 and the eye movement camera 60 using the external parameters.

[0116] In some embodiments, the processor 20 can be used to calibrate the external parameters of the lens barrel 51 relative to the eye movement camera 60 using the image and internal parameters; calculate the relative pose between the display module 70 and the eye movement camera 60 using the external parameters.

[0117] Specifically, the external parameters of the lens barrel 51 relative to the eye movement camera 60 refer to the position and orientation parameters of the lens barrel 51 relative to the eye movement camera 60. These parameters describe the relative position and orientation relationship between the lens barrel 51 and the eye movement camera 60, including rotation and translation parameters. Through calibration and the calibration process, the external parameters can be determined and used to correct image distortion, improve positioning accuracy, and achieve better visual effects. Using the internal parameters in combination with the external parameters makes it easier to calculate the relative pose between the display module 70 and the eye movement camera 60.

[0118] Please refer to Figure 10 , in some embodiments, sub-step 051 includes:

[0119] 0511: Extract the key feature points of the image;

[0120] 0512: Obtain the three-dimensional spatial coordinates of the key feature points in the coordinate system of the calibration camera and the two-dimensional coordinates on the eye movement camera image;

[0121] 0513: Calculate the external parameters of the lens barrel surface relative to the eye movement camera based on the three-dimensional spatial coordinates in the coordinate system of the calibration camera and the two-dimensional coordinates on the eye movement camera image.

[0122] In some embodiments, sub-steps 0511, 0512, and 0513 are implemented by the computing module 15, or rather, the computing module 15 can be used to extract key feature points of an image; obtain the three-dimensional spatial coordinates of the key feature points in the coordinate system of the calibration camera 50 and the two-dimensional coordinates on the image of the eye movement camera 60; calculate the external parameters of the surface of the lens barrel 51 relative to the eye movement camera 60 according to the three-dimensional spatial coordinates in the coordinate system of the calibration camera 50 and the two-dimensional coordinates on the image of the eye movement camera 60.

[0123] In some embodiments, the processor 20 can be used to extract key feature points of an image; obtain the three-dimensional spatial coordinates of the key feature points in the coordinate system of the calibration camera 50 and the two-dimensional coordinates on the image of the eye movement camera 60; calculate the external parameters of the surface of the lens barrel 51 relative to the eye movement camera 60 according to the three-dimensional spatial coordinates in the coordinate system of the calibration camera 50 and the two-dimensional coordinates on the image of the eye movement camera 60.

[0124] Specifically, five key feature points are extracted from each image, and the five key feature points are sorted in the order of middle, upper, lower, left, and right using pixel coordinates, corresponding to the three-dimensional spatial coordinates in the coordinate system of the calibration camera 50, and the relative pose relationship between the coordinate system of the calibration camera 50 and the coordinate system of the eye movement camera 60 is solved. Then, key point extraction is performed on all images respectively, and the relative pose relationship is solved to obtain the relative pose results of all images.

[0125] Among them, the relative pose includes rotation and translation components. According to the characteristics of image acquisition, for images acquired at different positions, the rotation component of the coordinate system of the calibration camera 50 and the coordinate system of the eye movement camera 60 does not change, only the translation component changes. Therefore, the pose calculation results of multiple images can be used to determine anomalies based on the rotation component, that is, images with abnormal rotation components are either excluded or retained, and then the average is taken as the final result.

[0126] Since the image plane of the calibration camera 50 is parallel to the virtual image plane of the display module 70, and the coordinate system of the calibration camera 50 is parallel to the coordinate system of the display module 70, the rotation relationship between the eye movement camera 60 and the calibration camera 50 is the same as the rotation relationship between the eye movement camera 60 and the display module 70. Additionally, according to the characteristics of eye tracking technology, the three-dimensional line of sight calculated by the eye tracking algorithm module is usually a three-dimensional vector relative to the eye movement camera 60, and only the relative rotation matrix between the eye movement camera 60 and the display module 70 is needed to convert the line of sight to the coordinate system of the display module 70. In this way, the external parameters of the eye movement camera 60 obtained do not depend on the user's subjective fixation on the control points on the display module 70, greatly reducing the interference of human factors on the accuracy and improving the calibration accuracy.

[0127] Please refer to Figure 11 andFigure 12 , in some embodiments, sub-step 0511 includes:

[0128] 05111: Use a multi-level edge detection algorithm to extract edges from the image to obtain edge lines;

[0129] 05112: Use an eight-connected domain to divide the edge lines and form connected domains, and at the same time eliminate interfering connected domains with an area smaller than a preset threshold;

[0130] 05113: Use the Hough line detection algorithm to extract the lines existing in the image and find four connected domains containing the most line points;

[0131] 05114: Find the points inside the four connected domains respectively, and calculate the average value to obtain the key feature points;

[0132] 05115: Repeat obtaining the key feature points until four key feature points are obtained;

[0133] 05116: Form a straight line with the two key feature points that are the farthest apart among the four key feature points, connect the remaining two key feature points as another straight line, and calculate the intersection point of the two straight lines as the fifth key feature point.

[0134] In some embodiments, sub-steps 05111, 05112, 05113, 05114, 05115, and 05116 are implemented by the calculation module 15. Or rather, the calculation module 15 can be used to extract edges from the image using a multi-level edge detection algorithm to obtain edge lines; use an eight-connected domain to divide the edge lines and form connected domains, and at the same time eliminate interfering connected domains with an area smaller than a preset threshold; use the Hough line detection algorithm to extract the lines existing in the image and find four connected domains containing the most line points; find the points inside the four connected domains respectively, and calculate the average value to obtain the key feature points; repeat obtaining the key feature points until four key feature points are obtained; form a straight line with the two key feature points that are the farthest apart among the four key feature points, connect the remaining two key feature points as another straight line, and calculate the intersection point of the two straight lines as the fifth key feature point.

[0135] In some embodiments, the processor 20 can be used to extract edges from an image using a multi-level edge detection algorithm to obtain edge lines; divide the edge lines using an eight-connected domain to form connected domains, and at the same time eliminate interfering connected domains with an area smaller than a preset threshold; use the Hough line detection algorithm to extract the lines existing in the image, and find four connected domains containing the most line points; respectively find the points inside the four connected domains, calculate the average value to obtain key feature points; repeat to obtain key feature points until four key feature points are obtained; form a straight line from the two key feature points with the farthest distance among the four key feature points, connect the remaining two key feature points as another straight line, and calculate the intersection point of the two straight lines as the fifth key feature point.

[0136] Specifically, the multi-level edge detection algorithm is a commonly used edge detection method. Its advantage is that it can improve the accuracy and precision of edge detection, and at the same time can effectively remove the influence of noise and detail information. In this embodiment, the Canny operator is selected as the multi-level edge detection algorithm.

[0137] The eight-connected domain means that starting from each pixel in the region, through the movement combination in eight directions (up, down, left, right, upper left, upper right, lower left, lower right), without exceeding the region, any pixel in the region can be reached. It is usually used in applications such as edge detection and image segmentation.

[0138] The Hough line detection algorithm is an algorithm used to detect lines in an image. For a straight line, its representation form in the Hough space is a point. Under a certain threshold, the more points connected to it, the higher the possibility of the straight line in the image.

[0139] Respectively finding the points inside the four connected domains means that in the selected four connected domains, for each connected domain, extract the points closest to the innermost part of the connected domain or the points near the innermost part of the connected domain. The innermost point of the connected domain refers to the point that is farthest from all boundary points in the connected domain.

[0140] The embodiment of the present invention also proposes a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by the processor 20, the calibration method of the eye movement device as described in any one of the above claims is implemented.

[0141] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0142] In the description of this specification, the description with reference to the terms "certain embodiments", "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiments or examples. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0143] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" can explicitly or implicitly include at least one of the features. In the description of the present application, "a plurality" means at least two, such as two, three, unless otherwise specifically and clearly defined.

[0144] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A calibration method for a display device, characterized in that, The display device includes a calibration camera, an eye movement camera, and a display module. The calibration method is used for calibrating between the display module and the eye movement camera, and the calibration method includes: Adjust the relative position relationship between the imaging component and the calibration camera so that the image plane of the imaging component is aligned with the lens barrel and the image plane of the calibration camera; Establish a coordinate system on the lens barrel; Adjust the position of the display module according to the pose of the imaging component so that the virtual image plane of the display module is aligned with the image plane; Collect multiple images of the lens barrel at different positions through the eye movement camera; Calculate the relative pose between the display module and the eye movement camera according to the images, the internal parameters of the eye movement camera, and the coordinate system.

2. The calibration method according to claim 1, wherein The imaging component includes a first calibration marking surface, the lens barrel of the calibration camera includes a second calibration marking surface, and the image plane of the calibration camera includes a third calibration marking surface; The adjusting the relative position relationship between the imaging component and the calibration camera so that the imaging component is aligned with the lens barrel and the image plane of the calibration camera includes: Adjust the pose of the imaging component so that the first calibration marking surface, the second calibration marking surface, and the third calibration marking surface coincide.

3. The calibration method according to claim 2, wherein The display module includes a fourth calibration marking surface. The adjusting the position of the display module so that the virtual image plane of the display module is aligned with the image plane includes: Adjust the display module so that the third calibration marking surface and the fourth calibration marking surface coincide.

4. The calibration method according to claim 1, characterized in that The display device further includes a six-axis adjustment platform, and the calibration method further includes: Adjust the pose of the imaging component and / or the display module through the six-axis adjustment platform.

5. The calibration method according to claim 1, wherein The collecting multiple images of the lens barrel at different positions through the eye movement camera includes: The eye movement camera collects multiple images of the lens barrel in a state where the virtual image plane and the image plane are parallel.

6. The calibration method according to claim 1, wherein The calibration method further includes: Use Zhang's calibration method to solve the internal parameters, and the internal parameters include at least one of focal length, principal point coordinates, and distortion coefficients.

7. The calibration method according to claim 1, characterized in that, The calculating the relative pose between the display module and the eye movement camera according to the images, the internal parameters of the eye movement camera, and the coordinate system includes: Calibrate the external parameters of the lens barrel relative to the eye movement camera using the images and the internal parameters; Calculate the relative pose between the display module and the eye movement camera using the external parameters.

8. The calibration method according to claim 7, wherein The calibrating the external parameters of the lens barrel relative to the eye movement camera using the images and the internal parameters includes: Extract the key feature points of the images; Obtain the three-dimensional spatial coordinates of the key feature points in the coordinate system of the calibration camera and the two-dimensional coordinates on the image of the eye movement camera; Calculate the external parameters of the lens barrel surface relative to the eye movement camera according to the three-dimensional spatial coordinates in the coordinate system of the calibration camera and the two-dimensional coordinates on the image of the eye movement camera.

9. The calibration method according to claim 8, wherein The extracting the key feature points of the images includes: Use a multi-level edge detection algorithm to perform edge extraction on the images to obtain edge lines; Use an eight-connected domain to divide the edge lines and form connected domains, and at the same time eliminate the interference connected domains with an area smaller than a preset threshold. The Hough line detection algorithm is used to extract the existing lines in the image, and four of the connected regions containing the most line points are found; Points inside the four connected regions are found respectively, and the average value is calculated to obtain the key feature points; Repeat to obtain the key feature points until four key feature points are obtained; A straight line is formed by the two key feature points with the farthest distance among the four key feature points, and the remaining two key feature points are connected as another straight line, and the intersection of the two straight lines is calculated as the fifth key feature point.

10. A calibration device for a display device, characterized in that, The display device includes a calibration camera, an eye movement camera and a display module, and the calibration device includes: An adjustment module for adjusting the relative position relationship between the imaging component and the calibration camera so that the image plane of the imaging component is aligned with the lens barrel and the image plane of the calibration camera; A building module for establishing a coordinate system on the lens barrel; An adjustment module for adjusting the position of the display module according to the pose of the imaging component so that the virtual image plane of the display module is aligned with the image plane; An acquisition module for acquiring multiple images of the lens barrel at different positions through the eye movement camera; A calculation module for calculating the relative pose between the display module and the eye movement camera according to the image, the internal parameters of the eye movement camera and the coordinate system.

11. A display device, characterized in that, It includes a processor and a memory, and the memory stores a computer program. When the computer program is executed by the processor, the processor realizes the calibration method of the eye movement device according to any one of claims 1-9.

12. A non-volatile computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the calibration method of the eye movement device according to any one of claims 1-9 is realized.

Images