Calibration Method and Calibration System for Optical Transmissive Display

Abstract

The present invention provides a calibration method for an optical transmissive display, comprising: using an image sensing device fixed at a standard position where a user's eyes view the optical transmissive display, wherein the image sensing device is used to capture a virtual image displayed by the optical transmissive display and a real image of the external environment; and calculating external parameters between the image sensing device and a virtual camera by using external parameters between the image sensing device and a plurality of real calibration patterns and external parameters between the virtual camera and the plurality of real calibration patterns. A calibration system is also proposed.

Description

Technical Field

[0001] The present invention relates to a calibration method and a calibration system for a display, and more particularly to a calibration method and a calibration system for an optical see-through display. Background Art

[0002] Augmented reality (AR) or mixed reality (MR) glasses project virtual images generated by an arithmetic unit into the field of view of a user through an optical see-through display, and the user can simultaneously see the virtual images and the physical environment to achieve a mixed reality in terms of visual perception. However, all the optical elements on the glasses are bound to have errors caused by manufacturing and assembly. In order to achieve an accurate virtual-real image fusion effect, all augmented reality or mixed reality glasses must generate relevant parameters through a calibration process before leaving the factory, so as to provide the arithmetic unit with optimized virtual images.

[0003] Most of the previous augmented reality or mixed reality glasses did not take into account the individual differences in the eye distance of users (i.e., the distance between the pupils of the two eyes), and used a predefined standard eye distance to calibrate the spatial relative relationship between the eyeballs and the display during the calibration process. When the eye distance of the user is significantly different from the standard eye distance defined by the manufacturer, the accuracy of virtual-real image fusion will be affected. Summary of the Invention

[0004] The present invention is directed to a calibration method for an optical see-through display, which can deduce the extrinsic parameters of the eyes and the corresponding optical see-through display, so that after the optical see-through display leaves the factory, it can generate optimized virtual-real fusion images suitable for the eye distance according to the pre-written extrinsic parameters and the set eye distance.

[0005] The present invention is directed to a calibration system, which can deduce the extrinsic parameters of the eyes and the corresponding optical see-through display, so that after the optical see-through display leaves the factory, it can generate optimized virtual-real fusion images suitable for the eye distance according to the pre-written extrinsic parameters and the set eye distance.

[0006] An embodiment of the present invention provides a calibration method for an optical transmissive display, comprising: using an image sensing device fixed at a standard position where a user's eyes view the optical transmissive display, wherein the image sensing device is used to capture a virtual image displayed by the optical transmissive display and a real image of the external environment; using the image sensing device to capture a virtual calibration pattern displayed by the optical transmissive display, and obtaining coordinate values of a plurality of first feature point images of the captured virtual calibration pattern; calculating a projection homography matrix for converting the image plane of the image sensing device to the image plane of the optical transmissive display using the coordinate values of the plurality of first feature point images and the coordinate values of the first feature points of the virtual calibration pattern; turning off the optical transmissive display, and using the image sensing device to capture a plurality of real calibration pattern images at different angles, wherein the real calibration pattern has a plurality of second feature points; calculating the internal parameters (intrinsic parameters) of the image sensing device and the external parameters between the image sensing device and the plurality of real calibration patterns using the coordinates of the plurality of second feature point images in the plurality of real calibration pattern images and the plurality of second feature points on the real calibration pattern; converting the coordinates of the plurality of feature point images in the plurality of real calibration pattern images to the coordinates of the projection points on the image plane of the optical transmissive display using the projection homography matrix; calculating the internal parameters of the virtual camera of the optical transmissive display and the external parameters between the virtual camera and the plurality of real calibration patterns using the coordinates of the plurality of projection points converted to the image plane of the optical transmissive display in the plurality of real calibration pattern images; and calculating the external parameters between the image sensing device and the virtual camera using the external parameters between the image sensing device and the plurality of real calibration patterns and the external parameters between the virtual camera and the plurality of real calibration patterns.

[0007] An embodiment of the present invention provides a calibration system for calibrating an optical transmissive display. The calibration system includes an image sensing device and a controller. The image sensing device is fixed at a standard position where a user's eyes view the optical transmissive display. The image sensing device is used to capture a virtual image displayed by the optical transmissive display and a real image of the external environment. The controller is coupled to the image sensing device and the optical transmissive display. The controller is configured to execute: commanding the image sensing device to capture a virtual calibration pattern displayed by the optical transmissive display and obtaining coordinate values of a plurality of first feature point images of the captured virtual calibration pattern; calculating a projective homography matrix for converting the image plane of the image sensing device to the image plane of the optical transmissive display using the coordinate values of the plurality of first feature point images and the coordinate values of the first feature points of the virtual calibration pattern; turning off the optical transmissive display and using the image sensing device to capture a plurality of real calibration pattern images at different angles, the real calibration pattern having a plurality of second feature points; calculating internal parameters of the image sensing device and external parameters between the image sensing device and the plurality of real calibration patterns using the coordinates of the plurality of second feature point images in the plurality of real calibration pattern images and the plurality of second feature points on the real calibration pattern; converting the coordinates of the plurality of feature point images in the plurality of real calibration pattern images to the coordinates of projected points on the image plane of the optical transmissive display using the projective homography matrix; calculating internal parameters of a virtual camera of the optical transmissive display and external parameters between the virtual camera and the plurality of real calibration patterns using the coordinates of the plurality of projected points converted to the image plane of the optical transmissive display in the plurality of real calibration pattern images; and calculating external parameters between the image sensing device and the virtual camera using the external parameters between the image sensing device and the plurality of real calibration patterns and the external parameters between the virtual camera and the plurality of real calibration patterns. In the calibration method and calibration system of the optical transmissive display according to the embodiments of the present invention, since the external parameters between the image sensing device and the virtual camera can be calculated using the above calibration steps, an optimized virtual-real fusion image suitable for the eye distance can be generated according to the pre-written external parameters and the set eye distance after the optical transmissive display leaves the factory. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, the above content and other objects, features and advantages of the present disclosure will become more clear. In the drawings:

[0009] Figure 1 It is a schematic optical path structure diagram of an optical transmissive display according to an embodiment of the present invention;

[0010] Figure 2 、 Figure 4 and Figure 5Schematic diagram of the architecture of the calibration system according to an embodiment of the present invention and schematic diagram of the flow of the calibration method of the optical transmissive display according to an embodiment of the present invention;

[0011] Figure 3 For Figure 2 Schematic diagram of the correspondence between the image captured by the image sensing device and the image plane of the optical transmissive display;

[0012] Figure 6 When the eye distance of the user changes Figure 1 Schematic diagram of the adjustment method of the optical transmissive display;

[0013] Figure 7 When the position of the user's eyes changes Figure 1 Schematic diagram of the adjustment method of the optical transmissive display. Detailed implementation manners

[0014] Now, reference will be made in detail to the exemplary embodiments of the present invention. Examples of the exemplary embodiments are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to denote the same or similar parts.

[0015] Figure 1 Schematic diagram of the optical path architecture of the optical transmissive display according to an embodiment of the present invention, and Figure 2 、 Figure 4 And Figure 5 Schematic diagram of the architecture of the calibration system according to an embodiment of the present invention and schematic diagram of the flow of the calibration method of the optical transmissive display according to an embodiment of the present invention. Figure 3 For Figure 2 Schematic diagram of the correspondence between the image captured by the image sensing device and the image plane of the optical transmissive display. Please refer to Figures 1 to 5 , the calibration system 200 of this embodiment (as shown in Figure 2 ) is used to calibrate the optical transmissive display 100 (as shown in Figure 1 ). The calibration system 200 includes an image sensing device 220 and a controller 210. The image sensing device 220 is fixed at the standard position where the user's eyes 50 (as shown in Figure 1 ) view the optical transmissive display 100. The image sensing device 220 is used to capture the virtual image 113 displayed by the optical transmissive display 100 and the real image of the external environment.

[0016] The optical transmissive display 100 can be various augmented reality displays. In one embodiment, the optical transmissive display 100 includes a virtual image projector 110, a beam splitter 120, and a partially transmissive and partially reflective mirror 130. The virtual image projector 110 may include a display panel 112 and at least one lens 114. The display panel 112 emits an image beam 111. The display panel 112 is, for example, an organic light-emitting diode (OLED) display panel, a liquid crystal display panel, a liquid-crystal-on-silicon panel (LCOS panel), a digital micro-mirror device (DMD), or other suitable display panel. The lens 114 is disposed on the transmission path of the image beam 111. The beam splitter 120 is disposed on the transmission path of the image beam 111 from the lens 114 and reflects at least a portion of the image beam 111 to the partially transmissive and partially reflective mirror 130. The beam splitter 120 can be a polarizing beam splitter (PBS) or a partially transmissive and partially reflective mirror. The partially transmissive and partially reflective mirror 130 is, for example, a semi-transmissive and semi-reflective curved mirror, which can reflect a portion of the image beam 111 back to the beam splitter 120, and at least a portion of this portion of the image beam 111 passes through the beam splitter 120 and is transmitted to the user's eye 50. In this way, the user's eye 50 will see a virtual image (i.e., the virtual image 113) in front of the partially transmissive and partially reflective mirror 130. On the other hand, a portion of the light 60 from an external object sequentially passes through the partially transmissive and partially reflective mirror 130 and the beam splitter 120 and is transmitted to the user's eye 50. In this way, the user can simultaneously see the virtual image 113 and the actual object in the outside world, achieving the effect of mixed reality (MR) or augmented reality (AR). Figure 1 The optical transmissive display 100 is just one example, and the present invention does not limit the architecture adopted by the optical transmissive display Figure 1 to be

[0017] The controller 210 is coupled to the image sensing device 220 and the optical transmissive display 100. In the present embodiment, the image sensing device 220 is, for example, a camera. The calibration method of the optical transmissive display 100 of the present embodiment can be performed using the calibration system 200. The calibration method of the optical transmissive display 100 includes using the image sensing device 220 to be fixed to a standard position where the user's eyes 50 view the optical transmissive display 100, wherein the image sensing device 220 is used to capture the virtual image 113 displayed by the optical transmissive display 100 and the physical image of the external environment. Then, the controller 210 is configured to perform the following steps, and the calibration method of the optical transmissive display 100 of the present embodiment also includes the following steps. First, as Figure 2 As shown, the controller 210 instructs the image sensing device 220 to capture the virtual calibration pattern 102 displayed on the optical transmissive display 100 and obtain coordinate values of a plurality of first feature point images of the captured virtual calibration pattern 102 .

[0018] For example, the image sensing device 220 can be a simulated human eye camera C, and the optical transmissive display 100 can project the virtual image 113 onto the optical transmissive display image plane p. The optical transmissive display image plane p can be considered to have a corresponding virtual camera D. The virtual camera D is used to simulate the virtual image projector 110 of the optical transmissive display 100. The internal parameter K of the simulated human eye camera C is C The next step is to find the internal parameters K of the virtual camera D. D And the external parameters between the simulated human eye camera C and the virtual camera D And verify its correctness. In one embodiment, the internal parameter K C With the internal parameter K D Including internal parameters such as focal length and principal point, while external parameters These include external parameters such as rotation matrices and translation vectors. In this example, these are the rotation matrices and translation vectors used to transform the coordinates of the simulated eye camera C to the virtual camera D, but the present invention is not limited thereto. The internal parameters mentioned in other sections of this specification all include internal parameters such as focal length and principal point, while the external parameters mentioned in other sections of this specification all include external parameters such as rotation matrices and translation vectors, but the present invention is not limited thereto.

[0019] First, refer to Figure 2 and Figure 3 , use the simulated human eye camera C to shoot the virtual calibration pattern 102 displayed on the optical transmissive display image plane p. The i-th first feature point in the simulated human eye camera C image is The i-th first feature point in the image plane p of the optical transmissive display is

[0020] Next, using the coordinate values of multiple first feature point images and the coordinate values of the first feature points of the virtual calibration pattern 102, the projection homography matrix for converting the image plane of the image sensing device 220 to the image plane p of the optical transmissive display is calculated. In one embodiment, the method for calculating the projection homography matrix for converting the image plane of the image sensing device 220 to the image plane p of the optical transmissive display is the minimum mean-square error (MMSE) method or the random sample consensus (RANSAC) method. For example, the homography matrix from the simulated human eye camera C to the image plane p of the optical transmissive display can be obtained using the minimum mean-square error method or the random sample consensus method.

[0021] Next, as Figure 4 shown, the optical transmissive display 100 is turned off, and the image sensing device 220 is used to capture multiple images of the physical calibration pattern 70 at different angles. The physical calibration pattern 70 has multiple second feature points. The physical calibration pattern 70 can be located on the physical calibration board P.

[0022] For example, when the optical transmissive display 100 is turned off, the simulated human eye camera C can directly capture the physical calibration board P, and capture j images of the physical calibration board P at different angles, where the physical calibration board P has the physical calibration pattern 70, M ij is the coordinate of the i-th second feature point on the j-th physical calibration board P position in the world coordinate system, is the coordinate of the i-th second feature point identified in the j-th calibration board image captured by the simulated human eye camera C in the image plane of the simulated human eye camera C.

[0023] After that, using the coordinate values of multiple second feature point images in multiple physical calibration pattern 70 images and the multiple second feature points on the physical calibration pattern 70, the internal parameters of the image sensing device 220 and the external parameters between the image sensing device 220 and the multiple physical calibration patterns 70 are calculated. In addition, the coordinate values of multiple feature point images in multiple physical calibration pattern 70 images are converted to the coordinate values of projection points on the image plane p of the optical transmissive display using the projection homography matrix. For example, is to use the homography matrix to convert to the coordinate values on the image plane of the virtual camera D

[0024] In addition, the internal parameters of the virtual camera D of the optical transmissive display 100 and the external parameters between the virtual camera D and the plurality of physical calibration patterns 70 are calculated by using the coordinates of the plurality of projection points converted to the image plane p of the optical transmissive display in the image of the plurality of physical calibration patterns 70. Furthermore, the external parameters between the image sensing device 220 and the virtual camera D are calculated by using the external parameters between the image sensing device 220 and the plurality of physical calibration patterns 70 and the external parameters between the virtual camera D and the plurality of physical calibration patterns 70. In one embodiment, the method for calculating the external parameters between the image sensing device 220 and the virtual camera D includes the Zhang-Zhengyou camera calibration algorithm.

[0025] For example, the Zhang-Zhengyou camera calibration algorithm (refer to the paper: "Zhang, Zhengyou. ″A flexible new technique for camera calibration.″ IEEE Transactions on pattern analysis and machine intelligence 22.11(2000): 1330-1334.") can be used, and the internal parameters K of the virtual camera D can be estimated from the internal parameters K of the virtual camera D can be estimated D and the external parameters between the simulated human eye camera C and the virtual camera D

[0026] Specifically, for the derivation of the Zhang-Zhengyou camera calibration algorithm, please refer to Figure 4 , K D (3×3) are the internal parameters of the virtual camera D (where 3×3 represents that K D is a 3×3 matrix), and are the rotation matrix and translation vector for converting the coordinate system of the physical calibration board P to the coordinate system of the virtual camera D (where 3×1 represents that is a 3×1 matrix, that is, a vector). According to the pinhole camera model and the coordinate system transformation matrix operation, the transformation relationship between the coordinate system of the physical calibration board P and the coordinate system of the virtual camera D can be expressed as Equation (1):

[0027]

[0028] where [x P y P z P 1] T is the homogeneous coordinate of one feature point on the plane of the physical calibration pattern 70 of the physical calibration board P, and [u v 1] Tis the homogeneous coordinate of the feature point projected onto the image plane of the virtual camera D. Since the physical calibration pattern 70 is a plane, z P = 0. Therefore, the homography matrix H for converting from the coordinate system of the physical calibration pattern 70 of the physical calibration plate P to the image plane coordinate system of the virtual camera D can be simplified to Equation (2):

[0029]

[0030] Let H = [h1 h2 h3], and Equation (2) can be rewritten as:

[0031] [h1 h2 h3] = λK D [r1 r2 t] (3)

[0032] By the properties of the rotation matrix, each row of the rotation matrix is orthogonal to each other and has a modulus of 1 (i.e., orthonormal), and Equations (4) and (5) can be derived:

[0033]

[0034]

[0035] The homography matrix H can be obtained from and as follows:

[0036]

[0037] where is the homography matrix from the physical calibration plate P to the simulated human eye camera C.

[0038] By Equations (4), (5), and (6), the internal parameter K D of the virtual camera D can be obtained using the Zhang Zhengyou calibration algorithm. Then, substituting K D into Equation (3), the external parameters between the virtual camera D and the physical calibration pattern 70 can be obtained. Adding the external parameters between the simulated human eye camera C and the physical calibration pattern 70 obtained using the pose estimation algorithm

[0039] Please refer to Figure 5, in one embodiment, the calibration system 200 further includes a tracking camera 230 coupled to the controller 210, and the controller 210 further performs the following steps. First, command the tracking camera 230 to capture a physical pattern of a known size (such as the physical calibration pattern 70), and obtain the external parameters between the tracking camera 230 and the physical pattern (such as the physical calibration pattern 70), where the tracking camera 230 and the image sensing device 220 have been calibrated, and obtain the internal parameters of the tracking camera 230 and the external parameters between the tracking camera 230 and the image sensing device 220. Then, convert the coordinates of the third feature points on the physical pattern (such as the physical calibration pattern 70) into the coordinates of the image captured by the tracking camera 230, then into the coordinates of the image captured by the image sensing device 220, then into the coordinates of the image of the virtual camera D, and finally into the coordinates of the virtual image (i.e., the virtual image 113 on the image plane p of the optical see-through display 100) displayed by the optical see-through display 100. The positions, third feature points, and standard positions (i.e., the standard positions where the eyes 50 view the optical see-through display 100, which is also the standard position where the image sensing device 220 is set) represented by them are on the same straight line in space to achieve the effect of fusing the virtual image 113 with the actual objects in the outside world. In one embodiment, the method for obtaining the external parameters between the tracking camera 230 and the physical pattern (such as the physical calibration pattern 70) is the pose estimation algorithm.

[0040] For example, in a usage scenario where it is necessary to detect actual objects for virtual-real image fusion, a tracking camera 230 can be added, marked here as the tracking camera T. On the premise that the tracking camera T and the simulated human eye camera C jointly complete the camera calibration procedure, the internal parameter K of the tracking camera T T , the internal parameter K of the simulated human eye camera C C and the external parameters between the two are all known. When the tracking camera T captures a pattern of a known size (such as the physical calibration pattern 70 of the physical calibration board P), the pose estimation algorithm can be used to calculate the external parameters between the two Convert the coordinates M of the i-th third feature point on the physical object (such as the physical calibration board P) i to the tracking camera coordinate system {T} as Then use the external parameters to convert to the simulated human eye camera coordinate system Then use the external parameters to convert to the virtual camera coordinate system {D} as Finally, use the internal parameter K of the virtual camera D obtained in the previous step DProject onto the virtual imaging plane (i.e., the image plane p of the optical see-through display) as which is the visual coincidence point between the simulated human eye camera C at the current position and M i . The relational expressions for the above conversion are as shown in the following equations (7) and (8):

[0041]

[0042]

[0043] Please refer to Figure 6 again. The controller 210 further performs the following steps: In response to a change in the user's interpupillary distance (i.e., the distance between the two eyes' pupils), a translation amount corresponding to the change in the interpupillary distance is added to the external parameters between the image sensing device 220 and the virtual camera D, and the position of the virtual image 113 displayed on the optical see-through display 100 is calculated based on the external parameters after adding this translation amount. In one embodiment, the direction of this translation amount is parallel to the direction of the line connecting the user's two eyes.

[0044] For example, when the interpupillary distance setting changes, taking the position at the time of calibration of the simulated human eye camera C as a reference, after the change in the interpupillary distance, it is equivalent to the simulated human eye camera C moving on the X C axis by At this time, only the external parameters between the simulated human eye camera C and the virtual camera D need to be corrected to and applied to equation (8), then the position of the simulated human eye camera at the C' position can be calculated. In this way, the virtual image 113 displayed on the optical see-through display 100 can be adaptively adjusted according to different interpupillary distances, thus ensuring the accuracy of the virtual and real image fusion. In this embodiment, the X C axis is, for example, parallel to the direction of the line connecting the user's two eyes, and the Z C axis is, for example, perpendicular to the image plane p of the optical see-through display, and d C→p is, for example, the distance from the simulated human eye camera C to the image plane p of the optical see-through display, that is, the distance d in Figure 2 .

[0045] In the calibration method and calibration system 200 of the optical see-through display 100 in this embodiment, since the external parameters between the image sensing device 220 and the virtual camera D can be calculated by using the above calibration steps, the optical see-through display 100 can generate optimized virtual and real fusion images suitable for the interpupillary distance according to the pre-written external parameters and the set interpupillary distance after leaving the factory. In this way, the optical see-through display 100 can be applied to users with different interpupillary distances, and users with different interpupillary distances can all see good virtual and real fusion images.

[0046] In another embodiment, as Figure 7 shown, the direction of the above translation amount is inclined with respect to the direction of the line connecting the user's two eyes. For example, when the eye position changes, taking the position at the time of calibration with the emulated eye camera C as a reference, after the eye position changes, it is equivalent to the emulated eye camera C moving by t C on the plane formed by the X C axis and the Z C axis (or in the space formed by the X C axis, the Y C axis and the Z C→C′ axis). At this time, only the external parameters between the emulated eye camera C and the virtual camera D need to be corrected to and applied to Equation (8), then the when the emulated eye camera is at position C' can be calculated. In this way, the virtual image 113 displayed by the optical see-through display 100 can be adaptively adjusted according to different eye positions, thus ensuring the accuracy of the virtual-real image fusion. In this embodiment, the X C axis is, for example, parallel to the direction of the line connecting the user's two eyes, and the Z C axis is, for example, perpendicular to the image plane p of the optical see-through display, and the Y C axis is, for example, perpendicular to the X C axis and the Z C axis, and d C→p is, for example, the distance from the emulated eye camera C to the image plane p of the optical see-through display.

[0047] In one embodiment, the controller 210 is, for example, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a programmable controller, a programmable logic device (PLD), or other similar devices or a combination of these devices, and the present invention is not limited thereto. In addition, in one embodiment, the functions of the controller 210 can be implemented as multiple program codes. These program codes are stored in a memory and executed by the controller 210. Alternatively, in one embodiment, the functions of the controller 210 can be implemented as one or more circuits. The present invention does not limit the implementation of the functions of the controller 210 in software or hardware.

[0048] In addition, Figures 2 to 4 or Figures 2 to 5The calibrated parameters can be stored in the memory of the optical transmissive display 100. Such a memory can be, for example, a flash memory, a solid-state drive, a magnetic disk, an optical disc, or various types of memories. In this way, after the optical transmissive display 100 leaves the factory, according to the user's needs, the parameters stored in the memory can be read by using the controller of the optical transmissive display 100 itself or an external controller connected externally to perform Figure 6 or Figure 7 the steps of to generate an optimized virtual-real fusion image according to different eye distances set by the user.

[0049] In summary, in the calibration method and calibration system of the optical transmissive display according to the embodiments of the present invention, since the external parameters between the image sensing device and the virtual camera can be calculated by using the above calibration steps, the optical transmissive display can generate an optimized virtual-real fusion image suitable for the eye distance according to the pre-written external parameters and the set eye distance after leaving the factory.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A calibration method for an optical transmissive display, characterized in that Including: Fixing an image sensing device at a standard position where a user's eyes view an optical transmissive display, wherein the image sensing device is used to capture a virtual image displayed by the optical transmissive display and a real image of the external environment; Using the image sensing device to capture a virtual calibration pattern displayed by the optical transmissive display, and obtaining coordinate values of a plurality of first feature point images of the captured virtual calibration pattern; Calculating a projective homography matrix for converting an image plane of the image sensing device to an image plane of the optical transmissive display by using the coordinate values of the plurality of first feature point images and the coordinate values of first feature points of the virtual calibration pattern; Turning off the optical transmissive display, and using the image sensing device to capture a plurality of real calibration pattern images at different angles, wherein the real calibration pattern has a plurality of second feature points; Calculating internal parameters of the image sensing device and external parameters between the image sensing device and the plurality of real calibration patterns by using coordinates of a plurality of second feature point images in the plurality of real calibration pattern images and the plurality of second feature points on the real calibration pattern; Converting coordinates of a plurality of feature point images in the plurality of real calibration pattern images to coordinates of projected points on the image plane of the optical transmissive display by using the projective homography matrix; Calculating internal parameters of a virtual camera of the optical transmissive display and external parameters between the virtual camera and the plurality of real calibration patterns by using coordinates of a plurality of projected points converted to the image plane of the optical transmissive display in the plurality of real calibration pattern images; And Calculating external parameters between the image sensing device and the virtual camera by using external parameters between the image sensing device and the plurality of real calibration patterns and external parameters between the virtual camera and the plurality of real calibration patterns.

2. The calibration method of the optical transmissive display according to claim 1, characterized in that, The virtual camera is used to simulate a virtual image projector of the optical transmissive display.

3. The calibration method of the optical transmissive display according to claim 1, wherein The method for calculating the projective homography matrix for converting the image plane of the image sensing device to the image plane of the optical transmissive display is the least mean square error method or the random sample consensus method.

4. The calibration method of the optical transmissive display according to claim 1, characterized in that Further including: Using a tracking camera to capture a real pattern with a known size, and obtaining external parameters between the tracking camera and the real pattern, wherein calibration has been performed between the tracking camera and the image sensing device, and obtaining internal parameters of the tracking camera and external parameters between the tracking camera and the image sensing device; and Converting coordinates of a third feature point on the real pattern to coordinates of an image captured by the tracking camera, then converting them to coordinates of an image captured by the image sensing device, then converting them to coordinates of an image of the virtual camera, and finally converting them to coordinates of a virtual image displayed by the optical transmissive display, and the position represented thereby, the third feature point, and the standard position are on the same straight line in space.

5. The calibration method of the optical transmissive display according to claim 4, characterized in that The method for obtaining external parameters between the tracking camera and the real pattern is an attitude estimation algorithm.

6. The calibration method of the optical transmissive display according to claim 1, characterized in that, Further including: In response to a change in the user's interpupillary distance, a translation amount corresponding to the change in the interpupillary distance is added to the external parameters between the image sensing device and the virtual camera, and the position of the virtual image displayed on the optical see-through display is calculated based on the external parameters after the translation amount is added.

7. The calibration method of the optical transmissive display according to claim 6, characterized in that, The direction of the translation amount is parallel to the direction of the line connecting the user's two eyes.

8. The calibration method of the optical transmissive display according to claim 6, characterized in that, The direction of the translation amount is inclined with respect to the direction of the line connecting the user's two eyes.

9. The calibration method of the optical transmissive display according to claim 1, wherein, The method for calculating the external parameters between the image sensing device and the virtual camera includes the Zhang-Zhengyou camera calibration algorithm.

10. A calibration system, characterized in that, For calibrating an optical see-through display, the calibration system includes: An image sensing device fixed at a standard position where the user's eyes view the optical see-through display, the image sensing device being configured to capture a virtual image displayed on the optical see-through display and a real image of the external environment; and A controller coupled to the image sensing device and the optical see-through display, the controller being configured to perform: Command the image sensing device to capture a virtual calibration pattern displayed on the optical see-through display, and obtain coordinate values of a plurality of first feature point images of the captured virtual calibration pattern; Calculate a projection homography matrix for converting the image plane of the image sensing device to the image plane of the optical see-through display using the coordinate values of the plurality of first feature point images and the coordinate values of the first feature points of the virtual calibration pattern; Turn off the optical see-through display, and use the image sensing device to capture a plurality of real calibration pattern images at different angles, the real calibration pattern having a plurality of second feature points; Calculate the internal parameters of the image sensing device and the external parameters between the image sensing device and the plurality of real calibration patterns using the coordinates of the plurality of second feature point images in the plurality of real calibration pattern images and the plurality of second feature points on the real calibration pattern; Convert the coordinates of the plurality of feature point images in the plurality of real calibration pattern images to the coordinates of the projection points on the image plane of the optical see-through display using the projection homography matrix; Calculate the internal parameters of the virtual camera of the optical see-through display and the external parameters between the virtual camera and the plurality of real calibration patterns using the coordinates of the plurality of projection points converted to the image plane of the optical see-through display in the plurality of real calibration pattern images; and Calculate the external parameters between the image sensing device and the virtual camera using the external parameters between the image sensing device and the plurality of real calibration patterns and the external parameters between the virtual camera and the plurality of real calibration patterns.

11. The calibration system according to claim 10, wherein, The virtual camera is used to simulate the virtual image projector of the optical see-through display.

12. The calibration system according to claim 10, wherein The method for calculating the projection homography matrix for converting the image plane of the image sensing device to the image plane of the optical see-through display is the least mean square error method or the random sample consensus method.

13. The calibration system according to claim 10, characterized in that, It further includes a tracking camera coupled to the controller, and the controller further performs: Command the tracking camera to photograph a physical pattern of a known size, and obtain the external parameters between the tracking camera and the physical pattern, wherein calibration has been performed between the tracking camera and the image sensing device, and obtain the internal parameters of the tracking camera and the external parameters between the tracking camera and the image sensing device; and Convert the coordinates of the third feature point on the physical pattern into the coordinates of the image captured by the tracking camera, then convert them into the coordinates of the image captured by the image sensing device, then convert them into the coordinates of the image of the virtual camera, and finally convert them into the coordinates of the virtual image displayed on the optical see-through display, and the position represented by it, the third feature point, and the standard position are on the same straight line in space.

14. The calibration system according to claim 13, wherein The method for obtaining the external parameters between the tracking camera and the physical pattern is an attitude estimation algorithm.

15. The calibration system according to claim 10, wherein The controller also executes: In response to a change in the user's interpupillary distance, increase the translation amount corresponding to the change in the interpupillary distance in the external parameters between the image sensing device and the virtual camera, and calculate the position of the virtual image displayed on the optical see-through display according to the external parameters after the translation amount is increased.

16. The calibration system according to claim 15, wherein The direction of the translation amount is parallel to the direction of the line connecting the user's two eyes.

17. The calibration system according to claim 15, characterized in that, The direction of the translation amount is inclined with respect to the direction of the line connecting the user's two eyes.

18. The calibration system according to claim 10, wherein, The method for calculating the external parameters between the image sensing device and the virtual camera includes the Zhang Zhengyou camera calibration algorithm.

Images