Relative pose calibration method, calibration device, calibration system and storage medium

Abstract

The invention discloses a relative pose calibration method, a calibration device, a calibration system and a computer readable storage medium. The relative pose calibration method comprises the following steps: S1, carrying out first calibration on a first camera and a second camera which are relatively fixed in position to obtain a first relative pose relationship between the first camera and the second camera; s2, the first camera is aligned with the view field of the display light machine, the second camera and the sensing camera are calibrated for the second time, a second relative pose relation between the second camera and the sensing camera is obtained, and the position is relatively fixed so that the second camera and the sensing camera can have a common-view area for calibration; and S3, calculating a third relative pose relationship between the display light machine and the perception camera according to the first relative pose relationship and the second relative pose relationship. Therefore, the calibration of the relative pose between the display light machine and the perception camera can be efficiently completed, and the calibration precision is high.

Description

Technical Field

[0001] This application relates to the technical field of head-mounted display devices, and particularly relates to a relative pose calibration method, a calibration device, a calibration system, and a computer-readable storage medium. Background Art

[0002] A head-mounted display device (HMD) relies on a camera group to perceive the surrounding environment. For example, a VST camera for video see-through, and a SLAM (Simultaneous Localization and Mapping) camera for detecting the spatial pose of the HMD. The information perceived by the camera group is reflected into the optical engine for near-eye display in the form of images or other forms, so as to achieve virtual display or mixed display. The above process is inseparable from the spatial calibration between the camera group and the optical engine for near-eye display. Currently, there is no feasible calibration technology between the camera group and the optical engine for near-eye display. Summary of the Invention

[0003] Embodiments of this application provide a relative pose calibration method, a calibration device, a calibration system, and a computer-readable storage medium to solve at least one of the above-mentioned technical problems.

[0004] The relative pose calibration method of the embodiments of this application is applied to a head-mounted display device, the head-mounted display device includes a display body, a sensing camera, and a display optical engine, the sensing camera and the display optical engine are both fixedly arranged on the display body, and the relative pose calibration method includes:

[0005] S1. Perform a first calibration on a first camera and a second camera with relatively fixed positions to obtain a first relative pose relationship between the first camera and the second camera;

[0006] S2. Align the field of view of the first camera with the display optical engine and perform a second calibration on the second camera and the sensing camera to obtain a second relative pose relationship between the second camera and the sensing camera, where the relatively fixed position enables the second camera and the sensing camera to have a common viewing area for calibration;

[0007] S3. Calculate a third relative pose relationship between the display optical engine and the sensing camera according to the first relative pose relationship and the second relative pose relationship.

[0008] In some embodiments, the first camera and the second camera are relatively fixedly connected through a rigid connecting body;

[0009] When the first camera is aligned with the field of view of the display optical engine, the rigid connection body can stagger the field of view range of the second camera from the occlusion range of the display body and the display optical engine, so as to have a common viewing area with the sensing camera.

[0010] In some embodiments, when the first camera is aligned with the field of view of the display optical engine, the first camera is located at the eye point position of the display optical engine.

[0011] In some embodiments, the display body includes a left-eye body, the sensing camera includes a left-eye camera, and the display optical engine includes a left-eye optical engine;

[0012] Aligning the first camera with the field of view of the display optical engine and performing a second calibration on the second camera and the sensing camera in S2 to obtain the second relative pose relationship between the second camera and the sensing camera includes:

[0013] S21. Align the first camera with the field of view of the left-eye optical engine and perform a second calibration on the second camera and the left-eye camera to obtain the second left relative pose relationship between the second camera and the left-eye camera, wherein the positions are relatively fixed so that the second camera and the left-eye camera have a common viewing area for calibration;

[0014] Calculating the third relative pose relationship between the display optical engine and the sensing camera according to the first relative pose relationship and the second relative pose relationship in S3 includes:

[0015] S31. Calculate the third left relative pose relationship between the left-eye optical engine and the left-eye camera according to the first relative pose relationship and the second relative pose relationship.

[0016] In some embodiments, the left-eye camera includes a left-eye video see-through camera and / or a left-eye spatial pose detection camera.

[0017] In some embodiments, the display body includes a right-eye body, the sensing camera includes a right-eye camera, and the display optical engine includes a right-eye optical engine;

[0018] Aligning the first camera with the field of view of the display optical engine and performing a second calibration on the second camera and the sensing camera in S2 to obtain the second relative pose relationship between the second camera and the sensing camera includes:

[0019] S22. Align the first camera with the right-eye optical machine's field of view and perform a second calibration on the second camera and the right-eye camera to obtain the second right relative pose relationship between the second camera and the right-eye camera, where the positions are relatively fixed such that the second camera and the right-eye camera have a common viewing area for calibration;

[0020] The calculation of the third relative pose relationship between the display optical machine and the perception camera according to the first relative pose relationship and the second relative pose relationship in S3 includes:

[0021] S32. Calculate the third right relative pose relationship between the right-eye optical machine and the right-eye camera according to the first relative pose relationship and the second relative pose relationship.

[0022] In some embodiments, the right-eye camera includes a right-eye video perspective camera and / or a right-eye spatial pose detection camera.

[0023] In some embodiments, the first calibration of the first camera and the second camera with relatively fixed positions in S1 to obtain the first relative pose relationship between the first camera and the second camera includes:

[0024] S11. Perform a first calibration on the first camera and the second camera with relatively fixed positions to obtain the first rotation matrix and the first translation matrix of the relative pose between the first camera and the second camera;

[0025] The second calibration of the second camera and the perception camera in S2 to obtain the second relative pose relationship between the second camera and the perception camera includes:

[0026] S23. Perform a second calibration on the second camera and the perception camera to obtain the second rotation matrix and the second translation matrix of the relative pose between the second camera and the perception camera;

[0027] The calculation of the third relative pose relationship between the display optical machine and the perception camera according to the first relative pose relationship and the second relative pose relationship in S3 includes:

[0028] S33. Calculate the third rotation matrix of the relative pose between the display optical machine and the perception camera according to the first rotation matrix and the second rotation matrix;

[0029] S34. Calculate the third translation matrix of the relative pose between the display optical machine and the perception camera according to the first translation matrix and the second translation matrix.

[0030] The relative pose calibration device according to the embodiments of the present application is applied to a head-mounted display device, which includes a display body, a perception camera, and a display optical engine. The perception camera and the display optical engine are both fixedly arranged on the display body. The relative pose calibration device includes:

[0031] A calibration module for performing a first calibration on a first camera and a second camera with relatively fixed positions to obtain a first relative pose relationship between the first camera and the second camera;

[0032] The calibration module is further configured to align the field of view of the first camera with the display optical engine and perform a second calibration on the second camera and the perception camera to obtain a second relative pose relationship between the second camera and the perception camera, wherein the relatively fixed positions enable the second camera and the perception camera to have a common viewing area for calibration;

[0033] A calculation module for calculating a third relative pose relationship between the display optical engine and the perception camera according to the first relative pose relationship and the second relative pose relationship.

[0034] The relative pose calibration system according to the embodiments of the present application includes one or more processors and a memory. The memory stores a computer program. When the computer program is executed by the processor, the relative pose calibration method according to any of the above embodiments is implemented.

[0035] The computer-readable storage medium according to the embodiments of the present application stores a computer program thereon, characterized in that when the program is executed by a processor, the relative pose calibration method according to any of the above embodiments is implemented.

[0036] In the relative pose calibration method, calibration device, calibration system, and computer-readable storage medium according to the embodiments of the present application, the field of view of the first camera is aligned with the display optical engine, the positions of the first camera and the second camera are relatively fixed, so that the second camera and the perception camera have a common viewing area for calibration. The pose of the display optical engine is represented by the pose of the first camera, and the third relative pose relationship between the display optical engine and the perception camera is calculated according to the first relative pose relationship between the first camera and the second camera and the second relative pose relationship between the second camera and the perception camera. In this way, the relative pose calibration between the display optical engine and the perception camera can be efficiently completed with high calibration accuracy.

[0037] Additional aspects and advantages of the embodiments of the present application will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the accompanying drawings, where:

[0039] Figure 1 is a schematic flowchart of a relative pose calibration method according to some embodiments of the present application;

[0040] Figure 2 is a schematic diagram of an application scenario where the relative pose calibration method according to some embodiments of the present application is applied to a head-mounted display device;

[0041] Figure 3 is a schematic diagram of the working process of the relative pose calibration method according to some embodiments of the present application;

[0042] Figure 4 is a schematic diagram of an application scenario where the relative pose calibration method according to some embodiments of the present application is applied to a head-mounted display device;

[0043] Figure 5 is a schematic diagram of an application scenario where the relative pose calibration method according to some embodiments of the present application is applied to a head-mounted display device;

[0044] Figure 6 is a schematic diagram of an application scenario where the relative pose calibration method according to some embodiments of the present application is applied to a head-mounted display device;

[0045] Figure 7 is a schematic diagram of a calibration board used in the relative pose calibration method according to some embodiments of the present application;

[0046] Figure 8 is a schematic flowchart of the relative pose calibration method according to some embodiments of the present application;

[0047] Figure 9 is a schematic flowchart of the relative pose calibration method according to some embodiments of the present application;

[0048] Figure 10 is a schematic flowchart of the relative pose calibration method according to some embodiments of the present application;

[0049] Figure 11 is a schematic diagram of the modules of a relative pose calibration device according to some embodiments of the present application;

[0050] Figure 12 is a schematic diagram of the modules of a relative pose calibration system according to some embodiments of the present application;

[0051] Figure 13 is a schematic diagram of the connection state between a computer-readable storage medium and a processor according to some embodiments of the present application. Detailed embodiments

[0052] The embodiments of the present application will be further described below in conjunction with the accompanying drawings. The same or similar reference numerals in the drawings denote the same or similar elements or elements having the same or similar functions throughout. In addition, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary and are only used to explain the embodiments of the present application and should not be construed as a limitation of the present application.

[0053] The realization of virtual display and mixed display by a head-mounted display device is inseparable from the spatial calibration of the VST camera and / or the SLAM camera and the display optical engine. The calibration result of the relative pose between the VST camera and / or the SLAM camera and the display optical engine will affect the accuracy of projecting the real picture captured by the VST camera onto the optical engine. Deviations in these accuracies will lead to incorrect user spatial perception, separation between virtual and reality, and even discomfort such as dizziness, affecting the user experience.

[0054] In the prior art, for the calibration method of the relative pose between the VST camera and / or the SLAM camera and the display optical engine, since the display optical engine blocks the calibration camera aligned with it behind, the calibration process is usually divided into two types:

[0055] The first type: The calibration card is in a fixed pose, and calibration needs to be carried out in two times. In the first time, the head-mounted display device is not placed, and the calibration camera directly shoots the calibration card; in the second time, after aligning the optical engine with the calibration camera, the VST camera and / or the SLAM camera shoot the calibration card again. Each calibration requires moving the head-mounted display device and the calibration camera to shoot more than 30 calibration pictures in different poses. Therefore, the optical engine needs to be aligned with the calibration camera more than 30 times, and the calibration time of one head-mounted display device is relatively long, the calibration efficiency is low, and because of repeatedly aligning the optical engine with the calibration camera, the error of the repeat accuracy will also lead to low calibration accuracy.

[0056] The second type: The optical engine and the calibration camera are aligned only once, and the VST camera and / or the SLAM camera and the calibration camera respectively shoot more than 30 calibration pictures in sequence. During the two shootings when the head-mounted display device is not placed (calibration camera shooting) and when the head-mounted display device is placed (VST camera and / or SLAM camera shooting), the robotic arm needs to place the calibration card at more than 30 different positions with a relatively high repeat accuracy to ensure that the position height is the same in the front and back shootings at each position. On the one hand, the cost of adding a high-precision robotic arm is required, and on the other hand, the cumulative effect of small accuracy errors after more than 30 times will also lead to a reduction in calibration accuracy.

[0057] Please refer to Figures 1 to 6, an embodiment of the present application provides a relative pose calibration method, which is applied to a head-mounted display device 100. The head-mounted display device 100 includes a display body 10, a sensing camera 20, and a display optical engine 30. The sensing camera 20 and the display optical engine 30 are both fixedly arranged on the display body 10. The relative pose calibration method includes:

[0058] S1. Perform the first calibration on the first camera 101 and the second camera 102 with relatively fixed positions to obtain the first relative pose relationship between the first camera 101 and the second camera 102;

[0059] S2. Align the field of view of the first camera 101 with the display optical engine 30 and perform the second calibration on the second camera 102 and the sensing camera 20 to obtain the second relative pose relationship between the second camera 102 and the sensing camera 20, where the relatively fixed positions enable the second camera 102 and the sensing camera 20 to have a common viewing area for calibration;

[0060] S3. Calculate the third relative pose relationship between the display optical engine 30 and the sensing camera 20 according to the first relative pose relationship and the second relative pose relationship.

[0061] In the relative pose calibration method of the embodiment of the present application, the field of view of the first camera 101 is aligned with the display optical engine 30, and the positions of the first camera 101 and the second camera 102 are relatively fixed, so that the second camera 102 and the sensing camera 20 have a common viewing area for calibration. The pose of the display optical engine 30 is characterized by the pose of the first camera 101, and the third relative pose relationship between the display optical engine 30 and the sensing camera 20 is calculated according to the first relative pose relationship between the first camera 101 and the second camera 102 and the second relative pose relationship between the second camera 102 and the sensing camera 20. In this way, the relative pose calibration between the display optical engine 30 and the sensing camera 20 can be efficiently completed, and the calibration accuracy is relatively high.

[0062] Specifically, the first camera 101 and the second camera 102 are relatively fixed in position and have a common viewing area. The common viewing area refers to the field of view area that multiple cameras can jointly capture, that is, in a multiple-camera system, the overlapping area of the field of view of each camera, which is also the field of view overlapping area. Here, it refers to the field of view overlapping area of the first camera 101 and the second camera 102. It can be understood that misalignment between cameras, object occlusion, etc. will all cause a small or no common viewing area.

[0063] The calibration board can adopt an Aruco calibration board as shown in Figure 7 . Of course, other types of calibration boards can also be adopted, which are not limited here. Place the calibration board in the common viewing area of the first camera 101 and the second camera 102 (the field of view directions of the first camera 101 and the second camera 102 are as shown in Figure 5The first camera 101 and the second camera 102 are fixed, and the calibration plate is moved within the common viewing area of the first camera 101 and the second camera 102. The first camera 101 and the second camera 102 shoot the calibration plate to perform a first calibration, thereby obtaining a first relative position relationship between the first camera 101 and the second camera 102.

[0064] After the first calibration is completed, a robotic arm or tooling can be used to place the head-mounted display device 100 in front of the field of view of the first camera 101, and align the first camera 101 with the field of view of the display light machine 30, so that the posture of the first camera 101 can represent the posture of the display light machine 30. At this time, since the positions of the first camera 101 and the second camera 102 are relatively fixed, the field of view of the second camera 102 is staggered from the occlusion range of the display body 10 and the display light machine 30, so that the second camera 102 and the perception camera 20 also have a common viewing area, and the second camera 102 and the perception camera 20 can simultaneously capture the calibration plate. Move the position of the calibration plate within the common viewing area of the second camera 102 and the perception camera 20 (the field of view direction of the second camera 102 and the perception camera 20 is as shown in FIG. Figure 4 The second calibration is performed to obtain a second relative position relationship between the second camera 102 and the perception camera 20. It should be noted that the position of the calibration plate does not need to remain consistent during the first and second calibrations.

[0065] Based on the first relative pose relationship between the first camera 101 and the second camera 102, and the second relative pose relationship between the second camera 102 and the perception camera 20, the relative pose relationship between the first camera 101 and the perception camera 20 can be calculated. Since the first camera 101 and the display optical engine 30 are aligned in view, the pose of the first camera 101 can represent the pose of the display optical engine 30, and the relative pose relationship between the first camera 101 and the perception camera 20 can represent the third relative pose relationship between the display optical engine 30 and the perception camera 20, thereby completing the relative pose calibration between the display optical engine 30 and the perception camera 20. This avoids repeated alignment of the first camera 101 and the display optical engine 30, allowing for efficient calibration of the relative pose between the display optical engine 30 and the perception camera 20. Furthermore, it avoids the problem of low calibration accuracy caused by repeated positioning errors of the calibration plate during the two consecutive captures, resulting in a highly accurate calibration result.

[0066] See also Figure 2 、 Figure 4 and Figure 5, in some embodiments, the first camera 101 and the second camera 102 are relatively fixedly connected through a rigid connecting body 103. When the first camera 101 is field-of-view aligned with the display optical engine 30, the rigid connecting body 103 can stagger the field-of-view range of the second camera 102 from the occlusion range of the display body 10 and the display optical engine 30, so as to have a common field-of-view area with the sensing camera 20.

[0067] Specifically, the first camera 101 and the second camera 102 are connected through a rigid connecting body 103, that is, a stable fixed connection is established between the first camera 101 and the second camera 102. While ensuring the relative fixed connection between the first camera 101 and the second camera 102, the rigid connecting body 103 can stagger the field-of-view range of the second camera 102 from the occlusion range of the display body 10 and the display optical engine 30 when the first camera 101 is field-of-view aligned with the display optical engine 30, so as to have a common field-of-view area with the sensing camera 20.

[0068] It can be understood that, on the premise of ensuring the above effects, different structural designs of the rigid connecting body 103 can be selected according to the different structures of the head-mounted display device 100. In one example, the length extension direction of the rigid connecting body 103 is perpendicular to the left-right direction of the head-mounted display device 100. For example Figure 4 and Figure 5 in, the left-right direction of the head-mounted display device 100 is the direction perpendicular to the paper surface, and the length extension direction of the rigid connecting body 103 is the vertical direction. Alternatively, the rigid connecting body 103 can be designed as a universal structure and can be adjusted to different shapes for different types of head-mounted display devices 100. For example, the rigid connecting body 103 can be designed as a C-shaped structure and the arc can be adjusted according to different types of head-mounted display devices 100.

[0069] It should be noted that Figure 5 in, in order to facilitate the display of the field-of-view alignment relationship between the first camera 101 and the display optical engine 30, part of the structure of the display body 10 is hidden (specifically, refer to Figure 4 and Figure 5 for comparison).

[0070] In some embodiments, when the first camera 101 is field-of-view aligned with the display optical engine 30, the first camera 101 is located at the eye point position of the display optical engine 30.

[0071] Specifically, the alignment of the first camera 101 with the display optical engine 30 can adopt a cross alignment method or a contour alignment method. For example, when using the cross alignment method to align the first camera 101 with the display optical engine 30, a cross is displayed on the display optical engine 30, and a cross is also displayed in the preview frame of the first camera 101. The display optical engine 30 is within the field of view of the first camera 101, so that the cross in the preview frame of the first camera 101 is aligned with the cross on the display optical engine 30 captured by the first camera 101. It can be understood that the above alignment only aligns the first camera 101 and the display optical engine 30 in the X-axis and Y-axis, and the distance between the first camera 101 and the display optical engine 30 also needs to be determined. The first camera 101 can be located at the eye point position of the display optical engine 30. The eye point position refers to the eye position where the user has the best experience when using the head-mounted display device 100. At this time, the distance between the aperture of the first camera 101 and the surface of the display optical engine 30 is the eye relief. When the first camera 101 is located at the eye point position of the display optical engine 30, it can simulate the receiving position of the user's eyes relative to the display optical engine 30. The field of view of the first camera 101 can represent the field of view of the user's eyes. When the user wears the head-mounted display device 100, the field of view of the display optical engine 30 is the field of view of the user's eyes. Thus, when performing pose calibration, the pose of the first camera 101 can be used as the pose of the display optical engine 30.

[0072] Please refer to Figure 2 、 Figure 5 and Figure 8 , in some embodiments, the display body 10 includes a left-eye body 11, the sensing camera 20 includes a left-eye camera 20a, and the display optical engine 30 includes a left-eye optical engine 31.

[0073] In S2, aligning the field of view of the first camera 101 with the display optical engine 30 and performing a second calibration on the second camera 102 and the sensing camera 20 to obtain the second relative pose relationship between the second camera 102 and the sensing camera 20 includes:

[0074] S21. Align the field of view of the first camera 101 with the left-eye optical engine 31 and perform a second calibration on the second camera 102 and the left-eye camera 20a to obtain the second left relative pose relationship between the second camera 102 and the left-eye camera 20a, where the positions are relatively fixed so that the second camera 102 and the left-eye camera 20a have a common viewing area for calibration.

[0075] In S3, calculating the third relative pose relationship between the display optical engine 30 and the sensing camera 20 according to the first relative pose relationship and the second relative pose relationship includes:

[0076] S31. Calculate the third left relative pose relationship between the left-eye optical engine 31 and the left-eye camera 20a according to the first relative pose relationship and the second relative pose relationship.

[0077] Specifically, the left eyepiece 11 refers to the portion of the head-mounted display device 100 that corresponds to the user's left eye when the head-mounted display device 100 is worn. When the head-mounted display device 100 is in operation, the image captured by the left-eye camera 20a is projected onto the left-eye camera 31 for display. There may be one or more left-eye cameras 20a. The above-described second positional relationship between the second camera 102 and the perception camera 20 specifically refers to the second positional relationship between the second camera 102 and the left-eye camera 20a. A robotic arm or tooling is used to place the head-mounted display device 100 in front of the field of view of the first camera 101. The fields of view of the first camera 101 and the left-eye camera 31 are aligned so that the position of the first camera 101 represents the position of the left-eye camera 31. The field of view of the second camera 102 is offset from the obstruction of the left eyepiece 11 and the left-eye camera 31 so that it has a common viewing area with the left-eye camera 20a, allowing the second camera 102 and the left-eye camera 20a to simultaneously capture the calibration plate. The second camera 102 and the left camera 20a are then calibrated a second time to obtain a second left relative pose relationship between the second camera 102 and the left camera 20a. Based on the first and second left relative pose relationships, a third left relative pose relationship between the left camera 31 and the left camera 20a is calculated, completing the relative pose calibration between the left camera 31 and the left camera 20a.

[0078] See also Figure 2 In some embodiments, the left-eye camera 20a includes a left-eye video perspective (VST) camera 21 and / or a left-eye spatial pose detection (SLAM) camera 22.

[0079] Specifically, the video perspective camera can capture a real-time view of the environment, which is then combined with computer image technology to be presented on the display light machine 30. The spatial posture detection camera can build a model of the environment by perceiving the surrounding environment during movement without prior information about the environment, thereby providing accurate positioning information for the head-mounted display device 100. The left-eye camera 20a may include only the left-eye video perspective camera 21; or, the left-eye camera 20a may include only the left-eye spatial posture detection camera 22; or, the left-eye camera 20a may include both the left-eye video perspective camera 21 and the left-eye spatial posture detection camera 22. In each of the above cases, the number of the left-eye video perspective camera 21 and the left-eye spatial posture detection camera 22 can be one or more, and there is no limitation here.

[0080] See also Figure 2 、 Figure 5 and Figure 9 In some embodiments, the display body 10 includes a right-eye body 12 , the perception camera 20 includes a right-eye camera 20 b , and the display light machine 30 includes a right-eye camera 32 .

[0081] S2 aligns the first camera 101 with the field of view of the display light machine 30 and performs a second calibration on the second camera 102 and the perception camera 20 to obtain a second relative position relationship between the second camera 102 and the perception camera 20, including:

[0082] S22, aligning the first camera 101 with the field of view of the right eye camera 32 and performing a second calibration on the second camera 102 and the right eye camera 20b to obtain a second right relative pose relationship between the second camera 102 and the right eye camera 20b, wherein the positions are relatively fixed so that the second camera 102 and the right eye camera 20b have a common viewing area for calibration;

[0083] Calculating a third relative posture relationship between the display optical engine 30 and the perception camera 20 based on the first relative posture relationship and the second relative posture relationship in S3 includes:

[0084] S32. Calculate a third right relative posture relationship between the right-eye camera 32 and the right-eye camera 20b based on the first relative posture relationship and the second relative posture relationship.

[0085] Specifically, the right-eye body 12 refers to the portion of the head-mounted display device 100 corresponding to the user's right eye when the head-mounted display device 100 is worn. When the head-mounted display device 100 is in operation, the image captured by the right-eye camera 20b is projected onto the right-eye camera 32 for display. There may be one or more right-eye cameras 20b. The above-described second positional relationship between the second camera 102 and the perception camera 20 specifically refers to the second positional relationship between the second camera 102 and the right-eye camera 20b. A robotic arm or tooling is used to place the head-mounted display device 100 in front of the field of view of the first camera 101. The fields of view of the first camera 101 and the right-eye camera 32 are aligned so that the position of the first camera 101 represents the position of the right-eye camera 32. The field of view of the second camera 102 is offset from the occlusion range of the right-eye body 12 and the right-eye camera 32 to create a common viewing area with the right-eye camera 20b, allowing the second camera 102 and the right-eye camera 20b to simultaneously capture the calibration plate. The second camera 102 and the right camera 20b are calibrated a second time to obtain a second left relative pose relationship between the second camera 102 and the right camera 20b. Based on the first relative pose relationship and the second right relative pose relationship, a third right relative pose relationship between the right camera 32 and the right camera 20b is calculated, completing the relative pose calibration between the right camera 32 and the right camera 20b.

[0086] See also Figure 2 In some embodiments, the right-eye camera 20 b includes a right-eye video perspective (VST) camera 23 and / or a right-eye spatial pose detection (SLAM) camera 24 .

[0087] Specifically, the right-eye camera 20b may only include the right-eye video perspective camera 23; alternatively, the right-eye camera 20b may only include the right-eye spatial pose detection camera 24; or the right-eye camera 20b may include both the right-eye video perspective camera 23 and the right-eye spatial pose detection camera 24 at the same time. In each of the above cases, the number of the right-eye video perspective camera 23 and the right-eye spatial pose detection camera 24 may be one or more, which is not limited herein.

[0088] Please refer to Figure 10 , in some embodiments, the first calibration of the relatively fixedly positioned first camera 101 and second camera 102 in S1 to obtain the first relative pose relationship between the first camera 101 and the second camera 102 includes:

[0089] S11. Perform the first calibration on the relatively fixedly positioned first camera 101 and second camera 102 to obtain the first rotation matrix and the first translation matrix of the relative pose between the first camera 101 and the second camera 102;

[0090] The second calibration of the second camera 102 and the sensing camera 20 in S2 to obtain the second relative pose relationship between the second camera 102 and the sensing camera 20 includes:

[0091] S23. Perform the second calibration on the second camera 102 and the sensing camera 20 to obtain the second rotation matrix and the second translation matrix of the relative pose between the second camera 102 and the sensing camera 20;

[0092] The calculation of the third relative pose relationship between the display optical engine 30 and the sensing camera 20 according to the first relative pose relationship and the second relative pose relationship in S3 includes:

[0093] S33. Calculate the third rotation matrix of the relative pose between the display optical engine 30 and the sensing camera 20 according to the first rotation matrix and the second rotation matrix;

[0094] S34. Calculate the third translation matrix of the relative pose between the display optical engine 30 and the sensing camera 20 according to the first translation matrix and the second translation matrix.

[0095] Specifically, both the first calibration and the second calibration can adopt a binocular or multi-camera calibration algorithm based on Zhang Zhengyou's calibration method. An example of the specific process of the first calibration of the first camera 101 and the second camera 102 is as follows: Adopt a binocular calibration algorithm based on Zhang Zhengyou's calibration method, move the position of the calibration board within the field of view of the first camera 101 and the second camera 102, and the first camera 101 and the second camera 102 simultaneously take more than 30 calibration board pictures. Extract the corner points from the calibration board pictures taken by the first camera 101 and the second camera 102, and calculate the rotation matrix R1 and translation matrix T1 of the first camera 101, as well as the rotation matrix R2 and translation matrix T2 of the second camera 102, respectively, according to the pixel coordinates and world coordinates of the corner points. Usually, the calculated rotation matrix and translation matrix are those of the first camera 101 and the second camera 102 relative to the calibration board. In the embodiments of the present application, taking the pose of the second camera 102 as the reference, making the rotation matrix R2 and translation matrix T2 of the second camera 102 be zero matrices, and calculating the rotation matrix and translation matrix of the first camera 101 relative to the second camera 102 as the rotation matrix R2 and translation matrix T2 of the first camera 101. Then, by subtracting the elements in the rotation matrix R1 and translation matrix T1 of the first camera 101 from the corresponding elements in the rotation matrix R2 and translation matrix T2 of the second camera 102, the first rotation matrix R 12 and the first translation matrix T 12 can be obtained. The calculation formula is as follows:

[0096] R 12 (i) = R1(i) - R2(i)

[0097] T 1a (i) = T1(i) - T2(i)

[0098] Wherein, R 12 (i) represents the i-th element in the first rotation matrix R 12 , R1(i) represents the i-th element in the rotation matrix R1 of the first camera 101, and R2(i) represents the i-th element in the rotation matrix R2 of the second camera 102; T 12 (i) represents the i-th element in the first translation matrix T 12 , T1(i) represents the i-th element in the translation matrix T1 of the first camera 101, and T2(i) represents the i-th element in the translation matrix T2 of the second camera 102. The first rotation matrix R 12 and the first translation matrix T 12 are the first relative pose relationship between the first camera 101 and the second camera 102.

[0099] After the first calibration is completed, a second calibration is performed on the second camera 102 and the perception camera 20 to obtain the second rotation matrix and the second translation matrix of the relative pose between the second camera 102 and the perception camera 20. The perception camera 20 includes a left-eye camera 20a and a right-eye camera 20b. That is, the second calibration includes the calibration between the second camera 102 and the left-eye camera 20a, and the calibration between the second camera 102 and the right-eye camera 20b. Obtaining the second rotation matrix and the second translation matrix of the relative pose between the second camera 102 and the perception camera 20 includes: obtaining the second left rotation matrix and the second left translation matrix of the relative pose between the second camera 102 and the left-eye camera 20a, and obtaining the second right rotation matrix and the second right translation matrix of the relative pose between the second camera 102 and the right-eye camera 20b.

[0100] An example of the calibration process between the second camera 102 and the left-eye camera 20a is as follows: Taking the calibration of the second camera 102 and a left-eye camera 20a as an example, a binocular calibration algorithm based on Zhang Zhengyou's calibration method is adopted. Move the position of the calibration board within the field of view of the second camera 102 and the left-eye camera 20a. The second camera 102 and the left-eye camera 20a simultaneously take more than 30 pictures of the calibration board. Extract the corner points of the calibration board pictures taken by the second camera 102 and the left-eye camera 20a, and calculate the rotation matrix R2 and the translation matrix T2 of the second camera 102, and the rotation matrix R3 and the translation matrix T3 of the left-eye camera 20a respectively according to the pixel coordinates and world coordinates of the corner points. Similar to the first calibration, taking the pose of the second camera 102 as the reference, making the rotation matrix R2 and the translation matrix T2 of the second camera 102 be zero matrices, and calculating the rotation matrix and the translation matrix of the left-eye camera 20a relative to the second camera 102 as the rotation matrix R3 and the translation matrix T3 of the left-eye camera 20a. Then, by subtracting the elements in the rotation matrix R3 and the translation matrix T3 of the left-eye camera 20a from the corresponding elements in the rotation matrix R2 and the translation matrix T2 of the second camera 102, the second left rotation matrix R 32 and the second left translation matrix T 32 can be obtained. The calculation formulas are as follows:

[0101] R 32 (i) = R3(i) - R2(i)

[0102] T 32 (i) = T3(i) - T2(i)

[0103] where, R 32 (i) represents the i-th element in the second left rotation matrix R 32 , R3(i) represents the i-th element in the rotation matrix R3 of the left-eye camera 20a; T 32 (i) represents the second left translation matrix T32 the i-th element in, and T3(i) represents the i-th element in the translation matrix T1 of the right-eye camera 20b. The second left rotation matrix R 32 and the second left translation matrix T 32 are the second left relative pose relationship between the second camera 102 and the left-eye camera 20a.

[0104] An example of the calibration process between the second camera 102 and the right-eye camera 20b is as follows: Taking the calibration of the second camera 102 and a right-eye camera 20b as an example, a binocular calibration algorithm based on Zhang's calibration method is adopted. Move the position of the calibration board within the field of view of the second camera 102 and the right-eye camera 20b, and the second camera 102 and the right-eye camera 20b simultaneously take more than 30 calibration board pictures. Extract the corner points of the calibration board pictures taken by the first camera 101 and the second camera 102, and calculate the rotation matrix R2 and translation matrix T2 of the second camera 102, as well as the rotation matrix R4 and translation matrix T4 of the right-eye camera 20b according to the pixel coordinates and world coordinates of the corner points. Similar to the first calibration, taking the pose of the second camera 102 as the reference, making the rotation matrix R2 and translation matrix T2 of the second camera 102 be zero matrices, and calculating the rotation matrix and translation matrix of the right-eye camera 20b relative to the second camera 102 as the rotation matrix R4 and translation matrix T4 of the right-eye camera 20b. Then, by subtracting the elements in the rotation matrix R4 and translation matrix T4 of the right-eye camera 20b from the corresponding elements in the rotation matrix R2 and translation matrix T2 of the second camera 102, the second right rotation matrix R 42 and the second right translation matrix T 42 can be obtained. The calculation formulas are as follows:

[0105] R 42 (i) = R4(i) - R2(i)

[0106] T 42 (i) = T4(i) - T2(i)

[0107] where, R 42 (i) represents the i-th element in the second right rotation matrix R 42 , R4(i) represents the i-th element in the rotation matrix R4 of the right-eye camera 20b; T 42 (i) represents the i-th element in the second right translation matrix T 42 , T4(i) represents the i-th element in the translation matrix T1 of the right-eye camera 20b. The second right rotation matrix R 42 and the second right translation matrix T 42That is the second right relative pose relationship between the second camera 102 and the right-eye camera 20b. The second relative pose relationship includes the second left relative pose relationship and the second right relative pose relationship.

[0108] After the second calibration, according to the first rotation matrix R 12 and the second rotation matrix, the third rotation matrix of the relative pose between the display optical machine 30 and the perception camera 20 can be calculated. Among them, according to the first rotation matrix R 12 and the second left rotation matrix R 32 the third left rotation matrix R 31 of the relative pose between the display optical machine 30 and the left-eye camera 20a can be calculated, and the calculation formula is as follows:

[0109] R 31 (i) = R3(i) - R1(i) = [R3(i) - R2(i)] - [R1(i) - R2(i)] = R 31 (i) - R 12 (i)

[0110] According to the first rotation matrix R 12 and the second right rotation matrix R 42 the third right rotation matrix R 41 of the relative pose between the display optical machine 30 and the right-eye camera 20b can be calculated, and the calculation formula is as follows:

[0111] R 41 (i) = R4(i) - R1(i) = [R4(i) - R2(i)] - [R1(i) - R2(i)] = R 41 (i) - R 12 (i)

[0112] The third rotation matrix includes the third left rotation matrix R 31 and the third right rotation matrix R 41 .

[0113] After the second calibration, according to the first translation matrix T 12 and the second translation matrix, the third translation matrix of the relative pose between the display optical machine 30 and the perception camera 20 can also be calculated. Among them, according to the first translation matrix T 12 and the second left translation matrix T 32 the third left translation matrix T 31 of the relative pose between the display optical machine 30 and the left-eye camera 20a can be calculated, and the calculation formula is as follows:

[0114] T 31 (i) = T3(i) - T1(i) = [T3(i) - T2(i)] - [T1(i) - T2(i)] = T 32 (i) - T12 (i)

[0115] According to the first translation matrix T 12 and the second right translation matrix T 42 the third right translation matrix T representing the relative pose between the display optical engine 30 and the right-eye camera 20b can be calculated. The calculation formula is as follows: 41 ,The calculation formula is as follows:

[0116] T 41 (i) = T4(i) - T1(i) = [T4(i) - T2(i)] - [T1(i) - T2(i)] = T 42 (i) - T 12 (i)

[0117] The third translation matrix includes a third left translation matrix T 31 and a third right translation matrix T 41 。Obtaining the third rotation matrix and the third translation matrix of the relative pose between the display optical engine 30 and the sensing camera 20 means obtaining the third relative pose relationship between the display optical engine 30 and the sensing camera 20.

[0118] The following describes the calculation process of the relative pose calibration method of the embodiments of the present application in practical applications.

[0119] Taking the calibration of the left optical engine 31 and the left-eye camera 20a of a head-mounted display device 100 as an example, with the pose of the second camera 102 as the reference, in the translation matrix of the second camera 102: T2(1) = 0, T2(2) = 0, T2(3) = 0. The world coordinates of the second camera 102 are (0, 0, 0, 1000).

[0120] The three elements in the translation matrix T1 of the first camera 101 obtained in the first calibration are: T1(1) = 3.75mm, T1(2) = -87.54mm, T1(3) = 4.61m. The world coordinates of the first camera 101 are (10.50, -22.64, 27.52, 999.30).

[0121] Then in the first translation matrix T 12 :

[0122] T 12 (1) = T1(1) - T2(1) = 3.75mm

[0123] T 12 (2) = T1(2) - T2(2) = -87.54mm

[0124] T 12 (3) = T1(3) - T2(3) = 4.61mm

[0125] The three elements in the translation matrix T3 of the left camera 20a obtained in the second calibration are: T3(1) = -15.14 mm, T3(2) = -120.38 mm, T3(3) = -63.33 mm. The world coordinates of the left camera 20a are (7.57, -20.61, -32.80, -999.22).

[0126] Then the second left translation matrix T 32 The three elements in it are:

[0127] T 32 (1) = T3(1) - T2(1) = -15.14 mm

[0128] T 32 (2) = T3(2) - T2(2) = -120.38 mm

[0129] T 32 (3) = T3(3) - T2(3) = -63.33 mm

[0130] Then the third left translation matrix T 31 The three elements in it are:

[0131] T 31 (1) = T 32 (1) - T 12 (1) = - -15.14 - 3.75 = -18.89 mm

[0132] T 31 (2) = T 32 (2) - T 12 (2) = - -120.38 - (-87.54) = -32.84 mm

[0133] T 31 (3) = T 32 (3) - T 12 (3) = - -63.33 - 4.61 = -67.94 mm

[0134] It should be noted that the third relative pose relationship between the display optical machine 30 and the perception camera 20 obtained by the relative pose calibration method according to the embodiments of the present application can be used for the perspective (Point Of View, POV) correction of the head-mounted display device 100 or the correction of other algorithms applied to the head-mounted display device 100, so that the image captured by the perception camera 20 is projected onto the display optical machine 30 with high precision, and the user has a better experience when wearing the head-mounted display device 100.

[0135] Please refer to Figure 11, an embodiment of the present application further provides a relative pose calibration device 200, which is applied to a head-mounted display device 100. The head-mounted display device 100 includes a display body 10, a sensing camera 20, and a display optical engine 30. The sensing camera 20 and the display optical engine 30 are both fixedly arranged on the display body 10. The relative pose calibration device 200 includes a calibration module 210 and a calculation module 220. The calibration module 210 is used to perform a first calibration on the first camera 101 and the second camera 102 with relatively fixed positions to obtain a first relative pose relationship between the first camera 101 and the second camera 102. The calibration module 210 is further used to align the fields of view of the first camera 101 and the display optical engine 30 and perform a second calibration on the second camera 102 and the sensing camera 20 to obtain a second relative pose relationship between the second camera 102 and the sensing camera 20, wherein the relatively fixed positions enable the second camera 102 and the sensing camera 20 to have a common viewing area for calibration. The calculation module 220 is used to calculate a third relative pose relationship between the display optical engine 30 and the sensing camera 20 according to the first relative pose relationship and the second relative pose relationship.

[0136] In some embodiments, the first camera 101 and the second camera 102 are relatively fixedly connected to each other through a rigid connecting body 103. When the fields of view of the first camera 101 and the display optical engine 30 are aligned, the rigid connecting body 103 can stagger the field of view range of the second camera 102 from the occlusion range of the display body 10 and the display optical engine 30.

[0137] In some embodiments, when the fields of view of the first camera 101 and the display optical engine 30 are aligned, the distance between the first camera 101 and the display optical engine 30 is equal to the exit pupil distance.

[0138] In some embodiments, the display body 10 includes a left-eye body 11. The sensing camera 20 includes a left-eye camera 20a. The display optical engine 30 includes a left-eye optical engine 31. The calibration module 210 is specifically used to align the fields of view of the first camera 101 and the left-eye optical engine 31 and perform a second calibration on the second camera 102 and the left-eye camera 20a to obtain a second left relative pose relationship between the second camera 102 and the left-eye camera 20a, wherein the relatively fixed positions enable the second camera 102 and the left-eye camera 20a to have a common viewing area for calibration. The calculation module 220 is specifically used to calculate a third left relative pose relationship between the left-eye optical engine 31 and the left-eye camera 20a according to the first relative pose relationship and the second relative pose relationship.

[0139] In some embodiments, the left-eye camera 20a includes a left-eye video see-through camera 21 and / or a left-eye spatial pose detection camera 22.

[0140] In some embodiments, the display body 10 includes a right-eye body 12. The sensing camera 20 includes a right-eye camera 20b. The display optical engine 30 includes a right-eye optical engine 32. The calibration module 210 is specifically configured to align the field of view of the first camera 101 with that of the right-eye optical engine 32 and perform a second calibration on the second camera 102 and the right-eye camera 20b to obtain a second right relative pose relationship between the second camera 102 and the right-eye camera 20b, wherein the positions are relatively fixed such that the second camera 102 and the right-eye camera 20b have a common viewing area for calibration. The calculation module 220 is specifically configured to calculate a third right relative pose relationship between the right-eye optical engine 32 and the right-eye camera 20b according to the first relative pose relationship and the second relative pose relationship.

[0141] In some embodiments, the right-eye camera 20b includes a right-eye video see-through camera 23 and / or a right-eye spatial pose detection camera 24.

[0142] In some embodiments, the calibration module 210 is specifically configured to perform a first calibration on the first camera 101 and the second camera 102 with relatively fixed positions to obtain a first rotation matrix and a first translation matrix of the relative pose between the first camera 101 and the second camera 102; perform a second calibration on the second camera 102 and the sensing camera 20 to obtain a second rotation matrix and a second translation matrix of the relative pose between the second camera 102 and the sensing camera 20. The calculation module 220 is specifically configured to calculate a third translation matrix of the relative pose between the display optical engine 30 and the sensing camera 20 according to the first translation matrix and the second translation matrix.

[0143] It should be noted that the foregoing explanations of the relative pose calibration method in the foregoing embodiments are equally applicable to the relative pose calibration device 200 of the embodiments of the present application, and will not be elaborated herein.

[0144] Please refer to Figure 12 , the embodiments of the present application further provide a relative pose calibration system 300, including one or more processors 310 and a memory 320. The memory 320 stores a computer program, and when the computer program is executed by the processor 310, the relative pose calibration method of any of the foregoing embodiments is implemented.

[0145] For example, when the computer program is executed by the processor 310, the following relative pose calibration method is implemented:

[0146] S1. Perform a first calibration on the first camera 101 and the second camera 102 with relatively fixed positions to obtain a first relative pose relationship between the first camera 101 and the second camera 102;

[0147] S2. Align the first camera 101 with the field of view of the display optical engine 30 and perform a second calibration on the second camera 102 and the sensing camera 20 to obtain the second relative pose relationship between the second camera 102 and the sensing camera 20. Among them, the positions are relatively fixed so that the second camera 102 and the sensing camera 20 have a common viewing area for calibration;

[0148] S3. Calculate the third relative pose relationship between the display optical engine 30 and the sensing camera 20 according to the first relative pose relationship and the second relative pose relationship.

[0149] For another example, when the computer program is executed by the processor 310, the following relative pose calibration method is implemented:

[0150] Aligning the first camera 101 with the field of view of the display optical engine 30 and performing a second calibration on the second camera 102 and the sensing camera 20 in S2 to obtain the second relative pose relationship between the second camera 102 and the sensing camera 20 includes:

[0151] S21. Align the first camera 101 with the field of view of the left-eye optical engine 31, stagger the field of view range of the second camera 102 from the occlusion range of the left-eye body 11 and the left-eye optical engine 31, and perform a second calibration on the second camera 102 and the left-eye camera 20a to obtain the second left relative pose relationship between the second camera 102 and the left-eye camera 20a;

[0152] Calculating the third relative pose relationship between the display optical engine 30 and the sensing camera 20 according to the first relative pose relationship and the second relative pose relationship in S3 includes:

[0153] S31. Calculate the third left relative pose relationship between the left-eye optical engine 31 and the left-eye camera 20a according to the first relative pose relationship and the second relative pose relationship.

[0154] It should be noted that the foregoing explanations of the relative pose calibration method and the relative pose calibration device 200 in the foregoing embodiments are equally applicable to the relative pose calibration system 300 of the embodiments of the present application, and will not be elaborated herein.

[0155] Please refer to Figure 13 , the embodiments of the present application further provide a computer-readable storage medium 400, on which a computer program 410 is stored. It is characterized in that when the program is executed by the processor 420, the relative pose calibration method of any of the foregoing embodiments is implemented.

[0156] For example, when the program is executed by the processor 420, the following relative pose calibration method is implemented:

[0157] S1. Perform the first calibration on the relatively fixed first camera 101 and second camera 102 to obtain the first relative pose relationship between the first camera 101 and the second camera 102;

[0158] S2. Align the field of view of the first camera 101 with the display optical machine 30, stagger the field of view range of the second camera 102 from the occlusion range of the display body 10 and the display optical machine 30, and perform the second calibration on the second camera 102 and the sensing camera 20 to obtain the second relative pose relationship between the second camera 102 and the sensing camera 20;

[0159] S3. Calculate the third relative pose relationship between the display optical machine 30 and the sensing camera 20 according to the first relative pose relationship and the second relative pose relationship.

[0160] For another example, when the program is executed by the processor 420, the following relative pose calibration method is implemented:

[0161] In S2, aligning the field of view of the first camera 101 with the display optical machine 30, staggering the field of view range of the second camera 102 from the occlusion range of the display body 10 and the display optical machine 30, and performing the second calibration on the second camera 102 and the sensing camera 20 to obtain the second relative pose relationship between the second camera 102 and the sensing camera 20 includes:

[0162] S21. Align the field of view of the first camera 101 with the left-eye optical machine 31, stagger the field of view range of the second camera 102 from the occlusion range of the left-eye body 11 and the left-eye optical machine 31, and perform the second calibration on the second camera 102 and the left-eye camera 20a to obtain the second left relative pose relationship between the second camera 102 and the left-eye camera 20a;

[0163] In S3, calculating the third relative pose relationship between the display optical machine 30 and the sensing camera 20 according to the first relative pose relationship and the second relative pose relationship includes:

[0164] S31. Calculate the third left relative pose relationship between the left-eye optical machine 31 and the left-eye camera 20a according to the first relative pose relationship and the second relative pose relationship.

[0165] It should be noted that the foregoing explanations of the relative pose calibration method and the relative pose calibration device 200 in the foregoing embodiments are equally applicable to the computer-readable storage medium 400 of the embodiments of the present application, and will not be elaborated herein.

[0166] In summary, in the relative pose calibration method, relative pose calibration device 200, relative pose calibration system 300, and computer-readable storage medium 400 according to the embodiments of the present application, the first camera 101 is aligned with the field of view of the display optical machine 30, and the positions of the first camera 101 and the second camera 102 are relatively fixed, so that the second camera 102 and the sensing camera 20 have a common viewing area for calibration. The pose of the display optical machine 30 is characterized by the pose of the first camera 101, and the third relative pose relationship between the display optical machine 30 and the sensing camera 20 is calculated according to the first relative pose relationship between the first camera 101 and the second camera 102 and the second relative pose relationship between the second camera 102 and the sensing camera 20. In this way, the relative pose calibration between the display optical machine 30 and the sensing camera 20 can be efficiently completed, and the calibration accuracy is relatively high.

[0167] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0168] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in the reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the art of the embodiments of the present application.

[0169] The logic and / or steps represented in the flowchart or otherwise described herein can, for example, be considered as a definable sequence of executable instructions for implementing logical functions, which can be embodied in any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device. For the purposes of this specification, a computer-readable storage medium can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection part having one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable storage medium can even be paper or other suitable media on which the program can be printed, as the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then stored in a computer memory.

[0170] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0171] Those of ordinary skill in the art can understand that all or part of the steps carried out in the methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments. In addition, in each of the embodiments of the present application, each functional unit can be integrated in a processing module, or each unit can exist physically alone, or two or more units can be integrated in a module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disc, etc.

[0172] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. A relative pose calibration method, applied to a head-mounted display device, characterized in that The head-mounted display device includes a display body, a perception camera, and a display optical engine. The perception camera and the display optical engine are both fixedly arranged on the display body. The relative pose calibration method includes: S1. Perform a first calibration on a first camera and a second camera with relatively fixed positions to obtain a first relative pose relationship between the first camera and the second camera; S2. Align the field of view of the first camera with the display optical engine and perform a second calibration on the second camera and the perception camera to obtain a second relative pose relationship between the second camera and the perception camera. Wherein, the relatively fixed position enables the second camera and the perception camera to have a common viewing area for calibration; S3. Calculate a third relative pose relationship between the display optical engine and the perception camera according to the first relative pose relationship and the second relative pose relationship.

2. The relative pose calibration method according to claim 1, wherein The first camera and the second camera are relatively fixedly connected through a rigid connecting body; When the field of view of the first camera is aligned with the display optical engine, the rigid connecting body can stagger the field of view range of the second camera from the occlusion range of the display body and the display optical engine to have a common viewing area with the perception camera.

3. The relative pose calibration method according to claim 1, wherein When the field of view of the first camera is aligned with the display optical engine, the first camera is located at the eye point position of the display optical engine.

4. The relative pose calibration method according to claim 1, wherein The display body includes a left-eye body, the perception camera includes a left-eye camera, and the display optical engine includes a left-eye optical engine; The step of aligning the field of view of the first camera with the display optical engine and performing a second calibration on the second camera and the perception camera in S2 to obtain a second relative pose relationship between the second camera and the perception camera includes: S21. Align the field of view of the first camera with the left-eye optical engine and perform a second calibration on the second camera and the left-eye camera to obtain a second left relative pose relationship between the second camera and the left-eye camera. Wherein, the relatively fixed position enables the second camera and the left-eye camera to have a common viewing area for calibration; The step of calculating a third relative pose relationship between the display optical engine and the perception camera according to the first relative pose relationship and the second relative pose relationship in S3 includes: S31. Calculate a third left relative pose relationship between the left-eye optical engine and the left-eye camera according to the first relative pose relationship and the second relative pose relationship.

5. The relative pose calibration method according to claim 4, wherein The left-eye camera includes a left-eye video see-through camera and / or a left-eye spatial pose detection camera.

6. The relative pose calibration method according to claim 1, characterized in that The display body includes a right-eye body, the perception camera includes a right-eye camera, and the display optical engine includes a right-eye optical engine; The step of aligning the field of view of the first camera with the display optical engine and performing a second calibration on the second camera and the perception camera in S2 to obtain a second relative pose relationship between the second camera and the perception camera includes: S22. Align the first camera with the right-eye optical machine's field of view and perform a second calibration on the second camera and the right-eye camera to obtain the second right relative pose relationship between the second camera and the right-eye camera. Herein, the positions are relatively fixed such that the second camera and the right-eye camera have a common viewing area for calibration; The calculation of the third relative pose relationship between the display optical machine and the sensing camera according to the first relative pose relationship and the second relative pose relationship in S3 includes: S32. Calculate the third right relative pose relationship between the right-eye optical machine and the right-eye camera according to the first relative pose relationship and the second relative pose relationship.

7. The relative pose calibration method according to claim 6, characterized in that The right-eye camera includes a right-eye video perspective camera and / or a right-eye spatial pose detection camera.

8. The relative pose calibration method according to claim 1, characterized in that, The first calibration of the first camera and the second camera with relatively fixed positions in S1 to obtain the first relative pose relationship between the first camera and the second camera includes: S11. Perform a first calibration on the first camera and the second camera with relatively fixed positions to obtain the first rotation matrix and the first translation matrix of the relative pose between the first camera and the second camera; The second calibration of the second camera and the sensing camera in S2 to obtain the second relative pose relationship between the second camera and the sensing camera includes: S23. Perform a second calibration on the second camera and the sensing camera to obtain the second rotation matrix and the second translation matrix of the relative pose between the second camera and the sensing camera; The calculation of the third relative pose relationship between the display optical machine and the sensing camera according to the first relative pose relationship and the second relative pose relationship in S3 includes: S33. Calculate the third rotation matrix of the relative pose between the display optical machine and the sensing camera according to the first rotation matrix and the second rotation matrix; S34. Calculate the third translation matrix of the relative pose between the display optical machine and the sensing camera according to the first translation matrix and the second translation matrix.

9. A relative pose calibration device, applied to a head-mounted display device, characterized in that The head-mounted display device includes a display body, a sensing camera, and a display optical machine. The sensing camera and the display optical machine are both fixedly arranged on the display body. The relative pose calibration device includes: A calibration module for performing a first calibration on the first camera and the second camera with relatively fixed positions to obtain the first relative pose relationship between the first camera and the second camera; The calibration module is further configured to align the first camera with the field of view of the display optical machine and perform a second calibration on the second camera and the sensing camera to obtain the second relative pose relationship between the second camera and the sensing camera. Herein, the positions are relatively fixed such that the second camera and the sensing camera have a common viewing area for calibration; A calculation module for calculating the third relative pose relationship between the display optical machine and the sensing camera according to the first relative pose relationship and the second relative pose relationship.

10. A head-mounted display device, characterized in that, It includes one or more processors and a memory, and the memory stores a computer program. When the computer program is executed by the processor, it implements the relative pose calibration method according to any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the relative pose calibration method according to any one of claims 1-8.

Images