Image-based calibration method and system for head-mounted display device

Abstract

The present application relates to a calibration method for an image-based head-mounted display device, wherein the method includes: a tracking camera and a calibration camera shooting a target device, respectively acquiring a first image and a second image; calculating a first extrinsic parameter of the calibration camera relative to the tracking camera based on the first image, the second image, the tracking camera intrinsic parameter, and the calibration camera intrinsic parameter; a virtual target rendering unit determining the position and posture of the tracking camera based on the first image, and generating a virtual target based on the position and posture, the first image, and the first extrinsic parameter and rendering it to the virtual camera, the virtual camera shooting the virtual target to acquire a third image; the head-mounted display device images the virtual target on the observation screen, and the calibration camera shoots the virtual target on the observation screen to acquire a fourth image; calculating perspective parameters based on the calibration camera intrinsic parameter, the first extrinsic parameter, the third image, and the fourth image, and calibrating the head-mounted display device based on the perspective parameters. Through this application, the problem of poor performance of the head-mounted display device calibration method is solved, and the calibration accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of head-mounted display device calibration, and in particular to an image-based head-mounted display device calibration method and system. Background Art

[0002] Augmented reality (AR) technology is currently widely used in fields such as medicine, industry, and the military. Optical see-through (OST) head-mounted displays (HMDs), as a carrier of AR technology, not only incorporate many AR technologies but also provide users with a better sense of immersion and experience than other carriers (such as mobile phones).

[0003] Unlike Video See-Through (VST), OST requires calibration of the system combining the human eye and HMD optics because the human eye is involved in the entire imaging process.

[0004] In related technology, the calibration method provided by Chinese patent CN1802586A projects a displayed reference point onto a calibration screen, manually adjusts its position to align it with the virtual reference point and the displayed reference point, and then obtains the calibration parameters. This method, relying on manual acquisition of corresponding reference points, is not only unsuitable for batch operations, but also manually adjusting the position to align the reference points is limited by the operator's control accuracy, resulting in large errors in the final reference point coordinates, which are reflected in the final calibration parameters.

[0005] The calibration method provided by Chinese patent CN105320271A is somewhat simplified. The external parameters of the tracking camera and the left and right virtual cameras are obtained by simultaneously observing a checkerboard calibration plate. However, the intrinsic parameters of the left and right virtual cameras are still calibrated manually. In order to simplify the complexity of manual calibration, it is assumed that the rotation error of the virtual camera itself is not large, so only the translation vector and eye distance are calibrated. However, this assumption is actually not valid. Especially when the OST calibration accuracy is high, the rotation error must be calibrated, otherwise the projected virtual image will deviate significantly from the actual scene.

[0006] Currently, no effective solution has been proposed to the problem of poor calibration effect of head-mounted display device calibration methods. Summary of the Invention

[0007] The embodiments of the present application provide an image-based calibration method, system, and computer device for a head-mounted display device, to at least address the problem of poor performance of head-mounted display device calibration methods in related arts.

[0008] In a first aspect, an embodiment of the present application provides an image-based calibration method for a head-mounted display device, the method comprising:

[0009] The tracking camera and the calibration camera shoot the target device to obtain a first image and a second image respectively;

[0010] A parameter calculation unit calculates a first extrinsic parameter of the calibration camera relative to the tracking camera based on the first image, the second image, an intrinsic parameter of the tracking camera, and an intrinsic parameter of the calibration camera;

[0011] A virtual target rendering unit determines a position and posture of the tracking camera based on the first image, generates a virtual target based on the position and posture, the first image, and the first external parameter, and renders the virtual target to a virtual camera, and the virtual camera photographs the virtual target to obtain a third image;

[0012] The head-mounted display device images the virtual target on an observation screen, and the calibration camera photographs the virtual target on the observation screen to obtain a fourth image;

[0013] The parameter calculation unit calculates perspective parameters according to the calibrated camera intrinsic parameters, the first extrinsic parameters, the third image, and the fourth image, and calibrates the head-mounted display device based on the perspective parameters.

[0014] In some embodiments, before the tracking camera and the calibration camera photograph the target device, the method further includes:

[0015] Build a calibration environment based on the target device, head display device, mask device, human head camera model and controller, where:

[0016] The calibration camera is set at the eye position of the human head camera model, the head display device and the mask device are installed on the human head camera model, the field of view of the head display device and the human head camera model surround the target device, and the controller is communicatively connected with the target device, the mask device, the head display device and the human head camera model.

[0017] In some embodiments, the mask device includes a light shielding plate, a substrate, a motor and a transmission device, wherein:

[0018] When the calibration camera captures the target device to obtain a second image, the controller sends a first operation instruction to instruct the visor to be set to a lowered state in which the visor does not block the lens of the head-mounted display device, and instructs the head-mounted display device to turn off the optical machine;

[0019] When the calibration camera captures the virtual target on the observation screen to obtain a fourth image, the controller sends a second operation instruction to instruct the visor to be set to a raised state to block the lens of the head-mounted display device, and instructs the head-mounted display device to turn on the optical machine.

[0020] In some embodiments, the virtual target rendering unit determines the pose of the tracking camera based on the first image, and generates a virtual target according to the pose, the first image, and the first external parameter, including:

[0021] A virtual target rendering unit acquires the first image, inputs the first image into a marker tracking library, and determines the position of the tracking camera relative to a world coordinate system through the marker tracking library;

[0022] transforming the marker information contained in the first image from a world coordinate system to a tracking camera coordinate system according to the pose;

[0023] Convert the marking information contained in the first image from the tracking camera coordinate system to the calibration camera coordinate system according to the first extrinsic parameter and calculate a transformation matrix;

[0024] The virtual target rendering generates a virtual target according to the marking information in the calibration camera coordinate system and the transformation matrix, wherein the virtual target has the same size as the target device.

[0025] In some embodiments, the parameter calculation unit calculates the perspective parameter according to the calibrated camera intrinsic parameter, the first extrinsic parameter, the third image, and the fourth image, including:

[0026] Calculating a second extrinsic parameter and an intrinsic parameter of the virtual camera relative to the calibration camera according to the third image and the fourth image;

[0027] Calculate a third extrinsic parameter of the tracking camera relative to the virtual camera according to the first extrinsic parameter, the second extrinsic parameter, and the intrinsic parameter of the virtual camera;

[0028] The third external parameter and the virtual camera internal parameter are combined and output as the perspective parameter.

[0029] In some embodiments, calibrating the head-mounted display device based on the perspective parameter includes:

[0030] The virtual target rendering unit receives the perspective parameter, generates a corrected virtual target according to the perspective parameter, and renders the virtual target to the virtual camera;

[0031] photographing the target device with a calibration camera to obtain a second image, and photographing the corrected virtual target on the observation screen to obtain a fourth image;

[0032] The corrected second image and fourth image are input to the parameter calculation unit, and the parameter calculation unit outputs accuracy index information of the perspective parameters.

[0033] In some embodiments, after calibrating the head-mounted display device based on the perspective parameters, the method further includes:

[0034] Providing a user with a customized parameter adjustment interface on the observation screen of the head-mounted display device through a fine-tuning module;

[0035] receiving interactive information input by a user on the custom parameter adjustment interface, adjusting the axis angle and field angle of the virtual camera according to the interactive information, and obtaining the current axis angle and field angle by the fine-tuning module after the second image and the fourth image overlap;

[0036] Save the current axis angle, field angle, and perspective parameters as high-precision perspective parameters.

[0037] In a second aspect, an embodiment of the present application provides an image-based calibration system for a head-mounted display device, the system comprising: a target device, a mask device, a human head camera model, a controller, and a head-mounted display device, wherein:

[0038] The head-mounted display device and the mask device are installed on the human head camera model, the field of view of the head-mounted display device and the human head camera model includes the target device, and the controller is communicatively connected with the target device, the mask device, the head-mounted display device and the human head camera model;

[0039] The head-mounted display device is used to capture the target device through a tracking camera to obtain a first image;

[0040] A calibration camera is provided at the eye position of the head camera model, and the calibration camera is used to simulate a human eye and photograph the target device to obtain a second image;

[0041] The controller is configured to calculate a first extrinsic parameter of the calibration camera relative to the tracking camera based on the first image, the second image, the tracking camera intrinsic parameter, and the calibration camera intrinsic parameter through a parameter calculation unit.

[0042] and determining, by a virtual target rendering unit, a position and posture of the tracking camera based on the first image, generating a virtual target according to the position and posture, the first image, and the first external parameter, rendering the virtual target to a virtual camera, and photographing the virtual target by the virtual camera to obtain a third image;

[0043] The head-mounted display device is used to image the virtual target on the observation screen, and instruct the calibration camera to capture the virtual target on the observation screen to obtain a fourth image;

[0044] The controller is further configured to calculate perspective parameters based on the calibrated camera intrinsic parameters, the first extrinsic parameters, the third image, and the fourth image, and calibrate the head-mounted display device based on the perspective parameters.

[0045] In some embodiments, the mask device includes a light shielding plate, a substrate, a motor, and a transmission device, wherein:

[0046] When the calibration camera captures the target device to obtain a second image, the controller sends a first operation instruction to instruct the visor to be set to a lowered state in which the visor does not block the lens of the head-mounted display device, and instructs the head-mounted display device to turn off the optical machine;

[0047] When the calibration camera captures the virtual target on the observation screen to obtain a fourth image, the controller sends a second operation instruction to instruct the visor to be set to a raised state to block the lens of the head-mounted display device, and instructs the head-mounted display device to turn on the optical machine.

[0048] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the image-based head-mounted display device calibration method as described in the first aspect above is implemented.

[0049] Compared to related technologies, the image-based head-mounted display device calibration method provided in the embodiments of the present application has the following beneficial effects:

[0050] 1. Because it uses a human head camera model and a calibration camera set at the eye position to simulate the human eye, calibration only requires the device to be mounted on the human head camera model, improving versatility and adapting to various types of head-mounted displays.

[0051] 2. The calibration method implemented in this paper adopts an automated approach throughout the process, without the need for manual intervention, thus avoiding the uncertainty caused by manual operation and improving the efficiency and accuracy of equipment calibration;

[0052] 3. To address the differences in eye position among different people, additional fine-tuning functions can enable wider adaptation and higher-precision correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0054] Figure 1 is a schematic diagram of a head camera model according to an embodiment of the present application;

[0055] Figure 2 is a schematic diagram of a coordinate system of a calibration system according to an embodiment of the present application;

[0056] Figure 3 is a flowchart of a calibration method for an image-based head-mounted display device according to an embodiment of the present application;

[0057] Figure 4 2 is a schematic diagram of an application environment of an image-based head-mounted display device calibration method according to an embodiment of the present application;

[0058] Figure 5 is a schematic diagram of a rendering process of a virtual target rendering unit according to an embodiment of the present application;

[0059] Figure 6 is a flow chart of parameter calculation according to an embodiment of the present application;

[0060] Figure 7 is a schematic diagram of a custom parameter adjustment interface according to an embodiment of the present application;

[0061] Figure 8 is a schematic diagram of another custom parameter adjustment interface according to an embodiment of the present application;

[0062] Figure 9 is a structural block diagram of a calibration method for an image-based head-mounted display device according to an embodiment of the present application;

[0063] Figure 10 2. A schematic diagram of a mask device of a calibration system for a head-mounted display device based on an image device according to an embodiment of the present application;

[0064] Figure 11 is a schematic diagram of a sunshade in a lowered state according to an embodiment of the present application;

[0065] Figure 12 is a schematic diagram of a sunshade in a raised state according to an embodiment of the present application;

[0066] Figure 13 It is a schematic diagram of the internal structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0068] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0069] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0070] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0071] Figure 1 is a schematic diagram of a head camera model according to an embodiment of the present application, such as Figure 1As shown, a calibration camera is set at the position of both eyes of the human head camera model, and the head display device is calibrated by simulating human eyes through the calibration camera.

[0072] Figure 2 is a schematic diagram of a coordinate system of a calibration system according to an embodiment of the present application, such as Figure 2 As shown in Figure 1, each component of the calibration system corresponds to a reference coordinate system. The tracking camera is typically a video camera in a head-mounted display system, corresponding to a camera coordinate system Ft(X, Y, Z). The imaging plane of the tracking camera corresponds to an image screen coordinate system Fti(U, V).

[0073] The head-mounted display device's display screen corresponds to a display screen coordinate system Fd(U, V), and the head-mounted display device's translucent observation screen corresponds to an observation image plane coordinate system Fdi(X, Y, Z). It should be noted that for AR devices, the display screen is a physical hardware device, while the translucent observation screen is a virtual plane observed through the display screen by a person wearing the AR device, generated through optical imaging. Furthermore, the virtual camera corresponds to a virtual camera coordinate system Fv(X, Y, Z), and the calibration camera's coordinate system Fcc(X, Y, Z) is represented by a dotted line.

[0074] Depend on Figure 2 It can be seen that after calibration, the coordinate system Fcc (X, Y, Z) of the calibration camera used to simulate the human eye coincides with the origin of the coordinate system Fv (X, Y, Z) of the virtual camera, but there is a deviation between its coordinate axes and the three coordinate axes of the corresponding virtual camera coordinate system. Therefore, this deviation needs to be further corrected.

[0075] This embodiment provides a method for calibrating a head-mounted display device based on an image. Figure 3 FIG. 1 is a flow chart of a calibration method for an image-based head-mounted display device according to an embodiment of the present application. Figure 3 As shown, the process includes the following steps:

[0076] S301, a tracking camera and a calibration camera photograph a target device to obtain a first image and a second image respectively;

[0077] The target device can be any open or custom calibration plate, such as checkerboard, Aruco, Charuco, Apriltags or other custom marker plates.

[0078] Furthermore, the tracking camera captures the first image for the purpose of subsequently obtaining the pose of the head-mounted display device relative to the target device. However, in this embodiment, methods for obtaining the pose include, but are not limited to, the aforementioned methods. Alternatively, the pose of the head-mounted display device relative to the target device may be obtained using an external online motion capture system such as Vicon.

[0079] S302, a parameter calculation unit calculates a first extrinsic parameter of the calibration camera relative to the tracking camera based on the first image, the second image, the tracking camera intrinsic parameter, and the calibration camera intrinsic parameter;

[0080] Optionally, the parameter calculation unit is deployed in the controller. In addition, the tracking camera and the calibration camera can be calibrated with internal parameters through automated or manual methods. Since the internal parameter calibration method is a relatively mature technology in the field and has no impact on the core invention of this application, it will not be described in detail in this embodiment.

[0081] S303, a virtual target rendering unit determines a position and posture of the tracking camera based on the first image, generates a virtual target based on the position and posture, the first image, and the first external parameter, and renders the generated virtual target to the virtual camera, which captures the virtual target to obtain a third image;

[0082] The virtual target rendering unit can generate a virtual target with the same size and pattern as the target device based on the virtual target image taken by the tracking camera and the posture of the tracking camera, and convert the virtual target to the coordinate system of the calibration camera according to the first external parameter (calibration camera relative to tracking camera).

[0083] Furthermore, a virtual camera is constructed at a default position that coincides with the calibration camera coordinate system, and the virtual target converted to the calibration camera coordinate system is rendered to the virtual camera, and the virtual target is photographed by the virtual camera to obtain a third image.

[0084] It should be noted that the virtual camera is created with a rendering engine (such as Unity3D, Unreal Engine, etc.) as the core component. Correspondingly, in this embodiment, the process of shooting the virtual target to obtain the third image is a figurative description, and the actual process is similar to saving the digital signal corresponding to the image in the form of a screenshot in an image environment.

[0085] S304: The head-mounted display device forms an image of a virtual target on the observation screen, and the calibration camera photographs the virtual target on the observation screen to obtain a fourth image;

[0086] The head-mounted display device images a virtual target on the observation screen through its own optical imaging system.

[0087] It should be noted that the first and second images captured above are multiple sets of images. During the acquisition of the first and second images, a controller is required to output control signals to change the position and angle of the target device. Furthermore, when the virtual camera captures the third and fourth images, multiple sets of images are acquired for parameter calculation, also while updating and generating multiple virtual targets.

[0088] S305: A parameter calculation unit calculates perspective parameters based on the calibrated camera intrinsic parameters, the first extrinsic parameters, the third image, and the fourth image, and calibrates the head-mounted display device based on the perspective parameters. The perspective parameters include the extrinsic parameters of the tracking camera relative to the virtual camera and the intrinsic parameters of the virtual camera.

[0089] Through steps S301 to S305 described above, this embodiment of the present application provides a fully automated calibration process. After placing the head-mounted display device on a human head camera model, the perspective parameters of the head-mounted display device can be obtained through this automated process without any manual operation. Compared to related head-mounted display device calibration methods, this avoids errors caused by manual operation and improves calibration effectiveness and efficiency.

[0090] In some of the embodiments, before calibrating the head-mounted display device, it is necessary to build a calibration environment based on a target device, a head-mounted display device, a mask device, a human head camera model, and a controller.

[0091] Figure 4 is a schematic diagram of an application environment of an image-based head-mounted display device calibration method according to an embodiment of the present application, such as Figure 4 As shown, a head-mounted display device 41 and a mask device 43 are installed on a human head camera model 42, a calibration camera is set at the eye position of the human head camera model 42, the field of view of the head-mounted display device 41 and the human head camera model 42 surrounds the target device 44, and a controller 45 is communicatively connected with the mask device 43, the head-mounted display device 41 and the human head camera model 42.

[0092] It should be noted that the method provided in this embodiment is applicable not only to optical head-mounted displays (such as optical waveguide devices and free-form surface devices), but also to mixed reality (MR) devices. It is applicable not only to single-eye but also binocular head-mounted displays. Furthermore, the controller 45 includes, but is not limited to, desktop computers, laptop computers, and industrial control computers.

[0093] In some embodiments, Figure 5 FIG. 1 is a schematic diagram of a rendering process of a virtual target rendering unit according to an embodiment of the present application. Figure 5 As shown, the virtual target rendering unit obtains a first image taken by the tracking camera, inputs the first image into a marker tracking library (such as a maker tracking library), and determines the position Tt<-w of the tracking camera relative to the world coordinate system through the marker tracking library;

[0094] Further, the marker information contained in the first image is converted from a world coordinate system to a tracking camera coordinate system according to the pose;

[0095] According to a first external parameter Tcc<-t of the calibration camera relative to the tracking camera, the marker information contained in the first image is converted from the tracking camera coordinate system to the calibration camera coordinate system and a transformation matrix is calculated;

[0096] A virtual target is generated based on the marking information and transformation matrix in the calibration camera coordinate system, and the image of the virtual target is transmitted to the optical machine of the head-mounted display device, wherein the virtual target has the same size as the target device.

[0097] In some embodiments, Figure 6 is a flow chart of parameter calculation according to an embodiment of the present application, such as Figure 6 As shown, the following steps are included:

[0098] The first step is to calculate the extrinsic parameter Tcc<-t(first extrinsic parameter) of the calibration camera relative to the tracking camera based on the target image (first image) taken by the tracking camera, the real target image (second image) taken by the calibration camera, the intrinsic parameters of the tracking camera, and the intrinsic parameters of the calibration camera.

[0099] The second step is to calculate the second external parameter Tv<-cc of the virtual camera relative to the calibration camera and the internal parameter projection matrix Pd<-v of the virtual camera based on the virtual target image (third image) taken by the virtual camera, the target virtual image (fourth image) taken by the calibration camera, and the internal parameters of the calibration camera.

[0100] In the third step, the third extrinsic parameter Tv-t of the tracking camera relative to the virtual camera is calculated based on the first extrinsic parameter Tcc<-t and the second extrinsic parameter Tv<-cc; and the third extrinsic parameter Tv-t and the virtual camera intrinsic parameter Pd<-v are saved as perspective parameters for subsequent calibration.

[0101] In some embodiments, calibrating a head-mounted display device based on perspective parameters includes: a virtual target rendering unit receiving perspective parameters, generating a corrected virtual target according to the perspective parameters, and rendering the result to a virtual camera; simultaneously, capturing a second image by calibrating a camera to capture a target device, and capturing a fourth image by capturing the corrected virtual target on an observation screen; inputting the corrected second and fourth images to a parameter calculation unit, which outputs accuracy index information of the perspective parameters, wherein the accuracy index information includes but is not limited to: average reprojection root mean square error, reprojection root mean square error variance, and maximum reprojection root mean square error.

[0102] In some embodiments, considering that each user has a different pupil distance and the positions of the left and right eyes and the calibration camera are also different when actually wearing the headset, there may still be a slight deviation between the virtual image seen and the actual image. This embodiment also provides a method for quickly improving the accuracy of the parameters of the head-mounted display device, including:

[0103] Providing a user with a custom parameter adjustment interface on the observation screen of the head-mounted display device through a fine-tuning module;

[0104] receiving interactive information input by the user on the custom parameter adjustment interface, adjusting the axis angle and field angle of the virtual camera according to the interactive information, and obtaining the current axis angle and field angle by the fine-tuning module after the second image and the fourth image overlap;

[0105] Save the current axis angle, field angle, and perspective parameters as high-precision perspective parameters.

[0106] Figure 7 is a schematic diagram of a custom parameter adjustment interface according to an embodiment of the present application, such as Figure 7 As shown, the fine-tuning module provides the user with a customized parameter adjustment interface on the display screen of the head-mounted display device, so that the user can customize the parameters of the virtual camera so that the user can comfortably watch the virtual image presented by the head-mounted display device;

[0107] Figure 8 is a schematic diagram of another custom parameter adjustment interface according to an embodiment of the present application, such as Figure 8 As shown, this solution is also applicable to dual-lens head-mounted display devices; further, the binocular imaging effect can be adjusted by adjusting parameters to eliminate problems such as blurring, ghosting, and inaccurate depth that cause user discomfort.

[0108] Through the custom adjustment function provided by this embodiment, after the user puts on the head-mounted display device, the virtual target rendering unit first renders a virtual target of the same size on the display screen of the head-mounted display device; secondly, the user looks at the virtual target and adjusts the three axis angles and field of view angle of the virtual camera at the position of the human eye through the fine-tuning module until the virtual image and the real image completely overlap.

[0109] This embodiment also provides an image-based head-mounted display device calibration system, which is used to implement the above-mentioned embodiments and preferred embodiments. Details already described are not repeated here. As used below, terms such as "module," "unit," and "subunit" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0110] Figure 9 is a structural block diagram of a calibration method for an image-based head-mounted display device according to an embodiment of the present application. Figure 9As shown, the system includes: a target device 90, a mask device 91, a human head camera model 92, a controller 94 and a head display device 93, wherein the head display device 93 and the mask device 91 are installed on the human head camera model 92, the field of view of the head display device 93 and the human head camera model 92 includes the target device 90, and the controller 94 is in communication with the target device 90, the mask device 91, the head display device 93 and the human head camera model 92;

[0111] The head-mounted display device 93 is used to capture the target device 90 through a tracking camera to obtain a first image;

[0112] A calibration camera is provided at the eye position of the head camera model 92, and the calibration camera is used to simulate a human eye and shoot the target device 90 to obtain a second image;

[0113] The controller 94 is configured to calculate a first external parameter of the calibration camera relative to the tracking camera based on the first image, the second image, the tracking camera internal parameter, and the calibration camera internal parameter through a parameter calculation unit.

[0114] and determining, by a virtual target rendering unit, a position and posture of the tracking camera based on the first image, generating a virtual target according to the position and posture, the first image, and the first external parameter, rendering the virtual target to the virtual camera, and instructing the virtual camera to photograph the virtual target 90 to obtain a third image;

[0115] The head-mounted display device 93 is used to image a virtual target on the observation screen and instruct the calibration camera to capture the virtual target on the observation screen to obtain a fourth image;

[0116] The controller 94 is further configured to calculate perspective parameters based on the calibrated camera intrinsic parameters, the first extrinsic parameters, the third image, and the fourth image, and calibrate the head-mounted display device based on the perspective parameters.

[0117] In some embodiments, Figure 10 FIG. 9 is a schematic diagram of a mask device 91 of a calibration system for a head-mounted display device based on an image device according to an embodiment of the present application. Figure 10 As shown, the mask device 91 includes a light shielding plate 910, a base plate 911, a motor 912 and a transmission device 913:

[0118] Further, Figure 11 is a schematic diagram of the sunshade in the lowered state according to an embodiment of the present application, as shown in FIG. Figure 11 As shown, when the calibration camera captures the target device 90 to obtain the second image, the controller 94 sends a first operating instruction, instructing the motor 912 to rotate forward, lowering the light shield 910 via the transmission device 913, and instructing the head-mounted display device 93 to turn off the optical engine. Thereafter, the calibration camera captures a real image of the target device 90 behind it through the lens of the head-mounted display device 93. This real image, i.e., the second image, is not interfered with by the virtual image.

[0119] Figure 12This is a schematic diagram showing light shield 910 in a raised state according to an embodiment of the present application. When the calibration camera captures a fourth image of a virtual target on the observation screen, controller 94 issues a second operating instruction, instructing motor 912 to reverse, lift light shield 910 via transmission 913, and instructing head-mounted display 63 to activate the optical engine. Subsequently, because the lenses of head-mounted display 93 are blocked by light shield 910, the real image background behind the lenses disappears, allowing the calibration camera to capture a clear virtual image of the virtual target.

[0120] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external terminal via a network connection. When executed by the processor, the computer program implements an image-based head-mounted display device calibration method. The display screen of the computer device may be a liquid crystal display or an electronic ink display. The input device of the computer device may be a touch layer covering the display screen, or may be buttons, a trackball, or a touchpad provided on the computer device housing, or may be an external keyboard, touchpad, or mouse.

[0121] In one embodiment, Figure 13 is a schematic diagram of the internal structure of a computer device according to an embodiment of the present application, such as Figure 13 As shown, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in FIG. Figure 13 As shown. The computer device includes a processor, a network interface, an internal memory, and a non-volatile memory connected via an internal bus, wherein the non-volatile memory stores an operating system, a computer program, and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with external terminals via a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program. When executed by the processor, the computer program implements an image-based head-mounted display device calibration method, and the database is used to store data.

[0122] Those skilled in the art will understand that Figure 13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0123] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for calibrating a head-mounted display device based on an image, characterized in that: The method comprises: The tracking camera and the calibration camera shoot the target device to obtain a first image and a second image respectively, wherein the calibration camera is set at the position of the eyes of the human head camera model, and the head display device is calibrated by simulating human eyes through the calibration camera; A parameter calculation unit calculates a first extrinsic parameter of the calibration camera relative to the tracking camera based on the first image, the second image, an intrinsic parameter of the tracking camera, and an intrinsic parameter of the calibration camera, wherein the tracking camera is a camera in the head-mounted display device system, which corresponds to a camera coordinate system, and an imaging plane of the tracking camera corresponds to an image screen coordinate system; A virtual target rendering unit determines a position and posture of the tracking camera based on the first image, generates a virtual target based on the position and posture, the first image, and the first external parameter, and renders the generated virtual target to a virtual camera. The virtual camera photographs the virtual target to obtain a third image. The virtual camera is constructed at a position that coincides with the calibration camera coordinate system by default. The virtual camera is created with a rendering engine as a core component. The process of photographing the virtual target to obtain the third image by the virtual camera is as follows: saving a digital signal corresponding to the image in the form of a screenshot in an image environment; The head-mounted display device images the virtual target on an observation screen, and the calibration camera photographs the virtual target on the observation screen to obtain a fourth image; The parameter calculation unit calculates perspective parameters according to the calibrated camera intrinsic parameters, the first extrinsic parameters, the third image, and the fourth image, and calibrates the head-mounted display device based on the perspective parameters.

2. The method according to claim 1, characterized in that Before the tracking camera and the calibration camera photograph the target device, the method further includes: Build a calibration environment based on the target device, head display device, mask device, human head camera model and controller, where: The calibration camera is set at the eye position of the human head camera model, the head display device and the mask device are installed on the human head camera model, the field of view of the head display device and the human head camera model surround the target device, and the controller is communicatively connected with the target device, the mask device, the head display device and the human head camera model.

3. The method according to claim 2, characterized in that The mask device includes a light shielding plate, a substrate, a motor and a transmission device, wherein: When the calibration camera captures the target device to obtain a second image, the controller sends a first operation instruction to instruct the visor to be set to a lowered state in which the visor does not block the lens of the head-mounted display device, and instructs the head-mounted display device to turn off the optical machine; When the calibration camera captures the virtual target on the observation screen to obtain a fourth image, the controller sends a second operation instruction to instruct the visor to be set to a raised state to block the lens of the head-mounted display device, and instructs the head-mounted display device to turn on the optical machine.

4. The method according to claim 1, wherein The virtual target rendering unit determines the posture of the tracking camera based on the first image, and generates a virtual target according to the posture, the first image and the first external parameter, including: A virtual target rendering unit acquires the first image, inputs the first image into a marker tracking library, and determines the position of the tracking camera relative to a world coordinate system through the marker tracking library; transforming the marker information contained in the first image from a world coordinate system to a tracking camera coordinate system according to the pose; Convert the marking information contained in the first image from the tracking camera coordinate system to the calibration camera coordinate system according to the first extrinsic parameter and calculate a transformation matrix; The virtual target rendering generates a virtual target according to the marking information in the calibration camera coordinate system and the transformation matrix, wherein the virtual target has the same size as the target device.

5. The method according to claim 1, wherein The parameter calculation unit calculates the perspective parameter according to the calibrated camera intrinsic parameter, the first extrinsic parameter, the third image, and the fourth image, including: Calculating a second extrinsic parameter and an intrinsic parameter of the virtual camera relative to the calibration camera according to the third image and the fourth image; Calculate a third extrinsic parameter of the tracking camera relative to the virtual camera according to the first extrinsic parameter, the second extrinsic parameter, and the intrinsic parameter of the virtual camera; The third external parameter and the virtual camera internal parameter are combined and output as the perspective parameter.

6. The method according to claim 1, characterized in that Calibrating the head-mounted display device based on the perspective parameter includes: The virtual target rendering unit receives the perspective parameter, generates a corrected virtual target according to the perspective parameter, and renders the virtual target to the virtual camera; photographing the target device with a calibration camera to obtain a second image, and photographing the corrected virtual target on the observation screen to obtain a fourth image; The corrected second image and fourth image are input to the parameter calculation unit, and the parameter calculation unit outputs accuracy index information of the perspective parameters.

7. The method according to claim 1, characterized in that After calibrating the head-mounted display device based on the perspective parameters, the method further includes: Providing a user with a customized parameter adjustment interface on the observation screen of the head-mounted display device through a fine-tuning module; receiving interactive information input by a user on the custom parameter adjustment interface, adjusting the axis angle and field angle of the virtual camera according to the interactive information, and obtaining the current axis angle and field angle by the fine-tuning module after the second image and the fourth image overlap; Save the current axis angle, field angle, and perspective parameters as high-precision perspective parameters.

8. A calibration system for a head-mounted display device based on an image, characterized in that: The system includes: a target device, a mask device, a human head camera model, a controller and a head display device, wherein: The head-mounted display device and the mask device are installed on the human head camera model, the field of view of the head-mounted display device and the human head camera model includes the target device, and the controller is communicatively connected with the target device, the mask device, the head-mounted display device and the human head camera model; The head-mounted display device is used to capture the target device through a tracking camera to obtain a first image; A calibration camera is provided at the eye position of the head camera model, and the calibration camera is used to simulate a human eye and photograph the target device to obtain a second image, wherein the calibration camera is provided at the eye position of the head camera model, and the head display device is calibrated by simulating the human eye through the calibration camera; The controller is configured to calculate, through a parameter calculation unit, a first extrinsic parameter of the calibration camera relative to the tracking camera based on the first image, the second image, an intrinsic parameter of the tracking camera, and an intrinsic parameter of the calibration camera, wherein the tracking camera is a camera in the head-mounted display device system, which corresponds to a camera coordinate system, and an imaging plane of the tracking camera corresponds to an image screen coordinate system; and determining the position and posture of the tracking camera based on the first image by a virtual target rendering unit, generating a virtual target according to the position and posture, the first image, and the first external parameter, and rendering the generated virtual target to a virtual camera, photographing the virtual target by the virtual camera to obtain a third image, constructing the virtual camera at a position that coincides with the calibration camera coordinate system by default, the virtual camera being created with a rendering engine as a core component, and photographing the virtual target by the virtual camera to obtain the third image in the following process: saving a digital signal corresponding to the image in the form of a screenshot in an image environment; The head-mounted display device is used to image the virtual target on the observation screen, and instruct the calibration camera to capture the virtual target on the observation screen to obtain a fourth image; The controller is further configured to calculate perspective parameters based on the calibrated camera intrinsic parameters, the first extrinsic parameters, the third image, and the fourth image, and calibrate the head-mounted display device based on the perspective parameters.

9. The system according to claim 8, characterized in that The mask device includes a light shielding plate, a substrate, a motor and a transmission device, wherein: When the calibration camera captures the target device to obtain a second image, the controller sends a first operation instruction to instruct the visor to be set to a lowered state in which the visor does not block the lens of the head-mounted display device, and instructs the head-mounted display device to turn off the optical machine; When the calibration camera captures the virtual target on the observation screen to obtain a fourth image, the controller sends a second operation instruction to instruct the visor to be set to a raised state to block the lens of the head-mounted display device, and instructs the head-mounted display device to turn on the optical machine.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the calibration method of the image-based head-mounted display device according to any one of claims 1 to 7 is implemented.

Images