Calibration detection method and system for head-mounted display device

Abstract

The present application relates to a calibration and detection method and system for a head-mounted display device. The calibration and detection method for a head-mounted display device includes: obtaining multiple sets of calibration object images obtained by synchronously photographing a calibration object with an observation camera and a tracking camera; jointly calibrating the observation camera and the tracking camera based on the multiple sets of calibration object images to obtain a joint calibration result; performing optical perspective calibration on the head-mounted display device based on the joint calibration result to obtain a target calibration result; obtaining a first image obtained by photographing the calibration object with the observation camera through a pair of eyeglass lenses; obtaining a second image obtained by photographing a virtual pattern drawn on a display screen of the head-mounted display device with the observation camera; and determining a calibration error of the target calibration result based on the first and second images. The method can implement an objective and standard optical perspective calibration verification process, thereby eliminating the need for manual intervention and improving the efficiency of calibration verification.

Description

Technical Field

[0001] The present application relates to the field of smart wearable devices, and in particular to a calibration detection method and system for head-mounted display devices. Background Art

[0002] Augmented reality (AR) or mixed reality (MR) glasses serve as core devices for human-computer interaction, playing a vital role in industrial manufacturing, healthcare, social entertainment, and education. Calibration of AR / MR glasses aims to align the spatial relationship between the virtual world presented on the AR / MR glasses and the real world, thereby ensuring an immersive user experience and accurate interaction.

[0003] Currently, after AR / MR glasses are calibrated for virtual and real-world integration, they are often manually put on and subjectively determine whether the calibration was successful based on the wearer's experience. This approach relies on manual intervention and subjective experience, making it inefficient.

[0004] There is currently no effective solution to the problem of low efficiency of calibration and verification in related technologies. Summary of the Invention

[0005] In this embodiment, a calibration detection method and system for a head-mounted display device are provided to solve the problem of low efficiency of calibration verification in related technologies.

[0006] In a first aspect, this embodiment provides a calibration and detection method for a head-mounted display device, which is used for a calibration device, wherein the calibration device includes an observation camera and a fixing bracket, wherein the fixing bracket is used to fix the observation camera and the head-mounted display device, and the head-mounted display device is fixed between the observation camera lens and a preset calibration object, so that the observation camera can photograph the preset calibration object through the glasses lens of the head-mounted display device; the method includes:

[0007] Acquire multiple sets of calibration object images obtained by synchronously photographing the calibration object with the observation camera and the tracking camera; wherein the tracking camera is a camera provided on the head-mounted display device;

[0008] performing joint calibration on the observation camera and the tracking camera according to the multiple sets of calibration object images to obtain a joint calibration result;

[0009] Based on the joint calibration result, performing optical perspective calibration on the head mounted display device to obtain a target calibration result;

[0010] Acquire a first image obtained by photographing the calibration object through the glasses lens by the observation camera;

[0011] Acquire a second image obtained by photographing a virtual pattern drawn on the display screen of the head-mounted display device with the observation camera; wherein the virtual pattern is obtained by drawing the calibration object based on the target calibration result;

[0012] A calibration error of the target calibration result is determined according to the first image and the second image.

[0013] In some embodiments, after determining the calibration error of the target calibration result, the method further includes:

[0014] When the calibration error is greater than a preset error threshold, determining that the current target calibration result does not meet the calibration qualification condition;

[0015] When the calibration error is less than or equal to the error threshold, it is determined that the current target calibration result meets the calibration qualification condition.

[0016] In some embodiments, when it is determined that the current target calibration result does not meet the calibration qualification condition, the method further includes:

[0017] Repeat the following process until the calibration error of the current target calibration result meets the calibration qualification condition:

[0018] Based on multiple sets of new calibration object images resynchronizedly captured by the observation camera and the tracking camera, the observation camera and the tracking camera are re-jointly calibrated, and based on the new joint calibration result, the head-mounted display device is re-optically perspective-calibrated, and the calibration error of the target calibration result is recalculated according to the new target calibration result.

[0019] In some embodiments, determining a calibration error of the target calibration result based on the first image and the second image includes:

[0020] Determining an associated feature pair corresponding to the first image and the second image; the associated feature pair is a feature pair in the first image and the second image that is associated with the same calibration feature on the calibration object;

[0021] A calibration error of the target calibration result is determined according to the pixel distance between the associated feature pairs.

[0022] In some embodiments, determining a calibration error of the target calibration result based on a pixel distance between the associated feature pairs includes:

[0023] Calculating the pixel distance between each set of associated feature pairs;

[0024] The pixel distances between all the associated feature pairs are counted to obtain the calibration error of the target calibration result.

[0025] In some embodiments, acquiring a plurality of sets of calibration object images obtained by synchronously photographing the calibration object with the observation camera and the tracking camera includes:

[0026] The observation camera and the tracking camera are moved to different positions in sequence, and the calibration object is photographed synchronously to obtain the calibration object images; wherein a group of the calibration object images corresponds to one position.

[0027] In some embodiments, based on the joint calibration result, optical perspective calibration is performed on the head mounted display device to obtain a target calibration result, including:

[0028] Drawing a virtual pattern corresponding to the calibration object on a display screen of the head-mounted display device;

[0029] determining the three-dimensional feature coordinates of the calibration object based on the virtual pattern, and acquiring a target image obtained by photographing the virtual pattern with the observation camera;

[0030] An optical perspective calibration result for the head-mounted display device is determined according to the three-dimensional feature coordinates, the target image, and the joint calibration result.

[0031] In a second aspect, this embodiment provides a server for performing calibration testing on a head-mounted display device based on a calibration device; the calibration device includes an observation camera and a fixing bracket, wherein the fixing bracket is used to fix the observation camera and the head-mounted display device, and the head-mounted display device is fixed between the observation camera lens and a preset calibration object, so that the observation camera can photograph the preset calibration object through the glasses lens of the head-mounted display device;

[0032] The server includes: an acquisition module, a joint calibration module, an optical perspective calibration module and a calibration detection module; wherein:

[0033] The acquisition module is configured to acquire a plurality of sets of calibration object images obtained by synchronously photographing the calibration object with the observation camera and the tracking camera; wherein the tracking camera is a camera provided on the head-mounted display device;

[0034] The joint calibration module is configured to perform joint calibration on the observation camera and the tracking camera according to the multiple sets of calibration object images to obtain a joint calibration result;

[0035] The optical perspective calibration module is configured to perform optical perspective calibration on the head mounted display device based on the joint calibration result to obtain a target calibration result;

[0036] The calibration detection module is used to obtain a first image obtained by the observation camera photographing the calibration object through the glasses lens; obtain a second image obtained by the observation camera photographing a virtual pattern drawn on the display screen of the head-mounted display device; wherein the virtual pattern is obtained by drawing the calibration object based on the target calibration result; and determine the calibration error of the target calibration result based on the first image and the second image.

[0037] In a third aspect, this embodiment provides a calibration and detection system for a head-mounted display device, comprising: a calibration device and the server described in the second aspect above;

[0038] The calibration device includes an observation camera and a fixing bracket, wherein the fixing bracket is used to fix the observation camera and the head-mounted display device, and the head-mounted display device is fixed between the observation camera lens and the preset calibration object, so that the observation camera can photograph the preset calibration object through the glasses lens of the head-mounted display device; the observation camera is communicatively connected to the server.

[0039] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored. When the program is executed by a processor, the calibration and detection method of the head-mounted display device described in the first aspect is implemented.

[0040] Compared to related technologies, this embodiment provides a calibration and detection method and system for a head-mounted display device. The calibration and detection method for a head-mounted display device includes obtaining multiple sets of calibration object images obtained by synchronously photographing a calibration object with an observation camera and a tracking camera; wherein the tracking camera is a camera installed on the head-mounted display device; based on the multiple sets of calibration object images, the observation camera and the tracking camera are jointly calibrated to obtain a joint calibration result; based on the joint calibration result, the head-mounted display device is optically perspective calibrated to obtain a target calibration result; obtaining a first image obtained by photographing the calibration object through the lens of the observation camera; obtaining a second image obtained by photographing a virtual pattern drawn on the display screen of the head-mounted display device with the observation camera; wherein the virtual pattern is drawn on the calibration object based on the target calibration result; and determining the calibration error of the target calibration result based on the first and second images. This method can achieve an objective and standard optical perspective calibration verification process, thereby eliminating the reliance on human intervention and improving the efficiency of calibration verification.

[0041] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 This is a hardware structure block diagram of a terminal of a calibration detection method for a head-mounted display device according to an embodiment of the present application;

[0044] Figure 2 is a flow chart of a calibration and detection method for a head-mounted display device according to an embodiment of the present application;

[0045] Figure 3 This is a schematic structural diagram of a calibration device according to an embodiment of the present application;

[0046] Figure 4 is a flow chart of a calibration and detection method for a head-mounted display device according to some embodiments of the present application;

[0047] Figure 5 This is a structural block diagram of a server in an embodiment of the present application;

[0048] Figure 6 Schematic diagram of the structure of the calibration detection system of the head-mounted display device according to the embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0050] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0051] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a block diagram of the hardware structure of the terminal of the calibration detection method for the head mounted display device of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 The processor 102 (only one is shown) and a memory 104 for storing data, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal may also include a transmission device 106 for communication functions and an input / output device 108. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0052] Memory 104 can be used to store computer programs, such as software programs and modules for application software, such as the computer program corresponding to the calibration and detection method for the head-mounted display device in this embodiment. Processor 102 executes the computer programs stored in memory 104 to perform various functional applications and data processing, thereby implementing the aforementioned method. Memory 104 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 104 may further include memory located remotely from processor 102, which can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0053] Transmission device 106 is used to receive or transmit data via a network. This network may include a wireless network provided by the terminal's communications provider. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0054] In this embodiment, a calibration and detection method for a head-mounted display device is provided. The calibration device includes an observation camera and a fixing bracket. The fixing bracket is used to fix the observation camera and the head-mounted display device. The head-mounted display device is fixed between the observation camera lens and a preset calibration object, so that the observation camera can photograph the preset calibration object through the glasses lens of the head-mounted display device. Figure 2 FIG. 1 is a flow chart of a calibration detection method for a head mounted display device according to this embodiment. Figure 2 As shown, the process includes the following steps:

[0055] Step S210 , obtaining a plurality of sets of calibration object images obtained by synchronously photographing the calibration object with an observation camera and a tracking camera; wherein the tracking camera is a camera provided on a head-mounted display device.

[0056] The head-mounted display (HMD) is a smart wearable device based on augmented reality (Optical See-Through, or OST) technology, which overlays virtual information onto the user's real field of view through transparent optical elements. For example, the HMD can be AR glasses or MR glasses.

[0057] To achieve accurate and natural interaction with a head-mounted display (HMD), optical perspective calibration is required. This optical perspective calibration specifically calibrates the spatial relationship between the virtual world presented on the HMD's display and the actual physical world. In other words, through a combined virtual-realistic calibration, the spatial alignment of the virtual world and the real physical world (hereafter referred to as the real world) is achieved.

[0058] The coordinates of the virtual world and the coordinates of the real world can have the following relationship:

[0059] ;

[0060] Among them, T RW is the conversion relationship between the coordinates of the virtual world and the coordinates of the real world; P W represents the three-dimensional real-world coordinates, p R Represents the two-dimensional virtual world coordinates, K R This parameter represents the internal parameters of the virtual world imaging camera, which can be inferred from the factory field of view (FOV) specifications of the head-mounted display's optical components. The virtual world imaging camera is a hypothetical camera that doesn't exist in the physical world and is used to draw virtual objects on the head-mounted display.

[0061] Among them, in this step, in order to achieve T RW To solve the problem, a calibration device consisting of an observation camera and a fixed bracket is introduced. Figure 3 FIG. 1 is a schematic diagram of the structure of a calibration device according to this embodiment. Figure 3 As shown, the calibration device includes an observation camera 31 and a fixing bracket 32. The fixing bracket 32 ​​can fix and support the observation camera 31 and the head-mounted display device 33. When the calibration device is used for joint calibration, the head-mounted display device 33 is fixedly supported between the lens of the observation camera 31 and the calibration object, so that the observation camera can simulate the perspective of the human eye using the head-mounted display device and photograph the calibration object through the glasses lens of the head-mounted display device 33; at the same time, the tracking camera ( Figure 3 (not shown) and simultaneously photographing the calibration object. This step utilizes the calibration device to position the observation camera, head-mounted display, and calibration object, taking into account the effect of the head-mounted display's eyeglass lenses on imaging. The support height of the observation camera and head-mounted display under the fixed bracket, as well as the distance between the observation camera and head-mounted display, can be adjusted by extending or retracting the fixed bracket.

[0062] Before completing the optical perspective calibration of the head-mounted display device, the observation camera and tracking camera must first be jointly calibrated to obtain the intrinsic parameters of the observation camera and the extrinsic parameters between the observation camera and tracking camera. In addition, the intrinsic parameters of the tracking camera can also be solved.

[0063] The observation camera may include a left-eye camera (hereinafter referred to as the left-eye observation camera) and a right-eye camera (hereinafter referred to as the right-eye observation camera). During joint calibration, the left-eye observation camera observes the calibration plate through the left lens of the head-mounted display device and captures the corresponding calibration object image; the right-eye observation camera observes the calibration plate through the right lens of the head-mounted display device and captures the corresponding calibration object image. A tracking camera may be provided on the head-mounted display device glasses. The tracking camera may include multiple tracking cameras, which may be provided on the side of the head-mounted display device glasses, in the middle of the head-mounted display device glasses, or in other locations, which are not limited in this embodiment. For example, a tracking camera may be provided on the left side of the left lens and on the right side of the right lens. The tracking camera located on the left side of the left lens serves as the left-eye tracking camera, and the tracking camera located on the right side of the right lens serves as the right-eye tracking camera.

[0064] The real-world 3D coordinate system can be represented based on the 3D coordinate system corresponding to a tracking camera on a head-mounted display device. For example, a 3D coordinate system can be established using the left tracking camera of the head-mounted display device to represent the real-world coordinate system. Then, during joint calibration, the extrinsic parameters between the left observation camera and the left tracking camera, the intrinsic parameters of the left observation camera, the extrinsic parameters between the right observation camera and the left tracking camera, and the intrinsic parameters of the right observation camera can be solved separately.

[0065] Specifically, during the joint calibration process, based on a pre-set automatic data acquisition program, the observation camera can be controlled to move synchronously with the tracking camera to different locations in a fixed manner relative to the head-mounted display device in the form of program instructions to synchronously capture the calibration object fixed in the scene. For example, the left-eye observation camera, the right-eye observation camera, the left-eye tracking camera, and the right-eye tracking camera can be triggered simultaneously to simultaneously capture the calibration object. The above-mentioned calibration device can be used to fix the observation camera and the head-mounted display device, and the observation camera and the head-mounted display device can be controlled to move synchronously and relatively statically to different pre-set locations to capture the calibration object.

[0066] It can be understood that the observation camera includes two left and right cameras, and the tracking camera includes two left and right cameras. Therefore, in a set of data collected at one time, there are four images of the calibration object taken by different cameras at the same time stamp. By moving to different points for synchronous shooting, multiple sets of calibration object images can be obtained. The multiple sets of calibration object images are then used to jointly calibrate the observation camera and the tracking camera. Since the observation camera shoots the calibration object through the glasses lens of the head-mounted display device, the internal parameters of the observation camera that are finally calibrated actually include the influence of the glasses lens of the head-mounted display device on the imaging.

[0067] Among them, the internal parameters of the observation camera may include parameters such as the focal length, principal point, and distortion coefficient of the observation camera; the external parameters between the observation camera and the tracking camera may include the relative position and posture relationship between the observation camera and the tracking camera.

[0068] Step S220 , performing joint calibration on the observation camera and the tracking camera according to the multiple sets of calibration object images to obtain a joint calibration result.

[0069] After obtaining multiple sets of calibration object images, the calibration features can be identified by performing image processing on the multiple sets of calibration object images. Based on the image coordinates of the identified calibration features in the corresponding calibration object images, the observation camera and tracking camera can be jointly calibrated. The calibration object can be a calibration plate, a calibration target AprilGrid, or other device that can be used for camera calibration. The image set on the calibration object can be a checkerboard pattern, dots, or a user-defined image suitable for camera calibration. The following description uses a calibration plate with a checkerboard pattern as an example.

[0070] The observation camera and tracking camera, supported by a fixed bracket, simultaneously capture the checkerboard grid on the calibration board at different locations, generating multiple sets of calibration object images. These multiple sets of calibration object images are processed to extract the checkerboard grid corners, and the corresponding corner coordinates of the checkerboard grid corners in each calibration object image are determined. Based on the coordinates of these corner points, the observation camera and tracking camera are jointly calibrated to obtain a joint calibration result. This joint calibration result includes the extrinsic parameters between the observation and tracking cameras, as well as the intrinsic parameters of the observation camera and the tracking camera.

[0071] Step S230: Based on the joint calibration result, perform optical perspective calibration on the head mounted display device to obtain a target calibration result.

[0072] Specifically, the target calibration result for the optical perspective calibration of the head-mounted display can be determined based on the aforementioned joint calibration results, for example, based on the intrinsic parameters of the observation camera, the extrinsic parameters between the observation camera and the tracking camera, the virtual image of the calibration object drawn on the display screen by the head-mounted display, and the captured image of the virtual image drawn on the display screen by the observation camera. That is, the spatial relationship between the virtual world and the real world can be determined. This target calibration result can include the transformation relationship between the observation camera coordinate system and the real-world coordinate system, and the transformation relationship between the observation camera coordinate system and the virtual-world coordinate system. Furthermore, the transformation relationship between the virtual-world coordinate system and the real-world coordinate system can be determined, thereby completing the optical perspective calibration.

[0073] Step S240: Acquire a first image obtained by photographing the calibration object through the eyeglass lens with an observation camera.

[0074] After completing optical perspective calibration, the target calibration results can be automatically verified. Specifically, a two-dimensional image captured by the observation camera through the eyeglass lens of the calibration object can be obtained as the first image, which serves as a reference for subsequent determination of the calibration error. The left-eye observation camera captures the calibration object through the left eye lens to obtain the left-eye first image L1; the right-eye observation camera captures the calibration object through the right eye lens to obtain the right-eye first image R1.

[0075] Step S250 , obtaining a second image obtained by photographing a virtual pattern drawn on the display screen of the head-mounted display device with an observation camera; wherein the virtual pattern is obtained by drawing the calibration object based on the target calibration result.

[0076] Based on the target calibration results, the head-mounted display device projects the calibration object onto the head-mounted display device's display screen. The real-world three-dimensional coordinates of the calibration features on the calibration object are known. Based on the target calibration results obtained in step S230, the head-mounted display device converts the known real-world three-dimensional coordinates of the calibration features into the display screen coordinate system for rendering, thereby forming a virtual pattern. The observation camera captures the virtual pattern on the display screen to obtain a second image. This second image may include a left-eye second image L2 captured by the left-eye observation camera and a right-eye second image R2 captured by the right-eye observation camera.

[0077] It is understandable that there is no fixed order for executing step S240 and step S250 . Step S240 and step S250 can be executed simultaneously, or step S250 can be executed before or after step S240 .

[0078] Step S260: determining a calibration error of the target calibration result according to the first image and the second image.

[0079] Specifically, the calibration error can be determined based on the pixel coordinates corresponding to the same feature of the actual calibration object in the first and second images. Once the calibration error is determined, it can be used to objectively assess whether the current target calibration result meets expectations, thereby achieving objective and accurate verification of the optical see-through calibration of the head-mounted display device.

[0080] If it is determined that the current target calibration result does not meet expectations, the observation camera and tracking camera can be re-synchronized to capture multiple sets of calibration object images. Joint calibration can then be re-performed based on the newly obtained calibration object images to obtain a new joint calibration result. The head-mounted display device is then re-optically calibrated based on the new joint calibration result to obtain a new target calibration result. The calibration error is then recalculated according to steps S240, S250, and S260 until the current target calibration result is determined to meet expectations based on the latest calibration error.

[0081] In related technologies, after completing optical see-through calibration, manual intervention is required to experience the calibration effect. Therefore, in the prior art, there is no standard, objective process for verifying optical see-through calibration. Instead, it relies on manual intervention and subjective experience, which is inefficient and unsuitable for the mass production environment of head-mounted display devices. This embodiment can implement an automatic, objective, and accurate optical see-through calibration verification solution based on the above-mentioned calibration device, and can accurately and quickly calculate the calibration error of the target calibration result, using this as an objective and quantitative calibration verification indicator. This can then guide subsequent processing procedures (recalibration or confirmation of calibration completion) based on this calibration verification indicator, thereby eliminating dependence on manual labor, improving the efficiency of calibration verification, and being suitable for the mass production environment of head-mounted display devices.

[0082] Furthermore, since this embodiment uses the observation camera to capture the calibration object through the head-mounted display's eyeglass lenses during combined calibration, the calibration data for this combined calibration is collected. Therefore, the combined calibration performed in this embodiment takes into account the effect of the eyeglass lenses on imaging. The target calibration results obtained from subsequent optical perspective calibration based on this combined calibration also improve the accuracy of optical perspective calibration compared to related technologies.

[0083] Through steps S210 to S260, multiple sets of calibration object images are obtained by synchronously photographing the calibration object with the observation camera and the tracking camera. The tracking camera is a camera installed on the head-mounted display device. Based on the multiple sets of calibration object images, the observation camera and the tracking camera are jointly calibrated to obtain a joint calibration result. Based on the joint calibration result, the head-mounted display device is optically perspective calibrated to obtain a target calibration result. A first image is obtained by photographing the calibration object through the eyeglass lens with the observation camera. A second image is obtained by photographing a virtual pattern drawn on the display screen of the head-mounted display device with the observation camera. The virtual pattern is drawn on the calibration object based on the target calibration result. The calibration error of the target calibration result is determined based on the first and second images. This method can achieve an objective and standard optical perspective calibration verification process, thereby eliminating the reliance on human intervention and improving the efficiency of calibration verification.

[0084] In one embodiment, after determining the calibration error of the target calibration result, the calibration detection method may further include:

[0085] When the calibration error is greater than the preset error threshold, it is determined that the current target calibration result does not meet the calibration qualification condition; when the calibration error is less than or equal to the error threshold, it is determined that the current target calibration result meets the calibration qualification condition.

[0086] That is, after obtaining the calibration error, the evaluation can be completed based on the following logic (ideally, the calibration error can be 0):

[0087] An error threshold can be set in advance. The calculated calibration error is compared with the error threshold. When the calibration error is greater than the error threshold, it indicates that the current optical perspective calibration of the head-mounted display device is unqualified and does not meet expectations. For example, according to the current target calibration result, the image observed by the user on the display screen when using the head-mounted display device may be blurry, so recalibration is required. When the calibration error is less than or equal to the error threshold, it indicates that the current target calibration result is in line with expectations. Drawing a pattern on the display screen based on the current target calibration result can enable the user to observe a clearer image when wearing the head-mounted display device, so the calibration can be ended.

[0088] The error threshold can be obtained based on measured data. For example, noise is added to the calibration result of the ideal optical perspective. The noise level exceeds the threshold value that blurs the image when the head-mounted display is worn, and the corresponding error threshold is determined based on the noise level.

[0089] Therefore, this embodiment uses the calibration error as an objective and quantitative verification indicator to achieve standard and accurate verification of whether the optical perspective calibration is qualified, thereby improving the verification efficiency of the optical perspective calibration.

[0090] In one embodiment, when it is determined that the current target calibration result does not meet the calibration qualification condition, the calibration detection method may further include:

[0091] Repeat the following process until the calibration error of the current target calibration result meets the calibration qualification conditions:

[0092] Based on multiple sets of new calibration object images re-collected by the observation camera and tracking camera, the observation camera and tracking camera are re-jointly calibrated, and the head-mounted display device is re-optically perspective-calibrated based on the new joint calibration results. The calibration error of the target calibration result is recalculated according to the new target calibration result.

[0093] In short, if the calibration error of the current target calibration result is greater than a preset error threshold, steps S210 to S260 of the above embodiment are re-executed until the calibration error of the latest target calibration result is less than or equal to the error threshold. In this way, the final optical see-through calibration result of the head-mounted display device can meet expectations.

[0094] This embodiment implements a standard, automatic, closed-loop calibration verification that does not require human intervention, thereby ensuring the calibration quality of the head-mounted display device.

[0095] Additionally, in one embodiment, according to step S260 above, determining a calibration error of the target calibration result according to the first image and the second image may include:

[0096] Determine the associated feature pairs corresponding to the first image and the second image; the associated feature pairs are feature pairs in the first image and the second image that are associated with the same calibration feature on the calibration object; and determine the calibration error of the target calibration result based on the pixel distance between the associated feature pairs.

[0097] This embodiment takes into account that for a head-mounted display device, if the result of the optical perspective calibration is accurate, then the first image obtained by the observation camera through the eyeglass lens to capture the calibration object and the second image captured by the observation camera from the display screen, in which the presentation position of the same calibration feature of the actual calibration object should overlap. In addition, since the coordinate unit in a two-dimensional image is pixel, this embodiment uses the pixel distance between the feature pairs associated with the same calibration feature on the calibration object in the first and second images as the standard metric for judging the optical perspective calibration result. Furthermore, based on this pixel distance, the calibration error of the target calibration result is determined.

[0098] The calibration object may include multiple calibration features, and multiple associated feature pairs can be determined from the first image and the second image, with different associated feature pairs associated with different calibration features on the calibration object. For example, a checkerboard calibration plate includes multiple checkerboard corner points, such as corner point A, corner point B, corner point C, etc. For corner point A, a set of associated feature pairs 1 can be determined from the first image and the second image. For corner point B, a set of associated feature pairs 2 can be determined from the first image and the second image. For corner point C, a set of associated feature pairs 3 can be determined from the first image and the second image, and so on. Subsequently, the pixel distance between each associated feature pair is calculated to determine the calibration error of the target calibration result.

[0099] Specifically, the associated feature pairs can be feature points corresponding to identical key points (corner points) on the calibration object in the first and second images, respectively. For example, for a checkerboard calibration plate, corner point detection can be performed on the first and second images using a corner point detection algorithm to determine the coordinates of each corner point in the first and second images. Furthermore, since each corner point is pre-assigned with a unique identifier (ID) or a specific set order, feature points corresponding to identical key points in the first and second images can be determined based on the corner point ID or the order in which the corner points are arranged in the image, thereby forming associated feature pairs.

[0100] This embodiment calculates the calibration error of the target calibration result based on the pixel distance of the same calibration feature in the first image and the second image, thereby achieving accurate verification of the target calibration result.

[0101] Specifically, in one embodiment, determining the calibration error of the target calibration result based on the pixel distance between the associated feature pairs may include:

[0102] Calculate the pixel distance between each set of associated feature pairs; perform statistics on the pixel distances between all associated feature pairs to obtain the calibration error of the target calibration result.

[0103] First, the pixel distance between each set of associated feature pairs can be calculated , which is represented by the pixel distance between the feature pairs corresponding to the same calibration feature i, the left-eye first image L1 and the left-eye second image L2 captured by the left-eye observation camera.

[0104] After that, the pixel distances corresponding to all the associated feature pairs are counted. For example, all the obtained pixel distances are averaged to obtain the calibration error:

[0105] ;

[0106] Where e is the calibration error and N is the number of calibration features.

[0107] This embodiment can obtain accurate calibration errors by statistically analyzing the pixel distances corresponding to all associated feature pairs, thereby achieving accurate verification of optical perspective calibration.

[0108] Additionally, in one embodiment, acquiring multiple sets of calibration object images obtained by synchronously photographing the calibration object with the observation camera and the tracking camera may specifically include:

[0109] The observation camera and the tracking camera are moved to different points in sequence, and the calibration object is photographed synchronously to obtain the calibration object image; one set of calibration object images corresponds to one point.

[0110] The calibration device is left stationary at a preset position. The positions of the observation camera and head-mounted display are adjusted, and the calibration object is positioned so that the observation camera can fully observe all calibration features on the object. The observation camera and tracking camera then capture the object at the same timestamp, generating a set of calibration object images. The observation camera and tracking camera are then moved synchronously to the next position. During this movement, the position of the two cameras remains unchanged, as both are secured by a fixed bracket and the tracking camera is mounted on the head-mounted display. At the next position, the observation camera and tracking camera simultaneously capture another set of calibration object images. Multiple sets of calibration object images can be obtained in this manner.

[0111] This embodiment can be based on a pre-set program and combined with the above-mentioned calibration device to complete the acquisition of multiple sets of calibration object images for joint calibration, thereby improving the convenience and efficiency of data acquisition. Joint calibration based on multiple sets of calibration object images can improve the accuracy of joint calibration.

[0112] Additionally, in one embodiment, performing optical perspective calibration on the head-mounted display device based on the joint calibration result to obtain a target calibration result may include:

[0113] A virtual pattern for the calibration object is drawn on the display screen of the head-mounted display device; based on the virtual pattern, the three-dimensional feature coordinates of the calibration object are determined, and a target image obtained by photographing the virtual pattern with an observation camera is obtained; and based on the three-dimensional feature coordinates, the target image, and the joint calibration result, an optical perspective calibration result for the head-mounted display device is determined.

[0114] The joint calibration results can include the intrinsic parameters of the observation camera and the extrinsic parameters between the observation camera and the tracking camera. Based on the extrinsic parameters between the observation camera and the tracking camera, the transformation relationship between the observation camera coordinate system and the real-world coordinate system can be determined.

[0115] The three-dimensional coordinate system R can be established on the plane where the display screen is located. Then, the calibration features of the virtual pattern on the display screen are obtained, and the three-dimensional coordinates under this three-dimensional coordinate line R are obtained, thereby obtaining the three-dimensional feature coordinates of the calibration object {P Ri}. i can represent the i-th calibration feature, such as the i-th checkerboard corner point. In addition, the virtual pattern displayed on the display screen can be photographed by an observation camera to obtain a two-dimensional image containing the virtual pattern as the target image. The two-dimensional image coordinates of the calibration feature in the image coordinate system of the target image can be extracted from the target image {p ci}, then, with the three-dimensional feature coordinates {P Ri}、2D image coordinates {p ci The system uses the observation camera's intrinsic parameters as input to solve the perspective-n-point (PnP) problem, ultimately determining the transformation between the observation camera coordinate system and the virtual world coordinate system. Then, based on the transformation between the observation camera coordinate system and the virtual world coordinate system, and the transformation between the real-world coordinate system and the observation camera coordinate system, the transformation between the real-world coordinate system and the virtual world coordinate system is determined, completing the optical perspective calibration for the head-mounted display.

[0116] This embodiment introduces an observation camera to split the virtual-reality combined calibration into the conversion relationship between the observation camera coordinate system and the virtual world coordinate system, and the conversion relationship between the observation camera coordinate system and the real-world coordinate system; based on the joint calibration results that take into account the influence of the glasses lenses of the head-mounted display device on the imaging, optical perspective calibration is finally achieved, thereby improving the accuracy of optical perspective calibration.

[0117] Figure 4 is a flow chart of a calibration detection method for a head mounted display device in some embodiments, such as Figure 4 As shown, the calibration detection method includes the following steps:

[0118] Step S401: Place the calibration device at rest, adjust the positions of the observation camera and the head-mounted display device at preset positions, set the position of the calibration object, and start the data acquisition program;

[0119] Step S402 , based on the data acquisition program, obtaining multiple sets of calibration object images obtained by synchronously photographing the calibration object at different positions using the observation camera and the tracking camera;

[0120] Step S403: performing joint calibration on the observation camera and the tracking camera based on the multiple sets of calibration object images to obtain a joint calibration result;

[0121] Step S404: Based on the joint calibration result, perform optical perspective calibration on the head mounted display device to obtain a target calibration result;

[0122] Step S405: Use an observation camera to photograph the calibration object through the glasses lens of the head-mounted display device to obtain a first image; wherein the first image includes a left-eye first image captured by the left-eye observation camera and a right-eye first image captured by the right-eye observation camera;

[0123] Step S406: drawing a virtual pattern of the calibration object on the display screen of the head mounted display device according to the target calibration result;

[0124] Step S407: photographing the virtual pattern on the display screen using an observation camera to obtain a second image; wherein the second image includes a left-eye second image captured by the left-eye observation camera and a right-eye second image captured by the right-eye observation camera;

[0125] Step S408: perform key point detection on the first image and the second image to determine the pixel coordinates and IDs of the calibration features of the calibration object on the first image and the second image respectively;

[0126] Step S409, determining the pixel distance of the associated feature pair based on the result of step S408;

[0127] Step S410, calculating the calibration error of the target calibration result according to the pixel distance;

[0128] In step S411 , it is determined whether the calibration error is less than or equal to a preset error threshold; if so, the process ends; otherwise, the process returns to step S401 .

[0129] It should be noted that the steps shown in the above process or the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in a different order than that shown, for example, step S405 and step S406.

[0130] The above steps S401 to S411 can realize an objective, standard and closed-loop optical perspective calibration verification process, thereby eliminating the dependence on manual participation and improving the efficiency of calibration verification.

[0131] Figure 5 is a structural block diagram of the server 50 of this embodiment, which is used to perform calibration detection on the head-mounted display device based on the calibration device; the calibration device includes an observation camera and a fixing bracket, wherein the fixing bracket is used to fix the observation camera and the head-mounted display device, and the head-mounted display device is fixed between the observation camera lens and the preset calibration object, so that the observation camera can shoot the preset calibration object through the glasses lens of the head-mounted display device; Figure 5As shown, the server 50 includes: an acquisition module 51, a joint calibration module 52, an optical perspective calibration module 53 and a calibration detection module 54; wherein:

[0132] An acquisition module 51 is used to acquire multiple sets of calibration object images obtained by synchronously photographing the calibration object with the observation camera and the tracking camera; wherein the tracking camera is a camera provided on the head-mounted display device; a joint calibration module 52 is used to jointly calibrate the observation camera and the tracking camera based on the multiple sets of calibration object images to obtain a joint calibration result; an optical perspective calibration module 53 is used to perform optical perspective calibration on the head-mounted display device based on the joint calibration result to obtain a target calibration result; a calibration detection module 54 is used to acquire a first image obtained by photographing the calibration object with the observation camera through the glasses lens; and acquire a second image obtained by photographing a virtual pattern drawn on the display screen of the head-mounted display device with the observation camera; wherein the virtual pattern is drawn based on the target calibration result; and determine the calibration error of the target calibration result based on the first image and the second image.

[0133] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0134] In this embodiment, a calibration detection system 60 for a head-mounted display device is provided. Figure 6 Schematic diagram of the structure of a calibration and detection system 60 for a head-mounted display device of this embodiment, the calibration and detection system 60 includes: a calibration device and the server 50 provided in the above embodiment;

[0135] The specific structure of the calibration device can be found in Figure 3 The calibration device may include an observation camera 31 and a fixing bracket 32, wherein the fixing bracket 32 ​​is used to fix the observation camera 31 and the head-mounted display device, and the head-mounted display device is fixed between the lens of the observation camera 31 and the preset calibration object, so that the observation camera 31 can shoot the preset calibration object through the glasses lens of the head-mounted display device; the observation camera 31 is communicatively connected to the server 50.

[0136] It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0137] In addition, in conjunction with the calibration and detection methods for head-mounted display devices provided in the above embodiments, this embodiment may also provide a storage medium for implementation. The storage medium stores a computer program; when executed by a processor, the computer program implements any of the calibration and detection methods for head-mounted display devices provided in the above embodiments.

[0138] It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0139] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0140] Obviously, the accompanying drawings are merely examples or embodiments of the present application. A person skilled in the art can also apply the present application to other similar situations based on these drawings without inventive effort. Furthermore, it is understandable that, although the work involved in this development process may be complex and lengthy, certain design, manufacturing, or production changes based on the technical content disclosed in this application are merely routine technical means for a person skilled in the art and should not be considered to constitute a deficiency in the disclosure of the present application.

[0141] The term "embodiment" as used in this application refers to specific features, structures, or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily mean that the embodiment is the same, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is understood, either explicitly or implicitly, by those skilled in the art that the embodiments described in this application can be combined with other embodiments when there is no conflict.

[0142] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A calibration and detection method for a head-mounted display device, characterized in that: For a calibration device, the calibration device includes an observation camera and a fixing bracket, wherein the fixing bracket is used to fix the observation camera and the head-mounted display device, and the head-mounted display device is fixed between the observation camera lens and a preset calibration object, so that the observation camera can photograph the preset calibration object through the glasses lens of the head-mounted display device; the method includes: Acquire multiple sets of calibration object images obtained by synchronously photographing the calibration object with the observation camera and the tracking camera; wherein the tracking camera is a camera provided on the head-mounted display device; performing joint calibration on the observation camera and the tracking camera according to the multiple sets of calibration object images to obtain a joint calibration result; Based on the joint calibration result, optical perspective calibration is performed on the head mounted display device to obtain a target calibration result, including: Drawing a virtual pattern corresponding to the calibration object on a display screen of the head-mounted display device; determining three-dimensional feature coordinates of the calibration object based on the virtual pattern, and obtaining a target image obtained by photographing the virtual pattern with the observation camera; and determining an optical perspective calibration result for the head-mounted display device by solving a perspective n-point method based on the three-dimensional feature coordinates, the two-dimensional image coordinates of the calibration feature in the target image, and the joint calibration result; Acquire a first image obtained by photographing the calibration object through the glasses lens by the observation camera; Acquire a second image obtained by photographing a virtual pattern drawn on the display screen of the head-mounted display device with the observation camera; wherein the virtual pattern is obtained by drawing the calibration object based on the target calibration result; determining a calibration error of the target calibration result according to the first image and the second image; When the calibration error is greater than a preset error threshold, determining that the current target calibration result does not meet the calibration qualification condition; When it is determined that the current target calibration result does not meet the calibration qualification condition, the following process is repeatedly performed until the calibration error of the current target calibration result meets the calibration qualification condition: Based on multiple sets of new calibration object images resynchronizedly captured by the observation camera and the tracking camera, the observation camera and the tracking camera are re-jointly calibrated, and based on the new joint calibration result, the head-mounted display device is re-optically perspective-calibrated, and the calibration error of the target calibration result is recalculated according to the new target calibration result.

2. The calibration detection method according to claim 1, characterized in that: The method further comprises: When the calibration error is less than or equal to the error threshold, it is determined that the current target calibration result meets the calibration qualification condition.

3. The calibration detection method according to claim 1, characterized in that: Determining a calibration error of the target calibration result according to the first image and the second image includes: Determining an associated feature pair corresponding to the first image and the second image; the associated feature pair is a feature pair in the first image and the second image that is associated with the same calibration feature on the calibration object; A calibration error of the target calibration result is determined according to the pixel distance between the associated feature pairs.

4. The calibration detection method according to claim 3, characterized in that: Determining a calibration error of the target calibration result according to the pixel distance between the associated feature pairs includes: Calculating the pixel distance between each set of associated feature pairs; The pixel distances between all the associated feature pairs are counted to obtain the calibration error of the target calibration result.

5. The calibration detection method according to claim 1, characterized in that: Acquiring multiple sets of calibration object images obtained by synchronously photographing the calibration object with the observation camera and the tracking camera, including: The observation camera and the tracking camera are moved to different positions in sequence, and the calibration object is photographed synchronously to obtain the calibration object images; wherein a group of the calibration object images corresponds to one position.

6. A server, characterized in that: Used to perform calibration testing on a head-mounted display device based on a calibration device; the calibration device includes an observation camera and a fixing bracket, wherein the fixing bracket is used to fix the observation camera and the head-mounted display device, and the head-mounted display device is fixed between the observation camera lens and a preset calibration object, so that the observation camera can photograph the preset calibration object through the glasses lens of the head-mounted display device; The server includes: an acquisition module, a joint calibration module, an optical perspective calibration module and a calibration detection module; wherein: The acquisition module is configured to acquire a plurality of sets of calibration object images obtained by synchronously photographing the calibration object with the observation camera and the tracking camera; wherein the tracking camera is a camera provided on the head-mounted display device; The joint calibration module is configured to perform joint calibration on the observation camera and the tracking camera according to the multiple sets of calibration object images to obtain a joint calibration result; The optical perspective calibration module is configured to perform optical perspective calibration on the head mounted display device based on the joint calibration result to obtain a target calibration result; and includes: Drawing a virtual pattern corresponding to the calibration object on a display screen of the head-mounted display device; determining three-dimensional feature coordinates of the calibration object based on the virtual pattern, and obtaining a target image obtained by photographing the virtual pattern with the observation camera; and determining an optical perspective calibration result for the head-mounted display device by solving a perspective n-point method based on the three-dimensional feature coordinates, the two-dimensional image coordinates of the calibration feature in the target image, and the joint calibration result; The calibration detection module is configured to obtain a first image obtained by photographing the calibration object through the glasses lens by the observation camera; obtain a second image obtained by photographing a virtual pattern drawn on the display screen of the head-mounted display device by the observation camera; wherein the virtual pattern is obtained by drawing the calibration object based on the target calibration result; determine a calibration error of the target calibration result based on the first image and the second image; when the calibration error is greater than a preset error threshold, determine that the current target calibration result does not meet the calibration qualification condition; when it is determined that the current target calibration result does not meet the calibration qualification condition, repeatedly perform the following process until the calibration error of the current target calibration result meets the calibration qualification condition: based on multiple sets of new calibration object images resynchronizedly acquired by the observation camera and the tracking camera, re-calibrate the observation camera and the tracking camera jointly, re-perform optical perspective calibration on the head-mounted display device based on the new joint calibration result, and recalculate the calibration error of the target calibration result based on the new target calibration result.

7. A calibration and detection system for a head-mounted display device, characterized in that: include: A calibration device and a server according to claim 6; The calibration device includes an observation camera and a fixing bracket, wherein the fixing bracket is used to fix the observation camera and the head-mounted display device, and the head-mounted display device is fixed between the observation camera lens and the preset calibration object, so that the observation camera can photograph the preset calibration object through the glasses lens of the head-mounted display device; The observation camera is communicatively connected to the server.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the calibration and detection method for a head-mounted display device according to any one of claims 1 to 5 are implemented.

Images