Calibration detection method and system for head-mounted display equipment

Abstract

The invention relates to a calibration detection method and system for head-mounted display equipment, and the method comprises the steps: obtaining a plurality of groups of calibration object images obtained through the synchronous shooting of a calibration object by an observation camera and a tracking camera; performing joint calibration on the observation camera and the tracking camera according to the plurality of groups of calibration object images to obtain a joint calibration result; based on the joint calibration result, performing optical perspective calibration on the head-mounted display device to obtain a target calibration result; obtaining a first image obtained by shooting the calibration object through the glasses lens by the observation camera; obtaining a second image obtained by shooting a virtual pattern drawn on the display screen of the head-mounted display device by the observation camera; and determining a calibration error of the target calibration result according to the first image and the second image. According to the invention, the objective and standard verification process of optical perspective calibration can be realized, so that the dependence on manual participation is eliminated, and the calibration verification efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of smart wearables, and in particular, to a calibration detection method and system for a head-mounted display device. Background Art

[0002] An augmented reality (AR) glasses or a mixed reality (MR) glasses can serve as a core device for human-computer interaction and play an important role in fields such as industrial manufacturing, medical treatment, social entertainment, and education. Calibrating the combination of virtual and real of the AR / MR glasses is to solve the problem of aligning the spatial relationship between the virtual world and the real world presented on the AR / MR glasses, so as to ensure the immersion and interaction accuracy of the user experience.

[0003] Currently, after the combination of virtual and real of the AR / MR glasses is calibrated, it is often manually worn, and subjectively determines whether the calibration is successful based on the wearing experience. This method relies on manual intervention and subjective experience, and has low efficiency.

[0004] In view of the problem of low efficiency in calibration verification in the related art, no effective solution has been proposed yet. Summary of the Invention

[0005] In this embodiment, a calibration detection method and system for a head-mounted display device are provided to solve the problem of low efficiency in calibration verification in the related art.

[0006] In a first aspect, in this embodiment, a calibration detection method for a head-mounted display device is provided for a calibration device. The calibration device includes an observation camera and a fixing bracket. The fixing bracket is used to fix the observation camera and the head-mounted display device. The head-mounted display device is fixed between the lens of the observation camera and a preset calibration object, so that the observation camera takes pictures of the preset calibration object through the glasses lens of the head-mounted display device. The method includes:

[0007] Obtaining multiple groups of calibration object images obtained by synchronously photographing the calibration object by the observation camera and a tracking camera, where the tracking camera is a camera disposed on the head-mounted display device;

[0008] Performing joint calibration on the observation camera and the tracking camera according to the multiple groups of calibration object images to obtain a joint calibration result;

[0009] Performing optical perspective calibration on the head-mounted display device based on the joint calibration result to obtain a target calibration result;

[0010] Obtaining a first image obtained by the observation camera taking pictures of the calibration object through the glasses lens;

[0011] Obtain a second image captured by the observation camera of a virtual pattern drawn on the display screen of the head-mounted display device; wherein, the virtual pattern is obtained by drawing for the calibration object based on the target calibration result;

[0012] Determine the calibration error of the target calibration result according to the first image and the second image.

[0013] In some embodiments, after determining the calibration error of the target calibration result, the method further includes:

[0014] When the calibration error is greater than a preset error threshold, determine that the current target calibration result does not meet the calibration qualification condition;

[0015] When the calibration error is less than or equal to the error threshold, determine that the current target calibration result meets the calibration qualification condition.

[0016] In some embodiments, when it is determined that the current target calibration result does not meet the calibration qualification condition, the method further includes:

[0017] Repeat the following process until the calibration error of the current target calibration result meets the calibration qualification condition:

[0018] Based on multiple groups of new calibration object images resynchronously collected by the observation camera and the tracking camera, re-perform joint calibration on the observation camera and the tracking camera, re-perform optical perspective calibration on the head-mounted display device based on the new joint calibration result, and recalculate the calibration error of the target calibration result according to the new target calibration result.

[0019] In some embodiments, determining the calibration error of the target calibration result according to the first image and the second image includes:

[0020] Determine the associated feature pairs corresponding to the first image and the second image; the associated feature pairs are feature pairs in the first image and the second image that are associated with the same calibration feature on the calibration object;

[0021] Determine the calibration error of the target calibration result according to the pixel distance between the associated feature pairs.

[0022] In some embodiments, determining the calibration error of the target calibration result according to the pixel distance between the associated feature pairs includes:

[0023] Calculate the pixel distance between each group of the associated feature pairs;

[0024] Statistically analyze the pixel distances between all the pairs of associated features to obtain the calibration error of the target calibration result.

[0025] In some embodiments, obtaining a plurality of sets of calibration object images obtained by synchronously photographing the calibration object by the observation camera and the tracking camera includes:

[0026] Obtaining the calibration object images obtained by synchronously photographing the calibration object when the observation camera and the tracking camera are sequentially moved to different positions; one set of the calibration object images corresponds to one of the positions.

[0027] In some embodiments, based on the joint calibration result, performing optical perspective calibration on the head-mounted display device to obtain a target calibration result, including:

[0028] Drawing a virtual pattern for the calibration object on the display screen of the head-mounted display device;

[0029] Based on the virtual pattern, determining the three-dimensional feature coordinates of the calibration object, and obtaining a target image obtained by photographing the virtual pattern by the observation camera;

[0030] According to the three-dimensional feature coordinates, the target image, and the joint calibration result, determining the optical perspective calibration result for the head-mounted display device.

[0031] In a second aspect, in the present embodiment, a server is provided for performing calibration detection on a head-mounted display device based on a calibration device; the calibration device includes an observation camera and a fixing bracket, wherein the fixing bracket is used to fix the observation camera and the head-mounted display device, and the head-mounted display device is fixed between the lens of the observation camera and a preset calibration object so that the observation camera photographs the preset calibration object through the spectacle lens of the head-mounted display device;

[0032] The server includes: an acquisition module, a joint calibration module, an optical perspective calibration module, and a calibration detection module; wherein:

[0033] The acquisition module is configured to acquire a plurality of sets of calibration object images obtained by synchronously photographing the calibration object by the observation camera and the tracking camera; wherein, the tracking camera is a camera disposed on the head-mounted display device;

[0034] The joint calibration module is configured to perform joint calibration on the observation camera and the tracking camera according to the plurality of sets of calibration object images to obtain a joint calibration result;

[0035] The optical perspective calibration module is configured to perform optical perspective calibration on the head-mounted display device based on the joint calibration result to obtain a target calibration result;

[0036] The calibration detection module is configured to obtain a first image obtained by the observation camera photographing the calibration object through the glasses lens; obtain a second image obtained by the observation camera photographing a virtual pattern drawn on the display screen of the head-mounted display device; wherein, the virtual pattern is drawn based on the target calibration result for the calibration object; and determine the calibration error of the target calibration result according to the first image and the second image.

[0037] In a third aspect, in this embodiment, a calibration detection system for a head-mounted display device is provided, including: a calibration device and the server described in the second aspect above;

[0038] The calibration device includes an observation camera and a fixing bracket. Among them, the fixing bracket is used to fix the observation camera and the head-mounted display device. The head-mounted display device is fixed between the lens of the observation camera and a preset calibration object, so that the observation camera photographs the preset calibration object through the glasses lens of the head-mounted display device; the observation camera is communicatively connected to the server.

[0039] In a fourth aspect, in this embodiment, a storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it implements the calibration detection method for the head-mounted display device described in the first aspect above.

[0040] Compared with the related art, in this embodiment, a calibration detection method and system for a head-mounted display device are provided. The calibration detection method for the head-mounted display device therein obtains multiple groups of calibration object images obtained by synchronously photographing a calibration object by an observation camera and a tracking camera; wherein, the tracking camera is a camera disposed on the head-mounted display device; performs joint calibration on the observation camera and the tracking camera according to the multiple groups of calibration object images to obtain a joint calibration result; performs optical perspective calibration on the head-mounted display device based on the joint calibration result to obtain a target calibration result; obtains a first image obtained by the observation camera photographing the calibration object through the glasses lens; obtains a second image obtained by the observation camera photographing a virtual pattern drawn on the display screen of the head-mounted display device; wherein, the virtual pattern is drawn based on the target calibration result for the calibration object; and determines the calibration error of the target calibration result according to the first image and the second image. It can implement an objective and standard verification process for optical perspective calibration, thereby eliminating the dependence on manual participation and improving the efficiency of calibration verification.

[0041] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more concise and understandable. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0043] Figure 1 is a hardware structure block diagram of a terminal of a calibration detection method for a head-mounted display device according to an embodiment of the present application;

[0044] Figure 2 is a flowchart of a calibration detection method for a head-mounted display device according to an embodiment of the present application;

[0045] Figure 3 is a schematic structural diagram of a calibration device according to an embodiment of the present application;

[0046] Figure 4 is a flowchart of a calibration detection method for a head-mounted display device according to some embodiments of the present application;

[0047] Figure 5 is a block diagram of the structure of a server according to an embodiment of the present application;

[0048] Figure 6 is a schematic structural diagram of a calibration detection system for a head-mounted display device according to an embodiment of the present application. Detailed Embodiments

[0049] To more clearly understand the purpose, technical solution and advantages of the present application, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments.

[0050] Unless otherwise defined, technical terms or scientific terms involved in this application shall have the general meanings understood by those with ordinary skills in the technical field to which this application belongs. In this application, words such as "a", "an", "one kind", "the", "these", etc. do not indicate a limitation in quantity, and they can be singular or plural. The terms "including", "containing", "having" and any variants thereof involved in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or device containing a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent in these processes, methods, products or devices. The terms "connected", "linked", "coupled" and other similar words involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly connected. The "plurality" involved in this application means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone these three situations. Usually, the character " / " indicates that the objects associated before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific sorting of the objects.

[0051] The method embodiment provided in this embodiment can be executed on a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is the hardware structure block diagram of the terminal of the calibration detection method of the head-mounted display device in this embodiment. As Figure 1 shown, the terminal may include one or more ( Figure 1 only one is shown in the figure) processors 102 and a memory 104 for storing data. Among them, the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above terminal. For example, the terminal may further include more or fewer components than Figure 1 shown in the figure, or have a different configuration from Figure 1 shown.

[0052] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the calibration detection method of the head-mounted display device in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above-mentioned method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories can be connected to the terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof.

[0053] The transmission device 106 is used to receive or send data via a network. The above-mentioned network includes the wireless network provided by the communication provider of the terminal. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one instance, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0054] In this embodiment, a calibration detection method for a head-mounted display device is provided, which is used for a calibration device. The calibration device includes an observation camera and a fixing bracket. Among them, the fixing bracket is used to fix the observation camera and the head-mounted display device. The head-mounted display device is fixed between the lens of the observation camera and a preset calibration object, so that the observation camera can take pictures of the preset calibration object through the spectacle lens of the head-mounted display device; Figure 2 is the flowchart of the calibration detection method of the head-mounted display device in this embodiment, as Figure 2 shown, this process includes the following steps:

[0055] Step S210, obtaining multiple groups of calibration object images obtained by the synchronous shooting of the calibration object by the observation camera and the tracking camera; among them, the tracking camera is a camera disposed on the head-mounted display device.

[0056] The head-mounted display device is an intelligent wearable device based on optical see-through (OST) augmented reality display technology, which superimposes virtual information on the user's real field of view through a transparent optical element. Exemplarily, the head-mounted display device can be an AR glasses or an MR glasses.

[0057] In order to achieve accurate and natural interaction using head-mounted display devices, it is necessary to perform optical perspective calibration on the head-mounted display device. The optical perspective calibration specifically calibrates the spatial relationship between the virtual world presented by the display screen of the head-mounted display device and the actual physical world. In other words, it is necessary to achieve spatial alignment between the virtual world and the real physical world (hereinafter referred to as the real world) through a combination of virtual and real calibration.

[0058] The coordinates of the virtual world and the coordinates of the real world can have the following relationship:

[0059] ;

[0060] Among them, T RW is the conversion relationship between the coordinates of the virtual world and the coordinates of the real world; W represents the three-dimensional real-world coordinates, p R Represents the two-dimensional virtual world coordinates, K R Represents the internal parameters of the virtual world imaging camera, which can be inferred from the factory field of view (FOV) parameters of the optical components of the head-mounted display device. The virtual world imaging camera is a camera that does not exist in the physical world and is imagined to be used to draw virtual object content on the display screen of the head-mounted display device.

[0061] Among them, in this step, in order to achieve T RW To solve the problem, a calibration device consisting of an observation camera and a fixed bracket was introduced. Figure 3 FIG. 1 is a schematic diagram of the structure of a calibration device of this embodiment. Figure 3 As shown, the calibration device includes an observation camera 31 and a fixing bracket 32. The fixing bracket 32 can fix and support the observation camera 31 and the head-mounted display device 33. When the calibration device is used for joint calibration, the head-mounted display device 33 is fixedly supported between the lens of the observation camera 31 and the calibration object, so that the observation camera can simulate the perspective of the human eye using the head-mounted display device, and shoot the calibration object through the glasses lens of the head-mounted display device 33; at the same time, the tracking camera ( Figure 3 The calibration object is photographed synchronously with the observation camera (not shown). This step uses the calibration device to set the positions of the observation camera, the head-mounted display device, and the calibration object in consideration of the influence of the glasses lens of the head-mounted display device on the imaging. Among them, the support height of the observation camera and the head-mounted display device under the support of the fixed bracket, as well as the distance between the observation camera and the head-mounted display device, can be adjusted by extending and retracting the fixed bracket.

[0062] Before completing the optical perspective calibration of the head-mounted display device, it is first necessary to jointly calibrate the observation camera and the tracking camera to obtain the internal parameters of the observation camera and the external parameters between the observation camera and the tracking camera. Additionally, the internal parameters of the tracking camera can also be solved for.

[0063] The observation camera can include a left-eye camera (hereinafter referred to as the left-eye observation camera) and a right-eye camera (hereinafter referred to as the right-eye observation camera). When performing the joint calibration, the left-eye observation camera is made to observe the calibration board through the left lens of the head-mounted display device and capture the corresponding calibration object images; the right-eye observation camera is made to observe the calibration board through the right lens of the head-mounted display device and capture the corresponding calibration object images. Tracking cameras can be provided on the glasses of the head-mounted display device. There can be multiple such tracking cameras, and they can be set on the side of the glasses of the head-mounted display device, or in the middle of the lenses of the glasses of the head-mounted display device, or at other positions, which is not limited in this embodiment. For example, one tracking camera is set on the left side of the left lens and one tracking camera is set on the right side of the right lens. The tracking camera on the left side of the left lens is used as the left-eye tracking camera, and the tracking camera on the right side of the right lens is used as the right-eye tracking camera.

[0064] The three-dimensional coordinate system of the real world can be characterized based on the three-dimensional coordinate system corresponding to one tracking camera on the head-mounted display device. For example, the three-dimensional coordinate system of the real world is established using the left-eye tracking camera of the head-mounted display device. Then, when performing the joint calibration, the external parameters between the left-eye observation camera and this left-eye tracking camera, the internal parameters of the left-eye observation camera, the external parameters between the right-eye observation camera and this left-eye tracking camera, and the internal parameters of the right-eye observation camera can be solved respectively.

[0065] Specifically, during the process of joint calibration, based on a pre-set data automatic acquisition program, in the form of program instructions, the observation camera can be controlled to move synchronously with the tracking camera to different positions in a manner fixed relative to the head-mounted display device, and the calibration object fixed in the scene is synchronously photographed. For example, the left-eye observation camera, the right-eye observation camera, the left-eye tracking camera, and the right-eye tracking camera can be triggered simultaneously to photograph the calibration object. Among them, with the help of the above-mentioned calibration device to fixedly support the observation camera and the head-mounted display device, the observation camera and the head-mounted display device can be controlled to move synchronously and relatively statically to different pre-set positions to photograph the calibration object.

[0066] It can be understood that the observation camera includes two cameras, a left camera and a right camera, and the tracking camera also includes two cameras, a left camera and a right camera. Therefore, in a set of data obtained by one acquisition, there are four calibration object images taken by different cameras at the same timestamp. Moving to different positions for synchronous shooting can obtain multiple sets of calibration object images. Then, the observation camera and the tracking camera are jointly calibrated using these multiple sets of calibration object images. Since the observation camera captures the calibration object through the glasses lens of the head-mounted display device, the internal parameters of the observation camera finally calibrated actually include the influence of the glasses lens of the head-mounted display device on imaging.

[0067] Among them, the internal parameters of the observation camera may include parameters such as the focal length, principal point, and distortion coefficient of the observation camera; the external parameters between the observation camera and the tracking camera may include the relative pose and attitude relationship between the observation camera and the tracking camera.

[0068] Step S220: Based on multiple sets of calibration object images, jointly calibrate the observation camera and the tracking camera to obtain a joint calibration result.

[0069] After obtaining multiple sets of calibration object images, image processing can be performed on the multiple sets of calibration object images to identify the calibration features therein. Based on the image coordinates of the identified calibration features in the corresponding calibration object images, the observation camera and the tracking camera are jointly calibrated. The calibration object can be a calibration board, a calibration target AprilGrid, or other devices that can be used for camera calibration. The image set on the calibration object can be a checkerboard, or dots, user-defined images suitable for camera calibration, etc. Next, an example will be given with a calibration board with a checkerboard pattern as the calibration object.

[0070] The observation camera and the tracking camera are supported by a fixed bracket and synchronously shoot the checkerboard on the calibration board at different positions to obtain multiple sets of calibration object images. Image processing is performed on these multiple sets of calibration object images to extract the checkerboard corner points and determine the corresponding corner point coordinates of the checkerboard corner points in each calibration object image. Based on each corner point coordinate, the observation camera and the tracking camera are jointly calibrated to obtain a joint calibration result. Among them, the joint calibration result may include the external parameters between the observation camera and the tracking camera, the internal parameters of the observation camera, and the internal parameters of the tracking camera.

[0071] Step S230: Based on the joint calibration result, perform optical perspective calibration on the head-mounted display device to obtain a target calibration result.

[0072] Specifically, based on the above joint calibration results, for example, according to the internal parameters of the observation camera, the external parameters between the observation camera and the tracking camera, combined with the virtual image of the calibration object drawn on the display screen of the head-mounted display device, and the captured image obtained by the observation camera shooting the virtual image drawn on the display screen, the target calibration result of the optical perspective calibration of the head-mounted display device can be determined, that is, the spatial relationship between the virtual world and the real world can be determined. The target calibration result can include: the conversion relationship between the observation camera coordinate system and the real world coordinate system, and the conversion relationship between the observation camera coordinate system and the virtual world coordinate system. Furthermore, the conversion relationship between the virtual world coordinate system and the real world coordinate system can be determined, thus completing the optical perspective calibration.

[0073] Step S240: Obtain the first image obtained by the observation camera shooting the calibration object through the glasses lens.

[0074] After completing the optical perspective calibration, the target calibration result can be automatically verified. Specifically, a two-dimensional image obtained by the observation camera shooting the calibration object through the glasses lens can be obtained as the first image, which is used as a reference for determining the calibration error subsequently. Among them, the left-eye observation camera shoots the calibration object through the left glasses lens to obtain the left-eye first image L1; the right-eye observation camera shoots the calibration object through the right glasses lens to obtain the right-eye first image R1.

[0075] Step S250: Obtain the second image obtained by the observation camera shooting the virtual pattern drawn on the display screen of the head-mounted display device; wherein, the virtual pattern is drawn for the calibration object based on the target calibration result.

[0076] The head-mounted display device projects and draws the calibration object on the display screen of the head-mounted display device according to the above target calibration result. Among them, the three-dimensional coordinates of the calibration features on the calibration object in the real world are known. The head-mounted display device converts the known three-dimensional coordinates of the calibration features in the real world obtained in the above step S230 to the display screen coordinate system for drawing, thereby forming a virtual pattern. The observation camera shoots the virtual pattern on the display screen, thereby obtaining the second image. The second image can include the left-eye second image L2 captured by the left-eye observation camera and the right-eye second image R2 captured by the right-eye observation camera.

[0077] It can be understood that there is no fixed order of execution between step S240 and step S250. Step S240 and step S250 can be executed simultaneously, step S250 can also be executed before step S240, or can be executed after step S240.

[0078] Step S260: Determine the calibration error of the target calibration result based on the first image and the second image.

[0079] Specifically, the calibration error can be determined according to the pixel coordinates corresponding to the same feature of the actual calibration object on the first image and the second image. After obtaining the calibration error, based on this calibration error, it can be objectively evaluated whether the current target calibration result meets the expectation, so as to achieve an objective and accurate verification of the optical see-through calibration of the head-mounted display device.

[0080] When it is determined that the current target calibration result has not met the expectation, the observation camera and the tracking camera can be resynchronized to take synchronous pictures of the calibration object to obtain multiple groups of calibration object images, and the new combined calibration is performed according to the newly obtained calibration object images to obtain a new combined calibration result. Then, based on the new combined calibration result, the optical see-through calibration of the head-mounted display device is performed again to obtain a new target calibration result. Then, according to the above steps S240, S250, and S260, the new calibration error is recalculated until it is determined that the current target calibration result meets the expectation according to the latest calibration error.

[0081] In the related art, after the optical see-through calibration is completed, manual participation is required to wear and experience the calibration effect. Therefore, in the prior art, there is no set of standard and objective processes for verifying the optical see-through calibration, which relies on manual intervention and subjective experience, with low efficiency and is not suitable for the mass production environment of head-mounted display devices. This embodiment can implement an automatic, objective, and accurate verification scheme for optical see-through calibration based on the above calibration device, and can accurately and quickly calculate the calibration error of the target calibration result, which is used as an objective and quantitative calibration verification index, so that the subsequent processing flow (re-calibration or confirmation of calibration completion) can be guided based on this calibration verification index. Therefore, it can eliminate the dependence on manual labor, improve the efficiency of calibration verification, and be applicable to the mass production environment of head-mounted display devices.

[0082] On the other hand, since in this embodiment, when performing the combined calibration, the observation camera takes pictures of the calibration object through the glasses lens of the head-mounted display device to collect the calibration data for the combined calibration. Therefore, the combined calibration performed in this embodiment takes into account the influence of the glasses lens on imaging, and the target calibration result obtained by performing the optical see-through calibration based on the combined calibration can also improve the accuracy of the optical see-through calibration compared with the related art.

[0083] Through the above steps S210 to S260, multiple sets of calibration object images obtained by the observation camera and the tracking camera synchronously photographing the calibration object are acquired; wherein, the tracking camera is a camera disposed on the head-mounted display device; based on the multiple sets of calibration object images, the observation camera and the tracking camera are jointly calibrated to obtain a joint calibration result; based on the joint calibration result, optical perspective calibration is performed on the head-mounted display device to obtain a target calibration result; a first image obtained by the observation camera photographing the calibration object through the spectacle lens is acquired; a second image obtained by the observation camera photographing a virtual pattern drawn on the display screen of the head-mounted display device is acquired; wherein, the virtual pattern is drawn for the calibration object based on the target calibration result; according to the first image and the second image, the calibration error of the target calibration result is determined. It can implement an objective and standard verification process for optical perspective calibration, thereby eliminating the dependence on manual participation and improving the efficiency of calibration verification.

[0084] In one embodiment, after determining the calibration error of the target calibration result, the above calibration detection method may further include:

[0085] When the calibration error is greater than a preset error threshold, it is determined that the current target calibration result does not meet the calibration qualification condition; when the calibration error is less than or equal to the error threshold, it is determined that the current target calibration result meets the calibration qualification condition.

[0086] That is, after obtaining the calibration error, the evaluation can be completed based on the following logic (ideally, the calibration error value can be 0):

[0087] An error threshold can be preset in advance. Compare the calculated calibration error with this error threshold. When the calibration error is greater than this error threshold, it indicates that the current optical perspective calibration of the head-mounted display device is unqualified and does not meet the expectation. For example, according to the current target calibration result, when the user uses the head-mounted display device, the image observed from the display screen may be blurred, so recalibration is required. When the calibration error is less than or equal to this error threshold, it indicates that the current target calibration result meets the expectation. Based on the current target calibration result, drawing a pattern on the display screen can enable the user to observe a relatively clear image when wearing the head-mounted display device, so the calibration can be ended.

[0088] Among them, the above error threshold can be obtained according to the measured data. For example, add noise to the calibration result of ideal optical perspective. Determine how much noise exceeds such that the image seen after wearing the head-mounted display device is blurred, and then determine the corresponding error threshold according to this noise.

[0089] Therefore, in this embodiment, the calibration error is used as an objectively quantifiable verification index to verify whether the optical perspective calibration is qualified in a standard and accurate manner, thereby improving the verification efficiency of the optical perspective calibration.

[0090] In one embodiment, when it is determined that the current target calibration result does not meet the calibration qualification condition, the above calibration detection method may further include:

[0091] Repeat the following process until the calibration error of the current target calibration result meets the calibration qualification condition:

[0092] Based on multiple groups of newly calibrated object images resynchronously collected by the observation camera and the tracking camera, re-perform joint calibration on the observation camera and the tracking camera, re-perform optical perspective calibration on the head-mounted display device based on the new joint calibration result, and recalculate the calibration error of the target calibration result according to the new target calibration result.

[0093] Briefly speaking, when the calibration error of the current target calibration result is greater than the preset error threshold, the steps S210 to S260 of the above embodiment are re-executed until the calibration error of the latest obtained target calibration result is less than or equal to the error threshold. In this way, the final optical perspective calibration result of the head-mounted display device can meet the expectations.

[0094] This embodiment realizes a standard, automatic, and human-free closed-loop calibration verification, thereby ensuring the calibration quality of the head-mounted display device.

[0095] Additionally, in one embodiment, according to step S260 above, determining the calibration error of the target calibration result based on the first image and the second image may include:

[0096] Determine the associated feature pairs corresponding to the first image and the second image; the associated feature pairs are feature pairs in the first image and the second image that are associated with the same calibration feature on the calibrated object; determine the calibration error of the target calibration result according to the pixel distance between the associated feature pairs.

[0097] This embodiment considers that for a head-mounted display device, if the result of the optical perspective calibration is accurate, then for the first image obtained by the observation camera shooting the calibrated object through the spectacle lens and the second image obtained by the observation camera shooting from the display screen, the presentation positions of the same calibration feature of the actual calibrated object should coincide. Also, since the coordinate unit in a two-dimensional image is a pixel. In this regard, this embodiment uses the pixel distance between the feature pairs in the first image and the second image that are associated with the same calibration feature on the calibrated object as the standard metric for evaluating the result of the optical perspective calibration. Furthermore, based on this pixel distance, the calibration error of the target calibration result is determined.

[0098] Among them, the calibration object may include multiple calibration features. Multiple pairs of associated features can be determined from the first image and the second image. Different pairs of associated features are associated with different calibration features on the calibration object. For example, on a checkerboard calibration board, there are multiple checkerboard corner points, such as corner point A, corner point B, corner point C, etc. For corner point A, a set of associated feature pair 1 can be determined from the first image and the second image. For corner point B, a set of associated feature pair 2 can be determined from the first image and the second image. For corner point C, a set of associated feature pair 3 can be determined from the first image and the second image, and so on. Then, calculate the pixel distance between each pair of associated features, and further determine the calibration error of the target calibration result.

[0099] Specifically, the pair of associated features can be the feature points corresponding to the same-named key points (corner points) on the calibration object in the first image and the second image respectively. For example, for a checkerboard calibration board, based on the corner point detection algorithm, the corner points of the first image and the second image can be detected respectively, and the coordinates of each corner point in the first image and the second image can be determined. Also, since each corner point is pre-set with a unique identifier (ID), or has a certain setting order, the feature points of the same-named key points in the first image and the second image can be determined based on the corner point ID or the arrangement order of the corner points on the image, so as to form a pair of associated features.

[0100] In this embodiment, based on the pixel distance of the same calibration feature in the first image and the second image respectively, the calculation of the calibration error of the target calibration result is realized, so as to realize the accurate verification of the target calibration result.

[0101] Specifically, in one embodiment, determining the calibration error of the target calibration result according to the pixel distance between the pairs of associated features may include:

[0102] Calculate the pixel distance between each group of pairs of associated features; perform statistics on the pixel distances between all pairs of associated features to obtain the calibration error of the target calibration result.

[0103] First, the pixel distance between each group of pairs of associated features can be calculated , which represents the pixel distance between the feature pairs corresponding to the same calibration feature i in the left-eye first image L1 and the left-eye second image L2 collected by the left-eye observation camera.

[0104] After that, perform statistics on the pixel distances corresponding to all pairs of associated features. For example, first calculate the average of all the obtained pixel distances to obtain the calibration error:

[0105] ;

[0106] where e is the calibration error and N is the number of calibration features.

[0107] This embodiment can accurately verify the optical perspective calibration by statistically analyzing the pixel distances corresponding to all associated feature pairs to obtain accurate calibration errors.

[0108] Additionally, in one embodiment, obtaining multiple sets of calibration object images obtained by synchronously photographing a calibration object with an observation camera and a tracking camera may specifically include:

[0109] Obtaining calibration object images obtained by synchronously photographing a calibration object when the observation camera and the tracking camera are successively moved to different positions; one set of calibration object images corresponds to one position.

[0110] Let the calibration device stand still at a preset position. Then, adjust the positions of the observation camera and the head-mounted display device, and set the position of the calibration object so that the observation camera can completely observe all calibration features on the calibration object. Let the observation camera and the tracking camera photograph the calibration object at the same timestamp to obtain a set of calibration object images; then, let the observation camera and the tracking camera synchronously move to the next position. When moving, since both the observation camera and the head-mounted display device are fixed by a fixed bracket and the tracking camera is set on the head-mounted display device, the positions between the observation camera and the tracking camera do not change. At the next position, the observation camera and the tracking camera synchronously photograph the calibration object to obtain another set of calibration object images. In this way, multiple sets of calibration object images can be obtained.

[0111] This embodiment can complete the acquisition of multiple sets of calibration object images for joint calibration based on a set program in combination with the above calibration device, thereby improving the convenience and efficiency of data acquisition. The joint calibration achieved based on multiple sets of calibration object images can improve the accuracy of joint calibration.

[0112] Additionally, in one embodiment, performing optical perspective calibration on the head-mounted display device based on the joint calibration result to obtain a target calibration result may include:

[0113] Drawing a virtual pattern for the calibration object on the display screen of the head-mounted display device; determining the three-dimensional feature coordinates of the calibration object based on the virtual pattern, and obtaining a target image obtained by the observation camera photographing the virtual pattern; determining the optical perspective calibration result for the head-mounted display device according to the three-dimensional feature coordinates, the target image, and the joint calibration result.

[0114] The joint calibration result may include the internal parameters of the observation camera and the external parameters between the observation camera and the tracking camera. The conversion relationship between the observation camera coordinate system and the real-world coordinate system can be determined according to the external parameters between the observation camera and the tracking camera.

[0115] A three-dimensional coordinate system R can be established on the plane where the display screen is located. Then, the calibration features of the virtual pattern on the display screen are obtained, and the three-dimensional coordinates under this three-dimensional coordinate system R are obtained, so as to obtain the three-dimensional feature coordinates {P Ri} of the calibration object. i can represent the i-th calibration feature, such as the i-th checkerboard corner point. In addition, an observation camera can be used to take pictures of the virtual pattern displayed on the display screen, so as to obtain a two-dimensional image containing the virtual pattern as the target image. The two-dimensional image coordinates {p ci} of the calibration feature in the image coordinate system of the target image can be extracted from the target image. Then, taking the three-dimensional feature coordinates {P Ri}, the two-dimensional image coordinates {p ci}, and the internal parameters of the observation camera as inputs, and solving the perspective n-point (Perspective-n-Point, abbreviated as PnP) problem, the conversion relationship between the observation camera coordinate system and the virtual world coordinate system is finally solved. Then, based on the conversion relationship between the observation camera coordinate system and the virtual world coordinate system, and the conversion relationship between the real world coordinate system and the observation camera coordinate system, the conversion relationship between the real world coordinate system and the virtual world coordinate system is solved, so as to complete the optical perspective calibration for the head-mounted display device.

[0116] In this embodiment, by introducing an observation camera, the calibration combining virtual and real is split into the conversion relationship between the observation camera coordinate system and the virtual world coordinate system, and the conversion relationship between the observation camera coordinate system and the real world coordinate system; based on the joint calibration result considering the influence of the glasses lens of the head-mounted display device on imaging, the optical perspective calibration is finally realized, thereby improving the accuracy of the optical perspective calibration.

[0117] Figure 4 is a flowchart of the calibration detection method for the head-mounted display device in some embodiments, as shown in Figure 4 . The calibration detection method includes the following steps:

[0118] Step S401, place the calibration device statically, adjust the positions of the observation camera and the head-mounted display device at preset points, set the position of the calibration object, and start the data acquisition program;

[0119] Step S402, based on the data acquisition program, obtain multiple groups of calibration object images obtained by the observation camera and the tracking camera synchronously taking pictures of the calibration object at different points;

[0120] Step S403, perform joint calibration on the observation camera and the tracking camera according to the multiple groups of calibration object images to obtain a joint calibration result;

[0121] Step S404, perform optical perspective calibration on the head-mounted display device based on the joint calibration result to obtain a target calibration result;

[0122] Step S405: Use the observation camera to take pictures of the calibration object through the glasses lens of the head-mounted display device to obtain a first image. The first image includes a left-eye first image captured by the left-eye observation camera and a right-eye first image captured by the right-eye observation camera.

[0123] Step S406: Draw a virtual pattern of the calibration object on the display screen of the head-mounted display device according to the target calibration result.

[0124] Step S407: Use the observation camera to take pictures of the virtual pattern on the display screen to obtain a second image. The second image includes a left-eye second image captured by the left-eye observation camera and a right-eye second image captured by the right-eye observation camera.

[0125] Step S408: Perform key-point detection on the first image and the second image to determine the pixel coordinates and IDs of the calibration features of the calibration object on the first image and the second image respectively.

[0126] Step S409: Determine the pixel distance of the associated feature pair according to the result of Step S408.

[0127] Step S410: Calculate the calibration error of the target calibration result according to the pixel distance.

[0128] Step S411: Determine whether the calibration error is less than or equal to a preset error threshold. If so, end the process; otherwise, return to execute Step S401.

[0129] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here. For example, Step S405 and Step S406.

[0130] The above Steps S401 to S411 can implement an objective, standard and closed-loop verification process for optical see-through calibration, thus eliminating the dependence on manual participation and improving the efficiency of calibration verification.

[0131] Figure 5 It is a structural block diagram of the server 50 in this embodiment. The server 50 is used to perform calibration detection on the head-mounted display device based on a calibration device. The calibration device includes an observation camera and a fixing bracket. The fixing bracket is used to fix the observation camera and the head-mounted display device. The head-mounted display device is fixed between the observation camera lens and a preset calibration object so that the observation camera can take pictures of the preset calibration object through the glasses lens of the head-mounted display device. As Figure 5As shown in the figure, the server 50 includes: an acquisition module 51, a joint calibration module 52, an optical perspective calibration module 53, and a calibration detection module 54; where:

[0132] The acquisition module 51 is configured to acquire multiple sets of calibration object images obtained by synchronously photographing a calibration object by an observation camera and a tracking camera; wherein, the tracking camera is a camera disposed on the head-mounted display device; the joint calibration module 52 is configured to perform joint calibration on the observation camera and the tracking camera according to the multiple sets of calibration object images to obtain a joint calibration result; the optical perspective calibration module 53 is configured to perform optical perspective calibration on the head-mounted display device based on the joint calibration result to obtain a target calibration result; the calibration detection module 54 is configured to acquire a first image obtained by the observation camera photographing the calibration object through the glasses lens; acquire a second image obtained by the observation camera photographing a virtual pattern drawn on the display screen of the head-mounted display device; wherein, the virtual pattern is drawn based on the target calibration result; and determine the calibration error of the target calibration result according to the first image and the second image.

[0133] It should be noted that each of the above modules can be a functional module or a program module, and can be implemented either by software or by hardware. For the modules implemented by hardware, each of the above modules can be located in the same processor; or each of the above modules can also be located in different processors in any combined form.

[0134] In this embodiment, a calibration detection system 60 for a head-mounted display device is provided. Figure 6 FIG. is a schematic structural diagram of the calibration detection system 60 for the head-mounted display device of this embodiment. The calibration detection system 60 includes: a calibration device and the server 50 provided in the above embodiment;

[0135] For the specific structure of the calibration device, reference can be made to Figure 3 , the calibration device may include an observation camera 31 and a fixing bracket 32. Among them, the fixing bracket 32 is used to fix the observation camera 31 and the head-mounted display device. The head-mounted display device is fixed between the lens of the observation camera 31 and a preset calibration object, so that the observation camera 31 can photograph the preset calibration object through the glasses lens of the head-mounted display device; the observation camera 31 is communicatively connected to the server 50.

[0136] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiment and the optional implementation manners, and will not be elaborated in this embodiment.

[0137] In addition, in combination with the calibration detection method of the head-mounted display device provided in the above embodiments, a storage medium can also be provided in this embodiment to implement it. A computer program is stored on the storage medium; when the computer program is executed by a processor, any one of the calibration detection methods of the head-mounted display device in the above embodiments is implemented.

[0138] It should be understood that the specific embodiments described here are only used to explain this application, rather than to limit it. According to the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0139] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data that have been authorized by the user or fully authorized by all parties.

[0140] Obviously, the drawings are only some examples or embodiments of the present application. For those of ordinary skill in the art, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it can be understood that although the work done during this development process may be complex and time-consuming, for those of ordinary skill in the art, certain design, manufacturing, or production changes based on the technical content disclosed in the present application are only conventional technical means and should not be regarded as insufficient disclosure of the present application.

[0141] The term "embodiment" in the present application means that the specific features, structures, or characteristics described in combination with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various positions in the specification and does not necessarily mean the same embodiment, nor does it mean being independent or alternative to other embodiments and mutually exclusive. Those of ordinary skill in the art can clearly or implicitly understand that the embodiments described in the present application can be combined with other embodiments without conflict.

[0142] The above-described embodiments only represent several implementation manners of the present application, and their descriptions are relatively specific and detailed, but should not be construed as a limitation on the scope of patent protection. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present application, several deformations and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A calibration detection method for a head-mounted display device, characterized in that, For a calibration device, the calibration device includes an observation camera and a fixing bracket. Among them, the fixing bracket is used to fix the observation camera and the head-mounted display device. The head-mounted display device is fixed between the lens of the observation camera and a preset calibration object, so that the observation camera takes pictures of the preset calibration object through the glasses lens of the head-mounted display device; the method includes: Obtain multiple groups of calibration object images obtained by synchronously taking pictures of the calibration object by the observation camera and the tracking camera; among them, the tracking camera is a camera arranged on the head-mounted display device; Based on the multiple groups of calibration object images, jointly calibrate the observation camera and the tracking camera to obtain a joint calibration result; Based on the joint calibration result, perform optical perspective calibration on the head-mounted display device to obtain a target calibration result; Obtain a first image obtained by the observation camera taking pictures of the calibration object through the glasses lens; Obtain a second image obtained by the observation camera taking pictures of a virtual pattern drawn on the display screen of the head-mounted display device; among them, the virtual pattern is drawn based on the target calibration result for the calibration object; Based on the first image and the second image, determine the calibration error of the target calibration result.

2. The calibration detection method according to claim 1, wherein, After determining the calibration error of the target calibration result, the method further includes: When the calibration error is greater than a preset error threshold, determine that the current target calibration result does not meet the calibration qualification condition; When the calibration error is less than or equal to the error threshold, determine that the current target calibration result meets the calibration qualification condition.

3. The calibration detection method according to claim 2, wherein When it is determined that the current target calibration result does not meet the calibration qualification condition, the method further includes: Repeat the following process until the calibration error of the current target calibration result meets the calibration qualification condition: Based on multiple groups of new calibration object images re-synchronously collected by the observation camera and the tracking camera, re-jointly calibrate the observation camera and the tracking camera, and based on the new joint calibration result, re-perform optical perspective calibration on the head-mounted display device, and re-calculate the calibration error of the target calibration result according to the new target calibration result.

4. The calibration detection method according to claim 1, wherein Based on the first image and the second image, determining the calibration error of the target calibration result includes: Determine the associated feature pairs corresponding to the first image and the second image; the associated feature pairs are feature pairs in the first image and the second image that are associated with the same calibration feature on the calibration object; Based on the pixel distance between the associated feature pairs, determine the calibration error of the target calibration result.

5. The calibration detection method according to claim 4, wherein, Based on the pixel distance between the associated feature pairs, determining the calibration error of the target calibration result includes: Calculate the pixel distance between each group of the associated feature pairs; Perform statistics on the pixel distances between all the associated feature pairs to obtain the calibration error of the target calibration result.

6. The calibration detection method according to claim 1, wherein Obtaining multiple groups of calibration object images obtained by synchronously taking pictures of the calibration object by the observation camera and the tracking camera includes: Obtain the calibration object images obtained by synchronously photographing the calibration object when the observation camera and the tracking camera are successively moved to different positions; one set of the calibration object images corresponds to one position.

7. The calibration detection method according to any one of claims 1 to 6, characterized in that Based on the joint calibration result, perform optical perspective calibration on the head-mounted display device to obtain a target calibration result, including: Draw a virtual pattern for the calibration object on the display screen of the head-mounted display device; Based on the virtual pattern, determine the three-dimensional feature coordinates of the calibration object, and obtain a target image obtained by the observation camera photographing the virtual pattern; According to the three-dimensional feature coordinates, the target image, and the joint calibration result, determine the optical perspective calibration result for the head-mounted display device.

8. A server, characterized in that, For calibrating and detecting a head-mounted display device based on a calibration device; the calibration device includes an observation camera and a fixing bracket, wherein the fixing bracket is used to fix the observation camera and the head-mounted display device, and the head-mounted display device is fixed between the lens of the observation camera and a preset calibration object, so that the observation camera photographs the preset calibration object through the spectacle lens of the head-mounted display device; The server includes: an acquisition module, a joint calibration module, an optical perspective calibration module, and a calibration detection module; wherein: The acquisition module is used to acquire multiple sets of calibration object images obtained by the observation camera and the tracking camera synchronously photographing the calibration object; wherein, the tracking camera is a camera arranged on the head-mounted display device; The joint calibration module is used to perform joint calibration on the observation camera and the tracking camera according to the multiple sets of calibration object images to obtain a joint calibration result; The optical perspective calibration module is used to perform optical perspective calibration on the head-mounted display device based on the joint calibration result to obtain a target calibration result; The calibration detection module is used to acquire a first image obtained by the observation camera photographing the calibration object through the spectacle lens; acquire a second image obtained by the observation camera photographing a virtual pattern drawn on the display screen of the head-mounted display device; wherein, the virtual pattern is drawn for the calibration object based on the target calibration result; and determine the calibration error of the target calibration result according to the first image and the second image.

9. A calibration detection system for a head-mounted display device, characterized in that, Including: A calibration device and the server according to claim 8; The calibration device includes an observation camera and a fixing bracket, wherein the fixing bracket is used to fix the observation camera and the head-mounted display device, and the head-mounted display device is fixed between the lens of the observation camera and a preset calibration object, so that the observation camera photographs the preset calibration object through the spectacle lens of the head-mounted display device; The observation camera is communicatively connected to the server.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the calibration detection method for the head-mounted display device according to any one of claims 1 to 7.

Images