Virtual-reality fusion calibration method and system based on optical tracker and AR glasses

Abstract

This invention relates to a virtual-real fusion calibration method and system based on an optical tracker and AR glasses. The method includes: designing virtual markers in a virtual scene of the AR glasses; placing real markers detected by the optical tracker in a real scene; capturing a calibration image containing both virtual and real markers using the camera of the AR glasses; and using the camera coordinate system of the AR glasses as a transformation medium to solve the transformation relationship between the virtual space coordinate system constructed by the AR glasses and the world coordinate system of the optical tracker based on the captured calibration image, thus completing the calibration. Compared with existing technologies, this invention has the advantages of convenient operation, shorter processing time, and greater user-friendliness.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a virtual-reality fusion calibration method and system based on an optical tracker and AR glasses. Background Art

[0002] Augmented Reality (AR) technology has become one of the hot topics in the field of computer applications. Augmented Reality is also known as augmented reality. AR is a relatively new technology that combines real-world information with virtual-world information. It uses computer and other scientific technologies to simulate and process physical information that is difficult to experience in the real world, and displays virtual information in the real world for effective application. Augmented Reality superimposes computer-generated virtual models on real scenes to achieve three-dimensional virtual-real fusion display and enhance users' perception of real scenes. With this advantage, augmented reality has great application prospects in the education industry, industrial production, and medical fields.

[0003] AR glasses are well-known product representatives in the field of AR technology. AR glasses can project virtual objects in space without affecting the user's observation of the real environment, giving users an experience of interacting with the real environment. AR glasses are an application product of augmented reality technology in the field of wearable computers. As an important medium for augmented reality, they have made great progress in recent years and are constantly approaching users, such as Microsoft's HoloLens glasses.

[0004] When users use AR glasses and want to display the virtual model in a certain posture more accurately at a certain position in the real scene, they need a reliable virtual-real calibration method to link the virtual space with the real scene. At present, many augmented reality calibration methods have problems such as inconvenient operation, long time consumption, and high difficulty. This series of problems have affected the widespread application of augmented reality technology to a certain extent. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a virtual-reality fusion calibration method and system based on optical tracker and AR glasses, which is easy to operate, takes less time and is more user-friendly.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A virtual-reality fusion calibration method based on an optical tracker and AR glasses, the method comprising:

[0008] Design virtual markers in the virtual scene of AR glasses;

[0009] Placing real markers detected by the optical tracker in the real scene;

[0010] Use the camera of AR glasses to capture a calibration image containing both virtual markers and real markers;

[0011] Using the camera coordinate system of AR glasses as the conversion medium, the conversion relationship between the virtual space coordinate system constructed by the AR glasses and the world coordinate system where the optical tracker is located is solved based on the captured calibration image to complete the calibration.

[0012] Preferably, the virtual marker and the real marker each include a plurality of calibration points.

[0013] Preferably, the specific steps of performing calibration based on the captured calibration image include:

[0014] Get the pixel coordinates P of the calibration points of the virtual marker from the calibration image virtual2D And the pixel coordinates P of the calibration points of the real marker real2D ;

[0015] Get the virtual three-dimensional coordinates P of the calibration point of the virtual marker in the virtual space coordinate system constructed by the AR glasses virtual3D , and the three-dimensional coordinates P of the calibration points of the real marker detected by the optical tracker in the world coordinate system real3D ;

[0016] Based on P real3D and P real2D Get the transformation matrix between the camera coordinate system of the AR glasses and the world coordinate system where the optical tracker is located

[0017] Based on P virtual3D and P virtual2D Get the transformation matrix between the camera coordinate system of the AR glasses and the virtual space coordinate system constructed by the AR glasses

[0018] Based on the transformation matrix and Get the transformation matrix between the virtual space coordinate system constructed by AR glasses and the world coordinate system where the optical tracker is located

[0019] Preferably, the virtual marker is an object constructed in the virtual scene of the AR glasses and includes multiple marking points, and the center of each marking point is the calibration point of the virtual marker.

[0020] Preferably, the real marker includes a plurality of marking spheres whose three-dimensional coordinates can be detected by the optical tracker, and the center of each marking sphere is the calibration point of the real marker.

[0021] Preferably, before using this method for virtual-reality fusion calibration, the internal parameters of the camera of the AR glasses need to be calibrated.

[0022] A virtual-real fusion calibration system, comprising:

[0023] Set up and design virtual markers in the virtual scene of AR glasses;

[0024] Setting up real markers in real scenes for optical tracker detection;

[0025] Controller: used to control the camera of AR glasses to capture calibration images containing both virtual markers and real markers;

[0026] Processor: Using the camera coordinate system of the AR glasses as the conversion medium, based on the captured calibration image, solves the conversion relationship between the virtual space coordinate system constructed by the AR glasses and the world coordinate system where the optical tracker is located to complete the calibration.

[0027] Preferably, the processor is used to execute a computer program stored in a storage medium to implement the specific steps of performing calibration based on the captured calibration image.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] (1) The present invention uses the camera coordinate system of AR glasses as a conversion medium. It only takes one photo to automatically complete the virtual-real fusion calibration through an algorithm, which is time-saving, low-difficulty, and more user-friendly.

[0030] (2) The virtual marker and the real marker of the present invention each include a number of calibration points, thereby improving the accuracy of subsequent coordinate conversion, and further improving the accuracy of virtual-real fusion calibration, thereby ensuring high-precision virtual-real fusion display. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flowchart of an implementation method of a virtual-reality fusion calibration method based on an optical tracker and AR glasses of the present invention;

[0032] Figure 2 Schematic diagram of coordinate system conversion of the present invention. DETAILED DESCRIPTION

[0033] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the following embodiments are merely illustrative in nature, and the present invention is not intended to limit its applicable objects or uses, and the present invention is not limited to the following embodiments.

[0034] Example

[0035] like Figure 1As shown, this embodiment provides a virtual-reality fusion calibration method based on an optical tracker and AR glasses, the method comprising:

[0036] Design virtual markers in the virtual scene of AR glasses;

[0037] Placing real markers detected by the optical tracker in the real scene;

[0038] Use the camera of AR glasses to capture a calibration image containing both virtual markers and real markers;

[0039] Using the camera coordinate system of AR glasses as the conversion medium, the conversion relationship between the virtual space coordinate system constructed by the AR glasses and the world coordinate system where the optical tracker is located is solved based on the captured calibration image to complete the calibration.

[0040] It should be noted that the internal parameters of the camera of AR glasses need to be calibrated only once for each device.

[0041] Both virtual markers and real markers include several calibration points respectively, among which: the virtual marker is an object constructed in the virtual scene of AR glasses and contains multiple marking points, and the center of each marking point is the calibration point of the virtual marker; the real marker includes multiple marking spheres whose three-dimensional coordinates can be detected by the optical tracker, and the center of each marking sphere is the calibration point of the real marker.

[0042] In this embodiment, the virtual markers may be multiple small balls, multiple marking points, polygons or chessboards, etc., which contain no less than 4 marking points, and the virtual 3D coordinates P of the marking point group in the virtual space coordinate system constructed by the AR glasses are recorded. virtual3D The number of marker balls installed on the real marker must be no less than 4, and the design of the relative positions between the marker balls must comply with the requirements of the official document of the optical tracker. Through the algorithm provided by it, the three-dimensional coordinates of each marker ball can be detected, that is, the three-dimensional coordinates P of the calibration point of the real marker detected by the optical tracker in the world coordinate system real3D .

[0043] The marker image recorded by the AR glasses needs to contain complete and clear virtual markers and real markers. Through image processing, the pixel coordinates of the center of the marker ball group on the real marker in the calibration image and the pixel coordinates of the marker point group used on the virtual marker can be obtained. Knowing the three-dimensional coordinates of the center of the real marker ball and the corresponding pixel coordinates, the algorithm can be used to solve the external parameters of the camera in the Tracker coordinate system when the image was taken. Similarly, the three-dimensional coordinates of the virtual marker point in the virtual space and the corresponding pixel coordinates in the image can be obtained, and the external parameters of the camera in the virtual space coordinate system can be solved. Using the camera space coordinate system as a conversion medium, the transformation of the coordinate systems of the virtual and real spaces can be obtained.

[0044] Specifically, the specific steps of performing calibration based on the captured calibration image include:

[0045] Get the pixel coordinates P of the calibration points of the virtual marker from the calibration image virtual2D And the pixel coordinates P of the calibration points of the real marker real2D ;

[0046] Get the virtual three-dimensional coordinates P of the calibration point of the virtual marker in the virtual space coordinate system constructed by the AR glasses virtual3D , and the three-dimensional coordinates P of the calibration points of the real marker detected by the optical tracker in the world coordinate system real3D ;

[0047] Based on P real3D and P real2D Get the transformation matrix between the camera coordinate system of the AR glasses and the world coordinate system where the optical tracker is located

[0048] Based on P virtual3D and P virtual2D Get the transformation matrix between the camera coordinate system of the AR glasses and the virtual space coordinate system constructed by the AR glasses

[0049] Based on the transformation matrix and Get the transformation matrix between the virtual space coordinate system constructed by AR glasses and the world coordinate system where the optical tracker is located

[0050] The specific coordinate transformation relationship in the above process is as follows Figure 2 As shown, O Camera is the camera coordinate system of AR glasses, O Virtual Virtual space coordinate system constructed for AR glasses, O Tracker is the world coordinate system of the optical tracker.

[0051] Based on the above, this embodiment also provides a virtual-reality fusion calibration system, including:

[0052] Set up and design virtual markers in the virtual scene of AR glasses;

[0053] Setting up real markers in real scenes for optical tracker detection;

[0054] Controller: used to control the camera of AR glasses to capture calibration images containing both virtual markers and real markers;

[0055] Processor: Using the camera coordinate system of the AR glasses as the conversion medium, based on the captured calibration image, solves the conversion relationship between the virtual space coordinate system constructed by the AR glasses and the world coordinate system where the optical tracker is located to complete the calibration.

[0056] The specific design of the virtual marker and the real marker has been described in detail in the above process and will not be repeated here. In addition, the processor in the system is used to execute the computer program stored in the storage medium, thereby implementing the specific steps of calibration based on the captured calibration image.

[0057] The present invention uses the camera coordinate system of AR glasses as the conversion medium. It only needs to take a photo and then automatically complete the calibration through the algorithm to achieve virtual-reality fusion display. It is time-saving, low-difficulty, and more user-friendly. The virtual markers and real markers each include a number of calibration points, thereby improving the accuracy of subsequent coordinate conversion, and then improving the accuracy of virtual-reality fusion calibration, ensuring high-precision virtual-reality fusion display.

[0058] The above embodiments are merely examples and do not limit the scope of the present invention. These embodiments can also be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the technical concept of the present invention.

Claims

1. A virtual-reality fusion calibration method based on an optical tracker and AR glasses, characterized in that: The method includes: Design virtual markers in the virtual scene of AR glasses; Placing real markers detected by the optical tracker in the real scene; Use the camera of AR glasses to capture a calibration image containing both virtual markers and real markers; Using the camera coordinate system of the AR glasses as the conversion medium, the conversion relationship between the virtual space coordinate system constructed by the AR glasses and the world coordinate system where the optical tracker is located is solved based on the captured calibration image to complete the calibration; Calibrate the internal parameters of the AR glasses' camera. Each device only needs to be calibrated once. Through image processing, the pixel coordinates of the center of the marker ball group on the real marker in the calibration image and the pixel coordinates of the marker point group used on the virtual marker are obtained; given the three-dimensional coordinates of the center of the real marker ball and the corresponding pixel coordinates, the algorithm is used to solve the external parameters of the camera in the Tracker coordinate system when the image is taken; similarly, the three-dimensional coordinates of the virtual marker point in the virtual space and the corresponding pixel coordinates in the image can be obtained, and the external parameters of the camera in the virtual space coordinate system can be solved; using the camera space coordinate system as a conversion medium, the transformation of the virtual and real space coordinate systems can be obtained; The specific steps of calibration based on the captured calibration image include: Get the pixel coordinates P of the calibration points of the virtual marker from the calibration image virtual2D And the pixel coordinates P of the calibration points of the real marker real2D ; Get the virtual three-dimensional coordinates P of the calibration point of the virtual marker in the virtual space coordinate system constructed by the AR glasses virtual3D , and the three-dimensional coordinates P of the calibration points of the real marker detected by the optical tracker in the world coordinate system real3D ; Based on P real3D and P real2D Get the transformation matrix between the camera coordinate system of the AR glasses and the world coordinate system where the optical tracker is located Based on P virtual3D and P virtual2D Get the transformation matrix between the camera coordinate system of the AR glasses and the virtual space coordinate system constructed by the AR glasses Based on the transformation matrix and Get the transformation matrix between the virtual space coordinate system constructed by AR glasses and the world coordinate system where the optical tracker is located 2. According to claim 1, a virtual-reality fusion calibration method based on an optical tracker and AR glasses is characterized in that: The virtual marker and the real marker each include a number of calibration points.

3. The virtual-reality fusion calibration method based on an optical tracker and AR glasses according to claim 2, characterized in that: The virtual marker is an object constructed in the virtual scene of the AR glasses and includes multiple marking points, and the center of each marking point is the calibration point of the virtual marker.

4. The virtual-reality fusion calibration method based on an optical tracker and AR glasses according to claim 2, characterized in that: The real marker includes a plurality of marking spheres whose three-dimensional coordinates can be detected by an optical tracker, and the center of each marking sphere is the calibration point of the real marker.

5. The virtual-reality fusion calibration method based on an optical tracker and AR glasses according to claim 1, characterized in that: Before using this method for virtual-reality fusion calibration, the internal parameters of the camera of the AR glasses need to be calibrated.

6. A virtual-reality fusion calibration system, characterized in that: include: Set up and design virtual markers in the virtual scene of AR glasses; Setting up real markers in real scenes for optical tracker detection; Controller: used to control the camera of AR glasses to capture calibration images containing both virtual markers and real markers; Processor: Using the camera coordinate system of the AR glasses as a conversion medium, based on the captured calibration image, solves the conversion relationship between the virtual space coordinate system constructed by the AR glasses and the world coordinate system where the optical tracker is located, and completes the calibration; The processor is used to execute a computer program stored in a storage medium, thereby implementing the method according to any one of claims 1 to 5.

Images