A fusion display method of a mixed reality glasses

Abstract

The application provides a fusion display method of a mixed reality glasses, comprising: establishing a world coordinate system, a reference frame coordinate system and a positioner coordinate system, and determining a coordinate point set of an object to be imaged; determining a first transformation relationship between the reference frame coordinate system and the positioner coordinate system; determining a second transformation relationship between the positioner coordinate system and an image coordinate system of the mixed reality glasses; calibrating the mixed reality glasses to obtain an imaging system intrinsic parameter transformation; and calculating a screen coordinate point set of the object to be imaged in a screen coordinate system of the mixed reality glasses according to the intrinsic parameter transformation relationship, the first transformation relationship, the second transformation relationship and the coordinate point set of the object to be imaged, and performing fusion display.

Description

Technical Field

[0001] The invention relates to the technical field of mixed reality glasses, and in particular to a fusion display method of mixed reality glasses. Background Art

[0002] Mixed reality technology (MR) is a further development of virtual reality technology. By presenting virtual scene information in real scenes, this technology establishes an interactive feedback information loop between the real world, the virtual world and the user to enhance the realism of the user experience.

[0003] Mixed reality glasses are a device that combines virtual reality and augmented reality technologies. When worn on the eyes, these glasses can overlay digital content onto the real world, creating a new, mixed, real and virtual interactive experience. Currently, the positioning of mixed reality glasses relies on advanced sensors and algorithms. However, these technologies can only overlay digital content on the real world, and cannot achieve the goal of the digital content overlaid and displayed by mixed reality glasses in the real world at a different location than the location in the real world, resulting in poor fusion of the overlaid digital content and the real world. Summary of the Invention

[0004] In view of the deficiencies in the prior art, the present invention provides a fusion display method for mixed reality glasses, which solves the above-mentioned problems in the prior art.

[0005] A fusion display method for mixed reality glasses, comprising:

[0006] Establishing the world coordinate system, reference frame coordinate system and positioner coordinate system, and determining the coordinate point set of the object to be imaged;

[0007] Determining a first transformation relationship between the reference frame coordinate system and the positioner coordinate system;

[0008] Determine a second transformation relationship between the locator coordinate system and the image coordinate system of the mixed reality glasses;

[0009] Calibrate the mixed reality glasses to obtain the intrinsic parameter transformation of the imaging system;

[0010] The screen coordinate point set of the object to be imaged in the screen coordinate system of the mixed reality glasses is calculated based on the internal parameter transformation relationship, the first change relationship, the second transformation relationship and the coordinate point set of the object to be imaged, and a fusion display is performed.

[0011] As an embodiment of the present invention, establishing a world coordinate system, a reference frame coordinate system, and a positioner coordinate system, and determining a coordinate point set of an object to be imaged, includes:

[0012] The world coordinate system and the reference frame coordinate system are established by the space positioner, the positioner coordinate system is established by the positioner provided on the mixed reality glasses, and the coordinate point set of the object to be imaged in the reference frame coordinate system is determined.

[0013] As an embodiment of the present application, the first transformation relationship between the reference frame coordinate system and the positioner coordinate system is determined, comprising:

[0014] The several reflective balls provided on the reference frame are tracked in real time by the space positioner, and the change relationship T reference2world from the reference frame coordinate system to the world coordinate system is obtained.

[0015] The several reflective balls provided on the positioner are tracked in real time by the space positioner, and the change relationship T world2locator from the world coordinate system to the positioner coordinate system is obtained.

[0016] The first transformation relationship is calculated based on T world2locator and T reference2world , and the calculation formula of the first transformation relationship is as follows:

[0017] T reference2locator = T world2locator · T reference2world

[0018] Wherein, T reference2locator is the first transformation relationship.

[0019] As an embodiment of the present application, the second transformation relationship T locator2image between the positioner coordinate system and the image coordinate system of the mixed reality glasses is determined, comprising:

[0020] The checkerboard coordinate system is established based on the preset checkerboard;

[0021] The checkerboard is photographed by the mixed reality glasses, and a plurality of photographed images are obtained, and the transformation relationship from the positioner coordinate system fixed on the mixed reality glasses to the world coordinate system is tracked and recorded by the space positioner;

[0022] The coordinates of the checkerboard coordinate system in the world coordinate system are obtained;

[0023] The second transformation relationship T locator2image from the positioner coordinate system to the image coordinate system of the mixed reality glasses is calculated based on the coordinates of the points on the plurality of photographed images in the image coordinate system, the coordinates of the points on the checkerboard coordinate system in the world coordinate system, and the transformation relationship from the positioner coordinate system to the world coordinate system.

[0024] As an embodiment of the present application, comprising:

[0025] Calculate the screen coordinate point set of the object to be imaged in the screen coordinate system of the mixed reality glasses. The calculation formula is as follows:

[0026] (x * ,y * ,n) T =T MR intrinsic ·T reference2locator ·T locator2image (x, y, z, n) T

[0027] Among them, (x * ,y * ,n) T Any element in the screen coordinate point set, (x, y, z, n) T Any element in the coordinate point set.

[0028] Compared with the existing technology, the present invention has the following beneficial effects: the screen coordinate point set of the object to be imaged in the screen coordinate system of the mixed reality glasses is calculated based on various transformation relationships, and the mixed reality glasses display the digital content that needs to be superimposed and displayed in the real world based on the screen coordinate point set, thereby achieving the accurate display of the digital content that needs to be superimposed and displayed in the corresponding position, and improving the fusion effect of the superimposed digital content and the real world. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a flow chart of an embodiment of the invention;

[0030] Figure 2 A schematic diagram of a spatial transformation process according to another embodiment of the invention;

[0031] Figure 3 A checkerboard diagram of another embodiment of the invention. DETAILED DESCRIPTION

[0032] The technical solution of the invention is further described below with reference to the accompanying drawings and embodiments.

[0033] like Figures 1 to 3 As shown, an embodiment of the invention proposes a fusion display method for mixed reality glasses, including:

[0034] Establishing the world coordinate system, reference frame coordinate system and positioner coordinate system, and determining the coordinate point set of the object to be imaged;

[0035] Determining a first transformation relationship between the reference frame coordinate system and the positioner coordinate system;

[0036] Determine a second transformation relationship between the locator coordinate system and the image coordinate system of the mixed reality glasses;

[0037] Calibrate the mixed reality glasses to obtain the intrinsic parameter transformation of the imaging system;

[0038] The screen coordinate point set of the object to be imaged in the screen coordinate system of the mixed reality glasses is calculated based on the internal parameter transformation relationship, the first change relationship, the second transformation relationship and the coordinate point set of the object to be imaged, and a fusion display is performed.

[0039] The working principle and working principle of the above technical solution: During actual use, the coordinate point set of the object to be imaged is determined by establishing a world coordinate system, a reference frame coordinate system and a locator coordinate system; after determining the first transformation relationship and the second transformation relationship from the reference frame coordinate system to the locator coordinate system, the mixed reality glasses are calibrated to obtain the internal parameter transformation of the imaging system; finally, according to various transformation relationships, the screen coordinate point set of the object to be imaged in the screen coordinate system of the mixed reality glasses is calculated, and the mixed reality glasses display the digital content that needs to be superimposed in the real world according to the screen coordinate point set, thereby realizing the accurate display of the digital content that needs to be superimposed in the corresponding position, thereby improving the fusion effect of the superimposed digital content and the real world.

[0040] In one embodiment, establishing a world coordinate system, a reference frame coordinate system, and a positioner coordinate system, and determining a set of coordinate points of an object to be imaged, includes:

[0041] Establish a world coordinate system and a reference frame coordinate system through a spatial locator, establish a locator coordinate system through a locator provided on the mixed reality glasses, and determine the coordinate point set of the object to be imaged in the reference frame coordinate system;

[0042] Determining a first transformation relationship between the reference frame coordinate system and the positioner coordinate system includes:

[0043] The spatial positioning device is used to track several reflective balls on the reference frame in real time, and the change relationship between the reference frame coordinate system and the world coordinate system is obtained. reference2world ;

[0044] The spatial locator is used to track several reflective balls on the locator in real time, and the change relationship between the world coordinate system and the locator coordinate system is obtained. world2locator ;

[0045] Based on T world2locator and T reference2world Calculate the first transformation relationship. The calculation formula of the first transformation relationship is as follows:

[0046] T reference2locator =T world2locator ·T reference2world

[0047] Among them, T reference2locator is the first transformation relationship.

[0048] The working principle and beneficial effects of the above technical solution are as follows: First, a world coordinate system is established through a spatial locator, and a reference frame coordinate system is established based on a reference frame through the spatial locator; the reference frame coordinate system is defined as a base coordinate system, and during the entire transformation process, the reference frame remains relatively stationary with respect to the real world; four reflective balls that can be tracked and identified by the spatial locator are installed on the reference frame, and through real-time tracking by the spatial locator, the transformation relationship T between the reference frame coordinate system, that is, the base coordinate system, and the world coordinate system can be obtained. reference2world ; Then, the locator coordinate system is established through the locator fixed on the mixed reality glasses; three reflective balls that can be tracked and identified by the spatial locator are installed on the locator. Through the real-time tracking of the spatial locator, the transformation relationship T from the world coordinate system to the locator coordinate system can be obtained. world2locator In this way, the first transformation relationship T from the reference frame coordinate system to the positioner coordinate system can be obtained through the spatial positioner reference2locator ; Among them, in the process of determining the coordinate point set of the object to be imaged, the feature points of the object to be imaged and the points on the lines are first collected.

[0049] In one embodiment, a second transformation relationship T is determined between the locator coordinate system and the image coordinate system of the mixed reality glasses. locator2image ,include:

[0050] Establish a checkerboard coordinate system based on the preset checkerboard;

[0051] The chessboard is photographed through mixed reality glasses to obtain a number of photographed images, and the transformation relationship between the coordinate system of the locator fixed on the mixed reality glasses and the world coordinate system is tracked and recorded by a spatial locator;

[0052] Get the coordinates of the chessboard coordinate system in the world coordinate system;

[0053] Based on the coordinates of the points on the captured images in the image coordinate system, the coordinates of the points on the checkerboard coordinate system in the world coordinate system, and the transformation relationship from the locator coordinate system to the world coordinate system, a second transformation relationship T from the locator coordinate system to the mixed reality glasses image coordinate system is calculated. locator2image .

[0054] The working principle and beneficial effects of the above technical solution are as follows: Figure 3 As shown, first, prepare a black and white checkerboard, and establish a checkerboard coordinate system;

[0055] Define an origin O on the chessboard, and the x-axis and y-axis serve as the origin, x-axis and y-axis of the chessboard space coordinate system respectively. y Axis, and take points A and B on the x-axis and y-axis respectively;

[0056] Use the camera function of the mixed reality glasses to take several (preferably 20) images of the chessboard from different angles. During the image capture process, keep the position of the chessboard unchanged. At the same time, use a spatial locator to track and record the transformation from the coordinate system of the locator fixed on the mixed reality glasses to the world coordinate system.

[0057] Get the coordinates of the chessboard coordinate system in the world coordinate system;

[0058] Obtain the coordinates of O, A, and B in the world coordinate system through the spatial locator Then the unit vector in the x-axis direction is:

[0059]

[0060] y The unit vector in the axis direction is:

[0061]

[0062] According to the right-hand rule, the unit vector n=u·v in the z-axis direction can be calculated.

[0063] The second transformation relationship T from the locator coordinate system to the mixed reality glasses image coordinate system can be calculated by using the coordinates of the points on the 20 captured images in the image coordinate system, the coordinates of the points on the checkerboard coordinate system in the world coordinate system, and the transformation relationship from the locator coordinate system to the world coordinate system obtained by tracking the spatial locator. locator2 image ;

[0064] Among them, the rotation matrix R is:

[0065]

[0066] The translation matrix T is:

[0067]

[0068] Then T locator2 image =R·T.

[0069] In one embodiment, the mixed reality glasses are calibrated to obtain an imaging system internal parameter transformation;

[0070] The working principle and beneficial effects of the above technical solution are as follows: in the image measurement process and computer vision application, in order to determine the mutual relationship between the three-dimensional geometric position of a point on the surface of a space object and the corresponding point in the image, a geometric model of the imaging of the photosensitive element needs to be established, the parameters of these geometric models are camera parameters, and these parameters related to the characteristics of the photosensitive element are called intrinsic parameters; the intrinsic parameters are often obtained through calibration of the photosensitive element, including focal length, image coordinate system, screen coordinate system and other parameters; and finally the intrinsic transformation relationship T of the mixed reality glasses imaging system is obtained MR intrinsic , T MR intrinsic The intrinsic transformation relationship is mainly obtained through the above-mentioned 20 chessboard images, and a calibration method based on the chessboard is used to obtain the intrinsic transformation relationship of the mixed reality glasses imaging system.

[0071] In one embodiment, the screen coordinate point set of the object to be imaged in the screen coordinate system of the mixed reality glasses is calculated, and the calculation formula is as follows:

[0072] (x * ,y * ,n) T = TMR intrinsic · Treference2locator ·T locator2image .(x,y,z,n) T

[0073] Where (x * ,y * ,n) T is any element in the screen coordinate point set, and (x, y, z, n) T is any element in the coordinate point set.

[0074] The working principle and beneficial effects of the above technical solution are as follows: the digital content of the object to be imaged, that is, the coordinate point set {(x, y, z)|x∈R, y∈R, z∈R) is represented, wherein the representation of a point in the coordinate point set is (x, y, z, n) T , that is, any element in the coordinate point set, and R represents the set of real numbers;

[0075] The screen coordinate point set in the screen coordinate system of the corresponding mixed reality glasses is represented as {(x * ,y * )|x * ∈R, y * ∈R}, wherein the representation of a point in the screen coordinate point set is (x * ,y * ,n) T , that is, any element in the screen coordinate point set, so that the superposition and fusion of the digital content and the corresponding real object in the real world are seen in the mixed reality glasses.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the invention and are not limiting. Although the invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the invention, which should all be covered by the scope of the claims of the invention.

Claims

1. A fusion display method for mixed reality glasses, characterized in that: include: Establishing a world coordinate system, a reference frame coordinate system, and a locator coordinate system, and determining a set of coordinate points of the object to be imaged; including: establishing the world coordinate system and the reference frame coordinate system using a spatial locator, establishing a locator coordinate system using a locator provided on the mixed reality glasses, and determining a set of coordinate points of the object to be imaged in the reference frame coordinate system; Determine the first transformation relationship between the reference frame coordinate system and the positioner coordinate system; including: using a spatial positioner to track a number of reflective balls on the reference frame in real time, and obtain the change relationship T from the reference frame coordinate system to the world coordinate system reference2world ; Use the spatial locator to track the reflective balls on the locator in real time to obtain the change relationship between the world coordinate system and the locator coordinate system T world2locator ; Based on T world2locator and T reference2world Calculate the first transformation relationship. The calculation formula of the first transformation relationship is as follows: T reference2locator =T world2locator ·T reference2world Among them, T reference2locator is the first transformation relationship; Determine a second transformation relationship between the locator coordinate system and the image coordinate system of the mixed reality glasses; including: establishing a checkerboard coordinate system based on a preset checkerboard; photographing the checkerboard through the mixed reality glasses to obtain a plurality of photographed images, and tracking and recording the transformation relationship between the locator coordinate system fixed on the mixed reality glasses and the world coordinate system through a spatial locator; obtaining the coordinates of the checkerboard coordinate system in the world coordinate system; and calculating a second transformation relationship T from the locator coordinate system to the image coordinate system of the mixed reality glasses based on the coordinates of points on the plurality of photographed images in the image coordinate system, the coordinates of the points on the checkerboard coordinate system in the world coordinate system, and the transformation relationship from the locator coordinate system to the world coordinate system. locator2image ; Calibrate the mixed reality glasses to obtain the intrinsic parameter transformation relationship of the imaging system; The screen coordinate point set of the object to be imaged in the screen coordinate system of the mixed reality glasses is calculated based on the internal parameter transformation relationship, the first change relationship, the second transformation relationship, and the coordinate point set of the object to be imaged, and a fusion display is performed; the screen coordinate point set of the object to be imaged in the screen coordinate system of the mixed reality glasses is calculated using the following calculation formula: (x * ,y * ,n) T =T MR intrinsic ·T reference2locator ·T locator2image ·(x,y,z,n) T Among them, (x * ,y * ,n) T Any element in the screen coordinate point set, (x, y, z, n) T is any element in the coordinate point set, T MR intrinsic is the internal parameter transformation relationship.

Images