Coordinate fusion calibration method and system based on XR technology

Abstract

The invention provides a coordinate fusion calibration method and system based on an XR technology, and the method comprises the following steps: S1, manually measuring a transformation matrix # imgabs0 #, S2, obtaining a transformation matrix # imgabs1 #, obtaining a transformation matrix # imgabs2 #, obtaining a transformation matrix # imgabs3 #, S3, carrying out the operation to obtain a transformation matrix # imgabs4 #, S4, carrying out the operation to obtain a transformation matrix # imgabs5 #, S5, carrying out the operation to obtain a transformation matrix # imgabs4 #, and S5, carrying out the operation to obtain a transformation matrix # imgabs4 #. S6, carrying out operation to obtain a transformation matrix # imgabs6 #; S6, carrying out operation to obtain a transformation matrix # imgabs7 # again, and adjusting the origin of the virtual world coordinate system to coincide with the origin of the coordinates of the real world; and S7, fusion is completed. The method has the advantages that the problem of position deviation of the virtual world and the real world of head display equipment in the prior art can be effectively solved, the relative transformation matrix of the head display camera C and the locator t is calculated through the method, then the fused transformation matrix # imgabs8 # is calculated, the origin of coordinates of the virtual world and the origin of coordinates of the real world are adjusted to coincide, and therefore the head display equipment is more accurate. And the exercise using accuracy of the user is ensured.

Description

Technical Field

[0001] The present invention relates to the field of XR technology, and in particular to a coordinate fusion calibration method and system based on XR technology. Background Art

[0002] Extended Reality (XR) refers to the combination of reality and virtuality through computers to create an environment where humans and machines can interact. It is also a general term for various technologies such as augmented reality (AR), virtual reality (VR), and mixed reality (MR). By integrating the visual interaction technologies of the three, it brings the experiencer an "immersive feeling" of seamless transition between the virtual world and the real world.

[0003] The existing head-mounted display device is provided with a camera, which projects a virtual image in front of the eyes, and cooperates with the head-mounted display handle to achieve a corresponding virtual immersive experience. The head-mounted display device has a built-in virtual coordinate system. When the head-mounted display device is started, an initial origin is set. When the user moves the head-mounted display, the image observed by the user in the virtual world will also move accordingly. Although the image in the virtual world will change with the movement of the user in the real world, the use of the built-in virtual coordinate system leads to deviations from the actual position coordinates in the real world, and it is difficult to show that the moving distance is equal to the moving distance of the virtual image. For some high-precision work that requires the cooperation of the head-mounted display device, it cannot meet the user's usage needs, which is not conducive to improving the competitiveness of the product. Summary of the invention

[0004] In order to solve the problems in the prior art, the present invention provides a coordinate fusion calibration method and system based on XR technology.

[0005] The present invention provides a coordinate fusion calibration method based on XR technology, comprising the following steps:

[0006] Step S1: Place the locator of the external positioning device and the head display handle H at designated positions respectively, generate the origin O of the real world coordinate system by the external positioning device, and manually measure the transformation matrix T of the head display handle H relative to the origin O of the real world coordinate system. o h ;

[0007] Step S2: Start the head display and generate the origin U of the virtual world coordinate system. According to the virtual world coordinate system, directly obtain the transformation matrix T of the head display handle H relative to the origin U of the virtual world coordinate system. u h , according to the virtual world coordinate system, directly obtain the transformation matrix T of the head display camera C relative to the origin U of the virtual world coordinate system uc , the locator t on the head display communicates with the locator of the external positioning device, and directly obtains the transformation matrix T of the locator t on the head display relative to the origin O of the real world coordinate system according to the real world coordinate system o t ;

[0008] Step S3, according to the transformation matrix T u h and the transformation matrix T o h Perform matrix transformation operations to obtain the transformation matrix T of the origin U of the virtual world coordinate system relative to the origin O of the real world coordinate system u o ;

[0009] Step S4, according to the transformation matrix T u o and the transformation matrix T u c Perform matrix transformation operations to obtain the transformation matrix T of the head-mounted display camera C relative to the locator t on the head-mounted display o c ;

[0010] Step S5: According to the transformation matrix T o c and the transformation matrix T o t Perform matrix transformation operations to obtain the transformation matrix T of the head-mounted camera C relative to the origin O of the real world coordinate system c t ;

[0011] Step S6: According to the transformation matrix T u o , transformation matrix T o t and the transformation matrix T c t Perform matrix transformation operations to re-obtain the transformation matrix T of the virtual world coordinate system origin U relative to the head display camera C u c , adjust the origin of the virtual world coordinate system to coincide with the origin of the real world coordinate system;

[0012] Step S7: The virtual world coordinate system and the real world coordinate system are integrated.

[0013] The present invention is further improved, and the specific steps of manual measurement in step 1 are as follows:

[0014] Step A1: Place the headset handle H on the table and identify a reference point P on the headset handle H;

[0015] Step A2: Use an external length measuring tool to measure the relative position of the reference point P and the locator of the external positioning device to obtain the transformation matrix T of the reference point P relative to the origin O of the real world coordinate system. o p ;

[0016] Step A3: The size of the headset controller model in the virtual world is consistent with the size of the real headset controller, and the transformation matrix T of the coordinates of the reference point P relative to the virtual world headset controller can be directly obtained in the virtual world coordinate system. h p ;

[0017] Step A4: According to the transformation matrix T o p and the transformation matrix T h p Perform matrix transformation operations to obtain the transformation matrix T of the headset handle relative to the origin O of the real world coordinate system o h .

[0018] The present invention is further improved, in step A4, the transformation matrix T o h By formula T o h =T o p (T h p ) -1 Calculated.

[0019] The present invention is further improved, in step A1, the head display handle H is placed on the plane where the locator of the external positioning device is located.

[0020] The present invention is further improved, in step S3, the transformation matrix T u o By formula T u o =T u h (T o h ) -1 Calculated.

[0021] The present invention is further improved, in step S4, the transformation matrix T o c By formula T o c =(T u o ) -1 T u c Calculated.

[0022] The present invention is further improved, in step S5, the transformation matrix T c t By formula T c t =(T o c ) -1 T o t Calculated.

[0023] The present invention is further improved, in step S6, the transformation matrix T u c By formula T u c =T u o T o t (T c t ) -1 Calculated.

[0024] The present invention also provides a system, including a head display, an external positioning device, and a head display handle, the head display handle is provided with a head display camera C and a locator t, and the external positioning device is provided with a locator.

[0025] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a coordinate fusion calibration method and system based on XR technology, which can effectively solve the problem of deviation between virtual position and real world position of head display devices in the prior art. By using this method, the transformation matrix T is manually measured. o h , and then obtain the transformation matrix T through the virtual world coordinates o h and the change matrix T u c , the locator of the external positioning device and the locator on the head display communicate to obtain the change matrix T o t , and then obtain the transformation matrix T through matrix transformation c t , and finally recalculate the transformation matrix T u c , adjusting the origin of the virtual world coordinate system to coincide with the origin of the real world coordinate system ensures the accuracy of the user's movement and can effectively improve production efficiency and product competitiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the solutions in the present application or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 This is a flow chart of a coordinate fusion calibration method based on XR technology of the present invention;

[0028] Figure 2 A system block diagram of the present invention;

[0029] Figure 3 Schematic diagram of matrix calculation for the headset controller in the real world coordinate system;

[0030] Figure 4 Schematic diagram for calculating the transformation matrix between the locator on the head display and the camera on the head display;

[0031] Figure 5 Schematic diagram for calculating the position of the virtual camera in the virtual world coordinate system. DETAILED DESCRIPTION

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field of this application; the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit the application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0033] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0035] like Figure 2 As shown, a system includes a head display, an external positioning device, and a head display handle. The head display handle is provided with a head display camera C and a locator t, and the external positioning device is provided with a locator.

[0036] Through the locator and locator t, the real-time construction of the real-world coordinate system is achieved.

[0037] The present invention provides a monitoring method, such as Figure 1-5 As shown, this method is used in the above system. The detailed implementation method of auxiliary rehabilitation monitoring in this example is as follows:

[0038] 1. Place the locator of the external positioning device and the head display handle H at the specified positions respectively, generate the origin O of the real world coordinate system with the external positioning device, and manually measure the transformation matrix T of the head display handle H relative to the origin O of the real world coordinate system o h .

[0039] By measuring the transformation matrix T o h It can provide a basis for the subsequent confirmation of the relative position calculation between the coordinate system of the locator of the external positioning device and the coordinate system of the virtual world in the head display.

[0040] The specific steps for manual measurement in step 1 are as follows:

[0041] Step A1: Place the headset handle H on the table and identify a reference point P on the headset handle H;

[0042] Step A2: Use an external length measuring tool to measure the relative position of the reference point P and the locator of the external positioning device to obtain the transformation matrix T of the reference point P relative to the origin O of the real world coordinate system. o p ;

[0043] For example, the following transformation matrix T is obtained by measuring o p ,

[0044]

[0045] Step A3: The size of the headset controller model in the virtual world is consistent with the size of the real headset controller, and the transformation matrix of the coordinates of the reference point P relative to the virtual world headset controller can be directly obtained in the virtual world coordinate system.

[0046]

[0047] Step A4: According to the transformation matrix T o p and the transformation matrix Perform matrix transformation operations to obtain the transformation matrix T of the headset handle relative to the origin O of the real world coordinate system o h .

[0048] Since the coordinate position of the headset handle in the virtual world coordinate system can be obtained more directly by the headset, the transformation matrix T can be easily obtained through matrix transformation. o h .

[0049] according to Figure 3 , for the transformation matrix T in step A4 o h By formula Calculated by matrix transformation.

[0050]

[0051] In step A1, the head display handle H is placed on the plane where the locator of the external positioning device is located. By placing it on the plane where the locator of the external positioning device is located, the measurement is convenient and the measurement accuracy can be improved.

[0052] 2. Start the headset and generate the origin U of the virtual world coordinate system. According to the virtual world coordinate system, directly obtain the transformation matrix T of the headset handle H relative to the origin U of the virtual world coordinate system u h , according to the virtual world coordinate system, directly obtain the transformation matrix T of the head display camera C relative to the origin U of the virtual world coordinate system u c , the locator t on the head display communicates with the locator of the external positioning device, and directly obtains the transformation matrix T of the locator t on the head display relative to the origin O of the real world coordinate system according to the real world coordinate system o t .

[0053] By directly deriving the transformation matrix T from the virtual world coordinate system built into the headset u h and the transformation matrix T u c , which provides a basis for the subsequent calculation of the transformation matrix of the origin U of the virtual world coordinate system relative to the origin O of the real world coordinate system.

[0054]

[0055]

[0056] 3. According to the transformation matrix T u h and the transformation matrix T o h Perform matrix transformation operations to obtain the transformation matrix T of the origin U of the virtual world coordinate system relative to the origin O of the real world coordinate system u o .

[0057] By calculating through matrix transformation, the transformation matrix T can be obtained u o , which provides the basis for the subsequent calculation of the transformation matrix of the head display camera C relative to the locator t on the head display.

[0058] according to Figure 4 , for the transformation matrix T u o By formula T u o =T u h (T o h ) -1 It is calculated that the transformation matrix T obtained in the above steps is u h and the transformation matrix T o h Calculate the transformation matrix T u o .

[0059] 4. According to the transformation matrix T u o and the transformation matrix T u c Perform matrix transformation operations to obtain the transformation matrix T of the head-mounted display camera C relative to the locator t on the head-mounted display o c .

[0060] By calculating through matrix transformation, the transformation matrix T can be obtained o c , which provides the basis for the subsequent transformation matrix of the head-mounted display camera C relative to the origin O of the real-world coordinate system.

[0061] according to Figure 4 , for the transformation matrix T o c By formula T o c =(T u o ) -1 T u c It is calculated that the transformation matrix T obtained in the above steps is u h and the transformation matrix T o h Calculate the transformation matrix T o c .

[0062] 5. According to the transformation matrix T o c and the transformation matrix T ot Perform matrix transformation operations to obtain the transformation matrix T of the head-mounted camera C relative to the origin O of the real world coordinate system c t .

[0063] By calculating through matrix transformation, the transformation matrix T can be obtained c t , to obtain the transformation matrix T of the virtual world coordinate system origin U relative to the head display camera C u c Provide the foundation.

[0064] according to Figure 4 , for the transformation matrix T c t By formula T c t =(T o c ) -1 T o t It is calculated that the transformation matrix T obtained in the above steps is o c and the transformation matrix T o t Calculate the transformation matrix T c t .

[0065]

[0066] 6. According to the transformation matrix T u o , transformation matrix T o t and the transformation matrix T c t Perform matrix transformation operations to re-obtain the transformation matrix T of the virtual world coordinate system origin U relative to the head display camera C u c , adjust the origin of the virtual world coordinate system to coincide with the origin of the real world coordinate system.

[0067] By calculating through matrix transformation, the transformation matrix T after the coordinate system fusion can be obtained again u c . Recalibrate the coordinate system in the virtual world to ensure that the coordinate system of the virtual world and the real world are the same.

[0068] according to Figure 5 , for the transformation matrix T u c By formula T u c =T u oT o t (T c t ) -1 It is calculated that the transformation matrix T obtained in the above steps is u o , transformation matrix T o t and the transformation matrix T c t Calculate the transformation matrix T o c .

[0069] 7. The virtual world coordinate system and the real world coordinate system are integrated.

[0070] Since the locator t and the head-mounted display camera are both fixed on the head-mounted display, T c t It is also fixed, that is, you only need to complete the calibration once to achieve the fusion of the virtual world coordinate system and the real world coordinate system, and then you can use it repeatedly in other scenes.

[0071] The recalibrated coordinate system in the headset allows the user to see the handle in the virtual world approach an object while the handle in the real world also approaches the object, and achieve synchronized touch, ensuring the uniformity of coordinate display after movement, which is convenient for use in surgical training in simulated operating rooms. Existing physical objects provide touch, and the headset is used to achieve accurate surgical training.

[0072] As can be seen from the above, the present invention provides a coordinate fusion calibration method and system based on XR technology, which can effectively solve the problem of deviation between virtual position and real world position of head display devices in the prior art. By using this method, the transformation matrix T is manually measured. o h , and then obtain the transformation matrix T through the virtual world coordinates o h and the change matrix T u c , the locator of the external positioning device and the locator on the head display communicate to obtain the change matrix T o t , and then obtain the transformation matrix T through matrix transformation c t , and finally recalculate the transformation matrix T u c , adjusting the origin of the virtual world coordinate system to coincide with the origin of the real world coordinate system ensures the accuracy of the user's movement and can effectively improve production efficiency and product competitiveness.

[0073] The specific implementation modes described above are preferred implementation modes of the present invention, and are not intended to limit the specific implementation scope of the present invention. The scope of the present invention includes but is not limited to the specific implementation modes, and all equivalent changes made according to the present invention are within the protection scope of the present invention.

Claims

1. A coordinate fusion calibration method based on XR technology, characterized in that: The steps include: Step S1: Place the locator of the external positioning device and the head display handle H at designated positions respectively, generate the origin O of the real world coordinate system by the external positioning device, and manually measure the transformation matrix of the head display handle H relative to the origin O of the real world coordinate system. Step S2: Start the head display and generate the origin U of the virtual world coordinate system. According to the virtual world coordinate system, directly obtain the transformation matrix of the head display handle H relative to the origin U of the virtual world coordinate system. According to the virtual world coordinate system, directly obtain the transformation matrix of the head display camera C relative to the origin U of the virtual world coordinate system The locator t on the headset communicates with the locator of the external positioning device, and directly obtains the transformation matrix of the locator t on the headset relative to the origin O of the real-world coordinate system according to the real-world coordinate system. Step S3, according to the transformation matrix and the transformation matrix Perform matrix transformation operations to obtain the transformation matrix of the origin U of the virtual world coordinate system relative to the origin O of the real world coordinate system Step S4, according to the transformation matrix and the transformation matrix Perform matrix transformation operations to obtain the transformation matrix of the head-mounted display camera C relative to the locator t on the head-mounted display Step S5, according to the transformation matrix and the transformation matrix Perform matrix transformation operations to obtain the transformation matrix of the head-mounted display camera C relative to the origin O of the real world coordinate system Step S6: According to the transformation matrix Transformation Matrix and the transformation matrix Perform matrix transformation operations to re-obtain the transformation matrix of the virtual world coordinate system origin U relative to the head display camera C Adjust the origin of the virtual world coordinate system to coincide with the origin of the real world coordinate system; Step S7: The virtual world coordinate system and the real world coordinate system are integrated.

2. The coordinate fusion calibration method based on XR technology according to claim 1, characterized in that: The specific steps of manual measurement in step 1 are as follows: Step A1: Place the head display handle H on the table and identify a reference point P on the head display handle H; Step A2: Use an external length measuring tool to measure the relative position of the reference point P and the locator of the external positioning device to obtain the transformation matrix of the reference point P relative to the origin O of the real world coordinate system. Step A3: The size of the headset controller model in the virtual world is consistent with the size of the real headset controller, and the transformation matrix of the coordinates of the reference point P relative to the virtual world headset controller can be directly obtained in the virtual world coordinate system. Step A4: According to the transformation matrix and the transformation matrix Perform matrix transformation operations to obtain the transformation matrix of the headset handle relative to the origin O of the real world coordinate system 3. The coordinate fusion calibration method based on XR technology according to claim 2, characterized in that: The transformation matrix in step A4 By formula Calculated.

4. The coordinate fusion calibration method based on XR technology according to claim 2, characterized in that: In step A1, the head display handle H is placed on the plane where the locator of the external positioning device is located.

5. The coordinate fusion calibration method based on XR technology according to claim 1, characterized in that: The transformation matrix in step S3 By formula Calculated.

6. The coordinate fusion calibration method based on XR technology according to claim 1, characterized in that: The transformation matrix in step S4 By formula Calculated.

7. The coordinate fusion calibration method based on XR technology according to claim 1, characterized in that: The transformation matrix in step S5 By formula Calculated.

8. The coordinate fusion calibration method based on XR technology according to claim 1, characterized in that: The transformation matrix in step S6 By formula Calculated.

9. A system for implementing any of the above-mentioned coordinate fusion calibration methods based on XR technology, characterized in that: It includes a head display, an external positioning device, and a head display handle. The head display handle is provided with a head display camera C and a locator t, and the external positioning device is provided with a locator.

Images