System and method for coordinate system alignment of multiple head mounted displays

Abstract

A system and method for coordinate system alignment of a plurality of head mounted displays is provided. The method includes capturing, by a first camera of a first head-mounted display, a first image of a second head-mounted display; performing first face detection on the first image through the first head-mounted display to obtain a first bounding box corresponding to the first coordinate system; aligning, by the first head-mounted display, a first coordinate system corresponding to the first head-mounted display with a second coordinate system corresponding to the second head-mounted display according to a first position of the first bounding box so as to update the first coordinate system; and displaying an output image through the first head-mounted display according to the updated first coordinate system.

Description

Technical Field

[0001] The present disclosure relates to extended reality (XR) technology, and more particularly, to a system and method for coordinate system alignment of a multi-head mounted display (HMD). Background Art

[0002] In order to perform coordinate system alignment for multiple HMDs, a third-party object is often used as a reference object. The HMD can perform coordinate system alignment by capturing images of the same reference object. However, in order to perform the method mentioned above, each HMD needs to be pre-configured with information about the reference object. That is, the type or position of the reference object cannot be freely changed. Therefore, the traditional method may not be convenient in some cases. Summary of the Invention

[0003] The present invention is directed to a system and method for coordinate system alignment of multiple HMDs.

[0004] The present invention is directed to a system for coordinate system alignment of multiple head mounted displays. The system includes a first head mounted display and a second head mounted display. The first head mounted display includes a first camera, wherein the first head mounted display is configured to: capture a first image of the second head mounted display through the first camera; perform a first face detection on the first image to obtain a first bounding box corresponding to a first coordinate system; align the first coordinate system corresponding to the first head mounted display with a second coordinate system corresponding to the second head mounted display according to a first position of the first bounding box so as to update the first coordinate system; and display an output image according to the updated first coordinate system.

[0005] In an embodiment of the present invention, the second head mounted display includes a second camera, wherein the second head mounted display is communicatively connected to the first head mounted display and is configured to: capture a second image of the first head mounted display through the second camera; perform a second face detection on the second image to obtain a second bounding box corresponding to a second coordinate system; and transmit information to the first head mounted display, wherein the information is associated with a second position of the second bounding box.

[0006] In an embodiment of the present invention, the first head mounted display aligns the first coordinate system with the second coordinate system according to the information.

[0007] In an embodiment of the present invention, the information includes a difference between a second position and a third position of the second head mounted display.

[0008] In one embodiment of the present invention, the first head-mounted display is further configured to: obtain the coordinates of the first bounding box; and calculate the first position based on the coordinates, the inverse matrix of the internal parameter matrix of the first camera, and the inverse matrix of the external parameter matrix of the first camera.

[0009] In one embodiment of the present invention, the first head-mounted display is further configured to: capture a plurality of images through the first camera; and perform a simultaneous localization and mapping algorithm based on the plurality of images to obtain the external parameter matrix.

[0010] In one embodiment of the present invention, the first head-mounted display is further configured to: obtain the depth information of the first head-mounted display through the first camera; and calculate the first position based on the depth information.

[0011] In one embodiment of the present invention, the first head-mounted display further includes a distance sensor and is further configured to: obtain the depth information of the first head-mounted display through the distance sensor; and calculate the first position based on the depth information.

[0012] The present invention is directed to a method for aligning coordinate systems of multiple head-mounted displays, including: capturing a first image of a second head-mounted display through a first camera of the first head-mounted display; performing a first face detection on the first image by the first head-mounted display to obtain a first bounding box corresponding to a first coordinate system; aligning the first coordinate system corresponding to the first head-mounted display with the second coordinate system corresponding to the second head-mounted display according to the first position of the first bounding box by the first head-mounted display so as to update the first coordinate system; and displaying an output image by the first head-mounted display according to the updated first coordinate system.

[0013] In one embodiment of the present invention, the method further includes: capturing a second image of the first head-mounted display through a second camera of the second head-mounted display, wherein the second head-mounted display is communicatively connected to the first head-mounted display; performing a second face detection on the second image by the second head-mounted display to obtain a second bounding box corresponding to a second coordinate system; and transmitting information from the second head-mounted display to the first head-mounted display, wherein the information is associated with the second position of the second bounding box.

[0014] Based on the above, the system of the present invention provides a method for aligning the coordinate systems of multiple HMDs only by looking at each other.

[0015] To make the foregoing more understandable, several embodiments accompanied by drawings are described in detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are included to provide a further understanding of the present disclosure, and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the present disclosure and, together with the description, are used to explain the principles of the present disclosure.

[0017] Figure 1 Schematic diagram showing a system for coordinate system alignment for multiple HMDS according to an embodiment of the present invention.

[0018] Figure 2 Flowchart showing a method for coordinate system alignment for multiple HMDS according to an embodiment of the present invention.

[0019] Figure 3 Schematic diagram showing face detection according to an embodiment of the present invention.

[0020] Figure 4 Flowchart showing a method for coordinate system alignment for multiple HMDS according to an embodiment of the present invention.

[0021] Explanation of reference numerals in the drawings

[0022] 10: System;

[0023] 30: Image;

[0024] 31: Person;

[0025] 32: Bounding box;

[0026] 100, 200, 300: Head-mounted display;

[0027] 110, 210: Processor;

[0028] 120, 220: Storage medium;

[0029] 130, 230: Transceiver;

[0030] 140, 240: Camera;

[0031] 150, 250: Distance sensor;

[0032] 160, 260: Display;

[0033] S201, S202, S203, S204, S205, S206, S401, S402, S403, S404: Steps. Detailed description

[0034] Figure 1A schematic diagram of a system 10 for coordinate alignment of multiple HMDS according to an embodiment of the present invention is shown, where the system 10 can be implemented in an XR system (e.g., a virtual reality (VR) system, an augmented reality (AR) system, or a mixed reality (MR) system). The system 10 may include a plurality of HMDS including HMD 100, HMD 200, and HMD 300. Each of the plurality of HMDS may have the same or similar hardware or software and may perform the same or similar functions. One of the plurality of HMDS may be a host HMD, and the other HMDS may be guest HMDS. For example, HMD 100 may be the host HMD, and HMD 200 or HMD 300 may be a guest HMD. The guest HMD may align its own camera coordinate system with the camera coordinate system of the host HMD.

[0035] HMD 100 may include a processor 110, a storage medium 120, a transceiver 130, a camera 140, and a display 160. In one embodiment, HMD 100 may further include a distance sensor 150. The processor 110 may be, for example, a central processing unit (CPU) or other programmable micro control unit (MCU), microprocessor, digital signal processor (DSP), programmable controller, application specific integrated circuit (ASIC), graphics processing unit (GPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field programmable gate array (FPGA), or other similar device or a combination of the above devices. The processor 110 may be coupled to the storage medium 120, the transceiver 130, the camera 140, the distance sensor 150, and the display 160.

[0036] The storage medium 120 can be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid state drive (SSD), or similar components, or a combination thereof, configured to record multiple modules or various applications executable by the processor 110.

[0037] The transceiver 130 can be configured to transmit or receive wired or wireless signals. The transceiver 130 can also perform operations such as low noise amplification, impedance matching, frequency mixing, upconversion or downconversion, filtering, amplification, etc. The HMD 100 can communicate with the HMD 200 or the HMD 300 via the transceiver 130.

[0038] The camera 140 can be a photographic device for capturing images. The camera 140 can include an image sensor, such as a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor.

[0039] The distance sensor 150 can obtain depth information of an object by capturing a stereoscopic image of the object or by detecting the time of flight (ToF) of the object (e.g., via an optical signal or a wireless signal).

[0040] The display 160 can form a virtual image on the retina of a person wearing the HMD 100.

[0041] The HMD 200 can include a processor 210, a storage medium 220, a transceiver 230, a camera 240, and a display 260. In one embodiment, the HMD 200 can further include a distance sensor 250. The processor 210 can be, for example, a CPU or other programmable MCU, microprocessor, DSP, programmable controller, ASIC, GPU, ALU, CPLD, FPGA, or other similar device, or a combination of the above devices. The processor 210 can be coupled to the storage medium 220, the transceiver 230, the camera 240, the distance sensor 250, and the display 260.

[0042] The storage medium 220 can be, for example, any type of fixed or removable RAM, ROM, flash memory, HDD, SSD, or similar components, or a combination thereof, configured to record multiple modules or various applications executable by the processor 210.

[0043] The transceiver 230 can be configured to transmit or receive wired or wireless signals. The transceiver 230 can also perform operations such as low-noise amplification, impedance matching, frequency mixing, upconversion or downconversion, filtering, amplification, etc. The HMD 200 can communicate with the HMD 100 or the HMD 300 via the transceiver 230.

[0044] The camera 240 can be a photographic device for capturing images. The camera 240 can include an image sensor, such as a CMOS sensor or a CCD sensor.

[0045] The distance sensor 250 can obtain depth information of an object by capturing a stereoscopic image of the object or by detecting the time of flight of the object (e.g., via an optical signal or a wireless signal).

[0046] The display 260 can form a virtual image on the retina of a person wearing the HMD 100.

[0047] Figure 2 A flowchart showing a method for coordinate system alignment for multiple HMDS according to an embodiment of the present invention, where the method can be implemented by the Figure 1 system 10 shown in the figure.

[0048] In step S201, the host HMD (i.e., the HMD 100) can detect one or more misaligned guest HMDS (i.e., the HMD 200 or the HMD 300). In one embodiment, the HMD 100 can detect the HMD 200 (or the HMD 300) by communicating with the HMD 200 via the transceiver 130.

[0049] In step S202, the host HMD (i.e., the HMD 100) can determine whether the number of misaligned guest HMDS is greater than 0. If the number of misaligned guest HMDS is greater than 0, then proceed to step S203. If the number of misaligned guest HMDS is equal to 0, stop the execution process of the method.

[0050] In step S203, the system 10 can notify the guest HMD (i.e., the HMD 200) which HMD is the host HMD (i.e., the HMD 100). In one embodiment, the identity of the host HMD or the guest HMD can be pre-stored in the storage medium 120 of the HMD 100 or the storage medium 220 of the HMD 200. In one embodiment, the identity of the host HMD or the guest HMD can be determined according to a command input by the user. For example, an HMD (e.g., the HMD 100, the HMD 200, or the HMD 300) can communicate with an input device (e.g., a keyboard, a touch screen, a camera, or a virtual keyboard in a virtual scene displayed by the HMD) via the transceiver of the HMD. The user can manipulate the input device to transmit a command to the HMD.

[0051] In step S204, the HMD 200 (or HMD 300) may guide the user of the HMD 200 to direct their gaze towards the person wearing the HMD 100, so that the HMD 200 can capture an image of the person wearing the HMD 100. For example, the HMD 200 may display a text message to the user through the display 260 to direct the user's gaze towards the person wearing the HMD 100.

[0052] In step S205, the HMDs (i.e., HMD 100, HMD 200, or HMD 300) may estimate each other's poses. Specifically, the HMD 200 may capture an image of the person wearing the HMD 100 through the camera 240, and may perform face detection on the image to obtain a bounding box corresponding to the coordinate system of the HMD 200. Figure 3 A schematic diagram showing face detection according to an embodiment of the present invention. The HMD 200 may capture an image 30, where the image 30 may include the HMD 100 and the person 31 wearing the HMD 100. The HMD 200 may perform face detection on the image 30 to obtain a bounding box 32 of the face of the person 31, where the bounding box 32 may correspond to the coordinate system of the HMD 200 (e.g., world coordinate system, camera coordinate system, or pixel coordinate system).

[0053] In one embodiment, the HMD 200 may obtain the depth information of the bounding box 32 through the camera 240 or the distance sensor 250. Thereafter, the HMD 200 may calculate the position of the bounding box 32 based on the depth information and the bounding box 32, as shown in equation (1), where [x w y w z w 1] is the 3D pose or position of the HMD 100 in the world coordinate system (or the coordinates of the bounding box 32), z c is the depth information of the HMD 100 (or the bounding box 32), I -1 is the extrinsic matrix for converting the coordinates of the bounding box 32 from the camera coordinate system of the HMD 200 to the world coordinate system, K -1 is the intrinsic matrix for converting the coordinates of the bounding box 32 from the pixel coordinate system to the camera coordinate system of the HMD 200, R I is the rotation matrix, T Iis a translation matrix, and [u v] is the 2D pose or position of the HMD 100 in the pixel coordinate system (or the coordinates of the bounding box 32). In one embodiment, the intrinsic matrix K or the extrinsic matrix I can be pre-configured into the HMD 200 or pre-stored in the storage medium 220. In one embodiment, the HMD 200 can capture multiple images through the camera 240 and can perform a simultaneous localization and mapping (SLAM) algorithm based on the multiple images to obtain the extrinsic matrix I.

[0054]

[0055] On the other hand, the HMD 100 can capture an image of the person wearing the HMD 200 through the camera 140, and can perform face detection on the image to obtain a bounding box corresponding to the coordinate system of the HMD 100. The HMD 100 can also obtain the depth information of the bounding box through the camera 140 or the distance sensor 150. The HMD 100 can calculate the position of the bounding box in the coordinate system of the HMD 100 according to the depth information of the bounding box, similar to the manner described in Equation (1).

[0056] In step S206, the visitor HMD (i.e., the HMD 200) can align the coordinate system of the visitor HMD with the coordinate system of the host HMD (i.e., the HMD 100) according to the estimated 3D pose of the host HMD by the visitor HMD and the information received from the host HMD, so as to update the coordinate system of the visitor HMD. The HMD 200 can display the output image (i.e., the rendered image generated by the HMD 200) via the display 260 according to the updated coordinate system of the HMD 200.

[0057] Specifically, the HMD 100 can transmit information to the HMD 200, where the information can include the difference between the position of the HMD 200 estimated by the HMD 100 and the position of the HMD 100. After the HMD 200 receives the information from the HMD 100, the HMD 200 can align the coordinate system of the HMD 200 with the coordinate system of the HMD 100 according to the position of the HMD 100 estimated by the HMD 200, the position of the HMD 200 estimated by the HMD 200, and the information received from the HMD 100, as shown in Equations (2) to (7), where P h (g) is the position of the HMD 200 estimated by the HMD 100 (i.e., the position of the HMD 200 corresponds to the world coordinate system of the HMD 100), P h (h) is the position of the HMD 100 estimated by the HMD 100 (i.e., the position of the HMD 100 corresponds to the world coordinate system of the HMD 100), Pg (g) is the position of the HMD 200 estimated by the HMD 200 (i.e., the position of the HMD 200 corresponds to the coordinate system of the HMD 200), P g (h) (i.e., as shown in Equation (1) [x w y w z w 1]) is the position of the HMD 100 estimated by the HMD 200 (i.e., the position of the HMD 100 corresponds to the world coordinate system of the HMD 200), T is the translation matrix and R is the rotation matrix. The HMD 200 can align the coordinate system of the HMD 200 with the coordinate system of the HMD 100 according to the translation matrix T and the rotation matrix R shown in Equations (6) to (7) so as to update the coordinate system of the HMD 200.

[0058]

[0059]

[0060]

[0061]

[0062] T = P h (g′) - P g (g)…(6)

[0063]

[0064] Figure 4 A flowchart showing a method for aligning coordinate systems for multiple HMDs according to an embodiment of the present invention, where the method can be implemented by the Figure 1 system 10 shown in the figure. In step S401, a first image of a second head-mounted display is captured by a first camera of the first head-mounted display. In step S402, a first face detection is performed on the first image by the first head-mounted display to obtain a first bounding box corresponding to the first coordinate system. In step S403, the first coordinate system corresponding to the first head-mounted display is aligned with the second coordinate system corresponding to the second head-mounted display according to the first position of the first bounding box by the first head-mounted display so as to update the first coordinate system. In step S404, an output image is displayed by the first head-mounted display according to the updated first coordinate system.

[0065] In summary, the system of the present invention provides a method for aligning the coordinate systems of multiple HMDS only by looking at each other. The visitor HMD can perform face detection to determine the position of the host HMD, and the visitor HMD can align its own camera coordinate system with the camera coordinate system of the host HMD based on the position. When implementing the present invention, there is no need to set up a third-party reference object on-site for coordinate system alignment.

[0066] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments without departing from the scope or spirit of the present disclosure. Given the foregoing, it is intended that the present disclosure cover modifications and variations provided that they fall within the scope of the appended claims and their equivalents.

Claims

1. A system for coordinate alignment of multiple head-mounted displays, characterized in that, Comprising: A first head-mounted display including a first camera; and A second head-mounted display, wherein the first head-mounted display is configured to: Capture a first image of the second head-mounted display through the first camera; Perform a first face detection on the first image to obtain a first bounding box corresponding to a first coordinate system; Align the first coordinate system corresponding to the first head-mounted display with a second coordinate system corresponding to the second head-mounted display according to a first position of the first bounding box so as to update the first coordinate system; And Display an output image according to the updated first coordinate system.

2. The system for coordinate alignment of multiple head-mounted displays according to claim 1, wherein the second head-mounted display includes a second camera, and the second head-mounted display is communicatively connected to the first head-mounted display and configured to: Capture a second image of the first head-mounted display through the second camera; Perform a second face detection on the second image to obtain a second bounding box corresponding to the second coordinate system; And Transmit information to the first head-mounted display, wherein the information is associated with a second position of the second bounding box.

3. The system for coordinate alignment of multiple head-mounted displays according to claim 2, wherein the first head-mounted display aligns the first coordinate system with the second coordinate system according to the information.

4. The system for coordinate alignment of multiple head-mounted displays according to claim 2, wherein the information includes a difference between the second position and a third position of the second head-mounted display.

5. The system for coordinate alignment of multiple head-mounted displays according to claim 1, wherein the first head-mounted display is further configured to: Obtain coordinates of the first bounding box; and Calculate the first position according to the coordinates, an inverse matrix of an internal parameter matrix of the first camera, and an inverse matrix of an external parameter matrix of the first camera.

6. The system for coordinate alignment of multiple head-mounted displays according to claim 5, wherein the first head-mounted display is further configured to: Capture a plurality of images through the first camera; and Perform a simultaneous localization and mapping algorithm according to the plurality of images to obtain the external parameter matrix.

7. The system for coordinate alignment of multiple head-mounted displays according to claim 5, wherein the first head-mounted display is further configured to: Obtain depth information of the first head-mounted display through the first camera; and Calculate the first position according to the depth information.

8. The system for coordinate alignment of multiple head-mounted displays according to claim 5, wherein the first head-mounted display further includes a distance sensor and the first head-mounted display is further configured to: Obtain depth information of the first head-mounted display through the distance sensor; and Calculate the first position according to the depth information.

9. A method for coordinate system alignment of multiple head-mounted displays, characterized in that, Comprising: Capture a first image of the second head-mounted display through a first camera of the first head-mounted display; Performing first face detection on the first image by the first head-mounted display to obtain a first bounding box corresponding to a first coordinate system; Aligning, by the first head-mounted display, the first coordinate system corresponding to the first head-mounted display with a second coordinate system corresponding to the second head-mounted display according to a first position of the first bounding box so as to update the first coordinate system; And Displaying, by the first head-mounted display, an output image according to the updated first coordinate system.

10. The method for coordinate system alignment for multiple head-mounted displays according to claim 9, further comprising: Capturing a second image of the first head-mounted display by a second camera of the second head-mounted display, wherein the second head-mounted display is communicatively connected to the first head-mounted display; Performing second face detection on the second image by the second head-mounted display to obtain a second bounding box corresponding to the second coordinate system; And Transmitting, by the second head-mounted display, information to the first head-mounted display, wherein the information is associated with a second position of the second bounding box.

Images