Registering local content between first and second augmented reality viewers

By introducing device coordinate system and rendering engine in augmented reality viewer, users can select feature points to view local content synchronously, solving the problem of synchronous viewing among multiple users and achieving synchronous rendering effect in the real world.

CN113544766BActive Publication Date: 2024-12-03MAGIC LEAP INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202080018919.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-04-19
Filing Date
2020-02-26
Publication Date
2024-12-03
Estimated Expiration
2040-02-26

AI Technical Summary

Technical Problem

Existing augmented reality viewing systems are difficult to achieve synchronous viewing of local content between multiple users, especially in cooperative projects or games, when users find it difficult to see the same local content in the same location at the same time.

Method used

By introducing device coordinate systems, registration marks, unified coordinate system alignment modules and rendering engines in the augmented reality viewer, users can select feature points to determine their own coordinate systems and display image data of local content through projectors to achieve the goal of Synchronous rendering of real-world objects such as tables.

Benefits of technology

It realizes the synchronous viewing of local content between multiple users, and users can see the same local content in the same location in the real world, enhancing the interactive experience of cooperative projects and games.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113544766B_ABST
    Figure CN113544766B_ABST
Patent Text Reader

Abstract

A method for viewing image data of local content is disclosed. An augmented reality view is created by storing a first device coordinate system (DCF), moving a first registration marker to select a first feature point (FP1) and a second feature point (FP2) on at least one real-world object that a user can view through a display. A unified coordinate system (UCS) alignment module stores the positions of the registration markers when FP1 and FP2 are selected, determines a user coordinate system (UCF) based on the positions of the first registration markers when FP1 and FP2 are selected, converts the DCF to the UCF, and displays the image data of the local content received by the projector on a first data source to the user through the display based on the conversion from the first DCF to the first UCF.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 817,318 filed on March 12, 2019 and U.S. Provisional Patent Application No. 62 / 836,417 filed on April 19, 2019, the entire contents of each of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to a viewing system having first and second augmented reality viewers and registering local content for viewing by the viewers. Background Art

[0004] Modern computing and display technology has facilitated the development of user interactive systems including "augmented reality" viewers. Such viewers typically have a head unit with a head-mounted frame that the user can secure to their head, and typically include two waveguides, one in front of each eye of the user. The waveguides are transparent so that ambient light from real-world objects can be transmitted through the waveguides and the user can see the real-world objects. Each waveguide is also used to transmit projection light from a projector to a corresponding eye of the user. The projection light forms an image on the retina of the eye. The retina of the eye thus receives both the ambient light and the projection light. The user simultaneously sees the real-world objects and one or more virtual objects created by the projection light.

[0005] In this way, users can see local content rendered in the real world. Two users in the room will each display their own local content. Sometimes it may be necessary for two users to see the same local content in the same location. For example, two users may want to see the same local content in the same location when collaborating on a project, playing a game, watching a sporting event, etc. Summary of the invention

[0006] The present invention provides a viewing system, comprising a first augmented reality viewer, comprising: a first display, allowing a first user to see a real world object, a first data source, for storing image data of local content, a first projector, displaying the image data of the local content to the first user through the first display when the first user views the real world object, a first processor, a first computer-readable medium connected to the first processor, a first set of visual data and algorithms on the first computer-readable medium and executable by the processor, which includes: a first device coordinate system DCF, a first registration mark, operable by the first user to select a first feature point FP1 and a second feature point FP2 on at least one of the real world objects, a first unified coordinate system UCS alignment module, having: a first feature point storage module, storing the position of the first registration mark when the FP1 and the FP2 are selected, a first user coordinate system calculator, determining a first user coordinate system UCF based on the position of the first registration mark when the FP1 and the FP2 are selected, a first converter, for converting the first DCF to the first UCF, and a first rendering engine, displaying the image data of the local content on the first data source based on the conversion from the first DCF to the first UCF.

[0007] The present invention also provides a method for viewing image data of local content, comprising: creating a first augmented reality view, including: storing a first device coordinate system DCF on a first computer-readable medium, moving a first alignment mark by the first user to select a first feature point FP1 and a second feature point FP2 on at least one of the real objects viewable by the user through a first display, and executing a first unified coordinate system UCS alignment module by: storing a position of the first alignment mark when the FP1 and the FP2 are selected, determining a first user coordinate system UCF based on the position of the first alignment mark when the FP1 and the FP2 are selected, converting the first DCF into the first UCF, and based on the conversion from the first DCF to the first UCF, displaying the image data of the local content received by the first projector on the first data source to the first user through the first display when the first user views the real-world object. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention is further described by way of example with reference to the accompanying drawings, in which:

[0009] Figure 1 is a viewing system used by first and second users according to an embodiment of the present invention and the table and rendered perspective view visible only to the users;

[0010] Figure 2is a schematic diagram showing components of an augmented reality viewer and one user's eye in cross-sectional plan view, other components of the augmented reality viewer in block diagram form, and an interior view of a table;

[0011] Figure 3 is similar to Figure 2 's view, with an emphasis on motion-based adjustment systems;

[0012] Figure 4 is similar to Figure 3 ,view of with a focus on systems for registering local content between users;

[0013] Figure 5 is a perspective view showing a table and various coordinate systems residing within the visual data and algorithms of a corresponding augmented reality viewer, and further showing the direction of gravity;

[0014] Figure 6 is similar to Figure 5 A view showing how a user selects a first and a second feature point;

[0015] Figure 7 is similar to Figure 6 , showing how to transform the coordinate system; and

[0016] Figure 8 is a block diagram of a machine in the form of a computer that may find application in the system of the present invention according to one embodiment of the present invention. DETAILED DESCRIPTION

[0017] Figure Figure 1 A first user 10 and a second user 12 are shown interacting with a viewing system 14 in accordance with one or more embodiments of the present invention. The viewing system 14 includes a respective augmented reality viewer 16 for each user 10 and 12, a network 18, and a server 20. The first and second users are located in an environment having a real object in the form of a table 22. Using the augmented reality viewer 16, the users 10 and 12 can view various renderings, including renderings of local content 24 and renderings of registration markers 26.

[0018] Each augmented reality viewer 16 includes a head unit 30, a waist pack 32, a cable connection 34, and a six degree of freedom (6dof) controller 36. The respective user 10 or 12 secures the head unit 30 to their head and secures the waist pack 32 remote from the head unit 30 to their waist (or other suitable location, such as a backpack). The cable connection 34 connects the head unit 30 to the waist pack 32. The head unit 30 includes technology for displaying one or more virtual objects to the user while allowing the user to see real objects such as the table 22. The waist pack 32 primarily includes the processing and communication capabilities of the augmented reality viewer 16. In another embodiment, the processing and communication capabilities can reside entirely in the head unit 30, so that the waist pack 32 is not required.

[0019] The waist pack 32 is connected to the network 18 via a wireless connection. The server 20 is connected to the network 18 and stores data representing local content. The waist pack 32 downloads the local content from the server 20 via the network 18. The waist pack 32 provides the data to the head unit 30 via a cable connection 34. The head unit 30 generally includes a display including a light source, which may be a laser or LED light or any other suitable light, and a waveguide that guides the light so that the light is received by the retina of each eye of the respective user 10 or 12. The display of the head unit 30 produces slightly different images for the left and right eyes of the user 10 or 12 so that the brain of the user 10 or 12 perceives the light as a three-dimensional image of the local content 24. The scale of the local content 24 and its location and distance from the user 10 or 12 are determined by the data representing the local content 24 and the various coordinate systems used to display the local content 24 to the respective user 10 or 12.

[0020] Local content 24 is displayed to users 10 and 12 at the same position relative to table 22. In the given example, local content 24 is the head of a character facing first user 10 from both the viewpoint of first user 10 and the viewpoint of second user 12.

[0021] The 6dof controller 36 includes a position transmitter, and the head unit 30 includes a position receiver. The position transmitter transmits a position signal, and the position receiver receives the position signal. The position signal includes data representing the position transmitter (and therefore the 6dof controller 36) relative to the position receiver (and therefore the head unit 30).

[0022] In use, the user 10 or 12 mounts the head unit 30 to their head and the waist pack 32 to their waist. The waist pack 32 downloads image data from the server 20 via the network 18. The user 10 or 12 can see the table 22 through the display of the head unit 30. A projector forming part of the head unit 30 receives the image data from the waist pack 32 and generates light based on the image data. The light travels through one or more waveguides forming part of the display of the head unit 30. The light then leaves the waveguide and propagates to the retina of the eye of the user 10 or 12. The projector generates light in a pattern that is replicated on the retina of the eye of the user 10 or 12. The light falling on the retina of the eye of the user 10 or 12 has a selected depth of field so that the user 10 or 12 perceives an image at a preselected depth behind the waveguide. In addition, the two eyes receive slightly different images so that the brain of the user 10 or 12 perceives one or more three-dimensional images at a selected distance from the head unit 30. In this example, both users 10 and 12 perceive local content 24 as an enhancement to table 22 .

[0023] Each user 10 or 12 can also perceive one of the registration marks 26 in three dimensions. The position of the registration mark 26 is always in a set direction relative to the 6dof controller 36 held by the corresponding user 10 or 12. Therefore, when the corresponding 6dof controller 36 is moved left to right and forward and backward, the registration mark 26 moves from left to right and forward and backward. The registration mark 26 attaches itself to the surface of the real object. When the user thus moves the 6dof controller 36, the registration mark 26 can, for example, move across the surface of the table 22. The user 10 or 12 can then select a specific point on the table 22, such as one or more corners of the table 22.

[0024] The local content 24 and the registration markers 26 are not visible from a drawing perspective and are only visible to the users 10 and 12 because they are using the augmented reality viewer 16. The local content 24 and the registration markers 26 are initially data structures that reside within the visual data and algorithms in the waist pack 32. The data structures then manifest themselves as light when a projector in the head unit 30 generates light based on the data structures. Although the local content 24 and the registration markers 26 do not exist in the three-dimensional space in front of the users 10 and 12 (e.g., are virtual objects), they are visible in the three-dimensional space in front of the users 10 and 12. Figure 1 Visualization of computer data in three-dimensional space is used throughout the description to illustrate how data structures that facilitate the rendering perceived by users 10 and 12 are interrelated within the data structures of the vision algorithm in waist pack 32.

[0025] Figure 2One of the augmented reality viewers 16 is shown in more detail, and includes a head unit 30, a 6dof controller 36, and visual data and algorithms 38. In some embodiments, the visual data and algorithms 38 reside primarily in Figure 1 32 in the waist pack. In other embodiments, the visual data and algorithm 38 may reside entirely within the head unit or may be split between the head unit and the waist pack.

[0026] Figure 2 Also included is a data source 40. In this example, the data source 40 includes Figure 1 The image data is received by the server 20 in the embodiment and then stored on the storage device of the waist pack 32. The image data may be, for example, three-dimensional image data that can be used to render the local content 24. The image data may be time-series data that allows the creation of videos that move in two or three dimensions in a predetermined or unpredictable sequence and may be located on real-world objects (e.g. Figure 1 On the table 22).

[0027] The visual data and algorithms 38 include a rendering engine 42 , a stereo analyzer 44 , a motion-based adjustment (Pixelstick) system 46 , and a uniform coordinate system (UCS) alignment module 48 .

[0028] The rendering engine 42 is connected to the data source 40, the motion-based adjustment system 46, and the UCS alignment module 48. The rendering engine 42 is capable of receiving inputs from various systems, the motion-based adjustment system 46 and the UCS alignment module 48 in this example.

[0029] like Figure 3 As shown, the motion-based adjustment system 46 includes a simultaneous localization and mapping (SLAM) system 52, a head unit to world coordinate system (frame) converter (transformer) 54, and a display adjustment algorithm 56. The rendering engine 42 is connected to the display adjustment algorithm 56. The display adjustment algorithm 56 receives input from the head unit to the world coordinate system converter 54. The head unit to the world coordinate system converter 54 is connected to the SLAM system 52. The SLAM system 52 is capable of receiving image data, analyzing the image data to determine objects within the image of the image data, and recording the position of the image data. The head unit to the world coordinate system converter 54 is capable of converting the head unit coordinate system to the world coordinate system.

[0030] The stereo analyzer 44 is connected to the rendering engine 42. The stereo analyzer 44 is capable of determining a left image data set and a right image data set from the data stream provided by the rendering engine 42.

[0031] The head unit 30 includes a head unit main body 60 and a display system 62. The head unit main body 60 has a shape to be worn on the head of the user. The display system 62 is fixed to the head unit main body 60.

[0032] Display system 62 includes left projector 66A and right projector 66B, left waveguide 70A and right waveguide 70B, and detection device 72.Left projector 66A and right projector 66B are connected to power supply.Each projector 66A or 66B has a corresponding input for providing image data of corresponding projector 66A or 66B.Corresponding projector 66A or 66B produces the light of two-dimensional pattern and emits light therefrom when powered on.Left waveguide 70A and right waveguide 70B are positioned to receive light from left projector 66A and right projector 66B respectively.Left waveguide 70A and right waveguide 70B are transparent waveguides.

[0033] The detection device 72 includes a head unit inertial motion unit (IMU) 80 and one or more head unit cameras 82. The head unit IMU 80 includes an accelerometer 84 (or more than one accelerometer), a gyroscope 86 (or more than one gyroscope), and a gravity sensor 88. The components of the IMU are typically formed in a semiconductor chip. The accelerometer and gyroscope can detect the movement of the head unit IMU 80 and the head unit body 60, including movement along three orthogonal axes and rotation around orthogonal axes. The gravity sensor 88 can detect the direction of gravity.

[0034] The head unit camera 82 continuously captures images from the environment around the head unit body 60. The images can be compared with each other to detect the movement of the head unit body 60 and the head of the user wearing the head unit body 60.

[0035] The SLAM system 52 is connected to the head unit camera 82. The head unit to world coordinate system converter 54 relies on data from the SLAM system 52 and the head unit IMU 80 to convert the coordinate system of the head unit 30 into the world coordinate system, i.e., a coordinate system that contains real objects (such as Figure 1 The coordinate system of the table 22).

[0036] In use, the user mounts the head unit body 60 to their head. For example, the components of the head unit body 60 may include a strap (not shown) that wraps around the back of the user's head. The left and right waveguides 70A, 70B are then located in front of the user's left and right eyes 120A, 120B.

[0037] The rendering engine 42 receives the image data from the data source 40. The rendering engine 42 inputs the image data to the stereo analyzer 44. The image data is Figure 1The stereo analyzer 44 analyzes the image data to determine a left image data set and a right image data set based on the image data. The left image data set and the right image data set are data sets representing two-dimensional images that are slightly different from each other, with the purpose of giving the user 10 a perception of three-dimensional rendering.

[0038] The stereo analyzer 44 inputs the left image data set and the right image data set into the left projector 66A and the right projector 66B. The left projector 66A and the right projector 66B then generate the left light pattern and the right light pattern. The components of the display system 62 are shown in a plan view, but it should be understood that when shown in a front view, the left light pattern and the right light pattern are two-dimensional patterns. Each light pattern includes a plurality of pixels. For the purpose of illustration, the light rays 124A and 126A from the two pixels are shown to leave the left projector 66A and enter the left waveguide 70A. The light rays 124A and 126A are reflected from the side of the left waveguide 70A. It is shown that the light rays 124A and 126A propagate from left to right in the left waveguide 70A by internal reflection, but it should be understood that the light rays 124A and 126A also propagate in the direction of entering the paper using a refractory and reflection system.

[0039] Light rays 124A and 126A exit the left optical waveguide 70A through a pupil 128A and then enter the left eye 120A through the pupil 130A of the left eye 120A. Light rays 124A and 126A then fall on the retina 132A of the left eye 120A. In this way, the left light pattern falls on the retina 132A of the left eye 120A. The user is given the perception that the pixels formed on the retina 132A are pixels 134A and 136A that the user perceives as being a certain distance away on the side of the left waveguide 70A opposite the left eye 120A. Depth perception is created by manipulating the focal length of the light.

[0040] In a similar manner, the stereo analyzer 44 inputs the right image data set to the right projector 66B. The right projector 66B transmits a right light pattern, which is represented by pixels in the form of light rays 124B and 126B. The light rays 124B and 126B are reflected within the right waveguide 70B and exit through the pupil 128B. The light rays 124B and 126B then enter through the pupil 130B of the right eye 120B and fall on the retina 132B of the right eye 120B. The pixels of light rays 124B and 126B are perceived as pixels 134B and 136B behind the right waveguide 70B.

[0041] The patterns produced on the retinas 132A and 132B are perceived as left and right images, respectively. The left and right images are slightly different from each other due to the function of the stereo analyzer 44. The left and right images are perceived as three-dimensional renderings in the user's mind.

[0042] As described above, the left waveguide 70A and the right waveguide 70B are transparent. Light of a real object from the left waveguide 70A and the right waveguide 70B on the side opposite to the eyes 120A and 120B can be projected through the left waveguide 70A and the right waveguide 70B and fall on the retinas 132A and 132B.

[0043] The head unit IMU 80 detects every movement of the user's head. For example, if the user 10 moves their head counterclockwise and simultaneously moves their body to the right along with their head, this movement will be detected by the accelerometer 84 and gyroscope 86 in the head unit IMU 80. The head unit IMU 80 provides the measurements from the accelerometer 84 and gyroscope 86 to the display adjustment algorithm 56. The display adjustment algorithm 56 calculates the placement value and provides the placement value to the rendering engine 42. The rendering engine 42 modifies the image data received from the data source 40 to compensate for the movement of the user's head. The rendering engine 42 provides the modified image data to the stereo analyzer 44 for display to the user 10.

[0044] As the user moves their head, the head unit camera 82 continuously captures images. The SLAM system 52 analyzes the images and identifies the images of objects within the images. The SLAM system 52 analyzes the movement of the objects to determine the pose position of the head unit body 60. The SLAM system 52 provides the pose position to the display adjustment algorithm 56. The display adjustment algorithm 56 uses the pose position to further refine the placement values ​​that the display adjustment algorithm 56 provides to the rendering engine 42. The rendering engine 42 thus modifies the image data received from the data source 40 based on a combination of the motion sensors in the head unit IMU 80 and the images captured by the head unit camera 82. As a practical example, if the user rotates their head to the right, the position of the local content 24 rotates to the left within the user's field of view, giving the user the impression that the position of the local content 24 remains stationary relative to the table 22 and other real world objects.

[0045] refer to Figure 4 , the UCS alignment module 48 includes a device coordinate system (DCF) 150, a controller to head unit converter 152, a user coordinate system (UCF) establishment system 154, UCF 156, and a DCF to UCF converter 158.

[0046] The DCF 150 is connected to the gravity sensor 88 through a combination of hardware, software and firmware. The gravity sensor 88 measures the direction of gravity relative to the head unit body 60. The DCF 150 is a coordinate system having three orthogonal axes, one of which is aligned with the direction of gravity.

[0047] UCF established systems 154 including Figure 1The registration mark 26, feature point position calculator 162, feature point storage module 164, and user coordinate system calculator 166 are shown. The feature point position calculator 162 is connected to the controller to the head unit converter 152 through a routine and receives input from the registration mark 26. The registration mark 26 is used to select the first feature point (FP1) and the second feature point (FP2). The feature point storage module 164 is connected to the feature point position calculator 162 and stores the positions of the first feature point FP1 and the second feature point FP2. The user coordinate system calculator 166 receives the positions of the feature points FP1 and FP2 from the memory and calculates the UCF 156 based on the positions of the first feature point FP1 and the second feature point FP2.

[0048] The 6dof controller 36 includes a controller body 170, a position transmitter 172, and a controller IMU 174. The controller body 170 has a shape that is convenient for the user to grasp. The position transmitter 172 and the controller IMU 174 are installed in the controller body 170. The battery (not shown) in the controller body 170 is used to power the position transmitter 172 and the controller IMU 174.

[0049] The head unit 30 includes a position receiver 176. The position receiver 176 is mounted to the head unit body 60 in a fixed position.

[0050] In use, the user holds the controller body 170 in one of their hands. The user can move the controller body 170 in three orthogonal directions and rotate the controller body 170 around three orthogonal axes. The position transmitter 172 continuously transmits position waves and the position receiver 176 receives the position waves. The position wave includes data indicating the position of the position transmitter 172 relative to the position receiver 176. Because the position transmitter 172 is stationary in the 6dof controller 36 and the position receiver 176 is stationary in the head unit 30, the position wave includes data indicating the relationship between the 6dof controller 36 and the head unit 30. The controller to the head unit converter 152 receives data from the position receiver 176 and calculates the position of the 6dof controller 36 relative to the head unit 30. The controller IMU 174 includes one or more accelerometers and one or more gyroscopes. The data from the controller IMU 174 is wirelessly transmitted from the 6dof controller 36 and received by the controller to the head unit converter 152 via a combination of hardware, firmware and software. The controller to head unit converter 152 combines the data from the position receiver 176 and the controller IMU 174 in a "fusion" routine. By fusing the data from the position transmitter 172 and the controller IMU 174, jitter is reduced compared to using the position transmitter 172 alone.

[0051] The registration mark 26 is provided to the rendering engine 42 and visualized to the user in a manner similar to the image data from the data source 40. As described above, the user can move the registration mark 26 by moving the 6dof controller 36. The user uses the registration mark 26 to select the first feature point FP1 and the second feature point FP2. When the user selects the first feature point FP1 and the second feature point FP2, the feature point position calculator 162 calculates the position of the corresponding feature point FP1 or FP2 in the world coordinate system. The feature point storage module 164 stores the positions of the first feature point FP1 and the second feature point FP2. The UCF calculator 166 then calculates the UCF 156 based on the positions of the first feature point FP1 and the second feature point FP2. The UCF calculator 166 then stores the UCF 156 in the memory.

[0052] DCF to UCF converter 158 converts DCF 150 to UCF 156 , resulting in DCF to UCF 178 .

[0053] Figure 1 Each of the users 10 and 12 in the augmented reality viewer 16 has a different DCF 150 and a different UCF 156, and therefore has a different DCF converted to UCF 178. The users 10 and 12 agree (e.g., verbally agree) on the positions of the first feature point FP1 and the second feature point FP2 they select on the table 22. The positions of the first feature point FP1 and the second feature point FP2 are different from the perspective of each user 10 and 12, and their mathematical positions are also calculated to be different within the feature point position calculator 162 of each augmented reality viewer 16. However, because the users 10 and 12 select the same feature points FP1 and FP2 in the real world, i.e., on the table 22, from the perspective of both users 10 and 12, the local content 24 can be rendered at the same position relative to the table 22.

[0054] Figure 5 The corresponding coordinate systems are shown before any user establishes a unified coordinate system. Each augmented reality viewer has a corresponding DCF (DCF1 and DCF2). One of the axes (the Y axis) is aligned with the direction of gravity (g), and the other two axes (the X axis and the Z axis) are fixed relative to the head unit 30. No user has established a UCF. For illustrative purposes only, arbitrary UCFs (UCF1 and UCF2) are shown in the figure and are only included in the figure to illustrate the coordinate system non-unity between the augmented reality viewers 16 used by the first user 10 and the second user 12 ( Figure 1 ).

[0055] Figure 6It is shown that UCF (UCF1 and UCF2) is established for each augmented reality viewer. Each user 10 and 12 selects a first feature point FP1 and a second feature point FP2 on the table 22 from their perspective. Then the corresponding augmented reality viewer of the corresponding user calculates the corresponding UCF (UCF1 and UCF2). Both UCFs (UCF1 and UCF2) are located and oriented at the same position on the table 22.

[0056] Figure 7 The corresponding DCF to UCF converter 158 ( Figure 4 ) is converted. Each augmented reality viewer converts its corresponding DCF (DCF1 or DCF2) into its corresponding UCF (UCF1 or UCF2). Based on the corresponding conversion from the corresponding DCF (DCF1 or DCF2) to the corresponding UCF (UCF1 or UCF2), the local content 24 ( Figure 1 ) is displayed to the corresponding user.

[0057] The viewing system 14 can be easily expanded to be used by three, four or more viewing augmented reality viewers used by three, four or more users. Each augmented reality viewer has its own DCF (DCF1, DCF2, DCF3... Dafne) and uses a corresponding pair of feature points (FP1 and FP2) to calculate a corresponding UCF (UCF1, UCF2, UCF3... UCFn). Then, each augmented reality viewer converts its corresponding DCF (e.g., DCF4) to a corresponding UCF (e.g., UCF4) as described above.

[0058] Figure 8 A diagrammatic representation of a machine in the exemplary form of a computer system 900 is shown within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein in accordance with some embodiments. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. Further, while a single machine is shown, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0059] The exemplary computer system 900 includes a processor 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both), a main memory 904 (e.g., a read-only memory (ROM), a flash memory, a dynamic random access memory (DRAM), such as a synchronous DRAM (SDRAM) or a Rambus DRAM (RDRAM), etc.), and a static memory 906 (e.g., a flash memory, a static random access memory (SRAM), etc.), which communicate with each other via a bus 908.

[0060] The computer system 900 may also include a disk drive unit 916 and a network interface device 920 .

[0061] The disk drive unit 916 includes a machine-readable medium 922 on which is stored one or more sets of instructions 924 (e.g., software) embodying any one or more of the methods or functions described herein. The software may also reside, completely or at least partially, within the main memory 904 and / or the processor 902 during execution by the computer system 900, the main memory 904 and the processor 902 also constituting machine-readable media.

[0062] Software may also be sent or received over network 18 via network interface device 920 .

[0063] The computer system 900 includes a driver chip 950 for driving the projector to generate light. The driver chip 950 includes its own data storage 960 and its own processor 962.

[0064] Although the machine-readable medium 922 is shown as a single medium in the exemplary embodiment, the term "machine-readable medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "machine-readable medium" should also be understood to include any medium that can store, encode, or carry a set of instructions executed by a machine and cause the machine to perform any one or more methods of the present invention. Therefore, the term "machine-readable medium" should be understood to include, but is not limited to, solid-state memories, optical and magnetic media, and carrier signals.

[0065] Although certain exemplary embodiments have been described and shown in the drawings, it is to be understood that these embodiments are merely illustrative and not restrictive of the invention, and that the invention is not limited to the specific construction and arrangements shown and described, since modifications may occur to those skilled in the art.

Claims

1. A viewing system, comprising: The first augmented reality viewer, including: The first head-mounted frame; A first gravity sensor, mounted to the first head-mounted frame, for detecting a direction of gravity; a first display mounted to the first head mounted frame to allow a first user to view real world objects; A first data source, used to store image data of local content; a first projector that displays image data of the local content to the first user through the first display when the first user views the real-world object; a first processor; a first computer-readable medium coupled to the first processor; and A first set of visual data and algorithms on the first computer readable medium and executable by the processor, wherein the first set of visual data and algorithms comprises: a first device coordinate system DCF, one axis of which is fixed relative to the gravity detected by the first gravity sensor; a first registration marker operable by said first user to select a first feature point FP1 and a second feature point FP2 on at least one of said real world objects; and The first unified coordinate system UCS alignment module, including: A first feature point storage module, which stores the position of the first registration mark when the FP1 and the FP2 are selected; a first user coordinate system calculator, which determines a first user coordinate system UCF based on the position of the first registration mark when the FP1 and the FP2 are selected; A first converter, configured to convert the first DCF into the first UCF; and A first rendering engine displays image data of the local content on the first data source based on conversion from the first DCF to the first UCF.

2. The viewing system according to claim 1, further comprising: A second augmented reality viewer comprising: a second head mounted frame; a second gravity sensor mounted to the second head-mounted frame for detecting a direction of gravity; a second display mounted to the second head mounted frame to allow a second user to view real world objects; A second data source is used to store image data of local content; a second projector that displays image data of the local content to the second user through the second display while the second user is viewing the real-world object; A second processor; a second computer-readable medium coupled to the second processor; and A second set of visual data and algorithms on the second computer readable medium and executable by the processor, comprising: a second device coordinate system DCF, one axis of which is fixed relative to the gravity detected by the second gravity sensor; a second registration mark operable by the second user to select the FP1 and the FP2 on the at least one of the real-world objects; and The second unified coordinate system UCS alignment module includes: A second feature point storage module, which stores the position of the second registration mark when the FP1 and the FP2 are selected; a second user coordinate system calculator, which determines a second user coordinate system UCF based on the position of the second registration mark when the FP1 and the FP2 are selected; a second converter, configured to convert the second DCF into the second UCF; and A second rendering engine displays image data of the local content on the second data source based on the conversion from the second DCF to the second UCF.

3. The viewing system according to claim 2, wherein: The first augmented reality viewer comprises: The first head-mounted frame is wearable on the head of the first user; and a first six degrees of freedom (6dof) controller movable by the first user relative to the first head mounted frame to select the FP1 and the FP2, Wherein, the first set of visual data and algorithms includes: a first feature point position calculator, which determines the position of the first registration mark when the FP1 and the FP2 are selected by determining the position of the first 6dof controller relative to the first head mounted frame when the FP1 and the FP2 are selected; and A first feature point storage module stores the position of the 6dof controller in the first DCF.

4. The viewing system according to claim 3, wherein: The second augmented reality viewer comprises: the second head-mounted frame, which is wearable on the head of the second user; and a second six degrees of freedom (6dof) controller movable by the second user relative to the second head mounted frame to select the FP1 and the FP2, Wherein, the second set of visual data and algorithms includes: a second feature point position calculator, which determines the position of the second registration mark when the FP1 and the FP2 are selected by determining the position of the 6dof controller relative to the second head mounted frame when the FP1 and the FP2 are selected; and A second feature point storage module stores the position of the second 6dof controller in the second DCF.

5. The viewing system of claim 1, wherein: The first augmented reality viewer further comprises: a first DCF determination routine executable by the first processor to calculate the first DCF that varies with movement of the first head mounted frame; and First DCF storage instructions executable by the first processor to store the first DCF on the first computer-readable medium.

6. The viewing system according to claim 5, wherein: The first augmented reality viewer further comprises: a first real object detection device that detects the location of at least one real object; a first world object identification routine executable by said first processor to identify the location of at least one point on a surface of said real object; a first world coordinate system determination routine executable by the first processor to calculate a first world coordinate system based on the at least one point; and A first world coordinate system stores instructions executable by the first processor to store the first world coordinate system on the computer-readable medium, wherein the DCF determination routine determines the DCF relative to the world coordinate system.

7. The viewing system of claim 6, wherein: The first real object detection device is a camera.

8. The viewing system of claim 6, wherein: The first real object detection device detects the locations of a plurality of real objects.

9. The viewing system of claim 6, wherein: The first world object identification routine identifies the locations of a plurality of points on a surface of the real object.

10. The viewing system of claim 9, wherein: The first world coordinate system determination routine calculates the first world coordinate system based on the plurality of points.

11. The viewing system of claim 5, wherein: The first augmented reality viewer further comprises: A first inertial measurement unit (IMU) is fixed to the first head mounted frame, the first IMU includes the first gravity sensor, and the DCF determination routine calculates the DCF coordinate system based on the first gravity direction.

12. The viewing system of claim 11, wherein: The first IMU includes at least one of: a first accelerometer and a first gyroscope.

13. A method for viewing image data of local content, comprising: Create the first augmented reality view, including: detecting a direction of gravity using a first gravity sensor mounted to a first head mounted frame relative to the first display; storing a first device coordinate system DCF on a first computer readable medium, the first device coordinate system DCF having one axis fixed relative to gravity detected by the first gravity sensor; moving, by a first user, a first registration mark to select a first feature point FP1 and a second feature point FP2 on at least one real object viewable by said first user through a first display; The first unified coordinate system UCS alignment module is executed by: storing a position of the first registration mark when the FP1 and the FP2 are selected; determining a first user coordinate system UCF based on the positions of the first registration marks when the FP1 and the FP2 are selected; converting the first DCF into the first UCF; and Based on the conversion from the first DCF to the first UCF, image data of local content received by the first projector on the first data source is displayed to the first user through the first display when the first user views a real-world object.

14. The method according to claim 13, further comprising: Create a second augmented reality view, including: detecting a direction of gravity using a second gravity sensor mounted to a second head mounted frame relative to a second display; storing a second device coordinate system DCF on a second computer readable medium, the second device coordinate system DCF having one axis fixed relative to gravity detected by the second gravity sensor; moving, by a second user, a second registration mark to select a first feature point FP1 and a second feature point FP2 on at least one of the real objects viewable by the second user through the second display; The second unified coordinate system UCS alignment module is executed by: storing a position of the second registration mark when the FP1 and the FP2 are selected; determining a second user coordinate system UCF based on the position of the second registration mark when the FP1 and the FP2 are selected; converting the second DCF into the second UCF; and Based on the conversion from the second DCF to the second UCF, image data of local content received by the second projector on the second data source is displayed to the second user through the second display when the second user views the real-world object.

15. The method according to claim 14, further comprising: Wearing the first head-mounted frame on the head of the first user; as well as The first user moves a first six-degree-of-freedom (6DOF) controller relative to the first head-mounted frame to select the FP1 and the FP2; executing a first feature point position calculator to determine the position of the first registration mark when the FP1 and the FP2 are selected by determining the position of the first 6dof controller relative to the first head mounted frame when the FP1 and the FP2 are selected; and A first feature storage module is executed to store the location of the 6dof controller in the first DCF.

16. The method according to claim 15, further comprising: wearing the second head-mounted frame on the head of the second user; as well as The second user moves a second six-degree-of-freedom (6DOF) controller relative to the second head-mounted frame to select the FP1 and the FP2; executing a second feature point position calculator to determine the position of the second registration mark when the FP1 and the FP2 are selected by determining the position of the second 6dof controller relative to the second head mounted frame when the FP1 and the FP2 are selected; as well as A second feature storage module is executed to store the location of the 6dof controller in the second DCF.

17. The method according to claim 13, further comprising: executing, with a first processor, a first DCF determination routine to calculate the first DCF that changes with movement of the first head mounted frame; as well as A first DCF storage instruction executable by the first processor is executed with the first processor to store the first DCF on the first computer-readable medium.

18. The method according to claim 17, further comprising: detecting a location of at least one real object with a first real object detection device; identifying, with a first world object identification routine executable by the first processor, a location of at least one point on a surface of the real object; calculating, with a first world object identification routine executable by said first processor, a first world coordinate system based on said at least one point; as well as The world coordinate system is stored on the computer-readable medium with first world coordinate system storage instructions executable by the first processor, wherein the DCF determination routine determines the DCF relative to the world coordinate system.

19. The method according to claim 18, wherein: The first real object detection device is a camera.

20. The method according to claim 18, wherein: The first real object detection device detects the locations of a plurality of real objects.

Citation Information

Patent Citations

  • Information processing apparatus, information processing system, and information processing method

    CN104205175A

  • Determining coordinate frames in a dynamic environment

    CN106462232A

  • Environmental Mapping System

    US20180190017A1

  • Virtual content scaling with a hardware controller

    US20180218545A1