Mixed reality presentation

Abstract

The mixed reality presentation system may include a head mounted device and a main unit for presenting a virtual image to a person wearing the head mounted device. The main unit may include a sensor for sensing the surrounding physical environment and the position and orientation of the head mounted device, a virtual image generator, and a transmitter. The virtual image generator is to generate a virtual image based on the sensed surrounding physical environment and the sensed position and orientation of the head mounted device. The transmitter is to transmit the generated virtual image to the head mounted device for presentation.

Description

Background Art

[0001] Virtual reality systems create computer-generated simulations of three-dimensional images or objects. Augmented reality systems display virtual images or objects on top of a real-world physical environment. Mixed reality systems virtually attach virtual images or objects to real-world physical objects or locations within a real-world physical environment, where changes to the real-world physical environment may result in changes to the virtual images being presented. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 is a block diagram schematically illustrating portions of an example mixed reality presentation system.

[0003] Figure 2 is a schematic diagram illustrating an example virtual image superimposed on a real-world physical environment and the virtual attachment of virtual objects to anchor features or real-world objects.

[0004] Figure 3 is a block diagram of an L-string portion of an example non-transitory computer-readable medium of an example virtual image generator.

[0005] Figure 4 is a flow chart of an example mixed reality rendering method.

[0006] Figure 5 is a block diagram that schematically connects the parts of an example head mounted device.

[0007] Figure 6 is a perspective view of an example mixed reality rendering system.

[0008] Figure 7 is a perspective view of an example mixed reality rendering system.

[0009] Figure 8 is a flow chart of an example mixed reality rendering method.

[0010] Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements. The drawings are not necessarily to scale, and the size of some parts may be exaggerated in order to more clearly illustrate the examples shown. In addition, the drawings provide examples and / or implementations consistent with the description; however, the description is not limited to the examples and / or implementations provided in the drawings. DETAILED DESCRIPTION

[0011] Disclosed herein are mixed reality presentation systems and mixed reality presentation methods that centralize the computing and sensing functions of the system in a main unit, thereby facilitating the use of head-mounted devices (also referred to as headphones or glasses) that may be more compact, lighter, more comfortable to wear, less expensive, and more battery efficient. The disclosed mixed reality presentation system and mixed reality presentation method centralize the computing and sensing functions of the system in a main unit by employing the main unit to sense both the surrounding physical environment and the position and orientation of the head-mounted device. The main unit itself performs the generation of virtual images based on the sensed surrounding physical environment and the sensed hardware position and orientation. The main unit transmits the generated virtual images to the head-mounted device for presentation by the head-mounted device.

[0012] In some embodiments, the head mounted device only presents or displays the virtual images (along with any audible presentation) that it receives from the main unit. The head mounted device, sometimes referred to as "video overlay glasses", includes a head mounted display mechanism that optically overlays computer generated images (virtual images with virtual objects) on top of the user's field of view through the eye mounted device. In some embodiments, the head mounted device does not include sensors for mapping the user's surroundings or for keeping the overlaid images "virtually attached" to real world objects. In some embodiments, the head mounted device itself does not perform any image generation. In some embodiments, the head mounted device itself does not capture images or provide any image data to the main unit. In some embodiments, the virtual images generated by the main unit and transmitted to the head mounted device omit any image data from the head mounted device.

[0013] In some embodiments, the head mounted device may additionally sense changes in its own position and / or orientation, wherein the head mounted device modifies the virtual image received from the main unit based on its own sensed motion. In such embodiments, the head mounted device may not generate a virtual image, but rather two-dimensionally shift the virtual image it receives. In such embodiments, the head mounted device may include an inertial motion unit (IMU), wherein the head mounted device performs a shift of the virtual image received from the main unit to compensate for any head mounted device motion detected by the IMU in the period of time since the virtual image was generated. Such a two-dimensional shift of the virtual image may involve far fewer computing components, computing power, and battery power than generating the virtual image itself.

[0014] In some embodiments, the head-mounted device may include sensors that sense the physical environment, wherein the sensed physical environment is transmitted to the main unit. In such an embodiment, the main unit uses information sensed by the head-mounted device (in the form of signals transmitted to the main unit) to track physical objects outside the field of view of the sensors of the main unit. In some embodiments, the resolution and / or depth of the sensors provided in the head-mounted device are much lower than the resolution and / or depth of the sensors used by the main unit to sense the surrounding physical environment. In some embodiments, the signals received from the sensors of the head-mounted device are not used directly by the main unit to generate virtual images, but are used to track physical objects outside the field of view of the main unit.

[0015] In some embodiments, the main unit may further include a user interaction sensor. The user interaction sensor detects user touch in the surrounding physical environment. In such an embodiment, the virtual image generated by the main unit and transmitted to the head mounted device is based on the signal from the user interaction sensor.

[0016] In some embodiments, the mixed reality presentation system is in the form of a desktop environment, or the method is implemented in a desktop environment. In one embodiment, the mixed reality presentation system is in the form of an accessory for a display and sensor system for desktop installation or support, and the accessory includes a main unit and a head-mounted device. In one such embodiment, the sensor of the main unit can be supported by a monitor, screen or display with a vertical surface. In some embodiments, the sensor and virtual image generator of the main unit can be combined as part of the monitor, screen or display, such as with an integrated computing system. In some embodiments, the sensor of the main unit can be supported by a monitor screen or display, and the computing components that perform virtual image generation based on signals from the sensors are provided in a separate fixed computing housing or unit.

[0017] In the illustrated example, a master unit different from the head-mounted device is responsible for tracking the video overlay glasses or head-mounted device, tracking the user's viewpoint, and creating a spatial map of the desktop and objects on its surface. The master unit further identifies and locates the horizontal plane of the desktop surface and the documents or objects on the surface. The master unit (such as with commercially available HP Aurasma / Reveal) further attaches virtual content to such identified locations, such as by adding virtual objects to the desktop and adding virtual content to the identified documents. The master unit can further present virtual content from the user's viewpoint position. In such a case, this may involve the final task of shifting the computer image based on movement data from the position / orientation sensor. The resulting overlay image can then be transmitted to the video overlay glasses and headsets.

[0018] An example mixed reality presentation system is disclosed, which may include a head mounted device and a main unit that presents a virtual image to a person wearing the head mounted device. The main unit may include a sensor that senses the surrounding physical environment and the position and orientation of the head mounted device, a virtual image generator, and a transmitter. The virtual image generator is to generate a virtual image based on the sensed surrounding physical environment and the sensed position and orientation of the head mounted device. The transmitter is to transmit the generated virtual image to the head mounted device for presentation.

[0019] An example mixed reality presentation method is disclosed. The method may include constructing a spatial map of a surrounding physical environment with a fixed master unit, tracking a position and orientation of a head mounted device with the fixed master unit, generating a virtual image based on the spatial map and the position and orientation of the head mounted device tracked with the fixed master unit, and transmitting the virtual image from the master unit to the head mounted device for presentation by the head mounted device.

[0020] An example mixed reality presentation system is disclosed, the example mixed reality presentation system including a non-transitory computer-readable medium containing processor instructions. The instructions direct the processor to: construct a spatial map of the surrounding physical environment with a fixed master unit; track the position and orientation of a head mounted device with the fixed master unit; generate a virtual image based on the spatial map and the position and orientation of the head mounted device tracked with the fixed master unit; and transmit the virtual image from the master unit to the head mounted device for presentation by the head mounted device.

[0021] Figure 1 is a block diagram schematically illustrating portions of an example mixed reality presentation system 20. The mixed reality presentation system 20 centralizes the computing and sensing functions of the system in a main unit, thereby facilitating the use of a head-mounted device (also referred to as a headset or glasses) that may be more compact, lighter, more comfortable to wear, less costly, and more battery efficient. The mixed reality presentation system 20 centralizes the computing and sensing functions of the system in a main unit by employing the main unit to sense both the surrounding physical environment and the position and orientation of the head-mounted device. The main unit itself performs the generation of virtual images based on the sensed surrounding physical environment and the sensed hardware position and orientation. The main unit transmits the generated virtual images to the head-mounted device for presentation by the head-mounted device. The mixed reality presentation system 20 includes a head-mounted device 30 and a main unit 50.

[0022] The head-mounted device 30 includes an accessory worn by a person to present virtual images, and in some embodiments, audio, to the person wearing the head-mounted device. The head-mounted device 30, sometimes referred to as "video overlay glasses," includes a head-mounted display that optically overlays computer-generated images (virtual images) on top of the user's field of view through eye-mounted devices. The head-mounted device 30 can be in the form of headphones or glasses that rest over the nose and in front of the eyes of the person wearing the head-mounted device. The head-mounted device 30 presents a virtual image 32 (shown in dotted lines) to the person. In the illustrated example, the head-mounted device 30 omits sensors for mapping the user's surroundings or for keeping the overlaid image virtually attached to real-world objects (these functions are performed by the main unit 50).

[0023] The main unit 50 performs most of the computing and sensing functions of the system 20. As a result, the head mounted device 30 can be more compact, lighter, more comfortable to wear, less expensive, and more battery efficient. The main unit 50 includes a sensor 52, a virtual image generator 54, and a transmitter 56.

[0024] The sensor 52 includes a sensor that senses a field of view 61 of the surrounding physical environment 60. The sensor 52 additionally senses the position and / or orientation of the head mounted device 30 (as indicated by the dashed line 62). In one embodiment, the sensor 52 includes (one or more) cameras and a depth camera sensor. The sensor 52 can be fixedly supported relative to the surrounding physical environment. In other embodiments, the sensor 52 can be mounted on a bracket, track, or other movable actuator close to the surrounding physical environment 60. The sensor 52 is different from the head mounted device 30 and is not carried by the head mounted device 30. Signals from the sensor 52 are transmitted to the virtual image generator 54.

[0025] The virtual image generator 54 generates a virtual image based on the sensed and mapped surrounding physical environment, the sensed position and orientation of the head mounted device 30, and based on instructions or programming that control the shape, size, color, surface markings or other visual attributes and position of the virtual object to be generated relative to the real world or surrounding physical environment. The virtual image generator 54 transmits the virtual image to the head mounted device 30 for presentation to the user.

[0026] As Figure 2As shown, the virtual image 32 transmitted to and presented by the head mounted device 30 may include a virtual object 62. The virtual object 62 may have some predefined relationship with a real world physical object 64 in the surrounding physical environment 60. The virtual image generator 54 maps the position of the object 64 based on the signal from the sensor 52 that captures the field of view 61 of the sensor 52. Based on the mapped position of the object 64 (as determined from the signal from the sensor 52) and the sensed position and / or orientation of the hardware 30, the virtual image generator 54 generates the virtual object 62, and further virtually attaches the virtual object 62 to the associated real world object 64.

[0027] Virtual attachment means that the relative position of the virtual object on the display of the head mounted device 30 is based on or defined by the mapped position and / or orientation of the associated real world object. In such an embodiment, the real world object acts as an anchor or anchoring feature for at least one virtual object or other virtual content. Figure 2 The virtual object 62 is illustrated as being displayed on top of or above the real-world physical object 64, but it should be appreciated that such a "virtual attachment" can be other relative positions of the virtual object 62 relative to the mapped physical object 64. Such a "virtual attachment" can involve a predefined spacing and / or angular offset of the virtual object 62 relative to the real-world physical object. In some embodiments, the "virtual attachment" can involve simulating physical interactions with real-world objects, such as a virtual object falling on a physical table, rolling across its surface and resting. In some embodiments, the mapped positioning of the real-world object 64 may cause the display of the virtual object 62 to be changed. For example, depending on the mapping or real-world position of the physical object 64, the virtual object 64 can be partially shown, such as where the virtual object 64 is to be illustrated as partially contained within the real-world physical object 64, protruding from the real-world physical object 64, or hidden by the real-world physical object 64.

[0028] As Figure 2 As further shown, the movement of real-world objects 64 within the surrounding physical environment 60 (as indicated by arrow 66) is tracked by the virtual image generator 54 of the main unit 50. The virtual image generator 54 changes the position (as indicated by arrow 67), shape and / or size of the displayed virtual object 62 based on the real-world repositioning of the tracked objects 64.

[0029] Virtual image generator 54 may include a processor following instructions provided in a non-transitory computer-readable medium. Figure 3is a block diagram illustrating an exemplary non-transitory computer readable medium 100 of the main unit 50 for directing the processor of the virtual image generator 54. The medium 100 may include various instruction modules, instructions or code sets for directing the processor to perform its reality presentation 200, such as Figure 4 As depicted in .

[0030] As Figure 3 As shown, the medium 100 includes spatial map builder instructions 104, hardware tracker instructions 108, virtual image generator instructions 112, and virtual image transmitter instructions 114. The spatial map builder instructions 104 direct the processor to perform Figure 4 The spatial map builder instructions 104 direct the processor to construct a spatial map of the surrounding physical environment 60. The construction of the spatial map is based on the signals from the sensors 52 and the physical object recognition instructions that identify the physical edges and boundaries of different physical objects in the surrounding physical environment. The instructions 104 further record each identified real-world physical object (such as object 64 or object 65 (in Figure 1 and Figure 2 60). As the real-world physical object 64 is repositioned and / or reoriented within the surrounding physical environment 60, the mapped or tracked position of the real-world physical object may change dynamically.

[0031] Head mounted device tracker instruction 108 directs the processor to perform Figure 4 208 of method 200 shown in FIG. Instructions 108 direct the processor to track the position and / or orientation of the head mounted device 30. The tracked position and orientation is transmitted to the virtual image generator 54 (in Figure 1 ). The position and / or orientation of the head mounted device 30 may affect the relative position and / or orientation of a generated virtual object that is part of a virtual image 32 transmitted to the head mounted device 30.

[0032] Virtual image generator instructions 112 direct the processor to perform Figure 4212 of the method 200 shown in FIG. The instructions 112 direct the processor to generate a virtual image containing a virtual object or multiple virtual objects for presentation by the head mounted device 30. The virtual object may be a three-dimensional object, a two-dimensional object, text, a graphic, or other virtual construct. The virtual objects forming the virtual image may have a processor-generated or computer-generated size, shape, and position based on a spatial map of the surrounding physical environment 60 and its real-world physical objects 64, 65 (or other real-world physical objects) and the position and / or orientation of the head mounted device 30, which indicates the viewpoint of the person wearing the head mounted device 30. The virtual objects are also primarily based on the nature of the mixed reality application responsible for generating them, such as interpretive text in a training application or a 3D model in a CAD application. When the virtual object moves or is reoriented within the surrounding physical environment, and / or when the head mounted device 30 is reoriented or re-oriented by the person wearing the head mounted device 30, the virtual image generator 54 following the virtual image generator instructions 112 may change the size, shape, and size or position of the computer-generated virtual object or other parts of the virtual image 32. Sensors 52 may also detect user interaction with a virtual object, which virtual image generator instructions 112 may use to cause the object to respond in some visual or audible manner, depending on the nature of the application.

[0033] Virtual image transmitter instructions 104 direct the processor to execute Figure 4 The instructions 104 direct the processor to output a control signal to the transmitter 56, thereby causing the transmitter 56 to transmit the virtual image to the head mounted device 34 in a wired or wireless manner for presentation by the head mounted device 30. In one embodiment, the transmitter 56 may include a Bluetooth transmitter. In other embodiments, the transmitter 56 may include other forms of wireless or wired transmitters. Because the virtual image is created by the virtual image generator 54 of the main unit 50 and transmitted to the head mounted device 30, rather than the head mounted device 30 generating the image, the head mounted device 30 simply presents such a virtual image, thereby allowing the head mounted device 30 to be less bulky, lighter in weight, and lower in cost.

[0034] Figure 5is a block diagram schematically illustrating a head mounted device 330. The head mounted device 330 may be employed as part of the system 20 described above, and may be used as part of the method 200. The head mounted device 330 utilizes a virtual image generated by the main unit 50, thereby allowing the head mounted device 30 to be less complex, lighter, less bulky, and less expensive. In the illustrated example, the head mounted device 330 senses a change in its own position and / or orientation (i.e., movement) before presenting the virtual image to a person wearing the head mounted device 330, and utilizes the sensed position and / or orientation change to adjust rather than generate the virtual image received from the main unit 50. The head mounted device 330 includes an inertial motion unit 370 and a display controller 374.

[0035] The inertial motion unit 370 includes a sensor that senses a change in the position and / or orientation of the head mounted device 330, which is related to the position and orientation of the field of view provided by the head mounted device 330 to the person wearing the head mounted device 330. In one embodiment, the inertial motion unit 370 includes one or more accelerometers. In one embodiment, the inertial motion sensor 370 includes a gyroscope. In other embodiments, the inertial motion unit 370 includes both an accelerometer and a gyroscope. In other embodiments, the inertial motion unit 370 may include other forms of sensors to sense the position and / or orientation of the head mounted device 330. The inertial motion unit 330 outputs a signal indicating the sensed change in position and / or orientation to the display controller 374.

[0036] The display controller 374 controls the display of the virtual image 32 received from the main unit 50 on the display portion 334 of the head mounted device 330. The display controller 374 includes a processor 376 and a non-transmitting computer readable medium 378. The processor 376 executes instructions provided on the medium 378. The medium 370 includes a computer readable medium containing instructions, such as in the form of programming or code, directing the processor 376 to output a signal to adjust the display of the virtual image 32 based on the signal from the inertial motion unit 370.

[0037] exist Figure 5In the example shown in , the head mounted device 330 is illustrated as receiving a virtual image 32 having a virtual object located at coordinates x, y presented by the display of the head mounted device 330. Based on the signal from the inertial motion unit 370, the processor 376 following the instructions contained in the memory 378 two-dimensionally shifts the virtual object of the virtual image 32 to the new coordinates x', y', so that the virtual image 32 is shifted and presented as a virtual image 32' on the display portion 334 of the head mounted device 330. Such a two-dimensional shift can improve the accuracy of the mixed reality presentation by correcting errors in the virtual image 32 caused by the time delay from when the sensor 52 senses the position and orientation of the head mounted device 332 and the time when the head mounted device 330 receives the virtual image based on the position of the head mounted device 330 originally sensed by the main unit 50. For example, at the time when the head mounted device 330 receives the virtual image 32, the person wearing the head mounted device 330 may have moved or reoriented his or her field of view and the head mounted device 330. By shifting the received virtual image based on a more current head mounted device position and / or orientation as sensed by inertial motion unit 370 , head mounted device 330 provides a more accurate mixed reality presentation.

[0038] In the illustrated example, the display controller 374 repositions or shifts the virtual image 32, but does not generate the virtual image 32 or its virtual objects. Accurate correction in the virtual image based on the detected head mounted device motion will use a full 3D transformation of the virtual image content. However, three factors lead to a 2D shift operation being an acceptable first approximation. First, the user's head motion is typically small compared to the frame rate (typically 60 Hz or higher) and latency of the hardware. Second, the user's head motion is primarily about the rotation of the neck. Third, most of the presented virtual content is within a limited distance range from the user (e.g., corresponding to a desktop). Therefore, for a first approximation, the virtual image can be viewed as being on the inner surface of a sphere centered on the user's head, and where the radius corresponds to the distance from the user's head to the desktop. The main unit 50 can also transmit parameters to the display controller 374 that are used to control 2D correction based on, for example, the main unit's determination of the distance of the head mounted device from the desktop. As a result, the head mounted device 330 can display virtual images more accurately relative to physical real world objects without incurring the cost, weight, and size associated with the head mounted device 330 generating the virtual image itself.

[0039] Figure 64 is a perspective view of a portion of an example mixed reality presentation system 420 employed as part of a mixed reality desktop system. The system 420 provides a simulated desktop environment that includes a mix of both real-world physical objects and virtual objects. In some embodiments, as described above, virtual objects can be virtually attached to specific real-world physical objects. In the illustrated example, for some, 20 is provided as part of an accessory for use with a desktop computing system having a flat desktop surface 402 with a monitor, screen, or display 408. The system 420 includes a head mounted device 330 (described above) and a main unit 450.

[0040] The main unit 450 is supported by the display 400 in a single box housing that also contains (one or more) head mounted device / glasses tracking sensors 452-1, (one or more) camera and depth camera sensors 452-2, user interaction sensors 453, and virtual image generator / transmitter 454. The tracking sensor 452-1 includes an infrared sensor, a camera optionally including a depth sensor, or other sensor that senses the position and / or orientation of the head mounted device 330 (as indicated by the dashed line 462). In one embodiment, the sensor 452-1 includes an optical tracking sensor that positions the head mounted device 330 in a work volume corresponding to a range of head positions of a desk worker. In one embodiment, the sensor can track one or more features, such as an infrared light or pattern marker on the head mounted device 430.

[0041] Sensor 452-2 senses or captures an image of the desktop 402 and those real-world objects within the field of view 461 of such cameras and the depth camera. In the illustrated example, sensors 452-1 and 452-2 are provided as part of a head that overhangs above the display 400 and protrudes forward to capture a field of view 461 covering a working area of ​​the surface 402 between the person wearing the head mounted device 330 and the display 400. The working area on top of the surface 402 may include real-world physical objects such as a keyboard, keypad or tablet computer 465-1, a computer mouse 465-2, and real-world physical documents, paper pads, books or papers 465-3. Real-world physical objects may also include the user's hands, forearms, etc. Each such real-world physical object (collectively referred to as object 465) is captured as part of an image by sensor 452-2.

[0042] The user interaction sensor 453 detects user touches in the surrounding physical environment 460. This can be used to supplement the system's detection of the user's hand gestures using sensor 452-2. In such an embodiment, the virtual image generated by the main unit and transmitted to the head mounted device is based on the signal from the user interaction sensor. In one embodiment, the user interaction sensor 453 may include a microphone coupled to the desktop, or as an infrared laser projected onto the desktop in a plane and an associated detector.

[0043] The virtual image generator / transmitter 454 corresponds to and is generally the same as the virtual image generator 54 and the transmitter 56 as described above. The virtual image generator / transmitter 454 may include the medium 100 and may implement the method 200. As described above, the virtual image generator / transmitter 454 uses signals from the camera or sensor 452-2 to generate a map 470 (shown as a grid) of the surrounding physical environment 460, which includes physical real-world objects 465 on the surface 402. The virtual image generator / transmitter 454 employs object recognition to identify and define different real-world objects based on edge, feature and contour identification. The characteristics of the identified real-world objects, such as their position and orientation relative to other real-world objects and in the general space of the environment 460, are determined and stored dynamically in real time. The virtual image generator / transmitter 454 tracks the movement and orientation of the different physical real-world objects so identified.

[0044] As described above, the virtual image generator / transmitter 454 further tracks the position and / or orientation of the head mounted device 330 using signals from the sensor 452-1. The virtual image generator / transmitter 454 determines the current field of view provided by the head mounted device 330 based on those signals received from the sensor 452-1. Using the determined field of view, the virtual image generator / transmitter 454 determines which portion of the mapped real-world physical environment 460 is within the field of view. By identifying which real-world physical objects are currently within the field of view of the head mounted device 330, the virtual image generator / transmitter 454 determines not only which virtual objects will also be within the same field of view, but also determines the size, shape, position and / or orientation of such virtual objects given the particular field of view provided by the head mounted device 330 in the mapped location of the object 465. Movement of real-world physical objects as seen in the field of view from the head mounted device 330 can result in changes in the characteristics of the virtual object or multiple virtual objects (for example, virtual object 432 in Figure 6), which is provided as part of the virtual image transmission head mounted device 330. For example, movement of real world physical objects can result in changes to the map 470, which can then also result in changes to the depicted position, orientation, size, or portion (such as when a virtual object would be blocked, hidden, or at least partially received by a real world object).

[0045] In some embodiments, any of the real-world physical objects 465 described above may be alternatively in the form of a virtual object. For example, a keyboard or touchpad 465-1 may be a virtual object. A document, tablet, or paper 465-3 may be a virtual object. A computer mouse 465-2 may be a virtual object. In such an embodiment, the user interaction sensor 453 may sense the interaction or manipulation of a person's hand or an item carried by a person's hand with such a virtual object. For example, a user may manipulate a stylus (or virtual stylus) relative to a virtual object 456-3. Based on such a sensed interaction, a virtual object (text) may be displayed on top of the virtual object 465-3, with the virtual document "written on top" then being stored in the memory of the system 420.

[0046] As described above, the sensing and computing operations of the system 420 are centralized in the main unit 450 rather than the head mounted device 330. In the illustrated example, both sensors in the virtual image generator are illustrated as being part of an accessory contained within a single housing. In other embodiments, the sensors 452-1 and 452-2 may be separate and distinct from the virtual image generator / transmitter 454. For example, the image generator / transmitter 454 may be software running on a personal computer to which the sensor 452 is attached as an accessory. The sensors 452-1 and 452-2 and the virtual image generator / transmitter 454 may be provided in various forms in various locations remote from the head mounted device 330.

[0047] Figure 7 is a perspective view of a portion of an illustrated example mixed reality presentation system 520. Mixed reality presentation system 520 is similar to mixed reality presentation system 420 described above, except that system 520 includes head mounted device 430 in place of head mounted device 330, and virtual image generator / transmitter 454 follows instructions that additionally utilize information provided by head mounted device 430. Those remaining components of system 520 that correspond to components of system 420 are similarly numbered. In the illustrated example, the previous real world physical objects 465-1, 465-2, and 465-3 are replaced instead by corresponding and functionally similar virtual objects 462-1 (virtual keyboard or tablet), 462-2 (virtual computer mouse), and 462-3 (virtual document, book, paper, etc.).

[0048] Head mounted device 430 is similar to head mounted device 330, except that head mounted device 430 additionally includes sensor 480 and transmitter 482. Sensor 480 includes a sensor such as a camera or other sensing device that senses the physical environment, such as the physical environment outside the field of view of sensor 452-1. Figure 7 As shown, because the sensor 480 is carried by the head mounted device 430, the user can adjust its field of view by simply revising his or her field of view. As a result, the sensor 480 can capture real world physical objects 465-4, 465-5 in the field of view 463 of the sensor 480, which extends beyond the field of view 461 of the sensor 452-1. Such images of the real world objects 465-4 and 465-5 can be transmitted by the transmitter 482 to the virtual image generator and the transmitter 454. In some embodiments, the transmitter 482 can also be used as a receiver (in the form of a transceiver) that also receives the virtual image generated by the main unit 450.

[0049] In one embodiment, the main unit 450 uses information sensed by the sensor 480 of the head mounted device 430 (in the form of signals transmitted to the main unit 450) to track physical objects outside the field of view 461 of the sensor 452-2 of the main unit 450. In some embodiments, the resolution and / or depth sensing capabilities of the sensor 480 provided in the head mounted device 430 are much lower than the resolution and / or depth sensing capabilities of the sensor employed by the main unit to sense the surrounding physical environment, wherein the image resolution is sufficient to identify, distinguish and track real world objects when the real world objects leave the field of view 461. In some embodiments, the image provided by the sensor 480 can provide a second viewpoint to the virtual image generator / transmitter 454, which captures the user-facing surface that is not visible to the sensor 452-2. In some embodiments, the signals received from the sensor 480 of the head mounted device 430 are not used directly by the main unit 450 to generate virtual images, but are only used in large part to track physical objects outside the field of view of the main unit 450.

[0050] Figure 8 6 is a flow chart of an example mixed reality presentation method 600, including actions taken by the head mounted device 430 (enclosed in the dashed box) compared to actions taken by the main unit 450 of the system 520. As indicated by box 602, the virtual image generator / transmitter 454 uses the depth camera 452-2 in the head of the desktop accessory or main unit 450 to build a spatial map of the desktop. The spatial map can be a 3D computer representation of the desktop and the objects on it.

[0051] As indicated by box 604, the virtual image generator / transmitter 454 further identifies an "anchored" feature on the desktop using the camera 452-2. Such identification may involve the location of the horizontal plane of the desktop surface and the identification of different documents and objects on the desktop surface. As indicated by box 608, and in some embodiments, where a head-mounted device 430 is used, images captured from the head-mounted device 430 can be received by the main unit 450, where the images are used to track or identify features or objects on the desktop that may otherwise be outside the field of view of the camera 452-2. As indicated by box 606, the virtual image generator / transmitter 454 further positions the virtual content relative to such an anchored feature. In other words, as described above, the virtual image generator session transmitter 454 attaches virtual content (such as a virtual object) to such an anchored feature or a real-world physical object.

[0052] As indicated by block 610, the virtual image generator / transmitter 454 further tracks the position and / or orientation of the glasses / head mounted device 430 (user viewpoint) based on the signal from the sensor 452-1. As indicated by block 612, the virtual image generator / transmitter 454 renders the virtual content from the user viewpoint and transmits the virtual image to the head mounted device 430 (also referred to as video overlay glasses). The virtual content or virtual objects are positioned and / or oriented based on any virtual attachment relative to an anchoring feature or real world physical object.

[0053] As indicated by block 616, display controller 374 of head mounted device 430 displays the virtual image received from main unit 450. As indicated by block 618, and in some operating modes, inertial motion unit 370 may sample data regarding the motion of head mounted device 430. As indicated by block 620, in such optional embodiments or operating modes, display controller 374 may additionally shift the virtual image, or its virtual content, based on the most recent, possibly more current, changes in the head mounted device position and orientation as detected by inertial motion unit 370.

[0054] Although the present disclosure has been described with reference to example embodiments, it will be appreciated by those skilled in the art that changes may be made in form and detail without departing from the scope of the subject matter claimed. For example, although different example embodiments may have been described as including features that provide one or more benefits, it is envisioned that the described features may be interchangeable or alternatively combined with each other in the described example embodiments or in other alternative embodiments. Because the technology of the present disclosure is relatively complex, not all changes in the technology are foreseeable. The present disclosure described with reference to example embodiments and set forth in the following claims is clearly intended to be as broad as possible. For example, unless specifically otherwise indicated, a claim recording a single specific element also covers a plurality of such specific elements. The terms "first", "second", "third", etc. in the claims only distinguish different elements, and unless otherwise indicated, are not specifically associated with a specific order or specific number of elements in the present disclosure.

Claims

1. A mixed reality presentation system, comprising: A head mounted device for presenting a virtual image to a person wearing the head mounted device; Main unit, including: Sensors for sensing the surrounding physical environment and the position and orientation of the head-mounted device; a virtual image generator for generating a virtual image based on the sensed surrounding physical environment and the sensed position and orientation of the head mounted device; and a transmitter for transmitting the generated virtual image to a head mounted device for presentation, wherein the main unit is in a single cabinet housing, and The head mounted device comprises: Inertial motion sensors for sensing movement of the head-mounted device; and A display controller is configured to modify a virtual image received from the main unit based on signals from the inertial motion sensor.

2. The system of claim 1, wherein the modification comprises a two-dimensional shift of the virtual image.

3. A system according to claim 1, wherein the sensor of the main unit has a field of view, and wherein the head-mounted device further includes a sensor for sensing the physical environment outside the field of view and transmitting a signal to the main unit for the main unit to use to track physical objects outside the field of view.

4. The system of claim 1, wherein the sensor of the main unit is supported by a display having a vertical surface.

5. The system according to claim 1, wherein the virtual image generator is to virtually attach the generated virtual image to an object of the sensed surrounding physical environment, so that the generated virtual image attached to the object is repositioned based on the sensed movement of the object.

6. The system of claim 1, wherein the sensor of the main unit comprises: Optical tracking sensors for locating and tracking the movement of the headset; as well as Image and depth cameras for sensing the surrounding environment.

7. The system of claim 6, wherein the sensor of the main unit further comprises a user interaction sensor for detecting a user touch in the surrounding physical environment.

8. The system of claim 1, wherein the main unit is part of a fixed desktop mounted display and sensor system.

9. A mixed reality presentation method, comprising: Use a fixed master unit to build a spatial map of the surrounding physical environment; Tracking the position and orientation of the head mounted device with a fixed master unit; generating a virtual image based on the spatial map and the position and orientation of the head mounted device tracked by the fixed master unit; transmitting a virtual image from the fixed master unit to the head mounted device for presentation by the head mounted device; sensing movement of the head mounted device using an inertial motion sensor of the head mounted device; as well as modifying, with a display controller of the head mounted device, a virtual image received from the main unit based on a signal from the inertial motion sensor, Wherein the fixed main unit is in a single box housing.

10. The method of claim 9, wherein the stationary main unit is a desktop accessory, and wherein the surrounding physical environment comprises a desktop.

11. The method of claim 9, wherein the virtual image generated by the main unit and transmitted to the head mounted device omits any image data from the head mounted device.

12. A mixed reality presentation system comprising a non-transitory computer readable medium containing instructions, the instructions directing a processor to perform the following operations: Use a fixed master unit to build a spatial map of the surrounding physical environment; Tracking the position and orientation of the head mounted device with a fixed master unit; generating a virtual image based on the spatial map and the position and orientation of the head mounted device tracked by the fixed master unit; transmitting a virtual image from the fixed master unit to the head mounted device for presentation by the head mounted device; sensing movement of the head mounted device using an inertial motion sensor of the head mounted device; as well as modifying, with a display controller of the head mounted device, a virtual image received from the main unit based on a signal from the inertial motion sensor, Wherein the fixed main unit is in a single box housing.

13. The mixed reality presentation system of claim 12, further comprising a fixed desktop accessory containing a non-transitory computer readable medium.

Images