Reality Mixer for Mixed Reality
Through the combination of MR mixer and graphics processor, the problem of adjusting aesthetics and behavioral parameters of real and virtual objects in the prior art is solved, personalized and immersive improvement of user experience, and a richer interaction method is provided.
Patent Information
- Application Number
- CN202210617820.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2016-10-21
- Filing Date
- 2017-02-10
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2037-02-10
AI Technical Summary
Existing mixed reality (MR) technologies are difficult to effectively adjust the aesthetic and behavioral parameters of real and virtual objects, resulting in insufficient immersion and personalization of the user experience.
Through the MR mixer, the input system is used to receive parameter values, combined with the graphics processor and display, the virtual image content is rendered to change the perceived reality of the real world environment, and the properties of the virtual object are adjusted through object recognition and audio mixing components to achieve reality and adjustment of dynamic aesthetics and behavioral parameters of virtual objects.
It realizes personalized adjustments to objects in a mixed reality environment, enhances the immersion and personalization of the user experience, and can automatically adjust the attributes of virtual and real objects according to user intentions, providing a richer interaction method.
Smart Images

Figure CN115016642B_ABST
Abstract
Description
[0001] This invention patent application is a divisional application of the invention patent application with international application number PCT / US2017 / 017278, international application date February 10, 2017, application number 201780004780.9 entering the Chinese national phase, and name “Reality mixer for mixed reality”. Technical Field
[0002] The present disclosure relates to a reality mixer for mixed reality. Background Art
[0003] Some display devices provide a mixed reality (MR) experience, in which real objects in the user's field of view are combined with computerized virtual imagery. Such devices can overlay informational text on real-world scenes, or, for example, use virtual content to augment the user's view of the world. Summary of the invention
[0004] One embodiment relates to a mixed reality display device including an input system, a display, and a graphics processor. The input system is configured to receive a parameter value, which is one of a plurality of values within a predetermined range that can be received by the input system. The display is configured to display virtual image content that adds an augmentation to a real world environment viewed by a user of the mixed reality display device. The graphics processor is operatively coupled to the input system and the display; it is configured to render the virtual image content to variably change the augmentation, variably changing the perceived realism of the real world environment in relation to the parameter value.
[0005] This summary is provided to introduce in simplified form some concepts that will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that address any or all of the disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] Figure 1 Aspects of an example near-eye display device and an off-board computer are shown.
[0007] Figure 2 A near-eye display device user is shown viewing a foreground object augmented by virtual image content.
[0008] Figure 3 Aspects of an example display window for a near-eye display device are shown.
[0009] Figure 4 and 5 Various aspects related to stereoscopic display of virtual image content are explained.
[0010] Figure 6 Various non-near-eye display devices are shown participating in a mixed reality (MR) demonstration.
[0011] Figure 7 The variation of two different fantasy parameters across a series of MR demonstrations is explained.
[0012] Figure 8 The selection of foreground objects to which MR blending is applied is illustrated.
[0013] Fig. 9 Parameter adjustment of selected foreground objects using MR blending is illustrated. DETAILED DESCRIPTION
[0014] The present disclosure relates to methods and configurations for manipulating presentation attributes of objects viewed in a mixed reality (MR) environment. In this specification, an 'MR environment' is an environment in which real-world images are visually augmented or otherwise modified by one or more virtual reality (VR) objects or components. The modified MR images are provided to the user via a head-mounted device or other display device. In addition to virtual images, MR also includes a mix of other presentation attributes, such as audio and tactile attributes. Accordingly, virtual audio and tactile content can be added, subtracted, or otherwise used to transform the sound and / or feel of the real world, which is then delivered to the user in a modified form. For example, a suitably configured display device can add a video overlay (e.g., a virtual hat) to an object or person seen in a real video feed. Sounds and / or feelings can also be added to the object or person. Some MR environments provide persistent, reasonable positioning of virtual objects and images relative to the real world. For example, a virtual plant arrangement placed on a real table can be continuously rendered as placed across multiple MR sessions.
[0015] In one embodiment, the hardware used to deliver a composite MR experience to a user may include a VR head-mounted device with a world-facing visual system and a microphone for capturing real-world images and audio. The image ultimately provided to the user is a composite image that combines a real-world video feed with one or more synthetic video overlays. Less immersive MR experiences can be achieved with other display devices having, for example, a world-facing visual system. In some examples, the display and visual system may be installed in an environment, such as a home, business, or vehicle. So-called 'augmented reality' (AR) head-mounted devices may also be used, in which at least a portion of the outside world can be directly viewed through the head-mounted device.
[0016] At the heart of the disclosed approach is the understanding that various virtualized aspects of the imagery (real or virtual) viewed by the user can be parameterized in various ways. Specifically, each object in an MR experience can have an independent set of parameters associated with it. Some parameters can control the virtual aesthetics of an object: shape, size, texture, transparency, blur, or presentation style—for example, from realistic to fantastical. Other parameters can control the behavior of virtual objects: for example, mass and associated kinematics, a person's physical strength or jumping ability, and sound properties such as loudness and pitch. Still other parameters can control the temporal evolution of a virtual object: is the object to be represented as it will look now, or as it will look at some time in the past, or as it may look at some time in the future? Because such attributes can be individually parameterized for different objects in an MR experience, the experience as a whole can be altered or adjusted by independently changing the associated parameter values of one or more objects.
[0017] The primary mechanism for changing parameter values that govern object properties is the MR mixer aspect of the present disclosure. The MR mixer provides a mechanism for adjusting various presentation properties of an object through the underlying parameter values associated with the object. Object parameter values can be changed individually or collectively. In this way, a user can adjust the mix of realism and fantasy in a MR experience in a similar way to mixing different audio tracks in a recording studio.
[0018] The user can access the MR mixer of the present invention via any suitable user interface (UI) modality. In one embodiment, the world-facing visual system includes both a depth imaging camera and a planar imaging camera. The visual system can be used to construct the world surface of objects in the user's field of view (FOV) in real time. An object recognition engine downstream of the visual system identifies various real-world objects, making them available for selection. In one example, the user directs his or her gaze to the object whose presentation properties are to be adjusted. Using a finger tap on the frame of the MR head-mounted device, the user initiates the selection of the object being gazed at. The user then speaks an audible command 'MR mixer', which brings up the MR mixer UI control for the selected object. In some instances, the selected object can be identified before the MR mixer UI control is displayed. Accordingly, the control can intelligently provide the ability to adjust only those properties belonging to the selected object.
[0019] As described above, the disclosed MR mixer may call a method for identifying real-world objects that may be included and manipulated within an MR experience. Object recognition may be used to identify people, objects, and environments that may be adjusted and to what level. Object recognition may utilize significant paint or labels, specific patterns, materials, forms, and the like. Real-world clothing, for example, may be bundled with a series of virtual instantiations. Real-world shoes, gloves, hats, rings, or any accessories may be associated with additional properties in the virtual world—for example, the ability to grow plants, burn things, draw world-to-world portals, or enable flight. Real-world Internet of Things (IoT) hardware—audio speakers, lights, appliances, controllers such as thermostats, and the like—may have more enhanced forms in virtual space.
[0020] The MR mixer concept goes well beyond the simple concept of video, audio, and haptic mixing. It extends to the dynamic adjustment of complex aesthetic and behavioral parameters of real and virtual objects in the MR environment. Complex aesthetic and behavioral parameters may include, for example, time and place. In one example, the time of day on the selected space can be changed by providing simulated lighting and weather and possibly modifying the structure and terrain. In addition, the immersive nature of the audio experience can be changed so that specific sounds can be listened to and turned off - for example, the sound of the entire experience is only one audio element (such as wind). Another example is to mix audio from different applications (e.g., network audio sources) as various tracks to the selected MR space. Assuming that virtual components - human avatars, virtual objects, etc. - are built in a progressive manner (i.e., 3D modeling, texturing, and lighting), the MR mixer can provide users with the ability to view the construction process as a time delay. This aspect becomes clearer when visualizing the changes made by multiple users (who can then experience the evolution of the space) in a common MR space. In one particular example, a user can image a map-like view of all known digital spaces accessible to them, with data about the terrain being traveled overlaid on top of it. In this way, the user can not only understand and remember past experiences, but also explore unseen but available worlds. The 'time' component of the mixer also applies to this example.
[0021] The MR mixer concepts disclosed herein also enable aiming and adjusting aesthetic and behavioral parameters of real and virtual objects moving within an MR environment. This capability leverages the core interaction model of deforming objects in motion, or aiming at objects while the user is in motion (e.g., in a car). For example, a user can aim and change a bird from a flock of birds in mid-flight. Once the change is applied, the entire flock of birds inherits the new properties. In other examples, a user can change the color, viscosity, virtual mass, or speed of a flowing liquid in real time.
[0022] In some embodiments, the MR mixer may automatically adjust the aesthetic and behavioral parameters of real and virtual objects based on explicit or implicit user signals. For example, certain objects may be brought into focus (visually, acoustically, tactilely) based on the user's perceived intent. A user may, for example, gaze across the room at a group of people chatting. Based on this clue, the mixer may allow the curious user to peek into a closed or distant room or 'world' to get a preview of the group's experience before joining the group. The explored world may be filtered based on the user's interests. For example, a social networking application that utilizes a social graph may be overlaid on an entire virtual world so that the user can see only those people who are in the user's social graph or share one or more attributes. Similarly, a virtual world may be filtered by a photo sharing application based on an image; the MR mixer may be configured to select more content from the world that 'looks like' the image.
[0023] In some embodiments, the MR mixer can be configured to automatically modify relevant aspects between real and virtual objects based on dynamic adjustments to aesthetic and behavioral parameters. In this way, the mixer can reconcile conflicts between two or more experiences that include elements of artificial intelligence (AI) or even natural self-navigation elements. For example, the MR mixer can enable a character from one video game to dock with a character from another video game.
[0024] Various aspects of the present disclosure will now be described by way of example and with reference to the embodiments listed above shown. In one or more embodiments, substantially identical components, process steps and other elements are identified in a coordinated manner and described in a minimally repetitive manner. However, it will be noted that each element identified equally may also be different to a certain extent. It will be further noted that the accompanying drawings included in the present disclosure are schematic and are usually not drawn to scale. Of course, the various drawing scales, aspect ratios, and the number of components shown in the accompanying drawings may be intentionally distorted to more easily see certain features or relationships.
[0025] Figure 1 Aspects of a near-eye display device 10 in an example embodiment are shown. The near-eye display device 10 is a binocular near-eye display device having see-through display windows 12R and 12L, on-ear speakers 14R and 14L, and a frame 16 configured to rest on the user's head. In other embodiments, the right and left display windows can be completely opaque for a fully immersive display experience. In some embodiments, the display windows 12 can be dynamically controlled via a darkening filter. The substantially see-through display windows can be switched to be completely opaque for a fully immersive MR experience, for example.
[0026] The near-eye display device 10 includes a right microdisplay 16R and a left microdisplay 16L. The right and left microdisplays are configured to project computerized virtual display images onto the right and left display windows 12R and 12L, respectively. The microdisplays are driven by a graphics processor 18 of an onboard computer 20. The onboard computer may be operatively coupled to various active and sensing components of the near-eye display device, including a communication system 22. In some embodiments, the graphics processor is configured to render a virtual display image projected onto the display window. In some embodiments, such rendering is controlled in whole or in part by data received through an onboard communication system. In some embodiments, such data may be transmitted by an off-board computer 20'. For example, the off-board computer may be a game console, a desktop computer, or a networked server system. In other embodiments, the off-board computer may be a laptop or tablet computer, a smart phone, a handheld gaming device, etc. The data for displaying or controlling the rendering of the display image may be transmitted from the off-board computer 20' to the onboard computer 20 in any manner. Signals encoded with such data may be carried on a wired or wireless communication link between the communication systems 22 of the respective computers.
[0027] When the near-eye display device 10 is in operation, the user of the display device views the right and left display images through his right eye and left eye, respectively. When the right and left display images are composed and presented in an appropriate manner, the user experiences the phantom of one or more virtual objects at a specified location and with specified 3D content and other display attributes. Such a virtual image can have any desired complexity; for example, it can include a complete virtual scene with both foreground and background portions.
[0028] Figure 1 The near-eye display device 10 includes an input system 24 configured to receive a parameter value. The parameter value can be received from a user of the near-eye display device or from an off-board computer 20'. The received parameter value can be one of a plurality of values in a predetermined (in some cases, numerical) range that can be received by the input system. Example ranges include negative five to positive five, zero to two hundred and fifty-six, zero to one hundred percent, and so on. In some embodiments, the parameter value can be one of three or more available values within a predetermined range. The range of values can be discrete (where the available parameter values are separated by large increments relative to the size of the range) or quasi-continuous (where the available parameter values are separated by small increments relative to the size of the range).
[0029] The input system 24 may be different in different embodiments of the present disclosure. In some embodiments, the input system may include a mechanical dial on the near-eye display device 10, or on a smart phone, game controller, or other device. For example, a user may increase a parameter value by rotating the dial clockwise, or decrease a parameter value by rotating the dial counterclockwise.
[0030] In some embodiments, user selection of parameter values is facilitated via one or more UI elements presented on near-eye display device 10 or on another display device. The one or more UI elements may be Figure 1 The microdisplay 16 in the embodiment of the present invention may be displayed, or presented by any other UI component of the device. In some scenarios, the UI control may appear only after the user invokes the MR blender, and disappear after the MR blending operation is completed.
[0031] In some embodiments, the input system 24 may include a touch sensor 24A disposed on the frame 16 or elsewhere of the near-eye display device 10. A user may slide or tap a finger on the touch sensor to enable input of available parameter values. In some examples, the touch sensor may emulate a touch-based virtual dial instantiated according to the application context. A user may increase a parameter value by tracing a clockwise path on the touch sensor, or decrease a parameter value by tracing a counterclockwise path.
[0032] In some embodiments, the input system 24 may include a camera operatively coupled to a downstream gesture recognition component 26. A world-facing depth imaging camera may, for example, be configured to capture a depth image of the environment in front of the user. A time-resolved sequence of depth images revealing the user's gestures may be fed to the gesture recognition component, which may then calculate parameter values. Alternatively or in addition, the input system may include a gaze tracking camera facing the user. The gaze tracking camera may be operatively coupled to a downstream gaze tracking engine 28. In some embodiments, gesture and / or gaze detection is used in conjunction with one or more UI elements displayed on the near-eye display device 10. Parameter input may require the user to aim at one UI element to increase a given parameter value, or aim at another UI element to decrease the parameter value. Alternatively, the user may speak the words 'increase' or 'decrease' to effect a corresponding change in the parameter value. To this end, the input system 24 may include a microphone operatively coupled to a downstream speech recognition component 30. In some embodiments, variable parameter values may be received via direct audio input from the user.
[0033] The input system 24 in the form of a world-facing depth imaging camera may be configured to image any or all aspects of the user's environment, including one or more real objects. The depth images from the world-facing depth imaging camera may be provided to the graphics processor 18 for the purpose of virtually reconstructing the environment. The term 'camera' refers herein to any machine vision component having at least one optical aperture and a sensor array configured to image a scene or subject. The depth imaging camera may be configured to acquire a time-resolved sequence of depth maps of a scene or subject. As used herein, the term 'depth map' refers to a corresponding area (X) of the imaged scene. i ,Y i) is a pixel array registered with the depth value Z i Indicates the depth of the corresponding area for each pixel. 'Depth' is defined as a coordinate parallel to the optical axis of the camera that increases with increasing distance from the camera. Operationally, a depth imaging camera may be configured to acquire 2D image data from which a depth map is obtained via downstream processing. The term 'depth video' herein refers to a time-resolved sequence of depth maps.
[0034] The configuration of the depth imaging camera may be different from one embodiment to another. In one example, brightness or color data from two stereoscopically oriented imaging arrays in a depth imaging camera may be co-registered and used to construct a depth map. More generally, one or more planar imaging cameras may be used to obtain depth coordinates by co-registration of the imaged features based on optical tomography. Hyperspectral planar imaging may be used with this scheme for improved feature differentiation. In other examples, an illumination source associated with a depth imaging camera may be configured to project a structured infrared illumination pattern including a plurality of discrete features (e.g., lines or dots) onto a subject. The imaging array in the depth imaging camera may be configured to image the structured illumination reflected back from the subject. Based on the spacing between adjacent features in various regions of the imaged subject, a depth map of the subject may be constructed. In yet other examples, the illumination source may project pulsed or otherwise modulated infrared (IR) or near-infrared illumination toward the subject. The sensor elements of the imaging array may be addressed to resolve the phase shift from each sensor element relative to the modulation of the illumination source. The phase shift (optionally converted to the depth domain) can be associated with the portion of the image corresponding to the addressed sensor element. In some embodiments, a rapid sequence of IR acquisitions can be used to obtain the phase shift. In other embodiments, a pair of imaging arrays in a depth imaging camera can be configured to detect pulsed illumination reflected back from a subject. Both arrays can include electronic shutters synchronized with the pulsed illumination, but the integration times of the two arrays can be different, so that the pixel-resolved time of flight (ToF) of the pulsed illumination from the illumination source to the subject and then to the two arrays can be distinguished based on the relative amount of light received in the corresponding elements of the two arrays.
[0035] In some embodiments, different planar imaging cameras and depth imaging cameras may be arranged to have parallel optical axes oriented in the same direction. In some embodiments, the image or video output from the planar imaging camera and the depth imaging camera may be co-registered and combined into a single (e.g., RGB+depth) data structure or stream. In an example where the depth imaging camera is a suitably configured ToF depth imaging camera, a data stream representing both depth and brightness (e.g., IR+depth) may be available by combining outputs that differ in phase.
[0036] continue Figure 1 In some embodiments, the input system 24 may be further configured to use gesture recognition to enable selection of an object from a plurality of objects viewed by the user. In addition, the input system may be configured to apply object recognition to the selected object to identify what kind of object it is. Using an object recognition algorithm, the object recognition engine 32 may be configured to compare the object parsed by the input system to a plurality of objects stored in a database or heuristically defined, and identify a match. For some objects and scenes, the object recognition engine may identify a general match (this object is a person). For other objects and scenes, the object recognition engine may identify a specific match (this object is my mother).
[0037] exist Figure 1 In an embodiment of the present invention, the near-eye display device 10 includes a position sensor 34 configured to sense the position and orientation of the near-eye display device relative to an object in the environment or some other reference point. The position sensor may include an inertial measurement unit (IMU, including one or more accelerometers, gyroscopes, and magnetometers) and / or a global positioning system (GPS) receiver. In some embodiments, the position sensor returns a six-degree-of-freedom (6DOF) estimate of three Cartesian coordinates of the near-eye display device, plus a rotation about each of the three Cartesian axes. The output of the position sensor can be used to map the position, size, and orientation of a (globally defined) virtual display object to the right and left display windows 12 of the near-eye display device.
[0038] Now turn to Figure 2 , each microdisplay 16 of the near-eye display device 10 may be configured to project virtual image content 36 at a predetermined location of an object 38 visible to a user 40, or along the line of sight of the object. The virtual image content may be configured to add augmentations to the real-world environment viewed by the user of the near-eye display device. In a see-through display embodiment, the virtual image content may be used to augment the real image of the object seen through the display window 12. In embodiments where the near-eye display device is fully immersive, the virtual image content may form the entire image of the object, along with any augmentations of the object that may be desired.
[0039] To these ends, the graphics processor 18 of the near-eye display device 10 may be configured to render virtual image content to variably vary the augmentation in a manner that variably varies the perceived reality of the real world environment. In other words, the virtual image content may impart a variable unrealistic quality to one or more objects viewed via the near-eye display device. In a perspective embodiment in which various real objects in the user's environment are viewed through a transparent display window 12, each microdisplay may be configured to superimpose this virtual image content on the object to impart a variable unrealistic quality. In a fully immersive embodiment in which the input system includes a world-facing camera configured to capture a primary image of the real environment, the primary image may be provided to the graphics processor, which then registers the virtual image content with the corresponding real object. Here, each microdisplay 16 is configured to integrally project a secondary image of the object so that the secondary image together with the virtual image content is viewable by the user. Because the graphics processor 18 is operatively coupled to the input system 24, the variable unrealistic quality (in each of the above embodiments) may be varied in relation to a parameter value received via the input system.
[0040] Figure 3 Aspects of the right or left display window 12 in a non-limiting embodiment are shown. The display window includes a backlight 42 and a liquid crystal display (LCD) matrix 46. The backlight may include a whole of light emitting diodes (LEDs) - for example, a white LED or a certain distribution of red, green and blue LEDs. The backlight may be placed so as to guide its emitted light through an LCD matrix, which forms a display image based on a control signal from a graphics processor 18. The LCD matrix may include numerous individually addressable pixels arranged on a rectangular grid or other geometric shapes. In some embodiments, pixels emitting red light may be juxtaposed with pixels emitting green and blue light in a matrix so that the LCD matrix forms a color image. In one embodiment, the LCD matrix may be a liquid crystal on silicon (LCOS) matrix. In other embodiments, a digital micromirror array may be used to replace the LCD matrix, or an active LED matrix may be used instead. In yet other embodiments, a scanning beam technique may be used to form right and left display images.
[0041] continue Figure 3 , the display window 12 also includes a gaze tracking engine configured to sense the pupil position of the right eye or left eye 48 of the user of the near-eye display device 10. Figure 3 In an embodiment of the invention, the gaze tracking engine takes the form of an imaging system facing the user, wherein an eye imaging camera 50 is arranged to image the user's eyes. Figure 3In the embodiment of the present invention, each display window 12 includes an on-axis illumination source 52 and an off-axis illumination source 54. Each illumination source emits infrared (IR) or near infrared (NIR) illumination in the wavelength band of high sensitivity of the eye imaging camera. The gaze tracking engine 28 of the onboard computer 20 is configured to track the user's gaze axis 56 using the output from the eye imaging camera.
[0042] With respect to gaze tracking, on-axis and off-axis lighting serve different purposes. Figure 3 As shown, off-axis illumination can create a specular glint 58 that is reflected from the cornea 62 of the user's eye. Off-axis illumination can also be used to illuminate the eye for a 'dark pupil' effect, where the pupil 60 appears darker than the surrounding iris 62. Conversely, on-axis illumination from an IR or NIR source can be used to create a 'bright pupil' effect, where the pupil appears brighter than the surrounding iris. More specifically, the IR or NIR illumination from the on-axis illumination source 52 illuminates the retroreflective tissue of the retina 64 of the eye, which reflects light back through the pupil, forming a bright image 66.
[0043] As described above, image data from the eye imaging camera 50 is communicated to the gaze tracking engine 28 of the onboard computer 20. There, the image data can be processed to resolve features such as the pupil center, the pupil outline, and / or one or more specular glints from the cornea. The locations of these features in the image data can be used as input parameters in a model (e.g., a polynomial model) that relates the feature locations to the gaze axis 56. The gaze vector can be used to target objects in the user's FOV, to actuate gaze-based UI controls, etc. In embodiments in which the gaze axes for the right and left eyes are determined, the gaze tracking engine can also be configured to calculate the user's focus as the intersection of the right and left gaze axes.
[0044] In most cases, the display image from the LCD matrix 46 will not be suitable for direct viewing by the user of the display system 10. Specifically, the display image may be offset from the user's eye, may have an undesirable vergence, and / or a very small exit pupil. By passing the display image through the horizontal and vertical pupil expanders 68, the display image is presented over an area that substantially covers the eye. This enables the user to see the display image over an appropriate range of horizontal and vertical offsets between the microdisplay and the eye. Figure 3 Also shown is an example placement of an input system 24B in the form of a world-facing depth imaging camera (see above).
[0045] The user's perception of the distance to the virtual display image is affected by the positional parallax between the right and left display images. This principle is achieved by Figure 4 This is illustrated by the example in . Figure 4Right and left image frames 70R and 70L are shown superimposed on each other for illustration purposes. The right and left image frames correspond to the image forming areas of the LCD matrix 46 of the right and left display windows, respectively. Thus, the right image frame surrounds the right display image 72R, while the left image frame surrounds the left display image 72L. If properly rendered, the right and left display images may appear as virtual images to the user. Figure 4 In the example of , the virtual image presents the surface of the individual renderable sites visible to the user. Now refer to Figure 5 , each site i in the visual surface has a corresponding pixel (X i ,Y i ) is associated with the depth coordinate Z i The required depth coordinate can be simulated in the following way.
[0046] Initially, a distance Z0 from the focal plane F of the near-eye display device 10 is selected. The left and right microdisplays are then configured to present their respective display images with a vergence appropriate for the selected distance. In one embodiment, Z0 may be set to "infinity" so that each microdisplay 16 presents a display image in the form of collimated light. In another embodiment, Z0 may be set to two meters, requiring each microdisplay to present a display image in the form of divergent light. In some embodiments, Z0 may be selected at design time and remain constant for all virtual images presented by the display device. In other embodiments, the microdisplay may be configured with electronically adjustable optical power to allow Z0 to vary dynamically depending on the range of distances at which the virtual image is presented.
[0047] When the distance Z0 from the focal plane has been established, a depth coordinate Z can be set for each location i on the visible surface. This is done by adjusting the positional disparity (relative to their respective image frames) of the two pixels corresponding to location i in the right and left display images. Figure 5 The pixel corresponding to position i in the right image frame is denoted as R i , and the corresponding pixel in the left image frame is denoted as L i .exist Figure 5 The position parallax is positive—that is, R is within L in the superimposed image frame. i This causes site i to appear behind the focal plane F. If the position parallax is negative, the site will appear in front of the focal plane. Finally, if the right and left display images are overlapped (no parallax, R i and L i coincide), then the location will appear to be exactly on the focal plane. Without limiting the present disclosure to any particular theory, the position parallax D can be related to Z, Z0, and the user's interpupillary distance (IPD) as follows:
[0048] D = IPD × (1-Z0 / Z).
[0049] In the above scheme, the positional parallax attempted to be introduced between corresponding locations of the right and left displayed images is "horizontal parallax" - that is, parallax parallel to the interpupillary axis of the user of display device 10. Horizontal parallax mimics the effect of real object depth on the human visual system, where the images of real objects received by the right and left eyes are naturally offset along the interpupillary axis.
[0050] Virtual images of any desired complexity can be rendered in this manner. In one embodiment, logic in the onboard computer 20 or offboard computer 20' maintains a model of the Cartesian space in front of the user in a reference frame fixed to the near-eye display device 10. The user's pupil position is mapped into this space, as are the image frames 44R and 44L positioned at a predetermined depth Z0. Then, by having each site i of the visible surface in the image have coordinates X in the common reference frame, the user's pupil position is mapped into this space, as are the image frames 44R and 44L positioned at a predetermined depth Z0. i ,,Y i , and Z i , construct a virtual image 74. For each position on the visible surface, construct two line segments - a first line segment to the pupil position of the user's right eye and a second line segment to the pupil position of the user's left eye. The pixel R corresponding to the position i in the right display image i is taken as the intersection point of the first line segment in the right image frame 44R. Similarly, the pixel L in the left display image i is taken to be the intersection of the second line segment in the left image frame 44L. This process automatically provides the appropriate amount of shifting and scaling to correctly render the visible surface, placing each location i at the desired distance from the user.
[0051] Any aspect of the foregoing description or drawings should not be interpreted in a limiting sense, as numerous variations are within the spirit and scope of the present disclosure. Figure 1 A near-eye display system 10 is shown in which a right display image appears behind a right display window and a left display image appears behind a left display window, although the right and left display images may also be formed on the same window. In a stereoscopic display device for use in, for example, a laptop computer or a home theater system, the right display image may be formed on a display screen using light of one polarization state, while the left display image may be formed on the same display screen using light of a different polarization state. Orthogonally aligned polarizing filters in the user's glasses may be used to ensure that the appropriate display image is received into the appropriate eye.
[0052] In some examples, a non-see-through display device or a near-eye display device 10 is used. Figure 6The non-near-eye display device shown in can provide confident MR. World-facing input systems can be implemented as cameras mounted on the back of smartphones, tablets, and laptops. These input systems can provide a realistic video feed from the approximate vantage point of the device user. The realistic video feed can then be combined with (or otherwise modified by) the virtual display content to generate a composite video feed. As described above, the position sensing component of the display-enabled device can be used to facilitate the mapping of virtual display content to local display coordinates. The composite video feed is then presented on the display in real time.
[0053] More specifically, Figure 6 Aspects of an example MR environment 76 are shown with three different non-wearable display devices configured as described above. Figure 6 , the environment is an indoor living space in which a single user 40 is immersed in an MR experience. In this case, the imagery includes a fantasy virtual reality (VR) display image 78, which can be seen on each of the display devices. The devices shown in this figure include a laptop computer 80, a tablet computer 82, and a television monitor 84. Each device can be enabled with a camera mounted on the back to capture real-life images behind the device.
[0054] Back to briefly reference Figure 1 , the near-eye display device 10 includes an audio mixing component 86 operatively coupled to the speakers 14R and 14L. The speakers can be configured to emit audio that appears to originate from a predetermined location of an object to be augmented. For example, the audio from each speaker can become louder as the speaker approaches the location, or can not be turned on at all until the object approaches within a threshold distance. Operatively coupled to the input system 24 and the speakers, the audio mixing component can be configured to synthesize an audio overlay to give a variable unrealistic quality to the real object. Likewise, the variable unrealistic quality can vary in relation to a parameter value.
[0055] Various parameters may be selectively adjusted to affect the balance between realism and augmentation. In some embodiments, the variable unreality quality controlled by the MR mixer may include one or more of the color, texture, lighting, shadows, opacity, blur, and pixelation of the object. In some embodiments, the variable unreality quality may include the manner of aesthetic presentation of the object. Here, the parameter values may range from realistic aesthetic presentation to fantasy aesthetic presentation.
[0056] exist Figure 7In the example of FIG, two different fantasy parameter values can be varied across the MR mixer presentation sequence. From left to right across the figure, the value of the background fantasy parameter B increases. From top to bottom in the figure, the value of the foreground fantasy parameter F increases. Thus, the upper left view box shows the user's unaltered view of the real world environment including the person holding a cell phone; and the lower right view box shows a highly fantasy augmentation of the real world environment. Figure 7 In the example of FIG. 1 , each fantasy parameter has eleven possible values, where zero represents an unmodified view and ten represents a completely fantasy view. As the foreground fantasy parameter F is adjusted upward from zero toward ten, the person holding the cell phone is incrementally covered with blueberries. As the background fantasy parameter B increases, the environment behind the person becomes increasingly decorated with stylized stars and planets.
[0057] In other embodiments, the variable unreality mass controlled by the MR mixer may include the virtualization mass and associated kinematics of the foreground object. More specifically, the extension may be configured to change the virtualization mass and associated kinematics of objects within the real world environment in response to a changing parameter value (which ranges from a realistic virtualization mass to a non-realistic virtualization mass). In some embodiments, the variable unreality mass may include the temporal evolution of the appearance of a foreground or background object, where the parameter value ranges from a past time to a future time. The extension may be changed so that the appearance of a person holding a cell phone may show, for example, an accelerated aging effect, or return to a younger state. In some embodiments, the variable unreality mass may include the relationship aspects of the foreground and background objects relative to one or more neighboring objects. Thus, the extension may be configured to change the relationship aspects according to changes in parameter values. In embodiments including speakers and audio mixing components, the variable virtual mass may include one or more of the loudness and pitch emitted by the object.
[0058] These and other adjustments may occur via the hardware configurations described above, or by any other suitable hardware configuration using any suitable process. For devices utilizing a world-facing camera and electronic display, the view is unchanged ( Figure 7 ) can be achieved by displaying the image seen by the world-facing camera without significant augmentation. Although shown in the context of adjusting the image of a person in front of a background, any viewed object or set of objects can be adjusted in a similar manner. Furthermore, adjustments to different kinds of parameters can be made together. Adjustments can be made collectively, for example, for fantasy parameters, resolution parameters, and opacity parameters.
[0059] The partially opaque effect can be achieved by estimating the background behind the foreground object seen and by partially transparent rendering covering the estimated background to expand the view of the foreground object. For full transparency, the MR mixer is used to effectively remove the foreground object from the scene. The full transparency effect can be achieved by estimating the background behind the foreground object seen and by fully opaque rendering covering the estimated background to expand the view of the foreground object. When using a perspective display, the fully opaque rendering can be achieved by selectively blocking the scene light so as not to reach the viewer's eyes. Similarly, the reduced resolution effect can be achieved by identifying the foreground object in the image captured by the world-facing camera and covering the pixelated image of only the foreground object in registration with the perspective view of the foreground object. For devices utilizing a world-facing camera and an electronic display, the reduced resolution effect can be achieved by identifying the foreground object in the image captured by the world-facing camera and pixelating the portion of the image corresponding to the identified foreground object.
[0060] Figure 8 and 9 A usage scenario is shown in which one or more parameters can be selectively adjusted to achieve a balance between realism and augmentation. These and other adjustments can be made by any suitable hardware configuration, using any suitable process. Figure 8 A seen scene is shown including a person 38. The figure shows a black dot 88 on the person, which schematically represents a gaze target (or another type of selection target) of the user 40. Figure 8 Also schematically shown is a frame tap gesture 90 performed by the user to select the person being viewed. Depending on the hardware capabilities of the device used to view the scene, a variety of different inputs (e.g., visual recognition gestures, voice, device input (e.g., buttons)) and peripheral device inputs (e.g., smartphone or game controller buttons) can be used to select any foreground object. Any suitable processing approach can be used to distinguish the foreground object from anything else in the scene (e.g., foreground / background analysis using depth information, edge detection, and machine learning recognizers).
[0061] Fig. 9 Parameter adjustment visual 92 is shown overlaid on the scene. The parameter adjustment visual may optionally be composed of a plurality of foreground objects (e.g., Fig. 9 The parameter adjustment visual may include an icon, text, or other indication of what type of adjustment will be made. Fig. 9 In the context of , the parameter adjustment visual includes an eyeball icon 94 for indicating that the opacity of the object being viewed will be changed. The black dot 88 indicates that the user's gaze target is the reduction icon of the parameter adjustment visual. In addition, Fig. 9A symbolic frame sliding gesture 96 performed by a user to reduce the opacity of a selected foreground object is schematically shown. In some embodiments, the user can perform a commit gesture, which is used to complete the adjustment. In a more specific example, the user can look at the ceiling and speak a 'change ceiling' command, thereby selecting the ceiling to adjust. As introduced above, the ceiling or other objects (foreground or background) can be identified using any suitable process. For example, the depth image can be analyzed to find a generally horizontal ceiling surface.
[0062] Other examples may include a theme selection visual overlay over the scene the user is viewing. Such an overlay may allow the viewer to quickly change the appearance of the entire scene between any of several possible theme presets. The theme selection may effectively hide the real background from view and create the illusion that the user is in a completely different environment. The theme presets may allow the viewer to make several changes to the scene at once, as opposed to changing each individual aspect independently.
[0063] In some embodiments, physical parameters or other behavioral parameters may be selectively adjusted. In contrast to the appearance-based parameters described above, behavioral parameters may affect the way the viewed object interacts with other objects and / or forces. For example, if the gravity parameter is lowered in a room, the real pinball in the room may be virtually erased and replaced with a virtual ball that bounces higher.
[0064] Some examples may include a dollhouse scene selection tool. The dollhouse selection tool allows the user to zoom in and / or out of the viewed scene. When performing this type of zooming, a modeled representation of the world may be visually presented to the user at a scale and perspective that may be selectively changed by the viewer (e.g., using gesture input, voice input, or mechanical input). In this way, the user is able to virtually view the world at different scales and from different perspectives without actually moving around the world. The user may then make changes to the visually viewed world, and such changes may be implemented when the viewer then moves (physically or virtually) to view the changed world.
[0065] As is apparent from the foregoing description, the methods and processes described herein may be bound to a computer system of one or more computers. Such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.
[0066] In simplified form Figure 1 1 is a non-limiting example of a computer system that is used to support the methods and processes described herein. Each computer 20 in the computer system includes a logic machine 98 and an instruction storage machine 100. The computer system also includes a display in the form of a microdisplay 16, a communication system 22, and Figure 1Various components not shown.
[0067] Each logic machine 98 includes one or more physical devices configured to execute instructions. For example, a logic machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, implement a technical effect, or otherwise obtain a desired result.
[0068] Each logic machine 98 may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for serial, parallel and / or distributed processing. The various components of the logic machine may optionally be distributed on two or more separate devices, which may be remotely located and / or configured for collaborative processing. Various aspects of the logic machine may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.
[0069] Each instruction storage machine 100 includes one or more physical devices configured to hold instructions executable by the associated logic machine 98 to implement the methods and processes described herein. When implementing such methods and processes, the state of the instruction storage machine may be transformed—for example, to hold different data. The instruction storage machine may include removable and / or built-in devices; it may include optical storage (e.g., CD, DVD, HD-DVD, Blu-ray Disc, etc.), semiconductor storage (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic storage (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), among others. The instruction storage machine may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices.
[0070] It will be understood that each instruction storage machine 100 includes one or more physical devices. However, aspects of the instructions described herein may alternatively be propagated via a communication medium (eg, electromagnetic signals, optical signals, etc.) that is not held for a finite duration by a physical device.
[0071] Aspects of the logic machine(s) and the instruction storage machine(s) may be integrated together in one or more hardware logic components. These hardware logic components may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASICs), program and application specific standard products (PSSP / ASSPs), systems on chips (SOCs), and complex programmable logic devices (CPLDs).
[0072] The terms "module", "program", and "engine" may be used to describe an aspect of a computer system implemented to perform a specific function. In some cases, a module, program, or engine may be instantiated via a logic machine that executes instructions held by an instruction storage machine. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" are intended to cover single or grouped executable files, data files, libraries, drivers, scripts, database records, etc.
[0073] It should be understood that a "service" as used herein is an application program that is executable across multiple user sessions. A service can be used for one or more system components, programs, and / or other services. In some embodiments, a service can run on one or more server computing devices.
[0074] The communication system 22 may be configured to communicatively couple the computer to one or more other machines. The communication system may include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication system may be configured to communicate via a wireless telephone network or a wired or wireless local or wide area network. In some embodiments, the communication system may allow the computer to send messages to other devices and / or receive messages from other devices via a network such as the Internet.
[0075] One aspect of the present disclosure relates to a mixed reality display device including an input system, a display, and a graphics processor. The input system is configured to receive a parameter value, which is one of a plurality of values within a predetermined range that can be received by the input system. The display is configured to display virtual image content that adds an augmentation to a real world environment viewed by a user of the mixed reality display device. The graphics processor is operatively coupled to the input system and the display. The graphics processor is configured to render the virtual image content to variably change the augmentation, variably changing the perceived realism of the real world environment in relation to the parameter value.
[0076] In some implementations, the display device further includes a user interface component configured to present a user interface element on the display device for allowing selection of the parameter value. In some implementations, the input system includes one or more of a camera operatively coupled to a gesture recognition component and a microphone operatively coupled to a voice recognition component. In some implementations, the input system includes a touch sensor. In some implementations, the virtual image content is displayed at a predetermined position of an object that can be viewed by a user of the display device, and the input system is further configured to allow selection of the object from a plurality of objects viewed by the user. In some implementations, the parameter value is one of three or more values within the range. In some implementations, the display is configured to project a secondary image of the object as a whole, and the secondary image can be seen by the user via the display device. In some implementations, the virtual image content is displayed at a predetermined position of an object that can be viewed by a user of the display device. Here, the input system includes a world-facing camera configured to capture a primary image of the object, and the primary image is provided to the graphics processor. In some implementations, the virtual image content is displayed at a predetermined location of an object in the real world environment, and the display is configured to superimpose the virtual image content on the object to give a variable non-realistic quality to the object. In some implementations, the display device also includes a speaker configured to emit audio; and an audio mixing component operatively coupled to the input system and the speaker, the audio mixing component configured to compose an audio overlay to give a variable non-realistic quality to the real world environment, the variable non-realistic quality varying in relation to the parameter value. In some implementations, the augmentation includes one or more of the color, texture, lighting, shadow, opacity, blur, and pixelation of a foreground or background object within the real world environment. In some implementations, the augmentation is configured to change the aesthetic presentation of an object within the real world environment, and the parameter value ranges from a realistic aesthetic presentation to a fantasy aesthetic presentation. In some implementations, the augmentation is configured to change the virtualization quality and associated kinematics of an object within the real world environment, and the parameter value ranges from a realistic virtualization quality to a non-realistic virtualization quality. In some implementations, the augmentation is configured to evolve the appearance of an object within the real-world environment over time, and the parameter value ranges from a past time to a future time. In some implementations, the augmentation is configured to change an aspect of an object's relationship relative to one or more neighboring objects in the real-world environment.
[0077] Another aspect of the present disclosure relates to a fully immersive binocular near-eye display device including an input system, a microdisplay, and a graphics processor. The input system is configured to receive a parameter value, which is one of a plurality of values within a predetermined range that can be received by the input system. The microdisplay is configured to display virtual image content at a predetermined location of an object, forming an image of the object that can be viewed by a user of the near-eye display device, the virtual image content adding an augmentation to the object. The graphics processor is operatively coupled to the input system and the microdisplay. The graphics processor is configured to render the virtual image content to variably change the augmentation, variably changing the perceived realism of the object in relation to the parameter value.
[0078] In some implementations, the input system includes a world-facing depth imaging camera configured to acquire a depth image of the object, and the depth image is provided to the graphics processor.
[0079] Another aspect of the present disclosure relates to a near-eye display device including an input system, a microdisplay, a speaker, a graphics processor, and an audio mixing component. The input system is configured to receive a parameter value, which is one of a plurality of values in a predetermined range that can be received by the input system. The microdisplay is configured to display virtual image content to a predetermined location that can be viewed by a user of the near-eye display device, and the virtual image content adds an expansion to the user's real world. The speaker is configured to emit audio at a predetermined location. The graphics processor is operationally coupled to the input system and the microdisplay. The graphics processor is configured to render the virtual image content to variably change the expansion in relation to the parameter value. The audio mixing component is operationally coupled to the input system and the speaker. The audio mixing component is configured to compose an audio overlay to give a variable virtual quality in relation to the parameter value.
[0080] In some implementations, the augmentation is configured to change one or more of loudness and pitch of sound emitted by an object in a predetermined location, and the display device further comprises a position sensor configured to sense a position and orientation of the display device relative to the object. In some implementations, the augmentation is configured to change a background image in the predetermined location.
[0081] It will be understood that the configuration and / or manner described herein is exemplary in nature, and these specific embodiments or examples herein should not be considered restrictive, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions illustrated and / or described may be performed in the order shown and / or described, in other orders, in parallel, or omitted. Likewise, the order of the above process may be changed.
[0082] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.
Claims
1. A mixed reality display device, comprising: an input system configured to receive a parameter value, the parameter value being one of a plurality of predetermined values receivable by the input system; an electronic display configured to display virtual image content that adds augmentation to a real world environment viewed by a user of the mixed reality display device, the virtual image content being persistently positioned relative to at least a portion of the real world environment, the display configured to superimpose the virtual image content on the portion of the real world environment to impart a non-realistic quality to the portion of the real world environment; as well as a graphics processor operatively coupled to the input system and the electronic display, the graphics processor configured to render the virtual image content to vary the augmentation and the non-reality quality in relation to the parameter value; wherein the virtual image content is displayed at a predetermined location of a moving object in the real world environment, and wherein the display is configured to superimpose the virtual image content on the moving object to impart a variable unrealistic quality to the moving object.
2. The mixed reality display device according to claim 1, characterized in that: Also included is a user interface component configured to present, on the display device, a user interface element for allowing selection of the parameter value from the plurality of predetermined values.
3. The mixed reality display device according to claim 1, characterized in that: The input system includes one or more of a camera operatively coupled to a gesture recognition component and a microphone operatively coupled to a speech recognition component.
4. The mixed reality display device according to claim 1, characterized in that: The input system includes a touch sensor.
5. The mixed reality display device according to claim 1, characterized in that: The parameter value is one of three or more predetermined values.
6. The mixed reality display device according to claim 1, characterized in that: Also includes: a speaker configured to emit audio; and an audio mixing component operatively coupled to the input system and the speaker, the audio mixing component configured to compose an audio overlay to impart a non-realistic quality to the real world environment, the non-realistic quality being related to the parameter value.
7. The mixed reality display device according to claim 1, characterized in that: The portion of the real-world environment is a human face in the real-world environment.
8. The mixed reality display device according to claim 7, characterized in that: The augmentation includes one or more of color, texture, lighting, shadows, opacity, blur, and pixelation superimposed on the face of the human body of the real world environment.
9. The mixed reality display device according to claim 7, characterized in that: The augmentation is configured to change an aesthetic presentation of a human face of the real-world environment, and wherein each of the plurality of predetermined values corresponds to a different aesthetic presentation of a human face of the real-world environment.
10. The mixed reality display device according to claim 7, characterized in that: The augmentation is configured to evolve the appearance of a face of the real-world environment over time.
11. The mixed reality display device according to claim 7, characterized in that: The augmentation is configured to alter an aspect of a relationship of a face relative to one or more neighboring objects in the real-world environment.
12. A smart phone comprising: a camera configured to capture video of a real-world environment; and A mixed reality display device as claimed in any one of claims 1 to 11.
13. The smart phone according to claim 12, wherein: The virtual image content is configured to change an aesthetic presentation of a person's face, and wherein each of the plurality of predetermined values corresponds to a different aesthetic presentation of the person's face.
14. The smart phone according to claim 13, wherein: Also included is a depth imaging camera configured to capture a depth image of the human face.
