System and method for augmented and virtual reality

Abstract

To provide a system that allows two or more users to interact within a virtual world comprising virtual world data.SOLUTION: A system comprises a computer network with one or more computing devices, each comprising memory, processing circuitry, and software that is stored at least in part in the memory and executable by the processing circuitry to process at least a portion of the virtual world data. At least a first portion of the virtual world data originates from a first user virtual world local to a first user, and the computer network is operable to transmit the first portion to a user device for presentation to a second user, such that the second user may experience the first portion from the location of the second user, thereby efficiently passing aspects of the first user virtual world to the second user.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] FIELD OF THEINVENTION The present invention relates generally to systems and methods configured to facilitate an interactive virtual or augmented reality environment for one or more users. [Background technology]

[0002] (background) Virtual and augmented reality environments are generated, in part, by a computer using data that represents the environment. This data may represent, for example, various objects that a user may sense and interact with. Examples of these objects include objects that are rendered and displayed for the user to see, audio that is played for the user to hear, and tactile (or haptic) feedback for the user to feel. Users may sense and interact with virtual and augmented reality environments through a variety of visual, auditory, and tactile means. Summary of the Invention [Means for solving the problem]

[0003] (summary) Embodiments of the present invention are directed to devices, systems, and methods for facilitating virtual reality and / or augmented reality interaction for one or more users.

[0004] One embodiment is directed to a user display device comprising: a housing frame mountable on a user's head; a first pair of cameras coupled to the housing frame for tracking the user's eye movements and estimating a focal depth based on the tracked eye movements; a projection module having a light generating mechanism for generating and modifying projected light associated with a display object app such that the display object appears in focus based on the estimated focal depth; a lens mounted on the housing frame; and a processor communicatively coupled to the projection module for communicating data associated with the display image to the projection module. The lens may be positioned in front of the user's eye and may include at least one transparent mirror for directing the projected light into the user's eye. The at least one transparent mirror may selectively allow transmission of light from the local environment.

[0005] The user display device may further include a second pair of cameras mountable on the housing frame for capturing field of view images of an eye corresponding to each of the second pair of cameras. The processor may calculate a head pose of the user based on the captured field of view images.

[0006] The projection module may include a scanning laser array for modifying a projected light beam associated with the display object based on the estimated depth of focus. The diameter of the projected light beam may be less than 0.7 mm.

[0007] In one embodiment, the first pair of cameras may comprise an infrared camera paired with an infrared light source to track the movement of each of the user's eyes. The user display device may further comprise a sensor assembly comprising at least one sensor for sensing at least one of the user's movement, the user's location, the user's direction, and the user's orientation. The at least one sensor may be an accelerometer, a compass, or a gyroscope. The processor may estimate the user's head pose based on at least one of the user's movement, the user's location, the user's direction, and the user's orientation. The user display device may comprise a GPS system. The user display device may further comprise a haptic interface device communicatively coupled to the projection module for providing tactile feedback. The user display device may further comprise an environmental sensing system for digitally reconstructing the user's environment.

[0008] The processor may be communicatively coupled to the computer network to transmit at least a portion of the virtual world data and to receive another portion of the virtual world data.

[0009] The user display device may be mountable on the head frame and may include an audio speaker module for outputting sound. The user display device may further include a microphone mountable on the housing frame and for capturing sounds local to the user.

[0010] The projection module may modify other projected lights associated with other objects that are not the display object such that the other objects appear blurred. The processor may render frames of the display object at a rate of at least 60 frames per second.

[0011] The display object may be at least one of a virtual object, a rendered physical object, an image, and a video.

[0012] In another embodiment, a method includes tracking eye movements of a user, estimating a focal depth of the user's eye based on the tracked eye movements, modifying a light beam associated with a display object based on the estimated focal depth such that the display object is rendered in focus, and projecting the modified light beam into the user's eye. A diameter of the projected light beam projected into the user's eye may be less than 0.7 mm.

[0013] The method may further include selectively enabling transmission of light from the user's local environment based on a visualization mode of the display object. The visualization mode may be one of an augmented reality mode, a virtual reality mode, and a combination of augmented and virtual reality modes.

[0014] The method may further include capturing a field of view image for each of the user's eyes. The captured field of view image may be used to estimate a head pose of the user. The captured field of view image may be used to transform at least one physical object into a physically rendered virtual object and display the physically rendered virtual object to the user.

[0015] The method may further include extracting a set of points within the captured field of view image and creating a criterion for at least one physical object within the captured field of view image based on the extracted set of points. The method may further include transmitting at least one of the extracted set of points and the created criterion to a cloud computer and tagging at least one of the extracted set of points and the created criterion with a type of object. The method may further include recognizing different physical objects as belonging to a type of object based on at least one of the tagged set of points associated with the type of object and the tagged created criterion associated with the type of object.

[0016] The method may further include sensing at least one of a user movement, a user location, a user direction, and a user orientation, and calculating a user posture based on the at least one of the sensed movement, the sensed location, the sensed direction, and the sensed orientation. The sensor may be at least one of an accelerometer, a compass, and a gyroscope.

[0017] The method may further include processing the virtual world data associated with the display object to a cloud network and transmitting at least a portion of the virtual world data associated with the display object to a second user located at the second location so that the second user may experience at least a portion of the virtual world data associated with the display object at the second location.

[0018] The method may further include sensing a physical object and modifying at least a portion of the virtual world data associated with the display object based on a predetermined relationship with the sensed physical object. The method further includes presenting the modified virtual world data to the second user.

[0019] The method may further include modifying another light associated with another object that is not the display object such that the other object appears blurred.

[0020] The method may further include receiving user input through a user interface and modifying the display object based on the received user input. The user interface may be at least one of a haptic interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device. The display object may be at least one of a virtual object, a rendered physical object, an image, and a video.

[0021] In another embodiment, a method includes interacting with a virtual world including virtual world data through a head-mounted user display device, the head-mounted user display device rendering a display image associated with at least a portion of the virtual world data to a user based on an estimated focal depth of the user's eyes, creating additional virtual world data originating from at least one of an interaction of the head-mounted user device with the virtual world and an interaction of the user with a physical environment of the user, and transmitting the additional virtual world data to a computer network. The virtual world may be presented in a two-dimensional or three-dimensional format.

[0022] The method may further include transmitting the additional virtual world data to a second user at the second location for presentation such that the second user can experience the additional virtual world data from the second location. The additional virtual world data may be associated with a field of view image captured through the head-mounted user display device. The additional virtual world data may be associated with at least one of a sensed movement of the user, a sensed location of the user, a sensed direction of the user, and a sensed orientation of the user. The additional virtual world data may be associated with physical objects sensed by the head-mounted user display device. The additional virtual world data may be associated with display objects having a predetermined relationship to the sensed physical objects.

[0023] The method may further include selecting an interface for enabling interaction between the user and the head-mounted user display device based on the user input, and rendering a display object associated with at least a portion of the virtual world data based on the selected interface. The selected interface may be one of a virtual reality mode, an augmented reality mode, a mixed reality mode, and a combination of the virtual reality and augmented reality modes.

[0024] In another embodiment, a method for enabling two or more users to interact with a virtual world including virtual world data includes displaying the virtual world through a first user display device in a first visualization mode for a first user, transmitting at least a portion of the virtual world data to a second user display over a computer network, and displaying the virtual world associated with the transmitted portion of the virtual world data in a second visualization mode on a second user display device for a second user. The first visualization mode may be different from the second visualization mode. The first visualization mode may be at least one of an augmented reality mode, a virtual reality mode, a mixed reality mode, and a combination of virtual reality and augmented reality modes.

[0025] In another embodiment, the method includes processing at least one of rendered physical image data associated with an image of a real physical object and virtual image data associated with a virtual display object based on a user selection, and selectively displaying to the user a selected combination of the real physical object as viewed by the user in real time, the rendered physical virtual object rendered based on the real physical object as viewed by the user in real time, and the virtual display object. At least one of the real physical object, the rendered physical virtual object, and the virtual display object may be selectively displayed based on a user input of a visualization mode. The visualization mode may be at least one of an augmented reality mode, a virtual reality mode, a mixed reality mode, and a combination of a virtual and augmented reality mode.

[0026] The method further includes receiving image data associated with another display object over a computer network and converting the image data into a data format compatible with the selected visualization mode so that a user can view the other display object in the selected visualization mode.

[0027] The method further includes selectively allowing transmission of light from the outside environment so that a user can view actual physical objects based on the selected visualization mode.

[0028] In another embodiment, a method includes selectively allowing transmission of light from an outside environment through a lens of a head-mounted user display device, the head-mounted user display device configured to display either an entirely virtual object, an entirely physical object, or a combination of virtual and physical objects.

[0029] The selective enabling of light transmission may be based on a desired visualization mode, where the desired visualization mode is one of an augmented reality mode, a virtual reality mode, a mixed reality mode, and a combination of augmented and virtual reality modes.

[0030] The method may further include allowing full transmission of light from the outside environment when the head-mounted user display device is turned off, such that the user views only the entire physical object.

[0031] The method may further include projecting a light beam associated with at least one display object having a particular shape into an eye of the user, and selectively enabling transmission of light from the outside environment based on the particular shape of the at least one display object such that the user views the display object along with physical objects in the outside environment. The method may further include preventing transmission of light from the outside environment such that the user is able to view only the entire virtual object.

[0032] In another embodiment, a method for enabling two or more users to interact with a virtual world including virtual world data includes creating a remote avatar for a first user to access the virtual world through a first user device at a first location, placing the remote avatar of the first user at a real geographic location such that the first user can experience the real geographic location through the first user device at the first location, and interacting with a second user accessing the virtual world through a second user device at the real geographic location through the remote avatar placed at the real geographic location. The first location may be different from the real geographic location, or the first location may be substantially the same as the real geographic location.

[0033] The remote avatar may have a predefined relationship to a physical object in the actual geographic location. The remote avatar may respond to environmental cues in the actual geographic location. The movement of the remote avatar may be controlled by a first user. The remote avatar may interact with a second user in the actual geographic location.

[0034] In another embodiment, a method includes capturing a field of view image of each of a user's eyes through a head-mounted user display device, extracting a set of points within the captured field of view image, associating the extracted set of points with a particular object, and recognizing different objects based on the associated set of points of the particular object.

[0035] Another embodiment is directed to a system for enabling two or more users to interact with a virtual world including virtual world data, the system comprising a computer network comprising one or more computing devices comprising a memory, a processing circuit, and software at least partially stored in the memory and executable by the processing circuit to process at least a portion of the virtual world data, where at least a first portion of the virtual world data originates from a first user virtual world that is local to the first user, and the computer network is operable to transmit the first portion to the user device for presentation to the second user such that the second user may experience the first portion from the location of the second user such that aspects of the first user virtual world are effectively passed on to the second user. The first and second users may be in different physical locations, or substantially the same physical location. At least a portion of the virtual world may be configured to change in response to a change in the virtual world data. At least a portion of the virtual world may be configured to change in response to a physical object sensed by the user device. The change in the virtual world data may represent a virtual object having a predetermined relationship to the physical object. The changes in the virtual world data may be presented to a second user device for presentation to a second user according to the predetermined relationship. The virtual world may be operable to be rendered by at least one of the computer server or the user device. The virtual world may be presented in a two-dimensional format. The virtual world may be presented in a three-dimensional format. The user device may be operable to provide an interface for enabling interaction between a user and the virtual world in an augmented reality mode. The user device may be operable to provide an interface for enabling interaction between a user and the virtual world in a virtual reality mode. The user device may be operable to provide an interface for enabling interaction between a user and the virtual world in a combination of augmented and virtual reality modes.The virtual world data may be transmitted over a data network. The computer network may be operable to receive at least a portion of the virtual world data from a user device. At least a portion of the virtual world data transmitted to the user device may comprise instructions for generating at least a portion of the virtual world. At least a portion of the virtual world data may be transmitted to a gateway for at least one of processing or distribution. At least one of the one or more computer servers may be operable to process the virtual world data distributed by the gateway.

[0036] Another embodiment is directed to a system for virtual and / or augmented user experiences in which remote avatars are animated based at least in part on data on a wearable device, with optional input from voice intonation and facial recognition software.

[0037] Another embodiment is directed to a system for virtual and / or augmented user experiences in which the camera pose or viewpoint position and vector may be located anywhere within a world sector.

[0038] Another embodiment is directed to a system for virtual and / or augmented user experiences in which a world, or portions thereof, may be rendered for a user to observe at various and selectable scales.

[0039] Another embodiment is directed to a system for virtual and / or augmented user experiences in which features such as points or parametric lines, in addition to pose-tagged images, can be used as basis data for a world model, from which a software robot or object recognizer can be used to create parametric representations of real-world objects, segmenting objects and tagging source features for manual inclusion in the world model.

[0040] Additional and other objects, features, and advantages of the present invention are set forth in the detailed description, drawings, and claims. The present specification also provides, for example, the following items: (Item 1) 1. A user display device, comprising: a housing frame that can be mounted on a user's head; a first pair of cameras coupled to the housing frame for tracking eye movements of the user and estimating a depth of focus based on the tracked eye movements; a projection module having a light generating mechanism for generating and modifying a projected light associated with a display object application based on the estimated depth of focus such that the display object appears in focus; A lens mounted on the housing frame; a processor communicatively coupled to the projection module for communicating data associated with the display image to the projection module; A device comprising: (Item 2) 2. The user display device of claim 1, wherein the lens further comprises at least one transparent mirror positioned in front of the user's eye for directing the projected light into the user's eye. (Item 3) 3. The user display device of item 2, wherein the at least one transparent mirror selectively allows the transmission of light from the local environment. (Item 4) 2. The user display device of item 1, further comprising a second pair of cameras mountable on the housing frame for capturing images of the field of view of an eye corresponding to each of the second pair of cameras. (Item 5) 5. The user display device of claim 4, wherein the processor calculates the user's head pose based on the captured field of view image. (Item 6) 2. The user display device of claim 1, wherein the projection module comprises a scanning laser array for modifying projected light associated with the display object based on the estimated focal depth. (Item 7) 7. The user display device of claim 6, wherein the projected light is a light beam having a diameter of less than 0.7 mm. (Item 8) 2. The user display device of claim 1, wherein the first pair of cameras comprises an infrared camera paired with an infrared light source to track movements of each of the user's eyes. (Item 9) 2. The user display device of claim 1, further comprising a sensor assembly comprising at least one sensor for sensing at least one of the user's movement, the user's location, the user's direction, and the user's orientation. (Item 10) 10. The user display device of item 9, wherein the at least one sensor is at least one of an accelerometer, a compass, and a gyroscope. (Item 11) 10. The user display device of claim 9, wherein the processor estimates the user's head pose based on at least one of the user's movement, the user's location, the user's direction, and the user's orientation. (Item 12) Item 1. The user display device of item 1, further comprising a GPS system. (Item 13) 13. The user display device of claim 1, wherein the processor is communicatively coupled to a computer network to transmit at least a portion of the virtual world data and receive another portion of the virtual world data. (Item 14) 2. The user display device of item 1, further comprising an audio speaker module mountable on the head frame for outputting sound. (Item 15) 2. The user display device of claim 1, further comprising a microphone mountable on the housing frame for capturing sounds local to the user. (Item 16) 2. The user display device of item 1, wherein the projection module modifies another projected light associated with another object that is not the display object so that the other object appears blurred. (Item 17) 2. The user display device of claim 1, wherein the processor renders frames of the display object at a rate of at least 60 frames per second. (Item 18) 2. The user display device of claim 1, further comprising a haptic interface device communicatively coupled to the projection module for providing tactile feedback. (Item 19) 2. The user display device of claim 1, wherein the display object is at least one of a virtual object, a rendered physical object, an image, and a video. (Item 20) 2. The user display device of item 1, further comprising an environmental sensing system for digitally reconstructing the user's environment. (Item 21) 1. A method comprising: Tracking the eye movements of a user; estimating a depth of focus of the user's eye based on the tracked eye movements; modifying a light beam associated with a display object based on the estimated depth of focus such that the display object is presented in focus; projecting the modified light beam into an eye of the user; A method comprising: (Item 22) 22. The method of claim 21, further comprising selectively enabling transmission of light from the user's local environment based on a visualization mode of the display object, the visualization mode being one of an augmented reality mode, a virtual reality mode, and a combination of augmented and virtual reality modes. (Item 23) 22. The method of claim 21, further comprising the step of capturing an image of the field of view of each of the user's eyes. (Item 24) 24. The method of claim 23, further comprising estimating a head pose of the user based on an image of at least one field of view of the user's eye. (Item 25) transforming at least one physical object in the captured field of view image into a physically rendered virtual object; displaying the physically rendered virtual object to the user; 24. The method of claim 23, further comprising: (Item 26) Extracting a set of points within the captured field of view image; creating a reference for at least one physical object within the captured field of view image based on the extracted point set; 24. The method of claim 23, further comprising: (Item 27) transmitting at least one of the extracted point set and the created reference to a cloud computer; tagging at least one of the extracted point set and the created criteria with a type of object; 27. The method of claim 26, further comprising: (Item 28) 28. The method of claim 27, further comprising the step of recognizing different physical objects as belonging to the type of object based on at least one of a tagged point set associated with the type of object and a tagged created criterion associated with the type of object. (Item 29) 22. The method of claim 21, wherein the projected light is a light beam having a diameter of less than 0.7 mm. (Item 30) sensing at least one of a movement of the user, a location of the user, a direction of the user, and an orientation of the user; calculating a posture of the user based on the at least one of the sensed movement, the sensed location, the sensed direction, and the sensed orientation; 22. The method of claim 21, further comprising: (Item 31) 31. The method of claim 30, wherein the at least one sensor is at least one of an accelerometer, a compass, and a gyroscope. (Item 32) processing virtual world data associated with the display object onto a computer network; transmitting at least a portion of the virtual world data associated with the display object to a second user located at a second location such that the second user may experience at least a portion of the virtual world data associated with the display object at the second location; 22. The method of claim 21, further comprising: (Item 33) Sensing a physical object; modifying at least a portion of virtual world data associated with the display object based on a predetermined relationship with the sensed physical object; 33. The method of claim 32, further comprising: (Item 34) Item 34. The method of item 33, further comprising the step of presenting the modified virtual world data to the second user. (Item 35) Item 22. The method of item 21, further comprising the step of modifying another light associated with another object that is not the display object so that the other object appears blurred. (Item 36) receiving user input through a user interface; modifying the display object based on the received user input; 22. The method of claim 21, further comprising: (Item 37) Item 37. The method of item 36, wherein the user interface is at least one of a haptic interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device. (Item 38) 22. The method of claim 21, wherein the display object is at least one of a virtual object, a rendered physical object, an image, and a video. (Item 39) 1. A method comprising: interacting with a virtual world including virtual world data through a head-mounted user display device, the head-mounted user display device rendering a display image to a user associated with at least a portion of the virtual world data based on an estimated focal depth of an eye of the user; creating additional virtual world data originating from at least one of an interaction of the head-mounted user device with the virtual world and an interaction of the user with a physical environment; transmitting said additional virtual world data to a computer network; A method comprising: (Item 40) 40. The method of claim 39, further comprising transmitting the additional virtual world data to a second user at a second location for presentation so that the second user can experience the additional virtual world data from the second location. (Item 40) 40. The method of claim 39, wherein the additional virtual world data is associated with a field of view image captured through the head-mounted user display device. (Item 41) 40. The method of claim 39, wherein the additional virtual world data is associated with at least one of the sensed movement of the user, the sensed location of the user, the sensed direction of the user, and the sensed orientation of the user. (Item 42) 40. The method of claim 39, wherein the additional virtual world data is associated with physical objects sensed by the head-mounted user display device. (Item 43) Item 43. The method of item 42, wherein the additional virtual world data is associated with a display object having a predetermined relationship to the sensed physical object. (Item 44) selecting an interface for enabling interaction between the user and the head mounted user display device based on user input; rendering a display object associated with at least a portion of the virtual world data based on the selected interface; 40. The method of claim 39, further comprising: (Item 45) Item 45. The method of item 44, wherein the selected interface is one of a virtual reality mode, an augmented reality mode, a mixed reality mode, and a combination of the virtual reality and augmented reality modes. (Item 46) 40. The method of claim 39, wherein the virtual world is presented in a two-dimensional format. (Item 47) 40. The method of claim 39, wherein the virtual world is presented in three-dimensional form. (Item 48) 1. A method for enabling two or more users to interact with a virtual world including virtual world data, comprising: displaying the virtual world through a first user display device in a first visualization mode for a first user; transmitting at least a portion of the virtual world data to a second user display over a computer network; displaying, at a second user display device of a second user, in a second visualization mode, a virtual world associated with the transmitted portion of the virtual world data; A method comprising: (Item 49) Item 49. The method of item 48, wherein the first visualization mode is different from the second visualization mode. (Item 50) Item 49. The method of item 48, wherein the first visualization mode is at least one of an augmented reality mode, a virtual reality mode, a mixed reality mode, and a combination of the virtual reality and augmented reality modes. (Item 51) Item 49. The method of item 48, wherein the second visualization mode is at least one of an augmented reality mode, a virtual reality mode, a mixed reality mode, and a combination of the virtual reality and augmented reality modes. (Item 52) 1. A method comprising: processing at least one of rendered physical image data associated with the image of the real physical object and virtual image data associated with the virtual display object based on the user's selection; selectively displaying to the user selected combinations of actual physical objects as viewed in real time by the user, rendered physical virtual objects that are rendered based on the actual physical objects as viewed in real time by the user, and the virtual display objects; A method comprising: (Item 53) Item 53. The method of item 52, wherein at least one of the actual physical object, the rendered physical virtual object, and the virtual display object is selectively displayed based on a user input of a visualization mode. (Item 54) Item 54. The method of item 53, wherein the visualization mode is at least one of an augmented reality mode, a virtual reality mode, a mixed reality mode, and a combination of the virtual and augmented reality modes. (Item 55) receiving image data associated with another display object over a computer network; converting the image data into a data format compatible with the selected visualization mode such that the user can view other display objects in the selected visualization mode; 54. The method of claim 53, further comprising: (Item 56) Item 53. The method of item 52, further comprising the step of selectively enabling transmission of light from an outside environment so that the user can view the actual physical object based on the selected visualization mode. (Item 57) 1. A method comprising: 1. A method comprising: selectively enabling transmission of light from an external environment through a lens of a head-mounted user display device, the head-mounted user display device being configured to display either an entirely virtual object, an entirely physical object, or a combination of virtual and physical objects. (Item 58) Item 58. The method of item 57, wherein the selective enabling of the light transmission is based on a desired visualization mode, and the desired visualization mode is one of an augmented reality mode, a virtual reality mode, a mixed reality mode, and a combination of augmented and virtual reality modes. (Item 59) Item 58. The method of item 57, further comprising the step of allowing full transmission of light from the outside environment when the head-mounted user display device is turned off such that the user views only the entire physical object. (Item 60) projecting a light beam associated with at least one display object having a particular shape into the eye of the user; selectively enabling transmission of light from the outside environment based on a particular shape of the at least one display object such that the user views the display object along with physical objects in the outside environment; 58. The method of claim 57, further comprising: (Item 61) Item 58. The method of item 57, further comprising the step of preventing the transmission of light from the outside environment so that the user views only the entire virtual object. (Item 62) 1. A method for enabling two or more users to interact with a virtual world including virtual world data, comprising: creating a remote avatar for a first user to access the virtual world through a first user device at a first location; placing a remote avatar of the first user at a real geographic location such that the first user can experience the real geographic location through the first user device at the first location; interacting, through a remote avatar located at the real geographic location, with a second user accessing the virtual world through a second user device at the real geographic location; A method comprising: (Item 63) Item 63. The method of item 62, wherein the first location is different from the actual geographic location. (Item 64) Item 63. The method of item 62, wherein the first location is substantially the same as the actual geographic location. (Item 65) Item 63. The method of item 62, wherein the remote avatar has a predetermined relationship to a physical object in the actual geographic location. (Item 66) 63. The method of claim 62, wherein the remote avatar responds to environmental cues in the actual geographic location. (Item 67) Item 63. The method of item 62, wherein the movement of the remote avatar is controlled by the first user. (Item 68) Item 63. The method of item 62, wherein the remote avatar can interact with a second user at the actual geographic location. (Item 69) 1. A method comprising: capturing an image of a field of view of each of the user's eyes through a head mounted user display device; Extracting a set of points within the captured field of view image; associating the extracted point set with a particular object; recognizing a different object based on the associated point set of the particular object; A method comprising: (Item 70) 1. A system for enabling two or more users to interact with a virtual world including virtual world data, comprising: a computer network comprising one or more computing devices, the one or more computing devices comprising a memory, a processing circuit, and software stored at least in part in the memory and executable by the processing circuit to process at least a portion of the virtual world data; At least a first portion of the virtual world data originates from a first user virtual world that is local to a first user, and the computer network is operable to transmit the first portion to a user device for presentation to the second user such that the second user may experience the first portion from the location of the second user such that aspects of the first user virtual world are effectively passed on to the second user. (Item 71) Item 71. The system of item 70, wherein the first and second users are in different physical locations. (Item 72) Item 71. The system of item 70, wherein the first and second users are in substantially the same physical location. (Item 73) Item 71. The system of item 70, wherein at least a portion of the virtual world changes in response to changes to the virtual world data. (Item 74) Item 71. The system of item 70, wherein at least a portion of the virtual world changes in response to physical objects sensed by the user device. (Item 75) Item 75. The system of item 74, wherein the changes to the virtual world data represent virtual objects having a predetermined relationship with the physical objects. (Item 76) Item 76. The system of item 75, wherein the changes to the virtual world data are presented to a second user device for presentation to the second user in accordance with the predetermined relationship. (Item 77) Item 71. The system of item 70, wherein the virtual world is operable to be rendered by at least one of the computer server or a user device. (Item 78) Item 71. The system of item 70, wherein the virtual world is presented in a two-dimensional format. (Item 79) Item 71. The system of item 70, wherein the virtual world is presented in three-dimensional format. (Item 80) Item 71. The system of item 70, wherein the user device is operable to provide an interface to enable interaction between a user and the virtual world in an augmented reality mode. (Item 81) Item 71. The system of item 70, wherein the user device is operable to provide an interface to enable interaction between a user and the virtual world in a virtual reality mode. (Item 82) Item 81. The system of item 80, wherein the user device is operable to provide an interface to enable interaction between a user and the virtual world in a combination of augmented and virtual reality modes. (Item 83) Item 71. The system of item 70, wherein the virtual world data is transmitted over a data network. (Item 84) Item 71. The system of item 70, wherein the computer network is operable to receive at least a portion of the virtual world data from a user device. (Item 85) Item 71. The system of item 70, wherein at least a portion of the virtual world data transmitted to the user device comprises instructions for generating at least a portion of the virtual world. (Item 86) Item 71. The system of item 70, wherein at least a portion of the virtual world data is transmitted to a gateway for at least one of processing or distribution. (Item 87) Item 87. The system of item 86, wherein at least one of the one or more computer servers is operable to process virtual world data delivered by the gateway. [Brief description of the drawings]

[0041] [Figure 1] FIG. 1 illustrates an exemplary embodiment of the disclosed system for facilitating an interactive virtual or augmented reality environment for multiple users. [Diagram 2] FIG. 2 illustrates an example of a user device for interacting with the system illustrated in FIG. [Diagram 3] FIG. 3 illustrates an exemplary embodiment of a mobile wearable user device. [Figure 4] FIG. 4 illustrates examples of objects viewed by a user when the mobile wearable user device of FIG. 3 is operating in an extended mode. [Diagram 5] FIG. 5 illustrates examples of objects viewed by a user when the mobile wearable user device of FIG. 3 is operating in virtual mode. [Figure 6] FIG. 6 illustrates examples of objects viewed by a user when the mobile wearable user device of FIG. 3 is operating in a mixed virtual interface mode. [Figure 7] FIG. 7 illustrates an embodiment in which two users located in different geographic locations each interact with the other user and a common virtual world through their respective user devices. [Figure 8] FIG. 8 illustrates an embodiment in which the embodiment of FIG. 7 is extended to include the use of a haptic device. [Figure 9A] FIG. 9A illustrates an example of mixed mode interaction where a first user is interacting with a digital world in a mixed virtual interface mode and a second user is interacting with the same digital world in a virtual reality mode. [Figure 9B]FIG. 9B illustrates another example of mixed mode interaction in which a first user interacts with a digital world in a mixed virtual interface mode and a second user interacts with the same digital world in an augmented reality mode. [Figure 10] FIG. 10 illustrates an example illustration of a user's field of view when interacting with the system in augmented reality mode. [Figure 11] FIG. 11 illustrates an example illustration of a user's view showing virtual objects triggered by physical objects when the user is interacting with the system in augmented reality mode. [Figure 12] FIG. 12 illustrates one embodiment of an integrated augmented and virtual reality configuration in which one user in an augmented reality experience visualizes the presence of another user in a virtual reality experience. [Figure 13] FIG. 13 illustrates one embodiment of a time and / or contingency based augmented reality experience configuration. [Figure 14] FIG. 14 illustrates one embodiment of a user display configuration suitable for virtual and / or augmented reality experiences. [Figure 15] FIG. 15 illustrates one embodiment of local and cloud-based computational collaboration. [Figure 16] FIG. 16 illustrates various aspects of the alignment configuration. [Figure 17] FIG. 17 illustrates an example of a family interacting with the digital world of a virtual and / or augmented reality system according to one game embodiment. [Figure 18] FIG. 18 illustrates an example illustration of a user's view of the digital world environment as seen by the user of FIG. [Figure 19] FIG. 19 illustrates a user present in the physical environment viewed by the user of FIG. 17 interacting with the same digital world through a wearable device. [Figure 20] FIG. 20 illustrates an example illustration of a user's field of view for the user of FIG. [Figure 21]FIG. 21 illustrates an illustrative example of another user's view, where another user who is also present in the physical environment viewed by the user of FIG. 17 interacts with the same digital world of the user of FIGS. 17 and 19 through a mobile device. [Figure 22] FIG. 22 illustrates an example illustration of a user's bird's-eye view of the environment of FIGS. [Diagram 23] FIG. 23 illustrates an example scenario in which multiple users interact with a virtual and / or augmented reality system. [Figure 24A] FIG. 24A illustrates an example embodiment of a mobile communication device for interacting with the system illustrated in FIG. [Figure 24B] FIG. 24B illustrates an example embodiment of the mobile communication device of FIG. 24A removably and operably coupled to an extension console. [Diagram 25] FIG. 25 illustrates one embodiment of coarse localization. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] (Detailed Description) Referring to Figure 1, system 100 is representative hardware for implementing the processes described below. This representative system comprises a computing network 105 consisting of one or more computer servers 110 connected through one or more high bandwidth interfaces 115. The servers in a computing network need not be co-located. The one or more servers 110 each comprise one or more processors for executing program instructions. The servers also include memory for storing program instructions and data used and / or generated by processes being executed by the servers under the direction of the program instructions.

[0043] The computing network 105 communicates data between the servers 110 and between the servers and one or more user devices 120 over one or more data network connections 130. Examples of such data networks include, but are not limited to, all manner of public and private data networks, both mobile and wired, including, for example, the many interconnections of such networks commonly referred to as the Internet. No particular media, topology, or protocol is intended to be implied by the diagram.

[0044] The user devices are configured to communicate directly with either the computing network 105, or the server 110. Alternatively, the user devices 120 communicate with the remote server 110, and optionally with other user devices locally, through a specially programmed local gateway 140 for processing data and / or communicating data between the network 105 and one or more local user devices 120.

[0045] As shown, the gateway 140 is implemented as a separate hardware component including a processor for executing software instructions and memory for storing software instructions and data. The gateway has its own wired and / or wireless connection to a data network for communicating with the server 110, which comprises the computing network 105. Alternatively, the gateway 140 can be integrated with a user device 120, which is worn or carried by a user. For example, the gateway 140 may be implemented as a downloadable software application that is installed and runs on a processor included in the user device 120. The gateway 140, in one embodiment, provides access to the computing network 105 to one or more users via the data network 130.

[0046] The servers 110 each include, for example, working memory and storage devices for storing data and software programs, a microprocessor for executing program instructions, graphics processors and other specialized processors for rendering and generating graphic, image, video, audio, and multimedia files. The computing network 105 may also include devices for storing data that is accessed, used, or created by the servers 110.

[0047] The server, and optionally software programs executing on the user device 120 and gateway 140, are used to generate a digital world (also referred to herein as a virtual world) in which the user interacts with the user device 120. The digital world is represented by data and processes that represent and / or define virtual, non-existent entities, environments, and conditions that can be presented to the user through the user device 120 for the user to experience and interact with. For example, any type of object, entity, or item that would appear to be physically present when instantiated in a scene viewed or experienced by the user may include a description of its appearance, its behavior, how the user is allowed to interact with it, and other characteristics. The data used to create the environment of the virtual world (including virtual objects) may include, for example, atmospheric data, terrain data, weather data, temperature data, location data, and other data used to define and / or represent the virtual environment. In addition, the data defining the various conditions governing the operation of the virtual world may include, for example, physical laws, time, spatial relationships, and other data that may be used to define and / or create the various conditions governing the operation of the virtual world (including virtual objects).

[0048] Entities, objects, conditions, properties, behaviors, or other features of the digital world will generally be referred to herein as objects (e.g., digital objects, virtual objects, rendered physical objects, etc.) unless the context dictates otherwise. Objects may be any type of animate or inanimate object, including, but not limited to, structures, plants, vehicles, people, animals, living things, machines, data, videos, text, photos, and other users. Objects may also be defined in the digital world to store information about items, behaviors, or conditions that actually exist in the physical world. Data that represents or defines an entity, object, or item, or stores its current state, will generally be referred to herein as object data. This data is processed by the server 110, or by the gateway 140 or user device 120, depending on the implementation, to instantiate an instance of the object and render the object in an appropriate manner for the user to experience the user device.

[0049] Programmers who develop and / or create the digital world create or define the objects and the conditions under which they are instantiated. However, the digital world may allow others to create or modify the objects. Once an object is instantiated, the state of the object may be permitted to be changed, controlled, or manipulated by one or more users experiencing the digital world.

[0050] For example, in one embodiment, the development, production, and management of the digital world is generally provided by one or more system administration programmers. In some embodiments, this may include the development, design, and / or execution of storylines, themes, and events in the digital world, as well as the delivery of the discourse through various forms of events and media, such as, for example, film, digital, network, mobile, augmented reality, and live entertainment. The system administration programmers may also handle the technical management, discussion management, and curation of the digital world and its associated user communities, as well as other tasks typically performed by network administration personnel.

[0051] A user interacts with one or more digital worlds using some type of local computing device, generally designated as a user device 120. Examples of such user devices include, but are not limited to, a smartphone, a tablet device, a heads-up display (HUD), a gaming console, or any other device capable of conveying data and providing an interface or display to a user, or a combination of such devices. In some embodiments, the user device 120 may include or communicate with local peripherals or input / output components, such as, for example, a keyboard, a mouse, a joystick, a game controller, a haptic interface device, a motion capture controller, a visual tracking device such as those available from Leap Motion, Inc. or from Microsoft under the trademark Kinect (RTM), audio equipment, voice equipment, a projector system, a 3D display, and holographic 3D contact lenses.

[0052] An example of a user device 120 for interacting with the system 100 is illustrated in Figure 2. In the exemplary embodiment shown in Figure 2, a user 210 may interact with one or more digital worlds through a smartphone 220. The gateway is implemented by a software application 230 stored and executing on the smartphone 220. In this particular example, the data network 130 includes a wireless mobile network that connects the user device (i.e., smartphone 220) to the computer network 105.

[0053] In one implementation of a preferred embodiment, the system 100 is capable of supporting a large number of concurrent users (e.g., millions of users) using some type of user device 120 to each interact with the same digital world or with multiple digital worlds.

[0054] The user device provides the user with an interface to enable visual, audible, and / or physical interaction between the user and the digital world generated by the server 110, including other users and objects (real or virtual) presented to the user. The interface provides the user with rendered views that can be seen, heard, or otherwise sensed, and the ability to interact with the views in real time. The manner in which the user interacts with the rendered views may be dictated by the capabilities of the user device. For example, if the user device is a smartphone, user interaction may be implemented by the user touching a touch screen. In another example, if the user device is a computer or gaming console, user interaction may be implemented using a keyboard or game controller. The user device may include additional components that enable user interaction, such as sensors, and objects and information (including gestures) detected by the sensors may be provided as input representing the user interaction with the virtual world using the user device.

[0055] The rendered views can be presented in a variety of formats, such as, for example, two-dimensional or three-dimensional visual displays (including projections), sound, and haptic or tactile feedback. The rendered views may be interacted with by the user in one or more modes, including, for example, augmented reality, virtual reality, and combinations thereof. The format of the rendered views, as well as the interface modes, may be dictated by one or more of the user device, data processing capabilities, user device connectivity, network capabilities, and system workload. Having multiple users simultaneously interacting with the digital world, and the real-time nature of the data exchange, is made possible by the computing network 105, the server 110, the gateway component 140 (optionally), and the user devices 120.

[0056] In one embodiment, the computing network 105 IS is comprised of a large-scale computing system having single and / or multi-core servers (i.e., servers 110) connected through high-speed connections (e.g., high-bandwidth interfaces 115). The computing network 105 may form a cloud or grid network. Each of the servers includes memory and is coupled with a computer-readable memory for storing software for implementing data to create, design, modify, or process objects in the digital world. These objects and their instantiations may be dynamic, coming into existence and going out of existence, changing over time, and changing in response to other conditions. Examples of dynamic capabilities of objects are generally discussed herein with respect to various embodiments. In some embodiments, each user interacting with the system 100 may also be represented as an object and / or collection of objects within one or more digital worlds.

[0057] The servers 110 in the computing network 105 also store computational state data for each of the digital worlds. The computational state data (also referred to herein as state data) may be a component of the object data and generally defines the state of an instance of an object at a given instance in time. Thus, the computational state data may change over time and may be affected by the actions of one or more users and / or programmers who maintain the system 100. When a user affects the computational state data (or other data that comprises the digital world), the user directly modifies or otherwise manipulates the digital world. If the digital world is shared or interacted with by other users, the user's actions may affect what is experienced by other users interacting with the digital world. Thus, in some embodiments, changes made to the digital world by a user will be experienced by other users interacting with the system 100.

[0058] Data stored in one or more servers 110 in the computing network 105 is transmitted or deployed, in one embodiment, at high speed and with low latency to one or more user devices 120 and / or gateway components 140. In one embodiment, the object data shared by the server may be complete or may be compressed and contain instructions to recreate the complete object data at the user's end, and may be rendered and visualized by the user's local computing device (e.g., gateway 140 and / or user device 120). Software executing on the server 110 of the computing network 105 may, in some embodiments, adapt the data it generates and sends to a particular user's device 120 for objects in the digital world (or any other data exchanged by the computing network 105) as a function of the user's particular device and bandwidth. For example, as a user interacts with the digital world through a user device 120, the server 110 may recognize the particular type of device being used by the user, the device's connectivity and / or the available bandwidth between the user device and the server, and appropriately determine and balance the size of the data being delivered to the device to optimize the user interaction. An example of this may include reducing the size of the transmitted data to a lower resolution quality so that the data may be displayed on a particular user device having a lower resolution display. In a preferred embodiment, the computing network 105 and / or gateway component 140 delivers data to the user device 120 at a rate sufficient to present an interface that operates at 15 frames per second or greater and with a data resolution of high resolution quality or greater.

[0059] The gateway 140 provides a local connection to the computing network 105 for one or more users. In some embodiments, it may be implemented by a downloadable software application running on the user device 120 or another local device such as that shown in FIG. 2. In other embodiments, it may be implemented by a hardware component (a component having a processor, with appropriate software / firmware stored on the component) in communication with the user device 120 but either not incorporated or attached thereto or incorporated with the user device 120. The gateway 140 communicates with the computing network 105 via the data network 130 and provides data exchange between the computing network 105 and one or more local user devices 120. As discussed in more detail below, the gateway component 140 may include software, firmware, memory, and processing circuitry and may be capable of processing data communicated between the network 105 and one or more local user devices 120.

[0060] In some embodiments, the gateway component 140 monitors and adjusts the rate at which data is exchanged between the user device 120 and the computer network 105 to enable optimal data throughput for a particular user device 120. For example, in some embodiments, the gateway 140 buffers and downloads both static and dynamic aspects of the digital world, even beyond the field of view presented to the user through an interface connected to the user device. In such embodiments, instances of static objects (structured data, software-implemented methods, or both) may be stored in memory (local to the gateway component 140, the user device 120, or both) and are referenced to the local user's current location as indicated by data provided by the computing network 105 and / or the user's device 120. Instances of dynamic objects, which may include, for example, intelligent software agents and objects controlled by other users and / or the local user, are stored in a high-speed memory buffer. Dynamic objects, which represent two- or three-dimensional objects within the view presented to the user, can be categorized into component shapes, such as, for example, static shapes that move but do not change, and dynamic shapes that change. The portions of the dynamic objects that are changing can be updated by a real-time thread high priority data stream from the server 110 through the computing network 105 managed by the gateway component 140. As an example of a priority thread data stream, data that is within the 60 degree field of view of the user's eye may be given higher priority than data that is more peripheral. Another example includes prioritizing dynamic characters and / or objects within the user's field of view over static objects in the background.

[0061] In addition to managing the data connection between the computing network 105 and the user device 120, the gateway component 140 may store and / or process data that may be presented to the user device 120. For example, the gateway component 140 may in some embodiments receive compressed data from the computing network 105, e.g., representing graphical objects to be rendered for viewing by a user, and perform advanced rendering techniques to reduce the data load transmitted from the computing network 105 to the user device 120. In another example where the gateway 140 is a separate device, the gateway 140 may store and / or process data for local instances of objects, rather than communicating the data to the computing network 105 for processing.

[0062] Referring now also to FIG. 3, the digital world may be experienced by one or more users in various forms that may depend on the capabilities of the user's device. In some embodiments, the user device 120 may include, for example, a smartphone, a tablet device, a head-up display (HUD), a gaming console, or a wearable device. In general, the user device will include a processor for executing program code stored in a memory on the device, coupled with a display, and a communication interface. An exemplary embodiment of a user device is illustrated in FIG. 3, where the user device comprises a mobile wearable device, i.e., a head-mounted display system 300. According to an embodiment of the present disclosure, the head-mounted display system 300 includes a user interface 302, a user sensing system 304, an environment sensing system 306, and a processor 308. Although the processor 308 is shown in FIG. 3 as a standalone component separate from the head-mounted system 300, in alternative embodiments, the processor 308 may be integrated with one or more components of the head-mounted system 300 or incorporated into other system 100 components, such as, for example, the gateway 140.

[0063] The user device presents the user with an interface 302 for interacting with and experiencing the digital world. Such interactions may involve the user and the digital world, one or more other users interacting with the system 100, and objects within the digital world. The interface 302 generally provides visual and / or audio sensory input (and in some embodiments physical sensory input) to the user. Thus, the interface 302 may include a speaker (not shown) and, in some embodiments, a display component 303 capable of enabling stereoscopic 3D viewing and / or 3D viewing that embodies more natural properties of the human visual system. In some embodiments, the display component 303 may comprise a transparent interface (such as a transparent OLED) that, when in an "off" setting, enables an optically correct view of the user's surrounding physical environment with little to no optical distortion or computing overlay. As discussed in more detail below, the interface 302 may include additional settings that enable various visual / interface capabilities and functionality.

[0064] The user sensing system 304 may, in some embodiments, include one or more sensors 310 operable to detect certain characteristics, properties, or information relating to an individual user wearing the system 300. For example, in some embodiments, the sensor 310 may include a camera or optical detection / scanning circuitry capable of detecting real-time optical properties / measurements of the user, such as, for example, one or more of pupil contraction / dilation, angulation / positioning of each pupil, sphericity, eye shape (as eye shape changes over time), and other anatomical data. This data may provide information (e.g., the user's visual focus) or may be used to calculate information that may be used by the head-mounted system 300 and / or the interface system 100 to optimize the user's viewing experience. For example, in one embodiment, the sensors 310 may each measure the pupil contraction rate of each of the user's eyes. This data may be transmitted to the processor 308 (or to the gateway component 140 or to the server 110), and the data may be used to determine the user's reaction to, for example, the brightness setting of the interface display 303. The interface 302 may be adjusted according to the user's response, for example, by dimming the display 303 if the user's response indicates that the brightness level of the display 303 is too high. The user sensing system 304 may include other components than those discussed above or illustrated in FIG. 3. For example, in some embodiments, the user sensing system 304 may include a microphone for receiving audio input from the user. The user sensing system may also include one or more infrared camera sensors, one or more visible spectrum camera sensors, structured light emitters and / or sensors, infrared light emitters, coherent light emitters and / or sensors, gyros, accelerometers, magnetometers, proximity sensors, GPS sensors, ultrasonic emitters and detectors, and a haptic interface.

[0065] The environmental sensing system 306 includes one or more sensors 312 for acquiring data from the physical environment around the user. Objects or information detected by the sensors may be provided as input to the user device. In some embodiments, this input may represent a user interaction with the virtual world. For example, a user viewing a virtual keyboard on a desk may use gestures with their fingers as if they were typing on the virtual keyboard. The moving finger movements may be captured by the sensors 312 and provided to the user device or system as input, which may be used to change the virtual world or to create new virtual objects. For example, the finger movements may be recognized (using a software program) as typing, and the recognized typing gestures may be combined with known locations of virtual keys on the virtual keyboard. The system may then render a virtual monitor that is displayed to the user (or other users interacting with the system), which displays the text being typed by the user.

[0066] The sensor 312 may include, for example, a generally outward-facing camera or a scanner for interpreting sight information, for example, through continuously and / or intermittently projected infrared structured light. The environmental sensing system 306 may be used to map one or more elements of the user's surrounding physical environment by detecting and registering the local environment, including static objects, dynamic objects, people, gestures, and various lighting, atmospheric, and acoustic conditions. Thus, in some embodiments, the environmental sensing system 306 may include image-based 3D reconstruction software incorporated into a local computing system (e.g., the gateway component 140 or the processor 308) and operable to digitally reconstruct one or more objects or information detected by the sensor 312. In one exemplary embodiment, the environmental sensing system 306 provides one or more of motion capture data (including gesture recognition), depth sensing, face recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, sound source localization, noise reduction, infrared or similar laser projection, as well as monochrome and / or color CMOS sensors (or other similar sensors), field of view sensors, and various other light-enhancing sensors. It should be understood that the environmental sensing system 306 may include other components than those discussed above or illustrated in FIG. 3. For example, in some embodiments, the environmental sensing system 306 may include a microphone for receiving audio from the local environment. The user sensing system may also include one or more infrared camera sensors, one or more visible spectrum camera sensors, structured light emitters and / or sensors, infrared light emitters, coherent light emitters and / or sensors, gyros, accelerometers, magnetometers, proximity sensors, GPS sensors, ultrasonic emitters and detectors, and haptic interfaces.

[0067] As mentioned above, the processor 308 may in some embodiments be integrated with other components of the head-mounted system 300, integrated with other components of the interface system 100, or may be a stand-alone device (wearable or separate from the user) as shown in FIG. 3. The processor 308 may be connected to various components of the head-mounted system 300 and / or components of the interface system 100 through a physical wired connection or through a wireless connection such as, for example, a mobile network connection (including cellular and data networks), Wi-Fi, or Bluetooth. The processor 308 may include memory modules, integrated and / or additional graphics processing units, wireless and / or wired Internet connectivity, and codecs and / or firmware capable of converting data from a source (e.g., the computing network 105, the user sensing system 304, the environmental sensing system 306, or the gateway component 140) into image and audio data, and the images / video and audio may be presented to the user via the interface 302.

[0068] The processor 308 handles data processing for the various components of the head-mounted system 300, as well as data exchange between the head-mounted system 300 and the gateway component 140, and in some embodiments the computing network 105. For example, the processor 308 may be used to buffer and process data streaming between the user and the computing network 105, thereby enabling a smooth, continuous, and high fidelity user experience. In some embodiments, the processor 308 may process data at a rate sufficient to achieve anywhere between 8 frames / second at 320×240 resolution to 24 frames / second at high resolution (1280×720), or even more, such as 60-120 frames / second and 4k resolution and higher (10k+ resolution and 50,000 frames / second). Additionally, the processor 308 may store and / or process data that may be presented to the user, rather than being streamed in real-time from the computing network 105. For example, the processor 308, in some embodiments, may receive compressed data from the computing network 105 and perform advanced rendering techniques (such as shading or shading) to reduce the data load transmitted from the computing network 105 to the user device 120. In another example, the processor 308 may store and / or process local object data rather than transmitting the data to the gateway component 140 or to the computing network 105.

[0069] The head-mounted system 300 may, in some embodiments, include various settings or modes that enable various visual / interface capabilities and functionality. The modes may be selected manually by the user or automatically by the components of the head-mounted system 300 or the gateway component 140. As mentioned above, one embodiment of the head-mounted system 300 includes an "off" mode in which the interface 302 does not provide substantially any digital or virtual content. In the off mode, the display component 303 may be transparent, thereby enabling an optically correct view of the user's surrounding physical environment with little to no optical distortion or computing overlay.

[0070] In one exemplary embodiment, the head-mounted system 300 includes an "augmented" mode in which the interface 302 provides an augmented reality interface. In the augmented mode, the interface display 303 may be substantially transparent, thereby allowing the user to view the local physical environment. At the same time, virtual object data provided by the computing network 105, the processor 308, and / or the gateway component 140 is presented on the display 303 in combination with the local physical environment.

[0071] 4 illustrates an example embodiment of objects viewed by a user when the interface 302 is operating in an enhanced mode. As shown in FIG. 4, the interface 302 presents a physical object 402 and a virtual object 404. In the embodiment illustrated in FIG. 4, the physical object 402 is an actual physical object that exists in the user's local environment, while the virtual object 404 is an object created by the system 100 and displayed via the user interface 302. In some embodiments, the virtual object 404 may be displayed at a fixed position or location within the physical environment (e.g., a virtual monkey standing next to a particular road sign located in the physical environment) or may be displayed to the user as an object located at a position relative to the user interface / display 303 (e.g., a virtual clock or thermometer visible in the top left corner of the display 303).

[0072] In some embodiments, the virtual object may be cued from or triggered by an object that is physically present within or outside the user's field of view. The virtual object 404 is cued from or triggered by the physical object 402. For example, the physical object 402 may actually be a stool, and the virtual object 404 may be displayed to the user (and in some embodiments to other users interacting with the system 100) as a virtual animal standing on the stool. In such an embodiment, the environmental sensing system 306 may use software and / or firmware stored, for example, in the processor 308, to recognize various features and / or shape patterns (captured by the sensor 312) that identify the physical object 402 as a stool. These recognized shape patterns, such as the top of the stool, may be used to trigger the placement of the virtual object 404. Other examples include walls, tables, furniture, cars, buildings, people, floors, plants, animals, and any object that can be seen may be used to trigger an augmented reality experience in some relationship to one or more objects.

[0073] In some embodiments, the particular virtual object 404 that is triggered may be selected by a user or may be selected automatically by other components of the head-mounted system 300 or the interface system 100. Additionally, in embodiments in which a virtual object 404 is automatically triggered, the particular virtual object 404 may be selected based on the particular physical object 402 (or characteristics thereof) from which the virtual object 404 is cued or triggered. For example, if the physical object is identified as a diving board extending over a pool, the triggered virtual object may be a living being wearing a snorkel, swimsuit, flotation device, or other related item.

[0074] In another exemplary embodiment, the head-mounted system 300 may include a "virtual" mode in which the interface 302 provides a virtual reality interface. In the virtual mode, the physical environment is omitted from the display 303, and virtual object data provided by the computing network 105, the processor 308, and / or the gateway component 140 is presented on the display 303. Omission of the physical environment may be achieved by physically blocking the visual display 303 (e.g., via a cover) or through a feature of the interface 302 in which the display 303 transitions to an opaque setting. In the virtual mode, live and / or stored visual and audio sensations may be presented to the user through the interface 302, and the user experiences and interacts with the digital world (digital objects, other users, etc.) through the virtual mode of the interface 302. Thus, the interface presented to the user in the virtual mode consists of virtual object data, including the virtual digital world.

[0075] Figure 5 illustrates an example embodiment of a user interface when the head-mounted interface 302 is operating in a virtual mode. As shown in Figure 5, the user interface presents a virtual world 500 made up of digital objects 510, which may include atmosphere, weather, terrain, structures, and people. Although not shown in Figure 5, the digital objects may also include, for example, plants, vehicles, animals, creatures, machines, artificial intelligence, location information, and any other objects or information that define the virtual world 500.

[0076] In another exemplary embodiment, the head-mounted system 300 may include a "mixed" mode, where various features of the head-mounted system 300 (as well as features of the virtual and augmented modes) may be combined to create one or more custom interface modes. In one example custom interface mode, the physical environment is omitted from the display 303, and virtual object data is presented on the display 303 in a manner similar to the virtual mode. However, in this example custom interface mode, the virtual objects may be fully virtual (i.e., they do not exist in the local physical environment), or they may be actual local physical objects that are rendered as virtual objects in the interface 302 instead of physical objects. Thus, in a particular custom mode (referred to herein as a mixed virtual interface mode), live and / or stored visual and audio sensations may be presented to the user through the interface 302, and the user experiences and interacts with a digital world that includes fully virtual objects and rendered physical objects.

[0077] FIG. 6 illustrates an exemplary embodiment of a user interface operating according to a mixed virtual interface mode. As shown in FIG. 6, the user interface presents a virtual world 600 consisting of full virtual objects 610 and rendered physical objects 620 (renderings of objects that are otherwise physically present in the scene). According to the example illustrated in FIG. 6, the rendered physical objects 620 include a building 620A, a ground 620B, and a platform 620C, and are shown with a thick outline 630 to indicate to the user that the objects are rendered. In addition, the full virtual objects 610 include an additional user 610A, a cloud 610B, a sun 610C, and a flame 610D above the platform 620C. It should be understood that the full virtual objects 610 may include, for example, atmosphere, weather, terrain, buildings, people, plants, vehicles, animals, creatures, machines, artificial intelligence, location information, and any other objects or information that define the virtual world 600 and that are not rendered from objects present in the local physical environment. Conversely, rendered physical object 620 is an actual local physical object that is rendered as a virtual object in interface 302. Thick outline 630 represents one embodiment for showing the rendered physical object to a user. As such, the rendered physical object may be shown using methods other than those disclosed herein, etc.

[0078] In some embodiments, the rendered physical objects 620 may be detected using the sensors 312 of the environment sensing system 306 (or using other devices such as a motion or image capture system) and converted into digital object data, for example, by software and / or firmware stored in the processing circuitry 308. Thus, as a user interacts with the system 100 in a mixed virtual interface mode, various physical objects may be displayed to the user as rendered physical objects. This may be particularly useful for allowing the user to interact with the system 100 while still being able to safely navigate the local physical environment. In some embodiments, the user may be able to selectively remove or add rendered physical objects to the interface display 303.

[0079] In another example custom interface mode, the interface display 303 may be substantially transparent, thereby allowing the user to view the local physical environment while various local physical objects are displayed to the user as rendered physical objects. This example custom interface mode is similar to the augmented mode, except that one or more of the virtual objects may be rendered physical objects, as discussed above with respect to the previous example.

[0080] The foregoing custom interface mode examples represent some example embodiments of the various custom interface modes that can be provided by the mixed mode of head-mounted system 300. Thus, various other custom interface modes may be created from various combinations of the features and functionality provided by the components of head-mounted system 300 and the various modes discussed above without departing from the scope of this disclosure.

[0081] The embodiments discussed herein merely illustrate some examples for providing interfaces operating in off, augmented, virtual, or mixed modes, and are not intended to limit the scope or content of each interface mode, or the functionality of the components of head-mounted system 300. For example, in some embodiments, virtual objects may include data displayed to a user (time, temperature, altitude, etc.), objects created and / or selected by system 100, objects created and / or selected by a user, or even objects representing other users interacting with system 100. Additionally, virtual objects may include extensions of physical objects (e.g., virtual statues growing out of a physical platform) and may be visually connected to or disconnected from physical objects.

[0082] Virtual objects may also be dynamic, changing over time, changing according to various relationships (e.g., location, distance, etc.) between the user or other users, physical objects, and other virtual objects, and / or changing according to other variables specified in the software and / or firmware of the head-mounted system 300, the gateway component 140, or the server 110. For example, in an embodiment, a virtual object may respond to a user device or its components (e.g., a virtual ball moves when a haptic device is placed next to it), physical or verbal user interactions (e.g., a virtual creature runs away when a user approaches it or speaks when a user talks to it), a chair is thrown at a virtual creature and the creature avoids the chair, other virtual objects (e.g., a first virtual creature reacts when it sees a second virtual creature), physical variables such as location, distance, temperature, time, or other physical objects in the user's environment (e.g., a virtual creature shown standing on a physical road becomes flattened when a physical car passes by).

[0083] The various modes discussed herein may be applied to user devices other than the head-mounted system 300. For example, an augmented reality interface may be provided via a mobile phone or tablet device. In such an embodiment, the phone or tablet may use a camera to capture the physical environment around the user and virtual objects may be overlaid on the phone / tablet display screen. In addition, virtual modes may be provided by displaying a digital world on the phone / tablet display screen. Thus, these modes may be mixed to create various custom interface modes as described above using the phone / tablet components discussed herein, as well as other components connected to or used in combination with the user device. For example, mixed virtual interface modes may be provided by a computer monitor, television screen, or a device lacking a camera operating in combination with a motion or image capture system. In this exemplary embodiment, the virtual world may be viewed from the monitor / screen and object detection and rendering may be performed by the motion or image capture system.

[0084] 7 illustrates an exemplary embodiment of the present invention in which two users located at different geographic locations each interact with the other user and a common virtual world through their respective user devices. In this embodiment, two users 701 and 702 are throwing a virtual ball 703 (a type of virtual object) back and forth, and each user is able to observe the other user's influence on the virtual world (e.g., each user observes the virtual ball change direction, be caught by the other user, etc.). Because the movement and location of the virtual object (i.e., virtual ball 703) is tracked by server 110 in computing network 105, system 100 may, in some embodiments, communicate to users 701 and 702 the exact location and timing of the arrival of ball 703 relative to each user. For example, if a first user 701 is located in London, user 701 may throw ball 703 to a second user 702 located in Los Angeles at a speed calculated by system 100. Thus, the system 100 may communicate the exact time and location of the ball's arrival to the second user 702 (e.g., via email, text message, instant message, etc.). As such, the second user 702 may use his device to watch the ball 703 arrive at a particular time and location. One or more users may also use geolocation mapping software (or the like) to track one or more virtual objects as they virtually travel the globe. An example of this may be a user wearing a 3D head-mounted display looking up into the sky and seeing a virtual airplane flying overhead, superimposed on the real world. The virtual airplane may be flown by the user, either by an intelligent software agent (software running on the user device or gateway), by other users, which may be locally and / or remotely, and / or a combination of these.

[0085] As mentioned above, the user device may include a haptic interface device that provides feedback (e.g., resistance, vibration, light, sound, etc.) to the user when the haptic device is determined by system 100 to be located at a physical spatial location relative to the virtual object. For example, the embodiment described above with respect to FIG. 7 may be extended to include the use of a haptic device 802, as shown in FIG.

[0086] In this exemplary embodiment, haptic device 802 may be displayed in the virtual world as a baseball bat. When ball 703 arrives, user 702 may swing haptic device 802 toward virtual ball 703. If system 100 determines that the virtual bat provided by haptic device 802 has "made contact" with ball 703, haptic device 802 may vibrate or provide other feedback to user 702, and virtual ball 703 may bounce off the virtual bat in a direction calculated by system 100 according to the detected speed, direction, and timing of the contact between the ball and the bat.

[0087] The disclosed system 100, in some embodiments, may facilitate mixed-mode interaction, where multiple users may interact with a common virtual world (and the virtual objects contained therein) using different interface modes (e.g., augmented, virtual, mixed, etc.). For example, a first user who interacts with a particular virtual world in a virtual interface mode may interact with a second user who interacts with the same virtual world in an augmented reality mode.

[0088] 9A illustrates an example in which a first user 901 (interacting with the digital world of the system 100 in a mixed virtual interface mode) and a first object 902 appear as virtual objects to a second user 922 interacting with the same digital world of the system 100 in a full virtual reality mode. As described above, when interacting with the digital world via the mixed virtual interface mode, local physical objects (e.g., the first user 901 and the first object 902) may be scanned and rendered as virtual objects in the virtual world. The first user 901 may be scanned, for example, by a motion capture system or similar device and rendered in the virtual world (by software / firmware stored in the motion capture system, the gateway component 140, the user device 120, the system server 110, or other devices) as a first rendered physical object 931. Similarly, the first object 902 may be scanned, for example, by the environmental sensing system 306 of the head-mounted interface 300 and rendered in the virtual world (by software / firmware stored in the processor 308, the gateway component 140, the system server 110, or other device) as a second rendered physical object 932. The first user 901 and the first object 902 are shown in the first portion 910 of FIG. 9A as physical objects in the physical world. In the second portion 920 of FIG. 9A, the first user 901 and the first object 902 are shown as a first rendered physical object 931 and a second rendered physical object 932 as they appear to a second user 922 interacting with the same digital world of the system 100 in full virtual reality mode.

[0089] FIG. 9B illustrates another exemplary embodiment of mixed mode interaction in which a first user 901 is interacting with a digital world in a mixed virtual interface mode, as discussed above, and a second user 922 is interacting with the same digital world (and the second user's local physical environment 925) in an augmented reality mode. In the embodiment of FIG. 9B, the first user 901 and the first object 902 are located at a first physical location 915, and the second user 922 is located at a different second physical location 925, separated by some distance from the first location 915. In this embodiment, the virtual objects 931 and 932 may be transposed in real time (or near real time) to a location in the virtual world that corresponds to the second location 925. Thus, the second user 922 may observe and interact with rendered physical objects 931 and 932, which represent the first user 901 and the first object 902, respectively, in the second user's local physical environment 925.

[0090] FIG. 10 illustrates an example illustration of a user's field of view when interacting with the system 100 in an augmented reality mode. As shown in FIG. 10, the user sees the local physical environment (i.e., a city with multiple buildings) as well as a virtual character 1010 (i.e., a virtual object). The location of the virtual character 1010 may be triggered by one or more 3D reference frames, such as 2D visual targets (e.g., billboards, postcards, or magazines) and / or buildings, cars, people, animals, airplanes, parts of buildings, and / or 3D physical objects, virtual objects, and / or combinations thereof. In the example illustrated in FIG. 10, known locations of buildings in the city may provide alignment references and / or information and key features for rendering the virtual character 1010. Additionally, the user's geospatial location (e.g., provided by GPS, attitude / position sensors, etc.) or movement location relative to the buildings may include data used by the computing network 105 to trigger transmission of data used to display the virtual character 1010. In some embodiments, the data used to display the virtual character 1010 may include a rendered character 1010 and / or instructions (executed by the gateway component 140 and / or the user device 120) for rendering the virtual character 1010 or portions thereof. In some embodiments, if the user's geospatial location is unavailable or unknown, the server 110, the gateway component 140, and / or the user device 120 may still display the virtual object 1010 using an estimation algorithm that uses the user's last known location as a function of time and / or other parameters to estimate where certain virtual and / or physical objects may be located. This may also be used to determine the location of any virtual objects in the event that the user's sensors become occluded and / or experience other malfunctions.

[0091] In some embodiments, the virtual character or virtual object may comprise a virtual statue, the rendering of which is triggered by a physical object. For example, referring now to FIG. 11, the virtual statue 1110 may be triggered by the actual physical platform 1120. The triggering of the statue 1110 may be in response to a visual object or feature (e.g., a fiducial, a design feature, a geometric shape, a pattern, a physical location, an elevation, etc.) detected by the user device or other components of the system 100. When a user views the platform 1120 without a user device, the user sees the platform 1120 without the statue 1110. However, when a user views the platform 1120 through a user device, the user sees the statue 1110 on the platform 1120, as shown in FIG. 11. The statue 1110 is a virtual object and therefore may be stationary, animated, change over time or relative to the user's viewing position, or even change depending on which particular user is viewing the statue 1110. For example, if the user is a small child, the statue may be a dog, whereas if the viewer is an adult male, the statue may be a large robot as shown in FIG. 11. These are examples of user-dependent and state-dependent experiences. This would allow one or more users to perceive one or more virtual objects, alone and / or in combination with physical objects, and experience customized and personalized versions of the virtual objects. The statue 1110 (or portions thereof) may be rendered by various components of the system, including, for example, software / firmware installed on the user device. Using data indicative of the position and pose of the user device in combination with the alignment features of the virtual object (i.e., statue 1110), the virtual object (i.e., statue 1110) forms a relationship with the physical object (i.e., platform 1120).For example, the relationship between one or more virtual objects and one or more physical objects may be a function of distance, positioning, time, geolocation, proximity to one or more other virtual objects, and / or any other functional relationship including any type of virtual and / or physical data. In some embodiments, image recognition software in the user device may further enhance the relationship between the digital and physical objects.

[0092] The interactive interfaces provided by the disclosed systems and methods may be implemented to facilitate a variety of activities, such as, for example, interacting with one or more virtual environments and objects, interacting with other users, and experiencing various forms of media content, including advertisements, musical concerts, and movies. However, the disclosed systems facilitate user interaction such that users do not merely watch or listen to media content, but rather actively participate in and experience the media content. In some embodiments, user participation may include modifying existing content or creating new content that is rendered in one or more virtual worlds. In some embodiments, the media content, and / or users creating content, may theme the creation of one or more virtual worlds.

[0093] In one embodiment, a musician (or other user) may create musical content that is rendered to users interacting with a particular virtual world. The musical content may include, for example, various singles, EPs, albums, videos, short films, and concert performances. In one embodiment, multiple users may interact with system 100 to simultaneously experience a virtual concert performed by a musician.

[0094] In some embodiments, produced media may contain a unique identifier code associated with a particular entity (e.g., a band, artist, user, etc.). The code may be in the form of a set of alphanumeric characters, a UPC code, a QR code, a 2D image trigger, a 3D physical object feature trigger, or other digital mark, as well as sound, image, and / or both. In some embodiments, the code may also be embedded in digital media that may be interacted with using system 100. A user may obtain the code (e.g., via payment of a fee) and redeem the code to access media content produced by the entity associated with the identifier code. Media content may be added or removed from the user's interface.

[0095] In one embodiment, to avoid the computational and bandwidth limitations of passing real-time or near real-time video data from one computing system to another with low latency, such as from a cloud computing system to a local processor coupled to a user, parametric information regarding various shapes and geometries may be transferred and utilized to define surfaces, while textures may be transferred and added to these surfaces to provide static or dynamic details, such as bitmap-based video details of an individual's face mapped onto the parametrically reconstructed facial geometry. As another example, if the system is configured to recognize an individual's face and knows that the individual's avatar is located in the augmented world, the system may be configured to pass the relevant world information and the individual's avatar information in one relatively large setup transfer, after which the remaining transfers to the local computing system, such as 308 depicted in FIG. 1 for local rendering, may be limited to parameter and texture updates, such as motion parameters of the individual's skeletal structure and movement bitmaps of the individual's face, at a much smaller bandwidth than the setup transfer or passing real-time video. Thus, cloud-based and local computing assets may be used in an integrated fashion, with the cloud handling computations that do not require relatively low latency and the local processing assets handling tasks where low latency is a priority; in such cases, the type of data transferred to the local system is preferably passed at a relatively low bandwidth, with some amount of such type of data (i.e., parametric information, textures, etc. as opposed to all real-time video).

[0096] Referring back to Figure 15, a schematic diagram illustrates the coordination between cloud computing assets (46) and local processing assets (308, 120). In one embodiment, the cloud (46) assets are operatively coupled, such as directly (40, 42), to one or both of the local computing assets (120, 308), such as processor and memory configurations that may be housed in structures configured to be coupled to a user's head (120) or belt (308), via wired or wireless networking (wireless being preferred for mobility, wired being preferred for some high bandwidth or large data capacity transfer that may be desired). These computing assets that are local to the user may likewise be operatively coupled to each other via wired and / or wireless connection configurations (44). In one embodiment, to maintain low inertia and small size of the head-mounted subsystem (120), the primary transfer between the user and the cloud (46) may be via a link between the belt-based subsystem (308) and the cloud, with the head-mounted subsystem (120) being primarily data tethered to the belt-based subsystem (308) using a wireless connection, such as, for example, an ultra-wideband ("UWB") connection as currently employed in personal computing peripheral connectivity applications.

[0097] With efficient local and remote processing collaboration and an appropriate display device for the user, such as the user interface 302 or user "display device" featured in FIG. 3, the display 14 described below with reference to FIG. 14, or a variation thereof, aspects of one world related to the user's current actual or virtual location may be transferred or "handed" to the user and efficiently updated. Indeed, in one embodiment, when one individual utilizes a virtual reality system ("VRS") in an augmented reality mode and another individual utilizes the VRS in a fully virtual mode to explore the same world that is local to the first individual, the two users may mutually experience the world in a variety of ways. For example, with reference to FIG. 12, a scenario similar to that described with reference to FIG. 11 is depicted with the addition of a visualization of the second user's avatar 2 flying through the augmented reality world depicted from the fully virtual reality scenario. 12 may be experienced and displayed in augmented reality to a first individual, with two augmented reality elements (statue 1110 and the second individual's flying bumblebee avatar 2) displayed in addition to the actual physical elements of the local world surroundings within the view, such as the ground, background buildings, statue platform 1120, etc. Dynamic updates may be utilized to allow the first individual to visualize the progress of the second individual's avatar 2 as the avatar 2 flies through the world local to the first individual.

[0098] Again, with a configuration as described above where there is one world model that resides on and can be distributed from cloud computing resources, such a world may be "handled" to one or more users in a relatively low bandwidth format that is preferable to trying to pass around real-time video data or the like. An individual standing near the statue (i.e., as shown in FIG. 12) may have their augmented experience informed by the cloud-based world model, a subset of which may be passed up to them and their local display device to complete the view. An individual sitting at a remote display device, which may be as simple as a personal computer located on a desk, can efficiently download the same section of information from the cloud and have it rendered on their display. In fact, an individual physically present in the park near the statue may take a remotely located friend for a walk in the park, with the friend participating through virtual and augmented reality. The system would need to figure out where the paths are, where the trees are, where the statues are, but with that information on the cloud, participating friends can download from the cloud side of the scenario and then start walking together as an augmented reality that is local to the individual actually in the park.

[0099] With reference to Figure 13, a time and / or contingency parameter based embodiment is depicted in which an individual engaging in a virtual and / or augmented reality interface, such as the user interface 302 or user display device featured in Figure 3, the display device 14 described below with reference to Figure 14, or variations thereof, is utilizing the system (4) and enters a coffee shop to order a cup of coffee (6). The VRS may be configured to utilize sensing and data collection capabilities locally and / or remotely to provide an augmented and / or virtual reality display enhancement for the individual, such as a highlighted location of the coffee shop door, or a bubble window of a related coffee menu (8). Upon receipt of the ordered cup of coffee by the individual, or upon detection of some other relevant parameter by the system, the system may be configured to display with the display device one or more time-based augmented or virtual reality images, videos, and / or sounds within the local environment, such as a Madagascar jungle view from the walls and ceiling, either static or dynamic, with or without jungle sounds and other effects (10). Such presentation to the user may be interrupted based on timing parameters (i.e., 5 minutes after a full coffee cup has been recognized and handed to the user, 10 minutes after the system recognizes the user walking through the front door of the store, etc.) or other parameters, such as the system's recognition that the user has finished drinking their coffee by noting the upside-down orientation of the coffee cup as the user takes a final sip of coffee from the cup, or the system's recognition that the user has left the front door of the store (12).

[0100] Referring to FIG. 14, one embodiment of a suitable user display device (14) is shown that includes a display lens (82) that may be mounted on the user's head or eye by a housing or frame (84). The display lens (82) may include one or more transparent mirrors positioned by the housing (84) in front of the user's eye (20) and may be configured to bounce the projected light (38) into the eye (20) to facilitate beam shaping while also allowing at least some light transmission from the local environment in an augmented reality configuration (in a virtual reality configuration, it may be desirable for the display system 14 to be able to block substantially all light from the local environment, such as by a darkened visor, blocking curtains, a full black LCD panel mode, or the like). In the depicted embodiment, two wide field machine vision cameras (16) are coupled to the housing (84) to image the environment around the user. In one embodiment, the cameras (16) are dual capture visible / infrared cameras. The depicted embodiment also includes a pair of scanning laser shaped wavefront (i.e., for depth) projector modules, along with viewing mirrors and optics configured to project light (38) into the eye (20) as shown. The depicted embodiment also includes two miniature infrared cameras (24) paired with infrared light sources (such as light emitting diodes "LEDs" 26) configured to track the user's eye (20) to aid in rendering and user input. The system (14) further features a sensor assembly (39) with X, Y, and Z axis accelerometer capabilities, as well as magnetic compass and X, Y, and Z axis gyro capabilities, and may preferably provide data at a relatively high frequency, such as 200 Hz. The depicted system (14) also includes a head pose processor (36), such as an ASIC (application specific integrated circuit), FPGA (field programmable gate array), and / or ARM processor (advanced reduced instruction set machine), which may be configured to calculate real or near real-time user head pose from the wide field of view image information output by the capture device (16).Also shown is another processor (32) configured to perform digital and / or analog processing to derive pose from gyro, compass, and / or accelerometer data from the sensor assembly (39). The depicted embodiment also features a GPS (37, Global Positioning Satellite) subsystem to assist with pose and positioning. Finally, the depicted embodiment includes a rendering engine (34), which may feature hardware executing software programs configured to provide rendering information that is local to the user to facilitate operation of the scanner and imaging into the user's eye for the user's view of the world. The rendering engine (34) is operatively coupled (81, 70, 76 / 78, 80, i.e., via wired or wireless connections) to the sensor pose processor (32), image pose processor (36), eye tracking camera (24), and projection subsystem (18) such that light of the rendered augmented and / or virtual reality objects is projected using the scanning laser array (18) in a manner similar to a retinal scanning display. The waveform of the projected light beam (38) may be bent or focused to match the desired focal length of the augmented and / or virtual reality object. A mini infrared camera (24) may be utilized to track the eye to assist in rendering and user input (i.e., where the user is looking, at what depth the user is focusing, the edge of the eye may be utilized to estimate focal depth, as discussed below). A GPS (37), gyro, compass, and accelerometer (39) may be utilized to provide heading and / or fast attitude estimates. The camera (16) images and attitude, along with data from associated cloud computing resources, may be utilized to map the local world and share the user's view with the virtual or augmented reality community. While most of the hardware in the display system (14) featured in FIG. 14 is depicted as directly coupled to the display (82) and the housing (84) adjacent to the user's eye (20), the depicted hardware components may be mounted on or housed within other components, such as belt-mounted components, as shown in FIG. 3, for example.In one embodiment, all of the components of the system (14) featured in FIG. 14 are directly coupled to the display housing (84), with the exception of the image pose processor (36), the sensor pose processor (32), and the rendering engine (34), and communication between the latter three and the remaining components of the system (14) may be by wireless communication, such as ultra-wideband, or by wired communication. The depicted housing (84) is preferably head-mounted and wearable by a user. It may also feature speakers, such as those inserted into the user's ears and utilized to provide the user with sounds that may be associated with an augmented or virtual reality experience, such as the jungle sounds referenced with reference to FIG. 13, and a microphone that may be utilized to capture sounds that are local to the user.

[0101] With regard to projecting light 38 into the user's eye 20, in one embodiment, the mini camera 24 may be utilized to measure where the center of the user's eye 20 is geometrically tangent, which generally coincides with the location of the eye's 20 focal point, or "depth of focus." The three-dimensional surface of all points the eye touches is called the "horopter." The focal distance may take on a finite number of depths, or may vary infinitely. Light projected from the vergence distance is believed to be focused on the subject's eye 20, while light in front of or behind the vergence distance is blurred. Furthermore, it has been discovered that spatially coherent light with a beam diameter of less than about 0.7 millimeters is properly resolved by the human eye, regardless of where the eye focuses. With this understanding in mind, to create the illusion of proper depth of focus, eye convergence may be tracked using a mini camera (24), and the rendering engine (34) and projection subsystem (18) may be used to render all objects on or near the horopter in focus, and all other objects out of focus to various degrees (i.e., using intentionally created blur). Perspective light directing optical elements configured to project coherent light into the eye may be provided by a supplier such as Lumus, Inc. Preferably, the system (14) renders to the user at a frame rate of about 60 frames per second or greater. As explained above, preferably, the mini camera (24) may be utilized for eye tracking, and software may be configured to take up multiple geometric shapes as well as focus position cues that serve as user input. Preferably, such a system is configured with brightness and contrast suitable for daytime or nighttime use. In one embodiment, such a system preferably has a latency of less than about 20 milliseconds for visual object alignment, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 arcminute, which is approximately the limit of the human eye.The display system (14) may be integrated with a localization system, which may involve a GPS element, eye tracking, compass, accelerometer, and / or other data sources to assist in position and attitude determination, and the localization information may be utilized to facilitate accurate rendering within the user's field of view of the relevant world (i.e., such information would facilitate the glasses knowing where they are relative to the real world).

[0102] Other suitable display devices include desktop and mobile computers, smartphones, which may be augmented with additional software and hardware features that facilitate or simulate 3D perspective viewing (e.g., in one embodiment, a frame may be removably coupled to the smartphone, the frame featuring a 200 Hz gyro and accelerometer sensor subset, two small machine vision cameras with wide field of view lenses, and an ARM processor to simulate some of the functionality of the configuration featured in FIG. 14 ), tablet computers, which may be augmented as described above for smartphones, These include, but are not limited to, tablet computers augmented with additional processing and sensing hardware, head-mounted systems that use smartphones and / or tablets to display augmented and virtual viewpoints (visual adaptation via magnifying optics, mirrors, contact lenses, or light structuring elements), non-see-through displays of light-emitting elements (LCD, OLED, vertical cavity surface emitting laser, steered laser beams, etc.), see-through displays that allow a person to simultaneously see the natural world and artificially generated images (e.g., light directing optics, clear and polarized OLEDs inserted into close focus contact lenses, steered laser beams, etc.), contact lenses with light-emitting elements (such as those available from Innovega, Inc. (Bellevue, WA) under the trademark Ioptik RTM, which may be combined with specialized complementary eyeglass components), implantable devices with light-emitting elements, and implantable devices that simulate the photoreceptors of the human brain.

[0103] With a system such as those depicted in Figures 3 and 14, 3D points may be captured from the environment and the pose (i.e., vector and / or origin position information relative to the world) of the camera capturing these images or points may be determined so that these points or images may be "tagged" or associated with this pose information. The points captured by the second camera may then be utilized to determine the pose of the second camera. In other words, the second camera may be oriented and / or localized based on a comparison with the tagged image from the first camera. This knowledge may then be utilized to extract textures (since there are two aligned cameras around), create maps, and create a virtual copy of the real world. Thus, at a basic level, in one embodiment, one has a personally worn system that can be utilized to capture both the 3D points and the 2D images that generated the points, which may then be sent out to cloud storage and processing resources. They may also be cached locally with the pose information embedded (i.e., caching the tagged images), so the cloud may have the tagged 2D images (i.e., tagged with the 3D pose) ready (i.e., in the cache available) with the 3D points. If the user is observing something dynamic, they may send additional information up to the cloud related to the movement (e.g., when looking at another individual's face, the user may take the texture map of the face and boost it to an optimized frequency, even though the surrounding world is otherwise essentially static).

[0104] The cloud system may be configured to store some points as pose-only fiducials to reduce overall eye-tracking computations. In general, it may be desirable to have some contour features to enable tracking of key items in the user's environment, such as walls, tables, etc., as the user moves around a room, and the user may wish to "share" the world and let other users into the room and see these points. Such useful and key points may be referred to as "fiducials" since they are extremely useful as fixation points. They relate to features that can be recognized by machine vision and that can be consistently and repeatedly extracted from the world on different pieces of user hardware. Thus, these fiducials may preferably be stored in the cloud for further use.

[0105] In one embodiment, it is preferable to have a relatively even distribution of criteria throughout the relevant universe, as these are the types of items that a camera can easily use to recognize locations.

[0106] In one embodiment, the associated cloud computing arrangement may be configured to periodically groom the database of 3D points and any associated metadata to use the best data from various users for both refining the criteria and creating the world. In other words, the system may be configured to obtain the best data set by using input from various users who view and work within the associated world. In one embodiment, the database is fractal in nature, and as users get closer to an object, the cloud passes on higher resolution information to such users. As users map an object more closely, that data is sent to the cloud, which can add new 3D points and image-based texture maps to the database if they are better than those previously stored in the database. All of this may be configured to happen simultaneously from many users.

[0107] As described above, an augmented or virtual reality experience may be based on recognizing certain kinds of objects. For example, to recognize and understand a particular object, it may be important to understand that such an object has depth. A recognizer software object ("recognizer") may be deployed on a cloud or local resource to specifically assist in the recognition of various objects on either or both platforms as a user navigates the data in the world. For example, if a system has world model data comprising a 3D point cloud and pose tagged images, and there is a desk with a number of points on it, as well as an image of a desk, there may be no determination that what is being observed is actually a desk, as a human would grasp it. In other words, a few 3D points in space, and an image from somewhere in the space showing most of the desk, may not be enough to instantly recognize that a desk is being observed. To assist with this identification, a specific object recognizer may be created that will go into the raw 3D point cloud, segment the point set, and extract, for example, the top surface of the desk. Similarly, perceivers may be created to segment walls from 3D points so that a user can change wallpaper in virtual or augmented reality, or remove a portion of a wall, and have an entrance to another room that is not actually there in the real world. Such perceivers may be thought of as software "robots" that operate within the world model's data, crawling the world model and seeding that world model with semantic information, or an ontology regarding what is believed to exist between points in space. Such perceivers or software robots may be configured such that their entire existence is to orbit the relevant world data and find what is believed to be a wall, or a chair, or other item. They may be configured to tag point sets with the functional equivalent of "this point set belongs to a wall" or may comprise a combination of point-based algorithms and pose tagging image analysis to manually inform the system regarding what is in the points.

[0108] Object recognizers may be created for many purposes of varying utility depending on the viewpoint. For example, in one embodiment, a specialty coffee shop such as Starbucks may invest in creating an accurate recognizer of Starbucks coffee cups within the relevant world of data. Such recognizers may be configured to crawl the world of high and low volume data searches for Starbucks coffee cups, so that they may be segmented and identified to the user when operating within the relevant neighborhood space (i.e., perhaps when the user sees a Starbucks coffee cup over a period of time, the user will be served coffee at a nearby Starbucks outlet). Once the cup is segmented, it may be quickly recognized when the user moves it onto their desk. Such recognizers may be configured to operate or operate on local resources and data, or both on the cloud and locally, depending on available computational resources, as well as cloud computing resources and data. In one embodiment, there is a global copy of the world model on the cloud, with hundreds of users contributing to the global model, but for smaller worlds or sub-worlds, such as a particular person's office in a particular town, most of the world may not care what that office looks like, so the system may be configured to trim the data and go to locally cache information that is deemed most locally relevant to a given user.

[0109] In one embodiment, for example, an object identified as often moving, such as a cup on a desk, as a user walks up to the desk, may be configured such that the relevant information (such as segmentation of a particular cup on a desk) resides solely on the local computing resource, not on the cloud, so that cloud models and transmission burdens do not need to be incurred between the cloud and local resources. Thus, the cloud computing resource may be configured to segment 3D points and images, thus resolving permanent (i.e., generally non-moving) objects from movable ones, which affects where the relevant data remains and where it is processed, removing the processing burden from the wearable / local system for some data related to more permanent objects, allowing one-time processing of locations that are then shared with an unlimited number of other users, allowing multiple data sources to simultaneously build a database of fixed and movable objects at a particular physical location, and segmenting objects from the background to create object-specific fiducials and texture maps.

[0110] In one embodiment, the system may be configured to query the user for input regarding the identity of an object (e.g., the system may present the user with questions such as "Is that a Starbucks coffee cup?") so that the user can train the system and enable the system to associate semantic information with objects in the real world. The ontology may provide guidance regarding what objects segmented from the world can do, how they behave, etc. In one embodiment, the system may feature a virtual or actual keypad, such as a wirelessly connected keypad, connectivity to a smartphone keypad, or the like, to facilitate certain user input into the system.

[0111] The system may be configured to share primitives (walls, windows, desk geometry, etc.) with any user who enters the room in virtual or augmented reality, and in one embodiment, that individual's system would be configured to take images from specific viewpoints and upload them to the cloud. The cloud can then be populated with old and new sets of data and run optimization routines to establish criteria that exist on individual objects.

[0112] GPS and other localization information may be used as inputs to such processing. Additionally, other computing systems and data, such as a person's online calendar or FacebookR™ account information, may be used as inputs (e.g., in one embodiment, the cloud and / or local system may be configured to analyze the contents of a user's calendar for flights, dates, and destinations so that over time, information can be moved from the cloud to the user's local system so that it is ready for the user's arrival time at a given destination).

[0113] In one embodiment, tags such as QR codes and the like may be inserted into the world for use with non-statistical pose calculations, security / access control, communication of special information, spatial messaging, non-statistical object recognition, etc.

[0114] In one embodiment, cloud resources may be configured to pass digital models of real and virtual worlds between users, as described above with reference to "Passable Worlds," with the models rendered by individual users based on parameters and textures. This reduces bandwidth relative to passing real-time video, enables rendering of virtual viewpoints of the view, and allows millions or more users to participate in a single virtual gathering without transmitting each of the data (such as video) they need to see, as the view is rendered by local computing resources.

[0115] A virtual reality system ("VRS") may be configured to register user position and field of view (together known as "attitude") through real-time metric computer vision using cameras, simultaneous localization and mapping techniques, maps, and one or more of data from sensors such as gyros, accelerometers, compasses, barometers, GPS, radio signal strength triangulation, signal time of flight analysis, LIDAR ranging, RADAR ranging, odometry, and sonar ranging. A wearable device system may be configured to simultaneously map and orient. For example, in an unknown environment, a VRS may be configured to gather information about the environment and identify suitable reference points for user attitude calculation, other points for world modeling, and images to provide a texture map of the world. The reference points may be used to optically calculate the attitude. As the world is mapped with more detail, more objects may be segmented and given their own texture maps, but the world can still be represented at low spatial resolution, preferably with simple polygons with low-resolution texture maps. Other sensors such as those discussed above may be utilized to assist in this modeling effort. The world may be fractal in nature, in that moving (through viewpoint, "surveillance" mode, zooming, etc.) or otherwise seeking a better view requires higher resolution information from cloud resources. Moving closer to an object captures higher resolution data, which may be transmitted to the cloud, where new data may be calculated and / or inserted at interstitial sites in the world model.

[0116] With reference to FIG. 16, the wearable system may be configured to capture image information and extract fiducials and recognized points (52). The wearable local system may calculate the pose using one of the pose calculation techniques described below. The cloud (54) may be configured to use the images and fiducials to segment the 3D object from a more static 3D background, the images providing a texture map of the object and the world (the texture may be real-time video). The cloud resources (56) may be configured to store and make available the static fiducials and textures for world registration. The cloud resources may be configured to trim the point cloud for optimal point density for registration. The cloud resources (60) may be configured to store and make available the object fiducials and textures for object registration and manipulation, the cloud may trim the point cloud for optimal density for registration. The cloud resources may be configured to use all valid points and textures to generate a fractal solid model of the object (62), the cloud may trim the point cloud information for optimal fiducial density. The cloud resources (64) may be configured to query users for training on identifying segmented objects and worlds, and the ontology database may use the answers to instill actionable properties into the objects and worlds.

[0117] The following specific alignment and mapping modes feature the terms "O attitude", which represents attitude determined from an optical or camera system, "s attitude", which represents attitude determined from sensors (i.e., a combination of GPS, gyro, compass, accelerometer, etc. data, as discussed above), and "MLC", which represents cloud computing and data management resources.

[0118] The "Orient" mode creates a base map of a new environment, the purpose of which is to establish the user's pose when the new environment is not mapped or (equivalently) connected to the MLC. In the Orient mode, the wearable system extracts points from the image, tracks them from frame to frame, and triangulates fiducials using the S pose (since there are no fiducials extracted from the image). The wearable system may also filter out bad fiducials based on the user's persistence. It should be understood that the Orient mode is the most basic mode of registration and mapping, and will be effective even for low-precision poses all the time. However, after the wearable system has been used in relative motion for at least a small amount of time, a minimum fiducial set will be established such that the wearable system is set to recognize objects and map the environment using the O pose. As soon as the O pose is certain (with the minimum fiducial set), the wearable set is configured to exit the Orient mode. The "Map and O pose" mode is used to map the environment. The purpose of the Map and O pose mode is to establish a high-precision S pose, map the environment, and provide the map and image to the MLC. In this mode, the O pose is calculated from the mature world reference downloaded from the MLC and / or determined locally. However, it should be understood that the S pose may be used as a check on the calculated O pose and may also be used to accelerate the calculation of the O pose. As before, the wearable system extracts points from the image, tracks them from frame to frame, triangulates the references using the O pose, and filters out bad references based on persistence. The remaining reference and pose tagged images are then provided to the MLC cloud. It should be understood that these functions (extracting points, filtering out bad references, and providing reference and pose tagged images) do not need to be done in real time, but may be done at a later time to conserve bandwidth.

[0119] The O pose is used to determine the user's pose (user location and field of view). The goal of the O pose is to establish a highly accurate pose in an already mapped environment using minimal processing power. The calculation of the O pose involves several steps. To estimate the pose at n, the wearable system is configured to use historical data collected from the S and O poses (n-1, n-2, n-3, etc.). The pose at n is then used to project fiducials onto the image captured at n and create an image mask from the projection. The wearable system extracts points from the masked region and calculates the O pose from the extracted points and the mature world fiducials. It should be appreciated that the processing load is significantly reduced by only searching / extracting points from a masked subset of a particular image. Going one step further, the O pose calculated at n and the S pose at n may be used to estimate the pose at n+1. The pose-tagged images and / or videos may be transmitted to the MLC cloud.

[0120] The "super-resolution" mode may be used to create a super-resolution image and a reference. The composite pose tagged image may be used to create a super-resolution image, which in turn may be used to improve the reference position estimate. It should be understood that iterative pose estimation from the super-resolution reference and image. The above steps may be performed in real-time on the wearable device or transmitted to the MLC cloud and performed at a later time.

[0121] In one embodiment, a VRS system may be configured to have certain base functionality as well as functionality facilitated by "apps" or applications that may be delivered through the VRS to provide certain specialized functionality. For example, the following apps may be installed on a target VRS to provide specialized functionality:

[0122] A "painterly rendering" app may be used by an artist to create image transformations that represent the world as they see it. A user may then activate these transformations on their user device so that they can view the world "through the artist's eyes." A "tabletop modeling" app may allow a user to build an object from physical objects placed on a table. A "virtual presence" app may be used to hand a virtual model of a space to another user who may then move around that space using a virtual avatar.

[0123] An "avatar emotion" app may be used to measure aspects such as subtle voice inflections, slight head movements, body temperature, and heart rate, and animate subtle effects on a virtual being avatar. Digitizing human state information and passing it to a remote avatar uses less bandwidth than video. In addition, such data can be mapped to emotion-capable non-human avatars. For example, a dog avatar can indicate excitement by wagging its tail based on an excited voice inflection.

[0124] An efficient mesh network may be desirable for moving data as opposed to sending everything back to a server. However, many mesh networks have suboptimal performance because the location information and topology are not well characterized. In one embodiment, the system may be utilized to determine the location of all users with a relatively high degree of accuracy, and thus a mesh network configuration may be utilized for high performance.

[0125] In one embodiment, the system may be utilized for search. With augmented reality, for example, users will generate and leave content related to many aspects of the physical world. Most of this content is not text and therefore not easily searchable by typical methods. The system may be configured to provide a facility for tracking personal and social network content for search and reference purposes.

[0126] In one embodiment, if a display device tracks 2D points through successive frames and then fits a vector-valued function to the time evolution of these points, it is possible to sample the vector-valued function at any point in time (e.g., between frames) or at some point in the near future (by projecting the vector-valued function forward). This allows for the creation of high-resolution post-processing and prediction of future poses before the next image is actually captured (e.g., it is possible to double the alignment speed without doubling the camera frame rate).

[0127] For body-fixed rendering (as opposed to head-fixed or world-fixed rendering), an accurate representation of the body is desirable. Rather than measuring the body, in one embodiment, it is possible to derive its location through the average position of the user's head. If the user's face faces forward most of the time, a multi-day average of head position will reveal that orientation. Together with the gravity vector, this provides a reasonably stable coordinate frame for body-fixed rendering. Using a current measure of head position relative to this long-term coordinate frame allows for consistent rendering of objects on / around the user's body without extra instrumentation. For implementation of this embodiment, a single registration average of the head direction vector may be started, and the running sum of the data divided by delta t will yield the current average head position. Keeping approximately five registrations, starting on days n-5, n-4, n-3, n-2, n-1, allows for the use of a running average of only the past "n" days.

[0128] In one embodiment, a view may be scaled down and presented to a user in a space smaller than it actually is. For example, in a situation where a view must be rendered in a huge space (i.e., a soccer stadium, etc.), an equivalent huge space may not exist or such a large space may be inconvenient for the user. In one embodiment, the system may be configured to scale down the view so that a user may observe the view in miniature. For example, an individual may have a video game from a God's eye view, or a world championship soccer match, and play it on an unscaled playing field, or on a playing field that is scaled and presented on a living room floor. The system may be configured to simply shift the viewpoint, scale, and associated adaptive distance.

[0129] The system may also be configured to draw the user's attention to particular items within the presented view by manipulating the focus of virtual or augmented reality objects, highlighting them, changing their contrast, brightness, scale, etc.

[0130] Preferably, the system may be configured to achieve the following modes:

[0131] In open space rendering mode, the system is configured to capture key points from a structured environment and fill the spaces in between with renderings. This mode may be used to create potential venues such as stages, output spaces, large interior spaces, etc.

[0132] In object wrapping mode, the system is configured to recognize 3D objects in the real world and then augment them. "Recognition" in this context may mean identifying a 3D object with sufficient accuracy to link an image to the 3D object. It should be understood that recognition in this context may mean either classifying a type of object (e.g., an individual's face) and / or classifying a particular instance of an object (e.g., Joe, an individual). With these principles in mind, the recognizer software can be used to recognize a variety of things, such as walls, ceilings, floors, faces, roads, skies, skyscrapers, ranch houses, tables, chairs, cars, road signs, billboards, doors, windows, bookshelves, etc. Some recognizer software programs may be Type I and have general functionality (e.g., "Put my video on that wall," "That's a dog," etc.), while other recognizer software programs may be Type II and have specific functionality (e.g., "My TV is on the living room wall 3.2 feet away from the ceiling," "That's Fido," etc.).

[0133] In body-centric rendering, any rendered virtual objects are fixed to the user's body. For example, some objects may float around the user's body (e.g., the user's belt). Achieving this requires knowing the position of the body, not just the head. However, the body position may be estimated by the head position. For example, the head is usually facing forward, parallel to the ground. The body position may also be made more accurate over time by using data obtained by long-term averaging of the user's head position.

[0134] For type II recognition objects (concrete functionality), a cross-section of the object is typically shown. Furthermore, type II recognition objects may be linked to an online database of various 3D models. When starting the recognition process, it is ideal to start with an object that has a publicly available 3D model, such as a car or utility.

[0135] The system may also be used for virtual presence, i.e., allowing a user to draw an avatar of a remote person in a particular open space. This may be considered a subset of the aforementioned "open space rendering." A user may create a rough geometry of a local environment and repeatedly send both the geometry and texture maps to others. However, the user must give permission for others to enter the environment. Subtle audio cues, hand tracking, and head movements may be sent to the remote avatar. Based on the aforementioned information, the avatar may be animated. It is to be appreciated that the creation of a virtual presence may be used with minimal and economical use of bandwidth.

[0136] The system may also be configured to create object "entrances" to another room. In other words, instead of showing an avatar in a local room, a recognized object (e.g., a wall) may be used as an entrance to another user's environment. Thus, multiple users may sit in their respective rooms and peer "through" the walls into the environments of other users.

[0137] The system may also be configured to create a dense digital model of an area when a group of cameras (people) view the scene from different perspectives. The model may be renderable from any vantage point as long as the area is viewed through at least one camera. For example, a wedding scene may be rendered through the vantage points of multiple users. It should be understood that the perceiving device may distinguish and map stationary objects differently from moving objects (e.g., walls have a stable texture map, people have a higher frequency moving texture map).

[0138] As the rich digital model is updated in real time, the view can be rendered from any viewpoint. Returning to the wedding example, attendees in the back may fly up to the front row for a better view. Or, off-site attendees can find their "seat" using their avatars, or invisibly if the host allows. Attendees may show or hide their moving avatars. It should be understood that this aspect is likely to require extremely high bandwidth. High frequency data may be streamed through the cloud over a high speed local wireless connection, while low frequency data may originate from the MLC cloud. In the above example, since all the attendees at the wedding have high accuracy location information, it may be trivial to create optimal routing paths for local networking.

[0139] For communication to the system or between users, simple silent messaging is often desirable. For example, a finger-coding keyboard may be used. In an optional embodiment, a haptic glove solution may provide improved performance.

[0140] To give the user a full virtual reality experience, the visual system is dimmed and the user is shown a view that does not overlap with the real world. In this mode, a registration system may still be needed to track the user's head position. There may be several modes that can be used to experience full virtual reality. For example, in a "couch" mode, the user may be able to fly. In a "walk" mode, real-world objects may be re-rendered as virtual objects so that the user does not collide with the real world.

[0141] As a general rule, rendering body parts is essential to the user's suspension of disbelief when navigating through a virtual world. This requires having a method to track and render body parts within the user's field of view (FOV). For example, an opaque visor may be a form of virtual reality with many image enhancement possibilities. In another embodiment, a wide field of view may give the user a rear-view. In yet another embodiment, the system may include various forms of "super vision," such as telescopic, see-through, infrared, God's-eye view, etc.

[0142] In one embodiment, the system for virtual and / or augmented user experiences is configured such that a remote avatar associated with a user may be animated based at least in part on data on the wearable device with input from sources such as voice intonation analysis and facial recognition analysis as performed by associated software modules. For example, referring again to FIG. 12, a bee avatar (2) may be animated to have a friendly smile based on facial recognition of a smile on the user's face, or based on a friendly tone or voice, as determined by software configured to analyze audio input to a microphone that may capture audio samples locally from the user. Additionally, the avatar character may be animated in a manner that makes the avatar likely to express a certain emotion. For example, in an embodiment in which the avatar is a dog, a happy smile or voice tone detected by a system local to the human user may be expressed by the avatar as the dog avatar wagging its tail.

[0143] With reference to Figures 17-22, various aspects of the composite gaming embodiment are illustrated in the context of a spy-type game that may be thematically oriented with some of the spy themes presented in connection with the characters promoted under the trade name "James Bond 007" (RTM). With reference to Figure 17, an illustration of a family (84) is depicted with a member of the family (85) controlling a character in the game by operating an input device (88), such as a gaming joystick or controller, operably coupled to a gaming computer or console (86), such as one based on a personal computer or dedicated gaming system, such as that commercially available under the trade name "PlayStation" (RTM). The gaming console (86) is operably coupled to a display (92) configured to show a user interface view (92) to the pilot / operator (85) and others who may be in the vicinity. Figure 18 illustrates one example of such a user interface view (92), with the subject game being played on or near a bridge in the city of London, England. The user interface view (92) for this particular player (85) is pure virtual reality in that all elements of the user interface displayed do not actually exist in the player's (85) living room, i.e., they are virtual elements displayed using a monitor or display (element 90 in FIG. 17). Referring again to FIG. 18, the depicted virtual reality view (92) features a view of the city of London, featuring bridges (102) and various buildings (98) and other architectural features, along with a depiction of a game character (118 - also referred to as "Agent 009" in this illustrative example) operated by the subject player (85), from a perspective view as shown in the user interface view (92) of FIG. 18. Also displayed to the player (85) are a communication display (96), a compass indicator (94), a character status indicator (114), a news tool user interface (104), a social networking tool user interface (132), and a messaging user interface (112).Also shown is a representation of another character in the game (122 - also referred to as "agent 006" in this illustrative example). As shown in user interface view 92, the system may be configured to present information deemed relevant to the presented scene, such as a message that agent 006 is approaching, along with a visually presented highlight around the agent 006 character, through messaging interface 112. The system may be configured to allow the operator 85 to change the perspective being utilized from time to time. For example, rather than the helicopter-like perspective view shown in FIG. 18 (92), in which the player's own character 118 is shown above and below, the player may decide to select a view from the eye-point of such a character, or one of many other possible views that may be calculated and presented.

[0144] 19, another illustrative view (144) shows an actual human player operating as the character "Agent 006" (140) wearing a head mounted display system (300) and associated local processing system (308) participating in the same game being played by an operator (e.g., player 85 in FIG. 17) in his living room, walking through the actual city of London for a mixed or augmented reality experience. In the depicted embodiment, while the player (140) is walking along a bridge wearing his augmented reality head mounted display (300), his local processing system (308) feeds various virtual reality elements to the display, as depicted, which are overlaid on the actual view of reality (i.e., the actual skyline and structures of London 138, etc.). He carries in his hands one or more real documents (142) which, in one embodiment, have been printed out and previously electronically communicated to him for use in the gaming scenario. Figure 20 shows an illustration of a view (146) from the eye point of view of the player (140) looking at the real document (142) to see the real London skyline (138), while also being presented with various virtual elements for an augmented reality view through his head mounted display (300). The virtual elements may include, for example, a communications display (126), a news display (128), one or more electronic communication or social networking tool displays (132), one or more player status indicators (134), a messaging interface (136), a compass orientation indicator (124), and one or more displays (148) of content, such as text, audio, or video content, that may be retrieved and presented in accordance with other displayed or captured information, such as text or photographs, featured within an actual document (142) carried by the player (140).Another nearby character, "Agent 009," who exists only in virtual reality, is presented in the augmented reality view (146) of the player (140) acting as the character "Agent 006," and may be labeled as such in the user interface for easy identification, as shown in FIG. 20.

[0145] Referring to Figure 21, a player's eye view (152) of another player (150) is presented who also happens to be in London (138) in real life, walking across the same bridge towards "Agent 006" player (140), but without an augmented reality head mounted display (e.g., element 300 of Figure 19). This player (150) need not have a head mounted augmented reality display, but in this embodiment carries a mobile communications device (154), such as a tablet or smartphone, which is wirelessly connected to the larger system and utilized as a "window" into the augmented reality world of the subject game, and which may be configured to present device augmented reality information about one or two other nearby players, who may be real (158) or virtual (160), in a limited user interface (156), along with other augmented reality display information (162), such as alerts or character information.

[0146] With reference to FIG. 22, a "bird's eye" or manned or unmanned aerial vehicle (or "UAV") view is presented (164). In one embodiment, the view (164) may be based on a virtual UAV operated by another player, i.e., one of the aforementioned players. The depicted view (164) may be presented in a fully virtual mode to a player who may be seated on a couch at home, for example, with a large computer display (90) or a head-mounted display (300). Alternatively, such a view may be presented to a player who happens to be in an airplane or other airborne vehicle as an augmented reality view (i.e., "augmented" or mixed, since at least a portion of the view would be actual reality for a person in such a position). The illustrated view (164) contains an interface area for an information dashboard (170) featuring relevant information, such as information about parties identified and identified within the view. The depicted view (164) also features virtual highlight information, such as an informational focus (168), the location and / or status of other players or characters (166), and / or other informational presentation (167).

[0147] 23, for illustrative purposes, another augmented reality scenario is presented with a view (172) featuring certain real reality elements, such as the room structure (174), a coffee table (180), a DJ table (178), and five real people (176, 188, 182, 184, 186), who are wearing head-mounted augmented reality interface hardware (300) so that they may experience their own augmented reality view of their surroundings, such as a virtual reality cartoon character (198) for one of the real people (188), a virtual reality Spanish dancer character (196), a storm trooper character (194), and a headpiece with round rabbit ears (192), respectively. Without the augmented reality interface hardware, the room would face the five real people as it would be with the furniture, the DJ table, and nothing unusual. When augmented reality interface hardware is involved, the system is configured so that the involved players or participants may experience a person who has decided to appear in effect as a storm trooper, a person who has decided to appear in effect as a Spanish dancer, a person who has decided to appear in effect as a cartoon character, and a person who has decided to appear as if they were in normal clothing, but who wishes their head to be visualized with a headpiece (192) with round rabbit ears. The system may also be configured to show a virtual musical score (190) that may only be visible to the DJ (176) through its augmented reality interface hardware (300), or certain virtual features associated with the actual DJ table (178), such as DJ table lighting features, that may be visible to people around using its augmented reality interface hardware (300).

[0148] With reference to Figures 24A and 24B, adaptations of a mobile communication device, such as a tablet computer or smartphone, may be utilized to experience augmented reality as a modified "window" into the augmented reality world of a subject game or experience created using the subject system. With reference to Figure 24A, a typical smartphone or tablet computing system mobile device (154) features a relatively simple visual user interface (156) and typically has a simple camera or two cameras. With reference to Figure 24B, the mobile computing device is removably and operably coupled to an augmentation console (218) configured to increase the augmented reality participation capabilities of the mobile computing device. For example, the depicted embodiment may include two player-oriented cameras (202) that may be utilized for eye tracking, four speakers (200) that may be utilized for simple high quality audio and / or directional sound shaping, two forward-oriented cameras (204) for machine vision, alignment, and / or localization, additional battery or power source capability (212), one or more input interfaces (214, 216) that may be positioned for easy utilization by the player grasping the coupled system, and a single input interface (218) that may be configured to provide feedback to the coupled system. The system may feature a haptic feedback device (222) for providing a user with a sense of touch to the object (in one embodiment, the haptic feedback device may be configured to provide two-axis feedback in the + or - direction for each axis, and to provide directional feedback; such a configuration may be utilized, for example, to assist an operator in keeping the system aimed at a particular target, etc.), one or more GPS or localization sensors (206), and / or one or more accelerometers, inertial measurement units, and / or gyros (208).

[0149] With reference to FIG. 25, in one embodiment, a system such as that depicted in FIG. 24B may be utilized to roughly locate participants in a Cartesian direction of X and Y (analogous to the Earth's coordinates of latitude and longitude) using GPS sensors and / or wireless triangulation (232). A rough orientation may be achieved using a compass and / or wireless orientation techniques (234). Once a rough location and orientation has been determined, the distributed system may be configured to load (i.e., via wireless communication) local feature mapping information onto local devices (i.e., the interconnected mobile communication system 154 and the expansion console 218, etc.). Such information may comprise geometric information such as, for example, horizon geometry, architectural geometry, waterway / planar element geometry, landscape geometry, and the like (236). The local and distributed systems may use a combination of coarse location, coarse orientation, and local feature map information to determine fine location and orientation characteristics (such as X, Y, and Z {highly similar} coordinates and 3-D orientation) (238), which may be used to enable the distributed systems to load fine pitch local feature mapping information into the local systems (i.e., the interconnected mobile communication systems 154 and the expansion consoles 218, etc.) to improve user experience and operation. Movement to different orientations and locations may be tracked using coarse location and orientation tools as well as locally deployed devices such as inertial measurement units, gyros, and accelerometers that may be coupled to mobile computing systems such as tablets or mobile phones that may be carried by participants (242).

[0150] The head mounted display components in the various aforementioned embodiments may comprise transparent video configurations of monocular or binocular display technology. Additionally, such components may comprise wearable or head mounted light field display systems in monocular or binocular configurations, including laser projection systems, where an image is projected onto the user's retina and focal depth information is provided per voxel and / or frame. The number of depth planes may preferably range from 2 to infinity or very large numbers. In one embodiment, 4 to 36 depth planes may be provided for a 3-D effect.

[0151] Real objects, such as the DJ table (178) featured in Figure 23, may be augmented with virtual reality surfaces, shapes, and / or functionality. For example, in one embodiment, a real button on such a device may be configured to open a virtual panel configured to interact with the real device and / or other devices, people, or objects.

[0152] A room (174), such as the party room depicted in Figure 23, may be extrapolated to be any room or space. The system may have any location from some known data (such as existing 2- or 3-dimensional data about other associated structures or things in the room) or may have new zero data, and a machine vision configuration utilizing a camera (204), such as that mounted on the controller console (218) of Figure 24B, can be utilized to capture additional data. Additionally, the system may be configured so that a group of people may gather available 2- or 3-dimensional map information sources.

[0153] In configurations where existing map information is available, such as 3D map data for the city of London, a user wearing a head mounted display or "sensory product" configuration (300) may be roughly located using GPS, compass, and / or other means (additional fixed tracking cameras, devices linked to other players, etc.). Fine alignment may then be achieved from the user's sensors using the known geometry of the physical location as a reference for such alignment. For example, in a building specific to London, looking at distance X, once the system has located the user within Y feet from GPS information and in direction C from compass and map M, the system may be configured to implement an alignment algorithm (somewhat similar to techniques utilized in robotic or computer-assisted surgery) to "lock in" the user's 3D location to within a certain degree of error E.

[0154] Fixed cameras may also be utilized in conjunction with head-mounted or sensory product systems. For example, in a party room such as that depicted in FIG. 23, a fixed camera mounted on one side of the room (174) may be configured to provide a live, ongoing view of the room and the moving people, giving remote participants a "live" digital remote presence view of the entire room so that their social interactions with both the virtual and physical people in the room are much richer. In such an embodiment, several rooms may be mapped to each other; i.e., the physical room and virtual room geometries may be mapped to each other. Additional augmentations or visuals may be created to map both the physical and virtual "meta" rooms with objects traversing throughout them, making them equal to, less than, or greater than the physical room, and a visually customized or "skinned" version of the room may then be made available to each user or participant (i.e., while being able to be in exactly the same physical or virtual room, the system may allow custom views by users. For example, a virtual storm trooper (194) in FIG. 23 may be at a party but have his environment mapped with a "Death Star" motif or skin, while a DJ (176) may have a skinned room as shown in FIG. 23 with the party environment. Thus, in a "shared cinematic reality" concept, each user has a matching view of some aspects of the room, but can all simultaneously modify certain variations (color, shape, etc.) to suit their personal preferences.

[0155] Various exemplary embodiments of the present invention are described herein. Reference is made to these examples in a non-limiting sense. They are provided to illustrate the broader and more appreciable aspects of the present invention. Various changes may be made to the described invention, and equivalents may be substituted without departing from the true spirit and scope of the present invention. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, process act, or step to the objective, spirit, or scope of the present invention. Moreover, as will be understood by those skilled in the art, each of the individual variations described and illustrated herein has discrete components and features that can be easily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present invention. All such modifications are intended to be within the scope of the claims associated with this disclosure.

[0156] The present invention includes methods that may be performed using the subject devices. The methods may include the act of providing such a suitable device. Such providing may be performed by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, access, position, configure, activate, power on, or otherwise act to provide the requisite device in the subject method. The methods described herein may be carried out in any order of the recited events that is logically possible, as well as the recited order of events.

[0157] Exemplary aspects of the invention have been described above, along with details regarding material selection and manufacture. As for other details of the invention, these may be understood in conjunction with the above-referenced patents and publications, and may generally be known or understood by those skilled in the art. The same may be true with respect to method-based aspects of the invention in terms of additional acts as typically or logically adopted.

[0158] In addition, although the present invention has been described with reference to several embodiments that optionally incorporate various features, the present invention is not limited to those described and indicated, as each variation of the present invention is considered. Various modifications may be made to the invention described, and equivalents (whether described herein or not included for some simplicity) may be substituted without departing from the true spirit and scope of the present invention. In addition, when a range of values ​​is provided, it is to be understood that all intervening values ​​between the upper and lower limits of that range, and any other stated or intervening values ​​within that stated range, are encompassed within the present invention.

[0159] It is also contemplated that any optional feature of the described inventive variation may be described and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that the same item is present in plural. More specifically, as used herein and in the claims associated herewith, the singular forms "a," "an," "the," and "the" include plural referents unless otherwise specified. In other words, the use of articles allows for "at least one" of the subject item in the above description as well as in the claims associated herewith. Furthermore, it is noted that such claims may be drafted to exclude any optional element. Thus, this statement is intended to serve as a predicate for the use of exclusive terms such as "solely," "only," and the like, or the use of "negative" limitations in connection with the recitation of claim elements.

[0160] Without using such exclusive terms, the term "comprising" in the claims associated with this disclosure shall allow for the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claims or whether the addition of features can be considered as a transformation of the nature of the elements recited in such claims. Except as specifically defined herein, all technical and scientific terms used herein are to be given the broadest possible commonly understood meaning while maintaining the validity of the claims.

[0161] The breadth of the present invention is not intended to be limited by the examples and / or subject specification provided, but rather is intended to be limited only by the scope of the claim language associated with this disclosure.

Claims

1. 1. A method comprising: capturing an image of a real physical object by one or more sensors of a wearable display system located within a first physical environment, the real physical object being located within a field of view of a first user and the location of the real physical object within the first physical environment being known to the wearable display system; providing, by the wearable display system, within the field of view of the first user, an interface having a first interface feature and a second interface feature, the first and second interface features, when activated, each configuring the wearable display system for a separate and distinct visualization mode of a plurality of visualization modes including a mixed reality mode, the mixed reality mode being a combination of a virtual reality mode and an augmented reality mode; rendering, by the wearable display system, a rendered physical object from the captured image of the actual physical object based at least in part on a digital model for the actual physical object, wherein the rendered physical object appears substantially identical to the actual physical object; and displaying, via the interface, the rendered physical object to the first user at a display location within a display in the mixed reality mode selected by the first user from the plurality of visualization modes, the display location corresponding to an area of ​​the display in which the actual physical object is viewable, the mixed reality mode comprising displaying a virtual object and the rendered physical object in the first physical environment of the first user, the rendered physical object being displayed in place of the actual physical object on the display of the wearable display system; capturing a first image of a first real physical object at a first location and at a first time by the one or more sensors of the wearable display system, the first real physical object being located within the field of view of the first user and the first location of the first real physical object being known to the wearable display system; rendering a first rendered physical object that appears substantially identical to the first actual physical object based at least in part on the first captured image of the first actual physical object; displaying the first rendered physical object to the first user in a first display location associated with the area for the rendered physical object in the display in the mixed reality mode selected by the first user from the plurality of visualization modes, wherein both the actual physical object and the first rendered physical object are perceived by the first user through the display, and the first rendered physical object is displayed in place of the first physical object in the display; identifying, by the one or more sensors of the wearable display system, a second real physical object at a second location and at a second time, the second real physical object being located within the field of view of the first user and the second location of the second real physical object being known to the wearable display system; generating an interaction between the second real physical object and the first rendered physical object relative to the rendered physical object; reflecting an effect of the interaction of the second actual physical object on the first rendered physical object by rendering a different first rendered physical object relative to the first physical object; A method comprising:

2. 1. A method comprising: capturing an image of a real physical object by one or more sensors of a wearable display system located within a first physical environment, the real physical object being located within a field of view of a first user and the location of the real physical object within the first physical environment being known to the wearable display system; providing, by the wearable display system, within the field of view of the first user, an interface having a first interface feature and a second interface feature, the first and second interface features, when activated, each configuring the wearable display system for a separate and distinct visualization mode of a plurality of visualization modes including a mixed reality mode, the mixed reality mode being a combination of a virtual reality mode and an augmented reality mode; rendering, by the wearable display system, a rendered physical object from the captured image of the actual physical object based at least in part on a digital model for the actual physical object, wherein the rendered physical object appears substantially identical to the actual physical object; and displaying, via the interface, the rendered physical object to the first user at a display location within a display in the mixed reality mode selected by the first user from the plurality of visualization modes, the display location corresponding to an area of ​​the display in which the actual physical object is viewable, the mixed reality mode comprising displaying a virtual object and the rendered physical object in the first physical environment of the first user, the rendered physical object being displayed in place of the actual physical object on the display of the wearable display system; scanning and rendering the first user as a first rendered user; determining a spatial relationship between the first user and the actual physical object located at a first geographic location; transposing the first rendered user and the rendered physical object to a different wearable display system of a different user in a second geographic location while maintaining the spatial relationship; rendering the first rendered user and the rendered physical object through the different wearable display system within a different field of view of the different user based at least in part on a result of transposing the first rendered user and the rendered physical object by the different wearable display system and the spatial relationship between the first rendered user and the rendered physical object; identifying, at the different wearable display system in the mixed reality mode of the plurality of visualization modes, an interaction by the different user with the first user or the actual physical object through the different wearable display system operating in an augmented reality mode of the plurality of visualization modes; A method comprising:

3. receiving, at the wearable display system in the mixed reality model, from the different wearable display system in the augmented reality mode, a response to the interaction by the first user through the wearable display system operating in the mixed reality mode of the plurality of visualization modes; transmitting a data stream in response to the second interaction from the wearable display system to the different wearable display system; and The method of claim 2 , further comprising:

4. 1. A method comprising: capturing an image of a real physical object by one or more sensors of a wearable display system located within a first physical environment, the real physical object being located within a field of view of a first user and the location of the real physical object within the first physical environment being known to the wearable display system; providing, by the wearable display system, within the field of view of the first user, an interface having a first interface feature and a second interface feature, the first and second interface features, when activated, each configuring the wearable display system for a separate and distinct visualization mode of a plurality of visualization modes including a mixed reality mode, the mixed reality mode being a combination of a virtual reality mode and an augmented reality mode; rendering, by the wearable display system, a rendered physical object from the captured image of the actual physical object based at least in part on a digital model for the actual physical object, wherein the rendered physical object appears substantially identical to the actual physical object; and displaying, via the interface, the rendered physical object to the first user at a display location within a display in the mixed reality mode selected by the first user from the plurality of visualization modes, the display location corresponding to an area of ​​the display in which the actual physical object is viewable, the mixed reality mode comprising displaying a virtual object and the rendered physical object in the first physical environment of the first user, the rendered physical object being displayed in place of the actual physical object on the display of the wearable display system; transmitting, in real time or near real time between the wearable display system and a display system operated by a second user located in a second physical environment remote from the first physical environment, a first data stream representing a first interaction by the first user with the rendered physical object relative to one or more actual physical objects in the first physical environment, to the display system operated by the second user, the display system comprising or operably coupled to a desktop display monitor or television set; receiving, at the wearable display system, a second data stream representing a first response by the second user to the first interaction; A method comprising:

5. identifying a physical feature on the actual physical object; Extending said physical characteristic with at least some functionality. Further comprising: The extending step includes at least Detecting the first interaction by the first user with the physical feature; responsive to detecting the interaction, rendering a virtual interface or panel in relation to the physical feature; detecting a second interaction by the first user with the virtual interface or panel; performing, by the wearable display system, an action to accomplish the function in response to the second interaction; and rendering in real time or near real time within the field of view of the first user through the wearable display system in the second physical environment remote from the first physical environment perceived by the first user using the wearable display system, the user object being dynamically controlled by the second user via the display system operatively coupled to or comprising the desktop display monitor or the television set and representing the second user; The method of claim 4, wherein the method is carried out by

6. creating a digital world for at least a portion of the first physical environment, the digital world including at least the rendered physical objects, the user object representing the second user, and a separate user object within the digital world representing the first user; Sharing the digital world between the wearable display system in the first physical environment and the display system in the second physical environment. The method of claim 4 further comprising:

7. receiving, in the wearable display system, the first interaction by the first user with one or more real physical objects in the physical environment in which the first user is located, the one or more real physical objects being different from the real physical objects from which the rendered physical object is created; transmitting, in near real-time, from the wearable display system to the display system, the first data stream for the first interaction by the first user with the one or more actual physical objects in the first physical environment that differ from the rendered physical objects, for rendering the first interaction on the desktop display monitor; Further comprising:

7. The method of claim 6, wherein at least a portion of the digital world is shared with the display system operated by the second user located in the second physical environment, the second user having the desktop display monitor or the television set or operably coupled to the desktop display monitor or the television set and remote from the first environment.

8. receiving, at the wearable display system in near real time, the second stream of data for a response by the second user to the first interaction by the first user using the display system to render the first interaction on the wearable display system; re-rendering, in near real-time, the user object to the second user using the second data stream to reflect the response by the second user to the first interaction; The method of claim 7 further comprising:

9. receiving, at the wearable display system in near real time, a third data stream for the second interaction by the second user via the user object controlled by the second user via the display system to render the second interaction on the wearable display system; re-rendering the user object for the second user in the wearable display system in near real-time using the third data stream to reflect the second interaction by the second user via the user object that is dynamically controlled by the second user in the second physical environment; The method of claim 8 further comprising:

Images