System and method for virtual reality and augmented reality

Abstract

To provide a system and a method for preferable virtual reality and augmented reality.SOLUTION: A computer implemented method of facilitating communications between first and second users, includes displaying, by a first head mounted device, a first virtual object to the first user wearing the first head mounted device. The method further includes displaying, by a second head mounted device, a second virtual object to the second user wearing the second head mounted device. The method further includes facilitating, by the first and second head mounted devices, communications between the first and second users by using the first and second virtual objects to simulate the first and second users being present in a common environment.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to connected mobile computing systems, methods, and configurations, and more particularly to mobile computing systems, methods, and configurations featuring at least one wearable component that may be utilized for virtual reality and / or augmented reality operations. [Background technology]

[0002] It is desirable for a mixed reality or augmented reality eyepiece display to be lightweight, low cost, have a small form factor, have a wide virtual image field of view, and be as transparent as possible. In addition, it is desirable to have a configuration that presents virtual image information in multiple focal planes (e.g., two or more) to be practical for a wide variety of use cases without exceeding acceptable tolerances for vergence-divergence and accommodation misalignment. Referring to FIG. 1, an augmented reality system is illustrated that features a head-mounted viewing component (2), a handheld controller component (4), and an interconnected auxiliary computing or controller component (6) that may be configured to be worn as a beltpack or equivalent on a user. Each of these components may be operatively coupled (10, 12, 14, 16, 17, 18) to each other and to other connected resources (8), such as cloud computing or cloud storage resources, via wired or wireless communication configurations, such as those defined by IEEE 802.11, Bluetooth (RTM), and other connectivity standards and configurations. For an augmented reality experience, various aspects of such components are described, such as various embodiments of two rendered optical elements (20) through which a user may see the world around them in addition to the visual components that may be produced by the associated system components. As illustrated in FIG. 1, such a system may also comprise various sensors configured to provide information about the user's surrounding environment, including, but not limited to, various camera type sensors and / or emitters (22, 24, 26) (such as monochrome, color / RGB, infrared, and / or thermal), a depth camera sensor (28), and / or a sound sensor (30) such as a microphone. There is a need for small and persistently connected wearable computing systems and assemblies such as those described herein that can be utilized to provide a user with the perception of a rich augmented reality experience, including experiences related to certain aspects of the natural world, such as, but not limited to, aspects of the many configurations of the modern corporate work environment world. Summary of the Invention [Means for solving the problem]

[0003] Embodiments are directed to connected mobile computing systems, methods, and configurations. In particular, embodiments are directed to mobile computing systems, methods, and configurations featuring at least one wearable component that may be utilized for virtual reality and / or augmented reality operations.

[0004] In one embodiment, a computer-implemented method of facilitating communication between a first and a second user includes displaying, by a first head mounted device, a first virtual object to a first user wearing the first head mounted device. The method also includes displaying, by a second head mounted device, a second virtual object to a second user wearing the second head mounted device. The method further includes using the first and second virtual objects, by the first and second head mounted devices, to facilitate communication between the first and second users to simulate the first and second users being in a common environment.

[0005] In one or more embodiments, the first virtual object is an avatar of the second user. The second virtual object may be an avatar of the first user. The method may include transmitting, by the first processor, the second virtual object to the second head-mounted device. The method may include transmitting, by the second processor, the first virtual object to the first head-mounted device.

[0006] In another embodiment, a computer-implemented method of sharing a virtual experience includes displaying, by a first head mounted device, a virtual object to a first user wearing the first head mounted device. The method also includes displaying, by a second head mounted device, the virtual object to a second user wearing the second head mounted device. The method further includes facilitating, by the first and second head mounted devices, communication between the first and second users related to the virtual object.

[0007] In one or more embodiments, a first user is located at a first geographic location and a second user is located at a second geographic location different from the first geographic location. The method may include modifying the virtual object by the first user. The method may include modifying the virtual object by the second user. The method may include communicating, by the first and second head mounted devices, the modifications to the virtual object between the first and second head mounted devices. The first and second head mounted devices may be communicatively coupled by a network.

[0008] In one or more embodiments, the method includes providing an input related to a virtual object by a first user through a first user interface device operably coupled to a first head-mounted device. The first user interface device may be selected from the group consisting of a tactile interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device. The method may include providing an input related to a virtual object by a second user through a second user interface device operably coupled to a second head-mounted device. The second user interface device may be selected from the group consisting of a tactile interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device.

[0009] In one or more embodiments, the method includes providing an input related to the virtual object by a sensor operably coupled to the first and second head-worn devices. The sensor may be selected from the group consisting of a camera and an optical detection device. The sensor may face outward. The sensor may face inward toward the user. The method may also include projecting infrared structured light by the sensor. The input may be selected from the group consisting of pupil size, pupil position, pupil angle, eye sphericity, eye shape, figure motion, and anatomical data associated with the user. The method may include determining, by the processor, a visual focus of the user using the input. The method may include mapping, by the processor, a portion of the user's environment using the input.

[0010] In one or more embodiments, the method includes providing environmental input related to the geographic location by environmental sensors operably coupled to the first and second head-worn devices. The environmental sensors may be selected from the group consisting of a camera, an optical detection device, a motion capture device, a depth measurement device, a facial recognition device, an object recognition device, an audio recognition device, a CMOS sensor, a field of view sensor, and an optical enhancement sensor. The environmental input may be selected from the group consisting of motion capture data, gesture recognition data, depth sensing data, facial recognition data, object recognition data, unique object feature recognition data, audio recognition data, voice recognition data, acoustic source localization data, noise reduction data, infrared light data, laser light data, CMOS sensor data, field of view data, and optical enhancement data.

[0011] In one or more embodiments, the virtual object corresponds to a computer-aided design file. The virtual object may correspond to a building information modeling system configuration. The virtual object may be a construction, and the first and second head-mounted devices may be configured to display a virtual tour of the construction to the first and second users, respectively. The virtual object may be a three-dimensional drawing of an object to be manufactured. The object to be manufactured may be a component of an automobile.

[0012] In one or more embodiments, the virtual object is associated with an educational experience. The virtual object may be selected from the group consisting of a laboratory object, an object from a library, a three-dimensional video, a scientific model, and a natural phenomenon. The first and second head-mounted devices may be configured to facilitate a one-to-one educational experience between the first and second users. The first and second head-mounted devices may be configured to facilitate a one-to-many educational experience.

[0013] In yet another embodiment, a computer-implemented method of sharing a real-world experience includes capturing, by a first head-mounted device, an image of a real-world object in an environment of a first user wearing the first head-mounted device. The method also includes generating, by a processor, a virtual object based on the image of the real-world object. The method further includes displaying, by a second head-mounted device, the virtual object to a second user wearing the second head-mounted device. The method also includes facilitating communication between the first user and the second user related to the real-world object, by the first and second head-mounted devices.

[0014] In one or more embodiments, the first user is located at a first geographic location and the second user is located at a second geographic location different from the first geographic location. The virtual object may correspond to a real-world object. The method also includes generating, by the processor, a mesh or a point cloud based on the image of the real world. The second head-mounted device may be configured to enable the second user to modify the virtual object. The first head-mounted device may be configured to display the first virtual object to the first user based on the second user modifying the virtual object. The first and second head-mounted devices may be configured to communicate modifications to the virtual object between the first and second head-mounted devices. The first and second head-mounted devices may be communicatively coupled by a network.

[0015] In one or more embodiments, the first and second head-mounted devices and processors are configured to display a view of the virtual object to the second user that is substantially identical to a view of the real-world object to the first user. The second user may have expertise related to the real-world object. The second user may have access to information about the real-world object.

[0016] In one or more embodiments, the method includes sensing, by a first head-mounted device, a texture of the real-world object. The method may also include generating, by a processor, a texture of the virtual object based on the texture of the real-world object. The method may further include displaying, by a second head-mounted device, the texture of the virtual object to a second user.

[0017] In one or more embodiments, the method includes detecting, by a vibration sensor operably coupled to the first head-mounted device, vibrations associated with the real-world object. The method may also include generating, by a processor, vibrations associated with the virtual object based on the vibrations associated with the real-world object. The method may further include providing, to a second user, the vibrations associated with the virtual object by a tactile interface device operably coupled to the second head-mounted device.

[0018] In one or more embodiments, the method includes detecting an odor associated with the real-world object by an olfactory sensor operably coupled to the first head-mounted device. The method may also include generating, by a processor, an odor associated with the virtual object based on the odor associated with the real-world object. The method may further include providing, to the second user, an odor associated with the virtual object by an olfactory interface device operably coupled to the second head-mounted device.

[0019] In one or more embodiments, the real-world object and the virtual object are both associated with a medical procedure at the first geographic location. Communication between the first and second users associated with the virtual objects may facilitate performance of the medical procedure by the first user. The medical procedure may be selected from the group consisting of a surgical procedure, a patient diagnosis, and an in-situ medical procedure.

[0020] In one or more embodiments, the real-world object and the virtual object are both related to an emergency incident at the first geographic location. The second user may have access to information about the first geographic location. The information about the first geographic location may be selected from the group consisting of a CAD drawing of the geographic location, an overview of gas and electrical architecture at the geographic location, and camera information at the geographic location. The second head-mounted device may be configured to prompt the second user to modify the first geographic location. The first geographic location may comprise a structure where a fire is occurring. The communication between the first user and the second user related to the real-world object may include eye tracking information about the first user or the second user.

[0021] In one or more embodiments, both the real-world object and the virtual object are associated with a retail transaction in the first geographic location. The retail transaction may be selected from the group consisting of tracking inventory, managing inventory, sales training, managing security scenarios, facilitating return transactions, and assisting customers with product selection.

[0022] In one or more embodiments, both the real-world object and the virtual object are associated with an entertainment experience. The first and second head-mounted devices may be configured to facilitate immersive news reporting. The first and second head-mounted devices may be configured to facilitate entertainment incorporating elements from a first geographic location of the first user. The method may include displaying a generated virtual object to the first user by the first head-mounted device, the virtual object may relate to a real-world object. The generated virtual object may be selected from the group consisting of text and audio-visual clips. The method may include receiving, by the first head-mounted device, a user input from the first user indicating the real-world object. The method may include displaying, on the first head-mounted device, the generated virtual object to the first user in response to receiving the user input from the first user indicating the real-world object. The user input may be selected from the group consisting of a user gaze, a user gesture, and a user manipulation of a controller operably coupled to the first head-mounted device.

[0023] The above and other embodiments of the present invention are described in the detailed description that follows. The present invention provides, for example, the following: (Item 1) 1. A computer-implemented method for facilitating communication between a first and a second user, the method comprising: displaying, by a first head-mounted device, a first virtual object to the first user wearing the first head-mounted device; displaying, by a second head-mounted device, a second virtual object to the second user wearing the second head-mounted device; facilitating communication between the first and second users using the first and second virtual objects by the first and second head-mounted devices to simulate the first and second users being present in a common environment; A method comprising: (Item 2) 2. The method of claim 1, wherein the first virtual object is an avatar of the second user. (Item 3) 2. The method of claim 1, wherein the second virtual object is an avatar of the first user. (Item 4) 2. The method of claim 1, further comprising transmitting, by a first processor, the second virtual object to the second head-mounted device. (Item 5) 2. The method of claim 1, further comprising transmitting, by a second processor, the first virtual object to the first head-mounted device. (Item 6) 1. A computer-implemented method for sharing a virtual experience, the method comprising: displaying, by a first head-mounted device, a virtual object to a first user wearing the first head-mounted device; displaying, by a second head-mounted device, the virtual object to a second user wearing the second head-mounted device; facilitating communication between the first user and a second user related to the virtual object by the first and second head-mounted devices; A method comprising: (Item 7) the first user is located in a first geographic location; 7. The method of claim 6, wherein the second user is located at a second geographic location different from the first geographic location. (Item 8) 8. The method of claim 7, further comprising modifying the virtual object by the first user. (Item 9) 8. The method of claim 7, further comprising modifying the virtual object by the second user. (Item 10) 8. The method of claim 7, further comprising communicating, by the first and second head mounted devices, modifications to the virtual object between the first and second head mounted devices. (Item 11) 8. The method of claim 7, wherein the first and second head-mounted devices are communicatively coupled by a network. (Item 12) 8. The method of claim 7, further comprising providing, by the first user, an input related to the virtual object via a first user interface device operably coupled to the first head-mounted device. (Item 13) Item 13. The method of item 12, wherein the first user interface device is selected from the group consisting of a tactile interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device. (Item 14) 8. The method of claim 7, further comprising providing, by the second user, an input related to the virtual object via a second user interface device operably coupled to the second head-mounted device. (Item 15) Item 15. The method of item 14, wherein the second user interface device is selected from the group consisting of a tactile interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device. (Item 16) 8. The method of claim 7, further comprising providing input related to the virtual object by sensors operably coupled to the first and second head-mounted devices. (Item 17) Item 17. The method of item 16, wherein the sensor is selected from the group consisting of a camera and an optical detection device. (Item 18) Item 17. The method of item 16, wherein the sensor faces outward. (Item 19) Item 17. The method of item 16, wherein the sensor faces inward toward a user. (Item 20) 17. The method of claim 16, further comprising projecting infrared structured light by the sensor. (Item 21) 17. The method of claim 16, wherein the input is selected from the group consisting of pupil size, pupil position, pupil angle, eye sphericity, eye shape, figure motion, and anatomical data associated with a user. (Item 22) 20. The method of claim 16, further comprising determining, by a processor, a visual focus of a user using the input. (Item 23) 20. The method of claim 16, further comprising: mapping, by a processor, a portion of the user's environment using the input. (Item 24) 8. The method of claim 7, further comprising providing environmental input related to a geographic location by an environmental sensor operably coupled to the first and second head-mounted devices. (Item 25) 25. The method of claim 24, wherein the environmental sensor is selected from the group consisting of a camera, an optical detection device, a motion capture device, a depth measurement device, a facial recognition device, an object recognition device, an audio recognition device, a CMOS sensor, a field of view sensor, and an optical enhancement sensor. (Item 26) 25. The method of claim 24, wherein the environmental input is selected from the group consisting of motion capture data, gesture recognition data, depth sensing data, facial recognition data, object recognition data, unique object feature recognition data, audio recognition data, voice recognition data, acoustic source localization data, noise reduction data, infrared light data, laser light data, CMOS sensor data, field of view data, and optical enhancement data. (Item 27) 8. The method of claim 7, wherein the virtual object corresponds to a computer-aided design file. (Item 28) 8. The method of claim 7, wherein the virtual object corresponds to a building information modeling system configuration. (Item 29) the virtual object is a construction; 8. The method of claim 7, wherein the first and second head-mounted devices are configured to display a virtual tour of the construction to the first and second users, respectively. (Item 30) 8. The method according to claim 7, wherein the virtual object is a three-dimensional drawing of an object to be manufactured. (Item 31) 31. The method according to claim 30, wherein the object to be manufactured is an automobile component. (Item 32) 8. The method of claim 7, wherein the virtual object is related to an educational experience. (Item 33) 33. The method of claim 32, wherein the virtual object is selected from the group consisting of laboratory objects, objects from a library, three-dimensional videos, scientific models, and natural phenomena. (Item 34) Item 33. The method of item 32, wherein the first and second head-mounted devices are configured to facilitate a one-to-one educational experience between the first and second users. (Item 35) Item 33. The method of item 32, wherein the first and second head-mounted devices are configured to facilitate a one-to-many educational experience. (Item 36) 1. A computer-implemented method for sharing a real-world experience, the method comprising: capturing, by a first head-mounted device, images of real-world objects in an environment of a first user wearing the first head-mounted device; generating, by a processor, a virtual object based on an image of the real-world object; displaying, by a second head-mounted device, the virtual object to a second user wearing the second head-mounted device; facilitating communication between the first user and a second user related to the real-world object by the first and second head-mounted devices; A method comprising: (Item 37) the first user is located in a first geographic location; Item 37. The method of item 36, wherein the second user is located at a second geographic location different from the first geographic location. (Item 38) Item 37. The method of item 36, wherein the virtual object corresponds to the real-world object. (Item 39) 37. The method of claim 36, further comprising generating, by the processor, a mesh or a point cloud based on the real world image. (Item 40) Item 37. The method of item 36, wherein the second head-mounted device is configured to allow the second user to modify the virtual object. (Item 41) Item 41. The method of item 40, wherein the first head-mounted device is configured to display a first virtual object to the first user based on the second user modifying the virtual object. (Item 42) Item 37. The method of item 36, wherein the first and second head mounted devices are configured to communicate modifications to the virtual object between the first and second head mounted devices. (Item 43) Item 37. The method of item 36, wherein the first and second head-mounted devices are communicatively coupled by a network. (Item 44) Item 37. The method of item 36, wherein the first and second head-mounted devices and the processor are configured to display a view of the virtual object to the second user that is substantially identical to a view of the real-world object to the first user. (Item 45) 37. The method of claim 36, wherein the second user has expertise related to the real-world object. (Item 46) Item 37. The method of item 36, wherein the second user has access to information about the real-world object. (Item 47) sensing, by the first head-mounted device, a texture of the real-world object; generating, by the processor, a texture for the virtual object based on a texture of the real-world object; displaying a texture of the virtual object to the second user by the second head-mounted device; and 37. The method of claim 36, further comprising: (Item 48) detecting, with a vibration sensor operably coupled to the first head-mounted device, a vibration associated with the real-world object; generating, by the processor, vibrations associated with the virtual object based on the vibrations associated with the real-world object; providing the vibrations associated with the virtual object to the second user via a tactile interface device operably coupled to the second head-mounted device; and 37. The method of claim 36, further comprising: (Item 49) detecting an odor associated with the real-world object with an olfactory sensor operably coupled to the first head-worn device; generating, by the processor, an odor associated with the virtual object based on the odor associated with the real-world object; providing the second user with the odor associated with the virtual object via an olfactory interface device operably coupled to the second head-mounted device; and 37. The method of claim 36, further comprising: (Item 50) 38. The method of claim 37, wherein the real-world object and the virtual object are both related to a medical procedure in the first geographic location. (Item 51) 51. The method of claim 50, wherein the communication between the first and second users related to the virtual object facilitates performance of the medical procedure by the first user. (Item 52) 51. The method of claim 50, wherein the medical procedure is selected from the group consisting of a surgical procedure, a patient diagnosis, and an in-situ medical procedure. (Item 53) 38. The method of claim 37, wherein the real-world object and the virtual object are both related to an emergency incident in the first geographic location. (Item 54) Item 54. The method of item 53, wherein the second user has access to information about the first geographic location. (Item 55) 55. The method of claim 54, wherein the information about the first geographic location is selected from the group consisting of a CAD drawing of the geographic location, an overview of gas and electrical architecture at the geographic location, and camera information at the geographic location. (Item 56) Item 54. The method of item 53, wherein the second head-mounted device is configured to prompt the second user to modify the first geographic location. (Item 57) Item 54. The method of item 53, wherein the first geographic location comprises a structure in which a fire is occurring. (Item 58) Item 54. The method of item 53, wherein the communication between the first user and the second user related to the real-world object includes eye tracking information regarding the first user or the second user. (Item 59) Item 38. The method of item 37, wherein the real-world object and the virtual object are both associated with a retail transaction in the first geographic location. (Item 60) 60. The method of claim 59, wherein the retail transaction is selected from the group consisting of tracking inventory, managing inventory, sales training, managing security scenarios, facilitating return transactions, and assisting customers with product selection. (Item 61) 38. The method of claim 37, wherein both the real-world object and the virtual object are associated with an entertainment experience. (Item 62) Item 62. The method of item 61, wherein the first and second head-mounted devices are configured to facilitate immersive news reporting. (Item 63) Item 62. The method of item 61, wherein the first and second head-mounted devices are configured to facilitate entertainment incorporating elements from a first geographic location of the first user. (Item 64) displaying, by the first head-mounted device, a virtual object generated by the first user; Item 62. The method of item 61, wherein the virtual object is related to the real-world object. (Item 65) Item 65. The method of item 64, wherein the generated virtual object is selected from the group consisting of text and audiovisual clips. (Item 66) receiving, by the first head-mounted device, user input from the first user indicative of the real-world object; displaying, on the first head mounted device, the generated virtual object to the first user in response to receiving the user input from the first user indicating the real world object; Item 65. The method of item 64, further comprising: (Item 67) Item 67. The method of item 66, wherein the user input is selected from the group consisting of a user gaze, a user gesture, and a user operation of a controller operably coupled to the first head-mounted device. [Brief description of the drawings]

[0024] The above and other aspects of the embodiments will be described in further detail with reference to the accompanying drawings, in which identical elements in different figures are referred to by common reference numerals.

[0025] [Figure 1] FIG. 1 diagrammatically depicts an augmented reality system, according to some embodiments.

[0026] [Diagram 2] FIG. 2 diagrammatically depicts a spatial computing system that allows multiple users to collaboratively view and / or manipulate virtual objects / images, according to some embodiments.

[0027] [Diagram 3] FIG. 3 diagrammatically depicts a spatial computing system that allows remote users to collaborate with local users, according to some embodiments.

[0028] [Figure 4] FIG. 4 is a block diagram that diagrammatically depicts an illustrative computing system, in accordance with some embodiments.

[0029] In order to more clearly understand how the above-listed and other advantages and objects of the various embodiments are obtained, a more detailed description of the embodiments is provided with reference to the accompanying drawings. It should be noted that the drawings are not drawn to scale, and that elements of similar structure or function are represented by like reference numerals throughout. It should be understood that these drawings depict only certain illustrated embodiments, and therefore should not be considered as limiting the scope of the embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS In various embodiments, a user may wear an augmented reality system such as that depicted in FIG. 1, which may also be referred to as a "spatial computing" system, in connection with interaction with the three-dimensional world around the user when such a system is operated (32). Such a system may, for example, include a head-mountable display component (2) and may feature environmental sensing capabilities as described above, such as various types of cameras that may be configured to map the user's surrounding environment or generate a "mesh" or "point cloud" for such environment. Spatial computing systems, including but not limited to systems providing augmented reality (AR), virtual reality (VR), and / or mixed reality (MR) display functionality, may be employed to facilitate various activities. For example, spatial computing systems may improve the efficiency of communication and collaboration for activities involving the interaction of multiple individuals, including individuals who may be geographically remote from one another.

[0031] Embodiments provide for the use of spatial computing systems to facilitate activities (e.g., between individuals) within various fields, including fields such as different industries with various types of business activities. Such fields may include, but are not limited to, one or more of the following: architecture, civil engineering and construction (AEC), manufacturing, automotive, healthcare, financial services, retail, media / entertainment / sports, energy and utilities, education, general business, travel / tourism / hospitality, agencies (e.g., developers), government, non-profit organizations, and / or telecommunications. For example, embodiments provide for business without barriers, such as, for example, shaking hands across cities, training teams across time zones, and reinventing the way people do business. For example, from realizing architectural drawings to transforming production and fulfillment operations, embodiments provide techniques through which spatial computing can revolutionize different types of work. Examples of architecture, engineering, and construction practices:

[0032] Bridging the gap between design and construction. Spatial computing systems can realize computer-aided design files ("CAD") and building information modeling system configurations ("BIM"), so that individuals can share sketches in a 3D environment, collaboratively fine-tune designs, and / or give virtual tours of a construction before it is built. In other words, in various embodiments, one or more users with a system such as that illustrated in FIG. 1 can collaboratively view certain aspects of a construction's architecture, plans, infrastructure, view simulations, and the like. For example, in one embodiment, an architect wearing a system such as that illustrated in FIG. 1 can conduct a simulated interactive live tour of a proposed new construction project for an investor based in Fiji who is also wearing a system such as that illustrated in FIG. 1, the two users being connected by cloud resources (8), such as internet connectivity, storage, and computational resources. Examples of health and fitness practices include:

[0033] Spatial computing systems can enable organizations and individuals to pursue the future of digital medicine; discover new training opportunities for the next generation of experts and help redefine health conditions; and enhance the way individuals (e.g., surgeons) analyze complex information and 3D images. For example, in various embodiments, a first user of a system such as the one illustrated in FIG. 1 can collaborate with another user of a similar system, each of which can be connected in an “expert onboard” or “colleague onboard” type of configuration via cloud resources (8) so that the second user can visualize the first user's activities as if the second user were looking over the first user's shoulder (and even better, so that the second user can literally have the same view as the first user, share audio as well, and communicate privately and securely with the first user). In one embodiment, using "Expert Onboard," a specialist surgeon may be virtually "called in" to assist more experienced field medics in wartime injury scenarios; the specialist surgeon may be able to help diagnose the patient scenario and instruct the field medics on what to do, what to examine, how to locate certain anatomical structures, etc.In other embodiments, remote experts with access to different computing resources and information may assist local resources, such as in fire and rescue scenarios, where local firefighters equipped with connected (8) systems such as those illustrated in FIG. 1 may assist in “specialized on-board” type fast-paced scenarios, where the local expert has access to CAD drawings of the building on fire, access to remote lock / unlock systems, and understanding of the gas and electrical architecture through connected computing resources that can be most easily and efficiently accessed by the remote expert using a large computer workstation with a high-speed keyboard and large flat monitor; in other words, the remote expert can conveniently and quickly utilize these resources, preferably accessed by the remote expert in the control station environment, but such information can be shared, visualized, and utilized through a combination of audio and visual information sharing through spatial computing connectivity shared by the users. The remote expert could literally say something like, "It's a gas line behind that little purple wall to your right. Pay attention. Go straight through that white wall and get the axe ahead. Yes, that's where you're looking. The next room is likely where the trapped person is. We're looking at "gaze" information from a local Internet of Things camera with infrared on the floor below the table." Manufacturing Examples

[0034] Spatial computing systems can revolutionize manufacturing processes in the automotive and other industries by allowing designers to create and collaborate in real time to enhance productivity. For example, in various embodiments, multiple users equipped with connected (8) systems such as those illustrated in FIG. 1 can collaborate and see / hear each other in a manner that is next to each other in the same room from different locations. They can share ideas, assist each other, and work in an "expert on board" or "colleague on board" scenario similar to that described above, they can dissect and examine 3D drawings, they can assist each other in the examination of 3D objects such as engines or car assembly components or the like. Retail Examples:

[0035] Spatial computing systems can be used to transform today's physical stores into tomorrow's immersive customer experiences. They allow peers in different cities to track and manage inventory and perform remote training. For example, in various embodiments, users in different locations with connected (8) systems such as those illustrated in FIG. 5 can collaborate together as if they were on the same retail floor or the same inventory warehouse, and they can assist each other in "expert onboard" or "colleague onboard" type configurations, such as what to do in emergency security scenarios, training experiences, or any kind of thing, how to handle a customer's retail return without proper documentation, or how to help a customer choose a very specific type of clothing, where a remote expert can be "appeared" using the system to assist a local peer who may have less experience or less specific knowledge in that regard. Examples in education:

[0036] Spatial computing systems can be used to transform classrooms (e.g., chemistry) into immersive learning experiences. They integrate laboratories and libraries into spatial learning, providing new ways to collaborate and interact. Teachers and students equipped with connected systems such as the one illustrated in FIG. 1 can see and hear audio on stunning three-dimensional video of objects such as organic chemistry diagrams, volcanoes, and any related objects, and they can also "share" the classroom when not actually residing in the same classroom. Additionally, remote teachers can silently assist one particular student in a new form of "expert teaching," where a remote teacher assists a local student in an "expert onboard" type scenario. Examples of media and entertainment examples:

[0037] Spatial computing systems can provide immersive news reporting and place-based entertainment that creates new ways for storytellers to connect with their audiences and develop content. For example, in various embodiments, a user of a connected (8) system such as that illustrated in FIG. 1 may be able to stroll through a park and receive and view media information about various things related to their location; for example, if the user gazes at a particular flower and has the system configured to provide additional media if the user presents a "thumbs up" gesture in front of the face near the flower (or, for example, uses one of the selection devices associated with handheld controller 4), the system may be configured to provide readable information about the flower, i.e., (e.g., by pattern recognition) what type it is, what is known about it, popular cultural uses of the flower, etc., according to the user's preferences. In fact, the user may wish to watch some movie clips regarding the use of such flowers, or even have a botanist "appear" in an "expert on board" type composition, or even have her husband appear from his remote location by using the connected (8) system so that he can see in a simulated way what she is seeing or "being there" and experience the scenario with her. Illustrative embodiments:

[0038] FIG. 2 provides an illustration according to one or more embodiments, where a computing environment allows multiple users to collaboratively view and / or manipulate virtual objects / images. For example, as described above, a spatial computing system can realize computer-aided design files ("CAD") and building information, building information modeling system configurations ("BIM"), so that individuals can share sketches in a 3D environment, collaboratively fine-tune designs, and / or give virtual tours of buildings before they are built; for example, as shown in FIG. 2, a user at location 1 can collaboratively view CAD files / models and / or virtual objects / structures with another user at location 2. Visual representations are displayed within the spatially organized 3D and / or 2D environment.

[0039] As illustrated in FIG. 3, a spatial computing system can enable a remote expert to become an "expert on-board" for any suitable situation or application. For example, in various embodiments, one user of a system such as the one illustrated may collaborate with another user of a similar system, each of which may be connected via networked resources, one of the users being an "expert on-board" or "colleague on-board." The second user may be able to visualize the first user's activities from the exact same perspective of the first user, or as if the second user were looking over the first user's shoulder. Other forms of media, such as audio, texture, vibration, smell, etc., may also be shared, depending on the particular types of sensors and / or interfaces implemented within the system / device.

[0040] 2 and 3 represent a physical environment and systems for implementing various processes described herein. The exemplary physical environment and systems of environment 100 include a user's physical environment as viewed by user 108 through head-mounted system 160. The exemplary system of environment 100 further includes access to 2D content via a system operably coupled to network 120. Network 120 may be the Internet, an internal network, a private cloud network, a public cloud network, etc. The system is also operably coupled to processor 170 via network 120. Although processor 170 is shown as a separate and isolated component from head-mounted system 160, in alternative embodiments, processor 170 may be integrated with one or more components of head-mounted system 160 and / or integrated within other system components within environment 100, such as, for example, computing network 125 and network 120 for accessing storage device 130. Processor 170 may be configured with software for receiving and processing information such as video, audio, and content received from head mounted system 160, local storage device 140, computing network 125, and storage device 130. The software may communicate with computing network 125 and storage device 130 via network 120. The software may be installed on processor 170, or in another embodiment, the features and functionality of the software may be integrated into processor 170. Processor 170 may also be configured with local storage device 140 to store information used by processor 170 for quick access without relying on information stored remotely on an external storage device from the vicinity of user 108. In other embodiments, processor 170 may be integrated within head mounted system 160.

[0041] The user's physical environment is the physical surroundings of the user 108 as the user moves around and views the user's physical environment through the head mounted system 160. The computing network 125 accesses the storage device 130 to retrieve and store content for display. In some embodiments, the local storage device 140 may provide content of interest to the user 108. The local storage device 140 may include, for example, a flash drive, a cache, a hard drive, a database, a file system, etc. Information stored in the local storage device 140 may include recently accessed or recently displayed content. The local storage device 140 enables the system to improve performance for the environment 100 by providing certain content locally to the software.

[0042] The software is stored in a non-transitory computer readable medium and includes a software program that performs the function of decomposing 2D content to be displayed in the user's physical environment 105. The software may run on a processor 170, which may be locally attached to the user 108, or in some other embodiments, the software 150 and processor 170 may be included in the head-mounted system 160. In some embodiments, some of the features and functionality of the software 150 may be stored and executed on a computing network 125 remote from the user 108.

[0043] Head-mounted system 160 may be a virtual reality (VR) or augmented reality (AR) head-mounted system that includes a user interface, a user sensing system, an environmental sensing system, and a processor (all not shown). Head-mounted system 160 presents an interface to user 108 for interacting with and experiencing the digital world. Such interactions may involve the user and the digital world, one or more other users interfacing with environment 100, and objects within the digital and physical worlds.

[0044] A user interface for the system may include at least one or a combination of a tactile interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device. A tactile interface device is a device that allows a human to interact with a computer through physical sensations and movements. Haptics refers to a type of human-computer interaction technology that involves tactile feedback or other physical sensations to perform an action or process on a computing device.

[0045] The user sensing system may include one or more sensors 162 operable to detect certain features, characteristics, or information about the user 108 wearing the head mounted system 160. For example, in some embodiments, the sensor 162 may include a camera or optical detection / scanning circuitry capable of detecting real-time optical properties / measurements of the user 108, such as one or more of the following: pupil contraction / dilation, angular measurement / positioning of each pupil, sphericity, eye shape (as eye shape changes over time), and other anatomical data. This data may be used by the head mounted system 160 to provide or calculate information (e.g., the user's visual focus) that can be used to enhance the user's viewing experience.

[0046] The environmental sensing system may include one or more sensors 164 for acquiring data from the user's physical environment 105. Objects or information detected by the sensors 164 may be provided as input to the head-mounted system 160. In some embodiments, this input may represent a user interaction with the virtual world. For example, a user (e.g., user 108) viewing a virtual keyboard on a tabletop (e.g., table 188) may gesture with their fingers as if the user is typing on the virtual keyboard. The motion of the moving fingers may be captured by the sensors 164 and provided as input to the head-mounted system 160, which may be used to change the virtual world or generate new virtual objects. The sensors 164 may include, for example, a generally outward-facing camera or scanner for interpreting scene information through, for example, persistently and / or intermittently projected infrared structured light. The environmental sensing system may be used to map one or more elements of the user's physical environment 105 around the user 108 by detecting and registering the local environment, including static objects, dynamic objects, people, gestures, and various lighting, atmospheric, and acoustic conditions. Thus, in some embodiments, the environmental sensing system may include image-based 3D reconstruction software embedded within a local computing system (e.g., processor 170) and operable to digitally reconstruct one or more objects or information detected by the sensor 164.

[0047] In one embodiment, the environmental sensing system provides one or more of the following: motion capture data (including gesture recognition), depth sensing, facial recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, acoustic source localization, noise reduction, infrared or similar laser projection, and monochrome and / or color CMOS sensors (or other similar sensors), field of view sensors, and various other optically enhanced sensors. It should be understood that the environmental sensing system may also include other components beyond those discussed above.

[0048] Processor 170 may in some embodiments be integrated with other components of head mounted system 160, integrated with other components of the system of environment 100, or may be a separate device (wearable or separate from user 108) as shown in FIG. 1. Processor 170 may be connected to various components of head mounted system 160 through a physical wired connection or through a wireless connection, such as, for example, a mobile network connection (including cellular and data networks), Wi-Fi, Bluetooth, or any other wireless connection protocol. Processor 170 may include memory modules, integrated and / or additional graphics processing units, wireless and / or wired Internet connectivity, and codecs and / or firmware capable of converting data from sources (e.g., computing network 125 and user and environmental sensing systems from head mounted system 160) into image and audio data, which may be presented to user 108 via a user interface (not shown).

[0049] Processor 170 handles data processing for the various components of head-mounted system 160 and data exchange between head-mounted system 160 and content displayed or accessed through computing network 125. For example, processor 170 may be used to buffer and process data delivery between user 108 and computing network 125, thereby enabling a smooth and sustained high-fidelity user experience. System Architecture Overview

[0050] 4 is a block diagram of an exemplary computing system 1400 suitable for implementing certain embodiments of the present disclosure. The computing system 1400 includes a bus 1406 or other communication mechanism for communicating information that interconnects subsystems and devices such as a processor 1407, a system memory 1408 (e.g., RAM), a static storage device 1409 (e.g., ROM), a disk drive 1410 (e.g., magnetic or optical), a communication interface 1414 (e.g., modem or Ethernet card), a display 1411 (e.g., CRT or LCD), and input devices 1412 (e.g., keyboard and mouse).

[0051] According to one embodiment of the invention, computing system 1400 performs specific operations by processor 1407 executing one or more sequences of one or more instructions contained in system memory 1408. Such instructions may be read into system memory 1408 from another computer-readable / usable medium, such as static storage device 1409 or disk drive 1410. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions to implement the present disclosure. Thus, embodiments of the present disclosure are not limited to any specific combination of hardware circuitry and / or software. In one embodiment, the term "logic" shall mean any combination of software or hardware used to implement all or a portion of the present disclosure.

[0052] As used herein, the terms "computer-readable medium" or "computer usable medium" refer to any medium that participates in providing instructions to the processor 1407 for execution. Such media may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as the disk drive 1410. Volatile media include dynamic memory, such as the system memory 1408.

[0053] Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards with a pattern of holes, paper tape, any other physical medium from which a computer can read, RAM, PROM, EPROM, flash EPROM, any other memory chip or cartridge, or any other medium.

[0054] In some embodiments of the present disclosure, execution of sequences of instructions for practicing the present disclosure is performed by a single computing system 1400. According to other embodiments of the present disclosure, two or more computing systems 1400 coupled by a communications link 1415 (e.g., a LAN, a PTSN, or a wireless network) may cooperate with each other to perform the sequences of instructions required to practice the present disclosure.

[0055] Computing system 1400 may transmit and receive messages, data, and instructions, including programs (i.e., application code), through communications link 1415 and communications interface 1414. Received program code may be executed by processor 1407 as it is received, and / or stored in disk drive 1410, or other non-volatile storage for later execution. Computing system 1400 may communicate to a database 1432 on external storage device 1431 through data interface 1433.

[0056] Various exemplary embodiments of the present invention are described herein. These examples are referred to in a non-limiting sense. They are provided to illustrate the more broadly applicable aspects of the present invention. Various modifications may be made to the described invention, and equivalents may be substituted without departing from the true spirit and scope of the present invention. In addition, many modifications may be made to adapt a particular situation, material, composition of matter, process, process acts, or steps to the objective, spirit, or scope of the present invention. Moreover, as will be understood by those skilled in the art, each of the individual variations described and illustrated herein has discrete components and features that can be easily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present invention. All such modifications are intended to be within the scope of the claims associated with this disclosure.

[0057] The present invention includes methods that may be practiced using the subject devices. The methods may include the act of providing such a suitable device. Such provisioning may be performed by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, approach, locate, configure, activate, launch, or otherwise act to provide the device needed in the subject method. The methods recited herein may be carried out in any order of the recited events, and in the recited order of events, that is logically possible.

[0058] Exemplary aspects of the invention, together with details regarding material selection and manufacture, have been described above. As for other details of the invention, these can be understood in conjunction with the patents and published documents referenced above, and generally understood or understood by those skilled in the art. The same can be true with respect to the method-based aspects of the invention in terms of additional acts as generally or theoretically adopted.

[0059] In addition, although the present invention has been described with reference to several embodiments that optionally incorporate various features, the present invention should not be limited to those described or shown as possible for each variation of the present invention. Various modifications and equivalents that can be made to the described invention (whether listed herein or not included for some brevity) can be substituted without departing from the true spirit and scope of the present invention. In addition, when a range of values ​​is provided, it is to be understood that each intervening value between the upper and lower limits of that range and any other stated or intervening value in that stated range is encompassed within the scope of the present invention.

[0060] It is also envisioned that any optional feature of the described inventive variation may be described or claimed independently or in combination with any one or more of the features described herein. Reference to a single object includes the possibility that there are multiple identical objects present. More specifically, as used in this specification and the claims associated herewith, the singular forms "a," "an," "said," and "the" include multiple supports unless specifically stated otherwise. In other words, the use of articles allows for "at least one" of the subject article in the above description and in the claims associated herewith. Furthermore, it is noted that such claims may be drafted to exclude any optional element. Thus, this description is intended to serve as a prelude for the use of such exclusive terms, such as "solely," "only," and equivalents, or the use of "negative" limitations in connection with the recitation of claim elements.

[0061] Without the use of such exclusive terms, the term "comprising" in claims associated with this disclosure shall permit the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claim, or the addition of features may be viewed as changing the nature of the elements recited in such claim. Except as specifically defined herein, all technical and scientific terms used herein shall be given their broadest possible, commonly understood meaning while maintaining claim legitimacy.

[0062] The scope of the present invention should not be limited to the examples and / or subject specification provided, but rather should be limited only by the scope of the claim language associated with this disclosure.

Claims

1. A system for sharing a virtual experience involving real-world objects and virtual objects, the system comprising: a first head mounted vision component including a first sensor; a first handheld controller component operably coupled to the first head mounted viewing component; a first computing component operably coupled to the first head-mounted viewing component and the first handheld controller component; a second head mounted vision component including a second sensor; and a second handheld controller component operably coupled to the second head mounted viewing component; and a second computing component operably coupled to the second head-mounted viewing component and the second handheld controller component; Equipped with the first sensor is configured to capture an image of a real-world object from a first user's view of the real-world object; the first computing component is configured to generate a view of a virtual object corresponding to the real-world object from the first user's view of the real-world object based on the captured image; the second head-mounted viewing component is configured to display to a second user wearing the second head-mounted viewing component a view of the virtual object from the first user's view of the real-world object; A system in which a view of the virtual object from the first user's view of the real-world object is displayed on the second head-mounted device in real time while capturing an image of the real-world object from the first user's view of the real-world object.

2. The first user is located at a first geographic location; The system of claim 1 , wherein the second user is located at a second geographic location different than the first geographic location.

3. The system of claim 2, wherein the first and second handheld controller components are configured to enable the first and second users, respectively, to modify the virtual object.

4. The system of claim 2, wherein the first and second computing components are configured to communicate modifications to the virtual object to each other.

5. The system described in claim 2, wherein the first and second head-mounted visibility components are communicatively coupled by a network.

6. The system of claim 2, wherein the first handheld controller component is configured to enable the first user to provide input related to the virtual object.

7. The system of claim 6, wherein the first handheld controller component is selected from the group consisting of a tactile interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device.

8. The system of claim 2, wherein the second handheld controller component is configured to enable the second user to provide input related to the virtual object.

9. The system of claim 8, wherein the second handheld controller component is selected from the group consisting of a tactile interface device, a keyboard, a mouse, a joystick, a motion capture controller, an optical tracking device, and an audio input device.

10. The system of claim 2, wherein the first and second sensors are configured to provide input related to the virtual object.

11. The system described in claim 10, wherein each of the first and second sensors is selected from the group consisting of a camera and an optical detection device.

12. The system of claim 10, wherein the input is selected from the group consisting of pupil size, pupil position, pupil angle, eye sphericity, eye shape, figure motion, and anatomical data associated with the user.

13. The system of claim 2, further comprising an environmental sensor operably coupled to the first and second head-mounted visibility components, the environmental sensor configured to provide environmental input related to a geographic location to capture an image of the real-world object.

14. The environmental sensor is selected from the group consisting of a camera, an optical detection device, a motion capture device, a depth measurement device, a face recognition device, an object recognition device, an audio recognition device, a CMOS sensor, a visual field sensor, and an optical enhancement sensor; 14. The system of claim 13, wherein the environmental input is selected from the group consisting of motion capture data, gesture recognition data, depth sensing data, facial recognition data, object recognition data, unique object feature recognition data, audio recognition data, voice recognition data, acoustic source localization data, noise reduction data, infrared light data, laser light data, CMOS sensor data, field of view data, and optical enhancement data.

15. A system for sharing a virtual experience involving real-world objects and virtual objects, the system comprising: a first head mounted vision component including a first sensor; a first handheld controller component operably coupled to the first head mounted viewing component; a first computing component operably coupled to the first head-mounted viewing component and the first handheld controller component; a second head mounted vision component including a second sensor; and a second handheld controller component operably coupled to the second head mounted viewing component; and a second computing component operably coupled to the second head-mounted viewing component and the second handheld controller component; Equipped with the first sensor is configured to capture an image of a real-world object from a first user's view of the real-world object; the first computing component is configured to generate a view of a virtual object corresponding to the real-world object from the first user's view of the real-world object based on the captured image; the first head-mounted viewing component is configured to display to the first user a view of the virtual object from the first user's view of the real-world object; the second head-mounted viewing component is configured to display to the second user wearing the second head-mounted viewing component a view of the virtual object from the first user's view of the real-world object; The system is configured to display a view of the virtual object from the first user's view of the real-world object on both the first head-mounted viewing component and the second head-mounted viewing component in real time while capturing an image of the real-world object from the first user's view of the real-world object.

16. A computer-implemented method for facilitating communication between a first and a second user, the method comprising: displaying, by a first head-mounted device, a first virtual object to the first user wearing the first head-mounted device; displaying, by a second head-mounted device, a second virtual object to the second user wearing the second head-mounted device; facilitating communication between the first and second users using the first and second virtual objects by the first and second head-mounted devices to simulate the first and second users being present in a common environment. A method comprising:

17. The method of claim 16, wherein the first virtual object is an avatar of the second user.

18. The method of claim 16, wherein the second virtual object is an avatar of the first user.

19. The method of claim 16, further comprising transmitting, by a first processor, the second virtual object to the second head-mounted device.

20. The method of claim 16, further comprising transmitting, by a second processor, the first virtual object to the first head-mounted device.

Images