Wide-ranging simultaneous remote digital presentation world

Abstract

To provide a preferred massive simultaneous remote digital presence world.SOLUTION: Various methods and apparatuses are described herein for enabling one or more users to interface with virtual or augmented reality environments. An example system includes a computing network having computer servers interconnected through high bandwidth interfaces to gateways for processing data and / or for enabling communication of data between the servers and one or more local user interface devices. The servers include memory, processing circuitry, and software for designing and / or controlling virtual worlds, as well as for storing and processing user data and data provided by other components of the system. One or more virtual worlds may be presented to a user through a user device for the user to take experience and interact.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 61 / 483,505, filed on May 6, 2011, and U.S. Provisional Patent Application No. 61 / 483,511, filed on May 6, 2011.

[0002] (Field of the Invention) The present invention generally relates to methods and apparatuses for enabling a two - way virtual or augmented reality environment for multiple users.

Background Art

[0003] (Background) Virtual and augmented reality environments are generated by a computer using, in part, data that describes the environment. This data may represent, for example, various objects that a user can perceive and interact with. Examples of these objects include objects that are rendered and displayed for a user to see, audio that is played for a user to hear, and haptic (or tactile) feedback for a user to feel. A user may perceive and interact with virtual and augmented reality environments via various visual, auditory, and tactile means.

Summary of the Invention

Means for Solving the Problems

[0004] The present disclosure describes various systems and methods for one or more users to interact with or participate in virtual or augmented reality environments.

[0005] In one exemplary embodiment, the system includes a computing network having computer servers interconnected to a gateway via a high-bandwidth interface for processing data and / or enabling data communication between the servers and one or more local user interface devices. The servers include memory, processing circuits, and software for designing and / or processing virtual worlds, as well as for storing and processing user data and data provided by other components of the system. One or more virtual worlds may be presented to the user through a user device for the user to experience and interact with. Multiple users may use the device to interact with one or more digital worlds simultaneously by each using the device to observe and interact with each other and with objects created within the digital worlds.

[0006] Embodiments of user devices include smartphones, tablet devices, head-up displays (HUDs), game consoles, or generally any other device capable of generating or communicating an interface to the user for communicating data, seeing, hearing, and / or touching. Generally, a user device would include a processor for executing program code stored in memory on the device, coupled with a visual display, and a communication interface. The interface enables visual, audible, and / or physical interaction between the user and the digital world, including other users and objects (real or virtual) presented to the user. In one embodiment, the user device comprises a head-mounted display system having an interface, a user sensing system, an environment sensing system, and a processor. This specification also provides, for example, the following items: (Item 1) A system for enabling one or more users to interact with a virtual world composed of virtual world data, wherein the system is A computer network comprising one or more computer servers, each of which comprises memory, processing circuits, and software, wherein the software is stored in the memory and is executable by the processing circuits to process at least a portion of the virtual world data. The computer network is capable of transmitting the virtual world data to a user device for presentation to a first user. At least a portion of the virtual world changes in response to changes in the virtual world data, A system in which at least a portion of the virtual world data is modified in response to physical objects perceived by the user device. (Item 2) The system described in item 1 describes a virtual object having a predetermined relationship with the physical object, which modifies the virtual world data. (Item 3) The system according to item 2, wherein the changes to the virtual world data are presented to a second user device in accordance with the predetermined relationship. (Item 4) The system described in any one of items 1 to 3, wherein the virtual world is operable to be rendered by at least one of the computer server or user device. (Item 5) The aforementioned virtual world is a system described in any one of items 1 to 4, presented in at least one of two-dimensional or three-dimensional forms. (Item 6) The system according to any one of items 1 to 5, wherein the user device is operable to provide an interface for enabling interaction between the user and the virtual world in at least one of augmented reality mode, virtual reality mode, or a combination of augmented reality mode and virtual reality mode. (Item 7) The aforementioned virtual world data is transmitted over a data network by a system described in any one of items 1 to 6. (Item 8) The computer network is capable of receiving at least a portion of the virtual world data from a user device, as described in any one of items 1 to 7. (Item 9) The system according to any one of items 1 to 8, wherein at least a portion of the virtual world data transmitted to the user device includes instructions for generating at least a portion of the virtual world. (Item 10) A system according to any one of items 1 to 9, wherein at least a portion of the virtual world data is transmitted to a gateway. (Item 11) A system for enabling one or more users to interact with a virtual world, wherein the system is The system includes a user device that presents the virtual world to the user and enables the user to interact with the virtual world, and the user device is Memory and Processing circuit and Software stored in the memory, which is executable by the processing circuit to render at least a portion of a virtual world from virtual world data received at least partially from a computer network, A display capable of operating to present the virtual world to the user, A communication interface capable of operating to communicate at least a portion of the virtual world data over a data network, A sensing system capable of operating to sense at least one of the user, a physical object, or the physical environment surrounding the user, Equipped with, The processing circuit is operable to execute the software to render changes to the virtual world in response to at least one of the sensed user, sensed physical object, or sensed physical environment. (Item 12) The modification of the virtual world comprises a virtual object, the virtual object having a predetermined relationship with the sensed user, physical object, or physical environment, as described in item 11. (Item 13) The system according to item 12, wherein the communication interface is operable to communicate the virtual object to the computer network. (Item 14) The aforementioned virtual world is a system described in any one of items 1 to 13, presented in at least one of two-dimensional or three-dimensional forms. (Item 15) The user device is part of the system according to any one of items 1 to 14, which enables interaction in at least one of the following modes: augmented reality mode, virtual reality mode, or a combination of augmented reality mode and virtual reality mode. (Item 16) The user device further comprises a device for providing tactile or haptic feedback, as described in any one of items 1 to 15. (Item 17) At least a portion of the aforementioned virtual world data is received from the gateway by a system described in any one of items 1 to 16. (Item 18) The gateway is a system according to any one of items 1 to 17, which is capable of distributing the virtual world data for processing. (Item 19) A computer implementation method, wherein the method is Presenting a virtual world to the user's device, Receiving sensor data generated by one or more sensors, wherein the one or more sensors are associated with the user device in response to gestures made by the user using the user device, and receiving Recognizing the gesture, generating a virtual object in response to the recognized gesture; presenting the virtual object to the user device; A method comprising: (Item 20) The method according to item 19, further comprising presenting the virtual object on a second user device. (Item 21) The method according to any one of items 1 to 20, further comprising establishing a relationship between the virtual object and a physical object in the vicinity of the user. (Item 22) A computer-implemented method, the method comprising: receiving, using a user device, sensory data generated by a sensor, the sensor being associated with the user device with respect to a physical object in the vicinity of the user; recognizing the object; generating, in response to the recognition of the object, a virtual object having a predetermined relationship with the physical object; transmitting the virtual object to a display associated with the user device for presentation to the user according to the predetermined relationship; A method comprising: (Item 23) The method according to item 22, further comprising transmitting the virtual object to a second display associated with a second user device for presentation to a second user according to the predetermined relationship. (Item 24) A computer-implemented method, the method comprising: storing data defining a digital world, the data defining one or more objects; receiving sensor data generated by sensors associated with a plurality of user devices, the sensor data describing at least one physical characteristic of the environment of the user device; In response to the sensor data, an instance of a default object is created for each of multiple users, To transmit the instance of the default object generated for the user to each of the multiple users. Methods that include... (Item 25) The method according to item 24, wherein the sensor data describes one or more physical characteristics such as position, user orientation, user movement, user device, environmental conditions, and physical objects in the vicinity of the user.

[0007] The aforementioned and other features and advantages of this disclosure will become even more apparent from the following detailed description of exemplary embodiments, which should be read in conjunction with the accompanying drawings. The detailed description and drawings are merely illustrative of this disclosure, rather than limiting the scope of the invention as defined by the accompanying claims and their equivalents.

[0008] Embodiments are illustrated as examples in the attached drawings, which are not necessarily drawn to a consistent scale, and similar numbers indicate similar parts. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 illustrates a typical embodiment of the disclosed system for enabling a two-way virtual or augmented reality environment for multiple users. [Figure 2] Figure 2 illustrates an example of a user device for interacting with the system shown in Figure 1. [Figure 3] Figure 3 illustrates an exemplary embodiment of a mobile wearable user device. [Figure 4] Figure 4 illustrates an example of an object visible to the user when the mobile wearable user device shown in Figure 3 is operating in augmentation mode. [Figure 5] Figure 5 illustrates an example of an object visible to the user when the mobile wearable user device shown in Figure 3 is operating in virtual mode. [Figure 6] Figure 6 illustrates an example of an object visible to the user when the mobile wearable user device shown in Figure 3 is operating in a mixed virtual interface mode. [Figure 7] Figure 7 illustrates an embodiment in which two users located in different geographical locations interact with the other user and a shared virtual world through their respective user devices. [Figure 8] Figure 8 illustrates an embodiment in which the embodiment of Figure 7 is expanded to include the use of a haptic device. [Figure 9A] Figure 9A illustrates an example of mixed-mode interaction, in which a first user interacts with the digital world in mixed virtual interface mode, and a second user interacts with the same digital world in virtual reality mode. [Figure 9B] Figure 9B illustrates another embodiment of mixed-mode interaction, in which a first user interacts with the digital world in mixed virtual interface mode, and a second user interacts with the same digital world in augmented reality mode. [Figure 10] Figure 10 illustrates an example of a user's field of view when interacting with the system in augmented reality mode. [Figure 11] Figure 11 illustrates an example of a user's field of view illustrating virtual objects triggered by physical objects when the user interacts with the system in augmented reality mode. [Modes for carrying out the invention]

[0010] Referring to Figure 1, System 100 is typical hardware for implementing the processes described below. This typical system comprises a computing network 105 consisting of one or more computer servers 110 connected via one or more high-bandwidth interfaces 115. The servers in the computing network do not need to be located in the same place. Each of the one or more servers 110 comprises one or more processors for executing program instructions. The servers also include memory for storing program instructions and data used and / or generated by processes executed by the servers under the direction of the program instructions.

[0011] The computing network 105 communicates data between servers 110 and between servers and one or more user devices 120 over one or more data network connections 130. Embodiments of such data networks include, but are not limited to, any and all types of public and private data networks, both mobile and wired, and include, for example, many interconnections of such networks commonly referred to as the Internet. No particular media, topology, or protocol is intended to be implied by the diagram.

[0012] The user device is configured to communicate directly with either the computing network 105 or the server 110. Alternatively, the user device 120 communicates locally with the remote server 110 and, optionally, with other user devices via a specially programmed local gateway 140 for processing data and / or for communicating data between the network 105 and one or more local user devices 120.

[0013] As illustrated, the gateway 140 is implemented as a separate hardware component including a processor for executing software instructions and memory for storing software instructions and data. The gateway has its own wired and / or wireless connectivity to a data network for communicating with a server 110 having a computing network 105. Alternatively, the gateway 140 can be integrated with a user device 120 worn or carried by the user. For example, the gateway 140 may be implemented as a downloadable software application installed and run on a processor included in the user device 120. In one embodiment, the gateway 140 provides access to the computing network 105 via a data network 130 to one or more users.

[0014] Each server 110 includes, for example, working memory and storage devices for storing data and software programs, a microprocessor for executing program instructions, a graphics processor and other specialized processors for rendering and generating graphics, images, video, audio, and multimedia files. The computing network 105 may also include devices for storing data accessed, used, or created by the servers 110.

[0015] Software programs running on the server, optionally on the user device 120 and gateway 140 are used to generate a digital world (also referred to herein as a virtual world) in which the user interacts with the user device 120. The digital world is represented by data and processes that describe and / or define virtual, non-existent entities, environments, and conditions that can be presented to the user via the user device 120 for the user to experience and interact with. For example, some type of object, entity, or item that would appear to exist physically when instantiated in a scene viewed or experienced by the user may include a description of its appearance, its behavior, how the user is permitted to interact with it, and other characteristics. Data used to create the environment of the virtual world (including virtual objects) may include, for example, atmospheric data, topographic data, meteorological data, temperature data, location data, and other data used to define and / or describe the virtual environment. In addition, data that defines the various conditions governing the behavior of the virtual world may include, for example, physical laws, time, spatial relationships, and other data that can be used to define and / or create the various conditions governing the behavior of the virtual world (including virtual objects).

[0016] Entities, objects, conditions, characteristics, behaviors, or other features of the digital world will generally be referred to as objects (e.g., digital objects, virtual objects, rendered physical objects, etc.) as herein, unless the context specifically indicates otherwise. Objects may be any type of living or non-living object, including, but not limited to, buildings, plants, vehicles, people, animals, living organisms, machines, data, videos, text, photographs, and other users. Objects may also be defined in the digital world to store information about items, behaviors, or conditions that actually exist in the physical world. Data that describes or defines entities, objects, or items, or stores their current state, will generally be referred to as object data as herein. This data is processed by the server 110, or, depending on the implementation, by the gateway 140 or user device 120, to instantiate instances of objects and render the objects in a manner appropriate for the user to experience them on a user device.

[0017] A programmer who develops and / or creates a digital world creates or defines objects and the conditions under which they are instantiated. However, the digital world can allow others to create or modify objects. Once an object is instantiated, its state may be allowed to be changed, controlled, or manipulated by one or more users experiencing the digital world.

[0018] For example, in one embodiment, the development, production, and management of the digital world are generally provided by one or more system administrator programmers. In some embodiments, this may include the development, design, and / or execution of storylines, themes, and events in the digital world, as well as the distribution of discourse through various forms of events and media, such as movies, digital, network, mobile, augmented reality, and live entertainment. The system administrator programmers may also handle the technical management, discussion management, and curation of the digital world and its associated user community, as well as other tasks typically performed by network administrator personnel.

[0019] The user generally interacts with one or more digital worlds using some type of local computing device designed as the user device 120. Embodiments of such a user device include, but are not limited to, smartphones, tablet devices, head-up displays (HUDs), game consoles, or any other devices capable of communicating data and providing an interface or display to the user, or combinations of such devices. In some embodiments, the user device 120 may include, or communicate with, local peripheral or input / output components such as, for example, a keyboard, mouse, joystick, game controller, haptic interface device, motion capture controller, audio equipment, voice equipment, projector system, 3D display, and holographic 3D contact lenses.

[0020] An embodiment of a user device 120 for interacting with system 100 is illustrated in Figure 2. In the exemplary embodiment shown in Figure 2, a user 210 may interact with one or more digital worlds through a smartphone 220. The gateway is implemented by a software application 230 stored and running on the smartphone 220. In this particular embodiment, the data network 130 includes a wireless mobile network connecting the user device (i.e., the smartphone 220) to the computer network 105.

[0021] In one preferred embodiment, the system 100 can support a large number of simultaneous users (e.g., millions of users) interacting with the same digital world or multiple digital worlds, respectively, using some type of user device 120.

[0022] The user device provides the user with an interface to enable visual, audible, and / or physical interaction between the user and the digital world generated by the server 110, including other users and objects (real or virtual) presented to the user. The interface provides the user with a rendered scene that can be seen, heard, or otherwise perceived, and the ability to interact with the scene in real time. The manner in which the user interacts with the rendered scene may be determined by the capabilities of the user device. For example, if the user device is a smartphone, the user interaction may be implemented by the user touching a touchscreen. In another embodiment, if the user device is a computer or game console, the user interaction may be implemented using a keyboard or game controller. The user device may include additional components that enable user interaction, such as sensors, and objects and information (including gestures) detected by the sensors may be provided as inputs describing the user interaction with the virtual world using the user device.

[0023] The rendered scene can be presented in various forms, such as two-dimensional or three-dimensional visual displays (including projections), sound, and tactile or haptic feedback. The rendered scene may be interacted with by the user in one or more modes, including, for example, augmented reality, virtual reality, and combinations thereof. The form of the rendered scene, as well as the interface mode, may be determined by one or more of the user device, data processing capabilities, user device connectivity, network capabilities, and system workload. The simultaneous interaction of multiple users with the digital world and the real-time nature of data exchange are made possible by the computing network 105, server 110, gateway components 140 (optional), and user device 120.

[0024] In one embodiment, the computing network 105 comprises a large-scale computing system having single and / or multicore servers (i.e., server 110) connected via high-speed connections (e.g., high-bandwidth interface 115). The computing network 105 may form a cloud or grid network. Each server includes memory and is linked to computer-readable memory for storing software for implementing data to create, design, modify, or process objects in the digital world. These objects and their instantiations may be dynamic, existing, disappearing, changing over time, and changing in response to other conditions. Embodiments of the dynamic capabilities of objects are generally discussed herein with respect to various embodiments. In some embodiments, each user interacting with the system 100 may also be represented as an object and / or a collection of objects in one or more digital worlds.

[0025] Server 110 in the computing network 105 also stores computed state data for each of the digital worlds. Computed state data (also referred to herein as state data) may be components of object data and generally define the state of an instance of an object in a given instance over time. Thus, computed state data may change over time and may be affected by the actions of one or more users and / or programmers who maintain the system 100. When a user affects computed state data (or other data including digital worlds), the user either directly modifies or otherwise manipulates the digital world. If the digital world is shared or interacted with by other users, the user's actions may affect what is experienced by other users interacting with the digital world. Thus, in some embodiments, changes made to the digital world by a user will be experienced by other users interacting with the system 100.

[0026] In one embodiment, data stored in one or more servers 110 within the computing network 105 is transmitted or unpacked to one or more user devices 120 and / or gateway components 140 at high speed and with low latency. In one embodiment, object data shared by the servers may be complete or compressed and contain instructions for recreating the complete object data on the user side, which may be rendered and visualized by the user's local computer device (e.g., gateway 140 and / or user device 120). In some embodiments, software running on the servers 110 of the computing network 105 may adapt the data it generates and transmits to a particular user's device 120 for objects in the digital world (or any other data exchanged by the computing network 105) as a function of the user's specific device and bandwidth. For example, when a user interacts with the digital world through a user device 120, the server 110 may recognize the specific type of device used by the user, the connectivity between the user device and the server, and / or the available bandwidth, and appropriately determine and balance the size of the data being delivered to the device to optimize user interaction. Embodiments of this may include reducing the size of transmitted data to a low-resolution quality so that the data can be displayed on a particular user device having a low-resolution display. In a preferred embodiment, the computing network 105 and / or gateway component 140 deliver data to the user device 120 at a speed sufficient to present an interface operating at 15 frames per second or higher and with a data resolution of high resolution or higher.

[0027] The gateway 140 provides local connectivity to the computing network 105 for one or more users. In some embodiments, it may be implemented by a downloadable software application running on the user device 120 or another local device such as the one shown in Figure 2. In other embodiments, it may be implemented by a hardware component (a component having a processor, with appropriate software / firmware stored on the component) which communicates with the user device 120 but is either not incorporated with it or not mounted with it, or is incorporated with the user device 120. The gateway 140 communicates with the computing network 105 via the data network 130 and provides data exchange between the computing network 105 and one or more local user devices 120. As will be discussed in more detail below, the gateway component 140 may include software, firmware, memory, and processing circuitry that may be capable of processing the data communicated between the network 105 and one or more local user devices 120.

[0028] In some embodiments, the gateway component 140 monitors and adjusts the rate of data exchanged between the user device 120 and the computer network 105 to enable optimal data processing capabilities for a particular user device 120. For example, in some embodiments, the gateway 140 buffers and downloads both static and dynamic aspects of the digital world, even those beyond the field of view presented to the user through the interface connected to the user device. In such embodiments, instances of static objects (structured data, software implementations, or both) may be stored in memory (local to the gateway component 140, the user device 120, or both) and referenced relative to the local user's current location, as indicated by the data provided by the computing network 105 and / or the user's device 120. Instances of dynamic objects, which may include intelligent software agents and objects controlled by other users and / or the local user, are stored in a high-speed memory buffer. Describing two-dimensional or three-dimensional objects in the view presented to the user, dynamic objects can be classified into component shapes such as static shapes that are moving but not changing, and dynamic shapes that are changing. Some of the changing dynamic objects can be updated by a real-time threaded high-priority data stream from the server 110 through the computing network 105, managed by the gateway component 140. As one embodiment of the priority threaded data stream, data within the 60-degree field of view of the user's eye may be given a higher priority than data that is more peripheral. Another embodiment includes prioritizing dynamic characters and / or objects within the user's field of view over static objects in the background.

[0029] In addition to managing data connections between the computing network 105 and the user device 120, the gateway component 140 may store and / or process data that may be presented to the user device 120. For example, in some embodiments, the gateway component 140 may receive compressed data from the computing network 105, for example, describing graphical objects to be rendered for user viewing, and perform advanced rendering techniques to reduce the data load transmitted from the computing network 105 to the user device 120. In another embodiment, where the gateway 140 is a separate device, the gateway 140 may store and / or process data of local instances of objects rather than communicating the data to the computing network 105 for processing.

[0030] Referring here to Figure 3, the digital world may be experienced by one or more users in various forms that may depend on the capabilities of the user's device. In some embodiments, the user device 120 may include, for example, a smartphone, a tablet device, a head-up display (HUD), a game console, or a wearable device. Generally, the user device will include a processor for executing program code stored in memory on the device, coupled with a display, and a communication interface. An exemplary embodiment of a user device is shown in Figure 3, which comprises a mobile wearable device, i.e., a head-mounted display system 300. According to embodiments of this disclosure, the head-mounted display system 300 includes a user interface 302, a user sensing system 304, an environment sensing system 306, and a processor 308. In alternative embodiments, the processor 308 is shown in Figure 3 as an isolated component separate from the head-mounted system 300, but the processor 308 may be integrated with one or more components of the head-mounted system 300, or may be incorporated into other system 100 components, such as a gateway 140.

[0031] The user device presents the user with an interface 302 for interacting with and experiencing the digital world. Such interaction may involve the user and the digital world, one or more other users interacting with system 100, and objects within the digital world. Interface 302 generally provides the user with image and / or audio sensory input (and physical sensory input in some embodiments). Thus, interface 302 may include a speaker (not shown) and, in some embodiments, a display component 303 capable of enabling stereoscopic 3D viewing and / or 3D viewing that embodies more natural characteristics of the human visual system. In some embodiments, the display component 303 may have a transparent interface (such as a transparent OLED) that enables an optically correct view of the physical environment around the user with virtually no optical distortion or computing overlay when in the "off" setting. As will be discussed in more detail below, interface 302 may include additional settings that enable various visual / interface performance and functionality.

[0032] In some embodiments, the user sensing system 304 may include one or more sensors 310 that are operable to detect certain features, characteristics, or information relating to an individual user wearing the system 300. For example, in some embodiments, the sensors 310 may include a camera or optical detection / scanning circuit capable of detecting real-time optical characteristics / measurements of the user, such as pupil constriction / expansion, angle measurement / positioning of each pupil, sphericity, eye shape (as changes in eye shape over time), and one or more other anatomical data. This data may provide information (e.g., the user's visual focus) that can be used by the head-mounted system 300 and / or interface system 100 to optimize the user's visual experience, or may be used to compute information. For example, in one embodiment, each sensor 310 may measure the pupil constriction rate of each of the user's eyes. This data may be transmitted to the processor 308 (or to the gateway component 140, or to the server 110), and the data may be used, for example, to determine the user's response to the brightness setting of the interface display 303. Interface 302 may be adjusted according to user responses, for example, by dimming the display 303 if the user's response indicates that the brightness level of the display 303 is too high. User sensing system 304 may include other components other than those discussed above or illustrated in Figure 3. For example, in some embodiments, user sensing system 304 may include a microphone for receiving voice input from the user. User sensing system may also include one or more infrared camera sensors, one or more visible spectrum camera sensors, structural light emitters and / or sensors, infrared light emitters, coherent light emitters and / or sensors, gyroscopes, accelerometers, magnetometers, proximity sensors, GPS sensors, ultrasonic emitters and detectors, and haptic interfaces.

[0033] The environmental sensing system 306 includes one or more sensors 312 for acquiring data from the physical environment surrounding the user. Objects or information detected by the sensors may be provided to the user device as input. In some embodiments, this input may represent user interaction with a virtual world. For example, a user viewing a virtual keyboard on a desk may use finger gestures as if typing on the virtual keyboard. The movement of the fingers may be captured by the sensors 312 and provided to the user device or system as input, which may be used to change the virtual world or to create new virtual objects. For example, the finger movements may be recognized as typing (using a software program), and the recognized typing gestures may be combined with known locations of virtual keys on the virtual keyboard. The system may then render a virtual monitor that is displayed to the user (or other users interacting with the system), which displays the text being typed by the user.

[0034] The sensor 312 may include, for example, a substantially outward-facing camera, or a scanner for interpreting scene information through, for example, continuously and / or intermittently projected infrared structured light. The environment sensing system 306 may be used to map one or more elements of the physical environment around the user by detecting and registering the local environment, including static objects, dynamic objects, people, gestures, and various lighting, atmospheric, and acoustic conditions. Accordingly, in some embodiments, the environment sensing system 306 may include image-based 3D reconstruction software that is integrated into a local computing system (e.g., gateway component 140 or processor 308) and is operable to digitally reconstruct one or more objects or information detected by the sensor 312. In one exemplary embodiment, the environment sensing system 306 provides motion capture data (including gesture recognition), depth sensing, face recognition, object recognition, unique object feature recognition, voice / audio recognition and processing, sound source localization, noise reduction, infrared or similar laser projection, and one or more of monochrome and / or color CMOS sensors (or other similar sensors), field of view sensors, and various other light-enhancing sensors. It should be understood that the environment sensing system 306 may include other components other than those discussed above or illustrated in Figure 3. For example, in some embodiments, the environment sensing system 306 may include a microphone for receiving sound from the local environment. The user sensing system may also include one or more infrared camera sensors, one or more visible spectrum camera sensors, structural light emitters and / or sensors, coherent light emitters and / or sensors, gyroscopes, infrared light emitters, accelerometers, magnetometers, proximity sensors, GPS sensors, ultrasonic emitters and detectors, and a tactile interface.

[0035] As described above, in some embodiments, the processor 308 may be integrated with other components of the head-mounted system 300, integrated with other components of the interface system 100, or may be an isolated device (wearable or separate from the user) as shown in Figure 3. The processor 308 may be connected to various components of the head-mounted system 300 and / or components of the interface system 100 via physical wired connections or via wireless connections such as mobile network connections (including cellular and data networks), Wi-Fi, or Bluetooth®. The processor 308 may include memory modules, integrated and / or additional graphics processing units, wireless and / or wired internet connectivity, and codecs and / or firmware capable of converting data from sources (e.g., computing network 105, user sensing system 304, environment sensing system 306, or gateway component 140) into image and audio data, which may be presented to the user via interface 302.

[0036] The processor 308 handles data processing for various components of the head-mounted system 300, as well as data exchange between the head-mounted system 300 and the gateway component 140, and in some embodiments, the computing network 105. For example, the processor 308 may be used to buffer and process data streaming between the user and the computing network 105, thereby enabling a smooth, continuous, and high-fidelity user experience. In some embodiments, the processor 308 may process data at a speed sufficient to achieve any of the following: 8 frames / second at 320x240 resolution to 24 frames / second at high resolution (1280x720), or even higher, such as 60-120 frames / second and 4k resolution or higher (10k+ resolution and 50,000 frames / second). In addition, the processor 308 may store and / or process data that can be presented to the user rather than being streamed in real time from the computing network 105. For example, in some embodiments, the processor 308 may receive compressed data from the computing network 105 and perform advanced rendering techniques (such as brightness or shading) to reduce the data load transmitted from the computing network 105 to the user device 120. In another embodiment, the processor 308 may store and / or process local object data rather than transmit the data to the gateway component 140 or to the computing network 105.

[0037] In some embodiments, the head-mounted system 300 may include various settings or modes that enable various visual / interface performance and functionality. The modes may be selected manually by the user or automatically by the components of the head-mounted system 300 or the gateway component 140. As described above, one embodiment of the head-mounted system 300 includes an "off" mode in which the interface 302 does not provide substantially any digital or virtual content. In off mode, the display component 303 may be transparent, thereby enabling an optically correct view of the user's surrounding physical environment with virtually no optical distortion or computing overlay.

[0038] In one exemplary embodiment, the head-mounted system 300 includes an "augmented" mode in which interface 302 provides an augmented reality interface. In augmented mode, the interface display 303 may be substantially transparent, thereby allowing the user to view the local physical environment. Simultaneously, virtual object data provided by the computing network 105, processor 308, and / or gateway component 140 is presented on the display 303 in combination with the local physical environment.

[0039] Figure 4 illustrates an exemplary embodiment of objects visible to the user when interface 302 is operating in augmentation mode. As shown in Figure 4, interface 302 presents physical objects 402 and virtual objects 404. In the embodiment illustrated in Figure 4, physical objects 402 are actual physical objects present in the user's local environment, while virtual objects 404 are objects created by system 100 and displayed via user interface 302. In some embodiments, virtual objects 404 may be displayed at a fixed position or location within the physical environment (e.g., a virtual monkey standing next to a specific road sign located within the physical environment), or they may be displayed to the user as objects located relative to the user interface / display 303 (e.g., a virtual clock or thermometer visible in the upper left corner of display 303).

[0040] In some embodiments, a virtual object may be signaled to or triggered by an object that is physically present within or outside the user's field of view. For example, a virtual object 404 may be signaled to or triggered by a physical object 402. For example, the physical object 402 may actually be a stool, and the virtual object 404 may be presented to the user (or, in some embodiments, to another user interacting with system 100) as a virtual animal standing on the stool. In such embodiments, the environment sensing system 306 may use software and / or firmware stored in, for example, processor 308 to recognize various features and / or shape patterns (captured by sensor 312) that identify the physical object 402 as a stool. These recognized shape patterns, for example, the top of the stool, may be used to trigger the placement of the virtual object 404. Other embodiments may use any object that can be seen, including walls, tables, furniture, cars, buildings, people, floors, plants, and animals, to trigger an augmented reality experience in some way with one or more objects.

[0041] In some embodiments, the specific virtual object 404 to be triggered may be selected by the user or automatically selected by other components of the head-mounted system 300 or the interface system 100. In addition, in embodiments where the virtual object 404 is automatically triggered, the specific virtual object 404 may be selected based on a specific physical object 402 (or its features) from which the virtual object 404 is signaled or triggered. For example, if the physical object is identified as a diving board extending over a pool, the triggered virtual object may be a creature wearing a snorkel, swimsuit, flotation device, or other related item.

[0042] In another exemplary embodiment, the head-mounted system 300 may include a “virtual” mode in which interface 302 provides a virtual reality interface. In virtual mode, the physical environment is omitted from display 303, and virtual object data provided by the computing network 105, processor 308, and / or gateway component 140 is presented on display 303. The omission of the physical environment may be achieved by physically blocking the visual display 303 (e.g., through a cover) or through features of interface 302, such that display 303 transitions to an opaque setting. In virtual mode, live and / or stored visual and auditory sensations may be presented to the user through interface 302, and the user experiences and interacts with the digital world (digital objects, other users, etc.) through the virtual mode of interface 302. Thus, the interface provided to the user in virtual mode consists of virtual object data, including the virtual digital world.

[0043] Figure 5 illustrates an exemplary embodiment of the user interface when the head-mounted interface 302 is operating in virtual mode. As shown in Figure 5, the user interface presents a virtual world 500 consisting of digital objects 510, which may include atmosphere, weather, terrain, buildings, and people. Although not shown in Figure 5, the digital objects may also include, for example, plants, vehicles, animals, organisms, machines, artificial intelligence, location information, and any other objects or information that define the virtual world 500.

[0044] In another exemplary embodiment, the head-mounted system 300 may include a “mixed” mode, and various features of the head-mounted system 300 (as well as features of the virtual and augmented modes) may be combined to create one or more custom interface modes. In one example of a custom interface mode, the physical environment is omitted from the display 303, and virtual object data is presented on the display 303 in a manner similar to that of the virtual mode. However, in this example of a custom interface mode, the virtual objects may be entirely virtual (i.e., they do not exist in the local physical environment), or they may be actual local physical objects that are rendered as virtual objects in the interface 302 instead of physical objects. Thus, in a particular custom mode (referred to herein as a mixed virtual interface mode), live and / or stored visual and auditory sensations may be presented to the user through the interface 302, allowing the user to experience and interact with a digital world that includes fully virtual objects and rendered physical objects.

[0045] Figure 6 illustrates an exemplary embodiment of a user interface operating according to a mixed virtual interface mode. As shown in Figure 6, the user interface presents a virtual world 600 consisting of fully virtual objects 610 and rendered physical objects 620 (alternatively, renderings of objects that physically exist in the scene). According to the embodiment illustrated in Figure 6, the rendered physical objects 620 include a building 620A, ground 620B, and platform 620C, and are shown with thick outlines 630 to indicate to the user that the objects are rendered. In addition, the fully virtual objects 610 include an additional user 610A, clouds 610B, sun 610C, and flames 610D on platform 620C. It should be understood that the fully virtual objects 610 may include, for example, the atmosphere, weather, terrain, buildings, people, plants, vehicles, animals, organisms, machines, artificial intelligence, location information, and any other objects or information that are not rendered from objects that exist in the local physical environment of the virtual world 600. Conversely, the rendered physical object 620 is an actual local physical object rendered as a virtual object within interface 302. The thick contour 630 describes one embodiment for showing the rendered physical object to the user. As such, the rendered physical object may be shown using methods other than those disclosed herein.

[0046] In some embodiments, the rendered physical objects 620 may be detected using the sensor 312 of the environment sensing system 306 (or using other devices such as a motion or image capture system) and converted into digital object data by software and / or firmware stored in the processing circuit 308, for example. Thus, when a user interacts with the system 100 in a mixed virtual interface mode, various physical objects may be displayed to the user as rendered physical objects. This can be particularly useful in allowing the user to interact with the system 100 while still being able to safely navigate the local physical environment. In some embodiments, the user may be able to selectively remove or add rendered physical objects to the interface display 303.

[0047] In another example of a custom interface mode, the interface display 303 may be substantially transparent, thereby allowing the user to perceive the local physical environment while various local physical objects are displayed to the user as rendered physical objects. This example of a custom interface mode is similar to the augmentation mode, except that one or more of the virtual objects may be rendered physical objects, as discussed above with respect to the previous embodiment.

[0048] The above-described examples of custom interface modes describe some exemplary embodiments of the various custom interface modes that can be provided by the mixed modes of the head-mounted system 300. Accordingly, various other custom interface modes may be created from various combinations of features and functionalities provided by the components of the head-mounted system 300 and the various modes discussed above, without departing from the scope of this disclosure.

[0049] The embodiments discussed herein merely illustrate several examples for providing interfaces operating in off, augmented, virtual, or mixed modes, and are not intended to limit the scope or content of each interface mode or the functionality of the components of the head-mounted system 300. For example, in some embodiments, virtual objects may include data displayed to the user (time, temperature, altitude, etc.), objects created and / or selected by system 100, objects created and / or selected by the user, or even objects describing other users interacting with system 100. In addition, virtual objects may include augmented physical objects (e.g., virtual images growing from a physical platform) and may be visually connected to or disconnected from physical objects.

[0050] A virtual object may also be dynamic, changing over time, according to various relationships (e.g., position, distance, etc.) between it and the user or other users, physical objects, and other virtual objects, and / or according to other variables specified in the software and / or firmware of the head-mounted system 300, gateway component 140, or server 110. For example, in one embodiment, a virtual object may respond to a user device or its components (e.g., a virtual ball moves when a tactile device is placed next to it), physical or verbal user interaction (e.g., a virtual creature runs away when a user approaches it or speaks when a user speaks to it), a chair is thrown at the virtual creature and it avoids the chair, other virtual objects (e.g., a first virtual creature reacts when it sees a second virtual creature), physical variables such as position, distance, temperature, time, or other physical objects in the user's environment (e.g., a virtual creature shown standing on a physical road flattens when a physical car passes by).

[0051] The various modes discussed herein may also be applied to user devices other than the head-mounted system 300. For example, the augmented reality interface may be provided via a mobile phone or tablet device. In such embodiments, the phone or tablet may use a camera to capture the physical environment around the user, and virtual objects may be overlaid on the phone / tablet display screen. In addition, the virtual mode may be provided by displaying the digital world on the phone / tablet display screen. Thus, these modes may be mixed to create various custom interface modes as described above, using the phone / tablet components discussed herein, as well as other components connected to or used in combination with the user device. For example, a mixed virtual interface mode may be provided by a computer monitor, television screen, or a camera-less device operating in combination with a motion or image capture system. In this exemplary embodiment, the virtual world may be viewed from the monitor / screen, and object detection and rendering may be performed by the motion or image capture system.

[0052] Figure 7 illustrates an exemplary embodiment of the present invention in which two users located in different geographical locations each interact with the other user and a common virtual world through their respective user devices. In this embodiment, the two users 701 and 702 throw a virtual ball 703 (a type of virtual object) back and forth, and each user can observe the other user's influence on the virtual world (for example, each user observes the virtual ball changing direction, being caught by the other user, etc.). Since the movement and position of the virtual object (i.e., the virtual ball 703) are tracked by a server 110 in a computing network 105, the system 100 may, in some embodiments, communicate to users 701 and 702 the exact location and timing of the ball 703's arrival for each user. For example, if the first user 701 is located in London, user 701 may throw the ball 703 to the second user 702, located in Los Angeles, at a speed calculated by the system 100. Therefore, system 100 may communicate the exact time and location of the ball's arrival to a second user 702 (e.g., via email, text message, instant message, etc.). Thus, the second user 702 may use their device to know that the ball 703 will arrive at a specific time and location. One or more users may also use geolocation mapping software (or similar) to track one or more virtual objects as they virtually travel around the Earth. An embodiment of this may be a user wearing a 3D head-mounted display that looks up at the sky and sees a virtual airplane flying overhead, superimposed on the real world. The virtual airplane may be flown by the user, an intelligent software agent (software running on the user device or gateway), other users who may be locally and / or remotely, and / or a combination thereof.

[0053] As described above, the user device may include a haptic interface device that provides feedback (e.g., resistance, vibration, light, sound, etc.) to the user when the system 100 determines that the haptic device is located in a physical spatial position relative to a virtual object. For example, the embodiment described above with respect to Figure 7 may be augmented to include the use of a haptic device 802, as shown in Figure 8. In this exemplary embodiment, the haptic device 802 may be represented in the virtual world as a baseball bat. When a ball 703 arrives, the user 702 may swing the haptic device 802 towards the virtual ball 703. If the system 100 determines that the virtual bat provided by the haptic device 802 has "contacted" the ball 703, the haptic device 802 may vibrate or provide other feedback to the user 702, and the virtual ball 703 may bounce the virtual bat in a direction calculated by the system 100 according to the detected velocity, direction, and timing of the contact between the ball and the bat.

[0054] In some embodiments, the disclosed system 100 may facilitate mixed-mode interaction, allowing multiple users to interact with a common virtual world (and virtual objects contained therein) using different interface modes (e.g., augmented reality, virtual reality, mixed reality, etc.). For example, a first user interacting with a particular virtual world in virtual interface mode may interact with a second user interacting with the same virtual world in augmented reality mode.

[0055] Figure 9A illustrates an embodiment in which a first user 901 (interacting with the digital world of system 100 in mixed virtual interface mode) and a first object 902 appear as virtual objects to a second user 922 (interacting with the same digital world of system 100 in full virtual reality mode). As described above, when interacting with the digital world via mixed virtual interface mode, local physical objects (e.g., the first user 901 and the first object 902) may be scanned and rendered as virtual objects in the virtual world. The first user 901 may be scanned, for example, by a motion capture system or similar device and rendered in the virtual world as a first rendered physical object 931 (by software / firmware stored in the motion capture system, gateway component 140, user device 120, system server 110, or other devices). Similarly, the first object 902 may be scanned, for example, by the environment sensing system 306 of the head-mounted interface 300 and rendered in the virtual world as a second rendered physical object 932 (by software / firmware stored in the processor 308, gateway component 140, system server 110, or other devices). The first user 901 and the first object 902 are shown as physical objects in the physical world in the first part 910 of Figure 9A. In the second part 920 of Figure 9A, the first user 901 and the first object 902 are shown as a first rendered physical object 931 and a second rendered physical object 932 when they appear to a second user 922 who interacts with the same digital world of the system 100 in full virtual reality mode.

[0056] Figure 9B illustrates another exemplary embodiment of mixed-mode interaction, in which a first user 901 interacts with the digital world in mixed-virtual interface mode, as discussed above, and a second user 922 interacts with the same digital world (and the second user's physical local environment 925) in augmented reality mode. In the embodiment of Figure 9B, the first user 901 and the first object 902 are located at a first physical location 915, and the second user 922 is located at a different second physical location 925, separated by some distance from the first location 915. In this embodiment, virtual objects 931 and 932 may be transposed in real time (or near real time) to their locations in the virtual world corresponding to the second location 925. Thus, the second user 922 may observe and interact with rendered physical objects 931 and 932 in the second user's physical local environment 925, which describe the first user 901 and the first object 902, respectively.

[0057] Figure 10 illustrates an example of a diagram illustrating the user's field of view when interacting with system 100 in augmented reality mode. As shown in Figure 10, the user sees a local physical environment (i.e., a city with multiple buildings) and a virtual character 1010 (i.e., a virtual object). The position of the virtual character 1010 may be triggered by one or more 3D reference coordinate systems such as 2D visual targets (e.g., signs, postcards, or magazines) and / or buildings, cars, people, animals, airplanes, parts of buildings, and / or 3D physical objects, virtual objects, and / or combinations thereof. In the embodiment illustrated in Figure 10, the known locations of buildings in the city may provide alignment references and / or information and key features for rendering the virtual character 1010. In addition, the user's geospatial position relative to the buildings (e.g., provided by GPS, attitude / position sensors, etc.) or moving position may include data used by the computing network 105 to trigger the transmission of data used to display the virtual character 1010. In some embodiments, the data used to display the virtual character 1010 may include instructions (executed by the gateway component 140 and / or the user device 120) for rendering the rendered character 1010 and / or a portion of the virtual character 1010. In some embodiments, if the user's geographical location is unavailable or unknown, the server 110, gateway component 140, and / or the user device 120 may still display the virtual object 1010 using an estimation algorithm that estimates where a particular virtual object and / or physical object could be located, using the user's last known location as a function of time and / or other parameters. This may also be used to determine the location of any virtual object if the user's sensors are blocked and / or experience other malfunctions.

[0058] In some embodiments, a virtual character or virtual object may have a virtual image, and the rendering of the virtual image is triggered by a physical object. For example, referring here to Figure 11, the virtual image 1110 may be triggered by an actual physical platform 1120. The triggering of the image 1110 may respond to a visual object or feature (e.g., reference point, design feature, geometric shape, pattern, physical location, altitude, etc.) detected by a user device or other components of the system 100. When a user views the platform 1120 without using a user device, the user sees the platform 1120 without the image 1110. However, when a user views the platform 1120 through a user device, the user sees the image 1110 on the platform 1120, as shown in Figure 11. The image 1110 is a virtual object and therefore may be stationary, active, change over time or relative to the user's viewing position, or even change depending on which particular user is viewing the image 1110. For example, if the user is a young child, the image may be a dog, and if the viewer is an adult male, the image may be a large robot as shown in Figure 11. These are examples of user-dependent and state-dependent experiences. This would allow one or more users to perceive one or more virtual objects, both alone and / or in combination with physical objects, and to experience customized and personalized versions of the virtual objects. Image 1110 (or a part thereof) may be rendered by various components of the system, including, for example, software / firmware installed on the user device. Using data indicating the position and orientation of the user device in combination with the alignment features of the virtual object (i.e., image 1110), the virtual object (i.e., image 1110) forms a relationship with the physical object (i.e., platform 1120).For example, the relationship between one or more virtual objects and one or more physical objects may be a function of distance, positioning, time, geographic location information, proximity to one or more other virtual objects, and / or any other functional relationship including any kind of virtual and / or physical data. In some embodiments, image recognition software in a user device may further enhance the relationship from digital objects to physical objects.

[0059] The interactive interfaces provided by the disclosed systems and methods may be implemented to facilitate various activities, such as interacting with one or more virtual environments and objects, interacting with other users, and experiencing various forms of media content, including advertisements, music concerts, and movies. However, the disclosed systems facilitate user interaction such that users not only view or listen to media content, but rather actively participate in and experience the media content. In some embodiments, user participation may include modifying existing content or creating new content to be rendered in one or more virtual worlds. In some embodiments, the media content, and / or the user creating the content, may be themed around the creation of one or more virtual worlds.

[0060] In one embodiment, a musician (or other user) may create musical content that is rendered to a user interacting with a specific virtual world. The musical content may include, for example, various singles, EPs, albums, videos, short films, and concert performances. In one embodiment, multiple users may interact with system 100 to simultaneously experience a virtual concert performed by the musician.

[0061] In some embodiments, the media produced may contain a unique identifier code associated with a specific entity (e.g., a band, artist, user, etc.). The code may be a set of alphanumeric characters, a UPC code, a QR code (registered trademark), a 2D image trigger, a 3D physical object feature trigger, or other digital marks, as well as in the form of sound, images, and / or both. In some embodiments, the code may also be embedded in digital media that can be interacted with using system 100. A user may obtain a code (e.g., by payment of a fee) and redeem the code to access media content produced by the entity associated with the identifier code. The media content may be added or removed from the user interface.

[0062] The embodiments disclosed herein are provided to illustrate one or more embodiments of methods and apparatus for enabling interactive virtual or augmented reality environments for multiple users. Modifications of the methods and apparatus disclosed herein may be made without departing from the scope of the disclosure, as described in the claims provided below. For example, various embodiments and models are discussed herein in relation to head-mounted display systems, but the various embodiments and models may also be applied to other user devices capable of providing the interfaces or capabilities discussed in relation to these particular embodiments.

Claims

1. A method that enables one or more users to interact with a virtual world, wherein the method is The structure of a wearable user device is attached to a part of a person's body, Using the processing circuit of the wearable user device, the software stored in memory is executed to render at least a portion of the virtual world from virtual world data received at least partially from a computer network. Using the display of the wearable user device, the virtual world is displayed to the user, Using the communication interface of the wearable user device, communicate at least a portion of the virtual world data over a computer network, The sensing system of the wearable user device is used to sense at least one of the user, a physical object, or the physical environment surrounding the user, wherein the sensing system includes a camera positioned to detect a measurement of the pupil angle of the user's eye. Using a gateway that connects the wearable user device to the computer network, the speed of data exchange between the wearable user device and the computer network is monitored and adjusted to enable optimal data processing capabilities for the wearable user device by determining whether the object to be rendered is a dynamic object or a static object, determining whether the object to be rendered is within the user's field of view having less than 60 degrees of pupil, and prioritizing dynamic objects or objects within the field of view over static objects or objects not within the field of view. Includes, A method wherein the processing circuit is operable to execute the software to render changes in the virtual world in response to at least one of the sensed user, sensed physical object, and sensed physical environment.

2. The method according to claim 1, wherein the changes in the virtual world include virtual objects, the virtual objects having predetermined relationships with the sensed user, physical objects, or physical environment.

3. The method according to claim 2, wherein the communication interface is operable to communicate the virtual object to the computer network.

4. The method according to claim 1, wherein the virtual world is presented in at least one of two-dimensional or three-dimensional formats.

5. The method according to claim 1, wherein the wearable user device enables interaction in at least one of augmented reality mode, virtual reality mode, or a combination of augmented reality mode and virtual reality mode.

6. The method according to claim 1, wherein the wearable user device further comprises a device for providing tactile or haptic feedback.

7. The method according to claim 1, wherein at least a portion of the virtual world data is communicated to and from a gateway.

8. The method according to claim 1, wherein the computer network comprises one or more computer servers, each comprising memory, processing circuits, and software stored in the memory and executable by the processing circuits to process at least a portion of the virtual world data, and the computer network is operable to transmit the virtual world data to a wearable user device for presentation to a first user.

9. The method according to claim 8, wherein the sensing system is an environmental sensing system coupled to the wearable user device configured to acquire data from the physical environment surrounding the user.

10. The method according to claim 9, wherein the environmental sensing system detects predetermined characteristics of static physical objects located outside the user.

11. The method according to claim 10, wherein at least one computer server of the computer network receives data from the environment sensing system for triggering the transmission of the virtual world data to the wearable user device.

12. The method according to claim 1, wherein the sensing system is configured such that at least a portion of the virtual world changes in response to a change in the virtual world data.

13. The method according to claim 12, wherein, in conjunction with changes in the virtual world in response to changes in the virtual world data, at least a portion of the virtual world data is changed in response to static physical objects located outside the user and sensed by the wearable user device.

14. The method according to claim 13, wherein the static physical object located outside the user includes mapped objects in the physical environment near the user.

15. The method according to claim 12, wherein the change in the virtual world data represents rendering at least one of a dynamic virtual object and a static virtual object based on a static physical object outside the user according to a predetermined relationship.

16. The method according to claim 1, wherein the changes in the virtual world data are presented to a second user device for presentation to a second user according to a predetermined relationship.

17. The method according to claim 1, wherein the virtual world is operable to be rendered by at least one computer server of the computer network or the wearable user device.

18. The method according to claim 1, wherein the virtual world is presented in at least one of two-dimensional or three-dimensional formats.

19. The method according to claim 1, wherein the wearable user device is operable to provide an interface for enabling interaction between the user and the virtual world in at least one of augmented reality mode, virtual reality mode, or a combination of augmented reality mode and virtual reality mode.

20. The method according to claim 1, wherein the structure that can be attached to a part of a person's body is configured to be a head-mounted structure.

Images