Method and System for 3D Model Sharing

Abstract

To provide systems and methods for decomposing and sharing 3D models.SOLUTION: In a 3D model sharing process 1200, a first version of a virtual three-dimensional model is displayed via a display of a wearable head device. A request is made to a host device for data associated with a second version of the virtual three-dimensional model, where the second version of the virtual three-dimensional model comprises a constituent part. It is determined whether the first version of the virtual three-dimensional model comprises the constituent part. In accordance with a determination that the first version of the virtual three-dimensional model does not comprise the constituent part, a request is made to the host device for data associated with the constituent part. The second version of the virtual three-dimensional model is displayed via the display of the wearable head device.SELECTED DRAWING: Figure 12

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of priority of U.S. Provisional Application No. 62 / 735,675, filed on September 24, 2018, the content of which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates to sharing 3D models between two or more computing systems, including mixed reality, imaging, and visualization systems.

Background Art

[0003] Modern computing and display technologies have facilitated the development of systems for so - called "virtual reality", "augmented reality", and "mixed reality" experiences, where digitally reproduced images are presented to a user in a manner that appears or is perceived to be real. Virtual reality (VR) scenarios typically involve the presentation of computer - generated virtual image information without transparency to other actual real - world visual inputs. Augmented reality (AR) scenarios typically involve the presentation of virtual image information as an augmentation to the visualization of the actual world around the user. Mixed reality (MR) is a type of augmented reality where physical and virtual objects co - exist and can interact in real - time. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies.

Summary of the Invention

Means for Solving the Problems

[0004] Embodiments of the present disclosure describe systems and methods for decomposing and sharing 3D models. In an exemplary method, a first version of a virtual three-dimensional model is displayed via a display of a wearable head device. A request is made to a host device regarding data associated with a second version of the virtual three-dimensional model, and the second version of the virtual three-dimensional model comprises components. It is determined whether the first version of the virtual three-dimensional model comprises components. In accordance with a determination that the first version of the virtual three-dimensional model does not comprise components, a request is made to the host device regarding data associated with the components. The second version of the virtual three-dimensional model is displayed via the display of the wearable head device. In accordance with a determination that the first version of the virtual three-dimensional model comprises components, a request is not made to the host device regarding data associated with the components. The present invention provides, for example, the following. (Item 1) A system comprising: a host computing system; a client computing system comprising a head-mounted display system and comprising: The host computing system is configured to: access a virtual three-dimensional model stored in a memory; decompose the three-dimensional model into one or more components; send a list of the one or more components to the client computing system; receive a component request from the client computing system; send to the client computing system one or more components corresponding to the component request one or more processors configured to execute a method comprising: and comprising: The client computing system is configured to: Receiving a list of the one or more components from the host computing system; Sending the component request to the host computing system; Receiving one or more of the components corresponding to the component request from the host computing system; Constructing a copy of the virtual 3D model from one or more of the components corresponding to the component request One or more processors configured to execute a method comprising A system comprising. (Item 2) The system according to item 1, wherein the component comprises mesh data. (Item 3) The system according to item 1, wherein the component comprises texture data. (Item 4) The system according to item 1, wherein the host computing system comprises a server. (Item 5) The system according to item 1, wherein the host computing system comprises a head-mounted display system. (Item 6) The system according to item 1, wherein the method further comprises storing the one or more components in a memory. (Item 7) The system according to item 1, wherein the method further comprises decomposing the one or more components. (Item 8) A method comprising Accessing a virtual 3D model stored in a memory; Decomposing the 3D model into one or more components; Sending a list of the one or more components to a client computing system comprising a head-mounted display; Receiving a component request from the client computing system; sending one or more components corresponding to the component requirements to the client computing system; receiving a list of the one or more components from a host computing system; sending the component requirements to the host computing system; receiving one or more of the components corresponding to the component requirements from the host computing system; constructing a copy of the virtual 3D model from one or more of the components corresponding to the component requirements A method comprising the above. (Item 9) The method according to item 8, wherein the component comprises mesh data. (Item 10) The method according to item 8, wherein the component comprises texture data. (Item 11) The method according to item 8, wherein the host computing system comprises a server. (Item 12) The method according to item 8, wherein the host computing system comprises a head-mounted display system. (Item 13) The method according to item 8, further comprising storing the one or more components in a memory. (Item 14) The method according to item 8, further comprising disassembling the one or more components. (Item 15) A non-transitory computer-readable medium storing one or more instructions that, when executed by one or more processors, cause the one or more processors to access a virtual 3D model stored in a memory; decompose the 3D model into one or more components; Sending the list of the one or more components to a client computing system comprising a head-mounted display; Receiving a component request from the client computing system; Sending one or more components corresponding to the component request to the client computing system; Receiving the list of the one or more components from a host computing system; Sending the component request to the host computing system; Receiving one or more of the components corresponding to the component request from the host computing system; Constructing a copy of the virtual 3D model from one or more of the components corresponding to the component request A non-transitory computer-readable medium for implementing a method including the above. (Item 16) The non-transitory computer-readable medium according to item 15, wherein the component comprises mesh data. (Item 17) The non-transitory computer-readable medium according to item 15, wherein the component comprises texture data. (Item 18) The non-transitory computer-readable medium according to item 15, wherein the host computing system comprises a server. (Item 19) The non-transitory computer-readable medium according to item 15, wherein the host computing system comprises a head-mounted display system. (Item 20) The non-transitory computer-readable medium according to item 15, wherein the method further comprises storing the one or more components in a memory. (Item 21) The non-transitory computer-readable medium according to item 15, wherein the method further comprises disassembling the one or more components. (Item 22) A method comprising: accessing a virtual 3D model stored in a memory; decomposing the virtual 3D model into one or more components; storing the one or more components in one or more arrays, wherein the one or more components are stored separately from the virtual 3D model; receiving a connection request from a wearable head device; sending a list of available components to the wearable head device; receiving a component request from the wearable head device; sending the requested component to the wearable head device based on the component request; A method. (Item 23) A method comprising: sending a connection request to a host device via a wearable head device; receiving a list of available components from the host device; sending a component request to the host device; receiving the requested component from the host device based on the component request; reconstructing a copy of the virtual 3D model based on the requested component; A method.

Brief Description of the Drawings

[0005] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims.

[0006]

Figure 1

[0007]

Figure 2

[0008]

Figure 3

[0009]

Figure 4

[0010]

Figure 5

[0011]

Figure 6A

[0012]

Figure 6B

[0013]

Figure 7

[0014]

Figure 8

[0015]

Figure 9

[0016]

Figure 10

[0017]

Figure 11

[0018]

Figure 12

[0019]

Figure 13

[0020]

Figure 14

[0021]

Figure 15

[0022]

Figure 16

[0023]

Figure 17

[0024]

Figure 18

[0025]

Figure 19A

Figure 19B

Figure 19C

[0026] Throughout the drawings, reference numerals may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate the exemplary embodiments described herein and are not intended to limit the scope of the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

[0027] Mixed Reality Environment

[0028] Like all people, users of a mixed reality system exist in the physical environment, i.e., the three - dimensional portion of the "real world," and all of its content that is perceivable by the user. For example, a user perceives the physical environment using one of the normal human senses, i.e., vision, sound, touch, taste, smell, and interacts with the physical environment by moving their own body within the physical environment. Locations within the physical environment can be described as coordinates in a coordinate space. For example, the coordinates can comprise latitude, longitude, and elevation relative to sea level, distances in three orthogonal dimensions from a reference point, or other suitable values. Similarly, a vector can describe a quantity having a direction and magnitude within the coordinate space.

[0029] A computing device can maintain a representation of a virtual environment, for example, within a memory associated with the device. As used herein, a virtual environment is a computational representation of a three-dimensional space. The virtual environment can include representations of any object, action, signal, parameter, coordinate, vector, or other characteristic associated with that space. In some embodiments, the circuitry (e.g., a processor) of the computing device can maintain and update the state of the virtual environment, i.e., the processor can determine the state of the virtual environment at a second time t1 based on data associated with the virtual environment and / or input provided by a user at a first time t0. For example, if an object within the virtual environment is located at a first coordinate at time t0, has certain programmed physical parameters (e.g., mass, coefficient of friction), and an input received from the user indicates that a force should be applied to the object in a direction vector, the processor can apply the laws of kinematics and use basic mechanics to determine the location of the object at time t1. The processor can use any suitable information known about the virtual environment and / or any suitable input (e.g., real-world parameters) to determine the state of the virtual environment at time t1. When maintaining and updating the state of the virtual environment, the processor can execute any suitable software, including software for creating and deleting virtual objects within the virtual environment, software for defining the behavior of virtual objects or characters within the virtual environment (e.g., scripts), software for defining the behavior of signals (e.g., audio signals) within the virtual environment, software for creating and updating parameters associated with the virtual environment, software for generating audio signals within the virtual environment, software for handling inputs and outputs, software for implementing network operations, software for applying asset data (e.g., animation data for moving virtual objects over time), or many other possibilities.

[0030] Output devices such as displays or speakers can present any or all aspects of the virtual environment to the user. For example, the virtual environment may include virtual objects (which may include representations of inanimate objects, people, animals, lights, etc.) that can be presented to the user. The processor can determine a view of the virtual environment (e.g., corresponding to a "camera" with origin coordinates, viewing axes, and frustum), and render the visible scene of the virtual environment corresponding to that view on the display. Any suitable rendering technique may be used for this purpose. In some embodiments, the visible scene may include only some of the virtual objects within the virtual environment and may exclude other virtual objects. Similarly, the virtual environment may include audio aspects that can be presented to the user as one or more audio signals. For example, virtual objects within the virtual environment may generate sounds arising from the location coordinates of the object (e.g., a virtual character can speak or cause sound effects), or the virtual environment may be associated with musical cues or ambient sounds that may or may not be associated with a particular location. The processor determines an audio signal corresponding to the "listener" coordinates, e.g., an audio signal that is mixed and processed to simulate an audio signal that would be heard by the listener at the listener coordinates corresponding to the composite of the sounds within the virtual environment, and can present the audio signal to the user via one or more speakers.

[0031] Since the virtual environment only exists as a computational construct, a user cannot directly perceive the virtual environment using their normal senses. Instead, a user can only indirectly perceive the virtual environment as presented to the user, for example, by way of a display, speakers, haptic output devices, and the like. Similarly, a user cannot directly touch, manipulate, or otherwise interact with the virtual environment, but can provide input data to a processor that can use device or sensor data, via an input device or sensor, to update the virtual environment. For example, a camera sensor can provide optical data indicating that a user is attempting to move an object within the virtual environment, and the processor can use that data to cause the object to respond within the virtual environment accordingly.

[0032] A mixed reality system can present a user with a mixed reality (“MR”) environment that combines aspects of the real and virtual environments, for example, using a see-through display and / or one or more speakers (e.g., that can be incorporated within a wearable head device). In some embodiments, one or more speakers can be external to a head-mounted wearable unit. As used herein, an MR environment is a simultaneous representation of a real environment and a corresponding virtual environment. In some examples, the corresponding real and virtual environments share a single coordinate space, and in some examples, the real coordinate space and one or more corresponding virtual coordinate spaces are related to each other by one or more transformation matrices (or other suitable representations). Thus, in some embodiments, a single coordinate can (in some examples, together with a transformation matrix) define a first location within the real environment and a second corresponding location within the virtual environment, and vice versa.

[0033] In an MR environment, a virtual object (e.g., within a virtual environment associated with the MR environment) can correspond to an actual object (e.g., within a physical environment associated with the MR environment). For example, if the physical environment of the MR environment includes an actual street lamp (actual object) at a location coordinate, the virtual environment of the MR environment may include a corresponding virtual street lamp (virtual object) at the corresponding location coordinate. As used herein, an actual object and its corresponding virtual object together constitute a "composite reality object." The virtual object does not need to exactly match or align with the corresponding actual object. In some embodiments, the virtual object can be a simplified version of the corresponding actual object. For example, if the physical environment includes an actual street lamp, the corresponding virtual object may include a cylinder having approximately the same height and radius as the actual street lamp (reflecting that the street lamp can be substantially cylindrical). Simplifying the virtual object in this way can enable computational efficiency and simplify the calculations to be performed on such virtual objects. Further, in some embodiments of the MR environment, not all actual objects within the physical environment may be associated with corresponding virtual objects. Similarly, in some embodiments of the MR environment, not all virtual objects within the virtual environment may be associated with corresponding actual objects. That is, some virtual objects may be in the virtual environment of the MR environment alone, without any real-world counterpart. In some embodiments, not all actual objects may be associated with corresponding actual objects.

[0034] In some embodiments, the virtual object may have characteristics that are sometimes significantly different from those of the corresponding real object. For example, while the real environment within the MR environment may comprise a green cactus with two arms, i.e., a spiny inanimate object, the corresponding virtual object within the MR environment may have the characteristics of a green two-armed virtual character with human facial features and a surly attitude. In this embodiment, the virtual object is similar to its corresponding real object in some characteristics (color, number of arms), but different from the real object in other characteristics (facial features, personality). Thus, virtual objects have the potential to represent real objects in a creative, abstract, exaggerated, or fanciful manner, or to endow inanimate real objects with behavior (e.g., human personality) that they would not otherwise have. In some embodiments, the virtual object may be a purely fictional creation with no real-world counterpart (e.g., perhaps a virtual monster within the virtual environment at a location corresponding to an empty space within the real environment).

[0035] Compared with a VR system that presents a virtual environment while obscuring the real environment, an MR system that presents an MR environment provides the advantage that the real environment remains perceptible while the virtual environment is being presented. Thus, the user of an MR system can use visual and audio cues associated with the real environment to experience and interact with the corresponding virtual environment. As an example, as described above, since the user of a VR system cannot directly perceive or interact with the virtual environment, they may struggle to perceive or interact with virtual objects displayed within the virtual environment. In contrast, the user of an MR system may find it intuitive and natural to interact with virtual objects by seeing, hearing, and touching the corresponding real objects within their real environment. This level of bidirectionality can enhance the user's immersion, connection, and engagement with the virtual environment. Similarly, by presenting the real and virtual environments simultaneously, an MR system can reduce negative psychological sensations (e.g., cognitive dissonance) and negative physical sensations (e.g., motion sickness) associated with VR systems. The MR system further offers many possibilities for applications that can extend or modify our real-world experiences.

[0036] FIG. 19A illustrates an exemplary real-world environment 1900 in which a user 1910 uses a mixed reality system 1912. The mixed reality system 1912 may include, for example, a display (e.g., a transparent display), one or more speakers, and one or more sensors (e.g., a camera), as described below. The illustrated real-world environment 1900 includes a rectangular room 1904A in which the user 1910 stands, and real-world objects 1922A (lamp), 1924A (table), 1926A (sofa), and 1928A (painting). The room 1904A further includes location coordinates 1906 that may be considered the origin of the real-world environment 1900. As shown in FIG. 19A, an environment / world coordinate system 1908 (including an x-axis 1908X, a y-axis 1908Y, and a z-axis 1908Z) with its origin at point 1906 (world coordinates) can define a coordinate space for the real-world environment 1900. In some embodiments, the origin 1906 of the environment / world coordinate system 1908 may correspond to the location where the mixed reality system 1912 is powered on. In some embodiments, the origin 1906 of the environment / world coordinate system 1908 may be reset during operation. In some examples, the user 1910 may be considered a real-world object within the real-world environment 1900, and similarly, the body parts of the user 1910 (e.g., hands, feet) may be considered real-world objects within the real-world environment 1900. In some examples, a user / listener / head coordinate system 1914 (including an x-axis 1914X, a y-axis 1914Y, and a z-axis 1914Z) with its origin at point 1915 (e.g., user / listener / head coordinates) can define a coordinate space for the user / listener / head where the mixed reality system 1912 is located. The origin 1915 of the user / listener / head coordinate system 1914 may be defined relative to one or more components of the mixed reality system 1912. For example, the origin 1915 of the user / listener / head coordinate system 1914 may be defined relative to the display of the mixed reality system 1912, such as during the initial calibration of the mixed reality system 1912. A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation can characterize the transformation between the user / listener / head coordinate system 1914 space and the environment / world coordinate system 1908 space.In some embodiments, the left ear coordinates 1916 and the right ear coordinates 1917 can be defined relative to the origin 1915 of the user / listener / head coordinate system 1914. A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix), or other suitable representation, can characterize the transformation between the left ear coordinates 1916 and the right ear coordinates 1917 and the user / listener / head coordinate system 1914 space. The user / listener / head coordinate system 1914 can simplify the representation of the location of the user's head or a head-mounted device relative to, for example, the environment / world coordinate system 1908. Using simultaneous localization and mapping (SLAM), visual odometry, or other techniques, the transformation between the user coordinate system 1914 and the environment coordinate system 1908 can be determined and updated in real time.

[0037] FIG. 19B illustrates an exemplary virtual environment 1930 corresponding to the real environment 1900. The illustrated virtual environment 1930 includes a virtual rectangular room 1904B corresponding to an actual rectangular room 1904A, a virtual object 1922B corresponding to an actual object 1922A, a virtual object 1924B corresponding to an actual object 1924A, and a virtual object 1926B corresponding to an actual object 1926A. The metadata associated with the virtual objects 1922B, 1924B, 1926B can include information derived from the corresponding actual objects 1922A, 1924A, 1926A. The virtual environment 1930 further includes a virtual monster 1932 that does not correspond to any actual object within the real environment 1900. The actual object 1928A within the real environment 1900 does not correspond to any virtual object within the virtual environment 1930. A persistent coordinate system 1933 (comprising an axis 1933X, a y-axis 1933Y, and a z-axis 1933Z) with its origin at point 1934 (persistent coordinates) can define a coordinate space for virtual content. The origin 1934 of the persistent coordinate system 1933 can be defined with respect to one or more actual objects such as the actual object 1926A. A matrix (which can include a translation matrix and a quaternion matrix or other rotation matrix), or other suitable representation, can characterize the transformation between the persistent coordinate system 1933 space and the environment / world coordinate system 1908 space. In some embodiments, the virtual objects 1922B, 1924B, 1926B, and 1932 can each have their own persistent coordinate points with respect to the origin 1934 of the persistent coordinate system 1933. In some embodiments, multiple persistent coordinate systems can exist, and the virtual objects 1922B, 1924B, 1926B, and 1932 can each have their own persistent coordinate points with respect to one or more persistent coordinate systems.

[0038] With respect to FIGS. 19A and 19B, the environment / world coordinate system 1908 defines a shared coordinate space for both the real environment 1900 and the virtual environment 1930. In the illustrated embodiment, the coordinate space has its origin at point 1906. Further, the coordinate space is defined by the same three orthogonal axes (1908X, 1908Y, 1908Z). Thus, a first location within the real environment 1900 and a second corresponding location within the virtual environment 1930 can be described with respect to the same coordinate space. This simplifies the identification and display of corresponding locations within the real and virtual environments because the same coordinates can be used to identify both locations. However, in some embodiments, the corresponding real and virtual environments need not use a shared coordinate space. For example, in some embodiments (not shown), a matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix), or other suitable representation, can characterize the transformation between the real environment coordinate space and the virtual environment coordinate space.

[0039] FIG. 19C illustrates an exemplary MR environment 1950 that simultaneously presents aspects of the real environment 1900 and the virtual environment 1930 to the user 1910 via the mixed reality system 1912. In the illustrated embodiment, the MR environment 1950 simultaneously presents real objects 1922A, 1924A, 1926A, and 1928A from the real environment 1900 (e.g., via the transmissive portion of the display of the mixed reality system 1912), and virtual objects 1922B, 1924B, 1926B, and 1932 from the virtual environment 1930 (e.g., via the active display portion of the display of the mixed reality system 1912) to the user 1910. As described above, the origin 1906 acts as the origin for the coordinate space corresponding to the MR environment 1950, and the coordinate system 1908 defines the x-axis, y-axis, and z-axis for the coordinate space.

[0040] In the illustrated embodiments, the composite reality object comprises a corresponding pair of real and virtual objects (i.e., 1922A / 1922B, 1924A / 1924B, 1926A / 1926B) that occupy corresponding locations within the coordinate space 1908. In some embodiments, both the real object and the virtual object may be visible to the user 1910 simultaneously. This may be desirable, for example, in cases where the virtual object presents information designed to augment the view of the corresponding real object (such as in a museum application where the virtual object presents the missing parts of an ancient damaged sculpture). In some embodiments, the virtual object (1922B, 1924B, and / or 1926B) may be displayed so as to occlude the corresponding real object (1922A, 1924A, and / or 1926A), for example, via active pixelization occlusion (using a pixelated occlusion shutter). This may be desirable, for example, in cases where the virtual object acts as a visual substitute for the corresponding real object (such as in a two-way reading application where an inanimate real object becomes a "living" character).

[0041] In some embodiments, a real object (e.g., 1922A, 1924A, 1926A) may be associated with virtual content or helper data that does not necessarily constitute a virtual object. The virtual content or helper data can facilitate the processing or handling of virtual objects within the composite reality environment. For example, such virtual content may include a two-dimensional representation of the corresponding real object, a custom asset type associated with the corresponding real object, or statistical data associated with the corresponding real object. This information can enable or facilitate calculations involving real objects without imposing unnecessary computational overhead.

[0042] In some embodiments, the presentation described above may also incorporate an audio aspect. For example, in the MR environment 1950, the virtual monster 1932 may be associated with one or more audio signals such as a footstep effect generated as the monster walks around the MR environment 1950. As further described below, the processor of the mixed reality system 1912 calculates an audio signal corresponding to the mixing and processed composite of all such sounds within the MR environment 1950, and can present the audio signal to the user 1910 via one or more speakers included in the mixed reality system 1912 and / or one or more external speakers.

[0043] (Mixed Reality Object Sharing)

[0044] A 3D digital model of an object may be displayed as a virtual object to one or more actual or digital people within an AR / VR / MR (hereinafter referred to as MR for simplicity) environment. For example, an automotive technician may desire to share the design of a new car with colleagues during their weekly team meeting. Conventionally, design sharing can be accomplished by providing several different views of the design so that viewers can imagine the 3D object. In this case, the automotive technician may print out perspective drawings to distribute to colleagues during the meeting. This can be useful for simple designs, but when the 3D object has a complex shape or design, it can be difficult for viewers to piece together the 3D object in their heads. Alternatively, or in addition, conventional methods of sharing 3D designs or models may involve creating a physical prototype. In this embodiment, the automotive technician would need to spend time and money creating a physical prototype. The physical prototype may make it easier to understand the designer's intent, but may only show a basic version of the design (viewers can only see the outside of the design, for example, they cannot open it to view internal components).

[0045] Embodiments of the disclosed systems and methods may provide improved 3D model sharing between computing systems. Continuing with the above example, the present invention may enable an automotive technician to create a new digital 3D model of a vehicle and then quickly and efficiently share the model with colleagues during a meeting. The present invention may also enable fast and easy ad-hoc sharing. For example, a manager of an automotive technician may desire to share a 3D design that the manager had previously designed in order to collaborate with the automotive technician. In some embodiments, the manager can access an old 3D vehicle design and share it with everyone attending the meeting without requiring people to download the design or click on a link to access it.

[0046] Typically, for example, in a game, 3D models (alternatively referred to as 3D assets or 3D content) are pre-loaded into the game application such that all of the 3D assets that will be visible to the user when the user starts the game are already on the device. When an update is required, the application will add new content offline. For example, the game may apply a patch when the game is not running, and when the application is next opened, the new content is installed and ready for use. This system of sharing a new 3D model between two or more computing systems is convenient for frequent, scheduled updates but is impractical when new 3D models are required frequently or unexpectedly.

[0047] An MR game or application can be collaborative, where multiple people, such as an entire group of students learning about the biological structure of frogs at school, contribute towards a common goal. If a teacher attempts to explain a concept and the students do not understand, it may be desirable for the teacher to display a virtual 3D model of a frog to the students so that what the teacher is explaining can be seen. Conventional systems would require the students to download the 3D model or close and re-enter the application so that the teacher can push a new frog model update to the students. Additionally, such methods may require a large amount of disk storage to accommodate all of the 3D asset data. An alternative method of accessing the 3D model without using disk space may be desirable.

[0048] These problems can be solved by a method and system for streaming data via a network between a server and one or more clients. The server may have access to one or more 3D models and may download and launch a 3D model sharing server application. The 3D model sharing application may decompose the model into component data parts, compress the component data parts, package them, and send them to a client over the network. The client may receive the components during the 3D model sharing client application runtime, reconstruct the 3D model, and be able to view the 3D model. In some embodiments, the runtime may be when the game or application loop is operating, when the screen is displaying application content, and / or when the computing system is operating with the application framework. Any number of clients may be part of the network. Any number of 3D models may be sent to the clients. Two or more clients may be one person with two or more devices (e.g., computing systems), or two or more people each with one device.

[0049] An advantage of the present application is to enable 3D content sharing during runtime.

[0050] The term "host" may be used synonymously with "server". The terms "3D model", "3D asset", "3D content", and "3D model" may be used synonymously. (Example of a 3D display of a wearable system)

[0051] A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present a 2D or 3D virtual image to a user. The image may be a still image, a frame of a video, or a video in a combination or equivalent. At least a part of the wearable system can be implemented on a wearable device that can present a VR, AR, or MR environment, alone or in combination, for user interaction. The wearable device can be used synonymously with an AR device (ARD). Further, for the purposes of the present disclosure, the term "AR" is used synonymously with the term "MR".

[0052] FIG. 1 depicts an illustration of a composite reality scenario with a virtual reality object and a physical object as viewed by a person. In FIG. 1, an MR scene 100 is depicted, and a user of MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, a user of MR technology also "sees" a robot statue 130 standing on the real-world platform 120 and a flying comic-like avatar character 140 that appears as an anthropomorphic bumblebee, but these elements do not exist in the real world.

[0053] For a 3D display to produce a true sense of depth, more specifically a simulated sense of surface depth, it may be desirable for each point within the field of view of the display to generate a focusing response corresponding to its virtual depth. If the focusing response for a display point does not correspond to the virtual depth of that point such that it is determined by both the convergence and stereoscopic binocular depth cues, the human eye may experience a focusing conflict, resulting in unstable imaging, harmful eye strain, headaches, and in the absence of focusing information, almost a complete lack of surface depth.

[0054] VR, AR, and MR experiences can be provided by a display system having a display that provides an image corresponding to a plurality of depth planes to a viewer. The images may vary for each depth plane (e.g., providing a somewhat different presentation of a scene or object), are separately focused by the viewer's eyes, and thereby serve to provide depth cues to the user based on the eye focusing required to focus on different image features of scenes located on different depth planes or based on observing different image features on different depth planes that are out of focus. As discussed anywhere herein, such depth cues can provide a believable perception of depth.

[0055] FIG. 2 illustrates an embodiment of a wearable system 200 that may be configured to provide an AR / VR / MR scene. Wearable system 200 may also be referred to as AR system 200. Wearable system 200 can include a display 220 and various mechanical and electronic modules and systems to support the functionality of display 220. Display 220 may be coupled to a frame 230, which can be worn by a user, wearer, or viewer 210. Display 220 can be positioned in front of the eyes of user 210. Display 220 can present AR / VR / MR content to the user. Display 220 can comprise a head-mounted display (HMD) that is worn on the user's head.

[0056] In some embodiments, the speaker 240 can be coupled to the frame 230 and positioned adjacent to the user's external auditory canal (in some embodiments, another speaker, not shown, can be positioned adjacent to the user's other external auditory canal to provide stereo / formable sound control). The display 220 can include an audio sensor (e.g., a microphone) 232 for detecting an audio stream from the environment and capturing ambient sound. In some embodiments, one or more other audio sensors, not shown, can be positioned to provide stereo sound reception. Stereo sound reception can be used to determine the location of the sound source. The wearable system 200 can perform speech or utterance recognition on the audio stream.

[0057] The wearable system 200 can include an outward-facing imaging system 464 (shown in FIG. 4) that observes the world within the user's surrounding environment. The wearable system 200 can also include an inward-facing imaging system 462 (shown in FIG. 4) that can track the user's eye movements. The inward-facing imaging system can track either the movement of one eye or both eyes. The inward-facing imaging system 462 may be attached to the frame 230 and may communicate electrically with a processing module 260 or 270 that can process the image information obtained by the inward-facing imaging system to determine, for example, the pupil diameter or orientation of the user 210's eyes, eye movement, or eye pose. The inward-facing imaging system 462 may include one or more cameras. For example, at least one camera may be used to image each eye. The images obtained by the cameras can be used to determine the pupil size or eye pose separately for each eye, thereby enabling the presentation of image information to each eye to be dynamically adjusted with respect to that eye.

[0058] As an example, the wearable system 200 can obtain an image of the user's posture using an outward-facing imaging system 464 or an inward-facing imaging system 462. The image may be a still image, a video frame, or a video.

[0059] The display 220 is operably coupled to the local data processing module 260 by means such as wired conductors or wireless connectivity, which can be mounted in various configurations, such as fixed to the frame 230, fixed to a helmet or hat worn by the user, built into headphones, or removably attached to the user 210 in another manner (e.g., in a backpack-type configuration, in a belt-coupled configuration). (250)

[0060] The local processing and data module 260 may comprise a digital memory such as a hardware processor and a non-volatile memory (e.g., flash memory), both of which may be utilized to assist in the processing, caching, and storage of data. The data may include (a) data captured from sensors such as an image capture device (e.g., a camera within an inward-facing imaging system and / or an outward-facing imaging system), an audio sensor (e.g., a microphone), an inertial measurement unit (IMU), an accelerometer, a compass, a global positioning system (GPS) unit, a wireless device, or a gyroscope (e.g., operatively coupled to the frame 230 or otherwise attachable to the user 210), or (b) potentially data obtained or processed using the remote processing module 270 or the remote data repository 280 for passage to the display 220 after processing or retrieval. The local processing and data module 260 may be operatively coupled to the remote processing module 270 or the remote data repository 280 via a communication link 262 or 264, such as a wired or wireless communication link, such that these remote modules may be available as resources to the local processing and data module 260. Additionally, the remote processing module 280 and the remote data repository 280 may be operatively coupled to each other.

[0061] In some embodiments, the remote processing module 270 may comprise one or more processors configured to analyze and process data or image information. In some embodiments, the remote data repository 280 may comprise a digital data storage facility, which may be available through other networking configurations in an Internet or "cloud" resource configuration. In some embodiments, all data may be capable of being stored and all calculations may be performed in the local processing and data module, enabling completely autonomous use from the remote modules. (Exemplary Components of a Wearable System)

[0062] Figure 3 schematically illustrates exemplary components of a wearable system. Figure 3 shows a wearable system 200 that may include a display 220 and a frame 230. An extended view 202 schematically illustrates various components of the wearable system 200. In one implementation, one or more of the components illustrated in Figure 3 may be part of the display 220. The various components can collect various data (e.g., audio or visual data, etc.) associated with a user of the wearable system 200 or the user's environment, either alone or in combination. It should be understood that other embodiments may have additional or fewer components depending on the use for which the wearable system is employed. Note that Figure 3 provides some basic concepts of some of the various components and types of data that may be collected, analyzed, and stored through the wearable system.

[0063] Figure 3 shows an exemplary wearable system 200 that may include a display 220. The display 220 can include a display lens 226 that can be mounted on a housing or frame 230 corresponding to the user's head or frame 230. The display lens 226 may include one or more transparent mirrors or diffractive optical elements positioned in front of the user's eyes 302, 304 by the housing 230, direct the projected light 338 into the eyes 302, 304, and be configured to allow transmission of at least some light from the local environment while facilitating beam shaping. The wavefront of the projected light beam 338 can diverge to match the desired focal length of the projected light. As shown, two wide field of view machine vision cameras 316 (also referred to as world cameras) are coupled to the housing 230 and can image the environment around the user. These cameras 316 can be dual capture visible / non-visible (e.g., infrared) light cameras. The cameras 316 may be part of an outward facing imaging system 464 shown in FIG. 4. Images obtained by the world cameras 316 can be processed by a pose processor 336. For example, the pose processor 336 can implement one or more object recognition devices 708 (e.g., shown in FIG. 7) and identify the pose of the user or another person within the user's environment or identify physical objects within the user's environment.

[0064] Continuing to refer to FIG. 3, a pair of optical projector modules with display optics and lenses configured to direct light 338 into eyes 302, 304 are shown. The depicted figure also shows two small infrared cameras 324 paired with infrared light (such as a light emitting diode “LED”) that can be configured to track the user's eyes 302, 304 and support rendering and user input. Camera 324 may be part of an inward-facing imaging system 462 shown in FIG. 4. Wearable system 200 may further feature a sensor assembly 339 with X, Y, and Z axis accelerometer capabilities and a magnetic compass and X, Y, and Z axis gyroscope capabilities, preferably providing data at a relatively high frequency such as 200 Hz. Sensor assembly 339 may be part of an IMU described with reference to FIG. 2A. The depicted system 200 may also include a head pose processor 336 such as an ASIC (application specific integrated circuit), FPGA (field programmable gate array), or ARM processor (advanced reduced instruction set machine) configured to calculate from wide field of view image information output from a capture device 316 of a real-time or near real-time user head pose. Head pose processor 336 may be a hardware processor and can be implemented as part of a local processing and data module 260 shown in FIG. 2A.

[0065] The wearable system may also include one or more depth sensors 234. Depth sensor 234 can be configured to measure the distance between objects in the environment and the wearable device. Depth sensor 234 may include a laser scanner (e.g., lidar), an ultrasonic depth sensor, or a depth sensing camera. In one implementation where camera 316 has depth sensing capabilities, camera 316 may also be considered a depth sensor 234.

[0066] Also shown is a processor 332 that can perform digital or analog processing and is configured to derive from gyroscope, compass, or accelerometer data from sensor assembly 339. Processor 332 may be part of the local processing and data module 260 shown in FIG. 2. Wearable system 200 as shown in FIG. 3 may also include a positioning system such as, for example, GPS 337 (Global Positioning System) and can assist with attitude and positioning analysis. Additionally, GPS may further provide remote base (e.g., cloud-based) information about the user's environment. This information may be used to recognize objects or information within the user's environment.

[0067] The wearable system may combine data obtained by GPS 337 and a remote computing system (e.g., remote processing module 270, another user's ARD, etc.) that can provide more information about the user's environment. As an example, the wearable system can determine the user's location based on GPS data and read out a world map that includes virtual objects associated with the user's location (e.g., by communicating with remote processing module 270). As another example, wearable system 200 can use world camera 316 (which may be part of the outward-facing imaging system 464 shown in FIG. 4) to monitor the environment. Based on the images obtained by world camera 316, wearable system 200 can detect objects within the environment (e.g., by using one or more object recognition devices 708 shown in FIG. 7). The wearable system can further use the data obtained by GPS 337 to interpret text.

[0068] The wearable system 200 may also include a rendering engine 334 configured to provide rendering information local to the user and to facilitate the operation of the scanner and the imaging into the user's eyes for the user's view of the world. The rendering engine 334 may be implemented by a hardware processor (e.g., a central processing unit or a graphics processing unit, etc.). In some embodiments, the rendering engine is part of the local processing and data module 260. The rendering engine 334 can be communicatively coupled to other components of the wearable system 200 (e.g., via a wired or wireless link). For example, the rendering engine 334 can be coupled to the eye camera 324 via the communication link 274 and to the projection subsystem 318 (which can project light into the user's eyes 302, 304 via a scanning laser array in a manner similar to a retinal scanning display) via the communication link 272. The rendering engine 334 can also communicate with other processing units, such as the sensor attitude processor 332 and the image attitude processor 336, respectively, via the links 276 and 294.

[0069] A camera 324 (e.g., a small infrared camera) may be used to track eye attitude and support rendering and user input. Some exemplary eye attitudes may include where the user is looking or the depth at which the user is focused (which can be estimated using eye convergence / divergence motion). A GPS 337, gyroscope, compass, and accelerometer 339 may be used to provide gross or high-speed attitude estimation. One or more of the cameras 316 can obtain images and attitudes that can be used to map the local environment and share the user view with others, in conjunction with data from associated cloud computing resources.

[0070] The exemplary components depicted in FIG. 3 are for illustrative purposes only. A plurality of sensors and other functional modules are shown together for ease of illustration and explanation. Some embodiments may include only one or a subset of these sensors or modules. Further, the location of these components is not limited to the positions depicted in FIG. 3. Some components may be mounted on or housed within other components such as belt-mounted components, handheld components, or helmet components. As an example, the image pose processor 336, the sensor pose processor 332, and the rendering engine 334 may be positioned within a belt pack and configured to communicate with other components of the wearable system via wireless communication such as ultra-wideband, Wi-Fi, Bluetooth®, or via wired communication. The depicted housing 230 is preferably head-mountable and wearable by a user. However, some components of the wearable system 200 may be worn on other parts of the user's body. For example, the speaker 240 may be inserted into the user's ear to provide sound to the user.

[0071] Regarding the projection of light 338 into the user's eyes 302, 304, in some embodiments, the camera 324 may generally be utilized to measure the location where the center of the user's eye is geometrically converged, which generally coincides with the position of the focus of the eye or the "depth of focus". The three-dimensional surface of all the points where the eyes converge may be referred to as the "horopter". The focal distance may take on a finite number of depths or may vary infinitely. Light projected from the convergence / divergence movement distance appears to be focused on the target eyes 302, 304, while light in front of or behind the convergence / divergence movement distance is blurred.

[0072] The human visual system is complex and it is difficult to provide a realistic perception of depth. An object viewer can perceive an object as three-dimensional due to the combination of convergence / divergence motion and accommodation. The convergence / divergence motion of two eyes with respect to each other (e.g., the rotational movement of the pupils towards or away from each other to converge the eye's line of sight and fixate on an object) is closely associated with the focusing (or "accommodation") of the eye's lens. Under normal conditions, a change in the focus of the eye's lens or the eye's accommodation to change the focus from one object to another object at a different distance will automatically cause a corresponding change in the convergence / divergence motion to the same distance under a relationship known as the "accommodation-convergence / divergence reflex". Similarly, a change in the convergence / divergence motion will, under normal conditions, induce a corresponding change in accommodation. A display system that provides a better match between accommodation and convergence / divergence motion can form a more realistic and comfortable simulation of a three-dimensional image.

[0073] Furthermore, spatially coherent light with a beam diameter of less than about 0.7 millimeters can be correctly resolved by the human eye regardless of where the eye is focused. Thus, in order to create an illusion of appropriate depth of focus, the eye's convergence / divergence motion may be tracked using camera 324, and the rendering engine 334 and projection subsystem 318 may be utilized to focus and render all objects on or near the convergence / divergence motion and render all other objects with varying degrees of variable resolution. Preferably, system 220 renders to the user at a frame rate of about 60 frames per second or more. As described above, preferably, camera 324 may be utilized for eye tracking and the software may be configured to take into account not only the convergence / divergence motion geometry but also a focus location queue for serving as a user input. Preferably, such a display system is configured using brightness and contrast suitable for daytime or nighttime use.

[0074] In some embodiments, the display system preferably has a latency of less than about 20 milliseconds, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 arc minute for visual object alignment, which, while not limited by theory, is considered to be approximately the limit of the human eye. The display system 220 may be integrated with a positioning system that may include a GPS element, optical tracking, a compass, an accelerometer, or other data sources to assist in position and orientation determination. The positioning information may be utilized to facilitate accurate rendering within the user's view of the relevant world (e.g., such information would facilitate the glasses' understanding of where they exist relative to the real world).

[0075] In some embodiments, the wearable system 200 is configured to display one or more virtual images based on the user's eye accommodation. Different from the conventional 3D display approach that forces the user to focus on where the image is projected, in some embodiments, the wearable system can be configured to automatically vary the focus of the projected virtual content to enable more comfortable viewing of the one or more images presented to the user. For example, if the user's eye has a current focus of 1 m, the image may be projected to coincide with the user's focus. If the user shifts the focus to 3 m, the image can be projected to coincide with the new focus. Thus, rather than forcing a predetermined focus on the user, the wearable system 200 of some embodiments can enable the user's eye to function in a more natural manner.

[0076] Such a wearable system 200 can eliminate or reduce the incidence of eye strain, headaches, and other physiological symptoms typically observed with virtual reality devices. To achieve this, various embodiments of the wearable system 200 can be configured to project virtual images at various focal distances through one or more variable focus elements (VFEs). In one or more embodiments, 3D perception can be achieved through a multi-plane focus system that projects the image onto a fixed focal plane away from the user. Other embodiments can employ variable plane focus, where the focal plane may reciprocate in the z-direction to match the current state of the user's focus.

[0077] In both the multi-plane focus system and the variable plane focus system, the wearable system 200 can employ eye tracking to determine the user's eye convergence / divergence motion, determine the user's current focus, and project the virtual image at the determined focus. In other embodiments, the wearable system 200 can include a light modulator that projects a variable focus light beam in a raster pattern across the retina through a fiber scanner or other light generation source. Thus, the ability of the display of the wearable system 200 to project an image at a variable focal distance may not only facilitate depth adjustment for the user to visually perceive an object in 3D, but may also be used to compensate for the user's eye abnormalities. In some other embodiments, the spatial light modulator may project the image to the user through various optical components. For example, as further described below, the spatial light modulator may project the image onto one or more waveguides, which then transmit the image to the user. (Waveguide stack assembly)

[0078] FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. Wearable system 400 can include a stack of waveguides or a stacked waveguide assembly 480 that can be utilized to provide three-dimensional perception to the eye / brain using a plurality of waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, wearable system 400 can correspond to wearable system 200 of FIG. 2, and FIG. 4A schematically shows some portions of that wearable system 200 in more detail. For example, in some embodiments, waveguide assembly 480 may be integrated within display 220 of FIG. 2.

[0079] Continuing to refer to FIG. 4, waveguide assembly 480 may also include a plurality of features 458, 456, 454, 452 between the waveguides. In some embodiments, features 458, 456, 454, 452 may be lenses. In other embodiments, features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., a cladding layer or structure for forming an air gap).

[0080] Waveguides 432b, 434b, 436b, 438b, 440b, or a plurality of lenses 458, 456, 454, 452 may be configured to transmit image information to the eye using various levels of wavefront curvature or ray divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. Image input devices 420, 422, 424, 426, 428 may each be utilized to input image information into waveguides 440b, 438b, 436b, 434b, 432b configured to disperse incident light across each individual waveguide for output toward the eye 410. Light may exit from the output surfaces of image input devices 420, 422, 424, 426, 428 and may be input into the corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be input into each waveguide and output across the entire field of sample exit pupil beams that may be directed toward the eye 410 at a particular angle (and amount of divergence) corresponding to a depth plane associated with a particular waveguide.

[0081] In some embodiments, image input devices 420, 422, 424, 426, 428 may each be discrete displays that each produce image information for input into the corresponding waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, image input devices 420, 422, 424, 426, 428 may be the output ends of a single multiplexed display that can send image information to each of image input devices 420, 422, 424, 426, 428, e.g., via one or more optical waveguides (such as an optical fiber cable).

[0082] The controller 460 can control the operation of the display and the image input devices 420, 422, 424, 426, 428. The controller 460 can include programming (e.g., instructions in a non-transitory computer-readable medium) that adjusts the timing and provision of image information to the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the controller 460 may be a single integrated device or a distributed system connected by a wired or wireless communication channel. The controller 460 may, in some embodiments, be part of the processing module 260 or 270 (illustrated in FIG. 2).

[0083] Waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Waveguides 440b, 438b, 436b, 434b, 432b may each be planar, or have another shape (e.g., curved), with a major top surface and a bottom surface and an edge extending between those major top and bottom surfaces. In the illustrated configuration, waveguides 440b, 438b, 436b, 434b, 432b may each include optical elements 440a, 438a, 436a, 434a, 432a configured to couple light out of the waveguide by diffracting or otherwise redirecting the light propagating within each individual waveguide. The externally coupled light can be output, for example, by the waveguide, at a location where the light propagating within the waveguide impinges on a light redirecting element such as a diffraction grating. The optical elements (440a, 438a, 436a, 434a, 432a) may be, for example, reflective or diffractive optical features. For ease of explanation and clarity of the drawings, optical elements 440a, 438a, 436a, 434a, 432a are shown disposed on the bottom major surfaces of waveguides 440b, 438b, 436b, 434b, 432b, but in some embodiments, optical elements 440a, 438a, 436a, 434a, 432a may be disposed on the top or bottom major surfaces, or may be disposed directly within the volume of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, optical elements 440a, 438a, 436a, 434a, 432a may be attached to a transparent substrate and formed within a layer of material forming waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, waveguides 440b, 438b, 436b, 434b, 432b may be a monolithic piece of material, and optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within the surface of that piece of material.

[0084] Continuing to refer to FIG. 4, in some embodiments, as discussed herein, each of the waveguides 440b, 438b, 436b, 434b, 432b can be configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is input into such waveguide 432b. The collimated light may represent an optically infinite focal plane. The next upper waveguide 434b may be configured to output collimated light that passes through a first lens 452 (e.g., a negative lens) before reaching the eye 410. The first lens 452 may be configured to create some convex wavefront curvature such that the eye / brain interprets the light originating from the next upper waveguide 434b as originating from a first focal plane that is closer inwardly toward the eye 410 from optically infinite. Similarly, the third upper waveguide 436b may be able to pass its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to create another incremental amount of wavefront curvature such that the eye / brain interprets the light originating from the third upper waveguide 436b as originating from a second focal plane that is even closer inwardly toward the person from optically infinite than the light from the next upper waveguide 434b.

[0085] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) can be similarly configured, and the top waveguide 440b in the stack sends its output through all of the lenses between it and the eye for the collective focusing power that represents the focal plane closest to the person. When viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensation lens layer 430 may be disposed at the top of the stack to compensate for the stack of lenses 458, 456, 454, 452 to compensate for the collective power of the lower lens stack 458, 456, 454, 452. Such a configuration can provide the same number of perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0086] Continuing to refer to FIG. 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to redirect light from their respective waveguides for a particular depth plane associated with the waveguide and output the light with an appropriate amount of divergence or collimation. As a result, waveguides having different associated depth planes may have different configurations of light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be three-dimensional or surface features configured to output light at specific angles. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be volume holograms, surface holograms, and / or diffraction gratings.

[0087] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be diffraction features or “diffractive optical elements” (also referred to herein as “DOEs”) that form a diffraction pattern. Preferably, the DOE has a relatively low diffraction efficiency such that only a portion of the light of the beam is deflected towards the eye 410 at each intersection of the DOE, while the remainder continues to move through the waveguide via total internal reflection. The light carrying the image information can thus be split into a plurality of associated output beams that exit the waveguide at a plurality of locations, resulting in a very uniform pattern of output emission towards the eye 304 with respect to this particular collimated beam that bounces within the waveguide.

[0088] In some embodiments, one or more DOEs can be switchable between an “on” state where they actively diffract and an “off” state where they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal in which microdroplets can be provided with a diffraction pattern, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract the incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts the incident light).

[0089] In some embodiments, the number and distribution of depth planes or depth of field may vary dynamically based on the pupil size or orientation of the viewer's eye. The depth of field may vary inversely with the pupil size of the viewer's eye. As a result, as the size of the pupil of the viewer's eye decreases, the depth of field increases such that one plane that was indistinguishable because its location in the plane was beyond the depth of focus of the eye becomes distinguishable and may appear more in focus with the corresponding increase in pupil size reduction and depth of field. Similarly, the number of separated depth planes used to present different images to the viewer may be decreased with a decreased pupil size. For example, a viewer may not be able to clearly perceive details of both a first depth plane and a second depth plane at one pupil size without adjusting the eye's focusing from one depth plane to the other. However, these two depth planes may be sufficiently in focus for the user at another pupil size without changing the focusing.

[0090] In some embodiments, the display system may vary the number of waveguides that receive image information based on a determination of pupil size or orientation or in response to receiving an electrical signal indicating a particular pupil size or orientation. For example, if the user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 460 (which may be an embodiment of the local processing and data module 260) can be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this can reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE for the waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide receives image information.

[0091] In some embodiments, it may be desirable to satisfy the condition that the emitted beam has a diameter less than the diameter of the viewer's eye. However, satisfying this condition can be difficult in light of the variability of the size of the viewer's pupil. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the emitted beam in response to determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the emitted beam may also decrease. In some embodiments, the emitted beam size may be varied using a variable aperture.

[0092] The wearable system 400 can include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 can be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes also referred to as the FOV camera. The FOV of the world camera may or may not be the same as the FOV of the viewer 210 that encompasses a portion of the world 470 that the viewer 210 perceives at a given time. For example, in some situations, the FOV of the world camera can be larger than the field of view of the viewer 210 of the wearable system 400. The entire area available for viewing or imaging by the viewer can be referred to as the field of regard (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400 since the wearer can move their body, head, or eyes and perceive substantially any direction within the space. In other contexts, the movement of the wearer may be more restricted, and accordingly, the wearer's FOR may contact a smaller solid angle. Images obtained from the outward-facing imaging system 464 can be used to track gestures made by the user (e.g., hand or finger gestures), detect objects within the world 470 in front of the user, and the like.

[0093] The wearable system 400 includes an audio sensor 232, such as a microphone, and can capture ambient sound. As described above, in some embodiments, one or more other audio sensors can be positioned to provide stereo sound reception useful for determining the location of the source of speech. As another example, the audio sensor 232 can comprise a directional microphone that can provide such useful directional information regarding the location where the audio source is located. The wearable system 400 can use information from both the outward-facing imaging system 464 and the audio sensor 230 when locating the source of speech or determining the active speaker at a particular moment. For example, the wearable system 400 can use voice recognition, alone or in combination with a reflected image of the speaker (such as seen in a mirror), to determine the identity of the speaker. As another example, the wearable system 400 can determine the location of a speaker within the environment based on the sound obtained from a directional microphone. The wearable system 400 can use a speech recognition algorithm to analyze the sound resulting from the location of the speaker, determine the content of the speech, and use voice recognition techniques to determine the identity of the speaker (e.g., name or other demographic information).

[0094] The wearable system 400 can also include an inward-facing imaging system 466 (e.g., a digital camera) that observes the user's movements such as eye movements and face movements. The inward-facing imaging system 466 can be used to capture an image of the eye 410 and determine the size and / or orientation of the pupil of the eye 304. The inward-facing imaging system 466 can be used to obtain an image for use in determining the direction the user is looking (e.g., eye posture), or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera can be used to independently determine the pupil size or eye posture of each eye separately, thereby enabling the presentation of image information to each eye to be dynamically adjusted with respect to that eye. In some other embodiments, only the pupil diameter or orientation of a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. The images obtained by the inward-facing imaging system 466 can be analyzed to determine the user's eye posture or mood, which can be used by the wearable system 400 to determine the audio or visual content to be presented to the user. The wearable system 400 can also use sensors such as an IMU, accelerometer, gyroscope, etc. to determine the head posture (e.g., head position or head orientation).

[0095] The wearable system 400 can include a user input device 466 through which a user can input commands to the controller 460 and interact with the wearable system 400. For example, the user input device 466 can include a trackpad, a touch screen, a joystick, a multi-degree of freedom (DOF) controller, a capacitance sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, a totem (e.g., functioning as a virtual user input device), etc. A multi-DOF controller can sense user input in some or all of the possible translations (e.g., left / right, forward / backward, or up / down) or rotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOF controller that supports translation can be referred to as 3DOF, while a multi-DOF controller that supports both translation and rotation can be referred to as 6DOF. In some cases, the user may use a finger (e.g., the thumb) to press or swipe on a touch sensor-based input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 can communicate with the wearable system 400 either wired or wirelessly. (Other components of the wearable system)

[0096] In many implementations, a wearable system may include, in addition to or instead of the components of the wearable system described above, other components. The wearable system may include, for example, one or more haptic devices or components. The haptic device or component may be operable to provide a tactile sensation to the user. For example, the haptic device or component may provide a tactile sensation of pressure or texture when touching virtual content (e.g., virtual objects, virtual tools, other virtual structures). The tactile sensation may reproduce the tactile sensation of a physical object represented by the virtual object, or may reproduce the sensation of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some implementations, the haptic device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the haptic device or component may be held by the user.

[0097] A wearable system may include, for example, one or more physical objects that are operable by a user and enable input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as pieces of metal or plastic, walls, or the surfaces of tables. In some implementations, a totem may not actually have any physical input structures (such as keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface so as to appear to the user to be on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad so as to appear to be resident on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad so as to appear on the surface of a thin rectangular plate of aluminum that serves as a totem. The rectangular plate itself may not have any physical keys or trackpads or sensors. However, the wearable system may detect user operations or interactions or touches using the rectangular plate as selections or inputs made via the virtual keyboard or virtual trackpad. The user input device 466 (shown in FIG. 4) may be an embodiment of a totem that may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. The user may use the totem alone or in combination with a gesture to interact with the wearable system or other users. (Exemplary Process of User Interaction with Wearable System)

[0098] Figure 5 is a process flow diagram of an example of a method 500 for interacting with a virtual user interface. Method 500 may be implemented by the wearable system described herein. Embodiments of method 500 are used by a wearable system and can detect a person or document within the FOV of the wearable system.

[0099] In block 510, the wearable system may identify a particular UI. The type of UI may be pre-determined by the user. The wearable system may identify that a particular UI needs to be captured based on user input (e.g., gesture, visual data, audio data, sensory data, direct command, etc.). The UI may be specific to a security scenario where the wearer of the system observes a user presenting a document to the wearer (e.g., at a traveler checkpoint). In block 520, the wearable system may generate data for the virtual UI. For example, data associated with the boundaries, general structure, shape, etc. of the UI may be generated. Additionally, the wearable system may determine map coordinates of the user's physical location so that the wearable system can display the UI in relation to the user's physical location. For example, if the UI is centered on the body, the wearable system may determine the coordinates of the user's physical standing position, head pose, or eye pose so that a ring UI can be displayed around the user or a planar UI can be displayed on a wall or in front of the user. In the security context described herein, the UI may be displayed as if it surrounds the traveler presenting the document to the wearer of the system so that the wearer can easily view the UI while looking at the traveler and the traveler's document. If the UI is centered on the hand, the map coordinates of the user's hand may be determined. These map points may be derived through the FOV camera, data received through sensory input, or any other type of collected data.

[0100] In block 530, the wearable system may send data from the cloud to the display, or the data may be sent from the local database to the display component. In block 540, the UI is presented to the user based on the sent data. For example, a light field display can project a virtual UI into one or both of the user's eyes. Once the virtual UI is created, the wearable system may simply wait for commands from the user in block 550 and generate more virtual content on the virtual UI. For example, the UI may be a body-centered ring around the user's body or the body of a person (e.g., a traveler) in the user's environment. The wearable system may then wait for commands (e.g., gestures, head or eye movements, voice commands, input from a user input device, etc.), and if recognized (block 560), the virtual content associated with the command may be presented to the user (block 570). (Examples of Avatar Rendering in Mixed Reality)

[0101] The wearable system may employ various mapping-related techniques to achieve high depth of field within the rendered light field. When mapping the virtual world, it is advantageous to capture all features and points in the real world and accurately depict virtual objects in relation to the real world. To achieve this, the FOV images captured from the user of the wearable system can be added to the world model by including new photos that convey information about various points and features of the real world. For example, the wearable system can collect a set of map points (such as 2D or 3D points), find new map points, and render a more accurate version of the world model. The world model of a first user can be communicated to a second user (e.g., via a network such as a cloud network) so that the second user can experience the world surrounding the first user.

[0102] FIG. 6A is a block diagram of another embodiment of a wearable system that may include an avatar processing and rendering system 690 in a composite reality environment. Wearable system 600 may be part of wearable system 200 shown in FIG. 2. In this embodiment, wearable system 600 can include a map 620 that may include at least a portion of the data in map database 710 (shown in FIG. 7). The map may be resident locally on the wearable system in part and may be resident in part in a networked storage location accessible by a wired or wireless network (e.g., within a cloud system). Pose process 610 is executed on a wearable computing architecture (e.g., processing module 260 or controller 460) and may utilize data from map 620 to determine the position and orientation of the wearable computing hardware or the user. The pose data may be calculated from data collected on-the-fly as the user experiences the system and moves within the world. The data may include images, data from sensors (e.g., an inertial measurement unit generally comprising accelerometer and gyroscope components), and surface information associated with objects in the real or virtual environment.

[0103] Sparse representations may be the output of a simultaneous localization and mapping (e.g., SLAM or vSLAM, which refers to configurations where the input is only image / vision) process. The system can be configured to find not only locations within the world where various components exist but also what the world consists of. Pose can be a building block that achieves many goals, including map ingestion and use of data from the map.

[0104] In one embodiment, the sparse point locations may not be entirely proper in themselves, and additional information may be required to produce a multi-focus AR, VR, or MR experience. Generally, a dense representation, which refers to depth map information, may be used at least in part to fill in the gaps. Such information may be calculated from a process called stereo 640, and the depth information may be determined using techniques such as triangulation or time-of-flight sensing. Image information and an active pattern (such as an infrared pattern created using an active projector), an image obtained from an image camera, or a hand gesture / totem 650 may serve as inputs to the stereo process 640. A significant amount of depth map information may be fused together, and some of it may be summarized using surface representations. For example, a mathematically definable surface may be an efficient (e.g., for large point clouds) and easy-to-summarize input to other processing devices such as a game engine. Thus, the output of the stereoscopic process (e.g., depth map) 640 may be combined in the fusion process 630. The pose 610 may similarly be an input to this fusion process 630, and the output of the fusion 630 becomes an input to the map ingestion process 620. Sub-surfaces may connect with each other in topographic mapping and may form larger surfaces, and the map becomes a large-scale hybrid of points and surfaces.

[0105] To address various aspects in the composite reality process 660, various inputs may be utilized. For example, in the embodiment depicted in FIG. 6A, game parameters are input, and the user of the system can determine that they are playing a monster battle game with one or more monsters in various locations, monsters that are dying or fleeing under various conditions (such as when the user shoots at the monster), walls or other objects in various locations, and equivalents. The world map may include information regarding the location of objects or semantic information of objects (e.g., classification such as whether the object is flat or round, horizontal or vertical, a table or a lamp, etc.), and the world map can be another beneficial input to the composite reality. Similarly, the posture with respect to the world can also serve as an input and can play an important role for almost any two-way system.

[0106] Control or input from the user can be another input to the wearable system 600. As described herein, user input can include visual input, gestures, totems, audio input, sensory input, etc. For example, in order to move around or play a game, the user may need to command the wearable system 600 regarding what they desire. In addition to simply moving themselves within the space, various forms of user control that can be utilized may exist. In one embodiment, an object such as a totem (e.g., a user input device) or a toy gun may be held by the user and tracked by the system. The system will preferably be configured to ascertain that the user is holding the item and understand the type of interaction the user is having with the item (e.g., if the totem or object is a gun, the system may be equipped with sensors such as an IMU to assist in determining what is occurring, including the location and orientation, and even when such an activity is not within the field of view of any of the cameras, and may be configured to understand whether the user is clicking a trigger or other sensing button or element).

[0107] Tracking or recognition of hand gestures may also provide input information. The wearable system 600 may be configured to track and interpret hand gestures with respect to button presses, left or right, stop, grip, hold gestures, etc. For example, in one configuration, a user may desire to flip through email or a calendar in a non-game environment or perform a "fist bump" with another person or player. The wearable system 600 may be configured to utilize a minimal amount of hand gestures, which may be dynamic or not. For example, the gesture may be a simple static gesture such as opening the hand for stop, raising the thumb for OK, lowering the thumb for not OK, or flipping the hand right or left or up / down for a directional command.

[0108] Eye tracking may be another input (e.g., tracking where the user is looking, controlling display technology, and rendering to a specific depth or range). In one embodiment, the convergence / divergence movement of the eyes may be determined using triangulation, and then the focus adjustment may be determined using a convergence / divergence movement / focus adjustment model developed for that particular person. Eye tracking may be performed by an eye camera to determine the line of sight of the eyes (e.g., the direction or orientation of one or both eyes). For example, other techniques such as measuring the electrical potential by electrodes placed near the eyes (e.g., electrooculogram recording) may also be used for eye tracking.

[0109] Speech tracking can be another input and can be used alone or in combination with other inputs (e.g., totem tracking, eye tracking, gesture tracking, etc.). Speech tracking may include speech recognition, voice recognition, either alone or in combination. System 600 can include an audio sensor (e.g., a microphone) that receives an audio stream from the environment. System 600 can incorporate voice recognition technology to determine the person speaking (e.g., whether the speech is from the wearer of the ARD, or from another person, or from a voice (e.g., recorded voice transmitted by a loudspeaker in the environment)), and speech recognition technology to determine the content being spoken. The local data and processing module 260 or the remote processing module 270 can process audio data from the microphone (or audio data within another stream, such as a video stream being viewed by the user, for example) and identify the content of the speech by applying various speech recognition algorithms, such as hidden Markov models, dynamic time warping (DTW)-based speech recognition, neural networks, deep learning algorithms such as deep feedforward and recurrent neural networks, end-to-end automatic speech recognition, machine learning algorithms (described with reference to FIG. 7), or other algorithms that use acoustic modeling or language modeling.

[0110] The local data and processing module 260 or the remote processing module 270 can also apply a voice recognition algorithm that can identify the identity of the speaker, such as whether the speaker is the user 210 of the wearable system 600 or another person the user is conversing with. Some exemplary voice recognition algorithms can include frequency estimation, hidden Markov models, Gaussian mixture models, pattern matching algorithms, neural networks, tabular representations, vector quantization, speaker diarization, decision trees, and dynamic time warping (DTW) techniques. Voice recognition techniques can also include anti-speaker techniques such as cohort models and world models. Spectral features may be used in representing speaker characteristics. The local data and processing module or the remote data processing module 270 can perform voice recognition using various machine learning algorithms described with reference to FIG. 7.

[0111] The implementation of the wearable system can use these user controls or inputs via the UI. UI elements (e.g., controls, pop-up windows, balloons, data entry fields, etc.) can be used, for example, to close the display of information, such as graphic or semantic information of an object.

[0112] Regarding a camera system, the exemplary wearable system 600 shown in FIG. 6A can include three pairs of cameras, namely, a relatively wide FOV or passive SLAM pair of cameras arranged on the sides of the user's face, and a different pair of cameras oriented in front of the user's face to handle the stereoscopic imaging process 640 and to capture hand gestures and totem / object tracking in front of the user's face. The pair of cameras for the FOV cameras and the stereoscopic process 640 may be part of the outward-facing imaging system 464 (shown in FIG. 4). The wearable system 600 can include an eye-tracking camera (which can be part of the inward-facing imaging system 462 shown in FIG. 4) oriented towards the user's eyes to triangulate eye vectors and other information. The wearable system 600 may also include one or more textured light projectors (such as an infrared (IR) projector) to project textures into the scene.

[0113] The wearable system 600 can include an avatar processing and rendering system 690. The avatar processing and rendering system 690 can be configured to generate, update, animate, and render an avatar based on context information. Some or all of the avatar processing and rendering system 690 can be implemented alone or in combination as part of the local processing and data module 260 or the remote processing modules 262, 264. In various embodiments, multiple avatar processing and rendering systems 690 (such as those implemented on different wearable devices) can be used to render the virtual avatar 670. For example, a wearable device of a first user may be used to determine the intent of the first user, while a wearable device of a second user can determine the characteristics of the avatar and render the avatar of the first user based on the intent received from the wearable device of the first user. The wearable device of the first user and the wearable device of the second user (or other such wearable devices) can communicate via a network, as will be described, for example, with reference to FIG. 9.

[0114] FIG. 6B illustrates an exemplary avatar processing and rendering system 690. The exemplary avatar processing and rendering system 690 can include, alone or in combination, a 3D model processing system 680, a context information analysis system 688, an avatar automatic scaler 692, an intent mapping system 694, a biological structure adjustment system 698, and a stimulus response system 696. The system 690 is intended to illustrate functionality for avatar processing and rendering and is not intended to be limiting. For example, in one implementation, one or more of these systems may be part of another system. For example, a part of the context information analysis system 688 may be part of the avatar automatic scaler 692, the intent mapping system 694, the stimulus response system 696, or the biological structure adjustment system 698, individually or in combination.

[0115] The context information analysis system 688 can be configured to determine environmental and object information based on one or more device sensors, as described with reference to FIGS. 2 and 3. For example, the context information analysis system 688 can use an image obtained by an imaging system 464 facing outward of the viewer of the user or the user's avatar to analyze the environment and the objects (including physical or virtual objects) in the user's environment or the environment in which the user's avatar is rendered. The context information analysis system 688 can analyze such an image, alone or in combination with data obtained from location data or a world map (e.g., maps 620, 710, 910), to determine the location and layout of the objects in the environment. The context information analysis system 688 can also access the biological characteristics of the user or humans in general to realistically animate the virtual avatar 670. For example, the context information analysis system 688 can generate a discomfort curve that can be applied to the avatar so that a part of the user's avatar's body (e.g., the head) cannot be in a position that is uncomfortable (or unrealistic) relative to other parts of the user's body (e.g., the avatar's head is not rotated 270 degrees). In one implementation, one or more object recognition devices 708 (shown in FIG. 7) may be implemented as part of the context information analysis system 688.

[0116] Avatar automatic scaler 692, intention mapping system 694, and stimulus response system 696, and biological structure adjustment system 698 can be configured to determine the characteristics of an avatar based on context information. Some exemplary characteristics of an avatar can include size, appearance, position, orientation, movement, pose, expression, etc. The avatar automatic scaler 692 can be configured to automatically scale the avatar so that the user does not have to view the avatar in an uncomfortable pose. For example, the avatar automatic scaler 692 can increase or decrease the size of the avatar and bring the avatar to the user's eye level so that the user does not have to look down at or look up at the avatar, respectively. The intention mapping system 694 can determine the intention of the user's interaction based on the environment in which the avatar is rendered and map the intention (rather than the exact user interaction) to the avatar. For example, the intention of the first user can be to communicate with the second user in a telepresence session (see, e.g., FIG. 9B). Typically, two people face each other when communicating. The intention mapping system 694 of the first user's wearable system can determine such a face-to-face intention that exists during the telepresence session, and cause the second user's avatar to be rendered towards the first user on the first user's wearable system. If the second user attempts to physically turn, instead of rendering the second user's avatar in the turned position (which would render the back of the second user's avatar to the first user), the intention mapping system 694 of the first user can continue to render the face of the second avatar to the first user, which is the presumed intention of the telepresence session (e.g., in this example, the face-to-face intention).

[0117] The stimulus response system 696 can identify an object of interest in the environment and determine the avatar's response to the object of interest. For example, the stimulus response system 696 can identify a sound source within the avatar's environment and automatically turn the avatar's direction to look at the sound source. The stimulus response system 696 can also determine a threshold end condition. For example, the stimulus response system 696 can return the avatar to its original posture after the sound source disappears or after a certain time period has elapsed.

[0118] The body structure adjustment system 698 can be configured to adjust the user's posture based on biological characteristics. For example, the body structure adjustment system 698 can be configured to adjust the relative position between the user's head and the user's torso or between the user's upper body and lower body based on a discomfort curve.

[0119] The 3D model processing system 680 can be configured to animate the virtual avatar 670 and render it on the display 220. The 3D model processing system 680 can include a virtual character processing system 682 and a movement processing system 684. The virtual character processing system 682 can be configured to generate a user's 3D model and update it (for creating and animating the virtual avatar). The movement processing system 684 can be configured to animate the avatar, for example, by changing the avatar's pose, moving the avatar around in the user's environment, or animating the avatar's facial expressions. As will be further described herein, the virtual avatar can be animated using rigging techniques. In some embodiments, the avatar is represented in two parts, namely, a surface representation (e.g., a deformable mesh) used to render the outward appearance of the virtual avatar, and a hierarchical set of interconnected joints (e.g., a core skeleton) for animating the mesh. In some implementations, the virtual character processing system 682 can be configured to edit or generate the surface representation, while the movement processing system 684 can be used to animate the avatar by moving the avatar, deforming the mesh, etc. (Example of Mapping the User's Environment)

[0120] FIG. 7 is a block diagram of an example of an MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual input 702 from a user's wearable system, stationary input 704 such as an indoor camera, sensory input 706 from various sensors, gestures, totems, eye tracking, user input, etc. from a user input device 466) from one or more user wearable systems (e.g., wearable system 200 or display system 220) or stationary indoor systems (e.g., indoor cameras, etc.). The wearable system may use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, motion sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.) to determine the location and various other attributes of the user's environment. This information may further be complemented with information from stationary cameras in the room that may provide images or various cues from different viewpoints. The image data obtained by the cameras (such as indoor cameras and / or cameras of outward-facing imaging systems) may be transformed into a set of mapping points.

[0121] One or more object recognition devices 708 may crawl through the received data (e.g., a set of points), recognize or map the points, tag the images, and attach semantic information to the objects using a map database 710. The map database 710 may comprise various points collected over time and their corresponding objects. The various devices and the map database may be interconnected with each other through a network (e.g., LAN, WAN, etc.) and may access the cloud.

[0122] Based on the present information and the set of points in the map database, the object recognition devices 708a - 708n may recognize objects in the environment. For example, the object recognition device may recognize faces, people, windows, walls, user input devices, televisions, documents (e.g., passports, driver's licenses, passports as described in the security embodiments herein), other objects in the user's environment, and the like. One or more object recognition devices may be specialized for objects with certain characteristics. For example, the object recognition device 708a may be used to recognize faces, while another object recognition device may be used to recognize documents.

[0123] Object recognition may be performed using various computer vision techniques. For example, a wearable system may analyze an image obtained by an outward-facing imaging system 464 (shown in FIG. 4) to perform scene reconstruction, event detection, video tracking, object recognition (e.g., of a person or document), object pose estimation, face recognition (e.g., from an image of a person in an environment or on a document), learning, indexing, motion estimation, or image analysis (e.g., identifying marks within a document such as a photo, signature, identification information, travel information, etc.). One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) techniques, Sequential Bayesian estimators (e.g., Kalman filter, Extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), Iterative Closest Point (ICP), Semi-Global Matching (SGM), Semi-Global Block Matching (SGBM), feature point histograms, various machine learning algorithms (e.g., support vector machine, k-nearest neighbor algorithm, naive Bayes, neural network (including convolutional or deep neural network), or other supervised / unsupervised models, etc.).

[0124] Object recognition can additionally or alternatively be performed by a variety of machine learning algorithms. Once trained, the machine learning algorithms can be stored by the HMD. Some examples of machine learning algorithms include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., Apriori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural network, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., Stacked Generalization, etc.), and / or other machine learning algorithms. In some embodiments, individual models can be customized for individual datasets. For example, a wearable device can generate or store a base model. The base model is used as a starting point and may generate additional models specific to a data type (e.g., a particular user within a telepresence session), a dataset (e.g., a set of additional images acquired of the user within a telepresence session), a conditional situation, or other variations. In some embodiments, the wearable HMD can be configured to generate a model for the analysis of aggregated data using multiple techniques. Other techniques may include steps that use predefined thresholds or data values.

[0125] Based on the book information and set of points within the map database, the object recognition devices 708a - 708n may recognize an object, complement the object with semantic information, and give the object operability. For example, when the object recognition device recognizes that the set of points is a door, the system may attach certain semantic information (e.g., the door has hinges and has a 90 - degree movement around the hinges). When the object recognition device recognizes that the set of points is a mirror, the system may attach semantic information that the mirror has a reflective surface that can reflect images of objects within the room. Semantic information can include, as described herein, the affordances of the object. For example, the semantic information may include the normal of the object. The system can assign a vector, the direction of which indicates the normal of the object. Over time, the map database can grow as the system (which may be resident locally or accessible through a wireless network) accumulates more data from the world. Once an object is recognized, the information may be transmitted to one or more wearable systems. For example, the MR environment 700 may include information about a scene occurring in California. The environment 700 may be transmitted to one or more users in New York. Based on data received from the FOV camera and other inputs, the object recognition device and other software components can map points collected from various images so that the scene can be accurately "passed" to a second user who may be present in a different part of the world, and can recognize objects, etc. The environment 700 may also use a topological map for location - specific purposes.

[0126] FIG. 8 is a process flow diagram of an example of a method 800 for rendering virtual content in relation to recognized objects. Method 800 describes a way in which a virtual scene can be presented to a user of a wearable system. The user can be geographically remote from the scene. For example, the user can be present in New York but may desire to view a scene currently taking place in California or may desire to go for a walk with a friend who lives in California.

[0127] In block 810, the wearable system may receive input regarding the user's environment from the user and other users. This can be accomplished through various input devices and knowledge already held within a map database. The user's FOV camera, sensors, GPS, eye tracking, etc. communicate information to the system in block 810. The system may, in block 820, determine sparse points based on this information. The sparse points can be used when determining pose data (e.g., head pose, eye pose, body pose, or hand gesture) that can be used to display and understand the orientation and position of various objects in the user's surroundings. The object recognition devices 708a - 708n may, in block 830, crawl through these collected points and use the map database to recognize one or more objects. This information may then be communicated to the user's individual wearable system in block 840, and a desired virtual scene may be displayed to the user accordingly in block 850. For example, a desired virtual scene (e.g., for a user in CA) may be displayed in appropriate orientation, position, etc. in relation to the various objects and other surroundings of the user in New York. (Exemplary communication between multiple wearable systems)

[0128] FIG. 9 schematically illustrates an overall system diagram depicting a plurality of user devices that interact with each other. Computing environment 900 includes user devices 930a, 930b, 930c. User devices 930a, 930b, and 930c can communicate with each other through network 990. Each of user devices 930a - 930c includes a network interface and can communicate with remote computing system 920 (which may also include network interface 971) via network 990. Network 990 may be a LAN, WAN, peer - to - peer network, wireless, Bluetooth®, or any other network. Computing environment 900 may also include one or more remote computing systems 920. Remote computing system 920 may include server computer systems that are clustered and located at different geographical locations. User devices 930a, 930b, and 930c may communicate with remote computing system 920 via network 990.

[0129] The remote computing system 920 may include a remote data repository 980 that can maintain information about a specific user's physical and / or virtual world. The data storage device 980 can store information related to the user, the user's environment (e.g., a world map of the user's environment), or the configuration of the user's avatar. The remote data repository may be an embodiment of the remote data repository 280 shown in FIG. 2. The remote computing system 920 may also include a remote processing module 970. The remote processing module 970 may be an embodiment of the remote processing module 270 shown in FIG. 2. The remote processing module 970 may include one or more processors that can communicate with user devices (930a, 930b, 930c) and the remote data repository 980. The processors can process information obtained from user devices and other sources. In some implementations, at least a portion of the processing or storage can be provided by a local processing and data module 260 (as shown in FIG. 2). The remote computing system 920 may enable a given user to share information about the specific user's own physical and / or virtual world with another user.

[0130] The user devices may be, individually or in combination, wearable devices (such as HMDs or ARDs), computers, mobile devices, or any other device. For example, user devices 930b and 930c may be embodiments of the wearable system 200 (or the wearable system 400 shown in FIG. 4) configured to present AR / VR / MR content as shown in FIG. 2.

[0131] One or more of the user devices can be used in conjunction with the user input device 466 shown in FIG. 4. The user devices can obtain information about the user and the user's environment (e.g., using the outward-facing imaging system 464 shown in FIG. 4). The user devices and / or the remote computing system 1220 can use the information obtained from the user devices to construct, update, and build a collection of images, points, and other information. For example, the user device can process the raw information obtained and send the processed information to the remote computing system 1220 for further processing. The user device can also send the raw information to the remote computing system 1220 for processing. The user device can receive the processed information from the remote computing system 1220 and provide final processing before projecting it to the user. The user device can also process the information obtained and pass the processed information to other user devices. The user device can communicate with the remote data repository 1280 while processing the information obtained. Multiple user devices and / or multiple server computer systems can be involved in the construction and / or processing of the obtained images.

[0132] Information about the physical world may evolve over time and may be based on information collected by different user devices. Models of the virtual world may also evolve over time and may be based on the inputs of different users. Such information and models may sometimes be referred to herein as a world map or a world model. As described with reference to FIGS. 6 and 7, the information obtained by the user devices may be used to construct a world map 910. The world map 910 may include at least a portion of the map 620 described in FIG. 6A. Various object recognition devices (e.g., 708a, 708b, 708c…708n) may be used to recognize objects and tag images and to attach semantic information to the objects. These object recognition devices are also described in FIG. 7.

[0133] The remote data repository 980 can be used to store data and facilitate the construction of the world map 910. The user device can constantly update information about the user's environment and receive information about the world map 910. The world map 910 may be created by the user or by another person. As discussed herein, the user devices (e.g., 930a, 930b, 930c) and the remote computing system 920 may construct and / or update the world map 910 alone or in combination. For example, the user device may communicate with the remote processing module 970 and the remote data repository 980. The user device may obtain and / or process information about the user and the user's environment. The remote processing module 970 may communicate with the remote data repository 980 and the user devices (e.g., 930a, 930b, 930c) and process information about the user and the user's environment. The remote computing system 920 may modify the information obtained by the user devices (e.g., 930a, 930b, 930c), such as selectively cropping the user's image, modifying the user's background, adding virtual objects to the user's environment, annotating the user's speech with auxiliary information, etc. The remote computing system 920 can transmit the processed information to the same and / or different user devices. (Exemplary Process for 3D Model Sharing)

[0134] Figure 10 illustrates an exemplary process 1000 for sharing 3D assets using the systems and methods described herein. At step 1001, a 3D sharing host application can access a 3D model file from a host system. The host may do this either by creating a new 3D asset on the host system or by obtaining a 3D asset from an external system. The 3D asset can originate in a 3D modeling application such as AutoCad or 3D Studio Max and can be of any file type. The host may utilize an asset import plug-in (e.g., ASSIMP) to enable graphic format exchange. The ASSIMP plug-in can support up to 40 different file formats such as, for example, Filmbox (FBX) in the.fbx format or Autodesk Max 3D modeling (3DS) in the.3ds format. In some embodiments, a custom program may be written to enable file format exchange. Any other suitable file format conversion method may be used to enable file reformating for use within the host 3D sharing application.

[0135] At step 1003, the 3D sharing host application can decompose the 3D asset into components. Examples of components may be geographic data, material data, vertex tables, one or more textures, triangle indices, or any other data used to ultimately define a complete representation of the 3D model.

[0136] In step 1005, the component can be sent to a client that is part of the network with the host. In some embodiments, there may be only one client. In some embodiments, there may be two or more clients. In a preferred embodiment, a local area network may be utilized using the UDP protocol. The network may be a LAN, WAN, peer-to-peer network, wireless, Bluetooth®, ad hoc, or any other network. In some embodiments, the network may be network 990 as described above. The host may be a remote computing system that may include server computer systems that are clustered and located in different geographical locations, as described above in connection with 920. The client may be user devices 930a, 930b, and / or 930c as described above. In some embodiments, there may be one host. In some embodiments, there may be two or more hosts. In some embodiments, each device connected to the network may in effect be both a host and a client that performs both sharing and receiving of 3D assets from other devices on the network.

[0137] In step 1007, the client can receive the component.

[0138] In step 1009, the client can reconstruct a 3D asset from the component received from the host. In some embodiments, the 3D asset can be stored in the memory of the client device. In some embodiments, the 3D asset may be saved for later use by the device.

[0139] In step 1011, the client can display the reconstructed 3D asset. In some embodiments, the client may not display the reconstructed 3D asset, or may display the asset at a later time, for example, in response to a user request. In some embodiments, the client may display the reconstructed 3D asset, but the user may not be able to view the reconstructed 3D asset because it is outside the user's FOV. (Exemplary 3D Model Sharing System Configuration)

[0140] FIG. 11 illustrates an exemplary 3D model sharing system configuration 1100 for sharing 3D assets using the systems and methods described herein. FIG. 11 depicts a system configuration 1100 that includes one client (which may be referred to as "remote") system 1116 and one server (which may be referred to as "host") system 1118. The client system and the host system need not be physically separated, and in some embodiments, for example, the client and the server can exist on the same physical device (e.g., as client and server threads executed in parallel by one or more processors of the same computer system). In practice, the 3D sharing system 1100 may generally require at least one host system 1118 that can function as a source of decomposed 3D assets. The host system 1118 may have authoritative control over the types and quantities of 3D models available within the 3D sharing application. The 3D sharing system 1100 may have any number (e.g., 0, 1, 10, etc.) of client systems 1116 operably connected to the host system 1118 through a communication network 1114. The communication network 1114 may be, for example, a LAN that utilizes the User Datagram Protocol (UDP). The host system 1118 may include one or more processors operably coupled to at least one display and capable of receiving communications from other computing systems. In some embodiments, the host system 1118 may be a PC desktop computer attached to a computer monitor using a network interface. In some embodiments, the host system 1118 may be a system 200, 930a, 930b, or 930c as described above. The user devices 200, 930a, 930b, and 930c can communicate with each other and with other computing systems through the communication network 1114.User devices 930a - 930c can each include a network interface (which may also include network interface 971) for communicating with a remote computing system such as 920 via network 1114. Client system 1116 may comprise one or more processors operably coupled to at least one display and capable of receiving communications from other computing systems. In some embodiments, client system 1116 may be a PC desktop computer attached to a computer monitor using a network interface. In some embodiments, client system 1118 may be a system 200, 930a, 930b, or 930c as described above. User devices 200, 930a, 930b, and 930c can communicate with each other and with other computing systems through communication network 1114. User devices 930a - 930c can each include a network interface (which may also include network interface 971) for communicating with a remote computing system such as 920 via network 1114.

[0141] The host system 1118 may have a 3D sharing host application installed. The 3D sharing host application may include three main modules, namely, an asset module 1108, a decomposition module 1110, and a transmission module 1112. The asset module 1108 may include one or more complete 3D models. The decomposition module 1110 may function to decompose the complete 3D model into components, optionally compress the components, and place the components into one or more arrays that are ready to be transmitted to different systems. The arrays may be stored in one or more libraries that may contain one or more decomposed 3D models. The transmission module 1112 may function to decompose the components from the decomposition module 1110 into parts that are transferable of data and may manage transmission protocols and processes. The transmission module 1112 may transmit the transferable parts of the data to other systems connected over the communication network 1114.

[0142] The client or remote system 1116 may have a 3D sharing client application installed. The 3D sharing client application may include three main modules, namely, a transmission module 1106, a reconstruction module 1104, and an asset module 1102. The transmission module 1106 may function to receive one or more decomposed 3D models and may manage the transmission and networking processes for the client system 1116. The transmission module 1106 may communicate with the 3D model sharing host application through a messaging protocol. These protocols may be custom protocols or may utilize commonly used protocols. The reconstruction module 1104 may function to reassemble the decomposed 3D models received from other systems over the network. The asset module 1102 may include one or more complete 3D models. (Exemplary Process for 3D Model Sharing)

[0143] Figure 12 illustrates an exemplary 3D model sharing process 1200 between a server and a client using the systems and methods described herein. Process 1200 may begin at step 1202 when a server system, such as server system 1118, opens a 3D model sharing host application.

[0144] In step 1204, the server may load one or more complete 3D model files into the 3D model sharing host application. The complete 3D model files may originate on the host system 1118 from different 3D modeling software downloaded on the host system 1118, such as AutoCad. In some embodiments, the 3D model files may be imported from an external source, such as an external hard drive, a remote computing system (e.g., 920 from FIG. 9 above), or a cloud storage device, or may originate from a 3D scanner or similar device. The 3D model files may be imported using ASSIMP or an equivalent plugin. The plugin may convert the 3D model files from one format to a different file format that is compatible with the 3D model sharing host application. Since the file formatting may be lost at the server during the decomposition process, as described in more detail below, the plugin may be utilized by only a host system, such as host system 1118.

[0145] In step 1206, the server system may decompose the 3D model into its constituent parts. The step of decomposing the 3D model can include the step of identifying the constituent parts within the 3D model. The step of decomposing the 3D model can be non-destructive in that it does not result in a change to the 3D model itself (e.g., loss of information within the 3D model) or a change to the way the 3D model is rendered for viewing. The step of decomposing the 3D model can include the step of applying data level changes to the 3D model and / or its constituent parts (e.g., applying a data formatting change, or compressing the constituent parts to facilitate sharing). The decomposition process may occur within the decomposition module 1110 as described above. A constituent part may be a subset of the data required to fully define the 3D model. The constituent part subset may be defined by an existing bundle of data comprising the 3D model, such as a vertex table, geographic data, material data, etc. In some embodiments, the constituent part subset may be defined in other ways, such as by a 3D model sharing application programmer, etc.

[0146] In step 1214, the client system may open a 3D model sharing client application. This may initiate a series of steps related to networking protocols, communication, and discovery.

[0147] In step 1216, the 3D model sharing client application may discover a 3D model sharing host over the network if the host exists. The network may be a LAN (e.g., UDP). In some embodiments, the local network is private. In some embodiments, cloud services are not used.

[0148] In step 1208, the server may receive a connection request from the client, and the server may accept the connection request.

[0149] In step 1210, the server may return a client information request to the client. The client information request may contain all lists within the library of the 3D model sharing host application. The client information request may alternatively be referred to as a table of asset holdings or content.

[0150] In step 1218, when the client compares with the 3D model sharing server application library, it may reply to the server using data that defines items missing from the 3D model sharing client application library.

[0151] In step 1212, the server may respond by sending to the client items that are missing from the 3D model sharing client application. In some embodiments, the client may have recently opened the 3D model sharing application and thus may start with zero items in its library. In this situation, the host may send the entire library to the client. In some embodiments, the library may be installed in memory and thus may be erased when the 3D model sharing application is closed. In some embodiments, the library may be stored on a disk such as the client system hard drive so that the 3D models can be accessed at a later date. In some embodiments, the 3D model data sent to the client is a data structure without a file format. At the end of this step, the 3D model sharing host library may match the 3D model sharing client library. In other words, both the host system and the client system may here contain the same components for the same set of 3D models.

[0152] In some embodiments, the client may update items that are legacy when compared to items stored within the server. For example, a client information request may include a version number associated with an individual asset within an asset inventory. One or more assets within the server's asset inventory may correspond to assets stored with the client. The client may identify items stored within the client asset library that have a version number less than the corresponding item within the server's asset inventory. The client may send data to the server identifying the item for which an update is requested, and the server may respond by sending the corresponding updated item to the client.

[0153] In step 1220, the client may reconstruct one or more 3D models from new items sent from the server. In some embodiments, the client may here have the same complete 3D model as the server.

[0154] In step 1222, a user of the client system may here view 3D models shared by the host system. (Exemplary 3D Model Sharing System Configuration)

[0155] FIG. 13 illustrates an exemplary 3D model sharing system configuration 1300 for sharing 3D assets using the systems and methods described herein.

[0156] The 3D model sharing system configuration 1300 may include a host system 1306 and a client system 1304 that are operably coupled by a communication link 1302 such as a network. The client system 1304 may be launched by opening a 3D model sharing client application and thus may operate within a runtime environment. The host system 1306 may be launched by opening a 3D model sharing host application and thus may operate within a runtime environment. FIG. 13 depicts one server system and one host system, which is not intended to be limiting in scope and is done for ease of illustration. In practice, the 3D model sharing system configuration 1300 may have only one host system 1306, more than one host system 1306, one client system 1304, more than one client system 1304, and / or one or more systems that function as both a host (e.g., transmitting 3D model data to other systems via communication 1302) and a client (e.g., receiving 3D model data from other systems via communication 1302). Further, in some embodiments, a single system may be capable of including both a host system and a client system. In some embodiments, the host system 1306 may be the host system 1118. In some embodiments, the client system 1304 may be the client system 1116. The host system 1306 may include one or more processors that are operably coupled to at least one display, e.g., a PC desktop computer attached to a computer monitor. In some embodiments, the host system 1306 may be the system 200, 930a, 930b, or 930c as described above. The client system 1304 may include one or more processors that are operably coupled to at least one display, e.g., a PC desktop computer attached to a computer monitor. In some embodiments, the client system 1304 may be the system 200, 930a, 930b, or 930c as described above.

[0157] The host system 1306 may include a server module 1316. The server module 1316 may manage functions and processes specific to a server, such as transmission and networking protocols. The server module 1316 may receive and process connection requests, such as in steps 1208, 1210, and / or 1212 of process 1200. In some embodiments, the server module 1316 may include a transmission module 1112.

[0158] The host system 1306 may include a host load module 1320. The load module may contain functions and processes that enable a 3D model sharing host application to import or load 3D assets into the 3D model sharing host application or onto the host system 1306. The load module 1320 may include an ASSIMP plugin. In some embodiments, step 1204 from process 1200 may occur within the load module 1320. In some embodiments, step 1001 from process 1000 may occur within the load module 1320. In some embodiments, the asset module 1108 may include the load module 1320.

[0159] The host system 1306 may include a host library manager module 1318. The library manager module 1318 may contain and manage data and processes corresponding to 3D assets already loaded in the 3D asset sharing host application. The library manager module 1318 may contain and manage data and processes corresponding to the library of the 3D asset sharing host application. This may include a decomposition process that maintains a library table of content, etc. In some embodiments, step 1206 from process 1200 may occur within the library manager module 1318. In some embodiments, step 1003 from process 1000 may occur within the library manager module 1318. In some embodiments, the library manager module 1318 may include a decomposition module 1110 from system 1100.

[0160] The host system 1306 may include a host world module 1322. The host world module 1322 may manage complete 3D assets, 3D virtual worlds, real-world meshing, etc., and the integration between them. In some embodiments, the host world module 1322 may be regarded as a scene graph, or a functional equivalent of a scene graph. In some embodiments, the world may be drawn from items obtained from a world library (i.e., fully reconstructed 3D models), and world objects can be created from them. In some embodiments, everything within the 3D virtual world can be represented as a world object. The world object may be part of a world "class". In some embodiments, the host world module 1322 contains the same data as the client world module 1312, by step 1222 of process 1200, or by step 1011 of process 1000 (i.e., after the 3D model has been locally shared and reconstructed at the client). The output from the host world module 1322 may be fed into the rendering path module 1324.

[0161] The host system 1306 may include a host rendering path module 1324. The rendering path module 1324 may receive data from the host world module 1322 as input, pass it through the host system 1306 rendering pipeline, and display one or more 3D models on the display of the host system 1306.

[0162] The client system 1304 may include a client module 1308. The client module 1304 may manage functions and processes specific to being a client, such as transmission and networking and / or server / client protocols. The client module 1308 may manage the server discovery process and other connection protocols. In some embodiments, this may include steps 1216 and / or 1218 in process 1200. In some embodiments, the client module 1308 may include a transmission module 1106.

[0163] The client system 1304 may include a client library manager module 1310. The library manager module 1310 may contain and manage data and processes corresponding to 3D asset components received from the 3D asset sharing host application. The library manager module 1310 may contain and manage data and processes corresponding to the library of the 3D asset sharing client application. This may include a reconstruction process. In some embodiments, step 1220 from process 1200 may occur within the library manager module 1310. In some embodiments, step 1009 from process 1000 may occur within the library manager module 1310. In some embodiments, the library manager module 1310 may include a reconstruction module 1104 from system 1100. In some embodiments, the client library contains the same data as the host library by step 1222 of process 1200 or by step 1011 of process 1000 (i.e., after the 3D model is locally shared and reconstructed at the client).

[0164] The client system 1304 may include a client world module 1312. The client world module 1312 may manage fully reconstructed 3D assets, 3D virtual worlds, real-world meshing, etc., and the integration between them. In some embodiments, the client world module 1312 may be regarded as a scene graph, or a functional equivalent of a scene graph. In some embodiments, the world is drawn from items obtained from a world library (i.e., fully reconstructed 3D models), and world objects are created from them. In some embodiments, everything within the 3D virtual world is represented as a world object. The world object may be part of a world "class". In some embodiments, the client world module 1312 can contain the same data as the host world module 1322, by step 1222 of process 1200, or by step 1011 of process 1000 (i.e., after the 3D model has been locally shared and reconstructed at the client). The output from the client world module 1312 may be fed into the rendering path module 1314.

[0165] The client system 1304 may include a client rendering path module 1314. The rendering path module 1314 may accept data from the client world module 1312 as input, pass it through the client system 1304 rendering pipeline, and may display one or more virtual 3D models on the display of the client system 1304.

[0166] The exemplary 3D model sharing system configuration 1300 can enable two or more devices to share a complete 3D model during runtime. This has the advantage of adding new assets to applications on a computing system faster than conventional systems where updates may have to wait until the application is closed. (Exemplary process for decomposing a 3D model)

[0167] Figure 14 illustrates an exemplary process 1400 for disassembling a complete 3D model using the systems and methods described herein. In some embodiments, the disassembly process 1400 may occur within the host library manager module 1318. In some embodiments, the disassembly process 1400 may occur within the disassembly module 1110. In some embodiments, the disassembly process 1400 may be step 1003 in process 1000. In some embodiments, the disassembly process 1400 may be step 1206 in process 1200.

[0168] Process 1400 may begin with a complete 3D model that can be loaded within a 3D model sharing host application. In step 1402, the 3D model sharing host application may copy one component of the 3D model and place the data representing the component into an array. Some exemplary components may be one or more vertex positions, vertex tables, geographic data, material data, textures, triangle index tables, etc. The component may be part of the data required to represent the complete 3D model.

[0169] In step 1404, the component may optionally be compressed. Standard compression techniques may be used.

[0170] In step 1406, the component may be packaged.

[0171] In step 1408, the package may be stored in one of the libraries of the 3D model sharing host application. The 3D model sharing host application may have one or more libraries used to store and organize the packages. In some embodiments, the library may comprise three libraries, namely a mesh library, a material library, and a texture library, as shown in FIG. 16. Alternative library configurations may be used. The package may be stored in the corresponding library (e.g., the texture may be stored in the texture library).

[0172] In step 1410, the process may check the 3D model and determine whether there are additional components that need to be added to the library. If there are additional components that have not been added to the library, the process can start over again from the beginning in step 1402 until all components have been added to the library. When all components have been added, the library may contain a complete record of the complete 3D model, as in step 1412.

[0173] In some embodiments, a 3D model can be decomposed into components that are predefined by a 3D model application. For example, a 3D model application (e.g., a computer-aided design program) can define a 3D model from the perspective of different data elements (e.g., meshes and mesh renderings), each of which can include one or more defined components (e.g., an array of triangles or a texture). In that case, the step of decomposing the 3D model can comprise applying operations (e.g., data formatting) to each component and storing it in memory. In some embodiments, the methods and systems described herein can be used to decompose a 3D model into one or more subgroups defined by a user. For example, with respect to a 3D model of a car, a user can specify a set of vertices / parts / structures as an "engine", a set of vertices / parts / structures as a "chassis", a set of vertices / parts / structures as a "drive train", and a set of vertices / parts / structures as "wheels". Each subgroup can then optionally be compressed and transmitted separately to a device. In some embodiments, a 3D model can be decomposed into subgroups defined by an algorithm. For example, an algorithm may be developed (e.g., manually, semi-automatically, and / or automatically, e.g., through machine learning) to identify a set of vertices / parts / structures as an "engine". The algorithm can first identify the entire 3D model (e.g., first identify the model as a car) before attempting to classify a set of vertices / parts / structures from a limited pool based on an initial identification. The algorithm can also attempt to directly classify a set of vertices / parts / structures. (Example of a complete 3D model)

[0174] FIG. 15 illustrates an exemplary complete 3D model 1500 that uses the systems and methods described herein. In some embodiments, the complete 3D model may be a complete set of data that describes a 3D renderable digital object. In some embodiments, the complete 3D model may be a set of data that represents a digital or virtual object that is viewable when added to a compatible rendering pipeline.

[0175] In 3D model component 1502, the 3D model can be assigned a name. The name may be assigned by the user, and / or may be the original file name used in 3D model creation software such as the file name given to the 3D model in AutoCad. The 3D model may have two main types of data, namely mesh data 1504 and mesh renderer data 1506. The mesh data may be further decomposed into subsets of data, or components such as component A 1508, component B 1510, component 1512, and component 1514. The material data 1516 may be further decomposed into subsets of data, or components such as material properties 1522, texture 1 1518, and texture 2 1522. The exact layout, categorization, and sub-grouping of the 3D model data can vary from that shown in FIG. 15. In some embodiments, any number of components, such as 2, 3, or 10 components, may exist under the mesh 1504. In some embodiments, instead of just one, multiple subsets of properties 1522 may exist under the material data 1516. In some embodiments, more or fewer texture components 1518, 1520 may exist than shown in FIG. 15. Any suitable number and categorization may be used to break down the complete 3D model into components.

[0176] In some embodiments, an exemplary complete 3D model 1500 can be decomposed using an exemplary decomposition process 1400. The complete 3D model 1500 can comprise one or more components (e.g., component A 1508, component B 1510, component C 1512, component D 1514, texture 1 1518, texture 2 1520, and / or material property 1522). One component of the complete 3D model 1500 (e.g., texture 1) can be copied in step 1402 of the decomposition process 1400. The component can optionally be compressed in step 1404 of the decomposition process 1400. The component can be packaged in step 1406 of the decomposition process 1400. The component can be stored in step 1408 of the decomposition process 1400 (e.g., within an exemplary set of libraries 1600). Steps 1402, 1404, 1406, and 1408 can be repeated for each component of the complete 3D model 1500 until all components of the complete 3D model 1500 are stored (e.g., within the set of libraries 1600). (Examples of Libraries in 3D Model Sharing Applications)

[0177] Figure 16 illustrates an exemplary set of libraries 1600 utilized by a 3D model sharing application to store components of a 3D model using the systems and methods described herein. The set of libraries 1600 may comprise one or more libraries for storing and organizing 3D model components. In some embodiments, there may be three libraries: a mesh library 1602, a material library 1604, and a texture library 1606. The mesh library 1602 may contain a set of components in the form of an array that describes a 3D model mesh. Exemplary arrays that may be stored within the mesh library are a vertex buffer array 1608, a normal buffer array 1610, a UV buffer array 1612, and / or a triangle array 1614. The mesh library may also optionally contain one or more string values such as a material "name" 1616, a bitmap texture 1 "name" 1618, and a bitmap texture 2 "name" 1620. These string values 1616, 1618, 1620 may reference other libraries such as the material library 1604 and the texture library 1606. In some embodiments, a complete 3D model may be represented when all of the data is pulled or referenced from the mesh library 1602. In some embodiments, a complete 3D model for a "name" may be represented when all of the data is pulled or referenced from all of the libraries 1602, 1604, 1606 that record or correspond to a single 3D model "name".

[0178] The material library 1604 may contain data representing material properties and property-related values for a 3D model. In some embodiments, the texture library 1606 may contain one or more textures 1622, 1624. In some embodiments, the texture library 1606 may contain one or more bitmaps. In some embodiments, the texture library 1606 may contain one or more UV maps, UV coordinates, or other UV space data.

[0179] All of the data contained in the library 1600 of 3D common applications may be combined so as to represent a complete renderable 3D model. (Exemplary process for reconstructing a 3D model)

[0180] FIG. 17 illustrates an exemplary process 1700 for reconstructing a complete 3D model from its components using the systems and methods described herein. In some embodiments, this process may be step 1220 in process 1200. In some embodiments, process 1700 may be step 1009 in process 1000. In some embodiments, process 1700 may occur within the reconstruction module 1104. In some embodiments, process 1700 may occur within the library manager module 1310.

[0181] Process 1700 may begin at step 1702 by creating an empty object, sometimes referred to as an empty game object. In some embodiments, the empty game object may comprise an empty volume within a space with a reference coordinate system. In some embodiments, the empty object is essentially a transformation existing within the space, i.e., a position and an orientation. In some embodiments, the empty object is an empty prism. In some embodiments, the empty object may correspond to 1502 in FIG. 15.

[0182] After the empty object is created at 1702, a new mesh 1704 and a new mesh renderer 1708 may be added to the empty game object. In some embodiments, the step of adding the new mesh 1704 may correspond to the mesh 1504 in FIG. 15, and the step of adding the new mesh renderer 1708 may correspond to the mesh renderer 1506 in FIG. 15.

[0183] Process 1700 may proceed to steps 1706 and 1710 where an empty mesh buffer may be filled with 3D model components corresponding to mesh data, and an empty mesh render buffer may be filled with 3D model components corresponding to mesh rendering data. In some embodiments, steps 1706 and / or 1710 may decompress compressed data. In some embodiments, the mesh data may correspond to components 1508, 1510, 1512, 1514 from FIG. 15. In some embodiments, the mesh rendering data may correspond to components 1518, 1520, 1522. The buffer may be filled with data from a library of 3D model applications corresponding to a single 3D model. In some embodiments, the empty mesh buffer may be filled with data from mesh library 1602 at 1706. In some embodiments, the empty mesh render buffer may be filled with data from material library 1604 and texture library 1606 at 1710. Process 1700 may complete when all data corresponding to a single 3D model (i.e., a single record within the library of 3D model sharing applications) has been added to the empty objects.

[0184] FIG. 18 illustrates an exemplary complete 3D model 1800 using the systems and methods described herein. In some embodiments, the complete 3D model 1800 may be the output of process 1700. In some embodiments, the complete 3D model 1800 may be a more detailed version of the exemplary complete 3D model 1500, where general configuration component A may be vertex buffer 1808, general configuration component B may be normal buffer 1810, general configuration component C may be UV buffer 1812, and general configuration component D may be triangle index 1814. In some embodiments, the triangle index 1814 may need to be filled last.

[0185] In some embodiments, the complete 3D model 1800 may be an input to process 1400 from which components are copied. In some embodiments, the complete 3D model 1800 may be stored as a world object within world modules 1322 and / or 1312. In some embodiments, the complete 3D model 1800 may be a 3D model loaded into the 3D model sharing server application at step 1204 of process 1200 and / or step 1001 of process 1000. In some embodiments, the complete 3D model 1800 may be output from step 1220 of process 1200 and / or step 1009 of process 1000. In some embodiments, the complete 3D model 1800 may be an asset stored within asset modules 1102 and / or 1108. The object "name" 1802 may match the original file name loaded into the 3D model sharing application, e.g., car.fbx, even if the 3D model is no longer in FBX file format. (Other Examples of 3D Model Sharing)

[0186] In some embodiments, the 3D model can originate from an MR device (e.g., the wearable system 200). The MR device can capture information about a physical object, generate a 3D model of the physical object, decompose the 3D model into components, and transmit the components to a client device, which can generate a 3D model based on the components. For example, the object recognition device 708 can receive input from an outward-facing imaging system 464, a lidar sensor, a depth sensor, an RGB camera, an infrared camera, a time-of-flight camera, and / or other sensors on the MR device. This input can be used in computer vision algorithms and / or machine learning training algorithms to generate a 3D model based on the physical object. The algorithm can generate the 3D model by first identifying the physical object and then generating the 3D model based on a library of 3D models associated with the physical object. In some embodiments, the 3D model can be generated from the physical object by assembling sub-groups based on pre-defined sub-groups within the library of 3D models. In some embodiments, the 3D model can be decomposed into sub-groups defined by the user. In some embodiments, the 3D model can be decomposed into sub-groups defined by the algorithm. For example, the algorithm may be developed (e.g., manually, semi-automatically, and / or automatically, e.g., through machine learning) to identify a set of vertices / parts / structures as sub-groups.

[0187] In some embodiments, the algorithm can also generate a 3D model directly from a physical object. For example, the algorithm can identify the surfaces and vertices of a physical object and generate a 3D model of the entire physical object based on the observed characteristics of the physical object. In some embodiments, the algorithm can identify the located surfaces and vertices of a physical object, generate a 3D model of the located portion, and / or define a subgroup of physical objects based on a library of subgroups. The algorithm can store data regarding the association of subgroups and form a complete 3D model based on the components.

[0188] The 3D model sharing systems and methods described in this disclosure can provide a specific solution to the technical problem of how to move a renderable 3D model from one computing system to a different computing system during runtime. This can present an improvement compared to existing techniques that only add new renderable 3D models to an application offline.

[0189] Although 3D models are discussed herein, it is also contemplated that the systems and methods described in this application can be applied to other digital assets besides 3D models. For example, 2D models can be decomposed, transferred, and reconstructed (e.g., into contours and textures) using the systems and methods described herein. In some embodiments, animations can be decomposed, transferred, and reconstructed (e.g., into a series of meshes that can be further decomposed as described herein) using the systems and methods described herein. In some embodiments, 3D models can be flattened into 2D models (e.g., for display on a 2D screen or for creating a 2D map). (Other Considerations)

[0190] The processes, methods, and algorithms described in this specification and / or depicted in the attached figures are each embodied in code modules that are executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, thereby being fully or partially automated. For example, a computing system can include a general-purpose computer (e.g., a server) or a dedicated computer, a dedicated circuit, etc., programmed with specific computer instructions. The code modules can be installed in a dynamic link library that can be compiled and linked into an executable program, or can be written in an interpreted type programming language. In some implementations, certain operations and methods can be implemented by circuits specific to a given function.

[0191] Furthermore, the functional implementations of the present disclosure are sufficiently mathematically, computationally, or technically complex that special-purpose hardware or one or more physical computing devices (utilizing appropriate specialized executable instructions) may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved or to provide the results substantially in real time. For example, an animation or video can include many frames, each frame can have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image processing tasks or applications in a commercially reasonable amount of time.

[0192] A code module or any type of data can be stored on any type of non-transitory computer-readable medium such as a physical computer storage device including a hard drive, solid state memory, random access memory (RAM), read only memory (ROM), optical disk, volatile or non-volatile storage device, combinations of the same, and / or equivalents. The methods and modules (or data) can also be transmitted as data signals generated on various computer-readable transmission media including wireless-based and wired / cabled-based media (e.g., as part of a carrier wave or other analog or digital propagated signal), and can take various forms (e.g., as part of a single or multiplexed analog signal or as multiple discrete digital packets or frames). The results of the disclosed process or process steps can be persistently or otherwise stored within any type of non-transitory tangible computer storage device or communicated via a computer-readable transmission medium.

[0193] Any process, block, state, step, or functionality in the flow diagrams described in and / or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in a process. The various processes, blocks, states, steps, or functionality can be combined, rearranged, added to, deleted from, modified from, or otherwise changed from the illustrative embodiments provided herein. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states can be performed in other sequences that are appropriate, e.g., sequentially, in parallel, or in some other manner. Tasks or events can be added to or removed from the disclosed illustrative embodiments. Further, the separation of the various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged in multiple computer products. Many implementation variations are possible.

[0194] The present process, method, and system may be implemented in a network (or distributed) computing environment. The network environment may include an enterprise-wide computer network, an intranet, a local area network (LAN), a wide area network (WAN), a personal area network (PAN), a cloud computing network, a cloud source computing network, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0195] The systems and methods of the present disclosure each have several innovative aspects, none of which individually participate in or are required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in the present disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Accordingly, the claims are not intended to be limited to the implementations shown herein, but should be accorded the broadest scope consistent with the present disclosure, the principles, and the novel features disclosed herein.

[0196] In the context of separate implementations, certain features described herein can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable secondary combination. Further, features may be described above as acting in a certain combination and may further be claimed as such, but one or more features from the claimed combination can in some cases be deleted from the combination, and the claimed combination can be directed to a secondary combination or a variation of a secondary combination. No single feature or group of features is necessary or essential to every embodiment.

[0197] In particular, conditional statements used herein such as “can,” “could,” “might,” “may,” “e.g.,” and equivalents, generally convey that one embodiment includes a certain feature, element, and / or step while another embodiment does not, unless specifically stated otherwise or understood otherwise within the context in which it is used. Thus, such conditional statements are not generally intended to imply that a feature, element, and / or step is required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps should be included or implemented in any particular embodiment, whether or not there is author input or prompting. The terms “comprising,” “including,” “having,” and equivalents are synonyms and are used inclusively in a non-limiting manner, excluding additional elements, features, acts, operations, etc. Also, the term “or” is used in its inclusive sense (and not in its exclusive sense), and thus, for example, when used to connect a list of elements, the term “or” means one, some, or all of the elements in the list. Additionally, the articles “a,” “an,” and “the” as used in this application and the appended claims should be construed to mean “one or more” or “at least one” unless otherwise specified.

[0198] As used herein, the phrase that refers to a list of items "at least one of" refers to any combination of those items, including a single element. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Connective phrases such as "at least one of X, Y, and Z" are generally understood in a context such that, unless specifically stated otherwise, they are used to convey that an item, term, etc. can be at least one of X, Y, or Z. Thus, such connective phrases are generally not intended to suggest that an embodiment requires that at least one of X, at least one of Y, and at least one of Z each be present.

[0199] Similarly, operations may be depicted in the drawings in a particular order, but it should be recognized that this is not necessary for achieving the desired result, such that the operations are performed in the particular order shown, or in a sequential order, or that all of the illustrated operations be performed. Additionally, the drawings may schematically depict one or more exemplary processes in the form of a flowchart. However, other operations not depicted can also be incorporated within the exemplary methods and processes schematically illustrated. For example, one or more additional operations can be performed before, after, simultaneously with, or between any of the illustrated operations. In addition, the operations can be rearranged or reordered in other implementations. In some situations, multitasking and parallel processing can be advantageous. Further, the separation of the various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims are performed in a different order and still achieve the desired result. (Exemplary Embodiment)

[0200] 1. A method comprising:

[0201] displaying, via a display of a head-mounted device, a first version of a virtual 3D model, the first version of the virtual 3D model comprising a first version of a component;

[0202] requesting, from a host device, data associated with a second version of the virtual 3D model, the second version of the virtual 3D model comprising a second version of the component;

[0203] determining, based on the second version of the component, whether the first version of the component requests an update;

[0204] upon determining that the first version of the component requests an update based on the second version of the component,

[0205] requesting, from the host device, data associated with the second version of the component;

[0206] displaying, via the display of the head-mounted device, the second version of the virtual 3D model;

[0207] upon determining that the first version of the component does not request an update based on the second version of the component,

[0208] refraining from requesting, from the host device, data associated with the second version of the component; A method as described above.

[0209] 2. The method according to embodiment 1, wherein the component comprises mesh data.

[0210] 3. The method according to Embodiment 1, wherein the component comprises texture data.

[0211] 4. The method according to Embodiment 1, wherein the host device is a server.

[0212] 5. The method according to Embodiment 1, wherein the head-mounted device is a first head-mounted device and the host device is a second head-mounted device.

[0213] 6. The method according to Embodiment 1, further comprising storing data associated with a second version of the component in a memory.

[0214] 7. The method according to Embodiment 1, further comprising deploying data associated with a second version of the component.

[0215] 8. A system comprising:

[0216] a head-mounted device comprising a display; and

[0217] one or more processors,

[0218] displaying, via the display, a first version of a virtual 3D model, the first version of the virtual 3D model comprising a first version of the component;

[0219] requesting, from a host device, data associated with a second version of the virtual 3D model, the second version of the virtual 3D model comprising a second version of the component;

[0220] determining, based on the second version of the component, whether the first version of the component requests an update;

[0221] In accordance with a determination that a first version of a component requires an update based on a second version of the component,

[0222] requesting data associated with the second version of the component from a host device;

[0223] displaying a second version of a virtual 3D model via a display;

[0224] In accordance with a determination that a first version of a component does not require an update based on a second version of the component,

[0225] refraining from requesting data associated with the second version of the component from a host device; one or more processors configured to perform a method comprising: A system comprising.

[0226] 9. The system according to embodiment 8, wherein the component comprises mesh data.

[0227] 10. The system according to embodiment 8, wherein the component comprises texture data.

[0228] 11. The system according to embodiment 8, wherein the host device is a server.

[0229] 12. The system according to embodiment 8, wherein the head-mounted device is a first head-mounted device and the host device is a second head-mounted device.

[0230] 13. The system according to embodiment 8, wherein the method further comprises storing data associated with the second version of the component in a memory.

[0231] 14. The system according to embodiment 8, wherein the method further comprises deploying data associated with the second version of the component.

[0232] A non-transitory computer-readable medium that, when executed by one or more processors, causes the one or more processors to

[0233] display a first version of a virtual 3D model via a display of a head-mounted device, the first version of the virtual 3D model comprising a first version of a component;

[0234] request data associated with a second version of the virtual 3D model from a host device, the second version of the virtual 3D model comprising a second version of the component;

[0235] determine whether the first version of the component requires an update based on the second version of the component;

[0236] based on the determination that the first version of the component requires an update based on the second version of the component,

[0237] request data associated with the second version of the component from the host device;

[0238] display the second version of the virtual 3D model via the display of the head-mounted device;

[0239] based on the determination that the first version of the component does not require an update based on the second version of the component,

[0240] refrain from requesting data associated with the second version of the component from the host device; store instructions for performing a method, the method including

[0241] 16. The component is the non-transitory computer-readable medium according to embodiment 15, comprising mesh data.

[0242] 17. The component is the non-transitory computer-readable medium according to embodiment 15, comprising texture data.

[0243] 18. The host device is a server, and the non-transitory computer-readable medium according to embodiment 15.

[0244] 19. The head-mounted device is a first head-mounted device, and the host device is a second head-mounted device, and the non-transitory computer-readable medium according to embodiment 15.

[0245] 20. The method further includes the step of storing data associated with a second version of the component in a memory, and the non-transitory computer-readable medium according to embodiment 15.

[0246] 21. The method further includes the step of deploying data associated with a second version of the component, and the non-transitory computer-readable medium according to embodiment 15.

[0247] 22. A method,

[0248] the step of accessing a virtual 3D model stored in a memory,

[0249] the step of determining one or more components of the virtual 3D model,

[0250] the step of storing one or more components in one or more arrays, wherein the one or more components are stored separately from the virtual 3D model,

[0251] the step of receiving a connection request from a head-mounted device,

[0252] the step of sending a list of available components to the head-mounted device,

[0253] Receiving component requirements from a head-mounted device;

[0254] Based on the component requirements, sending the requested components to the head-mounted device; A method comprising.

[0255] 23. The method according to embodiment 22, wherein the component comprises mesh data.

[0256] 24. The method according to embodiment 22, wherein the component comprises texture data.

[0257] 25. The method according to embodiment 22, further comprising compressing one or more components.

[0258] 26. The virtual 3D model is a first virtual 3D model, and the method according to embodiment 22 further comprises generating a copy of the virtual 3D model based on the requested components.

[0259] 27. The method according to embodiment 22 further comprises displaying the requested components via a display of the head-mounted device.

[0260] 28. The method further comprises

[0261] Creating a placeholder for a new virtual 3D model;

[0262] Updating the placeholder for the new virtual 3D model based on the requested components; The method according to embodiment 22 comprising.

[0263] 29. A system comprising

[0264] A head-mounted device;

[0265] One or more processors,

[0266] accessing a virtual three-dimensional model stored in memory;

[0267] determining one or more components of the virtual three-dimensional model;

[0268] storing one or more components in one or more arrays, the one or more components being stored separately from the virtual three-dimensional model;

[0269] receiving a connection request from a head-mounted device;

[0270] sending a list of available components to the head-mounted device;

[0271] receiving a component request from the head-mounted device;

[0272] sending the requested component to the head-mounted device based on the component request; one or more processors configured to execute a method comprising: A system comprising.

[0273] 30. The system according to embodiment 29, wherein the component comprises mesh data.

[0274] 31. The system according to embodiment 29, wherein the component comprises texture data.

[0275] 32. The system according to embodiment 29, wherein the method further comprises compressing one or more components.

[0276] 33. The virtual three-dimensional model is a first virtual three-dimensional model, and the system according to embodiment 29, wherein the method further comprises generating a copy of the virtual three-dimensional model based on the requested component.

[0277] 34. The method further includes, via a display of the head-mounted device, displaying the requested component, the system according to Embodiment 29.

[0278] 35. The method further

[0279] creating a placeholder for a new virtual 3D model, and

[0280] updating the placeholder for the new virtual 3D model based on the requested component, the system according to Embodiment 29. including, the system according to Embodiment 29.

[0281] 36. A non-transitory computer-readable medium that, when executed by one or more processors, causes the one or more processors to

[0282] access a virtual 3D model stored in a memory, and

[0283] determine one or more components of the virtual 3D model, and

[0284] store one or more components in one or more arrays, where the one or more components are stored separately from the virtual 3D model, the step of

[0285] receive a connection request from a head-mounted device, and

[0286] send a list of available components to the head-mounted device, and

[0287] receive a component request from the head-mounted device, and

[0288] send the requested component to the head-mounted device based on the component request, and A non-transitory computer-readable medium storing instructions for causing a method to be executed.

[0289] 37. The component is the non-transitory computer-readable medium according to embodiment 36, comprising mesh data.

[0290] 38. The component is the non-transitory computer-readable medium according to embodiment 36, comprising texture data.

[0291] 39. The method further includes, in the non-transitory computer-readable medium according to embodiment 36, a step of compressing one or more components.

[0292] 40. The virtual 3D model is a first virtual 3D model, and the method further includes, in the non-transitory computer-readable medium according to embodiment 36, a step of generating a copy of the virtual 3D model based on the requested components.

[0293] 41. The method further includes, in the non-transitory computer-readable medium according to embodiment 36, a step of displaying the requested components via a display of a head-mounted device.

[0294] 42. The method further includes

[0295] a step of creating a placeholder for a new virtual 3D model, and

[0296] a step of updating the placeholder for the new virtual 3D model based on the requested components, and in the non-transitory computer-readable medium according to embodiment 36.

[0297] 43. A system,

[0298] a host computing system, and

[0299] A client computing system comprising a head-mounted display system, comprising,

[0300] The host computing system accesses a virtual 3D model stored in memory,

[0301] decomposes the 3D model into one or more components,

[0302] and sends a list of the one or more components to the client computing system,

[0303] receives a component request from the client computing system,

[0304] and sends one or more components corresponding to the component request to the client computing system,

[0305] and is configured to execute a method comprising, one or more processors,

[0306] The client computing system receives a list of one or more components from the host computing system,

[0307] sends a component request to the host computing system,

[0308] receives one or more of the components corresponding to the component request from the host computing system,

[0309] and constructs a copy of the virtual 3D model from one or more of the components corresponding to the component request,

[0310] and is configured to execute a method comprising, one or more processors. System

[0311] 44. The component is the system according to Embodiment 43, comprising mesh data.

[0312] 45. The component is the system according to Embodiment 43, comprising texture data.

[0313] 46. The host computing system is a server, which is the system according to Embodiment 43.

[0314] 47. The host computing system comprises a head-mounted display system, which is the system according to Embodiment 43.

[0315] 48. The method further includes the step of storing one or more components in a memory, which is the system according to Embodiment 43.

[0316] 49. The method further includes the step of deploying one or more components, which is the system according to Embodiment 43.

[0317] 50. A method,

[0318] accessing a virtual 3D model stored in a memory;

[0319] decomposing the 3D model into one or more components;

[0320] sending a list of one or more components to a client computing system comprising a head-mounted display;

[0321] receiving a component request from the client computing system;

[0322] sending one or more components corresponding to the component request to the client computing system;

[0323] Receiving a list of one or more components from a host computing system;

[0324] Sending a component request to the host computing system;

[0325] Receiving one or more of the components corresponding to the component request from the host computing system;

[0326] Constructing a copy of a virtual 3D model from one or more of the components corresponding to the component request; A method comprising.

[0327] 51. The method according to embodiment 50, wherein the component comprises mesh data.

[0328] 52. The method according to embodiment 50, wherein the component comprises texture data.

[0329] 53. The method according to embodiment 50, wherein the host computing system is a server.

[0330] 54. The method according to embodiment 50, wherein the host computing system comprises a head-mounted display system.

[0331] 55. The method according to embodiment 50, further comprising storing one or more components in a memory.

[0332] 56. The method according to embodiment 50, further comprising deploying one or more components.

[0333] 57. A non-transitory computer-readable medium that, when executed by one or more processors, causes the one or more processors to

[0334] access a virtual 3D model stored in a memory;

[0335] The step of decomposing a three-dimensional model into one or more constituent parts,

[0336] The step of transmitting a list of one or more constituent parts to a client computing system equipped with a head-mounted display,

[0337] The step of receiving a component request from the client computing system,

[0338] The step of transmitting one or more constituent parts corresponding to the component request to the client computing system,

[0339] The step of receiving a list of one or more constituent parts from the host computing system,

[0340] The step of transmitting a component request to the host computing system,

[0341] The step of receiving one or more of the constituent parts corresponding to the component request from the host computing system,

[0342] The step of constructing a copy of the virtual three-dimensional model from one or more of the constituent parts corresponding to the component request, A non-transitory computer-readable medium storing one or more instructions for implementing a method, including the above.

[0343] 58. The constituent part is the non-transitory computer-readable medium according to embodiment 57, comprising mesh data.

[0344] 59. The constituent part is the non-transitory computer-readable medium according to embodiment 57, comprising texture data.

[0345] 60. The host computing system is a server, which is the non-transitory computer-readable medium according to embodiment 57.

[0346] 61. The host computing system is a non-transitory computer-readable medium according to Embodiment 57, comprising a head-mounted display system.

[0347] 62. The method further includes the step of storing one or more components in a memory, which is a non-transitory computer-readable medium according to Embodiment 57.

[0348] 63. The method further includes the step of deploying one or more components, which is a non-transitory computer-readable medium according to Embodiment 57.

[0349] 64. A system,

[0350] A host computing system, the host computing system

[0351] Accessing a first virtual 3D model via a host asset module;

[0352] Identifying one or more components via a decomposition module;

[0353] Copying one or more components to a library via a decomposition module;

[0354] Accessing one or more components via a host transmission module;

[0355] Transmitting one or more components to a client computing system via a host transmission module; A host computing system comprising one or more processors configured to execute a method including the above steps,

[0356] A client computing system, the client computing system comprising a head-mounted display system, the client computing system

[0357] receiving, via a client transmission module, one or more components from a host computing system;

[0358] creating, via a configuration module, an empty object;

[0359] adding, via the configuration module, one or more data types to the empty object;

[0360] adding, via the configuration module, one or more components to the empty object;

[0361] storing, via a client asset module, a copy of a virtual three-dimensional model; A client computing system comprising one or more processors configured to execute a method comprising: comprising

[0362] The host computing system and the client computing system are communicatively connected via a communication link. System.

[0363] 65. The system according to embodiment 64, wherein the component comprises mesh data.

[0364] 66. The system according to embodiment 64, wherein the component comprises texture data.

[0365] 67. The system according to embodiment 64, wherein the host computing system is a server.

[0366] 68. The host computing system is the system according to Embodiment 64, comprising a head-mounted display system.

[0367] 69. The method further includes the step of storing one or more components in a memory, of the system according to Embodiment 64.

[0368] 70. The method further includes the step of deploying one or more components, of the system according to Embodiment 64.

[0369] 71. A method,

[0370] which is a step in a host computing system, wherein the host computing system

[0371] accesses a first virtual 3D model via a host asset module;

[0372] identifies one or more components via a decomposition module;

[0373] copies one or more components to a library via a decomposition module;

[0374] accesses one or more components via a host transmission module;

[0375] transmits one or more components to a client computing system via a host transmission module; and is configured to execute a method, comprising steps of one or more processors.

[0376] which is a step in a client computing system, wherein the client computing system comprises a head-mounted display system, and the client computing system

[0377] Receiving, via a client transmission module, one or more components from a host computing system;

[0378] Creating, via a configuration module, an empty object;

[0379] Adding, via the configuration module, one or more data types to the empty object;

[0380] Adding, via the configuration module, one or more components to the empty object;

[0381] Storing, via a client asset module, a copy of a virtual 3D model; Steps, comprising one or more processors configured to execute a method; Including,

[0382] The host computing system and the client computing system are communicatively connected via a communication link; Method.

[0383] 72. The method according to embodiment 71, wherein the component comprises mesh data.

[0384] 73. The method according to embodiment 71, wherein the component comprises texture data.

[0385] 74. The method according to embodiment 71, wherein the host computing system is a server.

[0386] 75. The method according to embodiment 71, wherein the host computing system comprises a head-mounted display system.

[0387] 76. The method according to embodiment 71, further comprising storing one or more components in a memory.

[0388] The method further includes the step of deploying one or more components, the method according to embodiment 71.

[0389] 78. A non-transitory computer-readable medium that, when executed by one or more processors, causes the processor to

[0390] in a host computing system,

[0391] access a first virtual 3D model via a host asset module;

[0392] identify one or more components via a decomposition module;

[0393] copy one or more components to a library via a decomposition module;

[0394] access one or more components via a host transmission module;

[0395] send one or more components to a client computing system via a host transmission module;

[0396] a client computing system, the client computing system comprising a head-mounted display system, in the client computing system

[0397] receive one or more components from a host computing system via a client transmission module;

[0398] create an empty object via a configuration module;

[0399] add one or more data types to the empty object via a configuration module;

[0400] Adding one or more components to an empty object via a configuration module

[0401] Storing a copy of the virtual 3D model via a client asset module

[0402] Storing one or more instructions for implementing a method, the host computing system and the client computing system being communicatively connected via a communication link A non-transitory computer-readable medium

[0403] 79. The non-transitory computer-readable medium according to embodiment 78, wherein the component comprises mesh data

[0404] 80. The non-transitory computer-readable medium according to embodiment 78, wherein the component comprises texture data

[0405] 81. The non-transitory computer-readable medium according to embodiment 78, wherein the host computing system is a server

[0406] 82. The non-transitory computer-readable medium according to embodiment 78, wherein the host computing system comprises a head-mounted display system

[0407] 83. The non-transitory computer-readable medium according to embodiment 78, wherein the method further comprises storing one or more components in a memory

[0408] 84. The non-transitory computer-readable medium according to embodiment 78, wherein the method further comprises deploying one or more components

[0409] 85. A system

[0410] A host computing system

[0411] Accessing a first virtual 3D model via a load module;

[0412] Identifying one or more components via a host library manager module;

[0413] Copying one or more components to a library via a host library manager module;

[0414] Storing the first virtual 3D model via a host world module;

[0415] Rendering the first virtual 3D model via a host rendering pass module;

[0416] Accessing one or more components via a server module;

[0417] Transmitting one or more components to a client computing system via a server module; A host computing system comprising one or more processors configured to execute a method including:

[0418] A client computing system comprising a head-mounted display system, the client computing system:

[0419] Receiving one or more components from the host computing system via a client module;

[0420] Generating a copy of a virtual 3D model, wherein generating the virtual 3D model:

[0421] Steps of creating an empty object via a client library manager module,

[0422] including steps of adding one or more components to the empty object via a client library manager module, steps,

[0423] steps of storing a copy of a virtual 3D model via a client world module,

[0424] steps of rendering a copy of a virtual 3D model via a client rendering path module, A client computing system comprising one or more processors configured to execute a method including,

[0425] comprising, the host computing system and the client computing system being communicatively connected via a communication link, system.

[0426] 86. The system according to embodiment 85, wherein the component comprises mesh data.

[0427] 87. The system according to embodiment 85, wherein the component comprises texture data.

[0428] 88. The system according to embodiment 85, wherein the host computing system is a server.

[0429] 89. The system according to embodiment 85, wherein the host computing system comprises a head-mounted display system.

[0430] 90. The system according to embodiment 85, wherein the method further includes steps of storing one or more components in a memory.

[0431] 91. The method further includes the step of deploying one or more components, the system according to embodiment 85.

[0432] 92. A method comprising:

[0433] In a host computing system,

[0434] Accessing a first virtual 3D model via a load module;

[0435] Identifying one or more components via a host library manager module;

[0436] Copying one or more components to a library via a host library manager module;

[0437] Storing the first virtual 3D model via a host world module;

[0438] Rendering the first virtual 3D model via a host rendering path module;

[0439] Accessing one or more components via a server module;

[0440] Transmitting one or more components to a client computing system via a server module;

[0441] In a client computing system comprising a head-mounted display system,

[0442] Receiving one or more components from the host computing system via a client module;

[0443] A step of generating a copy of a virtual three-dimensional model, the step of generating a copy of the virtual three-dimensional model comprising:

[0444] creating an empty object via a client library manager module;

[0445] adding one or more components to the empty object via the client library manager module; step;

[0446] storing a copy of the virtual three-dimensional model via a client world module;

[0447] rendering a copy of the virtual three-dimensional model via a client rendering pass module;

[0448] including, wherein the host computing system and the client computing system are communicatively connected via a communication link; method.

[0449] 93. The method according to embodiment 92, wherein the component comprises mesh data.

[0450] 94. The method according to embodiment 92, wherein the component comprises texture data.

[0451] 95. The method according to embodiment 92, wherein the host computing system is a server.

[0452] 96. The method according to embodiment 92, wherein the host computing system comprises a head-mounted display system.

[0453] 97. The method according to embodiment 92, wherein the method further comprises storing one or more components in a memory.

[0454] 98. The method further includes the step of deploying one or more components, the method according to embodiment 92.

[0455] 99. A non-transitory computer-readable medium storing one or more instructions that, when executed by one or more processors, cause the processors to implement the method,

[0456] A host computing system, the host computing system comprising:

[0457] Accessing a first virtual 3D model via a load module;

[0458] Identifying one or more components via a host library manager module;

[0459] Copying one or more components to a library via a host library manager module;

[0460] Storing the first virtual 3D model via a host world module;

[0461] Rendering the first virtual 3D model via a host rendering pass module;

[0462] Accessing one or more components via a server module;

[0463] Transmitting one or more components to a client computing system via a server module; A host computing system comprising one or more processors configured to execute a method including:

[0464] A client computing system, the client computing system comprising a head-mounted display system, the client computing system comprising:

[0465] Receiving, via a client module, one or more components from a host computing system;

[0466] Generating a copy of a virtual 3D model, the generating of the copy of the virtual 3D model comprising:

[0467] Creating, via a client library manager module, an empty object;

[0468] Adding, via the client library manager module, one or more components to the empty object; steps;

[0469] Storing, via a client world module, the copy of the virtual 3D model;

[0470] Rendering, via a client rendering pass module, the copy of the virtual 3D model;

[0471] A client computer comprising one or more processors configured to perform a method comprising: comprising; The host computing system and the client computing system are communicatively connected via a communication link; A non-transitory computer-readable medium.

[0472] 100. The component is the non-transitory computer-readable medium according to embodiment 99, comprising mesh data.

[0473] 101. The component is the non-transitory computer-readable medium according to embodiment 99, comprising texture data.

[0474] 103. The host computing system is the non-transitory computer-readable medium according to Embodiment 99, which is a server.

[0475] 104. The host computing system is the non-transitory computer-readable medium according to Embodiment 99, which includes a head-mounted display system.

[0476] 105. The method further includes the step of storing one or more components in a memory, which is the non-transitory computer-readable medium according to Embodiment 99.

[0477] 106. The method further includes the step of deploying one or more components, which is the non-transitory computer-readable medium according to Embodiment 99.

[0478] 107. A system,

[0479] a non-transitory computer-readable memory that stores a 3D model including one or more components,

[0480] a host computing system configured to access the 3D model from the non-transitory computer-readable memory, and further

[0481] storing one of the one or more components of the 3D model in an array stored in the non-transitory computer-readable memory;

[0482] packaging the array in a package in a library of the host computing system; a host computing system including one or more processors configured to execute a method including the above, A system comprising the above.

[0483] 108. The component includes mesh data, which is the system according to Embodiment 107.

[0484] 109. The component is the system according to Embodiment 107, comprising texture data.

[0485] 110. The host computing system is the system according to Embodiment 107, which is a server.

[0486] 111. The host computing system is the system according to Embodiment 107, comprising a head-mounted display system.

[0487] 112. The method further includes the step of storing one or more components in a memory, which is the system according to Embodiment 107.

[0488] 113. The method further includes the step of deploying one or more components, which is the system according to Embodiment 107.

[0489] 114. A method,

[0490] storing, via a non-transitory computer-readable memory, a 3D model comprising one or more components;

[0491] accessing the 3D model from the non-transitory computer-readable memory;

[0492] storing one of the one or more components of the 3D model in an array stored in the non-transitory computer-readable memory;

[0493] packaging the array in a package within a library of the host computing system; and including.

[0494] 115. The component is the method according to Embodiment 114, comprising mesh data.

[0495] 116. The component is the method according to Embodiment 114, comprising texture data.

[0496] 117. The non-transitory computer-readable memory is the method according to Embodiment 114, which is part of a server.

[0497] 118. The non-transitory computer-readable memory is the method according to Embodiment 114, which is part of a head-mounted display system.

[0498] 119. The method further includes the step of storing one or more components in the memory, which is the method according to Embodiment 114.

[0499] 120. The method further includes the step of deploying one or more components, which is the method according to Embodiment 114.

[0500] 121. A non-transitory computer-readable medium that, when executed by one or more processors, causes the one or more processors to

[0501] store, via the non-transitory computer-readable memory, a 3D model comprising one or more components;

[0502] access the 3D model from the non-transitory computer-readable memory;

[0503] store one of the one or more components of the 3D model in an array stored in the non-transitory computer-readable memory;

[0504] package the array in a package within a library of a host computing system; and store one or more instructions that cause the implementation of a method including the above steps.

[0505] 122. The component comprises mesh data, which is the non-transitory computer-readable medium according to Embodiment 121.

[0506] 123. The component is the non-transitory computer-readable medium according to Embodiment 121, comprising texture data.

[0507] 124. The non-transitory computer-readable memory is the non-transitory computer-readable medium according to Embodiment 121, which is part of a server.

[0508] 125. The non-transitory computer-readable memory is the non-transitory computer-readable medium according to Embodiment 121, which is part of a head-mounted display system.

[0509] 126. The method further includes the step of storing one or more components in the memory, and is the non-transitory computer-readable medium according to Embodiment 121.

[0510] 127. The method further includes the step of deploying one or more components, and is the non-transitory computer-readable medium according to Embodiment 121.

[0511] Examples of components may be geographical data, material data, vertex tables, one or more textures, triangle indices, or any other data used to ultimately define a complete representation of a 3D model.

Claims

1. A system comprising one or more processors configured to perform a method, the method comprising receiving a virtual 3D model, decomposing the virtual 3D model into components, wherein the components comprise mesh data, texture data, or a combination thereof and are determined based on first data from one or more sensors of a head-mounted display system, transmitting to a client computing system in communication with a host computing system a library comprising the components, the library corresponding to the virtual 3D model, receiving a component request from the client computing system, transmitting to the client computing system one or more of the components corresponding to the component request, wherein the client computing system is configured to construct a view of the virtual 3D model based on the one or more of the components and further based on second data from the one or more sensors of the head-mounted display system.

2. The system of claim 1, further comprising the host computing system.

3. The system of claim 1, further comprising the head-mounted display system.

4. The system of claim 1, wherein the method further comprises storing in a memory one or more of the components.

5. The system of claim 1, wherein the method further comprises compressing one or more of the components.

6. The system of claim 1, wherein a physically integrated device comprises the system and further comprises the client computing system.

7. The system of claim 1, wherein the method further comprises receiving a connection request from the client computing system, granting the connection request from the client computing system.

8. A system comprising one or more processors configured to perform a method, the method comprising ​ ​ ​ Determining one or more parts to be included in a component requirement based on a comparison between a first library of components and a client library of components, wherein the components comprise mesh data, texture data, or a combination thereof, and are determined based on first data from one or more sensors of a head-mounted display system, sending the component requirement to a host computing system; receiving one or more components of the components corresponding to the component requirement from the host computing system; constructing a view of a virtual 3D model based on the one or more received components of the components and further based on second data from the one or more sensors of the head-mounted display system A system including.

9. The system according to claim 8, wherein the system comprises the head-mounted display system.

10. The system according to claim 8, wherein the method further includes storing the one or more components of the components in a memory.

11. The system according to claim 8, wherein the method further includes disassembling the one or more components of the components.

12. The system according to claim 8, wherein a physically integrated device comprises the system and further comprises the host computing system.

13. Determining the one or more parts to be included in the component requirement is further based on a version number associated with the one or more parts, according to the system of claim 8.

14. The method is sending a connection request to the host computing system; connecting to the host computing system according to receiving an indication of approval of the connection request from the host computing system The system according to claim 8, further including.

Images