Method and system for sharing three-dimensional models

CN113168737BActive Publication Date: 2024-11-22MAGIC LEAP INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980076776.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-09-24
Filing Date
2019-09-24
Publication Date
2024-11-22
Estimated Expiration
2039-09-24

Smart Images

  • Figure CN113168737B_ABST
    Figure CN113168737B_ABST
Patent Text Reader

Abstract

Examples of the present disclosure describe systems and methods for decomposing and sharing 3D models. In the example method, a first version of a virtual three-dimensional model is displayed via a display of a wearable head device. A request is made to a host device regarding data associated with a second version of the virtual three-dimensional model, wherein the second version of the virtual three-dimensional model includes a component. Determine whether the first version of the virtual three-dimensional model includes a component. Based on determining that the first version of the virtual three-dimensional model does not include a component, a request is made to the host device regarding data associated with the component. The second version of the virtual three-dimensional model is displayed via the display of the wearable head device. Based on determining that the first version of the virtual three-dimensional model includes a component, no request is made to the host device regarding data associated with the component.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 62 / 735,675, filed on September 24, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to sharing a three-dimensional model between two or more computing systems including a mixed reality system, an imaging system, and a visualization system. Background Art

[0004] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality," "augmented reality," and "mixed reality" experiences, in which digitally reproduced images are presented to a user in a manner such that they appear to be real or can be perceived as real. Virtual reality (VR) scenes generally involve the presentation of computer-generated virtual image information that has no transparency to other actual real visual input. Augmented reality (AR) scenes generally involve the presentation of virtual image information as an enhancement to the visualization of the real world around the user. Mixed reality (MR) is an augmented reality in which physical objects and virtual objects can coexist and interact in real time. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the invention

[0005] Examples of the present disclosure describe systems and methods for decomposing and sharing 3D models. In the example method, a first version of a virtual three-dimensional model is displayed via a display of a wearable head device. A request is made to a host device regarding data associated with a second version of the virtual three-dimensional model, wherein the second version of the virtual three-dimensional model includes constituent parts. Determine whether the first version of the virtual three-dimensional model includes constituent parts. Based on determining that the first version of the virtual three-dimensional model does not include constituent parts, a request is made to the host device regarding data associated with the constituent parts. The second version of the virtual three-dimensional model is displayed via a display of the wearable head device. Based on determining that the first version of the virtual three-dimensional model includes constituent parts, no request is made to the host device regarding data associated with the constituent parts. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims.

[0007] Figure 1 Illustration depicting a mixed reality scene with some virtual real objects and some physical objects viewed by a person.

[0008] Figure 2 An example of a wearable system is schematically shown.

[0009] Figure 3 Example components of a wearable system are schematically shown.

[0010] Figure 4 An example of a waveguide stack for a wearable device that outputs image information to a user is schematically shown.

[0011] Figure 5 is a process flow diagram of an example of a method for interacting with a virtual user interface.

[0012] Fig. 6A is a block diagram of another example of a wearable system that may include an avatar processing and rendering system.

[0013] Figure 6B Example components of an avatar processing and rendering system are shown.

[0014] Figure 7 is a block diagram of an example of a wearable system including various inputs into the wearable system.

[0015] Figure 8 is a process flow diagram of an example of a method for rendering virtual content with respect to an identified object.

[0016] Fig. 9 An overall system view depicting multiple wearable systems interacting with each other is schematically shown.

[0017] Fig.10 An example process for sharing 3D assets using the systems and methods described herein is shown.

[0018] Fig.11 An example 3D model sharing system configuration is shown for sharing 3D assets using the systems and methods described herein.

[0019] Fig.12 An example 3D model sharing process between a server and a client using the systems and methods described herein is shown.

[0020] Fig.13 An example 3D model sharing system configuration is shown for sharing 3D assets using the systems and methods described herein.

[0021] Fig.14 An example process of decomposing a full 3D model using the systems and methods described herein is shown.

[0022] Fig.15 An exemplary complete 3D model using the systems and methods described herein is shown.

[0023] Fig.16 A collection of libraries illustrating examples of storing components of 3D models for use in a 3D model sharing application using the systems and methods described herein.

[0024] Fig.17 An example process of reconstructing a complete 3D model from its constituent parts using the systems and methods described herein is shown.

[0025] Fig.18 An exemplary complete 3D model using the systems and methods described herein is shown.

[0026] Figures 19A-19C An example mixed reality environment is shown.

[0027] Reference numerals may be repeated throughout the drawings to indicate corresponding relationships between reference elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION

[0028] Mixed reality environment

[0029] Like all people, users of mixed reality systems exist in a real environment—that is, the three-dimensional portion of the "real world" that the user can perceive, as well as all of its contents. For example, the user perceives the real environment using one of the ordinary human senses—sight, hearing, touch, taste, smell—and interacts with the real environment by moving one's body around in the real environment. Positions in the real environment can be described as coordinates in a coordinate space; for example, coordinates can include latitude, longitude, and altitude relative to sea level; distance in three orthogonal dimensions from a reference point; or other suitable values. Similarly, a vector can describe a quantity having a direction and a magnitude in a coordinate space.

[0030] The computing device may maintain a representation of the virtual environment, for example, in a memory associated with the device. As used herein, a virtual environment is a computational representation of a three-dimensional space. The virtual environment may include representations of any object, action, signal, parameter, coordinate, vector, or other characteristics associated with the space. In some examples, the circuit (e.g., processor) of the computing device may maintain and update the state of the virtual environment; that is, the processor may determine the state of the virtual environment at a second time t1 based on data associated with the virtual environment and / or input provided by the user at a first time t0. For example, if an object in the virtual environment is located at a first coordinate at time t0 and has certain programmed physical parameters (e.g., mass, friction coefficient); and an input received from the user indicates that a force should be applied to the object in a direction vector; the processor may apply the laws of kinematics to determine the position of the object at time t1 using basic mechanics. The processor may use any suitable information known about the virtual environment and / or any suitable input (e.g., real-world parameters) to determine the state of the virtual environment at time t1. In maintaining and updating the state of the virtual environment, the processor may execute any suitable software, including software relating to the creation and deletion of virtual objects in the virtual environment; software (e.g., scripts) for defining the behavior of virtual objects or characters in the virtual environment; software for defining the behavior of signals (e.g., audio signals) in the virtual environment; software for creating and updating parameters associated with the virtual environment; software for generating audio signals in the virtual environment; software for processing input and output; software for implementing network operations; software for applying asset data (e.g., animation data that moves virtual objects over time); or many other possibilities.

[0031] An output device (such as a display or speaker) can present any or all aspects of the virtual environment to the user. For example, the virtual environment can include virtual objects that can be presented to the user (which can include representations of inanimate objects; people; animals; lights, etc.). The processor can determine a view of the virtual environment (e.g., corresponding to a "camera" having origin coordinates, view axes, and a frustum); and render a visual scene of the virtual environment corresponding to the view to the display. Any suitable rendering technology can be used for this purpose. In some examples, the visual scene can include only some virtual objects in the virtual environment and exclude certain other virtual objects. Similarly, the virtual environment can include audio aspects that can be presented to the user as one or more audio signals. For example, a virtual object in the virtual environment can generate sounds originating from the location coordinates of the object (e.g., a virtual character can speak or cause a sound effect); or the virtual environment can be associated with a music cue or ambient sound that may or may not be associated with a particular location. The processor can determine an audio signal corresponding to the "listener" coordinates—e.g., a synthesized audio signal corresponding to the sounds in the virtual environment, and which is mixed and processed to simulate the audio signal that the listener would hear at the listener coordinates—and present the audio signal to the user via one or more speakers.

[0032] Because the virtual environment exists only as a computing structure, the user cannot directly perceive the virtual environment using one's ordinary senses. Instead, the user can only perceive the virtual environment indirectly, as presented to the user, for example, through a display, speaker, tactile output device, etc. Similarly, the user cannot directly touch, manipulate, or otherwise interact with the virtual environment; but input data can be provided to the processor via an input device or sensor, which can use the device or sensor data to update the virtual environment. For example, a camera sensor can provide optical data that indicates that the user is trying to move an object in the virtual environment, and the processor can use this data to cause the object to respond accordingly in the virtual environment.

[0033] A mixed reality system can present to a user a mixed reality ("MR") environment that combines aspects of a real environment and a virtual environment, for example using a transmissive display and / or one or more speakers (which can, for example, be incorporated into a wearable head device). In some embodiments, the one or more speakers can be external to the head-mounted wearable unit. As used herein, an MR environment is a simultaneous representation of a real environment and a corresponding virtual environment. In some examples, the corresponding real and virtual environments share a single coordinate space; in some examples, the real coordinate space and one or more corresponding virtual coordinate spaces are related to each other by one or more transformation matrices (or other suitable representations). Thus, in some embodiments, a single coordinate (in some examples, together with a transformation matrix) can define a first position in the real environment and a second corresponding position in the virtual environment; or vice versa.

[0034] In an MR environment, a virtual object (e.g., in a virtual environment associated with the MR environment) may correspond to a real object (e.g., in a real environment associated with the MR environment). For example, if the real environment in the MR environment includes a real light pole (real object) at a position coordinate, the virtual environment in the MR environment may include a corresponding virtual light pole (virtual object) at the corresponding position coordinate. As used herein, a real object and its corresponding virtual object are combined to form a "mixed reality object". The virtual object does not need to be perfectly matched or aligned with the corresponding real object. In some examples, the virtual object may be a simplified version of the corresponding real object. For example, if the real environment includes a real light pole, the corresponding virtual object may include a cylinder having approximately the same height and radius as the real light pole (reflecting that the shape of the light pole may be approximately cylindrical). Simplifying virtual objects in this manner may allow for computational efficiency and may simplify calculations to be performed on such virtual objects. In addition, in some examples of the MR environment, not all real objects in the real environment may be associated with corresponding virtual objects. Similarly, in some examples of the MR environment, not all virtual objects in the virtual environment may be associated with corresponding real objects. That is, some virtual objects may only be in the virtual environment of the MR environment without any real-world counterparts. In some examples, not all real objects may be associated with corresponding real objects.

[0035] In some examples, a virtual object may have characteristics that are different (sometimes, completely different) from the characteristics of a corresponding real object. For example, when the real environment in the MR environment may include a green, two-armed cactus—a spiny inanimate object—the corresponding virtual object in the MR environment may have the characteristics of a green, two-armed virtual character with human facial features and a gruff demeanor. In this example, the virtual object is similar to its corresponding real object in some characteristics (color, number of arms); but is different from the real object in other characteristics (facial features, personality). In this way, the virtual object has the potential to represent the real object in a creative, abstract, exaggerated, or imaginative way; or to give behavior (e.g., a human personality) to otherwise inanimate the real object. In some examples, a virtual object may be a purely imaginary creation that has no real-world equivalent (e.g., a virtual monster in a virtual environment that may be in a position corresponding to an empty space in the real environment).

[0036] Compared with VR systems that present a virtual environment to users while blurring the real environment, mixed reality systems that present MR environments provide the advantage that the real environment remains perceptible when presenting the virtual environment. Therefore, users of mixed reality systems can use visual and audio cues associated with the real environment to experience and interact with the corresponding virtual environment. As an example, although users of VR systems may try to perceive or interact with virtual objects displayed in the virtual environment—because, as mentioned above, users cannot directly perceive or interact with the virtual environment—users of MR systems can find that they interact with virtual objects intuitively and naturally by looking at, hearing, and touching the corresponding real objects in his or her own real environment. This level of interaction can improve the user's sense of immersion, connection, and participation in the virtual environment. Similarly, by presenting the real environment and the virtual environment at the same time, mixed reality systems can reduce negative psychological feelings (e.g., cognitive impairment) and negative physical feelings (e.g., motion sickness) associated with VR systems. Mixed reality systems also provide many possibilities for applications that can enhance or change our experience of the real world.

[0037] Fig.19A An example real environment 1900 is shown in which a user 1910 uses a mixed reality system 1912. The mixed reality system 1912 may include a display (e.g., a transmissive display) and one or more speakers, as well as one or more sensors (e.g., a camera), such as described below. The real environment 1900 shown includes a rectangular room 1904A in which the user 1910 is standing; and real objects 1922A (lamp), 1924A (table), 1926A (couch), and 1928A (painting). The room 1904A also includes position coordinates 1906, which can be considered to be the origin of the real environment 1900. As shown in FIG. Fig.19AAs shown, an environment / world coordinate system 1908 (including an x-axis 1908X, a y-axis 1908Y, and a z-axis 1908Z) having its origin at point 1906 (world coordinates) can define a coordinate space for the real environment 1900. In some embodiments, the origin 1906 of the environment / world coordinate system 1908 can correspond to the location where the mixed reality system 1912 is powered on. In some embodiments, the origin 1906 of the environment / world coordinate system 1908 can be reset during operation. In some examples, the user 1910 can be considered a real object in the real environment 1900; similarly, the body parts of the user 1910 (e.g., hands, feet) can be considered real objects in the real environment 1900. In some examples, a user / listener / head coordinate system 1914 (including an x-axis 1914X, a y-axis 1914Y, and a z-axis 1914Z) having its origin at point 1915 (e.g., a user / listener / head coordinate) may define a coordinate space for a user / listener / head on which the mixed reality system 1912 is located. The origin 1915 of the user / listener / head coordinate system 1914 may be defined relative to one or more components of the mixed reality system 1912. For example, the origin 1915 of the user / listener / head coordinate system 1914 may be defined relative to a display of the mixed reality system 1912, such as during an initial calibration of the mixed reality system 1912. Matrices (which may include translation matrices and quaternion matrices or other rotation matrices) or other suitable representations may characterize the transformation between the user / listener / head coordinate system 1914 space and the environment / world coordinate system 1908 space. In some embodiments, the left ear coordinates 1916 and the right ear coordinates 1917 may be defined relative to the origin 1915 of the user / listener / head coordinate system 1914. A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix) or other suitable representation may characterize the transformation between the left ear coordinates 1916 and the right ear coordinates 1917 and the user / listener / head coordinate system 1914 space. The user / listener / head coordinate system 1914 may simplify the representation of a position relative to the user's head or relative to the head mounted device (e.g., relative to the environment / world coordinate system 1908). The transformation between the user coordinate system 1914 and the environment coordinate system 1908 may be determined and updated in real time using simultaneous localization and mapping (SLAM), visual odometry, or other techniques.

[0038] Fig.19BAn example virtual environment 1930 is shown that corresponds to the real environment 1900. The virtual environment 1930 shown includes a virtual rectangular room 1904B that corresponds to the real rectangular room 1904A; a virtual object 1922B that corresponds to the real object 1922A; a virtual object 1924B that corresponds to the real object 1924A; and a virtual object 1926B that corresponds to the real object 1926A. Metadata associated with the virtual objects 1922B, 1924B, 1926B may include information derived from the corresponding real objects 1922A, 1924A, 1926A. The virtual environment 1930 additionally includes a virtual monster 1932 that does not correspond to any real object in the real environment 1900. The real object 1928A in the real environment 1900 does not correspond to any virtual object in the virtual environment 1930. A persistent coordinate system 1933 (including an x-axis 1933X, a y-axis 1933Y, and a z-axis 1933Z) with its origin at point 1934 (persistent coordinates) can define a coordinate space for virtual content. The origin 1934 of the persistent coordinate system 1933 can be defined relative to one or more real objects (such as real object 1926A). A matrix (which can include a transformation matrix and a quaternion matrix or other rotation matrix) or other suitable representation can characterize the transformation between the persistent coordinate system 1933 space and the environment / world coordinate system 1908 space. In some embodiments, each of the virtual objects 1922B, 1924B, 1926B, and 1932 can have their own persistent coordinate point relative to the origin 1934 of the persistent coordinate system 1933. In some embodiments, there can be multiple persistent coordinate systems, and each of the virtual objects 1922B, 1924B, 1926B, and 1932 can have their own persistent coordinate point relative to one or more persistent coordinate systems.

[0039] about Fig.19A and 19B, the environment / world coordinate system 1908 defines a shared coordinate space for both the real environment 1900 and the virtual environment 1930. In the example shown, the coordinate space has an origin at point 1906. In addition, the coordinate space is defined by the same three orthogonal axes (1908X, 1908Y, 1902Z). Therefore, a first position in the real environment 1900 and a second corresponding position in the virtual environment 1930 can be described relative to the same coordinate space. This simplifies the identification and display of corresponding positions in the real and virtual environments because the same coordinates can be used to identify the two positions. However, in some examples, the corresponding real environment and virtual environment do not need to use a shared coordinate space. For example, in some examples (not shown), a matrix (which may include a transformation matrix and a quaternion matrix or other rotation matrix) or other suitable representation may characterize the transformation between the real environment coordinate space and the virtual environment coordinate space.

[0040] Fig.19C An example MR environment 1950 is shown that simultaneously presents aspects of a real environment 1900 and a virtual environment 1930 to a user 1910 via a mixed reality system 1912. In the example shown, the MR environment 1950 simultaneously presents real objects 1922A, 1924A, 1926A, and 1928A from the real environment 1900 (e.g., via a transmissive portion of a display of the mixed reality system 1912), and virtual objects 1922B, 1924B, 1926B, and 1932 from the virtual environment 1930 (e.g., via an active displayed portion of the display of the mixed reality system 1912) to the user 1910. As described above, the origin 1906 serves as the origin of a coordinate space corresponding to the MR environment 1950, and the coordinate system 1908 defines an x-axis, a y-axis, and a z-axis for the coordinate space.

[0041] In the example shown, the mixed reality object includes corresponding pairs of real objects and virtual objects (i.e., 1922A / 1922B, 1924A / 1924B, 1926A / 1926B) occupying corresponding positions in the coordinate space 1908. In some examples, both the real object and the virtual object can be visible to the user 1910 at the same time. This may be desirable in instances where, for example, the virtual object presents information designed to enhance the view of the corresponding real object (such as in a museum application where the virtual object presents a missing fragment of an ancient damaged sculpture). In some examples, the virtual object (1922B, 1924B, and / or 1926B) can be displayed (e.g., via pixelated occlusion using activation of a pixelated occlusion shutter) so as to occlude the corresponding real object (1922A, 1924A, and / or 1926A). This may be desirable in instances where, for example, the virtual object acts as a visual replacement for the corresponding real object (such as in an interactive storytelling application where an inanimate real object becomes a "living" character).

[0042] In some examples, real objects (e.g., 1922A, 1924A, 1926A) may be associated with virtual content or auxiliary data that may not necessarily constitute a virtual object. The virtual content or auxiliary data may facilitate the processing or handling of virtual objects in a mixed reality environment. For example, such virtual content may include a two-dimensional representation of a corresponding real object; a custom asset type associated with the corresponding real object; or statistics associated with the corresponding real object. This information may enable or facilitate computations involving real objects without incurring unnecessary computational overhead.

[0043] In some examples, the above-described presentation may also incorporate audio aspects. For example, in the MR environment 1950, the virtual monster 1932 may be associated with one or more audio signals, such as footstep sound effects generated when walking around the MR environment 1950. As further described below, the processor of the mixed reality system 1912 may calculate an audio signal corresponding to a mixed and processed synthesis of all such sounds in the MR environment 1950, and present the audio signal to the user 1910 via one or more speakers included in the mixed reality system 1912 and / or one or more external speakers.

[0044] Mixed reality object sharing

[0045] The 3D digital model of the object can be displayed as a virtual object to one or more real or digital people in an AR / VR / MR (hereinafter referred to as MR for simplicity) environment. For example, an automotive engineer may want to share a new car design with colleagues in their weekly team meeting. Traditionally, design sharing can be achieved by providing several different views of the design so that the viewer can imagine the 3D object. In this case, the automotive engineer can print out a perspective drawing to distribute to colleagues during the meeting. This can work for simple designs, but if the 3D object has a complex shape or design, it may be difficult for the viewer to piece together the 3D object in the viewer's brain. Alternatively or additionally, the traditional method of sharing a 3D design or model may involve creating a physical prototype. In this example, the automotive engineer will need to spend time and money to create a physical prototype. The physical prototype can make it easier to understand the designer's intentions, but may only show a basic version of the design (for example, the viewer may only see the outside of the design, but cannot open it to view the internal components).

[0046] Embodiments of the disclosed systems and methods may provide improved 3D model sharing between computing systems. Continuing with the example above, the present invention may enable an automotive engineer to create a new digital 3D model of a car and then quickly and efficiently share the model with colleagues during a meeting. The present invention may also enable quick and easy unplanned sharing. For example, a manager of an automotive engineer may wish to share a 3D design that the manager previously designed in order to collaborate with the automotive engineer. In some embodiments, the manager is able to access an old 3D car design and share it with everyone present at the meeting without requiring people to download the design or click a link to access it.

[0047] Typically, in gaming, for example, 3D models (optionally referred to as 3D assets or 3D content) are preloaded in a gaming application so that when a user launches the game, all of the 3D assets that the user will view are already on the device. When updates are needed, the application will add new content offline. For example, a game can apply a patch while the game is not running, and when the application is next opened, the new content is installed and ready to use. This system of sharing new 3D models with two or more computing systems is convenient for infrequent, planned updates, but is not practical when new 3D models are frequently or unexpectedly needed.

[0048] MR games or applications can be collaborative, where multiple people contribute to a common goal, such as a whole group of students in a school learning about the anatomy of a frog. If a teacher is trying to explain a concept and the students do not understand, it may be desirable for the teacher to show the students a virtual 3D model of the frog so that they can see what the teacher is describing. Traditional systems would require students to download the 3D model or exit and re-enter the application so that the teacher can push new frog model updates to the students. Additionally, such an approach may require a large amount of disk storage in order to accommodate all the 3D asset data. Alternative methods of accessing 3D models without using disk space may be desirable.

[0049] These problems can be solved by a method and system for streaming data on a network between a server and one or more clients. The server can access one or more 3D models and can have a downloaded and running 3D model sharing server application. The 3D model sharing application can divide the model into component data parts, compress, package the component data parts and send them to the client on the network. The client can receive the component parts, rebuild the 3D model and view the 3D model during the runtime of the 3D model sharing client application. In some embodiments, the runtime can be when the game or application loop is operating, when the screen is displaying the application content, and / or when the computing system is operating with the framework of the application. Any number of clients can be part of the network. Any number of 3D models can be sent to the client. Two or more clients can be one person with two or more devices (e.g., computing systems), or can be two or more people each having a device.

[0050] An advantage of the present application is that it enables 3D content sharing during runtime.

[0051] The term host may be used interchangeably with server.The terms 3D model, 3D asset, 3D content, and 3D model may be used interchangeably.

[0052] Example of 3D display for wearable systems

[0053] A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present a 2D or 3D virtual image to a user. The image can be a still image, a frame of a video, a combination of videos, etc. At least a portion of the wearable system can be implemented on a wearable device, which can present a VR, AR, or MR environment for interaction, alone or in combination. The wearable device can be used interchangeably as an AR device (ARD). In addition, for the purposes of this disclosure, the term "AR" and the term "MR" can be used interchangeably.

[0054] Figure 1An illustration depicting a mixed reality scene with some virtual real objects and some physical objects viewed by a person. Figure 1 , an MR scene 100 is depicted in which a user of MR technology sees a real-world park-like setting 110, which features people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of MR technology may also perceive that he "sees" a robotic statue 130 standing on the real-world platform 120, and an anthropomorphic, flying, cartoon-like avatar character 140 that appears to be a bumblebee, even though these elements do not exist in the real world.

[0055] In order for a 3D display to produce a realistic sense of depth, and more specifically, a simulated sense of surface depth, it may be desirable for each point in the display's field of view to generate an accommodation response that corresponds to its virtual depth. If the accommodation response to a displayed point does not correspond to the virtual depth of that point (as determined by the binocular depth cues of convergence and stereoscopic vision), the human eye may experience an accommodation conflict, resulting in imaging instability, detrimental eye fatigue, headaches, and almost complete loss of surface depth in the absence of accommodation information.

[0056] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. For each depth plane, the image may be different (e.g., providing a slightly different rendering of a scene or object) and may be focused separately by the viewer's eyes, thereby providing depth cues to the user based on the eye adaptation required to bring different image features of the scene located at different depth planes into focus, or based on observing different image features that are out of focus at different depth planes. As discussed elsewhere herein, such depth cues can provide reliable depth perception.

[0057] Figure 2 An example of a wearable system 200 that can be configured to provide an AR / VR / MR scene is shown. The wearable system 200 may also be referred to as an AR system 200. The wearable system 200 may include a display 220 and various mechanical and electronic modules and systems that support the functions of the display 220. The display 220 may be coupled to a frame 230, which may be worn by a user, wearer, or viewer 210. The display 220 may be placed in front of the eyes of the user 210. The display 220 may present AR / VR / MR content to the user. The display 220 may include a head-mounted display (HMD) worn on the user's head.

[0058] In some embodiments, speaker 240 may be coupled to frame 230 and placed adjacent to the user's ear canal (in some embodiments, another speaker not shown is placed adjacent to the user's other ear canal to provide stereo / plastic sound control). Display 220 may include an audio sensor (e.g., microphone) 232 for detecting an audio stream from the environment and capturing ambient sound. In some embodiments, one or more other audio sensors not shown may be placed to provide stereo reception. Stereo reception can be used to determine the location of the sound source. Wearable system 200 can perform sound or voice recognition on the audio stream.

[0059] The wearable system 200 may include an outward-facing imaging system 464 ( Figure 4 ), the outward-facing imaging system 464 observes the world around the user. The wearable system 200 may also include an inward-facing imaging system 462 ( Figure 4 ). The inward-facing imaging system can track the movement of one eye or the movement of both eyes. Inward-facing imaging system 462 can be attached to frame 230 and can be in electrical communication with processing module 260 or 270, which can process image information acquired by the inward-facing imaging system to determine eye movement or eye posture of user 210, such as pupil diameter or orientation of the eye. Inward-facing imaging system 462 can include one or more cameras. For example, at least one camera can be used to image each eye. The images acquired by the camera can be used to determine the pupil size or eye posture of each eye, respectively, thereby allowing image information to be presented to each eye to be dynamically adjusted for that eye.

[0060] As an example, wearable system 200 can acquire an image of the user's gesture using outward-facing imaging system 464 or inward-facing imaging system 462. The image can be a still image, a frame of a video, or a video.

[0061] The display 220 can be operably coupled 250 to a local data processing module 260, such as via a wired lead or a wireless connection, which can be mounted in a variety of configurations, such as fixedly attached to the frame 230, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, a belt-coupled configuration).

[0062] The local processing and data module 260 may include a hardware processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which may be used to assist in the processing, caching, and storage of data. The data may include data that is: a) captured from a sensor (which may, for example, be operably coupled to the frame 230 or otherwise attached to the user 210), such as an image capture device (e.g., a camera in an inward-facing imaging system or an outward-facing imaging system), an audio sensor (e.g., a microphone), an inertial measurement unit (IMU), an accelerometer, a compass, a global positioning system (GPS) unit, a radio, or a gyroscope; or b) acquired or processed using a remote processing module 270 or a remote data repository 280, and may be passed to the display 220 after such processing or acquisition. The local processing and data module 260 may be operably coupled to the remote processing module 270 or the remote data repository 280 via a communication link 262 or 264 (such as via a wired or wireless communication link) so that these remote modules can be used as resources for the local processing and data module 260. Furthermore, the remote processing module 280 and the remote data repository 280 may be operably coupled to each other.

[0063] In some embodiments, the remote processing module 270 may include one or more processors configured to analyze and process data or image information. In some embodiments, the remote data repository 280 may include a digital data storage facility that may be used via the Internet or other networking configurations configured as "cloud" resources. In some embodiments, all data may be stored and all calculations may be performed in the local processing and data module, allowing fully autonomous use from the remote module.

[0064] Example components of a wearable system

[0065] Figure 3 Example components of a wearable system are schematically shown. Figure 3 A wearable system 200 is shown, which may include a display 220 and a frame 230. An enlarged view 202 schematically illustrates various components of the wearable system 200. In some implementations, Figure 3 One or more of the components shown in can be part of display 220. The various components, alone or in combination, can collect various data (such as, for example, audio or visual data) associated with a user of wearable system 200 or the user's environment. It should be understood that other embodiments may have more or fewer components depending on the application for which the wearable system is used. Nevertheless, Figure 3 Provides a basic idea of ​​some of the various components and types of data that can be collected, analyzed, and stored by wearable systems.

[0066] Figure 3An example wearable system 200 is shown, which may include a display 220. The display 220 may include a display lens 226 that may be mounted to the user's head or to a housing or frame 230 corresponding to the frame 230. The display lens 226 may include one or more transparent mirrors or diffractive optical elements that are positioned by the housing 230 in front of the user's eyes 302, 304 and may be configured to direct projected light 338 into the eyes 302, 304 and assist in beam shaping while also allowing transmission of at least some light from the local environment. The wavefront of the projected light beam 338 may diverge to coincide with the desired focal length of the projected light. As shown, two wide field of view machine vision cameras 316 (also referred to as world cameras) may be coupled to the housing 230 to image the environment surrounding the user. These cameras 316 may be dual capture visible light / invisible (e.g., infrared) light cameras. The cameras 316 may be Figure 4 316. The images acquired by the world camera 316 may be processed by the gesture processor 336. For example, the gesture processor 336 may implement one or more object identifiers 708 (e.g., Figure 7 as shown) to recognize the gesture of a user or another person in the user's environment, or to recognize physical objects in the user's environment.

[0067] Continue to refer Figure 3 , showing a pair of light projector modules having display optics and lenses configured to direct light 338 into the eyes 302, 304. The depicted view also shows two miniature infrared cameras 324 paired with infrared lights (such as light emitting diodes "LEDs"), which can be configured to be able to track the user's eyes 302, 304 to support rendering and user input. The cameras 324 can be Figure 4 The wearable system 200 may further include a sensor assembly 339, which may include X, Y, and Z axis accelerometer functionality as well as a magnetic compass and X, Y, and Z axis gyroscope functionality, preferably providing data at a relatively high frequency (such as 200 Hz). The sensor assembly 339 may be a reference Figure 2 The depicted system 200 may also include a head pose processor 336, such as an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), or an ARM processor (advanced reduced instruction set machine), which may be configured to calculate real-time or near real-time user head pose based on the wide field of view image information output from the capture device 316. The head pose processor 336 may be a hardware processor and may be implemented as Figure 2 A portion of the local processing and data module 260 is shown in FIG.

[0068] The wearable system may also include one or more depth sensors 234. The depth sensor 234 may be configured to measure the distance between an object in the environment and the wearable device. The depth sensor 234 may include a laser scanner (e.g., a lidar), an ultrasonic depth sensor, or a depth sensing camera. In some implementations, if the camera 316 has depth sensing capabilities, the camera 316 may also be considered a depth sensor 234.

[0069] Also shown is a processor 332 which may be configured to perform digital or analog processing to derive a gesture from gyroscope, compass, or accelerometer data from sensor assembly 339. Processor 332 may be Figure 2 260. Figure 3 The wearable system 200 shown in FIG. 2 may also include a positioning system such as, for example, GPS 337 (Global Positioning System) to assist in posture and positioning analysis. In addition, GPS may further provide remote-based (e.g., cloud-based) information about the user's environment. This information may be used to identify objects or information in the user's environment.

[0070] The wearable system can combine the data acquired by GPS 337 with a remote computing system (such as, for example, remote processing module 270, another user's ARD, etc.) that can provide more information about the user's environment. As one example, the wearable system can determine the user's location based on the GPS data and retrieve a world map (e.g., by communicating with remote processing module 270) that includes virtual objects associated with the user's location. As another example, the wearable system 200 can use world camera 316 (which can be Figure 4 Based on the images acquired by world camera 316, wearable system 200 can detect objects in the environment (e.g., by using Figure 7 The wearable system may further use data acquired by GPS 337 to interpret features.

[0071] The wearable system 200 may also include a rendering engine 334 that may be configured to provide user-local rendering information to facilitate operation of the scanner and imaging into the user's eyes so that the user views the world. The rendering engine 334 may be implemented by a hardware processor (such as, for example, a central processing unit or a graphics processing unit). In some embodiments, the rendering engine is part of the local processing and data module 260. The rendering engine 334 may be communicatively coupled (e.g., via a wired or wireless link) to other components of the wearable system 200. For example, the rendering engine 334 may be coupled to the eye camera 324 via a communication link 274, and to the projection subsystem 318 via a communication link 272 (which may project light into the user's eyes 302, 304 via a scanning laser device in a manner similar to a retinal scanning display). The rendering engine 334 may also communicate with other processing units (such as, for example, a sensor gesture processor 332 and an image gesture processor 336) via links 276 and 294, respectively.

[0072] Camera 324 (e.g., a miniature infrared camera) may be used to track eye poses to support rendering and user input. Some example eye poses may include where the user is looking or the depth at which he or she is focusing (which may be estimated by the vergence of the eyes). GPS 337, gyroscope, compass, and accelerometer 339 may be used to provide rough or fast pose estimates. One or more of cameras 316 may acquire images and poses that may be used, along with data from associated cloud computing resources, to map the local environment and share the user's view with others.

[0073] Figure 3 The example components depicted in the figure are for illustration purposes only. For ease of illustration and description, multiple sensors and other functional modules are shown together. Some embodiments may include only one or a subset of these sensors or modules. In addition, the location of these components is not limited to Figure 3 . Some components may be mounted to or housed within other components (such as a belt-mounted component, a handheld component, or a helmet component). As an example, the image gesture processor 336, the sensor gesture processor 332, and the rendering engine 334 may be placed in a belt pack and configured to communicate with other components of the wearable system via wireless communications (such as ultra-wideband, Wi-Fi, Bluetooth, etc.) or via wired communications. The depicted housing 230 is preferably head-mountable and wearable by a user. However, some components of the wearable system 200 may be worn on other parts of the user's body. For example, the speaker 240 may be inserted into the user's ear to provide sound to the user.

[0074] With respect to the projection of light 338 into the user's eyes 302, 304, in some embodiments, the camera 324 can be used to measure the location where the center of the user's eyes tend geometrically, which generally coincides with the focal location or "focal depth" of the eyes. The three-dimensional surface of all points that the eyes tend toward can be called the "horopter". The focal length can take a certain number of depths, or can vary infinitely. Light projected from the vergence distance appears to be focused to the subject's eyes 302, 304, while light before or after the vergence distance is blurred.

[0075] The human visual system is complex, and providing a realistic sense of depth is challenging. Due to the combination of vergence and accommodation, a viewer of an object may perceive the object as three-dimensional. The vergence movement of the two eyes relative to each other (e.g., the rotational movement of the pupils toward or away from each other to converge the eyes' line of sight to fix on an object) is closely related to the focusing (or "accommodation") of the eye lenses. Under normal circumstances, the focus of the eye lenses is changed, or the eyes are accommodated, to automatically match the vergence changes to the same distance by changing the focus from one object to another object at another distance under a relationship known as the "accommodation-vergence reflex." Likewise, under normal circumstances, changes in vergence will trigger matching changes in accommodation. Display systems that provide a better match between accommodation and vergence can form more realistic and comfortable three-dimensional image simulations.

[0076] In addition, the human eye can correctly resolve spatially coherent light with a beam diameter less than about 0.7 mm, regardless of where the eye is focused. Therefore, in order to create the illusion of proper depth of focus, the vergence of the eye can be tracked with camera 324, and the rendering engine 334 and projection subsystem 318 can be used to render all objects at or near the focused vergence, as well as all other objects at varying degrees of resolution (e.g., using intentionally created blur). Preferably, the system 220 renders to the user at a frame rate of about 60 frames per second or higher. As described above, preferably, camera 324 can be used for eye tracking, and the software can be configured to pick up not only the vergence geometry but also the focus position cues for use as user input. Preferably, such a display system is configured with brightness and contrast suitable for use during the day or at night.

[0077] In some embodiments, the display system preferably has a latency of less than about 20 milliseconds for visual object alignment, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 arc minute, which, without being limited by theory, is believed to be about the limit of the human eye. The display system 220 can be integrated with a positioning system that can involve GPS elements, optical tracking, compasses, accelerometers, or other data sources to assist in position and posture determination; positioning information can be used to facilitate accurate rendering of the user's view of the relevant world (e.g., such information will help the glasses understand their position with respect to the real world).

[0078] In some embodiments, the wearable system 200 is configured to display one or more virtual images based on the adaptation of the user's eyes. In some embodiments, unlike existing 3D display methods that force the user to focus on where the image is projected, the wearable system can be configured to automatically change the focus of the projected virtual content to allow more comfortable viewing of one or more images presented to the user. For example, if the user's eyes have a current focus of 1m, the image can be projected to coincide with the user's focus. If the user moves the focus to 3m, the image can be projected to coincide with the new focus. Therefore, the wearable system 200 of some embodiments can allow the user's eyes to function in a more natural way rather than forcing the user to reach a predetermined focus.

[0079] Such a wearable system 200 may eliminate or reduce the occurrence of eyestrain, headaches, and other physiological symptoms typically observed with virtual reality devices. To achieve this, various embodiments of the wearable system 200 may be configured to project virtual images at varying focal lengths via one or more variable focus elements (VFEs). In one or more embodiments, 3D perception may be achieved via a multi-plane focus system that projects images at a fixed focal plane away from the user. Other embodiments may employ a variable plane focus, where the focal plane may be moved back and forth in the z direction to coincide with the user's current focus state.

[0080] In both the multi-plane focusing system and the variable plane focusing system, the wearable system 200 can use eye tracking to determine the convergence of the user's eyes, determine the user's current focus, and project the virtual image at the determined focus. In other embodiments, the wearable system 200 may include an optical modulator that projects a variable focus beam in a grating pattern variably onto the entire retina through a fiber scanner or other light source. Therefore, the ability of the display of the wearable system 200 to project images with varying focal lengths not only alleviates the user's focus adjustment when viewing 3D objects, but can also be used to compensate for the user's eye abnormalities. In some other embodiments, the spatial light modulator can project the image to the user through various optical components. For example, as further described below, the spatial light modulator can project the image onto one or more waveguides and then transmit the image to the user.

[0081] Waveguide stack components

[0082] Figure 4 An example of a waveguide stack for outputting image information to a user is shown. The wearable system 400 may include a stack of waveguides, or a stacked waveguide assembly 480, which may be used to provide a three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, the wearable system 400 may correspond to Figure 2 Wearable system 200, Figure 4 Some parts of the wearable system 200 are schematically shown in more detail. For example, in some embodiments, the waveguide assembly 480 can be integrated into Figure 2 in the display 220 .

[0083] Continue to refer Figure 4 , the waveguide assembly 480 may also include a plurality of features 458, 456, 454, 452 between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. Instead, they may simply be spacers (e.g., a cover layer or a structure for forming an air gap).

[0084] The waveguides 432b, 434b, 436b, 438b, 440b or the plurality of lenses 458, 456, 454, 452 may be configured to send image information to the eye at various levels of wavefront curvature or light divergence. Each waveguide stage may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. An image injection device 420, 422, 424, 426, 428 may be used to inject image information into the waveguides 440b, 438b, 436b, 434b, 432b, each of which may be configured to distribute incident light across each respective waveguide for output toward the eye 410. Light may exit the output surface of the image injection device 420, 422, 424, 426, 428 and may be injected into the corresponding input edge of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam (e.g., a collimated beam) can be injected into each waveguide to output an entire field of sample exit pupil beams that can be directed toward the eye 410 at a specific angle (and divergence) corresponding to a depth plane associated with a particular waveguide.

[0085] In some embodiments, image injection devices 420, 422, 424, 426, 428 are separate displays that each generate image information for injection into corresponding waveguides 440b, 438b, 436b, 434b, 432b, respectively. In some other embodiments, image injection devices 420, 422, 424, 426, 428 may be outputs of a single multiplexed display that may transmit image information to each of image injection devices 420, 422, 424, 426, 428, for example, via one or more light guides (such as optical cables).

[0086] Controller 460 may control the operation of displays and image injection devices 420, 422, 424, 426, 428. Controller 460 may include programming (e.g., instructions in a non-transitory computer readable medium) to adjust the timing and provide image information to waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, controller 460 may be a single integrated device, or a distributed system connected via wired or wireless communication channels. In some embodiments, controller 460 may be a processing module 260 or 270 (in Figure 2 ).

[0087] The waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each respective waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have another shape (e.g., curved) with a major top surface and a bottom surface and an edge extending between those major top and bottom surfaces. In the illustrated configuration, the waveguides 440b, 438b, 436b, 434b, 432b may each include an optical element 440a, 438a, 436a, 434a, 432a that may be configured to couple light out of the waveguide by diffracting or otherwise redirecting light propagating within each respective waveguide. The coupled out light may be output by the waveguide at a location where the light propagating in the waveguide strikes a light redirecting element (such as a diffraction grating). The optical elements (440a, 438a, 436a, 434a, 432a) may be, for example, reflective or diffractive optical features. Although shown disposed on the bottom major surface of the waveguides 440b, 438b, 436b, 434b, 432b for ease of description and clarity, in some embodiments, the optical elements 440a, 438a, 436a, 434a, 432a may be disposed on the top major surface or the bottom major surface, or may be disposed directly in the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the optical elements 440a, 438a, 436a, 434a, 432a may be formed as a layer of material that is attached to a transparent substrate to form the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be a single piece of material, and the optical elements 440a, 438a, 436a, 434a, 432a may be formed on a surface of or within the piece of material.

[0088] Continue to refer Figure 4As discussed herein, in some embodiments, each waveguide 440b, 438b, 436b, 434b, 432b can be configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye can be configured to transmit collimated light injected into such waveguide 432b to the eye 410. The collimated light can represent an optical infinity focal plane. The next waveguide up 434b can be configured to emit the collimated light before it passes through the first lens 452 (e.g., a negative lens) to the eye 410. The first lens 452 can be configured to produce a slightly convex wavefront curvature so that the eye / brain interprets the light from the next waveguide up 434b as coming from the first focal plane closer to the eye 410 from optical infinity. Similarly, the third upward waveguide 436b can pass its output light through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined optical power of the first lens 452 and the second lens 454 can be configured to produce another incremental wavefront curvature so that the eye / brain interprets light from the third waveguide 436b as coming from a second focal plane that is closer to the person from optical infinity inward than light from the next waveguide up 434b.

[0089] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) may be similarly configured, with the highest waveguide 440b in the stack sending its output through all lenses between it and the eye for the purpose of representing the aggregated optical power relative to the person's closest focal plane. To compensate for the stack of lenses 458, 456, 454, 452 when viewing / interpreting light from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be provided at the top of the stack to compensate for the total power of the underlying lens stack 458, 456, 454, 452. Such a configuration may provide as many perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optics of the waveguide and the focusing aspects of the lens may be static (e.g., not dynamic or electrically activated). In some alternative embodiments, one or both may be dynamic using electrically activated features.

[0090] Continue to refer Figure 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be configured to both redirect light out of its corresponding waveguide and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes can have different configurations of light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume or surface features that can be configured to output light at specific angles. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be volume holograms, surface holograms, and / or diffraction gratings.

[0091] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be diffractive features, or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOE has a relatively low diffraction efficiency, so that only a portion of the light of the beam is deflected away toward the eye 410 at each intersection of the DOE, while the rest of the light continues to move through the waveguide via total internal reflection. The light carrying the image information may thus be split into multiple related exit beams that exit the waveguide at multiple locations, and the result is a fairly uniform pattern of exit emissions toward the eye 304 for that particular collimated beam that bounces around within the waveguide.

[0092] In some embodiments, one or more DOEs can be switched between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE can include a polymer dispersed liquid crystal layer, wherein the droplets can include a diffraction pattern in a host medium, and the refractive index of the droplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the droplets can be switched to a refractive index that does not match the refractive index of the host medium (in which case the pattern actively diffracts incident light).

[0093] In some embodiments, the number and distribution of depth planes or depth of field can be dynamically changed based on the pupil size or orientation of the viewer's eye. The depth of field may be inversely proportional to the viewer's pupil size. As a result, as the size of the pupil of the viewer's eye decreases, the depth of field can increase, so that a plane that is not discernible because the position of the plane is beyond the focal depth of the eye may become discernible and appear more and more focused as the pupil size decreases and the depth of field increases accordingly. Similarly, the number of spaced-apart depth planes used to present different images to the viewer can be reduced as the pupil size decreases. For example, a viewer may not be able to clearly perceive details of both the first depth plane and the second depth plane at one pupil size without adjusting the eye's accommodation from one depth plane to another depth plane. However, both depth planes can be fully focused on the user at another pupil size at the same time without changing the accommodation.

[0094] In some embodiments, the display system may change the number of waveguides receiving image information based on a determination of pupil size or orientation, or based on receiving an electrical signal indicative of a particular pupil size or orientation. For example, if the user's eye cannot distinguish between two depth planes associated with two waveguides, the controller 460 (which may be an embodiment of the local processing and data module 260) may be configured or programmed to stop providing image information to one of the waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE of a waveguide is switchable between on and off states, the DOE may be switched to the off state when the waveguide does receive image information.

[0095] In some embodiments, it may be desirable to have the exit beam satisfy the condition of having a diameter less than the diameter of the viewer's eye. However, satisfying this condition may be a challenge given the variability of the viewer's pupil size. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the viewer's pupil size. For example, as the pupil size decreases, the size of the exit beam may also decrease. In some embodiments, a variable aperture may be used to vary the size of the exit beam.

[0096] The wearable system 400 may include an outward-facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. The portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes referred to as a FOV camera. The FOV of the world camera may or may not be the same as the FOV of the viewer 210, which covers the portion of the world 470 perceived by the viewer 210 at a given time. For example, in some cases, the FOV of the world camera may be larger than the viewer 210 of the wearable system 400. The entire area available for viewing or imaging by the viewer may be referred to as a field of interest (FOR). The FOR may include a solid angle of 4π radians around the wearable system 400, because the wearer can move his body, head, or eyes to perceive substantially any direction in space. In other contexts, the wearer's movements may be more limited, and thus the wearer's FOR may produce a smaller solid angle. Images obtained from outward-facing imaging system 464 may be used to track gestures made by a user (eg, hand or finger gestures), detect objects in world 470 in front of the user, and the like.

[0097] The wearable system 400 may include an audio sensor 232, such as a microphone, to capture ambient sounds. As described above, in some embodiments, one or more other audio sensors may be placed to provide stereo reception for determining the location of a speech source. As another example, the audio sensor 232 may include a directional microphone that may also provide such useful directional information about where the audio source is located. The wearable system 400 may use information from both the outward-facing imaging system 464 and the audio sensor 230 when locating the speech source, or determine active speakers at a particular moment, etc. For example, the wearable system 400 may use speech recognition alone or in combination with a reflected image of the speaker (e.g., as seen in a mirror) to determine the identity of the speaker. As another example, the wearable system 400 may determine the location of the speaker in the environment based on the sound obtained from the directional microphone. The wearable system 400 may employ a speech recognition algorithm to parse the sound from the speaker's location to determine the content of the speech, and use speech recognition technology to determine the identity of the speaker (e.g., name or other demographic information).

[0098] The wearable system 400 may also include an inward-facing imaging system 466 (e.g., a digital camera) that observes the user's movements, such as eye movements and facial movements. The inward-facing imaging system 466 may be used to capture images of the eye 410 to determine the size and / or orientation of the pupil of the eye 304. The inward-facing imaging system 466 may be used to obtain images for determining the direction in which the user is looking (e.g., eye posture) or biometric identification of the user (e.g., via iris recognition). In some embodiments, at least one camera may be used for each eye to independently determine the pupil size or eye posture of each eye, thereby allowing image information to be presented to each eye to be dynamically adjusted for that eye. In some other embodiments, only the pupil diameter or orientation of a single eye 410 may be determined (e.g., using only a single camera per pair of eyes), and it may be assumed to be similar for both eyes of the user. The images obtained by the inward-facing imaging system 466 may be analyzed to determine the eye posture or mood of the user, which the wearable system 400 may use to determine which audio or visual content should be presented to the user. Wearable system 400 may also use sensors such as an IMU, accelerometer, gyroscope, etc. to determine head pose (eg, head position or head orientation).

[0099] The wearable system 400 may include a user input device 466 through which a user may input commands to the controller 460 to interact with the wearable system 400. For example, the user input device 466 may include a touchpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a stick, a haptic device, a totem (e.g., acting as a virtual user input device), etc. The multi-DOF controller may sense user input with some or all possible translations (e.g., left / right, forward / backward, or up / down) or rotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOF controller that supports translational motion may be referred to as 3DOF, while a multi-DOF controller that supports translation and rotation may be referred to as 6DOF. In some cases, a user may use a finger (e.g., a thumb) to press or slide on a touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 can be held by a user's hand during use of the wearable system 400. The user input device 466 can communicate with the wearable system 400 by wire or wirelessly.

[0100] Other components of wearable systems

[0101] In many implementations, in addition to or in lieu of the components of the wearable system described above, the wearable system may include other components. The wearable system may, for example, include one or more haptic devices or components. The haptic device or haptic component may be operable to provide a sense of touch to the user. For example, when touching virtual content (e.g., a virtual object, a virtual tool, other virtual constructions), the haptic device or component may provide a sense of touch of pressure or texture. The sense of touch may replicate the feel of a physical object represented by the virtual object, or may replicate the feel of an imaginary object or character (e.g., a dragon) represented by the virtual content. In some implementations, the haptic device or component may be worn by the user (e.g., a glove wearable by the user). In some implementations, the haptic device or component may be held by the user.

[0102] The wearable system may, for example, include one or more physical objects that can be manipulated by a user to allow input to the wearable system or to interact with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, a piece of metal or plastic, a wall, a table surface. In some implementations, the totem may not actually have any physical input structure (e.g., a key, a trigger, a joystick, a trackball, a rocker switch). Instead, the totem may simply provide a physical surface, and the wearable system may render a user interface so that the user appears to be on one or more surfaces of the totem. For example, the wearable system may render images of a computer keyboard and a trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and a virtual trackpad to appear on the surface of a thin rectangular plate of aluminum used as a totem. The rectangular plate itself may not have any physical keys, trackpads, or sensors. However, the wearable system may detect the user's manipulation or interaction or touch of the rectangular plate as a selection or input via a virtual keyboard or a virtual trackpad. User input device 466 ( Figure 4 ) may be an embodiment of a totem, which may include a touchpad, a touch pad, a trigger, a joystick, a trackball, a rocker or a virtual switch, a mouse, a keyboard, a multi-degree-of-freedom controller, or another physical input device. A user may use a totem alone or in combination with gestures to interact with a wearable system or other users.

[0103] Example process of user interaction with wearable system

[0104] Figure 5 is a process flow diagram of an example of a method 500 for interacting with a virtual user interface. The method 500 may be performed by a wearable system as described herein. An embodiment of the method 500 may be used by a wearable system to detect a person or document in the FOV of the wearable system.

[0105] At box 510, the wearable system may identify a specific UI. The type of UI may be predetermined by the user. The wearable system may identify that a specific UI needs to be populated based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). The UI may be specific to a security scenario in which the wearer of the system is observing a user presenting a document to the wearer (e.g., at a travel checkpoint). At box 520, the wearable system may generate data for a virtual UI. For example, data associated with the boundaries, overall structure, shape, etc. of the UI may be generated. In addition, the wearable system may determine the map coordinates of the user's physical location so that the wearable system may display a UI related to the user's physical location. For example, if the UI is centered on the body, the wearable system may determine the coordinates of the user's body posture, head posture, or eye posture so that a ring UI may be displayed around the user, or a flat UI may be displayed on a wall or in front of the user. In the security context described herein, the UI may be displayed as if the UI surrounds a traveler who is presenting a document to the wearer of the system, so that the wearer may easily view the UI while viewing the traveler and the traveler's documents. If the UI is hand-centered, the map coordinates of the user's hand can be determined. These map points can be derived from data received by the FOV camera, sensory input, or any other type of collected data.

[0106] At box 530, the wearable system may send data from the cloud to the display, or may send data from a local database to the display component. At box 540, a UI is displayed to the user based on the data sent. For example, a light field display may project a virtual UI into one or both eyes of the user. Once the virtual UI has been created, at box 550, the wearable system may simply wait for a command from the user to generate more virtual content on the virtual UI. For example, the UI may be a body-centered ring around the user's body or the body of a person in the user's environment (e.g., a traveler). The wearable system may then wait for a command (gesture, movement of the head or eyes, voice command, input from a user input device, etc.), and if the command is recognized (box 560), the virtual content associated with the command may be displayed to the user (box 570).

[0107] Example of avatar rendering in mixed reality

[0108] The wearable system can employ various mapping-related techniques to achieve high depth of field in the rendered light field. When mapping out a virtual world, it is advantageous to know all the features and points in the real world to accurately depict virtual objects relative to the real world. To this end, the FOV image captured from the wearable system user can be added to the world model by including new pictures that convey information about various points and features of the real world. For example, the wearable system can collect a set of map points (e.g., 2D points or 3D points) and find new map points to render a more accurate version of the world model. The world model of the first user can be transmitted to the second user (e.g., via a network such as a cloud network) so that the second user can experience the world around the first user.

[0109] Fig. 6A is a block diagram of another example of a wearable system that may include an avatar processing and rendering system 690 in a mixed reality environment. The wearable system 600 may be Figure 2 In this example, the wearable system 600 may include a map 620, which may include a map database 710 ( Figure 7 ) in the wearable system. The map may reside partially locally on the wearable system and may reside partially in a networked storage location (e.g., in a cloud system) accessible via a wired or wireless network. Posture process 610 may be executed on a wearable computing architecture (e.g., processing module 260 or controller 460) and utilize data from map 620 to determine the location and orientation of the wearable computing hardware or user. As the user is experiencing the system and operating within the world, posture data may be calculated based on data collected on the fly. The data may include images, data from sensors (such as an inertial measurement unit, which typically includes accelerometer and gyroscope components), and surface information related to objects in a real or virtual environment.

[0110] Sparse point representations can be the output of a simultaneous localization and mapping (e.g., SLAM or vSLAM, which refers to configurations where the input is only images / vision) process. The system can be configured to find out not only where the individual components are in the world, but also what the world is made of. Poses can be a building block to achieve many goals, including populating maps and using data from maps.

[0111] In one embodiment, the sparse point locations themselves may not be completely sufficient, and further information may be required to produce a multi-focal AR, VR, or MR experience. A dense representation, often referred to as depth map information, can be used to at least partially fill the gap. Such information can be calculated from a process called "stereo" 640, where depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns created using active projectors), images acquired from image cameras, or hand gestures / totems 650 can be used as inputs to the stereo process 640. A large amount of depth map information can be fused together, and some of it can be summarized using surface representations. For example, a mathematically definable surface may be an effective (e.g., relative to a large point cloud) and understandable input to other processing devices (such as a game engine). Therefore, the output of the stereo process (e.g., depth map) 640 can be combined in a fusion process 630. The pose 610 can also be an input to the fusion process 630, and the output of the fusion 630 becomes the input to the fill map process 620. Sub-surfaces can be connected to each other to form larger surfaces, such as in topographic mapping, and the map becomes a large hybrid of points and surfaces.

[0112] To address various aspects of the mixed reality process 660, various inputs may be utilized. Fig. 6A In the embodiment shown in , game parameters can be input to determine that the user of the system is playing a monster battle game with one or more monsters at various locations, walls or other objects at various locations, etc., and the monster dies or flees under various conditions (such as if the user shoots the monster). The world map can include information about the location of objects or semantic information about the objects (e.g., classifications such as whether the object is flat or round, horizontal or vertical, a table or a lamp, etc.), and the world map can be another valuable input for mixed reality. Posture relative to the world can also become an input and can play a key role in almost all interactive systems.

[0113] Control or input from the user can be another input to the wearable system 600. As described herein, user input can include visual input, gestures, totems, audio input, sensory input, etc. In order to move around or play games, for example, the user may need to instruct the wearable system 600 about what he or she wants to do. In addition to moving itself in space, there can also be various forms of user controls that can be utilized. In one embodiment, a totem (e.g., a user input device) or an object such as a toy gun can be held by a user and tracked by the system. The system will preferably be configured to understand that the user is holding the item and understand what type of interaction the user is performing with the item (e.g., if the totem or object is a gun, the system can be configured to understand the position and orientation, and whether the user clicks a trigger or other sensing button or element that may be equipped with a sensor (such as an IMU), which can assist in determining what is happening even when such activity is not within the field of view of any camera).

[0114] Gesture tracking or recognition may also provide input information. The wearable system 600 may be configured to track and interpret gestures for button presses, for gestures to the left or right, to stop, to grasp, to hold, and the like. For example, in one configuration, a user may want to flip through email or a calendar in a non-gaming environment, or to have a "fist bump" with another person or player. The wearable system 600 may be configured to utilize a minimal amount of gestures, which may or may not be dynamic. For example, a gesture may be a simple static gesture, such as an open hand to indicate stop, a thumbs up to indicate yes, a thumbs down to indicate no; or a hand flip to the right or left or up / down to indicate a directional command.

[0115] Eye tracking can be another input (e.g., tracking where the user is looking to control the display technology to render at a specific depth or range). In one embodiment, the vergence of the eyes can be determined using triangulation, and then the accommodation can be determined using a vergence / accommodation model developed for that particular person. Eye tracking can be performed by an eye camera to determine eye gaze (e.g., the direction or orientation of one or both eyes). Other techniques can be used for eye tracking, such as, for example, measuring electrical potential through electrodes placed near the eyes (e.g., an electro-oculometer).

[0116] Voice tracking can be another input and can be used alone or in combination with other inputs (e.g., totem tracking, eye tracking, gesture tracking, etc.). Voice tracking can include voice recognition, sound recognition, alone or in combination. System 600 may include an audio sensor (e.g., a microphone) that receives an audio stream from the environment. System 600 can combine sound recognition technology to determine who is speaking (e.g., whether the voice is from the wearer of the ARD or another person or voice (e.g., whether the voice is from the wearer of the ARD or another person or voice (e.g., recorded voice sent from a speaker in the environment)) and voice recognition technology to determine what is being said. The local data and processing module 260 or the remote processing module 270 can process the audio data from the microphone (or audio data in another stream such as, for example, a video stream that the user is watching) to obtain the audio data by applying various speech recognition algorithms, such as, for example, hidden Markov models, dynamic time warping (DTW) based speech recognition, neural networks, deep learning algorithms (such as deep feedforward and recurrent neural networks), end-to-end automatic speech recognition, machine learning algorithms (refer to Figure 7 description), or use other algorithms such as acoustic modeling or language modeling to recognize speech content.

[0117] The local data and processing module 260 or the remote processing module 270 may also apply a speech recognition algorithm that can identify the identity of the speaker, such as whether the speaker is the user 210 of the wearable system 600 or someone else with whom the user is communicating. Some example speech recognition algorithms may include frequency estimation, hidden Markov models, Gaussian mixture models, pattern matching algorithms, neural networks, matrix representations, vector quantization, speaker binarization, decision trees, and dynamic time warping (DTW) techniques. Speech recognition techniques may also include anti-speaker techniques, such as peer group models and world models. Spectral features may be used to represent speaker characteristics. The local data and processing module or the remote data processing module 270 may use reference Figure 7 Describes various machine learning algorithms to perform speech recognition.

[0118] Implementations of the wearable system can use these user controls or inputs via the UI. UI elements (e.g., controls, pop-up windows, bubbles, data entry fields, etc.) can be used to, for example, turn off the display of information (e.g., graphical or semantic information of an object).

[0119] Regarding the camera system, Fig. 6A The example wearable system 600 shown in FIG. 6 may include three pairs of cameras: a relatively wide FOV or passive SLAM camera pair arranged on both sides of the user's face; a different pair of cameras oriented in front of the user, which are used to process the stereo imaging process 640 and capture gestures and totems / objects tracked in front of the user's face. The FOV cameras and the camera pair used for the stereo process 640 may be an outward-facing imaging system 464 ( Figure 4 Wearable system 600 may include an eye tracking camera (which may be Figure 4 The wearable system 600 may also include one or more texture light projectors (such as infrared (IR) projectors) to inject texture into the scene.

[0120] The wearable system 600 may include an avatar processing and rendering system 690. The avatar processing and rendering system 690 may be configured to generate, update, animate, and render an avatar based on contextual information. Some or all of the avatar processing and rendering systems 690 may be implemented, alone or in combination, as part of the local processing and data module 260 or the remote processing modules 262, 264. In various embodiments, multiple avatar processing and rendering systems 690 (e.g., as implemented on different wearable devices) may be used to render the virtual avatar 670. For example, a first user's wearable device may be used to determine the first user's intent, while a second user's wearable device may determine features of the avatar and render the first user's avatar based on the intent received from the first user's wearable device. As will be described with reference to Fig. 9 As described, the first user's wearable device and the second user's wearable device (or other such wearable devices) may communicate, for example, via a network.

[0121] Figure 6B An example avatar processing and rendering system 690 is shown. The example avatar processing and rendering system 690 may include, alone or in combination, a 3D model processing system 680, a contextual information analysis system 688, an avatar auto-scaler 692, an intent mapping system 694, an anatomical adjustment system 698, and a stimulus response system 696. The system 690 is intended to illustrate functionality for avatar processing and rendering and is not intended to be limiting. For example, in some implementations, one or more of these systems may be part of another system. For example, portions of the contextual information analysis system 688 may be part of the avatar auto-scaler 692, the intent mapping system 694, the stimulus response system 696, or the anatomical adjustment system 698, alone or in combination.

[0122] The context information analysis system 688 may be configured to Figure 2 and Figure 3One or more device sensors described herein may be used to determine environment and object information. For example, the context information analysis system 688 may use images acquired by the outward-facing imaging system 464 of a viewer of the user or the user's avatar to analyze the environment and objects (including physical or virtual objects) of the user's environment or rendering of the user's avatar's environment. The context information analysis system 688 may analyze such images alone or in combination with data acquired from location data or a world map (e.g., maps 620, 710, 910) to determine the location and layout of objects in the environment. The context information analysis system 688 may also typically access biological characteristics of a user or human to realistically animate the virtual avatar 670. For example, the context information analysis system 688 may generate a discomfort curve that may be applied to the avatar so that a portion of the user's avatar's body (e.g., the head) may not be in an uncomfortable (or unrealistic) position relative to other portions of the user's body (e.g., the avatar's head is not turned 270 degrees). In some implementations, one or more object identifiers 708 ( Figure 7 ) can be implemented as part of the contextual information analysis system 688.

[0123] The avatar auto-scaler 692, intent mapping system 694, stimulus response system 696, and anatomical adjustment system 698 may be configured to determine features of the avatar based on contextual information. Some example features of the avatar may include size, appearance, position, orientation, movement, posture, expression, and the like. The avatar auto-scaler 692 may be configured to automatically scale the avatar so that the user does not have to look at the avatar in an uncomfortable position. For example, the avatar auto-scaler 692 may increase or decrease the size of the avatar to place the avatar at the user's eye level so that the user does not need to look down at the avatar or up at the avatar, respectively. The intent mapping system 694 may determine the intent of the user's interaction and map the intent to the avatar (rather than the exact user interaction) based on the environment in which the avatar is rendered. For example, the intent of a first user may be to communicate with a second user in a remote presentation session (e.g., see Fig. 9 ). Typically, two people face each other when communicating. The intent mapping system 694 of the first user's wearable system can determine that such a face-to-face intent exists during the telepresence session, and can cause the first user's wearable system to render the second user's avatar as facing the first user. If the second user were to physically turn around, rather than rendering the second user's avatar in a turned position (which would result in the back of the second user's avatar being rendered to the first user), the first user's intent mapping system 694 can continue to render the second avatar's face to the first user, which is the inferred intent of the telepresence session (e.g., face-to-face intent in this example).

[0124] The stimulus response system 696 can identify an object of interest in the environment and determine the avatar's response to the object of interest. For example, the stimulus response system 696 can identify a sound source in the avatar's environment and automatically turn the avatar to look at the sound source. The stimulus response system 696 can also determine a threshold termination condition. For example, the stimulus response system 696 can return the avatar to its original pose after the sound source disappears or after a period of time has passed.

[0125] The anatomical adjustment system 698 can be configured to adjust the user's posture based on the biological characteristics. For example, the anatomical adjustment system 698 can be configured to adjust the relative position between the user's head and the user's torso or between the user's upper body and lower body based on the discomfort curve.

[0126] The 3D model processing system 680 can be configured to animate the display 220 and render the virtual avatar 670. The 3D model processing system 680 may include a virtual character processing system 682 and a movement processing system 684. The virtual character processing system 682 can be configured to generate and update the user's 3D model (for creating and animating the virtual avatar). The movement processing system 684 can be configured to animate the avatar, such as, for example, by changing the avatar's posture, by moving the avatar around in the user's environment or animating the avatar's facial expressions, etc. As will be further described herein, the virtual avatar can be animated using assembly techniques. In some embodiments, the avatar is represented in two parts: a surface representation (e.g., a deformable mesh) for rendering the external appearance of the virtual avatar and a hierarchical collection of interconnected joints (e.g., a core skeleton) for animating the mesh. In some implementations, the virtual character processing system 682 can be configured to edit or generate a surface representation, while the movement processing system 684 can be used to animate the avatar by moving the avatar, deforming the mesh, etc.

[0127] Example of mapping a user environment

[0128] Figure 7is a block diagram of an example of an MR environment 700. The MR environment 700 can be configured to receive input (e.g., visual input 702 from the user's wearable system, fixed input 704 such as a room camera, sensory input 706 from various sensors, gestures, totems, eye tracking, user input from user input device 466, etc.) from one or more user wearable systems (e.g., wearable system 200 or display system 220) or fixed room systems (e.g., room cameras, etc.). The wearable system can use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, motion sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.) to determine the location and various other attributes of the user environment. This information can be further supplemented with information from a fixed camera in the room, which can provide images or various cues from different perspectives. The image data acquired by a camera (such as a room camera and / or a camera of an outward-facing imaging system) can be reduced to a set of mapping points.

[0129] One or more object identifiers 708 may crawl through the received data (e.g., a collection of points) and identify or map the points, label images, attach semantic information to objects with the aid of a map database 710. The map database 710 may include various points and their corresponding objects collected over time. The various devices and map databases may be connected to each other via a network (e.g., LAN, WAN, etc.) to access the cloud.

[0130] Based on this information and the collection of points in the map database, object recognizers 708a to 708n can recognize objects in the environment. For example, an object recognizer can recognize faces, people, windows, walls, user input devices, televisions, documents (e.g., travel tickets, driver's licenses, passports as described herein in this security example), other objects in the user's environment, etc. One or more object recognizers can be specialized for objects with certain characteristics. For example, object recognizer 708a can be used to recognize faces, while another object recognizer can be used to recognize documents.

[0131] Object recognition can be performed using a variety of computer vision techniques. For example, the wearable system can analyze the imagery generated by the outward-facing imaging system 464 ( Figure 4The computer vision system can be used to perform scene reconstruction, event detection, video tracking, object recognition (e.g., people or documents), object pose estimation, facial recognition (e.g., images of people or documents from the environment), learning, indexing, motion estimation, or image analysis (e.g., identifying marks in documents, such as photos, signatures, identification information, travel information, etc.). One or more computer vision algorithms can be used to perform these tasks. Non-limiting examples of computer vision algorithms include: Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces method, Lucas-Kanade algorithm, Horn-Schunk algorithm, mean shift algorithm, visual simultaneous localization and mapping (vSLAM) technology, sequential Bayesian estimator (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive threshold (and other threshold techniques), iterative closest point (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histogram, various machine learning algorithms (such as, for example, support vector machine, k-nearest neighbor algorithm, naive Bayes, neural network (including convolutional or deep neural network) or other supervised / unsupervised models, etc.), etc.

[0132] Object recognition may be performed additionally or alternatively by various machine learning algorithms. After training, the machine learning algorithms may be stored by the HMD. Some examples of machine learning algorithms may include supervised or unsupervised machine learning algorithms, including regression algorithms (such as, for example, ordinary least squares regression), instance-based algorithms (such as, for example, learning vector quantization), decision tree algorithms (such as, for example, classification and regression trees), Bayesian algorithms (such as, for example, naive Bayes), clustering algorithms (such as, for example, k-means clustering), association rule learning algorithms (such as, for example, prior algorithms), artificial neural network algorithms (such as, for example, Perceptron), deep learning algorithms (such as, for example, Deep Boltzmann Machine or deep neural network), dimensionality reduction algorithms (such as, for example, principal component analysis), holistic algorithms (such as, for example, Stacked Generalization) and / or other machine learning algorithms. In some embodiments, separate models may be customized for individual data sets. For example, a wearable device may generate or store a basic model. The basic model may be used as a starting point for generating additional models specific to a data type (e.g., a specific user in a telepresence session), a data set (e.g., a set of additional images obtained by a user in a telepresence session), a conditional situation, or other variations. In some embodiments, the wearable HMD can be configured to utilize a variety of techniques to generate a model for analyzing aggregated data. Other techniques may include using predefined thresholds or data values.

[0133] Based on this information and a collection of points in a map database, object identifiers 708a to 708n can identify objects and supplement semantic information to bring objects to life. For example, if an object identifier identifies a set of points as a door, the system can attach some semantic information (e.g., a door has a hinge and has a 90-degree motion around the hinge). If an object identifier identifies a set of points as a mirror, the system can attach semantic information that the mirror has a reflective surface that can reflect the image of an object in the room. The semantic information can include the functional affordances of the objects described herein. For example, the semantic information can include the normal of an object. The system can assign a vector whose direction indicates the normal of an object. Over time, the map database grows as the system (which may reside locally or be accessible via a wireless network) collects more data from the world. Once an object is identified, the information can be transmitted to one or more wearable systems. For example, the MR environment 700 can include information about a scene occurring in California. The environment 700 can be transmitted to one or more users in New York. Based on data received from the FOV camera and other inputs, object recognizers and other software components can map points collected from the various images, recognize objects, etc. so that the scene can be accurately "delivered" to a second user, who may be in a different part of the world. Environment 700 may also use topological maps for positioning purposes.

[0134] Figure 8 8 is a process flow diagram of an example of a method 800 for rendering virtual content with respect to an identified object. The method 800 describes how a virtual scene may be presented to a user of a wearable system. The user may be geographically distant from the scene. For example, the user may be in New York, but may want to view a scene currently taking place in California, or may want to take a walk with a friend who lives in California.

[0135] At box 810, the wearable system may receive input about the user's environment from the user and other users. This may be accomplished through various input devices and knowledge already held in a map database. At box 810, the user's FOV camera, sensor, GPS, eye tracking, etc. communicate information to the system. At box 820, the system may determine sparse points based on this information. The sparse points may be used to determine pose data (e.g., head pose, eye pose, body pose, or hand gesture) that may be used to display and understand the orientation and position of various objects in the user's surroundings. At box 830, the object recognizers 708a-708n may crawl through these collected points and identify one or more objects using a map database. Then, at box 840, this information may be communicated to the user's separate wearable system, and at box 850, the desired virtual scene may be displayed to the user accordingly. For example, the desired virtual scene (e.g., a user in California) may be displayed with appropriate orientation, position, etc. relative to various objects and other surroundings of the user in New York.

[0136] Example communication between multiple wearable systems

[0137] Fig. 9 Schematically illustrates an overall system view depicting multiple user devices interacting with each other. Computing environment 900 includes user devices 930a, 930b, 930c. User devices 930a, 930b and 930c can communicate with each other through network 990. User devices 930a-930c can each include a network interface to communicate with remote computing system 920 (which can also include network interface 971) via network 990. Network 990 can be a LAN, WAN, peer-to-peer network, radio, Bluetooth or any other network. Computing environment 900 can also include one or more remote computing systems 920. Remote computing system 920 can include a cluster of server computer systems located in different geographical locations. User devices 930a, 930b and 930c can communicate with remote computing system 920 via network 990.

[0138] Remote computing system 920 may include remote data repository 980, which may maintain information about a particular user's physical and / or virtual worlds. Data repository 980 may store information about the user, the user's environment (e.g., a world map of the user's environment), or the configuration of the user's avatar. The remote data repository may be Figure 2 The remote computing system 920 may also include a remote processing module 970. The remote processing module 970 may be Figure 2 . The remote processing module 970 may include one or more processors that may communicate with the user devices (930a, 930b, 930c) and the remote data repository 980. The processors may process information obtained from the user devices and other sources. In some implementations, at least a portion of the processing or storage may be performed by the local processing and data module 260 (e.g., Figure 2 The remote computing system 920 may enable a given user to share information about the particular user's own physical and / or virtual world with another user.

[0139] The user device may be a wearable device (such as an HMD or ARD), a computer, a mobile device, or any other device, alone or in combination. For example, user devices 930b and 930c may be devices that can be configured to present AR / VR / MR content. Figure 2 The wearable system 200 (or Figure 4 An embodiment of a wearable system 400) is shown in FIG.

[0140] One or more user devices may be connected to Figure 4 The user device can be used together with the user input device 466 shown in FIG. The user device can obtain information about the user and the user's environment (e.g., using Figure 4 ). The user device and / or remote computing system 1220 can use the information obtained from the user device to construct, update, and build a collection of images, points, and other information. For example, the user device can process the acquired raw information and send the processed information to the remote computing system 1220 for further processing. The user device can also send the raw information to the remote computing system 1220 for processing. The user device can receive the processed information from the remote computing system 1220 and provide final processing before projecting it to the user. The user device can also process the acquired information and pass the processed information to other user devices. The user device can communicate with the remote data repository 1280 while processing the acquired information. Multiple user devices and / or multiple server computer systems can participate in the construction and / or processing of the acquired images.

[0141] Information about the physical world can evolve over time and can be based on information collected by different user devices. Models of the virtual world can also evolve over time and based on input from different users. Such information and models may sometimes be referred to herein as world maps or world models. Fig. 6A , 6B and Figure 7 As described above, the information obtained by the user device can be used to construct the world map 910. The world map 910 may include Fig. 6A Various object identifiers (e.g., 708a, 708b, 708c ... 708n) may be used to identify objects and label images, as well as attach semantic information to objects. These object identifiers are also Figure 7 Described in.

[0142] Remote data repository 980 can be used to store data and facilitate the construction of world map 910. User devices can continuously update information about user environments and receive information about world map 910. World map 910 can be created by users or others. As discussed herein, user devices (e.g., 930a, 930b, 930c) and remote computing system 920 can construct and / or update world map 910 alone or in combination. For example, user devices can communicate with remote processing module 970 and remote data repository 980. User devices can obtain and / or process information about users and user environments. Remote processing module 970 can communicate with remote data repository 980 and user devices (e.g., 930a, 930b, 930c) to process information about users and users' environments. The remote computing system 920 may modify the information acquired by the user device (e.g., 930a, 930b, 930c), such as, for example, selectively cropping the user's image, modifying the user's background, adding virtual objects to the user's environment, annotating the user's voice with auxiliary information, etc. The remote computing system 920 may send the processed information to the same and / or different user devices.

[0143] Example process of 3D model sharing

[0144] Fig.10An example process 1000 of sharing 3D assets using the systems and methods described herein is shown. At step 1001, a 3D sharing host application can access a 3D model file from a host system. The host can do so by creating a new 3D asset on the host system or obtaining the 3D asset from an external system. The 3D asset can be derived from a 3D modeling application, such as AutoCad or 3D Studio Max, and can have any file type. The host can utilize an asset import plug-in (e.g., ASSIMP) to implement graphic format exchange. The ASSIMP plug-in can support up to 40 different file formats, for example, Filmbox (FBX) in .fbx format or Autodesk Max 3D Modeling (3DS) in .3ds format. In some embodiments, a custom program can be written to implement file format exchange. Any other suitable file format conversion method can be used to implement file reformatting used within the host 3D sharing application.

[0145] The 3D sharing host application may decompose the 3D asset into component parts at step 1003. Examples of component parts may be geometry data, material data, vertex tables, one or more textures, triangle indices, or any other data used to ultimately define a complete representation of a 3D model.

[0146] At step 1005, the component may be sent to a client that is part of a network with a host. In some embodiments, there may be only one client. In some embodiments, there may be two or more clients. In a preferred embodiment, a local area network may be utilized using the UDP protocol. The network may be a LAN, WAN, peer-to-peer network, radio, Bluetooth, ad hoc, or any other network. In some embodiments, the network may be network 990 as described above. The host may be a remote computing system, which may include server computer systems that are clustered and located at different geographical locations, as described above in the context of 920. The client may be user devices 930a, 930b, and / or 930c as described above. In some embodiments, there may be one host. In some embodiments, there may be two or more hosts. In some embodiments, each device connected to the network may be both a host and a client, thereby enabling sharing and receiving 3D assets from other devices on the network.

[0147] At step 1007, the client may receive the component.

[0148] At step 1009, the client may reassemble the 3D asset from the components received from the host. In some embodiments, the 3D asset may be stored in a memory of the client device. In some embodiments, the 3D asset may be saved for later use by the device.

[0149] At step 1011, the client may display the reassembled 3D asset. In some embodiments, the client may not display the reassembled 3D asset, or may display the asset at a later time, such as upon user request. In some embodiments, the client may display the reassembled 3D asset, but the user may not be able to see the reassembled 3D asset because it is outside the user's FOV.

[0150] Example 3D model sharing system configuration

[0151] Fig.11 An example 3D model sharing system configuration 1100 is shown for sharing 3D assets using the systems and methods described herein. Fig.11A system configuration 1100 is depicted that includes one client (which may be referred to as a "remote") system 1116 and one server (which may be referred to as a "host") system 1118. The client system and the host system need not be physically separate systems; for example, in some embodiments, the client and server may exist on the same physical device (e.g., as client and server threads executed simultaneously by one or more processors of the same computer system). In practice, the 3D sharing system 1100 may generally require at least one host system 1118 that is capable of acting as a source of decomposed 3D assets. The host system 1118 may have authoritative control over the types and quantities of 3D models available within the 3D sharing application. The 3D sharing system 1100 may have any number (e.g., zero, one, ten, etc.) of client systems 1116 that are operably connected to the host system 1118 via a communication network 1114. The communication network 1114 may be, for example, a LAN utilizing the User Datagram Protocol (UDP). The host system 1118 may include one or more processors that are operably coupled to at least one display and capable of receiving communications from other computing systems. In some embodiments, the host system 1118 may be a PC desktop computer attached to a computer monitor with a network interface. In some embodiments, the host system 1118 may be the system 200, 930a, 930b or 930c as described above. The user devices 200, 930a, 930b and 930c may communicate with each other and other computing systems through the communication network 1114. The user devices 930a-930c may each include a network interface that communicates with a remote computing system (such as 920 (which may also include a network interface 971)) via the network 1114. The client system 1116 may include one or more processors that are operably coupled to at least one display and are capable of receiving communications from other computing systems. In some embodiments, the client system 1116 may be a PC desktop computer attached to a computer monitor with a network interface. In some embodiments, the client system 1118 may be the system 200, 930a, 930b or 930c as described above. The user devices 200, 930a, 930b and 930c may communicate with each other and other computing systems through the communication network 1114. User devices 930a - 930c may each include a network interface to communicate with a remote computing system such as 920 (which may also include network interface 971 ) via network 1114 .

[0152] The host system 1118 may have a 3D sharing host application installed. The 3D sharing host application may include three main modules: an asset module 1108, a decomposition module 1110, and a transmission module 1112. The asset module 1108 may include one or more complete 3D models. The decomposition module 1110 may be used to separate the complete 3D model into components, optionally compress the components, and place the components into one or more arrays that are ready to be sent to different systems. The arrays may be stored in one or more libraries that may contain one or more decomposed 3D models. The transmission module 1112 may be used to separate the components from the decomposition module 1110 into transferable data fragments, and may manage the transmission protocols and processes. The transmission module 1112 may send the transferable data fragments to other systems connected to the communication network 1114.

[0153] The client or remote system 1116 may have a 3D sharing client application installed. The 3D sharing client application may include three main modules: a transmission module 1106, a reassembly module 1104, and an asset module 1102. The transmission module 1106 may be used to receive one or more decomposed 3D models, and may manage the transmission and networking process of the client system 1116. The transmission module 1106 may communicate with the 3D model sharing host application via a messaging protocol. These protocols may be custom built protocols, or may utilize commonly used protocols. The reassembly module 1104 may be used to reassemble received decomposed 3D models from other systems on the network. The asset module 1102 may include one or more complete 3D models.

[0154] Example process for 3D model sharing

[0155] Fig.12 An example 3D model sharing process 1200 between a server and a client using the systems and methods described herein is shown. The process 1200 may begin at step 1202 when a server system, such as server system 1118, opens a 3D model sharing host application.

[0156] At step 1204, the server may load one or more complete 3D model files into the 3D model sharing host application. For example, the complete 3D model files may originate from the host system 1118 and are derived from different 3D modeling software (e.g., AutoCad) that is downloaded to the host system 1118. In some embodiments, the 3D model files may be downloaded from an external source (such as an external hard drive, a remote computing system (e.g., from above) Fig. 9920)) or imported from cloud storage. In some embodiments, the 3D model file can originate from a 3D scanner or similar device. The 3D model file can be imported using ASSIMP or an equivalent plug-in. The plug-in can convert the 3D model file from one format to a different file format compatible with the 3D model sharing host application. The plug-in can only be utilized by a host system (e.g., host system 1118) because the file formatting can disappear at the server during the decomposition process, as described in more detail below.

[0157] At step 1206, the server system may decompose the 3D model into its components. Decomposing the 3D model may include identifying the components within the 3D model. Decomposing the 3D model may be non-destructive because it does not result in changes to the 3D model itself (e.g., information loss in the 3D model) or changes in how the 3D model is rendered for viewing. Decomposing the 3D model may include applying data-level changes to the 3D model and / or its components (e.g., applying data formatting changes, or compressing components to facilitate sharing). The decomposition process may occur in the decomposition module 1110, as described above. The components may be subsets of data required to fully define the 3D model. The component subsets may be defined by pre-existing data bundles (such as vertex tables, geometric data, material data, etc.) that include the 3D model. In some embodiments, component subsets may be defined in other ways (such as by a 3D model sharing application programmer).

[0158] At step 1214, the client system may open a 3D model sharing client application. This may initiate a series of steps related to networking protocols, communications, and discovery.

[0159] At step 1216, if the host exists, the 3D model sharing client application can discover the 3D model sharing host via the network. The network can be a LAN (e.g., UDP). In some embodiments, the local network is private. In some embodiments, cloud services are not used.

[0160] At step 1208, the server may receive a connection request from the client, and the server may accept the connection request.

[0161] At step 1210, the server may send a client information request back to the client. The client information request may contain a list of everything in the library of the 3D model sharing host application. The client information request may alternatively be referred to as an asset list or table of contents.

[0162] At step 1218, the client may reply back to the server with data indicating which items are missing from the 3D model sharing client application library when compared with the 3D model sharing server application library.

[0163] At step 1212, the server may respond by sending the client the missing items from the 3D model sharing client application. In some embodiments, the client may have recently opened the 3D model sharing application and may therefore start with zero items in its library. In this case, the host may send its entire library to the client. In some embodiments, the library may be placed in a memory and may therefore be erased when the 3D model sharing application is closed. In some embodiments, the library may be saved to a disk, such as a client system hard drive, so the 3D models may be accessed at a later date. In some embodiments, the 3D model data sent to the client is a data structure without a file format. At the end of this step, the 3D model sharing host library may match the 3D model sharing client library. In other words, the host system and the client system may now contain the same components of the 3D model for the same group.

[0164] In some embodiments, the client may update outdated items when compared to items stored in the server. For example, the client information request may include version numbers associated with various assets in the asset list. One or more assets in the server's asset list may correspond to assets stored by the client. The client may identify an item stored in the client's asset library that has a lower version number than a corresponding item in the server's asset list. The client may send data to the server identifying the item that needs to be updated, and the server may respond by sending the corresponding updated item to the client.

[0165] At step 1220, the client may reassemble one or more 3D models based on the new items sent from the server. In some embodiments, the client may now have the same complete 3D model as the server.

[0166] At step 1222, the user of the client system can now view the 3D model shared by the host system.

[0167] Example 3D model sharing system configuration

[0168] Fig.13 An example 3D model sharing system configuration 1300 is shown for sharing 3D assets using the systems and methods described herein.

[0169] 3D model sharing system configuration 1300 may include a host system 1306 and a client system 1304 operatively coupled by a communication link 1302 (such as a network). Client system 1304 may have a 3D model sharing client application open and running, and thus operate in an operating environment. Host system 1306 may have a 3D model sharing host application open and running, and thus operate in an operating environment. Fig.13 A server system and a host system are depicted, but this is not meant to limit the scope and is done for ease of illustration. In practice, the 3D model sharing system configuration 1300 may have only one host system 1306, more than one host system 1306, one client system 1304, more than one client system 1304, and / or one or more systems that act as both a host (e.g., sending 3D model data to other systems via communication 1302) and a client (e.g., receiving 3D model data from other systems via communication 1302). In addition, in some embodiments, a single system may include both a host system and a client system. In some embodiments, the host system 1306 may be the host system 1118. In some embodiments, the client system 1304 may be the client system 1116. The host system 1306 may include one or more processors operably coupled to at least one display, such as a PC desktop computer attached to a computer monitor. In some embodiments, the host system 1306 may be the system 200, 930a, 930b, or 930c as described above. The client system 1304 may include one or more processors operably coupled to at least one display, such as a PC desktop computer attached to a computer monitor. In some embodiments, the client system 1304 may be the system 200, 930a, 930b, or 930c as described above.

[0170] Host system 1306 may include server module 1316. Server module 1316 may manage server-specific functions and processes, such as transport and networking protocols. Server module 1316 may receive and process connection requests, such as steps 1208, 1210, and / or 1212 in process 1200. In some embodiments, server module 1316 may include transport module 1112.

[0171] The host system 1306 may include a host loading module 1320. The loading module may contain functions and processes that enable the 3D model sharing host application to import or load 3D assets into the 3D model sharing host application or onto the host system 1306. The loading module 1320 may include an ASSIMP plug-in. In some embodiments, step 1204 in process 1200 may occur within the loading module 1320. In some embodiments, step 1001 in process 1000 may occur within the loading module 1320. In some embodiments, the asset module 1108 may include the loading module 1320.

[0172] Host system 1306 may include a host library manager module 1318. Library manager module 1318 may contain and manage data and processes corresponding to 3D assets that have been loaded into the 3D asset sharing host application. Library manager module 1318 may contain and manage data and processes corresponding to the library of the 3D asset sharing host application. This may include decomposition processes, maintaining a library table of content, etc. In some embodiments, step 1206 in process 1200 may occur in library manager module 1318. In some embodiments, step 1003 in process 1000 may occur in library manager module 1318. In some embodiments, library manager module 1318 may include decomposition module 1110 from system 1100.

[0173] The host system 1306 may include a host world module 1322. The host world module 1322 may manage all 3D assets, 3D virtual worlds, meshing of the real world, and the integration between these. In some embodiments, the host world module 1322 may be considered a scene graph, or a functional equivalent of a scene graph. In some embodiments, the world may be obtained from items from a world library (i.e., a completely reassembled 3D model) and world objects may be created from them. In some embodiments, everything in a 3D virtual world may be represented as a world object. A world object may be part of a world "class". In some embodiments, the host world module 1322 contains the same data as the client world module 1312, either through step 1222 in process 1200 or through step 1011 in process 1000 (i.e., after the 3D model has been shared and reassembled locally at the client). The output from the host world module 1322 may be fed into a renderer path module 1324.

[0174] The host system 1306 may include a host renderer pathway module 1324. The renderer pathway module 1324 may accept input data from the host world module 1322, place it through the host system 1306 renderer pipeline, and may display one or more 3D models on a display of the host system 1306.

[0175] The client system 1304 may include a client module 1308. The client module 1304 may manage functions and processes specific to being a client, such as transport and networking and / or server / client protocols. The client module 1308 may manage the server discovery process and other connection protocols. In some embodiments, this may include steps 1216 and / or 1218 in process 1200. In some embodiments, the client module 1308 may include the transport module 1106.

[0176] The client system 1304 may include a client library manager module 1310. The library manager module 1310 may contain and manage data and processes corresponding to 3D asset components that have been received from the 3D asset sharing host application. The library manager module 1310 may contain and manage data and processes corresponding to the library of the 3D asset sharing client application. This may include a reorganization process. In some embodiments, step 1220 in process 1200 may occur in the library manager module 1310. In some embodiments, the library manager module 1310 may include the reorganization module 1104 from the system 1100. In some embodiments, the client library contains the same data as the host library, either by step 1222 in process 1200 or by step 1011 in process 1000 (i.e., after the 3D model has been shared and reorganized locally at the client).

[0177] The client system 1304 may include a client world module 1312. The client world module 1312 may manage fully reassembled 3D assets, 3D virtual worlds, meshing of the real world, and the like, as well as the integration between these. In some embodiments, the client world module 1312 may be considered a scene graph, or a functional equivalent of a scene graph. In some embodiments, the world is derived from items from a world library (i.e., fully reassembled 3D models) and world objects are created from them. In some embodiments, everything in the 3D virtual world is represented as a world object. A world object may be part of a world "class". In some embodiments, the client world module 1312 may contain the same data as the host world module 1322, by step 1222 in process 1200 or step 1011 in process 1000 (i.e., after the 3D models have been shared and reassembled locally at the client). The output from the client world module 1312 may be fed into the renderer path module 1314.

[0178] The client system 1304 may include a client renderer pathway module 1312. The renderer pathway module 1314 may accept input data from the client world module 1312, place it through the client system 1304 renderer pipeline, and may display one or more virtual 3D models on a display of the client system 1304.

[0179] The example 3D model sharing system configuration 1300 can enable two or more devices to share a complete 3D model during runtime. This can have the advantage of adding new assets to applications on a computing system faster than traditional systems, where updates may need to wait until the application is closed.

[0180] Example process for decomposing a 3D model

[0181] Fig.14 An example process 1400 is shown for decomposing a complete 3D model using the systems and methods described herein. In some embodiments, the decomposition process 1400 can occur in the host library manager module 1318. In some embodiments, the decomposition process 1400 can occur in the decomposition module 1110. In some embodiments, the decomposition process 1400 can be step 1003 in the process 1000. In some embodiments, the decomposition process 1400 can be step 1206 in the process 1200.

[0182] Process 1400 may begin with a complete 3D model that can be loaded within a 3D model sharing host application. At step 1402, the 3D model sharing host application may copy a component of the 3D model and place data representing the component into an array. Some example components may be one or more vertex positions, vertex tables, geometry data, material data, textures, triangle index tables, etc. A component may be part of the data needed to represent a complete 3D model.

[0183] The components may optionally be compressed at step 1404. Standard compression techniques may be used.

[0184] At step 1406, the components may be packaged.

[0185] At step 1408, the package may be stored in one of the libraries of the 3D model sharing host application. The 3D model sharing host application may have one or more libraries for storing and organizing packages. In some embodiments, the library may include three libraries—a mesh library, a material library, and a texture library, such as Fig.16 As shown in . Alternative library configurations may be used. Packages may be stored in corresponding libraries (e.g., textures may be stored in a texture library).

[0186] At step 1410, the process can check the 3D model to see if there are additional components that need to be added to the library. If there are more components that have not been added to the library, the process can start again at step 1402 until all components are added to the library. When all components have been added, the library can contain a complete record of the complete 3D model, as in step 1412.

[0187] In some embodiments, the 3D model can be decomposed into components predefined by the 3D model application. For example, a 3D model application (e.g., a computer-aided design program) can define a 3D model based on different data elements (e.g., a mesh and a mesh renderer), each of which can include one or more defined components (e.g., a triangle array or a texture). Decomposing the 3D model in this case can constitute applying an operation (e.g., data formatting) to each component and storing it in a memory. In some embodiments, the methods and systems described herein can be used to decompose the 3D model into one or more subgroups defined by a user. For example, for a 3D model of a car, a user can specify a set of vertices / parts / structures as an "engine", a set of vertices / parts / structures as a "chassis", a set of vertices / parts / structures as a "drivetrain", and a set of vertices / parts / structures as a "wheel". Each subgroup can then be optionally compressed and sent to the device separately. In some embodiments, the 3D model can be decomposed into algorithmically defined subgroups. For example, an algorithm can be developed (e.g., manually, semi-automatically, and / or automatically through machine learning) to identify a set of vertices / parts / structures as an "engine". The algorithm may first identify the entire 3D model (e.g., first identify the model as a car) before attempting to classify the set of vertices / parts / structures from the finite pool based on the initial identification. The algorithm may also directly attempt to classify the set of vertices / parts / structures.

[0188] Example of a complete 3D model

[0189] Fig.15 A complete 3D model 1500 of an example using the systems and methods described herein is shown. In some embodiments, a complete 3D model may be a complete set of data describing a 3D renderable digital object. In some embodiments, a complete 3D model may be a set of data representing a digital or virtual object that can be displayed when added to a compatible renderer pipeline.

[0190] At the 3D model component 1502, the 3D model may be assigned a name. The name may be assigned by the user, the import process, and / or may be the original file name used in the 3D model creation software, such as the file name given to the 3D model in AutoCad. The 3D model may have two main types of data: mesh data 1504 and mesh renderer data 1506. The mesh data may be further divided into subsets, or components, of the data, such as component A 1508, component B 1510, component 1512, and component 1514. The material data 1516 may be further divided into subsets, or components, of the data, such as material properties 1522, texture 1 1518, and texture 2 1522. The exact layout, classification, and sub-grouping of the 3D model data may be related to the 3D model. Fig.15 In some embodiments, there may be any number of components under grid 1504, such as two, three, or ten components. In some embodiments, there may be multiple subsets of properties 1522 under material data 1516, rather than just one. In some embodiments, there may be more than one subset of properties 1522. Fig.15 More or fewer texture components 1518, 1520 are shown in FIG. Any suitable number and classification may be used to divide the complete 3D model into components.

[0191] In some embodiments, an exemplary complete 3D model 1500 may be decomposed using the exemplary decomposition process 1400. The complete 3D model 1500 may include one or more components (e.g., component A 1508, component B 1510, component C 1512, component D 1514, texture 1 1518, texture 2 1520, and / or material properties 1522). A component (e.g., texture 1) of the complete 3D model 1500 may be copied at step 1402 of the decomposition process 1400. The components may optionally be compressed at step 1404 of the decomposition process 1400. The components may be packaged at step 1406 of the decomposition process 1400. The components may be stored (e.g., in the exemplary library group 1600) at step 1408 of the decomposition process 1400. Steps 1402 , 1404 , 1406 , and 1408 may be repeated for each component in the complete 3D model 1500 until all components of the complete 3D model 1500 have been stored (eg, in library set 1600 ).

[0192] Example of a library in a 3D model sharing app

[0193] Fig.16A library group 1600 for storing component examples of 3D models used in a 3D model sharing application using the systems and methods described herein is shown. The library group 1600 may include one or more libraries to store and organize 3D model components. In some embodiments, there may be three libraries: a mesh library 1602, a material library 1604, and a texture library 1602. The mesh library 1602 may contain a set of components in the form of arrays that describe a 3D model mesh. An example array that may be stored in the mesh library is a vertex buffer array 1608, a normal buffer array 1610, a UV buffer array 1612, and / or a triangle array 1614. The mesh library may also optionally contain one or more string values, such as a material "name" 1616, a bitmap texture 1 "name" 1618, and a bitmap texture 2 "name" 1620. These string values ​​1616, 1618, 1620 may reference other libraries, such as a material library 1604 and a texture library 1604. In some embodiments, a complete 3D model may be represented when all data is pulled or referenced from the mesh library 1602. In some embodiments, a complete 3D model for a "name" may be represented when all data is pulled or referenced from all libraries 1602, 1604, 1606 corresponding to a record, or a single 3D model, "name".

[0194] The material library 1604 may contain data representing properties and values ​​associated with the properties of materials used for the 3D model. In some embodiments, the texture library 1606 may contain one or more textures 1622, 1624. In some embodiments, the texture library 1606 may contain one or more bitmaps. In some embodiments, the texture library 1606 may contain one or more UV maps, UV coordinates, or other UV space data.

[0195] All data contained in the 3D sharing application's library 1600 may be combined to represent a complete renderable 3D model.

[0196] Example process for reassembling a 3D model

[0197] Fig.17 An example process 1700 is shown for reassembling a complete 3D model from its constituent parts using the systems and methods described herein. In some embodiments, the process can be step 1220 in process 1200. In some embodiments, process 1700 can be step 1009 in process 1000. In some embodiments, process 1700 can occur in reassembly module 1104. In some embodiments, process 1700 can occur in library manager module 1310.

[0198] Process 1700 may begin at step 1702 by creating an empty object (sometimes referred to as an empty game object). In some embodiments, an empty game object may include an empty volume in space having a reference coordinate system. In some embodiments, an empty object is essentially a transformation—a position and orientation—that exists in space. In some embodiments, an empty object is an empty prism. In some embodiments, an empty object may correspond to Fig.15 1502 of them.

[0199] After creating the empty object at 1702, a new mesh 1704 and a new mesh renderer 1708 may be added to the empty game object. In some embodiments, adding a new mesh 1704 may correspond to Fig.15 The mesh 1504 in , and adding a new mesh renderer 1708 may correspond to Fig.15 The mesh renderer 1506 in .

[0200] Process 1700 may proceed to steps 1706 and 1710, where an empty mesh buffer may be filled with 3D model components corresponding to the mesh data, and an empty mesh renderer buffer may be filled with 3D model components corresponding to the mesh renderer data. In some embodiments, steps 1706 and / or 1710 may decompress the compressed data. In some embodiments, the mesh data may correspond to the 3D model components from the mesh renderer data. Fig.15 1508, 1510, 1512, 1514 of the components. In some embodiments, the mesh renderer data may correspond to the components 1518, 1520, 1522. The buffer may be filled with data corresponding to a single 3D model from the library of the 3D model application. In some embodiments, the empty mesh buffer may be filled 1706 with data from the mesh library 1602. In some embodiments, the empty mesh render buffer may be filled 1710 with data from the material library 1604 and the texture library 1602. When all data corresponding to a single 3D model (i.e., a single record within the library of the 3D model sharing application) has been added to the empty object, process 1700 may be complete.

[0201] Fig.18 A complete 3D model 1800 of an example using the systems and methods described herein is shown. In some embodiments, the complete 3D model 1800 may be an output of the process 1700. In some embodiments, the complete 3D model 1800 may be a more detailed version of the complete 3D model 1500 of the example, where the common component A may be a vertex buffer 1808, the common component B may be a normal buffer 1810, the common component C may be a UV buffer 1812, and the common component D may be a triangle index 1814. In some embodiments, the triangle index 1814 may need to be filled in last.

[0202] In some embodiments, the complete 3D model 1800 may be an input to process 1400, from which the components are copied. In some embodiments, the complete 3D model 1800 may be stored as a world object in world modules 1322 and / or 1312. In some embodiments, the complete 3D model 1800 may be a 3D model loaded into a 3D model sharing server application at step 1204 in process 1200 and / or step 1001 in process 1000. In some embodiments, the complete 3D model 1800 may be an output from step 1220 in process 1200 and / or step 1009 in process 1000. In some embodiments, the complete 3D model 1800 may be an asset stored in asset modules 1102 and / or 1108. The object "name" 1802 may match the original file name loaded into the 3D model sharing application, e.g., car.fbx, even though the 3D model is no longer in the FBX file format.

[0203] Other examples of 3D model sharing

[0204] In some embodiments, the 3D model can be derived from a MR device (e.g., wearable system 200). The MR device can capture information about a physical object, generate a 3D model of the physical object, decompose the 3D model into components, send the components to a client device, and the client device can generate a 3D model based on the components. For example, the object identifier 708 can receive input from an outward-facing imaging system 464, a radar sensor, a depth sensor, an RGB camera, an infrared camera, a time-of-flight camera, and / or other sensors on the MR device. The input can be used in a computer vision algorithm and / or a machine learning trained algorithm to generate a 3D model based on a physical object. The algorithm can generate a 3D model by first identifying the physical object and then generating a 3D model based on a library of 3D models associated with the physical object. In some embodiments, a subgroup can be assembled based on a predefined subgroup in a library of 3D models to generate a 3D model from a physical object. In some embodiments, the 3D model can be decomposed into subgroups defined by a user. In some embodiments, the 3D model can be decomposed into subgroups defined by the algorithm. For example, an algorithm may have been developed (eg, manually, semi-automatically, and / or automatically through machine learning) to identify sets of vertices / parts / structures that are subgroups.

[0205] In some embodiments, the algorithm can also generate a 3D model directly from a physical object. For example, the algorithm can identify the surfaces and vertices of a physical object and generate a 3D model of the entire physical object based on the observed properties of the physical object. In some embodiments, the algorithm can identify local surfaces and vertices of a physical object and generate a 3D model of the local portion based on a library of subgroups and / or define subgroups of the physical object. The algorithm can store data about the association of subgroups to form a complete 3D model based on the components.

[0206] The 3D model sharing system and method described in the present disclosure may provide a specific solution to the technical problem of how to move a renderable 3D model from one computing system to a different computing system during runtime. This may present an improvement over the prior art, which only adds new renderable 3D models to applications offline.

[0207] Although 3D models are discussed herein, it is also contemplated that the systems and methods described herein may be applied to other digital assets besides 3D models. For example, a 2D model may be decomposed, transferred, and reconstructed using the systems and methods described herein (e.g., into outlines and textures). In some embodiments, an animation may be decomposed, transferred, and reconstructed using the systems and methods described herein (e.g., into a series of meshes, each of which may be further decomposed as described herein). In some embodiments, a 3D model may be flattened into a 2D model (e.g., to display on a 2D screen or to create a 2D map).

[0208] Other considerations

[0209] Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying drawings may be embodied in a code module executed by one or more physical computing systems, hardware computer processors, dedicated circuits, and / or electronic hardware configured to execute dedicated and specific computer instructions, and may be automatically executed in whole or in part by the code module. For example, a computing system may include a general-purpose computer (e.g., a server) programmed with specific computer instructions or a dedicated computer, dedicated circuit, or the like. A code module may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language. In some implementations, specific operations and methods may be performed by circuits specific to a given function.

[0210] Furthermore, certain implementations of the functionality of the present disclosure are sufficiently mathematically, computationally, or technically complex that dedicated hardware or one or more physical computing devices (with appropriate dedicated executable instructions) may be necessary to perform the functionality or provide results in substantially real time, e.g., due to the amount or complexity of the computations involved. For example, an animation or video may include many frames, each with millions of pixels, and specially programmed computer hardware may be required to process the video data to provide a desired image processing task or application in a commercially reasonable amount of time. As another example, computing the weight map, rotation, and translation parameters of a skinning system by solving a constrained optimization problem for these parameters is highly computationally intensive (e.g., see reference 1 to 13). Fig.14 Example process 1400 described above).

[0211] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device including a hard drive, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disk, volatile or non-volatile storage device, a combination of the same or the like, and the like. The methods and modules (or data) may also be sent as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless-based and wired / cable-based media), and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device, or may be transmitted via a computer-readable transmission medium.

[0212] Any process, frame, state, step or function in the flowchart described herein and / or depicted in the accompanying drawings should be understood to represent a code module, code segment or code portion, which includes one or more executable instructions for implementing a specific function (e.g., logic or arithmetic) or a step in a process. Various processes, blocks, states, steps or functions can be combined, rearranged, added, deleted, modified or otherwise changed with the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps or states associated therewith can be performed in appropriate other sequences, for example, in serial, parallel or in some other manner. Tasks or events can be added to or removed from the disclosed example embodiments. In addition, the separation of various system components in the implementation described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

[0213] The processes, methods and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowd-sourced computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0214] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or requires the desired attributes disclosed herein. The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications to the implementations described in the present disclosure will be apparent, and the general principles defined herein can be applied to other implementations without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the implementations shown herein, but are consistent with the widest scope consistent with the present disclosure, principles, and novel features disclosed herein.

[0215] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from the claimed combination may be excluded from the combination in some cases, and the claimed combination may be used in subcombinations or variations of subcombinations. No single feature or group of features is required or essential for each and all embodiments.

[0216] Unless otherwise specified, or understood otherwise in the context of use, conditional language used herein, such as "can", "may", "possibly", "may", "for example", etc., is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not include certain features, elements, and / or steps. Therefore, such conditional language is generally not intended to imply that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments must include logic for deciding whether to include or perform these features, elements, and / or steps (whether or not there is author input or prompt) in any particular embodiment. The terms "comprise", "include", "have", etc. are synonymous and are used inclusively in an open-ended manner, and do not exclude additional elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusive sense (rather than in its exclusive sense), so when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, unless otherwise specified, the articles "a", "an", and "the" used in this application and the appended claims should be understood to mean "one or more" or "at least one".

[0217] As used herein, phrases referring to "at least one" of a list of items refer to any combination of those items, including single members. For example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B and C. Unless otherwise specifically stated, conjunctive language such as the phrase "at least one of X, Y, and Z" is otherwise understood by context as being generally used to convey that an item, term, etc. may be at least one of X, Y, or Z. Thus, such conjunctive language is generally not intended to imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z, respectively.

[0218] Similarly, although operations may be depicted in the drawings in a particular order, it should be appreciated that it is not necessary to perform these operations in the particular order shown or in sequence, or to perform all the operations shown, to achieve the desired result. In addition, the drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations not shown may be included in the schematically illustrated example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously, or between any of the illustrated operations. In addition, operations may be rearranged or reordered in other implementations. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above implementations should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products. In addition, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in different orders and still achieve the desired results.

[0219] Example Embodiments

[0220] 1. A method comprising:

[0221] displaying, via a display of a head mounted device, a first version of a virtual three-dimensional model, wherein the first version of the virtual three-dimensional model includes a first version of a component;

[0222] requesting data associated with a second version of the virtual three-dimensional model from a host device, wherein the second version of the virtual three-dimensional model includes a second version of the component;

[0223] determining whether the first version of the component needs to be updated based on the second version of the component;

[0224] Based on determining that the first version of the component requires an update based on the second version of the component:

[0225] requesting data associated with the second version of the component from the host device;

[0226] displaying the second version of the virtual three-dimensional model via the display of the head mounted device; and

[0227] Based on determining that the first version of the component does not require an update based on the second version of the component:

[0228] A request for data associated with the second version of the component from the host device is abandoned.

[0229] 2. The method of embodiment 1, wherein the component comprises grid data.

[0230] 3. The method of embodiment 1, wherein the component comprises texture data.

[0231] 4. The method of embodiment 1, wherein the host device is a server.

[0232] 5. The method of embodiment 1, wherein the head mounted device is a first head mounted device, and wherein the host device is a second head mounted device.

[0233] 6. The method of embodiment 1, further comprising storing the data associated with the second version of the component in a memory.

[0234] 7. The method of embodiment 1, further comprising decompressing the data associated with the second version of the component.

[0235] 8. A system comprising:

[0236] A head mounted device including a display;

[0237] One or more processors configured to execute a method comprising:

[0238] displaying, via the display, a first version of the virtual three-dimensional model, wherein the first version of the virtual three-dimensional model includes a first version of a component;

[0239] requesting data associated with a second version of the virtual three-dimensional model from a host device, wherein the second version of the virtual three-dimensional model includes a second version of the component;

[0240] determining whether the first version of the component needs to be updated based on the second version of the component;

[0241] Based on determining that the first version of the component requires an update based on the second version of the component:

[0242] requesting data associated with the second version of the component from the host device;

[0243] displaying the second version of the virtual three-dimensional model via the display; and

[0244] Based on determining that the first version of the component does not require an update based on the second version of the component:

[0245] A request for data associated with the second version of the component from the host device is abandoned.

[0246] 9. The system of embodiment 8, wherein the component comprises grid data.

[0247] 10. The system of embodiment 8, wherein the component comprises texture data.

[0248] 11. The system of embodiment 8, wherein the host device is a server.

[0249] 12. The system of embodiment 8, wherein the head mounted device is a first head mounted device, and wherein the host device is a second head mounted device.

[0250] 13. The system of embodiment 8, wherein the method further comprises storing the data associated with the second version of the component in a memory.

[0251] 14. The system of embodiment 8, the method further comprising decompressing the data associated with the second version of the component.

[0252] 15. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising:

[0253] displaying, via a display of a head mounted device, a first version of a virtual three-dimensional model, wherein the first version of the virtual three-dimensional model includes a first version of a component;

[0254] requesting data associated with a second version of the virtual three-dimensional model from a host device, wherein the second version of the virtual three-dimensional model includes a second version of the component;

[0255] determining whether the first version of the component needs to be updated based on the second version of the component;

[0256] Based on determining that the first version of the component requires an update based on the second version of the component:

[0257] requesting data associated with the second version of the component from the host device;

[0258] displaying the second version of the virtual three-dimensional model via the display of the head mounted device; and

[0259] Based on determining that the first version of the component does not require an update based on the second version of the component:

[0260] A request for data associated with the second version of the component from the host device is abandoned.

[0261] 16. The non-transitory computer readable medium of embodiment 15, wherein the component comprises mesh data.

[0262] 17. The non-transitory computer readable medium of embodiment 15, wherein the component comprises texture data.

[0263] 18. The non-transitory computer readable medium of embodiment 15, wherein the host device is a server.

[0264] 19. The non-transitory computer-readable medium of embodiment 15, wherein the head mounted device is a first head mounted device, and wherein the host device is a second head mounted device.

[0265] 20. The non-transitory computer readable medium of embodiment 15, the method further comprising storing the data associated with the second version of the component in a memory.

[0266] 21. The non-transitory computer readable medium of embodiment 15, the method further comprising decompressing the data associated with the second version of the component.

[0267] 22. A method comprising:

[0268] accessing a virtual three-dimensional model stored in a memory;

[0269] determining one or more components of the virtual three-dimensional model;

[0270] storing the one or more components in one or more arrays, wherein the one or more components are stored separately from the virtual three-dimensional model;

[0271] receiving a connection request from the head mounted device;

[0272] sending a list of available components to the head mounted device;

[0273] receiving a component request from the head mounted device; and

[0274] The requested component is sent to the head mounted device based on the component request.

[0275] 23. The method of embodiment 22, wherein the component comprises mesh data.

[0276] 24. The method of embodiment 22, wherein the component comprises texture data.

[0277] 25. The method of embodiment 22, further comprising compressing the one or more components.

[0278] 26. The method of embodiment 22, wherein the virtual three-dimensional model is a first virtual three-dimensional model, and wherein the method further comprises generating a copy of the virtual three-dimensional model based on the requested component.

[0279] 27. The method of embodiment 22, further comprising displaying the requested component via a display of the head mounted device.

[0280] 28. The method of embodiment 22, further comprising:

[0281] creating a placeholder for the new virtual three-dimensional model; and

[0282] The placeholder for the new virtual three-dimensional model is updated based on the requested component.

[0283] 29. A system, comprising:

[0284] Head mounted device;

[0285] One or more processors configured to execute a method comprising:

[0286] accessing a virtual three-dimensional model stored in a memory;

[0287] determining one or more components of the virtual three-dimensional model;

[0288] storing the one or more components in one or more arrays, wherein the one or more components are stored separately from the virtual three-dimensional model;

[0289] receiving a connection request from the head mounted device;

[0290] sending a list of available components to the head mounted device;

[0291] receiving a component request from the head mounted device; and

[0292] The requested component is sent to the head mounted device based on the component request.

[0293] 30. The system of embodiment 29, wherein the component comprises mesh data.

[0294] 31. The system of embodiment 29, wherein the component comprises texture data.

[0295] 32. The system of embodiment 29, wherein the method further comprises compressing the one or more components.

[0296] 33. The system of embodiment 29, wherein the virtual three-dimensional model is a first virtual three-dimensional model, and wherein the method further comprises generating a copy of the virtual three-dimensional model based on the requested component.

[0297] 34. According to the system of embodiment 29, the method also includes displaying the requested component via a display of the head-mounted device.

[0298] 35. The system of embodiment 29, wherein the method further comprises:

[0299] creating a placeholder for the new virtual three-dimensional model; and

[0300] The placeholder for the new virtual three-dimensional model is updated based on the requested component.

[0301] 36. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising:

[0302] accessing a virtual three-dimensional model stored in a memory;

[0303] determining one or more components of the virtual three-dimensional model;

[0304] storing the one or more components in one or more arrays, wherein the one or more components are stored separately from the virtual three-dimensional model;

[0305] receiving a connection request from the head mounted device;

[0306] sending a list of available components to the head mounted device;

[0307] receiving a component request from the head mounted device; and

[0308] The requested component is sent to the head mounted device based on the component request.

[0309] 37. The non-transitory computer readable medium of embodiment 36, wherein the component comprises mesh data.

[0310] 38. The non-transitory computer readable medium of embodiment 36, wherein the component comprises texture data.

[0311] 39. The non-transitory computer-readable medium of embodiment 36, wherein the method further comprises compressing the one or more components.

[0312] 40. The non-transitory computer-readable medium of embodiment 36, wherein the virtual three-dimensional model is a first virtual three-dimensional model, and wherein the method further comprises generating a copy of the virtual three-dimensional model based on the requested component.

[0313] 41. According to the non-transitory computer-readable medium of embodiment 36, the method also includes displaying the requested component via a display of the head-mounted device.

[0314] 42. The non-transitory computer readable medium of embodiment 36, wherein the method further comprises:

[0315] creating a placeholder for the new virtual three-dimensional model; and

[0316] The placeholder for the new virtual three-dimensional model is updated based on the requested component.

[0317] 43. A system comprising:

[0318] host computing system;

[0319] A client computing system including a head mounted display system;

[0320] The host computing system includes one or more processors configured to execute a method comprising:

[0321] accessing a virtual three-dimensional model stored in a memory;

[0322] decomposing the three-dimensional model into one or more component parts;

[0323] sending the list of one or more components to the client computing system;

[0324] receiving a component request from the client computing system;

[0325] sending the one or more components corresponding to the component request to the client computing system; and

[0326] The client computing system includes one or more processors configured to execute a method, the method comprising:

[0327] receiving said list of said one or more components from said host computing system;

[0328] sending the component request to the host computing system;

[0329] receiving, from the host computing system, one or more components corresponding to the component request;

[0330] The copy of the virtual three-dimensional model is composed by one or more components corresponding to the component request.

[0331] 44. The system of embodiment 43, wherein the component comprises grid data.

[0332] 45. The system of embodiment 43, wherein the component comprises texture data.

[0333] 46. ​​The system of embodiment 43, wherein the host computing system is a server.

[0334] 47. A system according to embodiment 43, wherein the host computing system includes a head-mounted display system.

[0335] 48. According to the system of embodiment 43, the method also includes storing the one or more components in a memory.

[0336] 49. According to the system of embodiment 43, the method also includes decompressing the one or more components.

[0337] 50. A method comprising:

[0338] accessing a virtual three-dimensional model stored in a memory;

[0339] decomposing the three-dimensional model into one or more component parts;

[0340] sending the list of the one or more components to a client computing system including a head mounted display;

[0341] receiving a component request from the client computing system;

[0342] sending the one or more components corresponding to the component request to the client computing system;

[0343] receiving said list of said one or more components from a host computing system;

[0344] sending the component request to the host computing system;

[0345] receiving from the host computing system one or more components corresponding to the component request; and

[0346] A copy of the virtual three-dimensional model is composed using one or more components corresponding to the component request.

[0347] 51. The method of embodiment 50, wherein the component comprises mesh data.

[0348] 52. The method of embodiment 50, wherein the component comprises texture data.

[0349] 53. The method of embodiment 50, wherein the host computing system is a server.

[0350] 54. A method according to embodiment 50, wherein the host computing system includes a head-mounted display system.

[0351] 55. The method of embodiment 50, further comprising storing the one or more components in a memory.

[0352] 56. The method of embodiment 50, further comprising decompressing the one or more components.

[0353] 57. A non-transitory computer readable medium storing one or more instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising:

[0354] accessing a virtual three-dimensional model stored in a memory;

[0355] decomposing the three-dimensional model into one or more component parts;

[0356] sending the list of the one or more components to a client computing system including a head mounted display;

[0357] receiving a component request from the client computing system;

[0358] sending the one or more components corresponding to the component request to the client computing system;

[0359] receiving said list of said one or more components from a host computing system;

[0360] sending the component request to the host computing system;

[0361] receiving from the host computing system one or more components corresponding to the component request; and

[0362] A copy of the virtual three-dimensional model is composed by one or more components corresponding to the component request.

[0363] 58. The non-transitory computer readable medium of embodiment 57, wherein the component comprises mesh data.

[0364] 59. The non-transitory computer readable medium of embodiment 57, wherein the component comprises texture data.

[0365] 60. The non-transitory computer readable medium of embodiment 57, wherein the host computing system is a server.

[0366] 61. The non-transitory computer-readable medium of embodiment 57, wherein the host computing system comprises a head-mounted display system.

[0367] 62. According to the non-transitory computer-readable medium of embodiment 57, the method also includes storing the one or more components in a memory.

[0368] 63. According to the non-transitory computer-readable medium of embodiment 57, the method also includes decompressing the one or more components.

[0369] 64. A system comprising:

[0370] A host computing system, wherein the host computing system comprises one or more processors configured to execute a method comprising:

[0371] accessing, via a host asset module, a first virtual three-dimensional model;

[0372] identifying one or more components via a decomposition module;

[0373] copying the one or more components to a library via the decomposition module;

[0374] accessing the one or more components via a host transport module;

[0375] transmitting the one or more components to a client computing system via the host transport module; and

[0376] The client computing system, wherein the client computing system comprises a head mounted display system, and wherein the client computing system comprises one or more processors configured to perform a method comprising:

[0377] receiving the one or more components from the host computing system via a client transport module;

[0378] Create empty objects by composing modules;

[0379] adding one or more data types to the empty object via the composition module;

[0380] adding one or more components to the empty object via the composition module;

[0381] storing a copy of the virtual three-dimensional model via a client asset module; and

[0382] The host computing system and the client computing system are communicatively connected via a communication link.

[0383] 65. The system of embodiment 64, wherein the component comprises grid data.

[0384] 66. The system of embodiment 64, wherein the component comprises texture data.

[0385] 67. The system of embodiment 64, wherein the host computing system is a server.

[0386] 68. A system according to embodiment 64, wherein the host computing system includes a head-mounted display system.

[0387] 69. According to the system of embodiment 64, the method also includes storing the one or more components in a memory.

[0388] 70. According to the system of embodiment 64, the method also includes decompressing the one or more components.

[0389] 71. A method comprising:

[0390] At a host computing system, wherein the host computing system comprises one or more processors configured to perform a method comprising:

[0391] accessing, via a host asset module, a first virtual three-dimensional model;

[0392] identifying one or more components via a decomposition module;

[0393] copying the one or more components to a library via the decomposition module;

[0394] accessing the one or more components via a host transport module;

[0395] transmitting the one or more components to a client computing system via the host transport module; and

[0396] At the client computing system, wherein the client computing system comprises a head mounted display system, and wherein the client computing system comprises one or more processors configured to perform a method comprising:

[0397] receiving the one or more components from the host computing system via a client transport module;

[0398] Create empty objects by composing modules;

[0399] adding one or more data types to the empty object via the composition module;

[0400] adding one or more components to the empty object via the composition module;

[0401] storing a copy of the virtual three-dimensional model via a client asset module; and

[0402] The host computing system and the client computing system are communicatively connected via a communication link.

[0403] 72. A method according to embodiment 71, wherein the component includes grid data.

[0404] 73. A method according to embodiment 71, wherein the component includes texture data.

[0405] 74. The method of embodiment 71, wherein the host computing system is a server.

[0406] 75. A method according to embodiment 71, wherein the host computing system includes a head-mounted display system.

[0407] 76. The method of embodiment 71, further comprising storing the one or more components in a memory.

[0408] 77. The method of embodiment 71, further comprising decompressing the one or more components.

[0409] 78. A non-transitory computer readable medium storing one or more instructions that, when executed by one or more processors, cause the processors to perform a method comprising:

[0410] At the host computing system:

[0411] accessing, via a host asset module, a first virtual three-dimensional model;

[0412] identifying one or more components via a decomposition module;

[0413] copying the one or more components to a library via the decomposition module;

[0414] accessing the one or more components via a host transport module;

[0415] transmitting the one or more components to a client computing system via the host transport module; and

[0416] At the client computing system, wherein the client computing system comprises a head mounted display system:

[0417] receiving the one or more components from the host computing system via a client transport module;

[0418] Create empty objects by composing modules;

[0419] adding one or more data types to the empty object via the composition module;

[0420] adding one or more components to the empty object via the composition module;

[0421] storing a copy of the virtual three-dimensional model via a client asset module; and

[0422] Wherein, the host computing system and the client computing system are communicatively connected via a communication link.

[0423] 79. The non-transitory computer-readable medium of embodiment 78, wherein the component comprises mesh data.

[0424] 80. The non-transitory computer readable medium of embodiment 78, wherein the component comprises texture data.

[0425] 81. The non-transitory computer-readable medium of embodiment 78, wherein the host computing system is a server.

[0426] 82. The non-transitory computer-readable medium of embodiment 78, wherein the host computing system comprises a head-mounted display system.

[0427] 83. According to the non-transitory computer-readable medium described in embodiment 78, the method also includes storing the one or more components in a memory.

[0428] 84. According to the non-transitory computer-readable medium of embodiment 78, the method also includes decompressing the one or more components.

[0429] 85. A system comprising:

[0430] A host computing system, wherein the host computing system comprises one or more processors configured to execute a method comprising:

[0431] accessing the first virtual three-dimensional model via the loading module;

[0432] identifying, via a host library manager module, one or more components;

[0433] copying the one or more components to a library via the host library manager module; storing the first virtual three-dimensional model via a host world module;

[0434] Rendering the first virtual three-dimensional model via a host renderer path module;

[0435] accessing the one or more components via a server module;

[0436] sending the one or more components to a client computing system via the server module; and

[0437] The client computing system, wherein the client computing system comprises a head mounted display system, and wherein the client computing system comprises one or more processors configured to perform a method comprising:

[0438] receiving the one or more components from the host computing system via a client module;

[0439] Generating a copy of the virtual three-dimensional model, wherein generating the copy of the virtual three-dimensional model comprises:

[0440] Creating an empty object via the client library manager module; and

[0441] adding, via the client library manager module, one or more components to the empty object;

[0442] storing the copy of the virtual three-dimensional model via a client world module;

[0443] rendering the copy of the virtual three-dimensional model via a client renderer pathway module;

[0444] Wherein, the host computing system and the client computing system are communicatively connected via a communication link.

[0445] 86. The system of embodiment 85, wherein the component comprises grid data.

[0446] 87. The system of embodiment 85, wherein the component comprises texture data.

[0447] 88. The system of embodiment 85, wherein the host computing system is a server.

[0448] 89. The system of embodiment 85, wherein the host computing system comprises a head mounted display system.

[0449] 90. According to the system of embodiment 85, the method also includes storing the one or more components in a memory.

[0450] 91. According to the system of embodiment 85, the method also includes decompressing the one or more components.

[0451] 92. A method comprising:

[0452] At the host computing system:

[0453] accessing the first virtual three-dimensional model via the loading module;

[0454] identifying, via a host library manager module, one or more components;

[0455] copying the one or more components to a library via the host library manager module; storing the first virtual three-dimensional model via a host world module;

[0456] Rendering the first virtual three-dimensional model via a host renderer path module;

[0457] accessing the one or more components via a server module;

[0458] sending the one or more components to a client computing system via the server module; and

[0459] At the client computing system, wherein the client computing system comprises a head mounted display system:

[0460] receiving the one or more components from the host computing system via a client module;

[0461] Generating a copy of the virtual three-dimensional model, wherein generating the copy of the virtual three-dimensional model comprises:

[0462] Creating an empty object via the client library manager module; and

[0463] adding, via the client library manager module, one or more components to the empty object;

[0464] storing the copy of the virtual three-dimensional model via a client world module;

[0465] rendering the copy of the virtual three-dimensional model via a client renderer pathway module;

[0466] Wherein, the host computing system and the client computing system are communicatively connected via a communication link.

[0467] 93. A method according to embodiment 92, wherein the component includes grid data.

[0468] 94. The method of embodiment 92, wherein the component comprises texture data.

[0469] 95. The method of embodiment 92, wherein the host computing system is a server.

[0470] 96. A method according to embodiment 92, wherein the host computing system includes a head-mounted display system.

[0471] 97. The method of embodiment 92, further comprising storing the one or more components in a memory.

[0472] 98. The method according to embodiment 92 further includes decompressing the one or more components.

[0473] 99. A non-transitory computer readable medium storing one or more instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising:

[0474] A host computing system, wherein the host computing system comprises one or more processors configured to execute a method comprising:

[0475] accessing the first virtual three-dimensional model via the loading module;

[0476] identifying, via a host library manager module, one or more components;

[0477] copying the one or more components to a library via the host library manager module; storing the first virtual three-dimensional model via a host world module;

[0478] Rendering the first virtual three-dimensional model via a host renderer path module;

[0479] accessing the one or more components via a server module;

[0480] sending the one or more components to a client computing system via the server module; and

[0481] The client computing system, wherein the client computing system comprises a head mounted display system, and wherein the client computing system comprises one or more processors configured to perform a method comprising:

[0482] receiving the one or more components from the host computing system via a client module;

[0483] Generating a copy of the virtual three-dimensional model, wherein generating the copy of the virtual three-dimensional model comprises:

[0484] Creating an empty object via the client library manager module; and

[0485] adding, via the client library manager module, one or more components to the empty object;

[0486] storing the copy of the virtual three-dimensional model via a client world module;

[0487] The copy of the virtual three-dimensional model is rendered via a client renderer pathway module; wherein the host computing system and the client computing system are communicatively connected via a communication link.

[0488] 100. The non-transitory computer-readable medium of embodiment 99, wherein the component comprises mesh data.

[0489] 101. The non-transitory computer-readable medium of embodiment 99, wherein the component comprises texture data.

[0490] 103. The non-transitory computer-readable medium of embodiment 99, wherein the host computing system is a server.

[0491] 104. The non-transitory computer-readable medium of embodiment 99, wherein the host computing system comprises a head-mounted display system.

[0492] 105. According to the non-transitory computer-readable medium described in embodiment 99, the method also includes storing the one or more components in a memory.

[0493] 106. According to the non-transitory computer-readable medium described in embodiment 99, the method also includes decompressing the one or more components.

[0494] 107. A system comprising:

[0495] a non-transitory computer-readable memory storing a three-dimensional model including one or more components, and

[0496] A host computing system comprising one or more processors configured to access the three-dimensional model from the non-transitory computer-readable memory and further configured to perform a method comprising:

[0497] storing one of the one or more components of the three-dimensional model in an array stored in the non-transitory computer readable memory,

[0498] The array is packaged into a package in a library of the host computing system.

[0499] 108. The system of embodiment 107, wherein the component comprises grid data.

[0500] 109. The system of embodiment 107, wherein the component comprises texture data.

[0501] 110. The system of embodiment 107, wherein the host computing system is a server.

[0502] 111. The system of embodiment 107, wherein the host computing system comprises a head mounted display system.

[0503] 112. According to the system of embodiment 107, the method also includes storing the one or more components in a memory.

[0504] 113. According to the system of embodiment 107, the method also includes decompressing the one or more components.

[0505] 114. A method comprising:

[0506] storing, via a non-transitory computer-readable memory, a three-dimensional model including one or more components;

[0507] accessing the three-dimensional model from the non-transitory computer-readable memory;

[0508] storing one of the one or more components of the three-dimensional model in an array stored in the non-transitory computer readable memory,

[0509] The array is packaged into a package in a library of the host computing system.

[0510] 115. A method according to embodiment 114, wherein the component includes grid data.

[0511] 116. A method according to embodiment 114, wherein the component includes texture data.

[0512] 117. The method of embodiment 114, wherein the non-transitory computer-readable memory is part of a server.

[0513] 118. A method according to embodiment 114, wherein the non-transitory computer-readable memory is part of a head-mounted display system.

[0514] 119. The method of embodiment 114, further comprising storing the one or more components in a memory.

[0515] 120. The method according to embodiment 114 further includes decompressing the one or more components.

[0516] 121. A non-transitory computer readable medium storing one or more instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising:

[0517] storing, via a non-transitory computer-readable memory, a three-dimensional model including one or more components;

[0518] accessing the three-dimensional model from the non-transitory computer-readable memory;

[0519] storing one of the one or more components of the three-dimensional model in an array stored in the non-transitory computer readable memory,

[0520] The array is packaged into a package in a library of the host computing system.

[0521] 122. The non-transitory computer-readable medium of embodiment 121, wherein the component comprises mesh data.

[0522] 123. The non-transitory computer-readable medium of embodiment 121, wherein the component comprises texture data.

[0523] 124. The non-transitory computer-readable medium of embodiment 121, wherein the non-transitory computer-readable memory is part of a server.

[0524] 125. The non-transitory computer-readable medium of embodiment 121, wherein the non-transitory computer-readable memory is part of a head-mounted display system.

[0525] 126. According to the non-transitory computer-readable medium described in embodiment 121, the method also includes storing the one or more components in a memory.

[0526] 127. According to the non-transitory computer-readable medium described in embodiment 121, the method also includes decompressing the one or more components.

[0527] Examples of components may be geometry data, material data, a vertex table, one or more textures, triangle indices, or any other data that is used to ultimately define a complete representation of a 3D model.

Claims

1. A system for sharing a three-dimensional model, comprising: host computing system; as well as A client computing system including a head mounted display system, The host computing system includes one or more processors configured to execute a method comprising: accessing a virtual three-dimensional model stored in a memory; Decomposing the virtual three-dimensional model into one or more components, wherein the one or more components include a group of components, and the group of components includes mesh data, texture data, or a combination thereof; determining whether the client computing system is connected to the host computing system; sending a library including the one or more components to the client computing system based on a determination that the client computing system is connected to the host computing system, the library corresponding to the virtual three-dimensional model; receiving a component request from the client computing system; and sending one or more components corresponding to the component request to the client computing system; The client computing system includes one or more processors configured to execute a method, the method comprising: receiving said library of said one or more components from said host computing system; determining a request portion to include in a request for the component based on a comparison of the library of the one or more components with a client library of components; sending the component request to the host computing system; receiving from the host computing system one or more components corresponding to the component request; and A copy of the virtual three-dimensional model is assembled based on the received one or more components.

2. The system according to claim 1, wherein: The host computing system includes a server.

3. The system according to claim 1, wherein: The host computing system includes a head-mounted display system.

4. The system according to claim 1, wherein the method further comprises: The one or more components are stored in a memory.

5. The system of claim 1, wherein the method further comprises: The one or more components are decompressed.

6. A method for sharing a three-dimensional model, comprising: accessing a virtual three-dimensional model stored in a memory; Decomposing the virtual three-dimensional model into one or more components, wherein the one or more components include a group of components, and the group of components includes mesh data, texture data, or a combination thereof; determining whether the client computing system is connected to the host computing system; sending a library including the one or more components to the client computing system including a head mounted display based on a determination that the client computing system is connected to the host computing system, the library corresponding to the virtual three-dimensional model; receiving a component request from the client computing system; sending one or more components corresponding to the component request to the client computing system; receiving said library of said one or more components from said host computing system; determining a request portion to include in a request for the component based on a comparison of the library of the one or more components with a client library of components; sending the component request to the host computing system; receiving from the host computing system one or more components corresponding to the component request; and A copy of the virtual three-dimensional model is assembled based on the received one or more components.

7. The method according to claim 6, wherein: The host computing system includes a server.

8. The method according to claim 6, wherein: The host computing system includes a head-mounted display system.

9. The method according to claim 6, further comprising: The one or more components are stored in a memory.

10. The method according to claim 6, further comprising: The one or more components are decompressed.

11. A non-transitory computer readable medium storing one or more instructions that, when executed by one or more processors, cause the one or more processors to perform a method for three-dimensional model sharing, the method: accessing a virtual three-dimensional model stored in a memory; Decomposing the virtual three-dimensional model into one or more components, wherein: The one or more components include a set of components, the set of components including mesh data, texture data, or a combination thereof; determining whether the client computing system is connected to the host computing system; sending a library including the one or more components to the client computing system including a head mounted display based on a determination that the client computing system is connected to the host computing system, the library corresponding to the virtual three-dimensional model; receiving a component request from the client computing system; sending one or more components corresponding to the component request to the client computing system; receiving said library of said one or more components from said host computing system; determining a request portion to include in a request for the component based on a comparison of the library of the one or more components with a client library of components; sending the component request to the host computing system; receiving, from the host computing system, one or more components corresponding to the component request; as well as A copy of the virtual three-dimensional model is assembled based on the received one or more components.

12. The non-transitory computer readable medium of claim 11, wherein: The host computing system includes a server.

13. The non-transitory computer readable medium of claim 11, wherein: The host computing system includes a head-mounted display system.

14. The non-transitory computer readable medium of claim 11, the method further comprising: The one or more components are stored in a memory.

15. The non-transitory computer readable medium of claim 11, the method further comprising: The one or more components are decompressed.