Method and system for sharing 3D models - Patents.com
The system decomposes and shares 3D models in real-time, addressing inefficiencies in existing technologies by enabling dynamic updates and reducing storage needs in mixed reality environments.
Patent Information
- Application Number
- JP2022139736
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-09-24
- Filing Date
- 2022-09-02
- Publication Date
- 2024-11-11
- Estimated Expiration
- 2039-09-24
AI Technical Summary
Existing technologies face challenges in efficiently sharing and updating three-dimensional models between computing systems, particularly in mixed reality environments, due to the complexity of model decomposition and the impracticality of frequent or unplanned updates, which often require large storage and inefficient data transfer methods.
A system and method for decomposing three-dimensional models into component parts, transmitting these components over a network, and reconstructing the models in real-time on client devices, allowing for efficient sharing and updating without the need for extensive storage or pre-loading.
Enables fast and efficient sharing of 3D models during runtime, reducing storage requirements and facilitating seamless collaboration in mixed reality applications by allowing dynamic updates and interactions.
Smart Images

Figure 0007580434000001 
Figure 0007580434000002 
Figure 0007580434000003
Abstract
Description
[Technical field]
[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 62 / 735,675, filed Sep. 24, 2018, the contents of which are incorporated herein by reference in their entirety.
[0002] The present disclosure relates to sharing three-dimensional models between two or more computing systems, including mixed reality, imaging, and visualization systems. [Background technology]
[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality" and "mixed reality" experiences, where digitally reproduced images are presented to a user in a manner that appears or can be perceived as real. Virtual reality (VR) scenarios typically involve the presentation of computer-generated virtual image information without transparency to other actual real-world visual inputs. Augmented reality (AR) scenarios typically involve the presentation of virtual image information as an augmentation to the visualization of the real world around the user. Mixed reality (MR) is a type of augmented reality in which physical and virtual objects can coexist and interact in real time. The systems and methods disclosed herein address various challenges associated with VR, AR and MR technologies. Summary of the Invention [Means for solving the problem]
[0004] An embodiment of the present disclosure describes a system and method for decomposing and sharing a 3D model. In an exemplary method, a first version of a virtual three-dimensional model is displayed via a display of a wearable head device. A request is made to a host device for data associated with a second version of the virtual three-dimensional model, the second version of the virtual three-dimensional model comprising a component part. It is determined whether the first version of the virtual three-dimensional model comprises a component part. In accordance with a determination that the first version of the virtual three-dimensional model does not comprise the component part, a request is made to the host device for data associated with the component part. The second version of the virtual three-dimensional model is displayed via a display of the wearable head device. In accordance with a determination that the first version of the virtual three-dimensional model comprises the component part, a request is not made to the host device for data associated with the component part. The present invention provides, for example, the following: (Item 1) 1. A system comprising: A host computing system; A client computing system having a head mounted display system; Equipped with The host computing system includes: Accessing a virtual three-dimensional model stored in a memory; decomposing the three-dimensional model into one or more component parts; transmitting the list of the one or more components to the client computing system; receiving a component request from the client computing system; transmitting to the client computing system one or more components corresponding to the component request; one or more processors configured to execute a method comprising: Equipped with The client computing system includes: receiving a list of the one or more components from the host computing system; sending the component request to the host computing system; receiving one or more of the components corresponding to the component request from the host computing system; constructing a copy of the virtual three-dimensional model from one or more of the component parts corresponding to the component part requests; one or more processors configured to execute a method comprising: A system comprising: (Item 2) Item 1 , the system comprising: (Item 3) Item 1 , the system comprising: (Item 4) 2. The system of claim 1, wherein the host computing system comprises a server. (Item 5) Item 1 , the host computing system comprising a head-mounted display system. (Item 6) 2. The system of claim 1, wherein the method further comprises storing the one or more components in a memory. (Item 7) 2. The system of claim 1, wherein the method further comprises disassembling the one or more components. (Item 8) 1. A method comprising: Accessing a virtual three-dimensional model stored in a memory; decomposing the three-dimensional model into one or more component parts; transmitting the list of the one or more components to a client computing system having a head mounted display; receiving a component request from the client computing system; transmitting to the client computing system one or more components corresponding to the component request; receiving a list of the one or more components from a host computing system; sending the component request to the host computing system; receiving one or more of the components corresponding to the component request from the host computing system; constructing a copy of the virtual three-dimensional model from one or more of the component parts corresponding to the component part requests; A method comprising: (Item 9) 9. The method of claim 8, wherein the component parts comprise mesh data. (Item 10) 9. The method of claim 8, wherein the component parts comprise texture data. (Item 11) 9. The method of claim 8, wherein the host computing system comprises a server. (Item 12) 9. The method of claim 8, wherein the host computing system comprises a head mounted display system. (Item 13) 9. The method of claim 8, further comprising storing the one or more components in a memory. (Item 14) 9. The method of claim 8, further comprising disassembling the one or more components. (Item 15) A non-transitory computer-readable medium having stored thereon one or more instructions that, when executed by one or more processors, cause the one or more processors to: Accessing a virtual three-dimensional model stored in a memory; decomposing the three-dimensional model into one or more component parts; transmitting the list of the one or more components to a client computing system having a head mounted display; receiving a component request from the client computing system; transmitting to the client computing system one or more components corresponding to the component request; receiving a list of the one or more components from a host computing system; sending the component request to the host computing system; receiving one or more of the components corresponding to the component request from the host computing system; constructing a copy of the virtual three-dimensional model from one or more of the component parts corresponding to the component part requests; A non-transitory computer readable medium for carrying out a method comprising: (Item 16) Item 16. The non-transitory computer-readable medium of item 15, wherein the component comprises mesh data. (Item 17) Item 16. The non-transitory computer-readable medium of item 15, wherein the component parts comprise texture data. (Item 18) Item 16. The non-transitory computer-readable medium of item 15, wherein the host computing system comprises a server. (Item 19) Item 16. The non-transitory computer-readable medium of item 15, wherein the host computing system comprises a head-mounted display system. (Item 20) 20. The non-transitory computer-readable medium of claim 15, wherein the method further comprises storing the one or more components in a memory. (Item 21) 20. The non-transitory computer-readable medium of claim 15, wherein the method further comprises disassembling the one or more component parts. (Item 22) 1. A method comprising: Accessing a virtual three-dimensional model stored in a memory; decomposing the virtual three-dimensional model into one or more component parts; storing the one or more components in one or more arrays, the one or more components being stored separately from the virtual three-dimensional model; receiving a connection request from a wearable head device; Sending a list of available components to the wearable head device; receiving a component request from the wearable head device; transmitting the requested component part to the wearable head device based on the component part request; A method comprising: (Item 23) 1. A method comprising: Sending a connection request to a host device via the wearable head device; receiving a list of available components from the host device; sending a component request to the host device; receiving a requested component from the host device based on the component request; reconstructing a copy of the virtual three-dimensional model based on the requested components; and A method comprising: [Brief description of the drawings]
[0005] The details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims.
[0006] [Figure 1] FIG. 1 depicts an illustration of a mixed reality scenario involving a virtual reality object and a physical object viewed by a person.
[0007] [Diagram 2] FIG. 2 illustrates diagrammatically an embodiment of a wearable system.
[0008] [Diagram 3] FIG. 3 diagrammatically illustrates example components of a wearable system.
[0009] [Figure 4] FIG. 4 diagrammatically illustrates an example of a waveguide stack of a wearable device for outputting image information to a user.
[0010] [Diagram 5] FIG. 5 is a process flow diagram of an embodiment of a method for interacting with a virtual user interface.
[0011] [Figure 6A] FIG. 6A is a block diagram of another embodiment of a wearable system that may include an avatar processing and rendering system.
[0012] [Figure 6B] FIG. 6B illustrates example components of an avatar processing and rendering system.
[0013] [Figure 7] FIG. 7 is a block diagram of an embodiment of a wearable system, including various inputs into the wearable system.
[0014] [Figure 8] FIG. 8 is a process flow diagram of an embodiment of a method for rendering virtual content in relation to recognized objects.
[0015] [Figure 9] FIG. 9 diagrammatically illustrates an overall system diagram depicting multiple wearable systems interacting with each other.
[0016] [Figure 10] FIG. 10 illustrates an example process of sharing a 3D asset using the systems and methods described herein.
[0017] [Figure 11] FIG. 11 illustrates an example 3D model sharing system configuration for sharing 3D assets using the systems and methods described herein.
[0018] [Figure 12] FIG. 12 illustrates an example 3D model sharing process between a server and a client using the systems and methods described herein.
[0019] [Figure 13] FIG. 13 illustrates an example 3D model sharing system configuration for sharing 3D assets using the systems and methods described herein.
[0020] [Figure 14] FIG. 14 illustrates an example process for decomposing a full 3D model using the systems and methods described herein.
[0021] [Figure 15] FIG. 15 illustrates an example full 3D model using the systems and methods described herein.
[0022] [Figure 16] FIG. 16 illustrates an example set of libraries that may be utilized in a 3D model sharing application to store components of a 3D model using the systems and methods described herein.
[0023] [Figure 17]FIG. 17 illustrates an example process for reconstructing a complete 3D model from its constituent parts using the systems and methods described herein.
[0024] [Figure 18] FIG. 18 illustrates an example full 3D model using the systems and methods described herein.
[0025] [Figure 19A] 19A-19C illustrate an example mixed reality environment. [Figure 19B] 19A-19C illustrate an example mixed reality environment. [Figure 19C] 19A-19C illustrate an example mixed reality environment.
[0026] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0027] Mixed Reality Environment
[0028] Like all people, users of a mixed reality system are present in a real environment, i.e., the three-dimensional portion of the "real world" and all of its content that is perceivable by the user. For example, the user perceives the real environment using one of the normal human senses, i.e., sight, sound, touch, taste, and smell, and interacts with the real environment by moving his or her own body within the real environment. Locations within the real environment can be described as coordinates in a coordinate space, e.g., the coordinates can comprise latitude, longitude, and elevation relative to sea level, distance in three orthogonal dimensions from a reference point, or other suitable values. Similarly, a vector can describe a quantity that has a direction and a magnitude within the coordinate space.
[0029] A computing device may maintain a representation of a virtual environment, for example, in a memory associated with the device. As used herein, a virtual environment is a computational representation of a three-dimensional space. A virtual environment may include a representation of any objects, actions, signals, parameters, coordinates, vectors, or other properties associated with that space. In some examples, a circuit (e.g., a processor) of a computing device may maintain and update the state of the virtual environment, i.e., the processor may determine the state of the virtual environment at a second time t1 based on data associated with the virtual environment and / or input provided by a user at a first time t0. For example, if an object in the virtual environment is located at a first coordinate at time t0 and has certain programmed physical parameters (e.g., mass, coefficient of friction), and input received from a user indicates that a force should be applied to the object in a directional vector, the processor may apply the laws of kinematics and use basic mechanics to determine the location of the object at time t1. The processor may use any suitable information known about the virtual environment and / or any suitable input (e.g., real-world parameters) to determine the state of the virtual environment at time t1. In maintaining and updating the state of the virtual environment, the processor may execute any suitable software, including software related to the creation and deletion of virtual objects within the virtual environment, software (e.g., scripts) for defining behavior of virtual objects or characters in the virtual environment, software for defining behavior of signals (e.g., audio signals) in the virtual environment, software for creating and updating parameters associated with the virtual environment, software for generating audio signals in the virtual environment, software for handling inputs and outputs, software for implementing network operations, software for applying asset data (e.g., animation data for moving a virtual object over time), or many other possibilities.
[0030] An output device, such as a display or speaker, can present any or all aspects of the virtual environment to the user. For example, the virtual environment may include virtual objects (which may include representations of inanimate objects, people, animals, lights, etc.) that may be presented to the user. The processor may determine a view of the virtual environment (e.g., corresponding to a "camera" with origin coordinates, viewing axis, and frustum) and render a viewable scene of the virtual environment corresponding to that view on the display. Any suitable rendering technique may be used for this purpose. In some examples, the viewable scene may include only some virtual objects in the virtual environment and exclude certain other virtual objects. Similarly, the virtual environment may include audio aspects that may be presented to the user as one or more audio signals. For example, a virtual object in the virtual environment may generate sounds that originate from the object's location coordinates (e.g., a virtual character may speak or cause a sound effect), or the virtual environment may be associated with musical cues or ambient sounds that may or may not be associated with a particular location. The processor can determine audio signals corresponding to "listener" coordinates, e.g., audio signals corresponding to a composite of sounds in the virtual environment and that are mixed and processed to simulate an audio signal that would be heard by a listener at the listener coordinates, and present the audio signals to the user via one or more speakers.
[0031] Because the virtual environment exists only as a computational construct, the user cannot directly perceive the virtual environment using their normal senses. Instead, the user can only indirectly perceive the virtual environment as presented to the user, for example, by a display, a speaker, a tactile output device, etc. Similarly, the user cannot directly touch, manipulate, or otherwise interact with the virtual environment, but can provide input data via input devices or sensors to a processor that can use the device or sensor data to update the virtual environment. For example, a camera sensor can provide optical data indicating that the user is attempting to move an object in the virtual environment, and the processor can use that data to cause the object to respond accordingly in the virtual environment.
[0032] A mixed reality system can present a user with a mixed reality ("MR") environment that combines aspects of real and virtual environments, for example, using a see-through display and / or one or more speakers (which may be incorporated, for example, into a wearable head device). In some embodiments, the one or more speakers may be external to the head-mounted wearable unit. As used herein, an MR environment is a simultaneous representation of a real environment and a corresponding virtual environment. In some examples, the corresponding real and virtual environments share a single coordinate space, and in some examples, the real coordinate space and the one or more corresponding virtual coordinate spaces are interrelated by one or more transformation matrices (or other suitable representations). Thus, in some embodiments, a single coordinate (in some examples, together with a transformation matrix) can define a first location in the real environment and a second corresponding location in the virtual environment, and vice versa.
[0033] In an MR environment, a virtual object (e.g., in a virtual environment associated with the MR environment) may correspond to a real object (e.g., in a real environment associated with the MR environment). For example, if the real environment of the MR environment comprises a real street lamp (real object) at a location coordinate, the virtual environment of the MR environment may comprise a corresponding virtual street lamp (virtual object) at a corresponding location coordinate. As used herein, a real object in combination with its corresponding virtual object together constitutes a "mixed reality object." A virtual object need not perfectly match or align with a corresponding real object. In some examples, a virtual object may be a simplified version of a corresponding real object. For example, if the real environment includes a real street lamp, the corresponding virtual object may comprise a cylinder of approximately the same height and radius as the real street lamp (reflecting that a street lamp may be approximately cylindrical). Simplifying virtual objects in this manner may enable computational efficiency and simplify the calculations to be performed for such virtual objects. Furthermore, in some examples of MR environments, not all real objects in the real environment may be associated with a corresponding virtual object. Similarly, in some embodiments of an MR environment, not all virtual objects in the virtual environment may be associated with a corresponding real-world object. That is, some virtual objects may exist alone in the virtual environment of the MR environment without any real-world counterpart. In some embodiments, not all real-world objects may be associated with a corresponding real-world object.
[0034] In some examples, a virtual object may have characteristics that are sometimes significantly different from those of a corresponding real object. For example, a real environment in an MR environment may comprise a green, two-armed cactus, i.e., a thorny inanimate object, while a corresponding virtual object in the MR environment may have characteristics of a green, two-armed virtual character with human facial features and a surly attitude. In this example, a virtual object resembles its corresponding real object in some characteristics (color, number of arms) but differs from the real object in other characteristics (facial features, personality). In this way, virtual objects have the potential to represent real objects in creative, abstract, exaggerated, or fantastical ways, or to impart behaviors (e.g., human personality) to otherwise inanimate real objects. In some examples, a virtual object may be a purely fantastical creation with no real-world counterpart (e.g., a virtual monster in a virtual environment, perhaps in a location that corresponds to an empty space in the real environment).
[0035] Compared to a VR system that presents a virtual environment to a user while obscuring the real environment, a mixed reality system that presents an MR environment offers the advantage that the real environment remains perceptible while the virtual environment is presented. Thus, a user of a mixed reality system can experience and interact with a corresponding virtual environment using visual and audio cues associated with the real environment. As an example, as described above, a user of a VR system may struggle to perceive or interact with a virtual object displayed in a virtual environment because the user cannot directly perceive or interact with the virtual environment, whereas a user of an MR system may find it intuitive and natural to interact with the virtual object by seeing, hearing, and touching the corresponding real object in his or her real environment. This level of interactivity can increase a user's immersion, association, and engagement with the virtual environment. Similarly, by presenting a real environment and a virtual environment simultaneously, a mixed reality system can reduce the negative psychological sensations (e.g., cognitive dissonance) and negative physical sensations (e.g., motion sickness) associated with a VR system. Mixed reality systems further offer many possibilities for applications that can augment or modify our experience of the real world.
[0036] 19A illustrates an exemplary real environment 1900 in which a user 1910 uses a mixed reality system 1912. The mixed reality system 1912 may comprise a display (e.g., a see-through display), one or more speakers, and one or more sensors (e.g., a camera), for example, as described below. The illustrated real environment 1900 comprises a rectangular room 1904A in which the user 1910 is standing, and real objects 1922A (lamp), 1924A (table), 1926A (sofa), and 1928A (painting). The room 1904A further comprises a location coordinate 1906, which may be considered as the origin of the real environment 1900. 19A , an environment / world coordinate system 1908 (comprising an x-axis 1908X, a y-axis 1908Y, and a z-axis 1908Z) with its origin at a point 1906 (world coordinates) may define a coordinate space for the real environment 1900. In some embodiments, the origin 1906 of the environment / world coordinate system 1908 may correspond to where the mixed reality system 1912 is powered on. In some embodiments, the origin 1906 of the environment / world coordinate system 1908 may be reset between operations. In some examples, the user 1910 may be considered a real object in the real environment 1900, and similarly, the body parts (e.g., hands, feet) of the user 1910 may be considered real objects in the real environment 1900. In some examples, a user / listener / head coordinate system 1914 (comprising an x-axis 1914X, a y-axis 1914Y, and a z-axis 1914Z) with its origin at a point 1915 (e.g., user / listener / head coordinates) may define a coordinate space for the user / listener / head in which the mixed reality system 1912 is located. The origin 1915 of the user / listener / head coordinate system 1914 may be defined relative to one or more components of the mixed reality system 1912. For example, the origin 1915 of the user / listener / head coordinate system 1914 may be defined relative to a display of the mixed reality system 1912, such as during an initial calibration of the mixed reality system 1912. A matrix (which may include a translation matrix and a quaternion matrix or other rotation matrix), or other suitable representation, may characterize the transformation between the user / listener / head coordinate system 1914 space and the environment / world coordinate system 1908 space.In some embodiments, left ear coordinates 1916 and right ear coordinates 1917 may be defined relative to an origin 1915 of a user / listener / head coordinate system 1914. Matrices (which may include translation matrices and quaternion or other rotation matrices), or other suitable representations, may characterize the transformation between the left ear coordinates 1916 and right ear coordinates 1917 and the user / listener / head coordinate system 1914 space. The user / listener / head coordinate system 1914 may simplify the representation of location relative to the user's head or head mounted device, for example, relative to the environment / world coordinate system 1908. Using simultaneous localization and mapping (SLAM), visual odometry, or other techniques, the transformation between the user coordinate system 1914 and the environment coordinate system 1908 may be determined and updated in real time.
[0037] 19B illustrates an example virtual environment 1930 corresponding to the real environment 1900. The shown virtual environment 1930 includes a virtual rectangular room 1904B corresponding to the real rectangular room 1904A, a virtual object 1922B corresponding to the real object 1922A, a virtual object 1924B corresponding to the real object 1924A, and a virtual object 1926B corresponding to the real object 1926A. Metadata associated with the virtual objects 1922B, 1924B, 1926B may include information derived from the corresponding real objects 1922A, 1924A, 1926A. The virtual environment 1930 additionally includes a virtual monster 1932 that does not correspond to any real object in the real environment 1900. A real object 1928A in the real environment 1900 does not correspond to any virtual object in the virtual environment 1930. A persistent coordinate system 1933 (comprising axes 1933X, y-axis 1933Y, and z-axis 1933Z) with its origin at point 1934 (persistent coordinates) may define a coordinate space for the virtual content. The origin 1934 of the persistent coordinate system 1933 may be defined relative to / with respect to one or more real objects, such as real object 1926A. A matrix (which may include translation matrices and quaternion matrices or other rotation matrices), or other suitable representation, may characterize the transformation between the persistent coordinate system 1933 space and the environment / world coordinate system 1908 space. In some embodiments, virtual objects 1922B, 1924B, 1926B, and 1932 may each have their own persistent coordinate point relative to the origin 1934 of the persistent coordinate system 1933. In some embodiments, there may be multiple persistent coordinate systems, and virtual objects 1922B, 1924B, 1926B, and 1932 may each have their own persistent coordinate points relative to one or more persistent coordinate systems.
[0038] With reference to Figures 19A and 19B, an environment / world coordinate system 1908 defines a shared coordinate space for both the real environment 1900 and the virtual environment 1930. In the illustrated embodiment, the coordinate space has its origin at point 1906. Furthermore, the coordinate space is defined by the same three orthogonal axes (1908X, 1908Y, 1908Z). Thus, a first location in the real environment 1900 and a second corresponding location in the virtual environment 1930 can be described with respect to the same coordinate space. This simplifies identifying and displaying corresponding locations in the real and virtual environments because the same coordinates can be used to identify both locations. However, in some embodiments, the corresponding real and virtual environments need not use a shared coordinate space. For example, in some embodiments (not shown), matrices (which may include translation matrices and quaternion matrices or other rotation matrices), or other suitable representations, can characterize the transformation between the real environment coordinate space and the virtual environment coordinate space.
[0039] 19C illustrates an example MR environment 1950 that simultaneously presents aspects of a real environment 1900 and a virtual environment 1930 to a user 1910 via a mixed reality system 1912. In the example shown, the MR environment 1950 simultaneously presents to the user 1910 real objects 1922A, 1924A, 1926A, and 1928A from the real environment 1900 (e.g., via a see-through portion of a display of the mixed reality system 1912) and virtual objects 1922B, 1924B, 1926B, and 1932 from the virtual environment 1930 (e.g., via an active display portion of a display of the mixed reality system 1912). As above, the origin 1906 acts as the origin for a coordinate space corresponding to the MR environment 1950, and the coordinate system 1908 defines the x-, y-, and z-axes for the coordinate space.
[0040] In the example shown, the mixed reality objects comprise corresponding pairs of real and virtual objects (i.e., 1922A / 1922B, 1924A / 1924B, 1926A / 1926B) that occupy corresponding locations in coordinate space 1908. In some examples, both real and virtual objects may be visible to user 1910 at the same time. This may be desirable, for example, in instances where a virtual object presents information designed to augment the view of the corresponding real object (such as in a museum application where the virtual object presents a missing portion of an ancient damaged sculpture). In some examples, the virtual objects (1922B, 1924B, and / or 1926B) may be displayed so as to occlude (e.g., via active pixelated occlusion using a pixelated occlusion shutter) the corresponding real objects (1922A, 1924A, and / or 1926A). This may be desirable, for example, in cases where a virtual object acts as a visual substitute for a corresponding real object (such as an interactive reading application, where inanimate real objects become "living" characters).
[0041] In some examples, real objects (e.g., 1922A, 1924A, 1926A) may be associated with virtual content or helper data that may not necessarily constitute virtual objects. The virtual content or helper data may facilitate processing or handling of the virtual objects in a mixed reality environment. For example, such virtual content may include a two-dimensional representation of the corresponding real object, a custom asset type associated with the corresponding real object, or statistical data associated with the corresponding real object. This information may enable or facilitate computations involving the real object without incurring unnecessary computational overhead.
[0042] In some examples, the presentation described above may also incorporate audio aspects. For example, in the MR environment 1950, the virtual monster 1932 may be associated with one or more audio signals, such as footstep effects, that are generated as the monster walks around the MR environment 1950. As described further below, a processor of the mixed reality system 1912 may calculate an audio signal that corresponds to a mixed and processed composite of all such sounds within the MR environment 1950, and present the audio signal to the user 1910 via one or more speakers included in the mixed reality system 1912 and / or one or more external speakers.
[0043] (Mixed reality object sharing)
[0044] A 3D digital model of an object may be displayed as a virtual object to one or more real or digital people in an AR / VR / MR (hereafter referred to as MR for simplicity) environment. For example, an automotive engineer may want to share a new car design with a colleague at his weekly team meeting. Traditionally, design sharing may be accomplished by providing several different views of the design so that the viewer can imagine the 3D object. In this case, the automotive engineer may print out perspective drawings to distribute to the colleague during the meeting. This may be useful for simple designs, but if the 3D object has a complex shape or design, it may be difficult for the viewer to piece together the 3D object in his or her mind. Alternatively or in addition, a traditional method of sharing a 3D design or model may involve creating a physical prototype. In this example, the automotive engineer would need to spend time and money creating a physical prototype. A physical prototype may make it easier to understand the designer's intent, but may only show a basic version of the design (a viewer can only see the outside of the design but cannot, for example, open it up to view the internal components).
[0045] Embodiments of the disclosed system and method may provide improved 3D model sharing between computing systems. Continuing with the above example, the present invention may enable an automotive engineer to create a new digital 3D model of a car and then quickly and efficiently share the model with colleagues during a meeting. The present invention may also enable fast and easy unplanned sharing. For example, the automotive engineer's manager may want to share a 3D design that the manager designed in advance to collaborate with the automotive engineer. In some embodiments, the manager can access the old 3D car design and share it with everyone attending the meeting without requiring people to download the design or click a link to access it.
[0046] Typically, for example, in games, 3D models (alternatively called 3D assets or 3D content) are preloaded into the game application so that when the user launches the game, all of the 3D assets that will be viewed by the user are already on the device. When updates are needed, the application will add new content offline. For example, a game may be patched when the game is not running, and the next time the application is opened, the new content is installed and ready for use. This system of sharing new 3D models with two or more computing systems is convenient for frequent planned updates, but is impractical when new 3D models are frequently or unexpectedly required.
[0047] An MR game or application may be collaborative, with multiple people contributing towards a common goal, such as an entire group of students learning about the anatomy of a frog at school. If a teacher is trying to explain a concept and the student does not understand, it may be desirable for the teacher to display a virtual 3D model of a frog to the student so that they can see what the teacher is explaining. A conventional system would require the student to download the 3D model or to quit and re-enter the application so that the teacher can push a new frog model update to the student. In addition, such a method may require a large amount of disk storage to house all of the 3D asset data. An alternative method of accessing the 3D model without using disk space may be desirable.
[0048] These problems may be solved by a method and system for streaming data over a network between a server and one or more clients. The server may have access to one or more 3D models and may download and launch a 3D model sharing server application. The 3D model sharing application may break the model down into constituent data parts, compress, package and send the constituent data parts to clients on the network. The clients may receive the constituent parts during a 3D model sharing client application runtime, reconstruct the 3D model and view the 3D model. In some embodiments, the runtime may be when a game or application loop is running, when a screen is displaying application content, and / or when a computing system is working with an application framework. Any number of clients may be part of the network. Any number of 3D models may be sent to the clients. The two or more clients may be one person with two or more devices (e.g., computing systems) or two or more people each with one device.
[0049] An advantage of the present application is that it enables 3D content sharing during runtime.
[0050] The term "host" may be used synonymously with server. The terms "3D model," "3D asset," "3D content," and "3D model" may be used synonymously. (Example of a 3D display for a wearable system)
[0051] A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present 2D or 3D virtual images to a user. The images may be still images, frames of video, or videos, in combination or the like. At least a portion of the wearable system can be implemented on a wearable device, which may present a VR, AR, or MR environment, alone or in combination, for user interaction. A wearable device can be used synonymously with an AR device (ARD). Additionally, for purposes of this disclosure, the term "AR" is used synonymously with the term "MR."
[0052] Figure 1 depicts an illustration of a mixed reality scenario with certain virtual reality objects and certain physical objects viewed by a person. In Figure 1, an MR scene 100 is depicted in which a user of the MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the MR technology also perceives that they "see" a robotic figure 130 standing on the real-world platform 120, and a flying cartoon-like avatar character 140 that appears to be an anthropomorphic bumblebee, although these elements do not exist in the real world.
[0053] In order for a 3D display to produce a true depth sensation, and more specifically, a simulated sensation of surface depth, it may be desirable to generate, for each point in the display's field of view, an accommodation response that corresponds to that point's virtual depth. If the accommodation response to a display point does not correspond to that point's virtual depth as determined by the binocular depth cues of convergence and stereopsis, the human eye may experience accommodation conflicts, resulting in unstable imaging, deleterious eye strain, headaches, and, in the absence of accommodative information, a near-complete lack of surface depth.
[0054] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images may be different for each depth plane (e.g., providing slightly different presentations of a scene or object) and may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on the ocular accommodation required to focus on different image features of a scene located on different depth planes, or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues can provide a believable perception of depth.
[0055] FIG. 2 illustrates an example of a wearable system 200 that may be configured to provide an AR / VR / MR scene. The wearable system 200 may also be referred to as an AR system 200. The wearable system 200 may include a display 220 and various mechanical and electronic modules and systems to support the functionality of the display 220. The display 220 may be coupled to a frame 230, which is wearable by a user, wearer, or viewer 210. The display 220 may be positioned in front of the eyes of the user 210. The display 220 may present AR / VR / MR content to the user. The display 220 may comprise a head-mounted display (HMD) that is worn on the user's head.
[0056] In some embodiments, a speaker 240 may be coupled to the frame 230 and positioned adjacent the user's ear canal (in some embodiments, another speaker, not shown, may be positioned adjacent the user's other ear canal to provide stereo / shapeable sound control). The display 220 may include an audio sensor (e.g., a microphone) 232 for detecting an audio stream from the environment and capturing ambient sounds. In some embodiments, one or more other audio sensors, not shown, may be positioned to provide stereo sound reception. Stereo sound reception may be used to determine the location of a sound source. The wearable system 200 may perform voice or speech recognition on the audio stream.
[0057] The wearable system 200 may include an outwardly facing imaging system 464 (shown in FIG. 4 ) that observes the world in the user's surrounding environment. The wearable system 200 may also include an inwardly facing imaging system 462 (shown in FIG. 4 ) that may track the user's eye movements. The inwardly facing imaging system may track either one eye's movements or both eyes' movements. The inwardly facing imaging system 462 may be mounted to the frame 230 and may be in electrical communication with the processing module 260 or 270 that may process image information obtained by the inwardly facing imaging system and determine, for example, the pupil diameter or orientation of the user's 210 eyes, eye movements, or eye posture. The inwardly facing imaging system 462 may include one or more cameras. For example, at least one camera may be used to image each eye. Images obtained by the cameras may be used to determine pupil size or eye posture for each eye separately, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye.
[0058] As an example, the wearable system 200 can obtain an image of the user's posture using an outward facing imaging system 464 or an inward facing imaging system 462. The image may be a still image, a frame of a video, or a video.
[0059] The display 220 is operably coupled, such as by wired leads or wireless connectivity, to a local data processing module 260, which may be mounted in a variety of configurations (250), such as fixedly attached to the frame 230, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack-type configuration, in a belt-coupled configuration).
[0060] The local processing and data module 260 may comprise a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data may include (a) data captured from sensors (e.g., that may be operatively coupled to the frame 230 or otherwise attached to the user 210), such as image capture devices (e.g., cameras in an inward-facing and / or outward-facing imaging system), audio sensors (e.g., microphones), inertial measurement units (IMUs), accelerometers, compasses, global positioning system (GPS) units, wireless devices, or gyroscopes, or (b) data obtained or processed using the remote processing module 270 or remote data repository 280, possibly for passing to the display 220 after processing or retrieval. The local processing and data module 260 may be operatively coupled to a remote processing module 270 or a remote data repository 280 by a communications link 262 or 264, such as via a wired or wireless communication link, such that these remote modules may be available as resources to the local processing and data module 260. In addition, the remote processing module 280 and the remote data repository 280 may be operatively coupled to each other.
[0061] In some embodiments, the remote processing module 270 may comprise one or more processors configured to analyze and process the data or image information. In some embodiments, the remote data repository 280 may comprise a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data may be stored and all calculations may be performed in the local processing and data module, allowing for fully autonomous use from the remote module. (Example Components of a Wearable System)
[0062] FIG. 3 diagrammatically illustrates example components of a wearable system. FIG. 3 shows a wearable system 200 that may include a display 220 and a frame 230. A blowup 202 diagrammatically illustrates various components of the wearable system 200. In some implementations, one or more of the components illustrated in FIG. 3 may be part of the display 220. The various components, alone or in combination, may collect various data associated with a user of the wearable system 200 or the user's environment (e.g., audio or visual data, etc.). It should be understood that other embodiments may have additional or fewer components depending on the application for which the wearable system is used. However, FIG. 3 provides a basic idea of some of the various components and types of data that may be collected, analyzed, and stored through the wearable system.
[0063] FIG. 3 illustrates an exemplary wearable system 200 that may include a display 220. The display 220 may include a display lens 226 that may be mounted to a housing or frame 230 that corresponds to the user's head or frame 230. The display lens 226 may include one or more transparent mirrors or diffractive optical elements positioned by the housing 230 in front of the user's eyes 302, 304 and may be configured to direct the projected light 338 into the eyes 302, 304 and facilitate beam shaping while also allowing transmission of at least some light from the local environment. The wavefront of the projected light beam 338 may diverge to match the desired focal length of the projected light. As illustrated, two wide-field machine vision cameras 316 (also referred to as world cameras) are coupled to the housing 230 and may image the environment around the user. These cameras 316 may be dual capture visible / non-visible (e.g., infrared) light cameras. The camera 316 may be part of the outward facing imaging system 464 shown in Figure 4. Images acquired by the world camera 316 may be processed by the pose processor 336. For example, the pose processor 336 may implement one or more object recognizers 708 (e.g., shown in Figure 7) to identify a pose of the user or another person in the user's environment, or to identify physical objects in the user's environment.
[0064] Continuing with reference to FIG. 3, a pair of light projector modules with display optics and lenses configured to direct light 338 into the eyes 302, 304 are shown. The depicted diagram also shows two miniature infrared cameras 324 paired with infrared lights (such as light emitting diodes "LEDs") that can be configured to track the user's eyes 302, 304 and support rendering and user input. The cameras 324 may be part of an inwardly facing imaging system 462 shown in FIG. 4. The wearable system 200 may further feature a sensor assembly 339 with X, Y, and Z axis accelerometer capabilities and magnetic compass and X, Y, and Z axis gyroscope capabilities, preferably providing data at a relatively high frequency such as 200 Hz. The sensor assembly 339 may be part of an IMU, described with reference to FIG. 2A. The depicted system 200 may also include a head pose processor 336, such as an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or ARM processor (Advanced Reduced Instruction Set Machine), that may be configured to calculate real-time or near real-time user head pose from the wide field of view image information output from the capture device 316. The head pose processor 336 may be a hardware processor and may be implemented as part of the local processing and data module 260 shown in FIG. 2A.
[0065] The wearable system may also include one or more depth sensors 234. The depth sensors 234 may be configured to measure the distance between objects in the environment and the wearable device. The depth sensors 234 may include a laser scanner (e.g., lidar), an ultrasonic depth sensor, or a depth-sensing camera. In some implementations where the camera 316 has depth-sensing capabilities, the camera 316 may also be considered a depth sensor 234.
[0066] Also shown is a processor 332, which may be configured to perform digital or analog processing and derive attitude from gyroscope, compass, or accelerometer data from the sensor assembly 339. The processor 332 may be part of the local processing and data module 260, shown in FIG. 2. The wearable system 200 as shown in FIG. 3 may also include a positioning system, such as, for example, a GPS 337 (Global Positioning System), to aid in attitude and positioning analysis. In addition, the GPS may further provide remote-based (e.g., cloud-based) information about the user's environment. This information may be used to recognize objects or information within the user's environment.
[0067] The wearable system may combine data obtained by the GPS 337 and a remote computing system (e.g., the remote processing module 270, another user's ARD, etc.), which may provide more information about the user's environment. As an example, the wearable system can determine the user's location based on the GPS data and retrieve (e.g., by communicating with the remote processing module 270) a world map including virtual objects associated with the user's location. As another example, the wearable system 200 can monitor the environment using the world camera 316 (which may be part of the outward-facing imaging system 464 shown in FIG. 4). Based on the images obtained by the world camera 316, the wearable system 200 can detect objects in the environment (e.g., by using one or more object recognizers 708 shown in FIG. 7). The wearable system can further interpret characters using data obtained by the GPS 337.
[0068] The wearable system 200 may also include a rendering engine 334, which may be configured to provide rendering information that is local to the user for the user's view of the world and facilitate the operation of the scanner and imaging into the user's eye. The rendering engine 334 may be implemented by a hardware processor (e.g., a central processing unit or a graphics processing unit, etc.). In some embodiments, the rendering engine is part of the local processing and data module 260. The rendering engine 334 may be communicatively coupled (e.g., via wired or wireless links) to other components of the wearable system 200. For example, the rendering engine 334 may be coupled to the eye camera 324 via communication link 274 and to the projection subsystem 318 (which may project light into the user's eyes 302, 304 via a scanning laser array in a manner similar to a retinal scanning display) via communication link 272. The rendering engine 334 may also communicate with other processing units, such as, for example, the sensor pose processor 332 and the image pose processor 336, via links 276 and 294, respectively.
[0069] A camera 324 (e.g., a small infrared camera) may be utilized to track eye pose and support rendering and user input. Some example eye poses may include where the user is looking or the depth at which they are focused (which may be estimated using eye convergence and divergence). A GPS 337, gyroscope, compass, and accelerometer 339 may be utilized to provide rough or fast pose estimation. One or more of the cameras 316, along with data from associated cloud computing resources, can obtain images and poses that may be utilized to map the local environment and share the user view with others.
[0070] The exemplary components depicted in FIG. 3 are for illustrative purposes only. Multiple sensors and other functional modules are shown together for ease of illustration and description. Some embodiments may include only one or a subset of these sensors or modules. Furthermore, the locations of these components are not limited to the locations depicted in FIG. 3. Some components may be mounted on or housed within other components, such as belt-mounted components, handheld components, or helmet components. As an example, the image pose processor 336, the sensor pose processor 332, and the rendering engine 334 may be located within a beltpack and configured to communicate with other components of the wearable system via wireless communication, such as ultra-wideband, Wi-Fi, Bluetooth, or via wired communication. The depicted housing 230 is preferably head-mountable and wearable by a user. However, some components of the wearable system 200 may be worn on other parts of the user's body. For example, the speaker 240 may be inserted into the user's ear to provide sound to the user.
[0071] With regard to projecting light 338 into the user's eyes 302, 304, in some embodiments, the camera 324 may be utilized to measure where the center of the user's eyes geometrically converges, which generally coincides with the location of the eye's focus or "depth of focus." The three-dimensional surface of all points at which the eyes converge may be referred to as the "monocular locus." The focal length may take on a finite number of depths or may vary infinitely. Light projected from the vergence distance appears focused on the subject's eyes 302, 304, while light in front of or behind the vergence distance is blurred.
[0072] The human visual system is complex and providing a realistic perception of depth is difficult. A viewer of an object may perceive the object as three-dimensional due to a combination of vergence and accommodation. The vergence of the two eyes relative to one another (e.g., the rotational movement of the pupils toward or away from one another to converge the gaze of the eyes and fixate on an object) is closely coupled to the focusing of the eye's lenses (or "accommodation"). Under normal conditions, a change in the focus of the eye's lenses or accommodation of the eye to change focus from one object to another at a different distance will automatically produce a corresponding change in vergence to the same distance, a relationship known as the "accommodation-vergence reflex." Similarly, a change in vergence will induce a corresponding change in accommodation under normal conditions. A display system that provides a better match between accommodation and vergence may produce a more realistic and comfortable simulation of a three-dimensional image.
[0073] Furthermore, spatially coherent light with a beam diameter of less than about 0.7 millimeters can be correctly resolved by the human eye regardless of where the eye focuses. Thus, to create the illusion of proper depth of focus, the vergence and divergence of the eye may be tracked using the camera 324, and the rendering engine 334 and projection subsystem 318 may be utilized to render all objects on or near the vergence and in focus, and all other objects at various degrees of variable resolution. Preferably, the system 220 renders to the user at a frame rate of about 60 frames per second or greater. As explained above, preferably, the camera 324 may be utilized for eye tracking, and software may be configured to take not only the vergence and divergence geometry, but also focus location cues to serve as user input. Preferably, such a display system is configured with a suitable brightness and contrast for daytime or nighttime use.
[0074] In some embodiments, the display system preferably has a latency of less than about 20 milliseconds for visual object alignment, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 arc minute, which is believed to be approximately the limit of the human eye, without being limited by theory. The display system 220 may be integrated with a localization system that may involve a GPS element, optical tracking, a compass, an accelerometer, or other data sources to aid in position and attitude determination. The localization information may be utilized to facilitate accurate rendering within the user's view of the relevant world (e.g., such information would facilitate knowing where the glasses are relative to the real world).
[0075] In some embodiments, the wearable system 200 is configured to display one or more virtual images based on the accommodation of the user's eyes. Unlike traditional 3D display approaches that force the user to focus where the image is projected, in some embodiments, the wearable system can be configured to automatically vary the focus of the projected virtual content to allow for a more comfortable viewing of one or more images presented to the user. For example, if the user's eyes have a current focus of 1m, the image may be projected to match the user's focus. If the user shifts the focus to 3m, the image can be projected to match the new focus. Thus, rather than forcing a predetermined focus on the user, the wearable system 200 of some embodiments can allow the user's eyes to function in a more natural manner.
[0076] Such a wearable system 200 may eliminate or reduce the incidence of eye strain, headaches, and other physiological symptoms typically observed with virtual reality devices. To accomplish this, various embodiments of the wearable system 200 may be configured to project virtual images at various focal distances through one or more variable focus elements (VFEs). In one or more embodiments, 3D perception may be achieved through a multi-planar focus system that projects images onto a fixed focal plane away from the user. Other embodiments may employ a variable planar focus, where the focal plane may move back and forth in the z-direction to match the user's current state of focus.
[0077] In both the multi-plane focus system and the variable plane focus system, the wearable system 200 may employ eye tracking to determine the vergence-divergence movement of the user's eyes, determine the user's current focus, and project the virtual image at the determined focus. In other embodiments, the wearable system 200 may include a light modulator that variably projects a light beam of varying focus in a raster pattern across the retina through a fiber scanner or other light generating source. Thus, the ability of the wearable system 200 display to project images at varying focal distances may be used to not only facilitate accommodation for the user to view objects in 3D, but also to compensate for the user's ocular anomalies. In some other embodiments, the spatial light modulator may project the image to the user through various optical components. For example, as described further below, the spatial light modulator may project the image onto one or more waveguides, which then transmit the image to the user. (Waveguide Stack Assembly)
[0078] FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. The wearable system 400 can include a stack of waveguides or a stacked waveguide assembly 480 that can be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, the wearable system 400 can correspond to the wearable system 200 of FIG. 2, of which FIG. 4A diagrammatically illustrates some portions of the wearable system 200 in more detail. For example, in some embodiments, the waveguide assembly 480 can be integrated into the display 220 of FIG. 2.
[0079] 4, the waveguide assembly 480 may also include a number of features 458, 456, 454, 452 between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers or structures to form air gaps).
[0080] The waveguides 432b, 434b, 436b, 438b, 440b, or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye with various levels of wavefront curvature or light beam divergence. Each waveguide level may be associated with a particular depth plane and configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 may be utilized to inject image information into the waveguides 440b, 438b, 436b, 434b, 432b, respectively, which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light may exit output surfaces of image injection devices 420, 422, 424, 426, 428 and be injected into corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be injected into each waveguide to output an entire sample exit pupil beam field that may be directed towards the eye 410 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide.
[0081] In some embodiments, image input devices 420, 422, 424, 426, 428 may be discrete displays that each produce image information for input into a corresponding waveguide 440b, 438b, 436b, 434b, 432b, respectively. In some other embodiments, image input devices 420, 422, 424, 426, 428 may be the output of a single multiplexed display that may send image information to each of image input devices 420, 422, 424, 426, 428, for example, via one or more optical conduits (such as fiber optic cables).
[0082] A controller 460 can control the operation of the display and image input devices 420, 422, 424, 426, 428. The controller 460 can include programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provision of image information to the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the controller 460 can be a single integrated device or a distributed system connected by a wired or wireless communication channel. The controller 460 can be part of the processing module 260 or 270 (illustrated in FIG. 2) in some embodiments.
[0083] The waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between the major top and bottom surfaces. In the illustrated configuration, the waveguides 440b, 438b, 436b, 434b, 432b may each include optical elements 440a, 438a, 436a, 434a, 432a that may be configured to outcouple light from the waveguide by diffracting or otherwise redirecting light propagating within each individual waveguide. The outcoupled light can be output, for example, by a waveguide where the light propagating in the waveguide strikes a light redirecting element, such as a diffraction grating. The optical elements (440a, 438a, 436a, 434a, 432a) can be, for example, reflective or diffractive optical features. Although shown disposed on the bottom major surfaces of the waveguides 440b, 438b, 436b, 434b, 432b for ease of explanation and clarity of drawing, in some embodiments the optical elements 440a, 438a, 436a, 434a, 432a can be disposed on the top or bottom major surfaces or directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the optical elements 440a, 438a, 436a, 434a, 432a may be formed in a layer of material that is attached to a transparent substrate and forms the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be a monolithic piece of material and the optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within that piece of material.
[0084] Continuing with reference to FIG. 4, as discussed herein, in some embodiments, each waveguide 440b, 438b, 436b, 434b, 432b can be configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is launched into such waveguide 432b. The collimated light may represent an optical infinity focal plane. The next upper waveguide 434b may be configured to send collimated light that passes through a first lens 452 (e.g., a negative lens) before it can reach the eye 410. The first lens 452 may be configured to create a slight convex wavefront curvature such that the eye / brain interprets the light originating from the next upper waveguide 434b as originating from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third upper waveguide 436b may have its output light pass through both the first lens 452 and the second lens 454 before reaching the eye 410. The combined refractive power of the first and second lenses 452 and 454 may be configured to create another incremental amount of wavefront curvature such that the eye / brain interprets the light originating from the third upper waveguide 436b as originating from a second focal plane that is closer inward from optical infinity towards the person than was the light from the next upper waveguide 434b.
[0085] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) can be similarly configured, with the highest waveguide 440b in the stack sending its output through all of the lenses between it and the eye for an aggregate focal force that represents the focal plane closest to the person. To compensate for the stack of lenses 458, 456, 454, 452 when viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be placed on top of the stack to compensate for the aggregate force of the lens stacks 458, 456, 454, 452 below. Such a configuration can provide as many perceived focal planes as there are waveguide / lens pairs available. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.
[0086] Continuing with reference to FIG. 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to redirect light from their respective waveguides and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes may have different configurations of light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be solid or surface features that may be configured to output light at a specific angle. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be solid holograms, surface holograms, and / or diffraction gratings.
[0087] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a can be diffractive features or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency so that only a portion of the light in the beam is deflected toward the eye 410 at each intersection of the DOE, while the remainder continues traveling through the waveguide via total internal reflection. The light carrying the image information can thus be split into several related exit beams that exit the waveguide at multiple locations, resulting in a very uniform pattern of exit emission toward the eye 304 for this particular collimated beam bouncing within the waveguide.
[0088] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal in which the microdroplets can comprise a diffractive pattern within a host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).
[0089] In some embodiments, the number and distribution of depth planes or depths of field may be dynamically varied based on the size or orientation of the pupil of the viewer's eye. The depth of field may vary inversely with the size of the viewer's pupil. As a result, as the size of the viewer's eye pupil decreases, the depth of field increases so that one plane that is indistinguishable because its location exceeds the focal depth of the eye may become distinguishable and appear more in focus with a reduction in pupil size and a corresponding increase in depth of field. Similarly, the number of spaced depth planes used to present different images to the viewer may be reduced with a reduced pupil size. For example, the viewer may not be able to clearly perceive the details of both the first and second depth planes at one pupil size without adjusting the accommodation of the eye from one depth plane to the other. However, these two depth planes may be simultaneously well-focused for the user at another pupil size without changing accommodation.
[0090] In some embodiments, the display system may vary the number of waveguides that receive image information based on a determination of a pupil size or orientation, or in response to receiving an electrical signal indicative of a particular pupil size or orientation. For example, if the user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 460 (which may be an embodiment of the local processing and data module 260) may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments in which the DOE for a waveguide is switchable between an on and off state, the DOE may be switched to the off state when the waveguide receives image information.
[0091] In some embodiments, it may be desirable to have the exit beam meet the condition of having a diameter less than the diameter of the viewer's eye. However, meeting this condition may be difficult in light of the variability in the size of the viewer's pupil. In some embodiments, this condition is met over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the exit beam may also decrease. In some embodiments, the exit beam size may be varied using a variable aperture.
[0092] The wearable system 400 may include an outwardly facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes also referred to as the FOV camera. The world camera FOV may or may not be the same as the viewer 210 FOV, which encompasses the portion of the world 470 that the viewer 210 perceives at a given time. For example, in some situations, the world camera FOV may be larger than the viewer 210's field of view of the wearable system 400. The entire area available for viewing or imaging by the viewer may be referred to as the ocular field of view (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400, since the wearer can move their body, head, or eyes and perceive virtually any direction in space. In other contexts, the wearer's movements may be more constrained and the wearer's FOR may correspondingly subtend a smaller solid angle. Images acquired from the outwardly facing imaging system 464 can be used to track gestures (e.g., hand or finger gestures) made by the user, detect objects in the world 470 in front of the user, etc.
[0093] The wearable system 400 can include an audio sensor 232, e.g., a microphone, to capture ambient sound. As described above, in some embodiments, one or more other audio sensors can be positioned to provide stereo sound reception useful in determining the location of the speech source. The audio sensor 232 can comprise, as another example, a directional microphone, which can provide such useful directional information regarding where the audio source is located. The wearable system 400 can use information from both the outwardly facing imaging system 464 and the audio sensor 230 in locating the speech source, or to determine the active speaker at a particular moment, etc. For example, the wearable system 400 can use voice recognition, alone or in combination with a reflected image of the speaker (e.g., as seen in a mirror), to determine the identity of the speaker. As another example, the wearable system 400 can determine the speaker's location in the environment based on sounds obtained from a directional microphone. The wearable system 400 can use speech recognition algorithms to analyze sounds originating from the speaker's location, determine the content of the speech, and use voice recognition techniques to determine the speaker's identity (e.g., name or other demographic information).
[0094] The wearable system 400 may also include an inwardly facing imaging system 466 (e.g., a digital camera) that observes user movements such as eye and face movements. The inwardly facing imaging system 466 may be used to capture images of the eye 410 and determine the size and / or orientation of the pupil of the eye 304. The inwardly facing imaging system 466 may be used to obtain images for use in determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera may be utilized for each eye independently to separately determine the pupil size or eye pose of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter or orientation of only a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. Images acquired by inwardly facing imaging system 466 may be analyzed to determine the user's eye posture or mood, which may be used by wearable system 400 to determine audio or visual content to be presented to the user. Wearable system 400 may also determine head pose (e.g., head position or head orientation) using sensors such as an IMU, accelerometer, gyroscope, etc.
[0095] The wearable system 400 may include a user input device 466 through which a user may input commands into the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a tactile device, a totem (e.g., serving as a virtual user input device), and the like. A multi-DOF controller may sense user input in some or all of the possible translations (e.g., left / right, forward / backward, or up / down) or rotations (e.g., yaw, pitch, or roll) of the controller. A multi-DOF controller that supports translations may be referred to as 3DOF, while a multi-DOF controller that supports translations and rotations may be referred to as 6DOF. In some cases, a user may use a finger (e.g., thumb) to press or swipe on the touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 in wired or wireless communication. (Other components of the wearable system)
[0096] In many implementations, the wearable system may include other components in addition to or as an alternative to the components of the wearable system described above. The wearable system may include, for example, one or more tactile devices or components. The tactile devices or components may be operable to provide a haptic sensation to the user. For example, the tactile device or component may provide a haptic sensation of pressure or texture upon touching the virtual content (e.g., a virtual object, virtual tool, other virtual structure). The haptics may replicate the haptic sensation of a physical object that the virtual object represents, or may replicate the sensation of an imaginary object or character (e.g., a dragon) that the virtual content represents. In some implementations, the tactile device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the tactile device or component may be held by the user.
[0097] A wearable system may include one or more physical objects that can be manipulated by, for example, a user to allow input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, e.g., pieces of metal or plastic, walls, the surface of a table. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface to appear to the user as being on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad to appear on the surface of a thin rectangular plate of aluminum that serves as the totem. The rectangular plate itself may not have any physical keys or trackpads or sensors. However, the wearable system may detect user manipulation or interaction or touch with the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. User input device 466 (shown in FIG. 4) may be an embodiment of the totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. A user may use the totem alone or in combination with posture to interact with the wearable system or other users. (Example process of user interaction with a wearable system)
[0098] 5 is a process flow diagram of an example method 500 for interacting with a virtual user interface. Method 500 may be implemented by a wearable system described herein. An embodiment of method 500 may be used by a wearable system to detect a person or document within the FOV of the wearable system.
[0099] In block 510, the wearable system may identify a particular UI. The type of UI may be predetermined by the user. The wearable system may identify that a particular UI needs to be captured based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). The UI may be specific to a security scenario where a wearer of the system is observing a user presenting a document to the wearer (e.g., at a traveler checkpoint). In block 520, the wearable system may generate data for a virtual UI. For example, data associated with the UI's boundaries, general structure, shape, etc. may be generated. In addition, the wearable system may determine map coordinates of the user's physical location so that the wearable system may display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine coordinates of the user's physical standing position, head pose, or eye pose so that a ring UI may be displayed around the user or a planar UI may be displayed on a wall or in front of the user. In the security context described herein, the UI may be displayed as if it were surrounding the traveler presenting a document to the wearer of the system, so that the wearer may easily view the UI while viewing the traveler and the traveler's document. If the UI is hand-centered, map coordinates of the user's hand may be determined. These map points may be derived through data received through the FOV camera, sensory input, or any other type of collected data.
[0100] In block 530, the wearable system may send data from the cloud to the display, or data may be sent from a local database to the display component. In block 540, a UI is displayed to the user based on the sent data. For example, a light field display can project the virtual UI into one or both of the user's eyes. Once the virtual UI is created, the wearable system may simply wait for commands from the user in block 550 and generate more virtual content on the virtual UI. For example, the UI may be a body-centered ring around the user's body or the body of a person (e.g., a traveler) in the user's environment. The wearable system may then wait for a command (gesture, head or eye movement, voice command, input from a user input device, etc.) and if it is recognized (block 560), the virtual content associated with the command may be displayed to the user (block 570). (An example of avatar rendering in mixed reality)
[0101] The wearable system may employ various mapping-related techniques to achieve a high depth of field in the rendered light field. When mapping a virtual world, it is advantageous to capture all features and points in the real world and accurately depict the virtual objects in relation to the real world. To this end, FOV images captured from a user of the wearable system can be added to the world model by including new photos that convey information about various points and features of the real world. For example, the wearable system can collect a set of map points (such as 2D or 3D points), find new map points, and render a more accurate version of the world model. The world model of the first user can be communicated (e.g., via a network such as a cloud network) to a second user so that the second user can experience the world surrounding the first user.
[0102] FIG. 6A is a block diagram of another example of a wearable system that may include an avatar processing and rendering system 690 in a mixed reality environment. The wearable system 600 may be part of the wearable system 200 shown in FIG. 2. In this example, the wearable system 600 may include a map 620 that may include at least a portion of the data in the map database 710 (shown in FIG. 7). The map may reside in part locally on the wearable system and in part in a networked storage location accessible by a wired or wireless network (e.g., in a cloud system). An attitude process 610 may run on the wearable computing architecture (e.g., processing module 260 or controller 460) and utilize data from the map 620 to determine the position and orientation of the wearable computing hardware or the user. The attitude data may be calculated from data collected on the fly as the user experiences the system and operates within the world. The data may include images, data from sensors (such as inertial measurement units, which generally include accelerometer and gyroscope components), and surface information associated with objects in a real or virtual environment.
[0103] The sparse representation may be the output of a simultaneous localization and mapping (e.g., SLAM or vSLAM, which refers to configurations where the input is image / vision only) process. The system can be configured to find not only where in the world various components reside, but also what the world consists of. Pose can be a building block that accomplishes many goals, including capturing maps and using data from maps.
[0104] In one embodiment, the sparse point locations may not be entirely correct by themselves, and further information may be required to produce a multifocal AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be utilized to fill this gap, at least in part. Such information may be calculated from a process called stereo 640, where the depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns created using an active projector), images obtained from an image camera, or hand gestures / totems 650 may serve as inputs to the stereo process 640. A significant amount of the depth map information may be fused together, some of which may be summarized using a surface representation. For example, a mathematically definable surface may be an efficient (e.g., for large point clouds) and easy to summarize input to other processing devices such as a game engine. Thus, the outputs of the stereo vision process (e.g., depth maps) 640 may be combined in a fusion process 630. The pose 610 may also be an input to this fusion process 630, whose output becomes the input to the map capture process 620. Sub-surfaces may connect with each other to form larger surfaces, such as in topographical mapping, and the map becomes a large-scale hybrid of points and surfaces.
[0105] Various inputs may be utilized to resolve various aspects in the mixed reality process 660. For example, in the embodiment depicted in FIG. 6A, game parameters may be input to determine that a user of the system is playing a monster battle game with one or more monsters in various locations, monsters dying or fleeing under various conditions (such as when the user shoots the monsters), walls or other objects in various locations, and the like. A world map may contain information about the location of objects or semantic information of objects (e.g., classification of whether an object is flat or round, horizontal or vertical, a table or a lamp, etc.), and may be another useful input to mixed reality. Attitude relative to the world may be an input as well, and may play an important role for almost any interactive system.
[0106] Controls or inputs from the user may be another input to the wearable system 600. As described herein, user inputs may include visual inputs, gestures, totems, audio inputs, sensory inputs, etc. To move around or play a game, for example, the user may need to command the wearable system 600 as to what he or she desires. There may be various forms of user control that may be utilized beyond simply moving oneself in space. In one embodiment, an object such as a totem (e.g., a user input device) or a toy gun may be held by the user and tracked by the system. The system would preferably be configured to know that the user is holding an item and understand the type of interaction the user is having with the item (e.g., if the totem or object is a gun, the system may be configured to understand the location and orientation, and whether the user is clicking a trigger or other sensitive button or element, which may be equipped with sensors such as an IMU that may help determine what is occurring even when such activity is not within the field of view of any of the cameras).
[0107] Tracking or recognition of hand gestures may also provide input information. The wearable system 600 may be configured to track and interpret hand gestures in terms of button presses, left or right, stop, grab, hold gestures, and the like. For example, in one configuration, a user may want to flip through email or calendar in a non-gaming environment, or "fist bump" with another person or player. The wearable system 600 may be configured to utilize a minimal amount of hand gestures, which may or may not be dynamic. For example, gestures may be simple static gestures such as an open hand for stop, a thumbs up for OK, a thumbs down for not OK, or a flip of the hand to the right or left or up / down for a directional command.
[0108] Eye tracking can be another input (e.g., to track where the user is looking, control display technology, render to a specific depth or range). In one embodiment, eye convergence may be determined using triangulation, and accommodation may then be determined using a convergence / accommodation model developed for that particular person. Eye tracking is performed by an eye camera, which can determine eye gaze (e.g., direction or orientation of one or both eyes). Other techniques, such as measurement of electrical potentials with electrodes placed near the eyes (e.g., electro-oculography), can also be used for eye tracking.
[0109] Speech tracking can be another input and can be used alone or in combination with other inputs (e.g., totem tracking, eye tracking, gesture tracking, etc.). Speech tracking can include speech recognition, voice recognition, alone or in combination. System 600 can include an audio sensor (e.g., a microphone) that receives an audio stream from the environment. System 600 can incorporate voice recognition technology to determine who is speaking (e.g., whether the speech is from the wearer of the ARD or from another person or from a voice (e.g., recorded voice transmitted by a loudspeaker in the environment)) and to determine what is being said. The local data and processing module 260 or the remote processing module 270 can process the audio data from the microphone (or audio data in another stream, such as, for example, a video stream being viewed by a user) and identify the content of the speech by applying various speech recognition algorithms, such as, for example, hidden Markov models, dynamic time warping (DTW) based speech recognition, neural networks, deep learning algorithms such as deep feedforward and recurrent neural networks, end-to-end automatic speech recognition, machine learning algorithms (described with reference to FIG. 7), or other algorithms that use acoustic or language modeling.
[0110] The local data and processing module 260 or the remote processing module 270 may also apply speech recognition algorithms that may identify the identity of the speaker, such as whether the speaker is the user 210 of the wearable system 600 or another person with whom the user is conversing. Some example speech recognition algorithms may include frequency estimation, hidden Markov models, Gaussian mixture models, pattern matching algorithms, neural networks, matrix representations, vector quantization, speaker diarization, decision trees, and dynamic time warping (DTW) techniques. Speech recognition techniques may also include anti-speaker techniques such as cohort models and world models. Spectral features may be used in representing speaker characteristics. The local data and processing module or the remote data processing module 270 may perform speech recognition using various machine learning algorithms, which are described with reference to FIG. 7.
[0111] An implementation of the wearable system can use these user controls or inputs via the UI. UI elements (e.g., controls, pop-up windows, callouts, data entry fields, etc.) can be used, for example, to dismiss the display of information, e.g., graphical or semantic information of an object.
[0112] With respect to the camera system, the exemplary wearable system 600 shown in FIG. 6A may include three pairs of cameras: a relative wide FOV or passive SLAM pair of cameras arranged on the sides of the user's face, and a different pair of cameras oriented in front of the user's face to handle the stereoscopic imaging process 640 and also to capture hand gestures and totem / object tracking in front of the user's face. The pair of cameras for the FOV cameras and stereoscopic process 640 may be part of the outward facing imaging system 464 (shown in FIG. 4). The wearable system 600 may include an eye tracking camera (which may be part of the inward facing imaging system 462 shown in FIG. 4) oriented towards the user's eyes to triangulate eye vectors and other information. The wearable system 600 may also include one or more textured light projectors (such as infrared (IR) projectors) to inject texture into the scene.
[0113] The wearable system 600 may comprise an avatar processing and rendering system 690. The avatar processing and rendering system 690 may be configured to generate, update, animate, and render avatars based on context information. Some or all of the avatar processing and rendering system 690 may be implemented as part of the local processing and data module 260 or the remote processing modules 262, 264, alone or in combination. In various embodiments, multiple avatar processing and rendering systems 690 (e.g., implemented on different wearable devices) may be used to render the virtual avatar 670. For example, a first user's wearable device may be used to determine the first user's intent, while a second user's wearable device may determine avatar characteristics and render the first user's avatar based on the intent received from the first user's wearable device. The first user's wearable device and the second user's wearable device (or other such wearable devices) can communicate over a network, for example, as will be described with reference to FIG. 9.
[0114] 6B illustrates an exemplary avatar processing and rendering system 690. The exemplary avatar processing and rendering system 690 may comprise, either alone or in combination, a 3D model processing system 680, a context information analysis system 688, an avatar auto-scaler 692, an intent mapping system 694, an anatomy adjustment system 698, and a stimulus response system 696. The system 690 is intended to illustrate functionality for avatar processing and rendering and is not intended to be limiting. For example, in some implementations, one or more of these systems may be part of another system. For example, parts of the context information analysis system 688 may be parts of the avatar auto-scaler 692, the intent mapping system 694, the stimulus response system 696, or the anatomy adjustment system 698, either alone or in combination.
[0115] The context information analysis system 688 can be configured to determine environment and object information based on one or more device sensors, described with reference to Figures 2 and 3. For example, the context information analysis system 688 can use images obtained by the outwardly facing imaging system 464 of a viewer of the user or the user's avatar to analyze the environment and objects (including physical or virtual objects) of the user's environment or the environment in which the user's avatar is rendered. The context information analysis system 688 can analyze such images, alone or in combination with data obtained from location data or world maps (e.g., maps 620, 710, 910), to determine the location and layout of objects in the environment. The context information analysis system 688 can also access biological characteristics of the user or humans in general to realistically animate the virtual avatar 670. For example, the context information analysis system 688 may generate an incongruity curve that may be applied to a user's avatar such that a part of the avatar's body (e.g., the head) cannot be in an incongruent (or unrealistic) position relative to other parts of the user's body (e.g., the avatar's head cannot be turned 270 degrees). In some implementations, one or more object recognizers 708 (shown in FIG. 7) may be implemented as part of the context information analysis system 688.
[0116] The avatar auto-scaler 692, the intent mapping system 694, and the stimulus response system 696, and the anatomy adjustment system 698 can be configured to determine characteristics of the avatar based on the context information. Some example characteristics of the avatar can include size, appearance, position, orientation, movement, posture, facial expression, etc. The avatar auto-scaler 692 can be configured to automatically scale the avatar so that the user does not have to view the avatar in an uncomfortable position. For example, the avatar auto-scaler 692 can increase or decrease the size of the avatar and bring the avatar to the user's eye level so that the user does not have to look down or up at the avatar, respectively. The intent mapping system 694 can determine the user's interaction intent based on the environment in which the avatar is rendered and map the intent (rather than the exact user interaction) to the avatar. For example, the intent of a first user can be to communicate with a second user in a telepresence session (see, e.g., FIG. 9B). Typically, two people face each other when communicating. The intent mapping system 694 of the first user's wearable system can determine such a facing intent that exists during the telepresence session and can cause the first user's wearable system to render the second user's avatar facing the first user. If the second user attempts to physically turn around, instead of rendering the second user's avatar in a turned around position (which would cause the back of the second user's avatar to be rendered towards the first user), the first user's intent mapping system 694 can continue to render the second avatar's face towards the first user, which is the inferred intent of the telepresence session (e.g., in this example, a facing intent).
[0117] The stimulus response system 696 can identify objects of interest in the environment and determine the avatar's response to the objects of interest. For example, the stimulus response system 696 can identify a source of sound in the avatar's environment and automatically reorient the avatar to look at the sound source. The stimulus response system 696 can also determine threshold exit conditions. For example, the stimulus response system 696 can cause the avatar to return to its original posture after the sound source disappears or after a period of time has passed.
[0118] The anatomical adjustment system 698 can be configured to adjust the user's posture based on the biological characteristics. For example, the anatomical adjustment system 698 can be configured to adjust the relative position between the user's head and the user's torso or between the user's upper body and lower body based on the discomfort curve.
[0119] The 3D model processing system 680 can be configured to animate the virtual avatar 670 and render it on the display 220. The 3D model processing system 680 can include a virtual character processing system 682 and a movement processing system 684. The virtual character processing system 682 can be configured to generate and update (to create and animate) a 3D model of the user. The movement processing system 684 can be configured to animate the avatar, for example, by changing the avatar's pose, by moving the avatar around in the user's environment, or by animating the avatar's facial expressions, etc. As will be further described herein, the virtual avatar can be animated using rigging techniques. In some embodiments, the avatar is represented in two parts: a surface representation (e.g., a deformable mesh) that is used to render the virtual avatar's outward appearance, and a hierarchical set of interconnected joints (e.g., a core skeleton) for animating the mesh. In some implementations, the virtual character processing system 682 can be configured to edit or generate surface representations, while the movement processing system 684 can be used to animate the avatar by moving the avatar, deforming meshes, etc. (Example of mapping the user's environment)
[0120] 7 is a block diagram of an example of a MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual inputs 702 from a user's wearable system, stationary inputs 704 such as room cameras, sensory inputs 706 from various sensors, gestures, totems, eye tracking, user inputs, etc. from user input devices 466) from one or more user wearable systems (e.g., wearable system 200 or display system 220) or stationary room systems (e.g., room cameras, etc.). The wearable systems can use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, movement sensors, depth sensors, GPS sensors, inward facing imaging systems, outward facing imaging systems, etc.) to determine the location and various other attributes of the user's environment. This information may be further supplemented with information from stationary cameras in the room, which may provide images from different perspectives or various cues. Image data acquired by the cameras (e.g., room cameras and / or cameras of the outward facing imaging systems) may be turned into a set of mapping points.
[0121] One or more object recognizers 708 can crawl through the received data (e.g., a collection of points), recognize or map the points, tag the images, and attach semantic information to the objects using a map database 710. The map database 710 may comprise various points and their corresponding objects collected over time. The various devices and the map database may be interconnected through a network (e.g., LAN, WAN, etc.) and accessible to the cloud.
[0122] Based on this information and the set of points in the map database, the object recognizers 708a-708n may recognize objects in the environment. For example, the object recognizers may recognize faces, people, windows, walls, user input devices, televisions, documents (e.g., travel documents, driver's licenses, passports as described in the security embodiments herein), other objects in the user's environment, etc. One or more object recognizers may be specialized for objects with certain characteristics. For example, object recognizer 708a may be used to recognize faces, while another object recognizer may be used to recognize documents.
[0123] Object recognition may be performed using various computer vision techniques. For example, the wearable system may analyze images acquired by the outward facing imaging system 464 (shown in FIG. 4) and perform scene reconstruction, event detection, video tracking, object recognition (e.g., people or documents), object pose estimation, face recognition (e.g., from images of people in the environment or on documents), learning, indexing, motion estimation, or image analysis (e.g., identifying indicia in documents, such as photographs, signatures, identification information, travel information, etc.), etc. One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale Invariant Feature Transform (SIFT), Speed Up Robust Features (SURF), Orientation FAST and Rotation BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, visual simultaneous localization and mapping (vSLAM) techniques, sequential Bayes estimators (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative nearest neighbor (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms (e.g., support vector machines, k-nearest neighbor algorithms, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.), and the like.
[0124] Object recognition can additionally or alternatively be performed by various machine learning algorithms. Once trained, the machine learning algorithms can be stored by the HMD. Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., a priori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural networks, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., stacked generalization, etc.), and / or other machine learning algorithms. In some embodiments, individual models can be customized for individual datasets. For example, the wearable device can generate or store a base model. The base model may be used as a starting point to generate additional models specific to a data type (e.g., a particular user in a telepresence session), a data set (e.g., a set of additional images acquired of a user in a telepresence session), a conditional situation, or other variations. In some embodiments, the wearable HMD may be configured to utilize multiple techniques to generate models for analysis of the aggregated data. Other techniques may include using predefined thresholds or data values.
[0125] Based on this information and the set of points in the map database, the object recognizer 708a-708n may recognize objects, complement the objects with semantic information, and give the objects movability. For example, if the object recognizer recognizes that a set of points is a door, the system may attach some semantic information (e.g., a door has a hinge and has 90 degree movement around the hinge). If the object recognizer recognizes that a set of points is a mirror, the system may attach semantic information that a mirror has a reflective surface that can reflect an image of an object in a room. The semantic information may include the affordances of the object, as described herein. For example, the semantic information may include the object's normal. The system may assign a vector, the direction of which indicates the object's normal. Over time, the map database may grow as the system (which may reside locally or be accessible through a wireless network) accumulates more data from the world. Once an object is recognized, the information may be transmitted to one or more wearable systems. For example, the MR environment 700 may include information about a scene taking place in California. The environment 700 may be transmitted to one or more users in New York. Based on the data received from the FOV cameras and other inputs, an object recognizer and other software components can map points collected from the various images, recognize objects, etc., so that the scene can be accurately "passed" to a second user who may be in a different part of the world. The environment 700 may also use a topology map for localization purposes.
[0126] 8 is a process flow diagram of an example method 800 for rendering virtual content in relation to recognized objects. Method 800 illustrates how a virtual scene may be presented to a user of a wearable system. The user may be geographically remote from the scene. For example, a user may be in New York but may wish to view a scene currently occurring in California, or may wish to go for a walk with a friend who resides in California.
[0127] In block 810, the wearable system may receive input from the user and other users regarding the user's environment. This may be accomplished through various input devices and knowledge already held in the map database. The user's FOV camera, sensors, GPS, eye tracking, etc. communicate information to the system in block 810. The system may determine sparse points based on this information in block 820. The sparse points may be used in determining pose data (e.g., head pose, eye pose, body pose, or hand gestures) that may be used in displaying and understanding the orientation and position of various objects in the user's surroundings. The object recognizers 708a-708n may crawl through these collected points in block 830 and recognize one or more objects using the map database. This information may then be communicated to the user's respective wearable system in block 840, and the desired virtual scene may be displayed to the user in response in block 850. For example, a desired virtual scene (eg, a user in CA) may be displayed in the proper orientation, position, etc., relative to various objects and other surroundings of the user in New York. (Example Communication Between Multiple Wearable Systems)
[0128] FIG. 9 diagrammatically illustrates an overall system diagram depicting multiple user devices interacting with each other. Computing environment 900 includes user devices 930a, 930b, 930c. User devices 930a, 930b, and 930c can communicate with each other through network 990. User devices 930a-930c each include a network interface and can communicate with remote computing system 920 (which may also include network interface 971) via network 990. Network 990 may be a LAN, WAN, peer-to-peer network, wireless, Bluetooth, or any other network. Computing environment 900 may also include one or more remote computing systems 920. Remote computing systems 920 may include server computer systems that are clustered and located in different geographic locations. User devices 930a, 930b, and 930c may communicate with remote computing system 920 via network 990.
[0129] The remote computing system 920 may include a remote data repository 980 that may maintain information about a particular user's physical and / or virtual world. The data storage 980 may store information related to the configuration of a user, the user's environment (e.g., a world map of the user's environment), or the user's avatar. The remote data repository may be an embodiment of the remote data repository 280 shown in FIG. 2. The remote computing system 920 may also include a remote processing module 970. The remote processing module 970 may be an embodiment of the remote processing module 270 shown in FIG. 2. The remote processing module 970 may include one or more processors that may be in communication with the user devices (930a, 930b, 930c) and the remote data repository 980. The processors may process information obtained from the user devices and other sources. In some implementations, at least a portion of the processing or storage may be provided by the local processing and data module 260 (as shown in FIG. 2). The remote computing system 920 may enable a given user to share information about the particular user's own physical and / or virtual world with another user.
[0130] The user devices may be, alone or in combination, a wearable device (such as an HMD or ARD), a computer, a mobile device, or any other device. For example, user devices 930b and 930c may be an embodiment of the wearable system 200 shown in FIG. 2 (or the wearable system 400 shown in FIG. 4) that may be configured to present AR / VR / MR content.
[0131] One or more of the user devices can be used in conjunction with the user input device 466 shown in FIG. 4. The user devices can acquire information about the user and the user's environment (e.g., using the outwardly facing imaging system 464 shown in FIG. 4). The user devices and / or the remote computing system 1220 can use the information acquired from the user devices to build, update, and construct a collection of images, points, and other information. For example, the user devices may process the raw information acquired and send the processed information to the remote computing system 1220 for further processing. The user devices may also send the raw information to the remote computing system 1220 for processing. The user devices may receive the processed information from the remote computing system 1220 and provide final processing before projecting to the user. The user devices may also process the acquired information and pass the processed information to other user devices. The user devices may communicate with the remote data repository 1280 while processing the acquired information. Multiple user devices and / or multiple server computer systems may be involved in the construction and / or processing of the acquired images.
[0132] Information about the physical world may evolve over time and may be based on information collected by different user devices. Models of the virtual world may also evolve over time and be based on inputs of different users. Such information and models may sometimes be referred to herein as world maps or world models. As described with reference to Figures 6 and 7, information obtained by user devices may be used to build a world map 910. The world map 910 may include at least a portion of the map 620 illustrated in Figure 6A. Various object recognizers (e.g., 708a, 708b, 708c...708n) may be used to recognize objects and tag images and to attach semantic information to objects. These object recognizers are also described in Figure 7.
[0133] The remote data repository 980 can be used to store data and facilitate the construction of the world map 910. The user devices can constantly update information about the user's environment and receive information about the world map 910. The world map 910 may be created by the user or by another person. As discussed herein, the user devices (e.g., 930a, 930b, 930c) and the remote computing system 920 may build and / or update the world map 910, alone or in combination. For example, the user devices may communicate with the remote processing module 970 and the remote data repository 980. The user devices may obtain and / or process information about the user and the user's environment. The remote processing module 970 may communicate with the remote data repository 980 and the user devices (e.g., 930a, 930b, 930c) and process information about the user and the user's environment. The remote computing system 920 can modify information obtained by the user devices (e.g., 930a, 930b, 930c), for example, by selectively cropping the user's image, modifying the user's background, adding virtual objects to the user's environment, annotating the user's speech with auxiliary information, etc. The remote computing system 920 can transmit the processed information to the same and / or different user devices. (Example process for sharing 3D models)
[0134] FIG. 10 illustrates an example process 1000 for sharing 3D assets using the systems and methods described herein. In step 1001, a 3D sharing host application can access a 3D model file from a host system. The host may do this by either creating a new 3D asset on the host system or by obtaining a 3D asset from an external system. The 3D asset may originate in a 3D modeling application such as AutoCad or 3D Studio Max and may be any file type. The host may utilize an asset import plug-in (e.g., ASSIMP) to enable graphic format exchange. The ASSIMP plug-in may support up to 40 different file formats, such as Filmbox (FBX) in .fbx format, or Autodesk Max 3D modeling (3DS) in .3ds format. In some embodiments, custom programs may be written to enable file format exchange. Any other suitable file format conversion method may be used to enable file reformatting for use within the host 3D sharing application.
[0135] In step 1003, the 3D sharing host application can break down the 3D asset into its constituent parts. Examples of constituent parts can be geographic data, material data, a vertex table, one or more textures, triangle indices, or any other data used to ultimately define a complete representation of the 3D model.
[0136] In step 1005, the components may be sent to a client that is part of a network with the host. In some embodiments, there may be only one client. In some embodiments, there may be more than one client. In a preferred embodiment, a local area network may be utilized utilizing the UDP protocol. The network may be a LAN, a WAN, a peer-to-peer network, wireless, Bluetooth, ad-hoc, or any other network. In some embodiments, the network may be network 990 as described above. The host may be a remote computing system that may include server computer systems that are clustered and located in different geographic locations as described above in connection with 920. The client may be user device 930a, 930b, and / or 930c as described above. In some embodiments, there may be one host. In some embodiments, there may be more than one host. In some embodiments, each device connected to the network may be both a host and a client, effectively both sharing and receiving 3D assets from other devices on the network.
[0137] In step 1007, the client may receive the component parts.
[0138] In step 1009, the client can reconstruct the 3D asset from the components received from the host. In some embodiments, the 3D asset can be stored in the memory of the client device. In some embodiments, the 3D asset may be saved for later use by the device.
[0139] In step 1011, the client may display the reconstructed 3D asset. In some embodiments, the client may not display the reconstructed 3D asset, or may display the asset at a later time, for example, in response to a user request. In some embodiments, the client may display the reconstructed 3D asset, but the user may not be able to see it because it is outside the user's FOV. (Example of 3D model sharing system configuration)
[0140] FIG. 11 illustrates an exemplary 3D model sharing system configuration 1100 for sharing 3D assets using the systems and methods described herein. FIG. 11 depicts the system configuration 1100 with one client (which may be referred to as a "remote") system 1116 and one server (which may be referred to as a "host") system 1118. The client system and the host system need not be physically separate systems, for example, in some embodiments, the client and the server can exist on the same physical device (e.g., as client and server threads executed in parallel by one or more processors of the same computer system). In practice, the 3D sharing system 1100 may generally require at least one host system 1118, which may act as a source of decomposed 3D assets. The host system 1118 may have authoritative control over the types and quantities of 3D models available within the 3D sharing application. The 3D sharing system 1100 may have any number (e.g., 0, 1, 10, etc.) of client systems 1116 operably connected to a host system 1118 through a communication network 1114. The communication network 1114 may be a LAN, for example, utilizing User Datagram Protocol (UDP). The host system 1118 may comprise one or more processors operably coupled to at least one display and capable of receiving communications from other computing systems. In some embodiments, the host system 1118 may be a PC desktop computer attached to a computer monitor with a network interface. In some embodiments, the host system 1118 may be a system 200, 930a, 930b, or 930c as described above. The user devices 200, 930a, 930b, and 930c may communicate with each other and with other computing systems through the communication network 1114.Each of the user devices 930a-930c can include a network interface (which may also include a network interface 971) for communicating with a remote computing system, such as 920, over the network 1114. The client system 1116 can comprise one or more processors operatively coupled to at least one display and capable of receiving communications from other computing systems. In some embodiments, the client system 1116 can be a PC desktop computer attached to a computer monitor with a network interface. In some embodiments, the client system 1118 can be the system 200, 930a, 930b, or 930c as described above. The user devices 200, 930a, 930b, and 930c can communicate with each other and with other computing systems through the communication network 1114. Each of the user devices 930a-930c can include a network interface (which may also include a network interface 971) for communicating with a remote computing system, such as 920, over the network 1114.
[0141] The host system 1118 may have a 3D sharing host application installed. The 3D sharing host application may include three main modules: an asset module 1108, a decomposition module 1110, and a transmission module 1112. The asset module 1108 may include one or more complete 3D models. The decomposition module 1110 may function to decompose the complete 3D model into its constituent parts, optionally compressing the constituent parts, and placing the constituent parts into one or more arrays ready to be transmitted to different systems. The arrays may be stored in one or more libraries, which may contain one or more decomposed 3D models. The transmission module 1112 may function to decompose the constituent parts from the decomposition module 1110 into transferable pieces of data and may manage the transmission protocol and process. The transmission module 1112 may transmit the transferable pieces of data to other systems connected over a communication network 1114.
[0142] The client or remote system 1116 may have a 3D sharing client application installed. The 3D sharing client application may comprise three main modules: a transmission module 1106, a reconstruction module 1104, and an assets module 1102. The transmission module 1106 may function to receive one or more exploded 3D models and may manage the transmission and networking process for the client system 1116. The transmission module 1106 may communicate with the 3D model sharing host application through messaging protocols. These protocols may be custom-made or may utilize commonly used protocols. The reconstruction module 1104 may function to reassemble exploded 3D models received from other systems on the network. The assets module 1102 may comprise one or more complete 3D models. (Example process for sharing 3D models)
[0143] 12 illustrates an example 3D model sharing process 1200 between a server and a client using the systems and methods described herein. The process 1200 may begin at step 1202 when a server system, such as the server system 1118, opens a 3D model sharing host application.
[0144] In step 1204, the server may load one or more complete 3D model files into the 3D model sharing host application. The complete 3D model files may originate on the host system 1118 from different 3D modeling software, such as, for example, AutoCad, that is downloaded onto the host system 1118. In some embodiments, the 3D model files may be imported from an external source, such as an external hard drive, a remote computing system (e.g., 920 from FIG. 9 above), or from cloud storage. In some embodiments, the 3D model files may originate from a 3D scanner or similar device. The 3D model files may be imported using an ASSIMP or equivalent plug-in. The plug-in may convert the 3D model files from one format to a different file format that is compatible with the 3D model sharing host application. The plug-in may be utilized only by the host system, such as, for example, the host system 1118, because the file formatting may be lost on the server during the decomposition process, as described in more detail below.
[0145] In step 1206, the server system may decompose the 3D model into its constituent parts. Decomposing the 3D model may include identifying the constituent parts within the 3D model. Decomposing the 3D model may be non-destructive in that it does not result in a change to the 3D model itself (e.g., loss of information within the 3D model) or a change to the way the 3D model is rendered for viewing. Decomposing the 3D model may include applying data level changes to the 3D model and / or its constituent parts (e.g., applying data formatting changes or compressing the constituent parts to facilitate sharing). The decomposition process may occur within the decomposition module 1110, as described above. A constituent part may be a subset of data required to fully define the 3D model. A constituent part subset may be defined by an existing bundle of data that comprises the 3D model, such as vertex tables, geographic data, material data, etc. In some embodiments, a constituent part subset may be defined in other ways, such as by a 3D model sharing application programmer.
[0146] In step 1214, the client system may open a 3D model sharing client application, which may initiate a series of steps related to networking protocols, communication, and discovery.
[0147] In step 1216, the 3D model sharing client application may discover the 3D model sharing host over a network, if a host exists. The network may be a LAN (e.g., UDP). In some embodiments, the local network is private. In some embodiments, a cloud service is not used.
[0148] In step 1208, the server may receive a connection request from a client, and the server may accept the connection request.
[0149] In step 1210, the server may send a client information request back to the client. The client information request may contain a list of everything in the library of the 3D model sharing host application. The client information request may alternatively be referred to as an asset holdings or table of contents.
[0150] In step 1218, the client may reply to the server with data that defines items that are missing from the 3D model sharing client application library when compared to the 3D model sharing server application library.
[0151] In step 1212, the server may respond by sending the client the items that are missing from the 3D model sharing client application. In some embodiments, the client may have recently opened the 3D model sharing application and therefore may start with zero items in its library. In this situation, the host may send its entire library to the client. In some embodiments, the library may be located in memory and therefore erased when the 3D model sharing application is closed. In some embodiments, the library may be saved to a disk, such as the client system hard drive, so that the 3D models may be accessed at a later date. In some embodiments, the 3D model data sent to the client is a data structure without a file format. At the end of this step, the 3D model sharing host library may match the 3D model sharing client library. In other words, both the host system and the client system may now contain the same components for the same set of 3D models.
[0152] In some embodiments, a client may update items that are out of date compared to items stored in the server. For example, a client information request may include a version number associated with an individual asset in an asset inventory. One or more assets in the server's asset inventory may correspond to assets stored with the client. The client may identify items stored in the client asset library that have a lower version number than the corresponding items in the server's asset inventory. The client may send data to the server identifying the items requesting an update, and the server may respond by sending the corresponding updated items to the client.
[0153] In step 1220, the client may reconstruct one or more 3D models from the new items sent from the server. In some embodiments, the client may now have a complete 3D model that is identical to the server.
[0154] In step 1222, a user of the client system may now view the 3D model shared by the host system. (Example of 3D model sharing system configuration)
[0155] FIG. 13 illustrates an example 3D model sharing system configuration 1300 for sharing 3D assets using the systems and methods described herein.
[0156] The 3D model sharing system configuration 1300 may comprise a host system 1306 and a client system 1304 operatively coupled by a communication link 1302, such as a network. The client system 1304 may have a 3D model sharing client application open and running, thus operating within the runtime environment. The host system 1306 may have a 3D model sharing host application open and running, thus operating within the runtime environment. While FIG. 13 depicts one server system and one host system, this is not intended to be limiting in scope and is done for ease of illustration. In practice, the 3D model sharing system configuration 1300 may have only one host system 1306, more than one host system 1306, one client system 1304, more than one client system 1304, and / or one or more systems that function as both a host (e.g., sending 3D model data to other systems via communication 1302) and a client (e.g., receiving 3D model data from other systems via communication 1302). Further, in some embodiments, a single system may comprise both a host system and a client system. In some embodiments, the host system 1306 may be the host system 1118. In some embodiments, the client system 1304 may be the client system 1116. The host system 1306 may comprise one or more processors operably coupled to at least one display, e.g., a PC desktop computer attached to a computer monitor. In some embodiments, the host system 1306 may be the system 200, 930a, 930b, or 930c as described above. The client system 1304 may comprise one or more processors operably coupled to at least one display, e.g., a PC desktop computer attached to a computer monitor. In some embodiments, the client system 1304 may be the system 200, 930a, 930b, or 930c as described above.
[0157] The host system 1306 may comprise a server module 1316. The server module 1316 may manage functions and processes specific to being a server, such as transmission and networking protocols. The server module 1316 may receive and process connection requests, such as steps 1208, 1210, and / or 1212 in process 1200. In some embodiments, the server module 1316 may comprise the transmission module 1112.
[0158] The host system 1306 may comprise a host load module 1320. The load module may contain functions and processes that allow the 3D model sharing host application to import or load 3D assets into the 3D model sharing host application or onto the host system 1306. The load module 1320 may comprise an ASSIMP plug-in. In some embodiments, step 1204 from process 1200 may occur within the load module 1320. In some embodiments, step 1001 from process 1000 may occur within the load module 1320. In some embodiments, the asset module 1108 may comprise the load module 1320.
[0159] The host system 1306 may comprise a host library manager module 1318. The library manager module 1318 may contain and manage data and processes corresponding to 3D assets already loaded into the 3D asset sharing host application. The library manager module 1318 may contain and manage data and processes corresponding to a library of the 3D asset sharing host application. This may include a decomposition process that maintains a library table of contents, etc. In some embodiments, step 1206 from process 1200 may occur within the library manager module 1318. In some embodiments, step 1003 from process 1000 may occur within the library manager module 1318. In some embodiments, the library manager module 1318 may comprise the decomposition module 1110 from the system 1100.
[0160] The host system 1306 may include a host world module 1322. The host world module 1322 may manage the complete 3D assets, the 3D virtual world, the meshing of the real world, etc., and the integration between them. In some embodiments, the host world module 1322 may be considered a scene graph, or a functional equivalent of a scene graph. In some embodiments, the world can be drawn from items (i.e., fully reconstructed 3D models) from a world library, from which world objects can be created. In some embodiments, everything in the 3D virtual world can be represented as a world object. A world object may be part of a world "class." In some embodiments, the host world module 1322 contains the same data as the client world module 1312 by step 1222 of process 1200 or by step 1011 of process 1000 (i.e., after the 3D model has been shared and reconstructed locally at the client). The output from the host world module 1322 may be fed into a rendering path module 1324.
[0161] The host system 1306 may include a host rendering path module 1324. The rendering path module 1324 may accept data from the host world module 1322 as input and pass it through the host system 1306 rendering pipeline and may display one or more 3D models on a display of the host system 1306.
[0162] The client system 1304 may comprise a client module 1308. The client module 1304 may manage functions and processes specific to being a client, such as transmission and networking and / or server / client protocols. The client module 1308 may manage server discovery processes and other connection protocols. In some embodiments, this may include steps 1216 and / or 1218 in process 1200. In some embodiments, the client module 1308 may comprise the transmission module 1106.
[0163] The client system 1304 may comprise a client library manager module 1310. The library manager module 1310 may contain and manage data and processes corresponding to the 3D asset components received from the 3D asset sharing host application. The library manager module 1310 may contain and manage data and processes corresponding to the library of the 3D asset sharing client application. This may include a reconstruction process. In some embodiments, step 1220 from process 1200 may occur within the library manager module 1310. In some embodiments, step 1009 from process 1000 may occur within the library manager module 1310. In some embodiments, the library manager module 1310 may comprise a reconstruction module 1104 from the system 1100. In some embodiments, the client library contains the same data as the host library by step 1222 of process 1200 or by step 1011 of process 1000 (i.e., after the 3D model is shared and reconstructed locally at the client).
[0164] The client system 1304 may include a client world module 1312. The client world module 1312 may manage the fully reconstructed 3D assets, the 3D virtual world, the meshing of the real world, and the like, and the integration between them. In some embodiments, the client world module 1312 may be considered a scene graph, or a functional equivalent of a scene graph. In some embodiments, the world is drawn from items (i.e., fully reconstructed 3D models) taken from a world library, from which world objects are created. In some embodiments, everything in the 3D virtual world is represented as a world object. A world object may be part of a world "class." In some embodiments, the client world module 1312 may contain the same data as the host world module 1322 by step 1222 of process 1200 or by step 1011 of process 1000 (i.e., after the 3D model has been shared and reconstructed locally at the client). The output from the client world module 1312 may be fed into a rendering path module 1314.
[0165] The client system 1304 may include a client rendering path module 1314. The rendering path module 1314 may accept data from the client world module 1312 as input and pass it through a client system 1304 rendering pipeline and may display one or more virtual 3D models on a display of the client system 1304.
[0166] The exemplary 3D model sharing system configuration 1300 can enable two or more devices to share a complete 3D model during runtime, which can have the advantage of adding new assets to an application on a computing system faster than traditional systems where updates may have to wait until the application is closed. (Example process for disassembling a 3D model)
[0167] 14 illustrates an example process 1400 for decomposing a full 3D model using the systems and methods described herein. In some embodiments, the decomposition process 1400 may occur within the host library manager module 1318. In some embodiments, the decomposition process 1400 may occur within the decomposition module 1110. In some embodiments, the decomposition process 1400 may be step 1003 in process 1000. In some embodiments, the decomposition process 1400 may be step 1206 in process 1200.
[0168] Process 1400 may begin with a complete 3D model that may be loaded into a 3D model sharing host application. In step 1402, the 3D model sharing host application may copy a component of the 3D model and place data representing the component into an array. Some example components may be one or more of vertex positions, a vertex table, geographic data, material data, textures, triangle index tables, etc. A component may be a portion of the data required to represent the complete 3D model.
[0169] The components may optionally be compressed in step 1404. Standard compression techniques may be used.
[0170] In step 1406, the components may be packaged.
[0171] In step 1408, the package may be stored in one of the libraries of the 3D model sharing host application. The 3D model sharing host application may have one or more libraries used to store and organize the packages. In some embodiments, the library may comprise three libraries, as shown in FIG. 16: a mesh library, a material library, and a texture library. Alternative library configurations may be used. The package may be stored in the corresponding library (e.g., textures may be stored in the texture library).
[0172] In step 1410, the process may check the 3D model to see if there are additional components that need to be added to the library. If there are more components that have not been added to the library, the process may start again from the beginning at step 1402 until all components have been added to the library. When all components have been added, the library may contain a complete record of the complete 3D model, as in step 1412.
[0173] In some embodiments, the 3D model can be decomposed into constituent parts predefined by the 3D model application. For example, a 3D model application (e.g., a computer-aided design program) can define a 3D model in terms of different data elements (e.g., meshes and mesh renderings), each of which may comprise one or more defined constituent parts (e.g., triangle arrays or textures). Decomposing the 3D model can then comprise applying an operation (e.g., data formatting) to each constituent part and storing it in memory. In some embodiments, the methods and systems described herein can be used to decompose the 3D model into one or more subgroups defined by a user. For example, for a 3D model of a car, a user can specify a set of vertices / parts / structures as "engine", a set of vertices / parts / structures as "chassis", a set of vertices / parts / structures as "drivetrain", and a set of vertices / parts / structures as "wheels". Each subgroup can then be optionally compressed and transmitted separately to the device. In some embodiments, the 3D model can be decomposed into algorithmically defined subgroups. For example, an algorithm may have been developed (e.g., manually, semi-automatically, and / or automatically, e.g., through machine learning) to identify a set of vertices / parts / structures as an "engine." The algorithm may first identify the entire 3D model (e.g., first identify the model as a car) before attempting to classify the set of vertices / parts / structures from a limited pool based on the initial identification. The algorithm may also attempt to classify the set of vertices / parts / structures directly. (Example of a fully 3D model)
[0174] 15 illustrates an example full 3D model 1500 using the systems and methods described herein. In some embodiments, a full 3D model may be a complete set of data describing a 3D renderable digital object. In some embodiments, a full 3D model may be a set of data representing a digital or virtual object that is capable of being displayed when added to a compatible rendering pipeline.
[0175] In the 3D model component 1502, the 3D model can be assigned a name. The name may be assigned by a user, an import process, and / or may be the original file name used in the 3D model creation software, such as the file name given to the 3D model in AutoCad. The 3D model may have two main types of data: mesh data 1504 and mesh renderer data 1506. The mesh data may be further decomposed into subsets of data or components, such as component A 1508, component B 1510, component 1512, and component 1514. The material data 1516 may be further decomposed into subsets of data or components, such as material properties 1522, texture 1 1518, and texture 2 1522. The exact layout, categorization, and subgrouping of the 3D model data may vary from that shown in FIG. 15. In some embodiments, there may be any number of components under mesh 1504, such as 2, 3, or 10 components. In some embodiments, there may be multiple subsets of properties 1522 under the material data 1516 instead of just one. In some embodiments, there may be more or fewer texture components 1518, 1520 than are shown in Figure 15. Any suitable number and categorization may be used to break down the complete 3D model into its component parts.
[0176] In some embodiments, the exemplary full 3D model 1500 can be decomposed using the exemplary decomposition process 1400. The full 3D model 1500 can comprise one or more components (e.g., component A 1508, component B 1510, component C 1512, component D 1514, texture 1 1518, texture 2 1520, and / or material properties 1522). One component (e.g., texture 1) of the full 3D model 1500 can be copied in step 1402 of the decomposition process 1400. The components can be optionally compressed in step 1404 of the decomposition process 1400. The components can be packaged in step 1406 of the decomposition process 1400. The components can be stored in step 1408 of the decomposition process 1400 (e.g., in the exemplary set of libraries 1600). Steps 1402, 1404, 1406, and 1408 may be repeated for each component of the complete 3D model 1500 until all components of the complete 3D model 1500 are stored (eg, in a set in the library 1600). (Example of library in 3D model sharing application)
[0177] FIG. 16 illustrates an example set of libraries 1600 utilized in a 3D model sharing application to store components of a 3D model using the systems and methods described herein. The set of libraries 1600 may comprise one or more libraries to store and organize the 3D model components. In some embodiments, there may be three libraries: a mesh library 1602, a material library 1604, and a texture library 1606. The mesh library 1602 may contain a set of components in the form of arrays that describe the 3D model mesh. Exemplary arrays that may be stored within the mesh library are a vertex buffer array 1608, a normal buffer array 1610, a UV buffer array 1612, and / or a triangle array 1614. The mesh library may also optionally contain one or more string values, such as a material "name" 1616, a bitmap texture 1 "name" 1618, and a bitmap texture 2 "name" 1620. These string values 1616, 1618, 1620 may reference other libraries, such as material library 1604 and texture library 1606. In some embodiments, a complete 3D model may be represented when all of the data is pulled or referenced from mesh library 1602. In some embodiments, a complete 3D model for a "name" may be represented when all of the data is pulled or referenced from all of the libraries 1602, 1604, 1606 that correspond to a record or single 3D model "name."
[0178] The material library 1604 may contain data representing material properties and property-related values for the 3D model. In some embodiments, the texture library 1606 may contain one or more textures 1622, 1624. In some embodiments, the texture library 1606 may contain one or more bitmaps. In some embodiments, the texture library 1606 may contain one or more UV maps, UV coordinates, or other UV space data.
[0179] All of the data contained in the library 1600 of 3D sharing applications may be combined to represent a complete renderable 3D model. (Example process for reconstructing a 3D model)
[0180] 17 illustrates an example process 1700 for reconstructing a complete 3D model from its constituent parts using the systems and methods described herein. In some embodiments, this process may be step 1220 in process 1200. In some embodiments, process 1700 may be step 1009 in process 1000. In some embodiments, process 1700 may occur within reconstruction module 1104. In some embodiments, process 1700 may occur within library manager module 1310.
[0181] Process 1700 may begin at step 1702 by creating an empty object, sometimes referred to as an empty game object. In some embodiments, an empty game object may comprise an empty volume in space with a reference coordinate system. In some embodiments, an empty object is essentially a transformation, i.e., position and orientation, that exists in space. In some embodiments, the empty object is an empty prism. In some embodiments, the empty object may correspond to 1502 of FIG. 15.
[0182] After the empty object is created 1702, a new mesh 1704 and a new mesh renderer 1708 may be added to the empty game object. In some embodiments, adding the new mesh 1704 may correspond to the mesh 1504 of Figure 15, and adding the new mesh renderer 1708 may correspond to the mesh renderer 1506 of Figure 15.
[0183] Process 1700 may proceed to steps 1706 and 1710 where an empty mesh buffer may be filled with 3D model components corresponding to the mesh data and an empty mesh renderer buffer may be filled with 3D model components corresponding to the mesh rendering data. In some embodiments, steps 1706 and / or 1710 may decompress the compressed data. In some embodiments, the mesh data may correspond to components 1508, 1510, 1512, 1514 from FIG. 15. In some embodiments, the mesh rendering data may correspond to components 1518, 1520, 1522. The buffer may be filled with data from a library of 3D models application corresponding to a single 3D model. In some embodiments, the empty mesh buffer may be filled 1706 with data from mesh library 1602. In some embodiments, the empty mesh render buffer may be filled 1710 with data from material library 1604 and texture library 1606. The process 1700 may be complete when all of the data corresponding to a single 3D model (i.e., a single record in the library of the 3D model sharing application) has been added to the empty object.
[0184] 18 illustrates an example full 3D model 1800 using the systems and methods described herein. In some embodiments, the full 3D model 1800 may be the output of the process 1700. In some embodiments, the full 3D model 1800 may be a more detailed version of the example full 3D model 1500, where general construction component A may be a vertex buffer 1808, general construction component B may be a normal buffer 1810, general construction component C may be a UV buffer 1812, and general construction component D may be a triangle index 1814. In some embodiments, the triangle index 1814 may need to be filled last.
[0185] In some embodiments, the complete 3D model 1800 may be an input to the process 1400 from which components are copied. In some embodiments, the complete 3D model 1800 may be stored as a world object in the world module 1322 and / or 1312. In some embodiments, the complete 3D model 1800 may be a 3D model that is loaded into a 3D model sharing server application in step 1204 of the process 1200 and / or step 1001 of the process 1000. In some embodiments, the complete 3D model 1800 may be an output from step 1220 of the process 1200 and / or step 1009 of the process 1000. In some embodiments, the complete 3D model 1800 may be an asset stored in the asset module 1102 and / or 1108. The object "name" 1802 may match the original file name that was loaded into the 3D model sharing application, e.g., car.fbx, even though the 3D model is no longer in FBX file format. (Another example of sharing 3D models)
[0186] In some embodiments, the 3D model can originate from an MR device (e.g., wearable system 200). The MR device can capture information about a physical object, generate a 3D model of the physical object, break down the 3D model into constituent parts, and send the constituent parts to a client device, which can generate the 3D model based on the constituent parts. For example, the object recognizer 708 can receive input from the outward-facing imaging system 464, a lidar sensor, a depth sensor, an RGB camera, an infrared camera, a time-of-flight camera, and / or other sensors on the MR device. This input can be used in computer vision algorithms and / or machine learning training algorithms to generate a 3D model based on the physical object. The algorithm can generate the 3D model by first identifying the physical object and then generating the 3D model based on a library of 3D models associated with the physical object. In some embodiments, the 3D model can be generated from the physical object by assembling subgroups based on predefined subgroups in the library of 3D models. In some embodiments, the 3D model can be broken down into subgroups defined by the user. In some embodiments, the 3D model can be decomposed into algorithmically defined subsets. For example, an algorithm may have been developed (e.g., manually, semi-automatically, and / or automatically, e.g., through machine learning) to identify sets of vertices / portions / structures as subsets.
[0187] In some embodiments, the algorithm can also generate 3D models directly from physical objects. For example, the algorithm can identify surfaces and vertices of a physical object and generate a 3D model of the entire physical object based on observed properties of the physical object. In some embodiments, the algorithm can identify localized surfaces and vertices of a physical object and generate a 3D model of the localized portions and / or define subgroups of the physical object based on a library of subgroups. The algorithm can store data regarding the association of the subgroups and form a complete 3D model based on the constituent parts.
[0188] The 3D model sharing system and method described in this disclosure may provide a specific solution to the technical problem of how to move renderable 3D models from one computing system to a different computing system during runtime, which may represent an improvement over existing techniques that only add new renderable 3D models to an application offline.
[0189] Although 3D models are discussed herein, it is also contemplated that the systems and methods described herein may be applied to other digital assets other than 3D models. For example, a 2D model may be decomposed (e.g., into contours and textures), transferred, and reconstructed using the systems and methods described herein. In some embodiments, an animation may be decomposed (e.g., into a series of meshes, each of which may be further decomposed as described herein), transferred, and reconstructed using the systems and methods described herein. In some embodiments, a 3D model may be flattened into a 2D model (e.g., for display on a 2D screen or to create a 2D map). (Other considerations)
[0190] Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying figures may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, and may be fully or partially automated thereby. For example, the computing systems may include general-purpose computers (e.g., servers) or special-purpose computers programmed with specific computer instructions, special-purpose circuits, etc. The code modules may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some implementations, certain operations and methods may be performed by circuitry specific to a given function.
[0191] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that special purpose hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may be required to implement the functionality, e.g., due to the amount or complexity of the calculations involved, or to provide results in substantially real-time. For example, an animation or video may contain many frames, each frame may have millions of pixels, and specifically programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.
[0192] The code modules or any type of data may be stored on any type of non-transitory computer readable medium, such as physical computer storage devices, including hard drives, solid state memory, random access memory (RAM), read only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations of the same, and / or the like. The methods and modules (or data) may also be transmitted as data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) generated over a variety of computer readable transmission media, including wireless-based and wired / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device, or communicated via a computer readable transmission medium.
[0193] Any process, block, state, step, or functionality in the flow diagrams described herein and / or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. Various processes, blocks, states, steps, or functionality can be combined, rearranged, added to, removed from, modified, or otherwise altered from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith can be performed in other sequences, e.g., serially, in parallel, or in some other manner, as appropriate. Tasks or events can be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems may generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.
[0194] The process, method, and system may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.
[0195] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in the present disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Thus, the claims are not intended to be limited to the implementations shown herein, but should be accorded the widest scope consistent with the present disclosure, the principles, and novel features disclosed herein.
[0196] Certain features described herein in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable subcombination. Furthermore, although features may be described above as acting in a combination and may even be initially claimed as such, one or more features from the claimed combination may in some cases be deleted from the combination and the claimed combination may be directed to a subcombination or variation of the subcombination. No single feature or group of features is required or essential to every embodiment.
[0197] In particular, conditional statements used herein, such as "can," "could," "might," "may," "eg," and the like, are generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not, unless specifically stated otherwise or understood otherwise within the context as used. Thus, such conditional statements are generally not intended to agree that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps should be included or performed in any particular embodiment, with or without authorial input or prompting. The terms "comprising," "including," "having," and the like, are synonymous and used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not its exclusive sense), so that, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, the articles "a," "an," and "the," as used in this application and the appended claims, unless otherwise specified, should be interpreted to mean "one or more" or "at least one."
[0198] As used herein, a phrase referring to a list of items "at least one of" refers to any combination of those items, including single elements. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Transitive phrases such as "at least one of X, Y, and Z" are generally understood differently in the context in which they are used to convey that an item, term, etc. may be at least one of X, Y, or Z, unless specifically stated otherwise. Thus, such transitive phrases are generally not intended to suggest that an embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.
[0199] Similarly, although operations may be depicted in the figures in a particular order, it should be appreciated that such operations need not be performed in the particular order depicted, or in sequential order, or that all of the depicted operations need not be performed to achieve desirable results. Additionally, the figures may diagrammatically depict one or more exemplary processes in the form of a flow chart. However, other operations not depicted may be incorporated within the diagrammatically depicted exemplary methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or during any of the depicted operations. Additionally, operations may be rearranged or reordered in other implementations. In some circumstances, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. ILLUSTRATIVE EMBODIMENTS
[0200] 1. A method comprising:
[0201] displaying a first version of the virtual three-dimensional model via a display of the head mounted device, the first version of the virtual three-dimensional model comprising a first version of the component part;
[0202] requesting data associated with a second version of the virtual three-dimensional model from the host device, the second version of the virtual three-dimensional model comprising a second version of the component part;
[0203] determining whether the first version of the component requires an update based on the second version of the component;
[0204] Pursuant to a determination that the first version of the component requires an update based on the second version of the component,
[0205] requesting data associated with a second version of the component part from the host device;
[0206] displaying a second version of the virtual three-dimensional model via a display of the head mounted device;
[0207] pursuant to a determination based on the second version of the component that the first version of the component does not require an update;
[0208] refraining from requesting data associated with the second version of the component from the host device; A method comprising:
[0209] 2. The method of embodiment 1, wherein the component comprises mesh data.
[0210] 3. The method of embodiment 1, wherein the constituent parts comprise texture data.
[0211] 4. The method of embodiment 1, wherein the host device is a server.
[0212] 5. The method of embodiment 1, wherein the head-mounted device is a first head-mounted device and the host device is a second head-mounted device.
[0213] 6. The method of embodiment 1, wherein the method further comprises storing data associated with the second version of the component part in the memory.
[0214] 7. The method of embodiment 1, wherein the method further comprises extracting data associated with the second version of the component part.
[0215] 8. A system comprising:
[0216] A head-mounted device having a display;
[0217] one or more processors,
[0218] displaying, via a display, a first version of the virtual three-dimensional model, the first version of the virtual three-dimensional model comprising a first version of the component part;
[0219] requesting data associated with a second version of the virtual three-dimensional model from the host device, the second version of the virtual three-dimensional model comprising a second version of the component part;
[0220] determining whether the first version of the component requires an update based on the second version of the component;
[0221] Pursuant to a determination that the first version of the component requires an update based on the second version of the component,
[0222] requesting data associated with a second version of the component part from the host device;
[0223] displaying a second version of the virtual three-dimensional model via the display;
[0224] pursuant to a determination based on the second version of the component that the first version of the component does not require an update;
[0225] refraining from requesting data associated with the second version of the component from the host device; One or more processors configured to execute a method, comprising: A system comprising:
[0226] 9. The system of embodiment 8, wherein the component comprises mesh data.
[0227] 10. The system of embodiment 8, wherein the component comprises texture data.
[0228] 11. The system of embodiment 8, wherein the host device is a server.
[0229] 12. The system of embodiment 8, wherein the head-mounted device is a first head-mounted device and the host device is a second head-mounted device.
[0230] 13. The system of embodiment 8, wherein the method further includes storing data associated with the second version of the component part in the memory.
[0231] 14. The system of embodiment 8, wherein the method further includes extracting data associated with the second version of the component part.
[0232] 15. A non-transitory computer-readable medium that, when executed by one or more processors, causes the one or more processors to:
[0233] displaying a first version of the virtual three-dimensional model via a display of the head mounted device, the first version of the virtual three-dimensional model comprising a first version of the component part;
[0234] requesting data associated with a second version of the virtual three-dimensional model from the host device, the second version of the virtual three-dimensional model comprising a second version of the component part;
[0235] determining whether the first version of the component requires an update based on the second version of the component;
[0236] Pursuant to a determination that the first version of the component requires an update based on the second version of the component,
[0237] requesting data associated with a second version of the component part from the host device;
[0238] displaying a second version of the virtual three-dimensional model via a display of the head mounted device;
[0239] pursuant to a determination based on the second version of the component that the first version of the component does not require an update;
[0240] refraining from requesting data associated with the second version of the component from the host device; A non-transitory computer readable medium storing instructions for carrying out a method, comprising:
[0241] 16. The non-transitory computer-readable medium of embodiment 15, wherein the component comprises mesh data.
[0242] 17. The non-transitory computer-readable medium of embodiment 15, wherein the component portion comprises texture data.
[0243] 18. The non-transitory computer-readable medium of embodiment 15, wherein the host device is a server.
[0244] 19. The non-transitory computer-readable medium of embodiment 15, wherein the head-mounted device is a first head-mounted device and the host device is a second head-mounted device.
[0245] 20. The non-transitory computer-readable medium of embodiment 15, wherein the method further comprises storing data associated with the second version of the component part in the memory.
[0246] 21. The non-transitory computer-readable medium of embodiment 15, wherein the method further comprises extracting data associated with the second version of the component part.
[0247] 22. A method comprising:
[0248] accessing a virtual three-dimensional model stored in a memory;
[0249] determining one or more components of the virtual three-dimensional model;
[0250] storing one or more components in one or more arrays, the one or more components being stored separately from the virtual three-dimensional model;
[0251] receiving a connection request from a head mounted device;
[0252] transmitting a list of available components to a head mounted device;
[0253] receiving a component request from a head mounted device;
[0254] transmitting the requested component parts to the head mounted device based on the component part request; A method comprising:
[0255] 23. The method of embodiment 22, wherein the component comprises mesh data.
[0256] 24. The method of embodiment 22, wherein the constituent parts comprise texture data.
[0257] 25. The method of embodiment 22, wherein the method further comprises compressing one or more of the constituent parts.
[0258] 26. The method of embodiment 22, wherein the virtual three-dimensional model is a first virtual three-dimensional model, and the method further includes generating a copy of the virtual three-dimensional model based on the requested component part.
[0259] 27. The method of embodiment 22, wherein the method further includes a step of displaying the requested component parts via a display of the head-mounted device.
[0260] 28. The method further comprises:
[0261] creating a placeholder for a new virtual three-dimensional model;
[0262] updating a placeholder for a new virtual three-dimensional model based on the requested component; 23. The method of embodiment 22, comprising:
[0263] 29. A system comprising:
[0264] A head-mounted device;
[0265] one or more processors,
[0266] accessing a virtual three-dimensional model stored in a memory;
[0267] determining one or more components of the virtual three-dimensional model;
[0268] storing one or more components in one or more arrays, the one or more components being stored separately from the virtual three-dimensional model;
[0269] receiving a connection request from a head mounted device;
[0270] transmitting a list of available components to a head mounted device;
[0271] receiving a component request from a head mounted device;
[0272] transmitting the requested component parts to the head mounted device based on the component part request; One or more processors configured to execute a method, comprising: A system comprising:
[0273] 30. The system of embodiment 29, wherein the component comprises mesh data.
[0274] 31. The system of embodiment 29, wherein the component comprises texture data.
[0275] 32. The system of embodiment 29, wherein the method further comprises compressing one or more components.
[0276] 33. The system of embodiment 29, wherein the virtual three-dimensional model is a first virtual three-dimensional model, and the method further includes generating a copy of the virtual three-dimensional model based on the requested component parts.
[0277] 34. The system of embodiment 29, wherein the method further includes a step of displaying the requested component parts via a display of the head-mounted device.
[0278] 35. The method further comprises:
[0279] creating a placeholder for a new virtual three-dimensional model;
[0280] updating a placeholder for a new virtual three-dimensional model based on the requested component; 30. The system of embodiment 29, comprising:
[0281] 36. A non-transitory computer-readable medium that, when executed by one or more processors, causes the one or more processors to:
[0282] accessing a virtual three-dimensional model stored in a memory;
[0283] determining one or more components of the virtual three-dimensional model;
[0284] storing one or more components in one or more arrays, the one or more components being stored separately from the virtual three-dimensional model;
[0285] receiving a connection request from a head mounted device;
[0286] transmitting a list of available components to a head mounted device;
[0287] receiving a component request from a head mounted device;
[0288] transmitting the requested component parts to the head mounted device based on the component part request; A non-transitory computer readable medium storing instructions for carrying out a method, comprising:
[0289] 37. The non-transitory computer-readable medium of embodiment 36, wherein the component comprises mesh data.
[0290] 38. The non-transitory computer-readable medium of embodiment 36, wherein the component portion comprises texture data.
[0291] 39. The non-transitory computer-readable medium of embodiment 36, wherein the method further comprises compressing one or more of the constituent parts.
[0292] 40. A non-transitory computer-readable medium as described in embodiment 36, wherein the virtual three-dimensional model is a first virtual three-dimensional model, and the method further includes a step of generating a copy of the virtual three-dimensional model based on the requested component parts.
[0293] 41. The non-transitory computer-readable medium of embodiment 36, wherein the method further comprises displaying the requested component parts via a display of the head-mounted device.
[0294] 42. The method further comprises:
[0295] creating a placeholder for a new virtual three-dimensional model;
[0296] updating a placeholder for a new virtual three-dimensional model based on the requested component; 37. The non-transitory computer-readable medium of embodiment 36, comprising:
[0297] 43. A system comprising:
[0298] A host computing system;
[0299] a client computing system having a head mounted display system; Equipped with
[0300] The host computing system
[0301] accessing a virtual three-dimensional model stored in a memory;
[0302] Decomposing the three-dimensional model into one or more component parts;
[0303] sending a list of one or more components to a client computing system;
[0304] receiving a component part request from a client computing system;
[0305] sending, to a client computing system, one or more component parts corresponding to the component part request; one or more processors configured to execute a method,
[0306] The client computing system
[0307] receiving a list of one or more components from a host computing system;
[0308] sending a component request to a host computing system;
[0309] receiving one or more of the components corresponding to the component request from a host computing system;
[0310] constructing a copy of the virtual three-dimensional model from one or more of the component parts corresponding to the component part requests; comprising one or more processors configured to execute a method, system.
[0311] 44. The system of embodiment 43, wherein the component comprises mesh data.
[0312] 45. The system of embodiment 43, wherein the component comprises texture data.
[0313] 46. The system of embodiment 43, wherein the host computing system is a server.
[0314] 47. The system of embodiment 43, wherein the host computing system comprises a head-mounted display system.
[0315] 48. The system of embodiment 43, wherein the method further comprises storing one or more components in the memory.
[0316] 49. The system of embodiment 43, wherein the method further comprises a step of deploying one or more components.
[0317] 50. A method comprising:
[0318] accessing a virtual three-dimensional model stored in a memory;
[0319] Decomposing the three-dimensional model into one or more component parts;
[0320] transmitting the list of one or more components to a client computing system having a head mounted display;
[0321] receiving a component part request from a client computing system;
[0322] sending, to a client computing system, one or more component parts corresponding to the component part request;
[0323] receiving a list of one or more components from a host computing system;
[0324] sending a component request to a host computing system;
[0325] receiving one or more of the components corresponding to the component request from a host computing system;
[0326] constructing a copy of the virtual three-dimensional model from one or more of the component parts corresponding to the component part requests; A method comprising:
[0327] 51. The method of embodiment 50, wherein the component comprises mesh data.
[0328] 52. The method of embodiment 50, wherein the constituent parts comprise texture data.
[0329] 53. The method of embodiment 50, wherein the host computing system is a server.
[0330] 54. The method of embodiment 50, wherein the host computing system comprises a head-mounted display system.
[0331] 55. The method of embodiment 50, wherein the method further comprises storing one or more components in a memory.
[0332] 56. The method of embodiment 50, wherein the method further comprises a step of deploying one or more components.
[0333] 57. A non-transitory computer-readable medium that, when executed by one or more processors, causes the one or more processors to:
[0334] accessing a virtual three-dimensional model stored in a memory;
[0335] Decomposing the three-dimensional model into one or more component parts;
[0336] transmitting the list of one or more components to a client computing system having a head mounted display;
[0337] receiving a component part request from a client computing system;
[0338] sending, to a client computing system, one or more component parts corresponding to the component part request;
[0339] receiving a list of one or more components from a host computing system;
[0340] sending a component request to a host computing system;
[0341] receiving one or more of the components corresponding to the component request from a host computing system;
[0342] constructing a copy of the virtual three-dimensional model from one or more of the component parts corresponding to the component part requests; A non-transitory computer-readable medium storing one or more instructions that cause a method to be performed, comprising:
[0343] 58. The non-transitory computer-readable medium of embodiment 57, wherein the component comprises mesh data.
[0344] 59. The non-transitory computer-readable medium of embodiment 57, wherein the component portion comprises texture data.
[0345] 60. The non-transitory computer-readable medium of embodiment 57, wherein the host computing system is a server.
[0346] 61. The non-transitory computer-readable medium of embodiment 57, wherein the host computing system comprises a head-mounted display system.
[0347] 62. The non-transitory computer-readable medium of embodiment 57, wherein the method further comprises storing one or more components in the memory.
[0348] 63. The non-transitory computer-readable medium of embodiment 57, wherein the method further comprises deploying one or more components.
[0349] 64. A system comprising:
[0350] A host computing system, comprising:
[0351] accessing a first virtual three-dimensional model via a host assets module;
[0352] Identifying, via a decomposition module, one or more component parts;
[0353] copying, via a decomposition module, one or more constituent parts into a library;
[0354] accessing one or more components via a host transmission module;
[0355] transmitting, via a host transmission module, one or more components to a client computing system; A host computing system comprising one or more processors configured to execute a method comprising:
[0356] 1. A client computing system, the client computing system comprising a head mounted display system, the client computing system comprising:
[0357] receiving one or more components from a host computing system via a client transmission module;
[0358] creating an empty object via a configuration module;
[0359] adding, via a configuration module, one or more data types to the empty object;
[0360] adding, via a composition module, one or more composition parts to the empty object;
[0361] storing, via a client asset module, a copy of the virtual three-dimensional model; A client computing system comprising one or more processors configured to execute a method comprising: Equipped with
[0362] the host computing system and the client computing system are communicatively connected via a communications link; system.
[0363] 65. The system of embodiment 64, wherein the component comprises mesh data.
[0364] 66. The system of embodiment 64, wherein the component comprises texture data.
[0365] 67. The system of embodiment 64, wherein the host computing system is a server.
[0366] 68. The system of embodiment 64, wherein the host computing system comprises a head-mounted display system.
[0367] 69. The system of embodiment 64, wherein the method further comprises storing one or more components in the memory.
[0368] 70. The system of embodiment 64, wherein the method further comprises a step of deploying one or more components.
[0369] 71. A method comprising:
[0370] A step in a host computing system, the host computing system comprising:
[0371] accessing a first virtual three-dimensional model via a host assets module;
[0372] Identifying, via a decomposition module, one or more component parts;
[0373] copying, via a decomposition module, one or more constituent parts into a library;
[0374] accessing one or more components via a host transmission module;
[0375] transmitting, via a host transmission module, one or more components to a client computing system; The method includes the steps of:
[0376] 13. A method for controlling a head mounted display system comprising:
[0377] receiving one or more components from a host computing system via a client transmission module;
[0378] creating an empty object via a configuration module;
[0379] adding, via a configuration module, one or more data types to the empty object;
[0380] adding, via a composition module, one or more composition parts to the empty object;
[0381] storing, via a client asset module, a copy of the virtual three-dimensional model; The method includes the steps of: Including,
[0382] the host computing system and the client computing system are communicatively connected via a communications link; method.
[0383] 72. The method of embodiment 71, wherein the component comprises mesh data.
[0384] 73. The method of embodiment 71, wherein the constituent parts comprise texture data.
[0385] 74. The method of embodiment 71, wherein the host computing system is a server.
[0386] 75. The method of embodiment 71, wherein the host computing system comprises a head-mounted display system.
[0387] 76. The method of embodiment 71, wherein the method further comprises storing one or more components in a memory.
[0388] 77. The method of embodiment 71, wherein the method further comprises a step of deploying one or more components.
[0389] 78. A non-transitory computer-readable medium that, when executed by one or more processors, causes the processors to:
[0390] In the host computing system,
[0391] accessing a first virtual three-dimensional model via a host assets module;
[0392] Identifying, via a decomposition module, one or more component parts;
[0393] copying, via a decomposition module, one or more constituent parts into a library;
[0394] accessing one or more components via a host transmission module;
[0395] transmitting, via a host transmission module, one or more components to a client computing system;
[0396] 1. A client computing system comprising a head mounted display system, comprising:
[0397] receiving one or more components from a host computing system via a client transmission module;
[0398] creating an empty object via a configuration module;
[0399] adding, via a configuration module, one or more data types to the empty object;
[0400] adding, via a composition module, one or more composition parts to the empty object;
[0401] storing, via a client asset module, a copy of the virtual three-dimensional model;
[0402] a host computing system and a client computing system communicatively coupled via a communication link; A non-transitory computer-readable medium.
[0403] 79. The non-transitory computer-readable medium of embodiment 78, wherein the component comprises mesh data.
[0404] 80. The non-transitory computer-readable medium of embodiment 78, wherein the component portion comprises texture data.
[0405] 81. The non-transitory computer-readable medium of embodiment 78, wherein the host computing system is a server.
[0406] 82. The non-transitory computer-readable medium of embodiment 78, wherein the host computing system comprises a head-mounted display system.
[0407] 83. The non-transitory computer-readable medium of embodiment 78, wherein the method further comprises storing one or more components in the memory.
[0408] 84. The non-transitory computer-readable medium of embodiment 78, wherein the method further comprises deploying one or more components.
[0409] 85. A system comprising:
[0410] 1. A host computing system comprising:
[0411] accessing a first virtual three-dimensional model via a load module;
[0412] identifying, via a host library manager module, one or more component parts;
[0413] copying, via a host library manager module, one or more component parts to a library;
[0414] storing, via a host world module, the first virtual three-dimensional model;
[0415] Rendering the first virtual three-dimensional model via a host rendering path module;
[0416] accessing one or more components via a server module;
[0417] transmitting, via the server module, one or more components to a client computing system; A host computing system comprising one or more processors configured to execute a method comprising:
[0418] 1. A client computing system, the client computing system comprising a head mounted display system, the client computing system comprising:
[0419] receiving, via a client module, one or more components from a host computing system;
[0420] Generating a copy of the virtual three-dimensional model, the generating the virtual three-dimensional model comprising:
[0421] creating an empty object via a client library manager module;
[0422] adding, via a client library manager module, one or more components to the empty object; Steps and
[0423] storing, via a client world module, a copy of the virtual three-dimensional model;
[0424] rendering a copy of the virtual three-dimensional model via a client rendering path module; A client computing system comprising one or more processors configured to execute a method comprising:
[0425] wherein the host computing system and the client computing system are communicatively connected via a communication link; system.
[0426] 86. The system of embodiment 85, wherein the component comprises mesh data.
[0427] 87. The system of embodiment 85, wherein the component comprises texture data.
[0428] 88. The system of embodiment 85, wherein the host computing system is a server.
[0429] 89. The system of embodiment 85, wherein the host computing system comprises a head-mounted display system.
[0430] 90. The system of embodiment 85, wherein the method further includes storing one or more components in the memory.
[0431] 91. The system of embodiment 85, wherein the method further includes a step of deploying one or more components.
[0432] 92. A method comprising:
[0433] In the host computing system,
[0434] accessing a first virtual three-dimensional model via a load module;
[0435] identifying, via a host library manager module, one or more component parts;
[0436] copying, via a host library manager module, one or more component parts to a library;
[0437] storing, via a host world module, the first virtual three-dimensional model;
[0438] Rendering the first virtual three-dimensional model via a host rendering path module;
[0439] accessing one or more components via a server module;
[0440] transmitting, via the server module, one or more components to a client computing system;
[0441] 1. A client computing system comprising a head mounted display system, comprising:
[0442] receiving, via a client module, one or more components from a host computing system;
[0443] Generating a copy of the virtual three-dimensional model, the step of generating a copy of the virtual three-dimensional model comprising:
[0444] creating an empty object via a client library manager module;
[0445] adding, via a client library manager module, one or more components to the empty object; Steps and
[0446] storing, via a client world module, a copy of the virtual three-dimensional model;
[0447] rendering a copy of the virtual three-dimensional model via a client rendering path module;
[0448] wherein the host computing system and the client computing system are communicatively connected via a communication link; method.
[0449] 93. The method of embodiment 92, wherein the component comprises mesh data.
[0450] 94. The method of embodiment 92, wherein the constituent parts comprise texture data.
[0451] 95. The method of embodiment 92, wherein the host computing system is a server.
[0452] 96. The method of embodiment 92, wherein the host computing system comprises a head-mounted display system.
[0453] 97. The method of embodiment 92, wherein the method further comprises storing one or more components in a memory.
[0454] 98. The method of embodiment 92, wherein the method further comprises a step of deploying one or more components.
[0455] 99. A non-transitory computer-readable medium storing one or more instructions that, when executed by one or more processors, cause the processors to perform a method,
[0456] A host computing system, comprising:
[0457] accessing a first virtual three-dimensional model via a load module;
[0458] identifying, via a host library manager module, one or more component parts;
[0459] copying, via a host library manager module, one or more component parts to a library;
[0460] storing, via a host world module, the first virtual three-dimensional model;
[0461] Rendering the first virtual three-dimensional model via a host rendering path module;
[0462] accessing one or more components via a server module;
[0463] transmitting, via the server module, one or more components to a client computing system; A host computing system comprising one or more processors configured to execute a method comprising:
[0464] 1. A client computing system, the client computing system comprising a head mounted display system, the client computing system comprising:
[0465] receiving, via a client module, one or more components from a host computing system;
[0466] Generating a copy of the virtual three-dimensional model, the step of generating a copy of the virtual three-dimensional model comprising:
[0467] creating an empty object via a client library manager module;
[0468] adding, via a client library manager module, one or more components to the empty object; Steps and
[0469] storing, via a client world module, a copy of the virtual three-dimensional model;
[0470] rendering a copy of the virtual three-dimensional model via a client rendering path module;
[0471] A client computer comprising one or more processors configured to execute a method including: Equipped with the host computing system and the client computing system are communicatively connected via a communications link; A non-transitory computer-readable medium.
[0472] 100. The non-transitory computer-readable medium of embodiment 99, wherein the component comprises mesh data.
[0473] 101. The non-transitory computer-readable medium of embodiment 99, wherein the component portion comprises texture data.
[0474] 103. The non-transitory computer-readable medium of embodiment 99, wherein the host computing system is a server.
[0475] 104. The non-transitory computer-readable medium of embodiment 99, wherein the host computing system comprises a head-mounted display system.
[0476] 105. The non-transitory computer-readable medium of embodiment 99, wherein the method further comprises storing one or more components in the memory.
[0477] 106. The non-transitory computer-readable medium of embodiment 99, wherein the method further comprises deploying one or more components.
[0478] 107. A system comprising:
[0479] a non-transitory computer readable memory that stores a three-dimensional model comprising one or more components;
[0480] a host computing system configured to access the three-dimensional model from the non-transitory computer readable memory; and
[0481] storing one of the one or more components of the three-dimensional model in an array stored in a non-transitory computer readable memory;
[0482] packaging the array in a package in a library of a host computing system; A host computing system comprising one or more processors configured to execute a method comprising: A system comprising:
[0483] 108. The system of embodiment 107, wherein the component comprises mesh data.
[0484] 109. The system of embodiment 107, wherein the component comprises texture data.
[0485] 110. The system of embodiment 107, wherein the host computing system is a server.
[0486] 111. The system of embodiment 107, wherein the host computing system comprises a head-mounted display system.
[0487] 112. The system of embodiment 107, wherein the method further includes storing one or more components in the memory.
[0488] 113. The system of embodiment 107, wherein the method further includes a step of deploying one or more components.
[0489] 114. A method comprising:
[0490] storing, via a non-transitory computer readable memory, the three-dimensional model comprising the one or more component parts;
[0491] accessing the three-dimensional model from a non-transitory computer readable memory;
[0492] storing one of the one or more components of the three-dimensional model in an array stored in a non-transitory computer readable memory;
[0493] packaging the array in a package in a library of a host computing system; A method comprising:
[0494] 115. The method of embodiment 114, wherein the component comprises mesh data.
[0495] 116. The method of embodiment 114, wherein the component comprises texture data.
[0496] 117. The method of embodiment 114, wherein the non-transitory computer-readable memory is part of the server.
[0497] 118. The method of embodiment 114, wherein the non-transitory computer-readable memory is part of the head-mounted display system.
[0498] 119. The method of embodiment 114, wherein the method further comprises storing one or more components in a memory.
[0499] 120. The method of embodiment 114, wherein the method further comprises the step of deploying one or more components.
[0500] 121. A non-transitory computer-readable medium that, when executed by one or more processors, causes the one or more processors to:
[0501] storing, via a non-transitory computer readable memory, the three-dimensional model comprising the one or more component parts;
[0502] accessing the three-dimensional model from a non-transitory computer readable memory;
[0503] storing one of the one or more components of the three-dimensional model in an array stored in a non-transitory computer readable memory;
[0504] packaging the array in a package in a library of a host computing system; A non-transitory computer-readable medium storing one or more instructions that cause a method to be performed, comprising:
[0505] 122. The non-transitory computer-readable medium of embodiment 121, wherein the component comprises mesh data.
[0506] 123. The non-transitory computer-readable medium of embodiment 121, wherein the component portion comprises texture data.
[0507] 124. The non-transitory computer-readable medium of embodiment 121, wherein the non-transitory computer-readable memory is part of the server.
[0508] 125. The non-transitory computer-readable medium of embodiment 121, wherein the non-transitory computer-readable memory is part of a head-mounted display system.
[0509] 126. The non-transitory computer-readable medium of embodiment 121, wherein the method further comprises storing one or more components in the memory.
[0510] 127. The non-transitory computer-readable medium of embodiment 121, wherein the method further comprises deploying one or more components.
[0511] Examples of components may be geographic data, material data, a vertex table, one or more textures, a triangle index, or any other data used to ultimately define a complete representation of the 3D model.
Claims
1. 1. A system comprising: one or more processors configured to perform the method; The method comprises: Accessing a virtual three-dimensional model stored in a memory; Decomposing the virtual three-dimensional model into constituent parts, The component parts are mesh data, texture data, or a combination thereof; and based on first data from one or more sensors of the head mounted display system; determining whether a client computing system is in communication with a host computing system; transmitting, in accordance with a determination that the client computing system is in communication with the host computing system, a library comprising the components to the client computing system, the library corresponding to the virtual three-dimensional model; receiving a component request from the client computing system; transmitting to the client computing system one or more of the components corresponding to the component request; Including, The client computing system is configured to construct a view of the virtual three-dimensional model based on the one or more of the components and further based on second data from the one or more sensors of the head mounted display system.
2. The system of claim 1 , wherein the system comprises the host computing system.
3. The system described in claim 1, further comprising the head-mounted display system.
4. The system of claim 1 , wherein the method further comprises storing the one or more of the components in the memory.
5. The system of claim 1 , wherein the method further comprises compressing the one or more of the components.
6. The system of claim 1 , wherein a physically integrated device comprises the system and further comprises the client computing system.
7. Determining whether the client computing system is in communication with the system includes: receiving a connection request from the client computing system; accepting the connection request from the client computing system; The system of claim 1 , comprising:
8. 1. A system comprising: one or more processors configured to perform the method; The method comprises: receiving a library of component parts from a host computing system, the library of component parts comprising component parts corresponding to the virtual three-dimensional model; The component parts are mesh data, texture data, or a combination thereof; and based on first data from one or more sensors of the head mounted display system; determining one or more parts to be included in a component part request based on a comparison of the library of component parts to a client library of component parts; sending the component request to the host computing system; receiving one or more of the components corresponding to the component request from the host computing system; constructing a view of the virtual three-dimensional model based on the received one or more of the components and further based on second data from the one or more sensors of the head mounted display system; Including, the system.
9. The system of claim 8 , wherein the system comprises the head mounted display system.
10. The system of claim 8 , wherein the method further comprises storing the one or more of the components in a memory.
11. The system of claim 8 , wherein the method further comprises disassembling the one or more of the components.
12. The system of claim 8 , wherein a physically integrated device comprises the system and further comprises the host computing system.
13. The system of claim 8 , wherein determining the one or more parts to be included in the configuration part request is further based on a version number associated with the one or more parts.
14. The method comprises: sending a connection request to the host computing system; connecting to the host computing system in response to receiving an indication of acceptance of the connection request from the host computing system; The system of claim 8 further comprising:
Citation Information
Patent Citations
Method and device for supporting design of control panel
JP1994314318A
Plant design support system
JP1999328243A
Parts sales system
JP2002092349A
Electronic catalog device for mold part
JP2004054363A
Design support system, design support method, and program thereof
JP2014228921A
