A Skeletal System for Animating Virtual Avatars

Abstract

To provide a skeleton system for animating a virtual avatar.SOLUTION: A skinning parameter used for animating a virtual avatar can include mesh weighting of a skeleton and joint conversion. A system and a method are provided for determining the skinning parameter by using an optimization process according to a restriction based on a relation between skeleton joints to be understood by a person or anatomically motivated. An input to the optimization process can contain a higher order skeleton, and the applied restriction can dynamically change during the optimization. The skinning parameter can be used in a linear blend skinning (LBS) usage in an augmented reality.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to U.S. Patent Application No. 62 / 674,908, filed May 22, 2018, and entitled "SKELETAL SYSTEMS FOR ANIMATING VIRTUAL AVATARS," which is incorporated by reference in its entirety.

[0002] The present disclosure relates to virtual and augmented reality, including mixed reality, imaging, and visualization systems, and more particularly to skinning systems and methods for animating virtual characters such as avatars. [Background technology]

[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality", "augmented reality" and "mixed reality" experiences, in which digitally reproduced images are presented to a user in a manner that appears or can be perceived as real. Virtual reality (VR) scenarios typically involve the presentation of computer-generated virtual image information without transparency to other actual real-world visual inputs. Augmented reality (AR) scenarios typically involve the presentation of virtual image information as an augmentation to the visualization of the real world around the user. Mixed reality (MR) is a type of augmented reality in which physical and virtual objects can coexist and interact in real time. The systems and methods disclosed herein address various challenges associated with VR, AR and MR technologies. Summary of the Invention [Means for solving the problem]

[0004] Skinning parameters used to animate the virtual avatar can include skeletal mesh weights and joint transformations. Systems and methods are provided for determining the skinning parameters using an optimization process according to constraints based on human-understandable or anatomically motivated relationships between skeletal joints. Inputs to the optimization process can include high-order skeletons, and the applied constraints can change dynamically during optimization. The skinning parameters can be used in linear blend skinning (LBS) applications in augmented reality, games, movies, or visual effects.

[0005] Examples of systems and methods for improving or optimizing mesh (e.g., skin) weighting and joint transformations for animation of virtual avatars are provided. The joint transformations can include multiple (e.g., six) degrees of freedom. The systems and methods can receive target pose training data from a variety of input sources (e.g., photogrammetric scans, expertly crafted sculptures, simulations, or biometrically derived models) and can generate high-fidelity weighting and transformations for use in linear blend skinning (LBS). The systems and methods can compute the weighting and joint transformations (e.g., rotations and translations) by solving a constrained optimization problem. The optimization can be constrained to only those solutions that provide high fidelity under generalized conditions, including the use of novel or real-time captured animation capabilities. This can result in a reduction in the number of samples (often expensive and time-consuming to generate) or joint transformations used to meet a quality metric.

[0006] Although certain embodiments of the present systems and methods are described in the context of augmented, mixed, or virtual reality, they are not so limited and can be applied to games, movies, or visual effects (VFx).

[0007] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this summary nor the following detailed description purports to define or limit the scope of the inventive subject matter. For example, the present application provides the following: (Item 1) 1. A system comprising: 1. A non-transitory computer storage device, comprising: a target mesh representing a pose of a virtual avatar, the target mesh comprising a plurality of target mesh vertices; An initial skeleton having a plurality of joints; Skinning parameters that represent the deformation of the skeleton to the posture of the virtual avatar; a set of constraints associated with said skinning parameters or said joints; a non-transitory computer storage device configured to store the a hardware processor in communication with the non-transitory computer storage device, the hardware processor comprising: accessing the target mesh, the initialization skeleton, and the set of constraints; performing a constrained optimization process to determine the skinning parameters that provide a match to the target mesh, the constrained optimization process subject to the set of constraints; Outputting the skinning parameters. a hardware processor programmed to A system comprising: (Item 2) 2. The system of claim 1, wherein the target mesh is determined from a photographic scan of a human or animal subject in a certain pose. (Item 3) Item 4. The system of item 1, wherein the target mesh is determined from an expert sculpt, a computational simulation of the pose, or a biometrically derived model. 2. The system of claim 1, wherein the initialization skeleton comprises a higher-order skeleton. (Item 5) 2. The system of claim 1, wherein the initialized skeleton comprises a low-order skeleton in a base or neutral pose. (Item 6) 2. The system of claim 1, wherein the initialized skeleton further comprises a skin cluster mapping of vertices to the joints of the skeleton. (Item 7) Item 2. The system of item 1, wherein the skinning parameters comprise a weighting map that measures the amount of influence of the joints on the vertices. (Item 8) 2. The system of claim 1, wherein the skinning parameters comprise six degree-of-freedom parameters associated with the movement of each joint. (Item 9) Item 10. The system of item 8, wherein the six degree of freedom parameters include rotation and translation. Item 11. The system of item 1, wherein the hardware processor is further programmed to calculate a radial basis function (RBF) formula associated with the skinning parameters. 2. The system of claim 1, wherein the constrained optimization process is programmed to implement a falloff configured to blend results from a first domain into a second domain. (Item 12) 2. The system of claim 1, wherein the constrained optimization process comprises linear or quadratic programming. (Item 13) Item 2. The system of item 1, wherein the constrained optimization process comprises a nonlinear solver that utilizes first or second derivative estimates. (Item 14) The above set of constraints is: A non-negativity constraint on the weights of the weight map, A regularization constraint on the total influence of all joints on a vertex, the maximum number of non-zero weights for any vertex, Constraints in which the rotation parameters associated with the joint translations represent 3D rotations, or Constraints on the rotation parameters associated with joint movements to have orthogonal matrices Item 1. The system of item 1, comprising one or more of the following: (Item 15) The above set of constraints is: A constraint that restricts child joints to be a fixed distance from their parent joints, a smoothness constraint that restricts nearby vertices of the mesh to have similar weights in the weight map; Geometric constraints that represent the anatomical properties of the joint; Geometric constraints representing joint length, lateral size, or range of motion (ROM); A constraint that locks a subset of the joints from being updated during the constrained optimization process. Constraints that joint positions can only translate along specific axes, constraints that lock the joint properties during the constrained optimization process; a constraint that sets the weight for a vertex to zero if the joint is located more than a threshold distance from said vertex; or A constraint incorporating a segmentation set that relates a set of one or more vertices to a set of one or more joints. Item 1. The system of item 1, comprising one or more of the following: (Item 16) 2. The system of claim 1, wherein the set of constraints comprises constraints incorporating a segmentation set that associates a set of one or more vertices with a set of one or more joints, and the constrained optimization process is programmed to act only on joints or vertices within the segmentation set. (Item 17) 2. The system of claim 1, wherein the hardware processor is programmed to dynamically update the set of constraints during the constrained optimization process. (Item 18) 1. A method comprising: under the control of a hardware processor in communication with the non-transitory storage device; accessing a target mesh representing a target pose for a virtual avatar; accessing an initialization mesh for use in matching said target mesh; Accessing a set of constraints; and performing a constrained optimization to determine skinning parameters representative of the target pose, the constrained optimization being performed according to the set of constraints, the constrained optimization being initialized with the initialization mesh; Outputting the skinning parameters. A method comprising: (Item 19) 1. A method for automatically painting weights and determining bone locations for linear blend skinning, comprising: under the control of a hardware processor in communication with the non-transitory storage device; Solving an error minimization equation based on an initial set of linear blend skinning weights, a high-order skeleton, and a target pose. Including, A method, wherein the above equations are subject to constraints understandable by a human. (Item 20) 1. A method comprising: Under control of a hardware processor in communication with the non-transitory storage device; Using an algorithm to algorithmically derive skin weights and joint positions Including, The inputs to the algorithm are an initial set of linear blend skinning weights, a high-order skeleton, and a target pose. A method, wherein the algorithm is subject to a set of constraints that are based on one or more human understandable constraints. (Item 21) 1. A system for automatically painting weights and determining bone locations for linear blend skinning, the system comprising: A hardware processor in communication with a non-transitory storage device, the hardware processor comprising: Solve an error minimization equation based on an initial set of linear blend skinning weights, a high-order skeleton, and a target pose. A hardware processor that is programmed to Equipped with The above equations are subject to constraints that are understandable by a human,system. (Item 22) 1. A system comprising: A hardware processor in communication with a non-transitory storage device, the hardware processor comprising: Using an algorithm to algorithmically derive skin weights and joint positions A hardware processor that is programmed to Equipped with The inputs to the algorithm are an initial set of linear blend skinning weights, a high-order skeleton, and a target pose. The algorithm is subject to a set of constraints that are based on one or more human understandable constraints. [Brief description of the drawings]

[0008] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims.

[0009] [Figure 1] FIG. 1 depicts an illustration of a mixed reality scenario involving a virtual reality object and a physical object viewed by a person.

[0010] [Diagram 2] FIG. 2 illustrates diagrammatically an embodiment of a wearable system.

[0011] [Diagram 3] FIG. 3 diagrammatically illustrates example components of a wearable system.

[0012] [Figure 4] FIG. 4 diagrammatically illustrates an example of a waveguide stack of a wearable device for outputting image information to a user.

[0013] [Diagram 5] FIG. 5 is a process flow diagram of an embodiment of a method for interacting with a virtual user interface.

[0014] [Figure 6A] FIG. 6A is a block diagram of another embodiment of a wearable system that may include an avatar processing and rendering system.

[0015] [Figure 6B] FIG. 6B illustrates example components of an avatar processing and rendering system.

[0016] [Figure 7] FIG. 7 is a block diagram of an embodiment of a wearable system, including various inputs into the wearable system.

[0017] [Figure 8] FIG. 8 is a process flow diagram of an embodiment of a method for rendering virtual content in relation to recognized objects.

[0018] [Figure 9A] FIG. 9A diagrammatically illustrates an overall system diagram depicting multiple wearable systems interacting with each other.

[0019] [Figure 9B] FIG. 9B illustrates an exemplary telepresence session.

[0020] [Figure 10] FIG. 10 illustrates an example of an avatar as perceived by a user of a wearable system.

[0021] [Figure 11A] Figure 11A illustrates an example of a low-order core (or base) scaffold. Higher order scaffolds can be added to the low-order core scaffold (see, for example, the example shown in Figure 11B).

[0022] [Figure 11B] 11B illustrates an example of a skeleton for animating an avatar. The skeleton includes a core (or base) skeleton (shown in solid lines) with higher order biomechanical linkages (shown in dashed lines) between a hierarchical set of interconnected joints.

[0023] [Figure 12] FIG. 12 illustrates an example of joint disassembly for a hand deformity.

[0024] [Figure 13] FIG. 13 illustrates an example of articular disassembly for jaw deformation.

[0025] [Figure 14] FIG. 14 is a flow chart for an exemplary method for joint disassembly.

[0026] Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] overview A virtual avatar may be a virtual representation of a real or fictional person (or fictional or anthropomorphized object) within an AR / VR / MR environment. For example, during a telepresence session in which two AR / VR / MR users interact with each other, the viewer may perceive another user's avatar within the viewer's environment, thereby creating a tangible sense of the other user's presence within the viewer's environment. Avatars may also provide a way for users to interact with each other and do things together within a shared virtual environment. For example, a student attending an online class may perceive and interact with avatars of other students or teachers in the virtual classroom. As another example, a user playing a game within an AR / VR / MR environment may see and interact with avatars of other players in the game.

[0028] Embodiments of the disclosed systems and methods may provide improved animation of avatars and more realistic interaction between a user of a wearable system and avatars in the user's environment. Although the examples in this disclosure generally describe animating humanoid avatars, similar techniques can also be applied to animals, fictional creatures, objects, and the like.

[0029] The wearable device can include a display for presenting an interactive VR / AR / MR environment, including a high-fidelity digital avatar. Creating a high-fidelity digital avatar can take weeks or months of work by a specialized team and can utilize numerous high-quality digitized photo scans of a human model. Embodiments of the disclosed technology have the ability to create a high-quality or high-fidelity avatar (or digital representation in general) of any human, animal, character, or object. To accomplish this, embodiments of the disclosed process are faster and less resource intensive, while still maintaining accurate output.

[0030] As an example, a digital representation of a human (or generally any animal or deformable object such as clothing or hair) may include a skeleton and an overlying mesh (e.g., to represent the outer surface, which may be skin, clothing, etc.). Bones may be connected to other bones at joints, and bones may have certain mesh vertices assigned to them such that as the bone moves, the assigned vertices move automatically with the bone. Each vertex may have multiple bone assignments, with the vertex's motion being interpolated from a combination of bone movements. This initial movement is called a "skin cluster" and generally captures the coarse movement. (Note that bones and skeletons are digital constructs and do not necessarily correspond to actual bones in the human body.) A subsequent step in modeling a human (a virtual representation, sometimes also referred to herein as an avatar) may be needed to capture the finer movement of the skin, sometimes also referred to herein as a surface or mesh. This subsequent step represents the variance from the initial coarse movement to capture the finer movement of the skin.

[0031] To animate avatars, wearable devices can utilize the Linear Blend Skinning (LBS) technique, in which the vertices of the avatar's mesh are transformed based on a linearly weighted sum of the joint rotations and translations of the underlying skeleton. Computing the joint weights, rotations, and translations (collectively referred to as skinning parameters) for a range of avatar poses is a challenging problem.

[0032] A human animator can assign mesh vertices to bones by setting the corresponding weighting of each vertex for each bone. Human animators make these assignments based on the animator's unique subjective artistic vision, and there are as many manual skinning techniques as there are animators. This assignment process can be a laborious, time-consuming process that is not well suited to creating skins for use in real-time applications such as augmented, composite, or virtual reality.

[0033] This application describes computational systems and methods that may determine a target pose from various sources (e.g., photogrammetric scans of a model, expert sculpts, simulations, or biometrically derived models) and calculate skinning parameters for that pose. The systems and methods may apply constrained optimization techniques that are seeded with an initial target pose, a higher-order skeleton, and skin clusters and generate skinning parameters according to human-understandable and biologically motivated constraints (e.g., joint-to-joint lengths may be constrained because real-world bones do not easily change their length). The systems and methods may be particularly applicable in real-time contexts and may reduce or minimize the need for user (e.g., animator) input. These computational systems and methods are based on concrete, objective, and non-conventional rules that may be applied algorithmically to generate skinning parameters in a manner that differs from how a human animator subjectively performs animations using the animator's subjective artistic vision.

[0034] Thus, various implementations of systems and methods for computing skinning parameters for automated avatar animation will be provided below. (Example of a 3D display for a wearable system)

[0035] A wearable system (also referred to herein as an augmented reality (AR) system) can be configured to present 2D or 3D virtual images to a user. The images may be still images, frames of video, or videos, in combination or the like. At least a portion of the wearable system can be implemented on a wearable device, which may present a VR, AR, or MR environment, alone or in combination, for user interaction. A wearable device can be used synonymously with an AR device (ARD). Additionally, for purposes of this disclosure, the term "AR" is used synonymously with the term "MR."

[0036] Figure 1 depicts an illustration of a mixed reality scenario with certain virtual reality objects and certain physical objects viewed by a person. In Figure 1, an MR scene 100 is depicted in which a user of the MR technology sees a real-world park-like setting 110 featuring people, trees, buildings in the background, and a concrete platform 120. In addition to these items, the user of the MR technology also perceives that they "see" a robotic figure 130 standing on the real-world platform 120, and a flying cartoon-like avatar character 140 that appears to be an anthropomorphic bumblebee, although these elements do not exist in the real world.

[0037] In order for a 3D display to produce a true depth sensation, and more specifically, a simulated sensation of surface depth, it may be desirable to generate, for each point in the display's field of view, an accommodation response that corresponds to that point's virtual depth. If the accommodation response to a display point does not correspond to that point's virtual depth as determined by the binocular depth cues of convergence and stereopsis, the human eye may experience accommodation conflicts, resulting in unstable imaging, deleterious eye strain, headaches, and, in the absence of accommodative information, a near-complete lack of surface depth.

[0038] VR, AR, and MR experiences can be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images may be different for each depth plane (e.g., providing slightly different presentations of a scene or object) and may be focused separately by the viewer's eyes, thereby serving to provide depth cues to the user based on ocular accommodation required to focus on different image features of a scene located on different depth planes, or based on observing different image features on different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a believable perception of depth.

[0039] FIG. 2 illustrates an example of a wearable system 200, which can be configured to provide an AR / VR / MR scene. The wearable system 200 can also be referred to as an AR system 200. The wearable system 200 includes a display 220 and various mechanical and electronic modules and systems to support the functionality of the display 220. The display 220 can be coupled to a frame 230, which is wearable by a user, wearer, or viewer 210. The display 220 can be positioned in front of the eyes of the user 210. The display 220 can present AR / VR / MR content to the user. The display 220 can comprise a head-mounted display (HMD) worn on the user's head.

[0040] In some embodiments, a speaker 240 is coupled to the frame 230 and positioned adjacent the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent the user's other ear canal to provide stereo / shapeable sound control). The display 220 can include an audio sensor (e.g., a microphone) 232 to detect audio streams from the environment and capture ambient sounds. In some embodiments, one or more other audio sensors, not shown, are positioned to provide stereo sound reception. Stereo sound reception can be used to determine the location of a sound source. The wearable system 200 can perform voice or speech recognition on the audio stream.

[0041] The wearable system 200 may include an outwardly facing imaging system 464 (shown in FIG. 4 ) that observes the world in the user's surrounding environment. The wearable system 200 may also include an inwardly facing imaging system 462 (shown in FIG. 4 ) that may track the user's eye movements. The inwardly facing imaging system may track either one eye's movements or both eyes' movements. The inwardly facing imaging system 462 may be mounted to the frame 230 and may be in electrical communication with the processing module 260 or 270 that may process image information obtained by the inwardly facing imaging system and determine, for example, the pupil diameter or orientation of the user's 210 eyes, eye movements, or eye posture. The inwardly facing imaging system 462 may include one or more cameras. For example, at least one camera may be used to image each eye. Images obtained by the cameras may be used to determine pupil size or eye posture for each eye separately, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye.

[0042] As an example, the wearable system 200 can obtain an image of the user's posture using an outward facing imaging system 464 or an inward facing imaging system 462. The image may be a still image, a frame of a video, or a video.

[0043] The display 220 can be operably coupled (250) to a local data processing module 260, which can be mounted in a variety of configurations, such as fixedly attached to the frame 230, by wired leads or a wireless connection, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise removably attached to the user 210 (e.g., in a backpack configuration, in a belt-attached configuration).

[0044] The local processing and data module 260 may comprise a hardware processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data may include a) data captured from sensors (e.g., that may be operatively coupled to the frame 230 or otherwise attached to the user 210), such as an image capture device (e.g., a camera in an inward-facing or outward-facing imaging system), audio sensor (e.g., a microphone), an inertial measurement unit (IMU), an accelerometer, a compass, a global positioning system (GPS) unit, a wireless device, or a gyroscope, or b) data obtained or processed using the remote processing module 270 or remote data repository 280, possibly for passing to the display 220 after processing or retrieval. The local processing and data module 260 may be operatively coupled to a remote processing module 270 or a remote data repository 280 by a communications link 262 or 264, such as via a wired or wireless communication link, such that these remote modules are available as resources to the local processing and data module 260. In addition, the remote processing module 280 and the remote data repository 280 may be operatively coupled to each other.

[0045] In some embodiments, the remote processing module 270 may comprise one or more processors configured to analyze and process the data or image information. In some embodiments, the remote data repository 280 may comprise a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use from the remote module. (Example Components of a Wearable System)

[0046] FIG. 3 diagrammatically illustrates example components of a wearable system. FIG. 3 illustrates a wearable system 200, which may include a display 220 and a frame 230. A blowup 202 diagrammatically illustrates various components of the wearable system 200. In some implementations, one or more of the components illustrated in FIG. 3 may be part of the display 220. The various components, alone or in combination, may collect various data associated with a user of the wearable system 200 or the user's environment (e.g., auditory or visual data, etc.). It should be understood that other embodiments may have additional or fewer components depending on the application for which the wearable system is used. However, FIG. 3 provides a basic idea of ​​some of the various components and the types of data that may be collected, analyzed, and stored through the wearable system.

[0047] FIG. 3 illustrates an exemplary wearable system 200, which may include a display 220. The display 220 may include a display lens 226, which may be mounted to a housing or frame 230 that corresponds to the user's head, or a frame 230. The display lens 230 may include one or more transparent mirrors positioned in front of the user's eyes 302, 304 by the housing 230, and may be configured to bounce the projected light 338 into the eyes 302, 304 and facilitate beam shaping while also allowing transmission of at least some light from the local environment. The wavefront of the projected light beam 338 may be bent or focused to match a desired focal length of the projected light. As illustrated, two wide-field machine vision cameras 316 (also referred to as world cameras) are coupled to the housing 230 and may image the environment around the user. These cameras 316 may be dual capture visible / non-visible (e.g., infrared) light cameras. The camera 316 may be part of the outward facing imaging system 464 shown in Figure 4. Images acquired by the world camera 316 may be processed by the pose processor 336. For example, the pose processor 336 may implement one or more object recognizers 708 (e.g., shown in Figure 7) to identify a pose of the user or another person in the user's environment, or to identify physical objects in the user's environment.

[0048] Continuing with reference to FIG. 3, a pair of scanning laser shaped wavefront (e.g., for depth) light projector modules with display mirrors and optics configured to project light 338 into the eyes 302, 304 are shown. The depicted diagram also shows two miniature infrared cameras 324 paired with infrared lights (such as light emitting diodes "LEDs") configured to track the user's eyes 302, 304 and support rendering and user input. The cameras 324 may be part of an inwardly facing imaging system 462 shown in FIG. 4. The wearable system 200 may further feature a sensor assembly 339, which may include X, Y, and Z axis accelerometer capabilities and magnetic compass and X, Y, and Z axis gyroscope capabilities, and preferably provide data at a relatively high frequency, such as 200 Hz. The sensor assembly 339 may be part of an IMU, as described with reference to FIG. 2A. The depicted system 200 may also include a head pose processor 336, such as an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or ARM processor (Advanced Reduced Instruction Set Machine), which may be configured to calculate real-time or near real-time user head pose from the wide FOV image information output from the capture device 316. The head pose processor 336 may be a hardware processor and may be implemented as part of the local processing and data module 260 shown in FIG. 2A.

[0049] The wearable system may also include one or more depth sensors 234. The depth sensors 234 may be configured to measure a distance between an object in the environment and the wearable device. The depth sensors 234 may include a laser scanner (e.g., lidar), an ultrasonic depth sensor, or a depth-sensing camera. In some implementations where the camera 316 has depth-sensing capabilities, the camera 316 may also be considered a depth sensor 234.

[0050] Also shown is a processor 332 configured to perform digital or analog processing and derive attitude from gyroscope, compass, or accelerometer data from the sensor assembly 339. The processor 332 may be part of a local processing and data module 260, shown in FIG. 2. The wearable system 200 may also include a positioning system, such as, for example, a GPS 337 (Global Positioning System), as shown in FIG. 3, to aid in the attitude and positioning analysis. In addition, the GPS may further provide remote-based (e.g., cloud-based) information about the user's environment. This information may be used to recognize objects or information within the user's environment.

[0051] The wearable system may combine data obtained by the GPS 337 and a remote computing system (e.g., the remote processing module 270, another user's ARD, etc.), which may provide more information about the user's environment. As an example, the wearable system may determine the user's location based on the GPS data and retrieve a world map (e.g., by communicating with the remote processing module 270) that includes virtual objects associated with the user's location. As another example, the wearable system 200 may monitor the environment using a world camera 316 (which may be part of the outward-facing imaging system 464 shown in FIG. 4). Based on the images obtained by the world camera 316, the wearable system 200 may detect objects in the environment (e.g., by using one or more object recognizers 708 shown in FIG. 7). The wearable system may further use data obtained by the GPS 337 to interpret characters.

[0052] The wearable system 200 may also include a rendering engine 334, which may be configured to provide local rendering information to the user for the user's view of the world and facilitate operation of the scanner and imaging into the user's eye. The rendering engine 334 may be implemented by a hardware processor (e.g., a central processing unit or a graphics processing unit, etc.). In some embodiments, the rendering engine is part of the local processing and data module 260. The rendering engine 334 may be communicatively coupled (e.g., via wired or wireless links) to other components of the wearable system 200. For example, the rendering engine 334 may be coupled to the eye camera 324 via communication link 274 and to the projection subsystem 318 (which may project light into the user's eyes 302, 304 via a scanning laser array in a manner similar to a retinal scanning display) via communication link 272. The rendering engine 334 may also communicate with other processing units, such as, for example, the sensor pose processor 332 and the image pose processor 336, via links 276 and 294, respectively.

[0053] A camera 324 (e.g., a small infrared camera) may be utilized to track eye pose and support rendering and user input. Some example eye poses may include where the user is looking or the depth at which they are focused (which may be estimated using eye vergence). A GPS 337, gyroscope, compass, and accelerometer 339 may be utilized to provide a rough or fast pose estimate. One or more of the cameras 316 may obtain images and poses that, together with data from associated cloud computing resources, may be utilized to map the local environment and share the user view with others.

[0054] The exemplary components depicted in FIG. 3 are for illustrative purposes only. Multiple sensors and other functional modules are shown together for ease of illustration and description. Some embodiments may include only one or a subset of these sensors or modules. Furthermore, the locations of these components are not limited to the locations depicted in FIG. 3. Some components may be mounted or stored within other components, such as belt-mounted components, handheld components, or helmet components. As an example, the image pose processor 336, the sensor pose processor 332, and the rendering engine 334 may be located within a beltpack and configured to communicate with other components of the wearable system via wireless communication, such as ultra-wideband, Wi-Fi, Bluetooth, or via wired communication. The depicted housing 230 is preferably head-mountable and wearable by a user. However, some components of the wearable system 200 may be worn on other parts of the user's body. For example, the speaker 240 may be inserted into the user's ear to provide sound to the user.

[0055] With regard to projecting light 338 into the user's eyes 302, 304, in some embodiments, the camera 324 may be utilized to measure where the center of the user's eyes is geometrically converged, which generally coincides with the location of the eye's focus or "depth of focus." The three-dimensional surface of all points at which the eyes converge may be referred to as the "monocular locus." The focal length may take on a finite number of depths or may vary infinitely. Light projected from the vergence distance appears focused on the subject's eyes 302, 304, while light in front of or behind the vergence distance is blurred. Examples of wearable devices and other display systems of the present disclosure are also described in U.S. Patent Publication No. 2016 / 0270656, which is incorporated herein by reference in its entirety.

[0056] The human visual system is complex and providing a realistic perception of depth is difficult. A viewer of an object may perceive the object as three-dimensional due to a combination of vergence and accommodation. Vergence movements of the two eyes relative to one another (e.g., pupil rotation such that the pupils move toward or away from one another, converging the gaze of the eyes to fixate on an object) are closely linked to the focusing (or "accommodation") of the eye's lenses. Under normal conditions, a change in focus of the eye's lenses or accommodation of the eye to change focus from one object to another at a different distance will automatically produce a matching change in vergence at the same distance, a relationship known as the "accommodation-vergence reflex." Similarly, a change in vergence will induce a matching change in accommodation under normal conditions. A display system that provides better matching between accommodation and vergence may produce a more realistic and comfortable simulation of a three-dimensional image.

[0057] Furthermore, spatially coherent light with a beam diameter of less than about 0.7 millimeters can be correctly resolved by the human eye regardless of where the eye is focused. Thus, to create the illusion of proper depth of focus, eye vergence-divergence movements may be tracked using the camera 324, and the rendering engine 334 and projection subsystem 318 may be utilized to render all objects on or near the monocular locus in focus and all other objects variable degrees out of focus (e.g., using intentionally created blur). Preferably, the system 220 renders to the user at a frame rate of about 60 frames per second or greater. As explained above, preferably the camera 324 may be utilized for eye tracking, and software may be configured to take up not only vergence-divergence geometry, but also focus location cues to serve as user input. Preferably, such a display system is configured with a suitable brightness and contrast for daytime or nighttime use.

[0058] In some embodiments, the display system preferably has a latency of less than about 20 milliseconds for visual object alignment, an angular alignment of less than about 0.1 degrees, and a resolution of about 1 arc minute, which is believed to be approximately the limit of the human eye, without being limited by theory. The display system 220 may be integrated with a localization system, which may involve a GPS element, optical tracking, a compass, an accelerometer, or other data sources to aid in position and attitude determination. The localization information may be utilized to facilitate accurate rendering within the user's view of the relevant world (e.g., such information would facilitate the glasses knowing their location relative to the real world).

[0059] In some embodiments, the wearable system 200 is configured to display one or more virtual images based on the accommodation of the user's eyes. Unlike traditional 3D display approaches that force the user to focus where the image is projected, in some embodiments, the wearable system is configured to automatically vary the focus of the projected virtual content to allow for more comfortable viewing of one or more images presented to the user. For example, if the user's eyes have a current focus of 1m, the image may be projected to match the user's focus. If the user shifts the focus to 3m, the image is projected to match the new focus. Thus, rather than forcing a predetermined focus on the user, the wearable system 200 of some embodiments allows the user's eyes to function in a more natural manner.

[0060] Such a wearable system 200 may eliminate or reduce the incidence of eye strain, headaches, and other physiological symptoms typically observed with virtual reality devices. To accomplish this, various embodiments of the wearable system 200 are configured to project virtual images at variable focal distances through one or more variable focus elements (VFEs). In one or more embodiments, 3D perception may be achieved through a multi-planar focus system that projects images onto a fixed focal plane from the user. Other embodiments employ a variable planar focus, where the focal plane is moved back and forth in the z-direction to match the user's current state of focus.

[0061] In both the multi-plane focus system and the variable plane focus system, the wearable system 200 may employ eye tracking to determine the vergence-divergence movement of the user's eyes, determine the user's current focus, and project the virtual image at the determined focus. In other embodiments, the wearable system 200 includes a light modulator that variably projects a variably focused light beam in a raster pattern across the retina through a fiber scanner or other light generating source. Thus, the wearable system 200 display's ability to project images at variable focal distances not only facilitates accommodation for the user to view objects in 3D, but may also be used to compensate for the user's ocular anomalies, as further described in U.S. Patent Publication No. 2016 / 0270656, which is incorporated herein by reference in its entirety. In some other embodiments, the spatial light modulator may project the image to the user through various optical components. For example, as further described below, the spatial light modulator may project the image onto one or more waveguides, which then transmit the image to the user. (Waveguide Stack Assembly)

[0062] FIG. 4 illustrates an example of a waveguide stack for outputting image information to a user. The wearable system 400 includes a stack of waveguides or a stacked waveguide assembly 480 that can be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides 432b, 434b, 436b, 438b, 4400b. In some embodiments, the wearable system 400 may correspond to the wearable system 200 of FIG. 2, of which FIG. 4 illustrates several portions in more detail. For example, in some embodiments, the waveguide assembly 480 may be integrated into the display 220 of FIG. 2.

[0063] 4, the waveguide assembly 480 may also include a number of features 458, 456, 454, 452 between the waveguides. In some embodiments, the features 458, 456, 454, 452 may be lenses. In other embodiments, the features 458, 456, 454, 452 may not be lenses. Rather, they may simply be spacers (e.g., cladding layers or structures to form air gaps).

[0064] The waveguides 432b, 434b, 436b, 438b, 440b or multiple lenses 458, 456, 454, 452 may be configured to transmit image information to the eye with various levels of wavefront curvature or light beam divergence. Each waveguide level may be associated with a particular depth plane and configured to output image information corresponding to that depth plane. Image injection devices 420, 422, 424, 426, 428 may be utilized to inject image information into the waveguides 440b, 438b, 436b, 434b, 432b, respectively, which may be configured to disperse incident light across each individual waveguide for output toward the eye 410. Light exits output surfaces of image launch devices 420, 422, 424, 426, 428 and is launched into corresponding input edges of waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, a single beam of light (e.g., a collimated beam) may be launched into each waveguide to output an entire field of cloned collimated beams that are directed towards the eye 410 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0065] In some embodiments, the image input devices 420, 422, 424, 426, 428 are each discrete displays that generate image information for input into a respective waveguide 440b, 438b, 436b, 434b, 432b, etc. In some other embodiments, the image input devices 420, 422, 424, 426, 428 are the output of a single multiplexed display that may, for example, send image information to each of the image input devices 420, 422, 424, 426, 428 via one or more optical conduits (such as fiber optic cables).

[0066] A controller 460 controls the operation of the stacked waveguide assembly 480 and the image input devices 420, 422, 424, 426, 428. The controller 460 includes programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provision of image information to the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the controller 460 may be a single integrated device or a distributed system connected by a wired or wireless communication channel. The controller 460 may be part of the processing module 260 or 270 (illustrated in FIG. 2) in some embodiments.

[0067] The waveguides 440b, 438b, 436b, 434b, 432b may be configured to propagate light within each individual waveguide by total internal reflection (TIR). The waveguides 440b, 438b, 436b, 434b, 432b may each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between the major top and bottom surfaces. In the illustrated configuration, the waveguides 440b, 438b, 436b, 434b, 432b may each include a light extraction optical element 440a, 438a, 436a, 434a, 432a configured to extract light from the waveguide by redirecting the light and propagating it within each individual waveguide and outputting image information from the waveguide to the eye 410. The extracted light may also be referred to as out-coupled light, and the light extraction optical element may also be referred to as out-coupling optical element. The extracted light beam is output by the waveguide where the light propagating in the waveguide strikes the light redirecting element. The light extraction optical element (440a, 438a, 436a, 434a, 432a) may be, for example, a reflective or diffractive optical feature. Although shown disposed on the bottom major surface of the waveguides 440b, 438b, 436b, 434b, 432b for ease of explanation and clarity of drawing, in some embodiments the light extraction optical element 440a, 438a, 436a, 434a, 432a may be disposed on the top or bottom major surface, or may be disposed directly within the volume of the waveguides 440b, 438b, 436b, 434b, 432b. In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed in a layer of material that is attached to a transparent substrate and forms the waveguides 440b, 438b, 436b, 434b, 432b. In some other embodiments, the waveguides 440b, 438b, 436b, 434b, 432b may be monolithic pieces of material and the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be formed on and / or within the material pieces.

[0068] Continuing with reference to FIG. 4, as discussed herein, each waveguide 440b, 438b, 436b, 434b, 432b is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide 432b closest to the eye may be configured to deliver collimated light to the eye 410 as it is launched into such waveguide 432b. The collimated light may represent an optical infinity focal plane. The next upper waveguide 434b may be configured to send collimated light that passes through a first lens 452 (e.g., a negative lens) before it can reach the eye 410. The first lens 452 may be configured to generate a slight convex wavefront curvature so that the eye / brain interprets the light originating from the next upper waveguide 434b as originating from a first focal plane closer inward from optical infinity toward the eye 410. Similarly, the third upper waveguide 436b passes its output light through both a first lens 452 and a second lens 454 before reaching the eye 410. The combined refractive powers of the first and second lenses 452 and 454 may be configured to produce another incremental amount of wavefront curvature such that the eye / brain interprets the light emerging from the third waveguide 436b as emerging from a second focal plane that is closer inward from optical infinity towards the person than was the light from the next upper waveguide 434b.

[0069] Other waveguide layers (e.g., waveguides 438b, 440b) and lenses (e.g., lenses 456, 458) are similarly configured, with the highest waveguide 440b in the stack sending its output through all of the lenses between it and the eye for an aggregate focal force representing the focal plane closest to the person. To compensate for the stack of lenses 458, 456, 454, 452 when viewing / interpreting light originating from the world 470 on the other side of the stacked waveguide assembly 480, a compensating lens layer 430 may be placed on top of the stack to compensate for the aggregate force of the lens stacks 458, 456, 454, 452 below. Such a configuration provides as many perceived focal planes as there are waveguide / lens pairs available. Both the light extraction optical elements of the waveguides and the focusing sides of the lenses may be static (e.g., not dynamic or electroactive). In some alternative embodiments, one or both may be dynamic using electroactive features.

[0070] Continuing with reference to FIG. 4, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be configured to redirect light from its respective waveguide and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes may have different configurations of light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be solid or surface features that may be configured to output light at a specific angle. For example, the light extraction optical elements 440a, 438a, 436a, 434a, 432a may be solid holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published June 25, 2015, which is incorporated by reference in its entirety.

[0071] In some embodiments, the light extraction optical elements 440a, 438a, 436a, 434a, 432a are diffractive features or "diffractive optical elements" (also referred to herein as "DOEs") that form a diffraction pattern. Preferably, the DOEs have a relatively low diffraction efficiency so that only a portion of the light in the beam is deflected toward the eye 410 at each intersection of the DOE, while the remainder continues traveling through the waveguide via total internal reflection. The light carrying the image information is thus split into several related exit beams that exit the waveguide at multiple locations, which can result in a very uniform pattern of exit emission toward the eye 304 for this particular collimated beam bouncing within the waveguide.

[0072] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal in which microdroplets comprise a diffractive pattern in a host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0073] In some embodiments, the number and distribution of depth planes or depths of field may be dynamically varied based on the size or orientation of the pupil of the viewer's eye. The depth of field may vary inversely with the size of the viewer's pupil. As a result, as the size of the viewer's eye pupil decreases, the depth of field increases so that one plane that is indistinguishable because its location exceeds the focal depth of the eye may become distinguishable and appear more focused with a reduction in pupil size and a corresponding increase in depth of field. Similarly, the number of spaced depth planes used to present different images to the viewer may be reduced with a reduced pupil size. For example, the viewer may not be able to clearly perceive the details of both the first and second depth planes at one pupil size without adjusting the accommodation of the eye from one depth plane to the other. However, these two depth planes may be sufficient to simultaneously focus the user at another pupil size without changing accommodation.

[0074] In some embodiments, the display system may vary the number of waveguides that receive image information based on a determination of a pupil size or orientation, or in response to receiving an electrical signal indicative of a particular pupil size or orientation. For example, if the user's eye is unable to distinguish between two depth planes associated with two waveguides, the controller 460 (which may be an embodiment of the local processing and data module 260) may be configured or programmed to stop providing image information to one of these waveguides. Advantageously, this may reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments in which the DOE for a waveguide is switchable between an on and off state, the DOE may be switched to the off state when the waveguide receives image information.

[0075] In some embodiments, it may be desirable to have the exit beam meet the condition of having a diameter less than the diameter of the viewer's eye. However, meeting this condition may be difficult in light of the variability in the size of the viewer's pupil. In some embodiments, this condition is met over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the size of the viewer's pupil. For example, as the pupil size decreases, the size of the exit beam may also decrease. In some embodiments, the exit beam size may be varied using a variable aperture.

[0076] The wearable system 400 may include an outwardly facing imaging system 464 (e.g., a digital camera) that images a portion of the world 470. This portion of the world 470 may be referred to as the field of view (FOV) of the world camera, and the imaging system 464 is sometimes also referred to as the FOV camera. The world camera FOV may or may not be the same as the viewer 210 FOV, which encompasses the portion of the world 470 that the viewer 210 perceives at a given time. For example, in some situations, the world camera FOV may be larger than the viewer 210's field of view of the wearable system 400. The entire area available for viewing or imaging by the viewer may be referred to as the ocular field of view (FOR). The FOR may include a solid angle of 4π steradians surrounding the wearable system 400, since the wearer can move their body, head, or eyes and perceive virtually any direction in space. In other contexts, the wearer's movements may be more constrained and the wearer's FOR may correspondingly subtend a smaller solid angle. Images obtained from the outwardly facing imaging system 464 can be used to track gestures (e.g., hand or finger gestures) made by the user, detect objects in the world 470 in front of the user, etc.

[0077] The wearable system 400 can include an audio sensor 232, e.g., a microphone, to capture ambient sound. As described above, in some embodiments, one or more other audio sensors can be positioned to provide stereo sound reception useful in determining the location of the speech source. The audio sensor 232 can comprise, as another example, a directional microphone, which can also provide such useful directional information regarding where the audio source is located. The wearable system 400 can use information from both the outwardly facing imaging system 464 and the audio sensor 230 in locating the speech source or to determine the active speaker at a particular moment, etc. For example, the wearable system 400 can use voice recognition, alone or in combination with a reflected image of the speaker (e.g., as seen in a mirror), to determine the identity of the speaker. As another example, the wearable system 400 can determine the location of the speaker in the environment based on sounds obtained from a directional microphone. The wearable system 400 can use speech recognition algorithms to analyze sounds originating from the speaker's location, determine the content of the speech, and use voice recognition techniques to determine the speaker's identity (e.g., name or other demographic information).

[0078] The wearable system 400 may also include an inwardly facing imaging system 466 (e.g., a digital camera) that observes user movements such as eye and face movements. The inwardly facing imaging system 466 may be used to capture images of the eye 410 and determine the size and / or orientation of the pupil of the eye 304. The inwardly facing imaging system 466 may be used to obtain images for use in determining the direction the user is looking (e.g., eye pose) or for biometric identification of the user (e.g., via iris identification). In some embodiments, at least one camera may be utilized for each eye independently to separately determine the pupil size or eye pose of each eye, thereby allowing the presentation of image information to each eye to be dynamically adjusted for that eye. In some other embodiments, the pupil diameter or orientation of only a single eye 410 (e.g., using only a single camera per pair of eyes) is determined and assumed to be similar for both eyes of the user. Images obtained by inwardly facing imaging system 466 may be analyzed to determine the user's eye posture or mood, which may be used by wearable system 400 to determine audio or visual content to be presented to the user. Wearable system 400 may also determine head pose (e.g., head position or head orientation) using sensors such as an IMU, accelerometer, gyroscope, etc.

[0079] The wearable system 400 may include a user input device 466 through which a user may input commands into the controller 460 and interact with the wearable system 400. For example, the user input device 466 may include a trackpad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a tactile device, a totem (e.g., serving as a virtual user input device), and the like. A multi-DOF controller may sense user input in possible translation (e.g., left / right, forward / backward, or up / down) or rotation (e.g., yaw, pitch, or roll) of some or all of the controller. A multi-DOF controller that supports translation may be referred to as 3DOF, while a multi-DOF controller that supports translation and rotation may be referred to as 6DOF. In some cases, a user may use a finger (e.g., thumb) to press or swipe on a touch-sensitive input device to provide input to the wearable system 400 (e.g., to provide user input to a user interface provided by the wearable system 400). The user input device 466 may be held by the user's hand during use of the wearable system 400. The user input device 466 may communicate with the wearable system 400 in wired or wireless communication. (Other components of the wearable system)

[0080] In many implementations, the wearable system may include other components in addition to or as an alternative to the components of the wearable system described above. The wearable system may include, for example, one or more tactile devices or components. The tactile devices or components may be operable to provide a haptic sensation to the user. For example, the tactile device or component may provide a haptic sensation of pressure or texture upon touching the virtual content (e.g., a virtual object, virtual tool, other virtual structure). The haptics may replicate the sensation of a physical object that the virtual object represents, or may replicate the sensation of an imaginary object or character (e.g., a dragon) that the virtual content represents. In some implementations, the tactile device or component may be worn by the user (e.g., a user-wearable glove). In some implementations, the tactile device or component may be held by the user.

[0081] A wearable system may include one or more physical objects that can be manipulated by, for example, a user to allow input to or interaction with the wearable system. These physical objects may be referred to herein as totems. Some totems may take the form of inanimate objects, such as, for example, pieces of metal or plastic, walls, the surface of a table, etc. In some implementations, a totem may not actually have any physical input structures (e.g., keys, triggers, joysticks, trackballs, rocker switches). Instead, a totem may simply provide a physical surface, and the wearable system may render a user interface to appear to the user as being on one or more surfaces of the totem. For example, the wearable system may render an image of a computer keyboard and trackpad to appear to reside on one or more surfaces of the totem. For example, the wearable system may render a virtual computer keyboard and virtual trackpad to appear on the surface of a thin rectangular plate of aluminum that may serve as a totem. The rectangular plate itself does not have any physical keys or trackpads or sensors. However, the wearable system may detect user manipulation or interaction or touch with the rectangular plate as a selection or input made via a virtual keyboard or virtual trackpad. User input device 466 (shown in FIG. 4) may be an embodiment of the totem, which may include a trackpad, touchpad, trigger, joystick, trackball, rocker or virtual switch, mouse, keyboard, multi-degree-of-freedom controller, or another physical input device. A user may use the totem alone or in combination with posture to interact with the wearable system or other users.

[0082] Examples of tactile devices and totems usable with the wearable devices, HMDs, and display systems of the present disclosure are described in U.S. Patent Publication No. 2015 / 0016777, which is incorporated by reference in its entirety. (Example process of user interaction with a wearable system)

[0083] 5 is a process flow diagram of an example method 500 for interacting with a virtual user interface. Method 500 may be implemented by a wearable system described herein. An embodiment of method 500 may be used by a wearable system to detect a person or document within the FOV of the wearable system.

[0084] In block 510, the wearable system may identify a particular UI. The type of UI may be predetermined by the user. The wearable system may identify that a particular UI needs to be captured based on user input (e.g., gestures, visual data, audio data, sensory data, direct commands, etc.). The UI may be specific to a security scenario where a wearer of the system is observing a user presenting a document to the wearer (e.g., at a traveler checkpoint). In block 520, the wearable system may generate data for a virtual UI. For example, data associated with the extent, general structure, shape of the UI, etc. may be generated. In addition, the wearable system may determine map coordinates of the user's physical location so that the wearable system may display the UI in relation to the user's physical location. For example, if the UI is body-centered, the wearable system may determine the coordinates of the user's physical standing position, head pose, or eye pose so that a ring UI may be displayed around the user or a planar UI may be displayed on a wall or in front of the user. In the security context described herein, the UI may be displayed as if it were surrounding the traveler presenting a document to the wearer of the system, so that the wearer may easily view the UI while viewing the traveler and the traveler's document. If the UI is hand-centered, map coordinates of the user's hand may be determined. These map points may be derived through data received through the FOV camera, sensory input, or any other type of collected data.

[0085] In block 530, the wearable system may send data from the cloud to the display, or data may be sent from a local database to the display component. In block 540, a UI is displayed to the user based on the sent data. For example, a light field display may project the virtual UI into one or both of the user's eyes. Once the virtual UI is created, the wearable system may simply wait for a command from the user to generate more virtual content on the virtual UI in block 550. For example, the UI may be a body-centered ring around the user's body or the body of a person (e.g., a traveler) in the user's environment. The wearable system may then wait for a command (gesture, head or eye movement, voice command, input from a user input device, etc.) and if it is recognized (block 560), the virtual content associated with the command may be displayed to the user (block 570). (An example of avatar rendering in mixed reality)

[0086] The wearable system may employ various mapping-related techniques to achieve a high depth of field in the rendered light field. When mapping a virtual world, it is advantageous to capture all features and points in the real world and accurately depict the virtual objects in relation to the real world. To this end, FOV images captured from a user of the wearable system can be added to the world model by including new photos that convey information about various points and features of the real world. For example, the wearable system can collect a set of map points (such as 2D or 3D points), find new map points, and render a more accurate version of the world model. The world model of the first user can be communicated (e.g., via a network such as a cloud network) to a second user so that the second user can experience the world surrounding the first user.

[0087] FIG. 6A is a block diagram of another example of a wearable system that may include an avatar processing and rendering system 690 into a mixed reality environment. The wearable system 600 may be part of the wearable system 200 shown in FIG. 2. In this example, the wearable system 600 may include a map 620, which may include at least a portion of the data in the map database 710 (shown in FIG. 7). The map may reside in part locally on the wearable system and in part in a networked storage location (e.g., in a cloud system) accessible by a wired or wireless network. An attitude process 610 may run on the wearable computing architecture (e.g., processing module 260 or controller 460) and utilize data from the map 620 to determine the position and orientation of the wearable computing hardware or the user. The attitude data may be calculated from data collected on the fly as the user experiences the system and operates within the world. The data may include images, data from sensors (such as inertial measurement units, which generally include accelerometer and gyroscope components), and surface information associated with objects in a real or virtual environment.

[0088] The sparse representation may be the output of a simultaneous localization and mapping (e.g., SLAM or vSLAM, which refers to configurations where the input is image / vision only) process. The system can be configured to find not only where in the world various components are located, but also what the world consists of. Pose can be a building block that accomplishes many goals, including capturing maps and using data from maps.

[0089] In one embodiment, the sparse point locations may not be entirely adequate by themselves, and further information may be required to produce a multi-focal AR, VR, or MR experience. A dense representation, generally referring to depth map information, may be utilized to fill this gap, at least in part. Such information may be calculated from a process referred to as stereo 640, where depth information is determined using techniques such as triangulation or time-of-flight sensing. Image information and active patterns (such as infrared patterns created using an active projector), images obtained from an image camera, or hand gestures / totems 650 may serve as inputs to the stereo process 640. A significant amount of depth map information may be fused together, some of which may be summarized with a surface representation. For example, a mathematically definable surface may be an efficient (e.g., compared to a large point cloud) and easy-to-understand input to other processing devices such as a game engine. Thus, the outputs of the stereo process (e.g., depth map) 640 may be combined in a fusion process 630. The pose 610 may also be an input to this fusion process 630, the output of which becomes the input to the map capture process 620. The sub-surfaces may interconnect to form larger surfaces, as in topographic mapping, and the map becomes a large-scale hybrid of points and surfaces.

[0090] Various inputs may be utilized to resolve various aspects in the mixed reality process 660. For example, in the embodiment depicted in FIG. 6A, game parameters may be input to determine that a user of the system is playing a monster battle game with one or more monsters in various locations, monsters dying or fleeing under various conditions (such as when the user shoots the monsters), walls or other objects in various locations, and the like. A world map may contain information about the location of objects or semantic information of objects (e.g., classification of whether an object is flat or round, horizontal or vertical, table or lamp, etc.), and may be another useful input to mixed reality. Attitude relative to the world is also an input and plays an important role for almost any interactive system.

[0091] Controls or inputs from the user are another input to the wearable system 600. As described herein, user inputs can include visual inputs, gestures, totems, audio inputs, sensory inputs, etc. To move around or play a game, for example, the user may need to command the wearable system 600 as to what he or she wants. There are various forms of user control that can be utilized beyond simply moving oneself in space. In one embodiment, an object such as a totem (e.g., a user input device) or a toy gun may be held by the user and tracked by the system. The system would preferably be configured to know that the user is holding an item and understand the type of interaction the user is having with the item (e.g., if the totem or object is a gun, the system may be configured to understand the location and orientation and whether the user is clicking a trigger or other sensed button or element (which may be equipped with sensors such as IMUs that can help determine what is happening even when the activity is not within the field of view of any of the cameras)).

[0092] Tracking or recognition of hand gestures may also provide input information. The wearable system 600 may be configured to track and interpret hand gestures in terms of button presses, left or right, stop, grab, hold gestures, etc. For example, in one configuration, a user may want to flip an email or calendar in a non-gaming environment, or "fist bump" with another person or player. The wearable system 600 may be configured to utilize a minimal amount of hand gestures, which may or may not be dynamic. For example, gestures may be simple static gestures, such as opening the hand for stop, thumbs up for OK, thumbs down for not OK, or flipping the hand to the right or left or up / down for directional commands.

[0093] Eye tracking is another input (e.g., tracking where the user is looking, controlling display technology, rendering to a specific depth or range). In one embodiment, eye vergence may be determined using triangulation, and then accommodation may be determined using a vergence / accommodation model developed for that particular person. Eye tracking is performed by an eye camera, which can determine eye gaze (e.g., direction or orientation of one or both eyes). Other techniques can be used for eye tracking, such as, for example, measurement of electrical potentials with electrodes placed near the eyes (e.g., electro-oculography).

[0094] Speech tracking may be another input that may be used alone or in combination with other inputs (e.g., totem tracking, eye tracking, gesture tracking, etc.). Speech tracking may include speech recognition, voice recognition, alone or in combination. System 600 may include an audio sensor (e.g., a microphone) that receives an audio stream from the environment. System 600 may incorporate voice recognition technology to determine who is speaking (e.g., whether the speech is from the wearer of the ARD or another person or voice (e.g., recorded voice transmitted by a loudspeaker in the environment)) and to determine what is being said. The local data and processing module 260 or the remote processing module 270 can process the audio data from the microphone (or audio data in another stream, such as, for example, a video stream being viewed by a user) and identify the content of the speech by applying various speech recognition algorithms, such as, for example, hidden Markov models, dynamic time warping (DTW) based speech recognition, neural networks, deep learning algorithms such as deep feedforward and recurrent neural networks, end-to-end automatic speech recognition, machine learning algorithms (described with reference to FIG. 7), or other algorithms that use acoustic or language modeling.

[0095] The local data and processing module 260 or the remote processing module 270 can also apply speech recognition algorithms, which can identify the identity of the speaker, such as whether the speaker is the user 210 of the wearable system 600 or another person with whom the user is conversing. Some example speech recognition algorithms can include frequency estimation, hidden Markov models, Gaussian mixture models, pattern matching algorithms, neural networks, matrix representations, vector quantization, speaker diarization, decision trees, and dynamic time warping (DTW) techniques. Speech recognition techniques can also include anti-speaker techniques such as cohort models and world models. Spectral features may be used in representing speaker characteristics. The local data and processing module or the remote data processing module 270 can perform speech recognition using various machine learning algorithms, which are described with reference to FIG. 7.

[0096] An implementation of the wearable system can use these user controls or inputs via the UI. UI elements (e.g., controls, pop-up windows, callouts, data entry fields, etc.) can be used, for example, to dismiss the display of information, e.g., graphical or semantic information of an object.

[0097] With respect to the camera system, the exemplary wearable system 600 shown in FIG. 6A may include three paired cameras: a pair of relatively wide FOV or passive SLAM cameras arranged on the sides of the user's face, and a different paired camera oriented in front of the user to handle the stereo imaging process 640 and also capture hand gestures and totem / object tracking in front of the user's face. The paired cameras for the FOV cameras and stereo process 640 may be part of the outward facing imaging system 464 (shown in FIG. 4). The wearable system 600 may include an eye tracking camera (which may be part of the inward facing imaging system 462 shown in FIG. 4) oriented towards the user's eyes to triangulate eye vectors and other information. The wearable system 600 may also include one or more textured light projectors (such as infrared (IR) projectors) to inject texture into the scene.

[0098] The wearable system 600 may comprise an avatar processing and rendering system 690. The avatar processing and rendering system 690 may be configured to generate, update, animate, and render avatars based on the context information. Some or all of the avatar processing and rendering system 690 may be implemented as part of the local processing and data module 260 or the remote processing modules 262, 264, alone or in combination. In various embodiments, multiple avatar processing and rendering systems 690 (e.g., as implemented on different wearable devices) may be used to render the virtual avatar 670. For example, a first user's wearable device may be used to determine the first user's intent, while a second user's wearable device may determine avatar characteristics and render the first user's avatar based on the intent received from the first user's wearable device. The first user's wearable device and the second user's wearable device (or other such wearable devices) can communicate, for example, via a network, as will be described with reference to Figures 9A and 9B.

[0099] FIG. 6B illustrates an exemplary avatar processing and rendering system 690. The avatar processing and rendering system 690 can implement the mesh transfer technique described with reference to FIGS. 11-14C. The exemplary avatar processing and rendering system 690 can comprise, either alone or in combination, a 3D model processing system 680, a context information analysis system 688, an avatar auto-scaler 692, an intent mapping system 694, an anatomical structure adjustment system 698, and a stimulus response system 696. The system 690 is intended to illustrate and not limit functionality for avatar processing and rendering. For example, in an implementation, one or more of these systems may be part of another system. For example, parts of the context information analysis system 688 may be parts of the avatar auto-scaler 692, the intent mapping system 694, the stimulus response system 696, or the anatomical structure adjustment system 698, either alone or in combination.

[0100] The context information analysis system 688 can be configured to determine environment and object information based on one or more device sensors, described with reference to Figures 2 and 3. For example, the context information analysis system 688 can use images acquired by the outwardly facing imaging system 464 of a viewer of the user or the user's avatar to analyze the environment and objects (including physical or virtual objects) of the user's environment or the environment in which the user's avatar is rendered. The context information analysis system 688 can analyze such images, alone or in combination with data acquired from location data or world maps (e.g., maps 620, 710, 910), to determine the location and layout of objects in the environment. The context information analysis system 688 can also access biological characteristics of the user or humans in general to realistically animate the virtual avatar 670. For example, the context information analysis system 688 can generate an incongruity curve, which can be applied to the user's avatar so that parts of the avatar's body (e.g., the head) are not in an incongruent (or unrealistic) position relative to other parts of the user's body (e.g., the avatar's head is not turned 270 degrees). In some implementations, one or more object recognizers 708 (shown in FIG. 7) may be implemented as part of the context information analysis system 688.

[0101] The avatar auto-scaler 692, the intent mapping system 694, the stimulus response system 696, and the anatomy adjustment system 698 can be configured to determine characteristics of the avatar based on the context information. Some example characteristics of the avatar can include size, appearance, position, orientation, movement, posture, expression, etc. The avatar auto-scaler 692 can be configured to automatically scale the avatar so that the user does not have to view the avatar in an uncomfortable position. For example, the avatar auto-scaler 692 can increase or decrease the size of the avatar and bring the avatar to the user's eye level so that the user does not have to look down or up at the avatar, respectively. The intent mapping system 694 can determine the user's interaction intent based on the environment in which the avatar is rendered and map the intent (rather than the exact user interaction) to the avatar. For example, the intent of a first user can be to communicate with a second user in a telepresence session (see, e.g., FIG. 9B). Typically, two people face each other when communicating. The intent mapping system 694 of the first user's wearable system can determine such a facing intent that exists during the telepresence session and can cause the first user's wearable system to render the second user's avatar facing the first user. If the second user attempts to physically turn around, instead of rendering the second user's avatar in a turned around position (which would cause the back of the second user's avatar to be rendered towards the first user), the first user's intent mapping system 694 can continue to render the second avatar's face towards the first user, which is the inferred intent of the telepresence session (e.g., in this example, a facing intent).

[0102] The stimulus response system 696 can identify objects of interest in the environment and determine the avatar's response to the objects of interest. For example, the stimulus response system 696 can identify a sound source in the avatar's environment and automatically reorient the avatar to look at the sound source. The stimulus response system 696 can also determine threshold exit conditions. For example, the stimulus response system 696 can cause the avatar to return to its original posture after the sound source disappears or after a period of time has passed.

[0103] The anatomical structure adjustment system 698 can be configured to adjust the user's posture based on the biological characteristics. For example, the anatomical structure adjustment system 698 can be configured to adjust the relative position between the user's head and the user's torso or between the user's upper body and lower body based on the discomfort curve.

[0104] The 3D model processing system 680 can be configured to animate the virtual avatar 670 and render it on the display 220. The 3D model processing system 680 can include a virtual character processing system 682 and a movement processing system 684. The virtual character processing system 682 can be configured to generate and update a 3D model of the user (to create and animate the virtual avatar). The movement processing system 684 can be configured to animate the avatar, for example, by changing the avatar's pose, by moving the avatar in the user's environment, or by animating the avatar's facial expressions, etc. As will be further described herein, the virtual avatar can be animated using rigging techniques. In some embodiments, the avatar is represented in two parts: a surface representation (e.g., a deformable mesh) that is used to render the virtual avatar's outward appearance, and a hierarchical set of joints (e.g., a core skeleton) that are interconnected to animate the mesh. In some implementations, the virtual character processing system 682 can be configured to edit or generate the surface representation, while the movement processing system 684 can be used to animate the avatar by moving the avatar, deforming meshes, etc. For example, in some implementations, the movement processing system 684 performs an embodiment of the joint resolution process described herein (e.g., process 1400 described with reference to FIG. 14 ). (Example of mapping the user's environment)

[0105] 7 is a block diagram of an example of a MR environment 700. The MR environment 700 may be configured to receive inputs (e.g., visual inputs 702 from a user's wearable system, stationary inputs 704 such as room cameras, sensory inputs 706 from various sensors, gestures, totems, eye tracking, user inputs, etc. from user input devices 466) from one or more user wearable systems (e.g., wearable system 200 or display system 220) or stationary room systems (e.g., room cameras, etc.). The wearable systems can use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, movement sensors, depth sensors, GPS sensors, inward-facing imaging systems, outward-facing imaging systems, etc.) to determine the location and various other attributes of the user's environment. This information may be further supplemented with information from stationary cameras in the room, which may provide images from different perspectives or various cues. Image data acquired by the cameras (e.g., room cameras and / or cameras of the outward-facing imaging systems) may be reduced to a set of mapping points.

[0106] One or more object recognizers 708 can crawl through the received data (e.g., a collection of points), recognize or map the points, tag the images, and associate semantic information with the objects using a map database 710. The map database 710 may comprise various points and their corresponding objects collected over time. The various devices and the map database may be interconnected through a network (e.g., LAN, WAN, etc.) and accessible to the cloud.

[0107] Based on this information and the set of points in the map database, the object recognizers 708a-708n may recognize objects in the environment. For example, the object recognizers may recognize faces, people, windows, walls, user input devices, televisions, documents (e.g., travel documents, driver's licenses, passports as described in the security embodiments herein), other objects in the user's environment, etc. One or more object recognizers may be specialized for objects with certain characteristics. For example, object recognizer 708a may be used to recognize faces, while another object recognizer may be used to recognize documents.

[0108] Object recognition may be performed using various computer vision techniques. For example, the wearable system may analyze images acquired by the outward facing imaging system 464 (shown in FIG. 4) and perform scene reconstruction, event detection, video tracking, object recognition (e.g., people or documents), object pose estimation, face recognition (e.g., from images of people in the environment or on documents), learning, indexing, motion estimation, or image analysis (e.g., identifying indicia in documents, such as photographs, signatures, identification information, travel information, etc.), etc. One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale Invariant Feature Transform (SIFT), Speed ​​Up Robust Features (SURF), Orientation FAST and Rotation BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, visual simultaneous localization and mapping (vSLAM) techniques, sequential Bayes estimators (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), iterative nearest neighbor (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histograms, various machine learning algorithms (e.g., support vector machines, k-nearest neighbor algorithms, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.), and the like.

[0109] Object recognition can additionally or alternatively be performed by various machine learning algorithms. Once trained, the machine learning algorithms can be stored by the HMD. Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., a priori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural networks, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., stacked generalization, etc.), and / or other machine learning algorithms. In some embodiments, individual models can be customized for individual datasets. For example, the wearable device can generate or store a base model. The base model may be used as a starting point to generate additional models specific to a data type (e.g., a particular user in a telepresence session), a data set (e.g., a set of additional images acquired of a user in a telepresence session), a conditional situation, or other variations. In some embodiments, the wearable HMD can be configured to generate models for analysis of the aggregated data using multiple techniques. Other techniques may include using predefined thresholds or data values.

[0110] Based on this information and the set of points in the map database, the object recognizer 708a-708n may recognize the object, complement the object with semantic information, and bring the object to life. For example, if the object recognizer recognizes that the set of points is a door, the system may attach some semantic information (e.g., a door has a hinge and has 90 degree movement around the hinge). If the object recognizer recognizes that the set of points is a mirror, the system may attach semantic information that the mirror has a reflective surface that can reflect an image of the object in the room. The semantic information may include the affordances of the object as described herein. For example, the semantic information may include the object's normal. The system may assign a vector, the direction of which indicates the object's normal. Over time, the map database grows as the system (which may reside locally or be accessible through a wireless network) accumulates more data from the world. Once the object is recognized, the information may be transmitted to one or more wearable systems. For example, the MR environment 700 may contain information about a scene occurring in California. The environment 700 may be transmitted to one or more users in New York. Based on the data received from the FOV cameras and other inputs, the object recognizer and other software components can map points collected from the various images, recognize objects, etc., so that the scene can be accurately "passed" to a second user who may be in a different part of the world. The environment 700 may also use a topology map for localization purposes.

[0111] 8 is a process flow diagram of an example method 800 for rendering virtual content in relation to recognized objects. Method 800 illustrates how a virtual scene may be presented to a user of a wearable system. The user may be geographically remote from the scene. For example, a user may be in New York but may wish to view a scene currently occurring in California, or may wish to go for a walk with a friend who is in California.

[0112] In block 810, the wearable system may receive input from the user and other users regarding the user's environment. This may be accomplished through various input devices and knowledge already held in a map database. The user's FOV camera, sensors, GPS, eye tracking, etc. communicate information to the system in block 810. The system may determine sparse points based on this information in block 820. The sparse points may be used in determining pose data (e.g., head pose, eye pose, body pose, or hand gestures) that may be used in displaying and understanding the orientation and location of various objects in the user's surroundings. The object recognizers 708a-708n may crawl through these collected points in block 830 and recognize one or more objects using the map database. This information may then be communicated to the user's respective wearable system in block 840, and the desired virtual scene may be displayed to the user in block 850 as appropriate. For example, a desired virtual scene (eg, a user in CA) may be displayed in the proper orientation, position, etc., relative to various objects and other surroundings of the user in New York. (Example Communication Between Multiple Wearable Systems)

[0113] FIG. 9A diagrammatically illustrates an overall system diagram depicting multiple user devices interacting with each other. Computing environment 900 includes user devices 930a, 930b, 930c. User devices 930a, 930b, and 930c can communicate with each other through network 990. User devices 930a-930c each include a network interface and can communicate with remote computing system 920 (which may also include network interface 971) via network 990. Network 990 may be a LAN, WAN, peer-to-peer network, wireless, Bluetooth, or any other network. Computing environment 900 may also include one or more remote computing systems 920. Remote computing systems 920 may include server computer systems that are clustered and located in different geographic locations. User devices 930a, 930b, and 930c may communicate with remote computing system 920 via network 990.

[0114] The remote computing system 920 may include a remote data repository 980, which may maintain information about a particular user's physical and / or virtual world. The data storage 980 may store information related to the configuration of the user, the user's environment (e.g., a world map of the user's environment), or the user's avatar. The remote data repository may be an embodiment of the remote data repository 280 shown in FIG. 2. The remote computing system 920 may also include a remote processing module 970. The remote processing module 970 may be an embodiment of the remote processing module 270 shown in FIG. 2. The remote processing module 970 may include one or more processors, which may communicate with the user devices (930a, 930b, 930c) and the remote data repository 980. The processors may process information obtained from the user devices and other sources. In some implementations, at least a portion of the processing or storage may be provided by the local processing and data module 260 (as shown in FIG. 2). The remote computing system 920 may enable a given user to share information about the particular user's own physical and / or virtual world with another user.

[0115] The user devices may be, alone or in combination, a wearable device (such as an HMD or ARD), a computer, a mobile device, or any other device. For example, user devices 930b and 930c may be an embodiment of the wearable system 200 shown in FIG. 2 (or the wearable system 400 shown in FIG. 4), which can be configured to present AR / VR / MR content.

[0116] One or more of the user devices can be used in conjunction with the user input device 466 shown in FIG. 4. The user device can acquire information about the user and the user's environment (e.g., using the outwardly facing imaging system 464 shown in FIG. 4). The user device and / or the remote computing system 1220 can use the information acquired from the user device to build, update, and construct a collection of images, points, and other information. For example, the user device may process the acquired raw information and send the processed information to the remote computing system 1220 for further processing. The user device may also send the raw information to the remote computing system 1220 for processing. The user device may receive the processed information from the remote computing system 1220 and provide final processing before projecting it to the user. The user device may also process the acquired information and pass the processed information to other user devices. The user device may communicate with the remote data repository 1280 while processing the acquired information. Multiple user devices and / or multiple server computer systems may be involved in building and / or processing the acquired images.

[0117] Information about the physical world may evolve over time and may be based on information collected by different user devices. Models of the virtual world may also evolve over time and be based on inputs of different users. Such information and models may sometimes be referred to herein as world maps or world models. As described with reference to Figures 6 and 7, information obtained by the user devices may be used to build a world map 910. The world map 910 may include at least a portion of the map 620 illustrated in Figure 6A. Various object recognizers (e.g., 708a, 708b, 708c...708n) may be used to recognize objects and tag images and to add semantic information to objects. These object recognizers are also described in Figure 7.

[0118] The remote data repository 980 can be used to store data and facilitate the construction of the world map 910. The user device can constantly update information about the user's environment and receive information about the world map 910. The world map 910 may be created by the user or another person. As discussed herein, the user devices (e.g., 930a, 930b, 930c) and the remote computing system 920, alone or in combination, may construct and / or update the world map 910. For example, the user device may communicate with the remote processing module 970 and the remote data repository 980. The user device may obtain or process information about the user and the user's environment. The remote processing module 970 may communicate with the remote data repository 980 and the user devices (e.g., 930a, 930b, 930c) and process information about the user and the user's environment. The remote computing system 920 can modify information obtained by a user device (e.g., 930a, 930b, 930c), for example, by selectively cropping an image of the user, modifying the user's background, adding virtual objects to the user's environment, annotating the user's speech with auxiliary information, etc. The remote computing system 920 can transmit the processed information to the same or a different user device. (Example of a telepresence session)

[0119] FIG. 9B depicts an example in which two users of separate wearable systems are conducting a telepresence session. Two users (named Alice 912 and Bob 914 in this example) are shown in this figure. The two users are wearing their respective wearable devices 902 and 904, which may include an HMD (e.g., display device 220 of system 200), described with reference to FIG. 2, to represent the virtual avatar of the other user in the telepresence session. The two users can conduct a telepresence session using the wearable devices. Note that the vertical line in FIG. 9B separating the two users is intended to illustrate that Alice 912 and Bob 914 may (but need not) be in two different locations while they communicate via telepresence (e.g., Alice may be in her office in Atlanta, while Bob is outside in Boston).

[0120] As described with reference to FIG. 9A, the wearable devices 902 and 904 may communicate with each other or with other user devices and computer systems. For example, Alice's wearable device 902 may communicate with Bob's wearable device 904, for example, via a network 990 (shown in FIG. 9A). The wearable devices 902 and 904 can track the user's environment and movements within the environment (e.g., via a separate outwardly facing imaging system 464 or one or more location sensors) and speech (e.g., via a separate audio sensor 232). The wearable devices 902 and 904 can also track the user's eye movements or gaze based on data obtained by the inwardly facing imaging system 462. In some situations, the wearable device may also capture or track the user's facial expressions or other bodily movements (e.g., arm or leg movements) when the user is in the vicinity of a reflective surface, and the outwardly facing imaging system 464 may capture a reflected image of the user and observe the user's facial expressions or other bodily movements.

[0121] The wearable devices can use the obtained information of the first user and the environment to animate a virtual avatar that will be rendered by the wearable device of the second user and create a tangible sense of the first user's presence within the second user's environment. For example, the wearable devices 902 and 904 and the remote computing system 920, alone or in combination, may process images or movements of Alice for presentation by Bob's wearable device 904, or may process images or movements of Bob for presentation by Alice's wearable device 902. As described further herein, the avatar can be rendered based on contextual information, such as, for example, the user's intent, the user's environment or the environment in which the avatar is rendered, or other biological characteristics of the human.

[0122] Although the examples refer to only two users, the techniques described herein should not be limited to two users. Multiple users (e.g., two, three, four, five, six, or more) using wearable devices (or other telepresence devices) may participate in a telepresence session. A particular user's wearable device may present the other users' avatars to that particular user during the telepresence session. Additionally, although the examples in this figure show the users standing in the environment, the users are not required to stand. Any of the users may be standing, sitting, kneeling, lying, walking, running, or in any position or state of movement during the telepresence session. The users may also be present in physical environments other than those described in the examples herein. The users may be present in separate environments or in the same environment while conducting a telepresence session. Not all users are required to wear their individual HMDs during the telepresence session. For example, Bob 914 wears a wearable device 904 while Alice 912 may use other image acquisition and display devices such as a webcam and a computer screen. (Example of a virtual avatar)

[0123] FIG. 10 illustrates an example of an avatar as perceived by a user of a wearable system. The exemplary avatar 1000 shown in FIG. 10 can be an avatar of Alice 912 (shown in FIG. 9B) standing behind a physical plant in a room. The avatar can include various characteristics such as, for example, size, appearance (e.g., skin color, complexion, hairstyle, clothing, and facial features such as wrinkles, moles, blemishes, acne, dimples, etc.), position, orientation, movement, posture, facial expression, etc. These characteristics may be based on a user associated with the avatar (e.g., Alice's avatar 1000 may have some or all of the characteristics of the real person Alice 912). As described further herein, the avatar 1000 can be animated based on context information, which can include adjustments to one or more of the characteristics of the avatar 1000. Although generally described herein as representing the physical appearance of a person (e.g., Alice), this is for purposes of illustration and not limitation. The avatar of Alice may represent the appearance of another real or fictional human being other than Alice, an anthropomorphized object, a creation, or any other real or fictional representation. Additionally, the plants in Figure 10 need not be physical, but may be virtual representations of plants presented to the user by the wearable system. Additional or different virtual content than that shown in Figure 10 may also be presented to the user. (Example of a rigging system for a virtual character)

[0124] An animated virtual character, such as a human avatar, can be represented in whole or in part as a polygon mesh in computer graphics. A polygon mesh, or simply "mesh" for short, is a collection of points in a modeled three-dimensional space. The mesh can form a polygonal object whose surface defines the body or shape of the virtual character (or a portion thereof). A mesh can include any number of points (within practical limits that may be imposed by available computing power), but a finer mesh with more points can generally depict more realistic virtual characters with finer details that may more closely resemble real people, animals, objects, and the like. FIG. 10 shows an example of a mesh 1010 around the eye of an avatar 1000. A mesh can include any number of points (within practical limits that may be imposed by available computing power), but a finer mesh with more points can generally depict more realistic virtual characters with finer details that may more closely resemble real people, animals, objects, and the like. 10 shows an example of a mesh 1010 around the eyes of an avatar 1000. The mesh 1010 may be relatively fine (e.g., have a relatively large number of points) to facilitate high-quality simulation of the complex curves and movements that occur within the eye region.

[0125] Each point in the mesh can be defined by coordinates in the modeled three-dimensional space. The modeled three-dimensional space can be, for example, a Cartesian space addressed by (x,y,z) coordinates. The points in the mesh are the vertices of polygons, which make up the polygonal object. Each polygon represents a surface, or face, of the polygonal object and is defined by an ordered set of vertices, and the edges of each polygon are straight edges that connect the ordered set of vertices. In some cases, the polygon vertices in a mesh can differ from geometric polygons in that they are not necessarily coplanar in the 3D graphic. In addition, the vertices of a polygon in a mesh can be colinear, in which case the polygon has zero area (referred to as a degenerate polygon).

[0126] In some embodiments, meshes are composed of three-vertex polygons (i.e., triangles or "triangles" for short) or four-vertex polygons (i.e., quadrilaterals or "quads" for short). However, higher order polygons may also be used in some meshes. Meshes are typically quad-based in direct content creation (DCC) applications (e.g., applications such as Maya (available from Autodesk, Inc.) or Houdini (available from Side Effects Software Inc.), which are primarily designed for creating and manipulating 3D computer graphics), while meshes are typically triangle-based in real-time applications.

[0127] To animate a virtual character, its mesh is deformed by moving some or all of its vertices to new positions in space at various instants. The deformations can represent both large-scale movements (e.g., limb movements) and fine-scale movements (e.g., facial movements). These and other deformations can be based on real-world models (e.g., photogrammetric scans of actual humans that implement body movements, joint movements, facial distortions, expressions, etc.), artistically-directed evolutions (which may be based on real-world sampling), combinations thereof, or other techniques. In the early days of computer graphics, mesh deformations could be accomplished manually by independently setting new positions for the vertices, but given the size and complexity of modern meshes, it is typically desirable to produce deformations using automated systems and processes. The control systems, processes, and techniques for producing these deformations are referred to as rigging, or simply, "rigs." The exemplary avatar processing and rendering system 690 of FIG. 6B includes a 3D model processing system 680, which can implement rigging.

[0128] Rigging for virtual characters can use a skeletal system to aid in mesh deformation. A skeletal system includes a collection of joints, which correspond to points of articulation for a mesh. In the context of rigging, joints are sometimes also referred to as "bones", despite the distinction between these terms when used in the anatomical sense. Joints in a skeletal system can move or otherwise change relative to one another according to transformations, which may be applied to the joints. The transformations can include translations or rotations in space and other movements. Joints can be assigned hierarchical relationships (e.g., parent-child relationships) relative to one another. These hierarchical relationships can allow one joint to inherit transformations or other properties from another joint. For example, a child joint in a skeletal system can inherit a transformation assigned to its parent joint, causing the child joint to move with the parent joint.

[0129] A skeletal system for a virtual character can be defined with joints in appropriate locations, and with appropriate local rotation axes, degrees of freedom, etc., to allow a desired set of mesh deformations to be performed. Once a skeletal system is defined for a virtual character, each joint can be assigned an amount of influence it has on various vertices in the mesh, in a process called "skinning." This can be done by assigning a weight value to each vertex for each joint in the skeletal system. When a transformation is applied to any given joint, the vertices under its influence can be automatically moved or otherwise modified based on that joint transformation by an amount that can depend on its individual weight value.

[0130] A rig can include multiple skeletal systems. One type of skeletal system is a core skeleton (also referred to as a low-order skeleton), which can be used to control large-scale movement of a virtual character. FIG. 11A illustrates an example of a low-order core skeleton 1100a. For example, for a human avatar, the core skeleton can resemble a human anatomical skeleton. A core skeleton for rigging purposes may not map exactly to an anatomically correct skeleton, but may have a subset of joints in similar locations with similar orientation and movement properties. Higher order joints and biomechanical linkages can be added to the core skeleton 1100a to build a skeleton that can perform more animation functions. FIG. 11B illustrates an example of a skeletal system 1100, with the core skeleton shown in solid lines and higher order joints shown in dashed lines.

[0131] As briefly mentioned above, a skeletal system of joints can be hierarchical, with, for example, parent-child relationships between joints. When a transformation (e.g., a change in position and / or orientation) is applied to a particular joint in the skeletal system, the same transformation can be applied to all other lower level joints in the same hierarchy. For example, in the case of a rig for a human avatar, a core skeleton may include separate joints for the avatar's shoulders, elbows, and wrists. Of these, the shoulder joint may be assigned the highest level in the hierarchy, while the elbow joint may be assigned as a child of the shoulder joint, and the wrist joint may be assigned as a child of the elbow joint. Thus, when a particular translation and / or rotation transformation is applied to the shoulder joint, the same transformation can also be applied to the elbow joint and the wrist joint, such that they are translated and / or rotated in the same manner as the shoulder.

[0132] Despite the connotation of the name, a skeletal system in a rig does not necessarily represent an anatomical skeleton. In rigging, a skeletal system can represent various hierarchies used to control deformations of a mesh. For example, hair may be represented as a series of joints in a hierarchical chain, skin movement resulting from the distortion of an avatar's face (which may represent facial expressions such as smiling, frowning, laughing, speaking, blinking, etc.) may be represented by a series of facial joints controlled by a facial rig, muscle deformations may be modeled by joints, and clothing movement may be represented by a grid of joints.

[0133] A rig for a virtual character may include multiple skeletal systems, some of which may drive the movement of the others. A lower-order skeletal system is one that drives one or more higher-order skeletal systems. Conversely, a higher-order skeletal system is one that is driven or controlled by a lower-order skeletal system. For example, the movement of a character's core skeleton may be manually controlled by an animator, but the core skeleton may in turn drive or control the movement of the higher-order skeletal systems. For example, higher-order helper joints (which may not have an anatomical analog in the physical skeleton) may be provided to refine the mesh deformation resulting from the movement of the core skeleton. Transformations applied to these and other joints in the higher-order skeletal system may be algorithmically derived from transformations applied to the lower-order skeleton. The higher-order skeleton may represent, for example, muscle, skin, fat, clothing, hair, or any other skeletal system that does not require direct animation control.

[0134] As previously discussed, a transformation can be applied to a joint in a skeletal system to perform mesh deformation. In the context of rigging, a transformation includes a function that receives one or more given points in 3D space and produces an output of one or more new 3D points. For example, a transformation can receive one or more 3D points that define a joint and output one or more new 3D points that define the transformed joint. A joint transformation can include, for example, a translation component, a rotation component, and a scale component.

[0135] A translation is a transformation that moves a set of one or more specified points in the modeled 3D space by a specified amount without changing the orientation or size of the set of points. A rotation is a transformation that rotates a set of one or more specified points in the modeled 3D space about a specified axis by a specified amount (e.g., rotate all points in a mesh 45 degrees about the z-axis). An affine transformation (or six degrees of freedom (DOF) transformation) is one that includes only translations and rotations. The application of an affine transformation can be viewed as moving a set of one or more points in space without changing their size (although their orientation can change).

[0136] A scale transformation, on the other hand, modifies one or more defined points in a modeled 3D space by scaling their individual coordinates by a defined value. This changes the size and / or shape of the transformed set of points. A uniform scale transformation scales each coordinate by the same amount, while a non-uniform scale transformation can independently scale the (x,y,z) coordinates of a defined point. A non-uniform scale transformation can be used to provide contraction and stretching effects, such as those that may result from muscle action. Yet another type of transformation is a shear transformation. A shear transformation modifies a set of one or more defined points in a modeled 3D space by translating the coordinates of the point by different amounts based on the distance of that coordinate from an axis.

[0137] When a transformation is applied to a joint, moving it, the vertices under the influence of that joint are also moved. This results in a deformation of the mesh. As discussed above, the process of assigning weights and quantifying the influence of each joint on each vertex is called skinning (or sometimes "weight painting" or "skin weighting"). Weights are typically values ​​between 0 (meaning no influence) and 1 (meaning full influence). Some vertices in a mesh may only be influenced by a single joint. In that case, those vertices are assigned a weight value of 1 for that joint, and their positions are changed based on the transformation assigned to that specific joint and not the others. Other vertices in a mesh may be influenced by multiple joints. In that case, separate weights are assigned to those vertices for all of the influencing joints, and the sum of the weights per vertex is equal to 1. The positions of those vertices are changed based on the transformations assigned to all of their influencing joints.

[0138] Making weight assignments for all of the vertices in a mesh can be extremely labor intensive, especially as the number of joints increases. Balancing the weights and achieving the desired mesh deformation in response to transformations applied to the joints can be very difficult even for highly trained experts. For real-time applications, the task can be further complicated by the fact that many real-time systems also impose limits on the number of joints (generally eight or fewer) that can be weighted to a particular vertex. Such limits are typically imposed for purposes of efficiency within a graphics processing unit (GPU).

[0139] The term "skinning" also refers to the process of actually deforming a mesh using assigned weights based on transformations applied to joints in a skeletal system. For example, a series of core skeletal joint transformations may be defined by an animator to produce a desired character movement (e.g., a running movement or a dance step). Once a transformation is applied to one or more of the joints, new positions are calculated for the vertices under the influence of the transformed joints. The new position for any given vertex is typically calculated as a weighted average of all the joint transformations that affect that particular vertex. There are many algorithms used to calculate this weighted average, but the most common, and the one used in most real-time applications due to its simplicity and ease of control, is linear blend skinning (LBS). In linear blend skinning, a new position for each vertex is calculated using each joint transformation for which that vertex has a non-zero weighting. The new vertex coordinates resulting from each of these joint transformations are then summed in proportion to the individual weights assigned to that vertex for each joint. In reality, there are well-known limitations to LBS, and much of the work in creating high quality rigs (e.g., for movies) goes into finding and overcoming these limitations. Many helper articulated systems are designed specifically for this purpose.

[0140] In addition to skeletal systems, "blend shapes" can also be used in rigging to produce mesh deformations. A blend shape (sometimes called a "morph target" or simply a "shape") is a deformation applied to a set of vertices in a mesh, where each vertex in the set moves in a specified direction by a specified amount based on a weighting. Each vertex in the set can have its own custom motion for a specific blend shape, and moving the vertices in the set simultaneously will produce the desired shape. The custom motion for each vertex in a blend shape can be specified by a "delta", which is a vector that represents the amount and direction of XYZ motion applied to that vertex. Blend shapes can be used, for example, to produce facial deformations, moving eyes, lips, eyebrows, noses, dimples, etc., to name only a few possibilities.

[0141] Blendshapes are useful for deforming meshes in artistically oriented ways. They offer a large amount of control as precise shapes can be sculpted or captured from a scan of the model. However, the benefits of blendshapes come at the cost of needing to store deltas for every vertex within the blendshape. For animated characters with fine meshes and many blendshapes, the amount of delta data can become significant.

[0142] Each blend shape can be applied to a defined degree by using blend shape weightings. These weights typically range from 0 (the blend shape is not applied at all) to 1 (the blend shape is fully active). For example, a blend shape for moving a character's eyes can be applied with a small weighting to move the eyes a small amount, or with a large weighting to create a larger eye movement.

[0143] Rigs may be applied to multiple blendshapes to combine with each other to achieve a desired complex deformation. For example, to produce a smile, rigs may be applied to blendshapes to pull the corners of the mouth, lift the upper lip, lower the lower lip, and move the eyes, eyebrows, nose, and dimples. The desired shape from combining two or more blendshapes is known as a combined shape (or simply, a "combo").

[0144] One problem that can arise from applying two blendshapes in combination is that the blendshapes may affect some of the same vertices. When both blendshapes are active, the result is called a dual transform or "breakaway model". The solution to this is typically the corrective blendshape. Corrective blendshapes are specialized blendshapes that do not represent the desired deformation relative to a neutral point, but rather represent the desired deformation relative to the currently applied deformation. Corrective blendshapes (or simply "corrective") can be applied based on the weighting of the blendshapes they correct. For example, the weighting for a corrective blendshape can be made proportional to the weighting of the underlying blendshape that triggers the application of the corrective blendshape.

[0145] Corrective blendshapes can also be used to correct skinning anomalies or improve the quality of a deformation. For example, a joint may represent the motion of a specific muscle, but cannot represent all the nonlinear behavior of skin, fat, and muscle as a single transformation. Applying a correction or a series of corrections as the muscle activates can result in a more attractive and convincing deformation.

[0146] Rigs are built in layers, with lower and simpler layers often driving higher layers. This applies to both skeletal systems and blendshape deformations. For example, as already mentioned, rigging for an animated virtual character may include a higher skeletal system controlled by a lower skeletal system. There are many ways to control a higher skeletal system or blendshapes based on a lower skeletal system, including constraints, logic systems, and pose-based deformations.

[0147] A constraint is typically a system in which a particular object or joint transformation controls one or more components of the transformation applied to another joint or object. Many different types of constraints exist. For example, an aim constraint changes the rotation of a target transformation to point in a specific direction or to a specific object. A parent constraint acts as a virtual parent-child relationship between pairs of transformations. A position constraint constrains a transformation to a specific point or object. An orientation constraint constrains a transformation to a specific rotation of an object. Additional discussion of constraints that may be used in the disclosed joint resolution embodiments is provided below.

[0148] Logic systems are systems of mathematical equations that, given a set of inputs, produce some output. They are specified rather than learned. For example, a blendshape value may be defined as the product of two other blendshapes (this is an example of a compensating shape known as a combination or combo shape).

[0149] Pose-based deformations can also be used to control higher-order skeletal systems or blendshapes. The pose of a skeletal system is defined by a set of transformations (e.g., rotations and translations) for all joints in that skeletal system. Pose can also be defined for a subset of joints in a skeletal system. For example, an arm pose can be defined by transformations applied to the shoulder, elbow, and wrist joints. A Pose Space Deformer (PSD) is a system that is used to determine a deformation output for a particular pose based on one or more "distances" between that pose and a defined pose. These distances can be a metric that characterizes the degree to which one of the poses differs from the others. A PSD can include a pose interpolation node, which, for example, takes a set of joint rotations (which define the poses) as input parameters and in turn outputs normalized per-pose weights to drive a deformer such as a blendshape. A pose interpolation node can be implemented in a variety of ways, including with a radial basis function (RBF). RBFs can implement machine-learned mathematical approximations of functions. An RBF can be trained using a set of inputs and their associated expected outputs. The training data can be, for example, multiple sets of joint transformations (defining specific poses) and corresponding blendshapes (or linear skins) to be applied in response to those poses. Once the function is learned, new inputs (e.g., poses) can be given and their expected outputs can be efficiently calculated. RBFs are a type of artificial neural network. RBFs can be used to drive higher level components of a rig based on the state of lower level components. For example, the pose of a core skeleton can drive helper joints and compensations at the higher level.

[0150] These control systems can be chained together to implement complex behaviors. As an example, an eye rig may contain two "look around" values ​​for horizontal and vertical rotation. These values ​​can be passed through some logic to determine the exact rotation of the eye joint transform, which in turn can be used as input to an RBF that controls a blend shape that changes the shape of the eyelid to match the eye position. The activation values ​​of these shapes can be used to drive other components of the facial expression, using additional logic, etc.

[0151] The goal of a rigging system is typically to provide a mechanism for producing compelling, high-fidelity transformations based on a simple, human-understandable control system. For real-time applications, the goal is typically to provide a rigging system that is simple enough to be run in real-time, e.g., on the VR / AR / MR system 200, while resulting in as few compromises as possible to the final quality. In some embodiments, the 3D model processing system 680 executes the rigging system to animate the avatar in real-time within the mixed reality environment 100, interactively (with a user of the VR / AR / MR system) and providing appropriate contextual avatar behavior (e.g., intent-based behavior) within the user's environment. (Introduction of a skeletal system for geometric deformation of virtual avatars)

[0152] As discussed herein, rigging includes techniques for transferring information about an avatar's body transformations (e.g., facial contortion, arm movement, leg movement, torso movement, head rotation and tilt, hand movement including grasping and pointing, etc.) onto a mesh of vertices. Animating a mesh (e.g., animating an avatar) may include deforming the mesh by moving some or all of the vertices that form the mesh to new positions in 3D space. The example avatar processing and rendering system 690 of FIG. 6B includes a 3D model processing system 680, which can implement a rig (e.g., a control system for deforming the mesh and animating the avatar). As a mesh may include a large number of vertices, a rig typically provides common desired transformations as computerized commands, making it easier to control the mesh.

[0153] For high-end visual effects productions such as movies, there may be ample production time for the rig to perform extensive mathematical calculations to achieve highly realistic animation effects. In other words, this context typically prioritizes high quality at the expense of high speed.

[0154] However, for real-time applications (such as in mixed reality), speed of deformation may be very advantageous and different rigging techniques may be used. In other words, the present context typically prioritizes high speed at the expense of high quality. Rigs often utilize deformations that rely on a skeletal system, linear blend skinning, or optionally blend shapes. Techniques for generating parameters that a rig may use for avatar skinning transformations are described below.

[0155] Linear blend skinning (LBS) is a popular technique for animating digital puppets in which a mesh (or other geometric object) is deformed based on a driven skeleton consisting of a hierarchy of joint transformations. Figure 11B illustrates an example of a skeleton 1100 with a hierarchical order of joints. A mesh (e.g., with polygonal vertices) can be draped over the skeleton and used to represent the skin or clothing of an avatar or the deformable surface of another virtual object.

[0156] The relationship between the skeleton joints and the base (neutral) pose of the skeleton can be used to define an association between each vertex of the mesh and each of the joints in the skeleton. For a new pose of the skeleton, a transformed mesh (representing the new pose) can be calculated by determining the relative transformation (e.g., rotation and translation) of each of the joints from its base pose to the new pose. Not all joints affect all vertices. For example, moving an avatar's left leg will typically not cause any movement of the avatar's right hand. However, moving an avatar's right forearm will typically cause the avatar's right hand to move. A weight map, comprising a set of per-joint and per-vertex weights, can be used to measure the amount of influence of any joint on any vertex. If a joint does not affect a vertex, the associated weight is zero.

[0157] For each joint, the joint relative transformation is applied to each vertex that has a non-zero weight for that joint, and the new vertex coordinates are stored. The final vertex position for all vertices is the weighted sum of the transformed vertex positions for each joint where the joint has a non-zero weight. Thus, as the skeleton moves, the rig can move the mesh vertices (e.g., representing the skin or clothing that covers the skeleton) via the LBS skinning parameters. Additional technical details of LBS are described below.

[0158] One potential difficulty in LBS or any direct skinning method is determining the weighting map for each joint. In practice, these are often set manually in a process known as weight painting, which is time and labor intensive and prone to user error. The complexity increases as the number of joints that affect a given region of the mesh increases. In addition, in real-time applications there is often a difficulty in limiting the maximum number of joints that can affect a specific part of the geometry, which in turn requires careful evaluation and tradeoffs to achieve high quality under strict real-time limits.

[0159] In some cases, LBS tends to be a relatively low quality method for deforming a mesh, so blendshapes are sometimes added on top of LBS to achieve higher quality deformation of a mesh. Blendshapes move one or more vertices according to a control value between 0 (e.g., neutral or rest shape) and 1 (e.g., full blendshape) and can have specific blendshape values ​​for specific training poses, but are not directly tied to the skeleton of a rig. Using LBS and blendshapes requires a lot of calculations (e.g., LBS calculations, blendshape calculations, for many vertex and bone movements, etc.), which can be computationally difficult for a rigging control system, especially in real time.

[0160] Thus, one goal for the skinning techniques described in detail below is to develop a system that provides high quality, well-generalized skinning, based on training data for poses, and that operates in a real-time context. In part to achieve this goal, embodiments of the system are highly user-configurable. While many rigging systems are designed to require minimal user input, such systems often suffer from common drawbacks that make them undesirable for MR or real-time applications. For example, some common disadvantages include one or more of the following:

[0161] One example of a disadvantage is the dependency on a large number of joints: although multiple joint transformations may reduce the error of the skinning approximation, they do not meet the limits enforced in real-time systems, do not generalize well to new animations, require more control calculations, and are generally not easily understood by humans, limiting their ability to be debugged in context.

[0162] A second example of a disadvantage is that problems of overfitting may arise, which may require large amounts of input data or difficult to understand control statistics to prevent.

[0163] Embodiments of the disclosed systems and methods reduce or avoid the likelihood of some or all of these disadvantages by utilizing human-understandable constraints that describe the properties of the resulting system. These constraints allow the system to (optionally) receive human user input to achieve results based on a better human understanding of domain-specific knowledge. (Example of training data)

[0164] Parameters (e.g., weight maps, rotations, and translations) that represent the transformations of the joints of the skinning system can be extracted from training data that represents a set or sequence of target poses (e.g., facial expressions, body gestures, movements (e.g., sitting, walking, running, etc.)). The skinning parameters can be learned from the training data using various machine learning or artificial intelligence techniques (described further below). The process of learning the skinning parameters from training data is sometimes also referred to as joint decomposition.

[0165] The training data comprises a mesh that shares topology with a base neutral mesh used for animation of the avatar, but that has been deformed to a target pose used to skin the avatar. The training data can represent poses or blendshapes for the avatar. The training data can come from any available source, including one or more of the following:

[0166] Scanning ground truth. The results of the machine learning model can either under-fit or over-fit the training data depending on the amount, type, and format of the training data set. The training data can be generated by scanning the object (e.g., a model) in a series of poses. The scanned data geometrically represents the object in three dimensions in the pose sequence with respect to the articulation points and the deformation ground truth. Imaging scanning can be performed, for example, by placing the object in a photogrammetry capture stage with multiple cameras (e.g., two or more) surrounding and pointed at the object. The cameras can be synchronized with each other and capture images that can be converted into a 3D scan of the posed object. For static poses, a single scan can be generated. However, for avatars that can move and express emotions, scans can be performed while the object assumes training poses that include basic expression units (e.g., smiling, frowning, making gestures, walking, etc.). These scans can be used to create a training data set that is used to build skinning parameters that can be used to create a convincing digital avatar of the object in action.

[0167] For example, the scanned images can be used to generate pose correspondences between the avatar's low-order core skeleton (e.g., see example low-order skeleton 1100a in FIG. 11A or solid line portion of skeleton 1100 in FIG. 11B) and the geometric space of the scanned data. The resulting training data can be used to train a joint resolution system (which determines skinning parameters from the training data) in the generalized statistical space. For example, improved skinning parameters can be determined from an initial set of skinning parameters. The resulting statistical space can represent any animation or joint motion of the avatar and can be evaluated in real time, for example, using a radial basis function (RBF).

[0168] Professionally carved sculpture. This method is often used when real-world image capture data is unavailable, impractical, or too expensive to obtain, or is of poor quality. It is also used when a more artistically oriented focus is desired, such as in 3D animation. This form of data is labor intensive, often interpreted by humans with scarce resources, and can be prone to error.

[0169] simulation. Clothing and tissue simulations are often used in visual effects production to achieve high fidelity and lifelike deformations. While these practices often produce optimal results, they can be computationally intensive and not suitable for real-time applications. In an implementation of the joint decomposition system, deformations such as skin or clothing can be simulated using real-time graphics. By running these simulations through a range of motion (ROM), the system can decompose the deformation space into statistical models of blendshapes and weights. The weights of the statistical models can then be interpolated using RBF for real-time execution.

[0170] Biometrically derived models. Biomechanical higher order skeletons for common types of bones and muscles can be represented in the character's skeletal space via joints and LBS. Muscles exhibit finite volume properties, but the stress and strain rates on muscles can be modeled in the scale matrices of the joints used in the representation. Bones are more focused in rigid space, but require more mechanical properties and structure to achieve the desired motion. The mechanics of bones and muscles can then be captured by joint decomposition. (Overview of Linear Skinning Systems and Methods)

[0171] Linear skinning (or linear blend skinning) is a method of transforming the positions of a set of points in 3D space based on the transformation of a set of joints. The points can be the vertices of a mesh used to animate the avatar. In what follows, a skeleton has J joints, indexed by j = 1, 2, ..., J, and there are a total of V points of interest, indexed by i = 1, 2, ..., V. Each joint can be transformed by a rotation R j and translation T j For a given point v, i is the formula R j v i +T j Then, we use the joint j to find the new point v i ' can be converted to

[0172] For each joint and point, weight w ij can represent the influence of the jth joint on the ith point. The weights quantify the amount of influence that a particular joint has on a particular point. All weights w ij The collection of weights is sometimes referred to as a weight map. The individual weights in a weight map may satisfy some constraints. For example, each weight is typically a non-negative value w ij ≧0, and the total influence of all J joints on any point i can be normalized to 1, i.e., [ka] It becomes.

[0173] Skinning parameters (e.g. weight map, rotation R j , and translation T j There may also be other constraints on M. For example, the number of non-zero weights for any point may be limited to a maximum value to reduce the computational burden on the processor animating the mesh and to reduce overfitting. The maximum number M may be, for example, 4 or 8 or 12, and can be selected based on the memory model of the processor (e.g., a commercial Graphics Processing Unit (GPU)) used for animation. A rotation R is used to represent a valid rotation in 3D space. j may be expressed in a number of different mathematical ways, for example as an orthogonal matrix, which avoids shearing or scaling due to the transformation.

[0174] point v i The transformed coordinates v i ' is the sum of the weighted transformations for each concrete joint. [ka] (Exemplary Joint Disassembly)

[0175] The goal of the joint resolution process is to find the transformed position v i ' (computed from equation (1)) and the reference positions v of these points at the target poses in the training data (described above). i TD The skinning parameters w are adjusted to reduce or minimize the error between ij , R j , and T jThe decomposition process can be formulated as a constrained optimization problem, where the error can be expressed as a Euclidean or L2-norm between the transformed positions and the target pose positions in the training data. Other error functions can also be used (e.g., median or L1-norm). Various constraints on the solution to the constrained optimization problem are described above, and additional constraints are described below.

[0176] For example, constrained optimization can reduce or minimize the error E, which in some embodiments (using the L2-norm) can be expressed as follows: [ka]

[0177] Depending on the constraints, optimization methods can include linear or quadratic programming. However, reducing or minimizing errors when there are many rotational components subject to complex constraints often requires advanced nonlinear programming methods. For example, some techniques use first and second derivative estimates and apply nonlinear solvers that operate for arbitrary constraints. Other techniques are iterative, where alternate pointwise weights or joint transformations are locked and other components are optimized. In various implementations, optimization techniques include gradient descent, block coordinate descent, downhill simplex, simulated annealing, Levenberg-Marquardt optimization, least squares, nonlinear optimization, heuristic or metaheuristic (e.g., Tabu) search, weighted absolute orientation solvers, etc.

[0178] The constrained optimization algorithm can receive as input an initialization skeleton (e.g., initial positions of the skeleton joints), a mapping of mesh vertices (e.g., skin clusters) to the joints, and a mesh representing a target pose (or multiple target poses) for the avatar. The constrained optimization algorithm calculates the transformed position v i' (e.g., calculated from equation (1)) and the reference positions of these points at the target pose in the training data (e.g., calculated from equation (2)) converge (e.g., the error E falls below an error threshold for the transformed positions v i The output of the constrained optimization algorithm is the skinning parameters (e.g., w ij , R j , and T j ) In some embodiments, the joint resolution system can additionally or alternatively output an updated skeleton with joint or vertex positions that reflect a fit to the target training pose. In some embodiments, the joint resolution system can additionally or alternatively output a radial basis function (RBF) formulation of the skinning parameters.

[0179] Some embodiments of the joint disassembly system differ from conventional systems based on the inputs used for initialization and the constraints under which the system operates.

[0180] For example, many decomposition systems take only the deformed mesh as input and infer the number and location of joints and their mesh weights from this data. Some systems even attempt to generate a hierarchical skeleton from the input training data. While such systems can work, in practice the unconstrained nature leads to incorrect behavior (e.g., biometrically irrational solutions) and lack of generalization ability, either through overfitting or finding unacceptable local minima in larger contexts. (Example Inputs to the Joint Decomposition Process)

[0181] A skinned mesh requires a mesh and a bone hierarchy, and bone transformations drive all deformations on the mesh. In practice, working directly with the bone hierarchy to directly deform the mesh can be disadvantageous. Often there are many extra bones that are included to obtain better deformations of the avatar. Animating these extra bones directly can significantly slow down the animation task. In addition, these bones often involve aspects of the avatar, such as volume preservation, that are difficult for a human animator to properly understand, but are typically easy for a computer to compute. For automated animation tasks, such as motion capture, it can be impractical to keep track of many of these bones, which may not directly correspond to any single actual physical element. Furthermore, higher layers are often used for optional components, such as hair or clothing, that can be modified or replaced on an existing character without changing the base animation.

[0182] Instead, in practice, a simple lower-order skeleton may be made progressively more complex by the addition of procedurally animated components into a higher-order skeleton. FIG. 11A shows an example of a lower-order skeleton 1100a, and FIG. 11B shows an example of a higher-order skeleton 1100 (dashed line). These control systems include new bones and create the higher-order skeleton and control system, which animates these new bones based on transformations of the lower-order skeleton used as input. For example, the lower-order skeleton may represent bones of the avatar's body that correspond to the avatar's actual skeleton. The higher-order skeleton may include bones that represent the avatar's muscles. These additional bones may be driven by the position of the lower-order skeleton. For example, the biceps "bone" position may be controlled by the corresponding forearm bone in the lower-order skeleton, which represents, among other things, the elbow angle at that point in the animation.

[0183] These low to high skeletal systems can be very complex, involving, for example, volume preservation, dynamics (e.g., muscle / fat jiggling), and other systems. In many animation rigs, there can be multiple layers of low and high skeletons, with the higher output of one skeletal system becoming the lower input of the next. As an example, imagine an avatar where the lowest skeleton represents the avatar's "real" skeleton, a second skeletal system represents the muscles, and a third skeletal system represents the fat layered on top of the muscles. Additional skeletal systems can also be added to represent hair, clothing, or props.

[0184] Thus, in some implementations, a lower-order skeleton may comprise a set of bones and joints that represent or are similar to the anatomical bones and joints (and / or their interconnections) of an avatar to be animated. A higher-order skeleton (with components added onto the lower-order skeleton) may represent muscles, ligaments, fat, hair, clothing, etc. A lower-order skeleton may comprise a portion of a set of bones or joints that define a digital character's skeleton (e.g., similar to the lower-order skeleton 1100a shown in FIG. 11A, which is similar to bones in an actual human skeleton). A lower-order skeleton may represent the base of a hierarchical skeletal structure. A higher-order skeleton may be considered the difference between a complete skeleton (e.g., the skeletal system 1100 shown in FIG. 11B) and a lower-order skeleton 1100a.

[0185] Some joint breakdown systems may start with no skeleton at all and infer all bones and bone relationships from animation. Some such systems may be disadvantageous because they do not allow for the specification of constraints that allow certain advantageous embodiments described herein to generalize well.

[0186] When performing joint decomposition, the system or method can determine the weights and motions of the final layer of deformation, e.g., the highest order skeleton. The system and method can receive at least a hierarchy of higher order skeletons as input, so the system and method can perform optimization. However, in some cases, it may not be necessary for all bones in the higher order skeleton to be initialized. For example, the system and method can be initialized with a base skeleton pose with additional higher order bones set to default values. The system and method can determine the best transformation of the additional bones (e.g., starting from default values).

[0187] Thus, the present systems and methods may receive higher order skeletal structure as input, but may not require the pose of the higher order skeletal structure.

[0188] An embodiment of the joint resolution system can be given an initial skeleton, with optional target pose or skin weighting. The skeleton used to initialize the constrained optimization algorithm can be a higher-order skeleton that represents joint and mesh positions close to the target pose, rather than a base skeleton in a neutral or resting pose (which is typically not related to the target pose). Starting the resolution in this way, rather than a blank slate, can provide much more flexibility and reduces or prevents the likelihood of mis-learning. This also leads to more efficient optimization because the input skeleton and skinning parameters are already close to the desired result (i.e., to match the target pose). In practice, the initial starting skeleton data can be a very rough match to the desired target. For example, a mesh can be segmented such that each vertex is assigned to a single bone (see, for example, FIG. 12 for an example of hand segmentation 1200). Such a skinning solution is generally not acceptable, but can be used as an initial input and be sufficient to orient the joint resolution system in the desired direction. (Example linear skinning constraint)

[0189] Although a base form for linear skinning can be formulated as equation (1), the optimization equations can be very complex. This is due to the interdependencies implied by the constraints. The various joint resolution systems described herein can enforce various constraints, some of which can be user-selectable or dynamically changed during joint resolution. Embodiments of the joint resolution system can receive user input and better guide the system in finding a biometrically reasonable solution, and better define the problem space in which the optimization techniques will operate.

[0190] Constraints common to many LBS joint decomposition systems are discussed above, including non-negativity and regularization constraints for the weight maps, and the rotation parameters representing valid rotations (e.g., R j (x,y,i) is an orthogonal matrix, and the maximum number of nonzero weights per vertex, M.

[0191] Other constraints include parent-child relationships between joints in a hierarchical skeletal system. For example, the wrist joint can be a child of the forearm joint, and so on. Such constraints can enforce rigid bone-like behavior by restricting a child joint to a fixed position relative to its parent's local space (e.g., a position that is a fixed distance away from the parent joint to represent the bone's rigid behavior). This fixed position can be estimated on a pose-by-pose basis or using all target poses. Using all target poses can be advantageous to fit the skeleton to a captured character. In many hierarchical skeletons, the length of a joint relative to its parent is fixed during animation, and only the orientation is allowed to change. This constraint is similar to real-world bones, which do not easily change their length.

[0192] Relationships between joints can further complicate the optimization equations. For example, a parent-child relationship between two joints j1 and j2 can be expressed as the rotation and translation parameters R j1 , and T j1 is the rotation and translation parameter R j2 , and T j2It may mean that the weights of different points are related to each other, and this has to be taken into account during the constrained optimization. Similarly, if the weights of different points are related to each other, the optimization equation system should include these relationships.

[0193] Smoothness constraints may enforce similar weighting of neighboring points in a joint or vertex hierarchy. For example, the system may restrict vertices to use similar weighting with vertices that are topologically close on the mesh. Smoothness constraints may advantageously create smoother skin deformations. Vertices may be topologically close to another vertex if they are neighbors or neighbors of neighbors (e.g., within a threshold number of edges that are less than 3, 5, 7, 10, 12, etc. away on the mesh). Smoothness may be constrained by not allowing weights of vertices that are topologically close to each other to differ from each other by more than some smoothness factor.

[0194] In some implementations, joints may be constrained by a geometric model (e.g., length, lateral size, limits on range of motion (ROM) relative to neighboring joints, etc.) that represents the underlying anatomical structure that the joint represents. For example, joints representing phalanges may be given a radius and length and a range of motion so that they do not bend backwards on the upper surface of the hand. Applying joint constraints during the decomposition process may be advantageous and lead to a more biometrically reasonable joint decomposition.

[0195] One or more other novel constraints may additionally or alternatively be incorporated into the disclosed system embodiments. The following provides a non-limiting list of various constraints that may be used during the decomposition process.

[0196] Joint positions can be locked to prevent them from being estimated or changed during the constrained optimization process. For example, a joint (or a group of joints or the vertices associated with those joints) can be locked when the decomposition process optimizes another subset of the skeleton or mesh (and then unlocked when the process optimizes that joint or group of joints). For example, a leg joint can be locked while solving for skinning parameters for a wrist joint.

[0197] By extending the joint parent-child relationships described above, joint positions can be constrained to translate only along a particular axis or axes of the parent joint to produce a more robust rig. For example, joint positions can be locked or unlocked along specific axes, as described above, to determine bone length.

[0198] Similar to locking or unlocking joint positions, joint orientations, radii, lengths, etc. may also be locked or unlocked in a similar manner.

[0199] A joint or vertex can be limited to weighting only for joints that are geometrically close. For example, a vertex on an arm may have zero weighting for joints in a leg, reflecting the anatomical constraint that leg and arm movement are independent. A joint or vertex may be considered geometrically close when a particular joint tends to move when another joint moves (e.g., a hand tends to move when the arm to which it is attached moves, but the hand does not move when the other arm moves).

[0200] Joint distance constraints can also be applied. By utilizing a joint geometric model, the system can calculate the distance of a vertex from a joint and use this distance in the optimization procedure. For example, a vertex can be excluded from use of a joint (e.g., weighting is set to 0) if the distance is too large, e.g., above a threshold. As another example, the weighting can be non-zero only for vertices within a threshold distance of the joint. The distance between joints (or vertices) can be calculated as a distance traversed along the skeleton of the mesh (e.g., see skeleton 1100 shown in FIG. 11B), for example, since the joints / vertices can represent a hierarchical order of interconnections.

[0201] Vertex and joint selection constraints can also be applied. The system can define multiple enumerated selections of vertices and joints, which can be combined in various ways. For example, a vertex selection can be locked to a specific weighting, or weighted to a specific selection of joints, or excluded from the optimization process entirely.

[0202] Weights can be optimized or locked per joint, per vertex, or per joint-vertex pair through the use of segmentation sets, which associate a specific vertex or set of vertices with a set of specific joints.

[0203] The list of constraints above is not intended to be comprehensive. The system can receive user input to select the constraints to be applied or the parameters used for the constraints (e.g., distance threshold, segmentation set, etc.). The constraints (or constraint parameters) can change dynamically during joint decomposition. For example, the distance threshold may start at a relatively small value (e.g., for nearby joints) and then increase as the joint decomposition progresses. Thus, the system can be highly configurable to better define the problem space in which the joint decomposition will operate. (Example application to a real-time rigging system)

[0204] The statistical pose space used for joint decomposition is considered and planned from end to end. The results of joint decomposition are deterministic depending on the information with which the decomposition is seeded. If joint decomposition is allowed to generate skinning transformations without constraints, the results may not be usable or human interpretable. An embodiment of the joint decomposition technique refines or optimizes the behavior of higher order skeletons to match training data (e.g., ground truth scan data). By seeding the joint decomposition system with valid information, the system tends to avoid mismatches and produce results that are qualitative and practical for applications in the domain of mixed reality or real-time rigging systems for games.

[0205] Segmentation refers to creating a mapping between a set of concrete vertices and a set of joints. This mapping can isolate and localize the results of joint decomposition. Segmentation can also increase the performance of the system due to the reduced number of calculations required to achieve a converged result. For example, as explained above, joint decomposition can lock the weights for all joints except those found in the segmented set, and only the associated vertices can be analyzed and the LBS skinning parameters optimized.

[0206] As an example, if an animator wishes to decompose only the hand, a segmentation set can be generated for each individual finger, for example to prevent the middle finger joint from affecting the ring or index finger, and to localize the results of the decomposition within the skeletal hierarchy such that joints that biomechanically control vertices are optimized while joints that do not (e.g., outside the segmentation set) are locked.

[0207] The joint decomposition can (optionally) be integrated with a falloff radius, which blends the results for a particular region with an existing skin cluster or LBS. For example, the falloff can feather the skinning parameters found for one set of joints to another set of joints. As an example, the falloff may utilize three regions: a first region where the skinning transformation is calculated, a second, more distant region that is not modified by the decomposition (e.g., locked), and an intermediate region between the first and second regions where the falloff parameters smoothly transition the skinning parameters from the first region (e.g., within the segmentation set) to the second region (e.g., outside the segmentation set). The falloff may implement a weighted average in the intermediate region.

[0208] FIG. 12 illustrates an example of joint decomposition for hand deformation. In this example, the joint decomposition used a falloff and segmentation set per finger. In the hand 1200, the different cross-hatchings represent the segmentation or binding of vertices to a particular set of finger joints. The hand 1200 can be used as an input to initialize the joint decomposition process, and the hand 1208 represents the target hand pose (in this case the blendshape for the desired hand deformation). The hand 1204 shows an example result from the joint decomposition of the hand. The hand 1212 is overlaid with a heatmap showing areas (in different shades of grey) where the error between the target hand 1208 and the LBS decomposed hand 1204 is greater (darker grey) or less (lighter grey). The LBS joint decomposition provides a good match to the target hand 1208, with some error in the finger pads behind the thumb and between the index and middle fingers.

[0209] Another aspect of the user-directed capabilities of joint decomposition is the ability to set transformation limits for one or more of the joints. For example, during the decomposition process, the rotation and translation parameters of the joints (e.g., R j and T j) may be updated up to 6 DOF and, in conjunction with constrained optimization of mesh weighting, allowed to match the conformal nature of the target. If constraints are not imposed on the optimization, the joints may overfit to the target training data. Overfitting can have poor prediction and computational performance and be sensitive to even slight variations in the input training data. Undesirable variance in the output from overfitting can appear as high frequency artifacts, commonly referred to as jumping or popping.

[0210] This overfitting is undesirable and may be considered unusable within statistical space and pose space transformation applications. By applying constraints, the optimization brings about practical thresholds to reduce or avoid popping resulting from overfitting. Various such constraints are described above (e.g., joint position or orientation locks, joint distance constraints, segmentation sets, etc.). For example, constraints that limit the joints of the segmentation to move within a subset of 3D space (e.g., a range of x, y, or z values, or a range of yaw, pitch, or roll angles) can be applied to make the joint behavior conform to the way the actual joints move biometrically.

[0211] For a biomechanical rigging with a joint system, e.g., skeleton 1100, refined or optimal position or rotation information for that joint and its associated skin cluster weighting can be determined and applied in the joint decomposition. If a joint is not provided or targeted, the system (or via user input) can generate a joint that can be added to the skin cluster.

[0212] 13 illustrates an example of joint decomposition for jaw deformation. In this example, three extra joints were added to the jaw contour, as opposed to a single joint for the entire jaw, and constraints were imposed on the degrees of freedom in which the extra jaw contour joints may move. Head 1300 shows the input skin (mesh) used to initialize the joint decomposition, head 1312 shows the target pose (e.g., in this example, a blendshape), and head 1304 shows the result of the joint decomposition. Head 1308 is overlaid with a heatmap showing an example of the error between the decomposition result 1304 and the target 1312. The match is very good throughout most of the face, with areas of larger error in the lower neck. Additional Aspects and Advantages of the Joint Disassembly System and Method

[0213] The problem of high quality, well-generalized real-time mesh deformation of digital characters can be solved by deriving mesh (e.g., skin) weights and joint positions from training data. Mesh weights and joint positions can be determined by seeding a constrained optimization engine with an initial target shape, high-order skeleton, and skin clusters using human-understandable constraints (e.g., utilizing error minimization).

[0214] Embodiments of joint decomposition systems and methods for improved or high-quality linear blend skinning (LBS) are provided that can algorithmically "paint" weights for linear blend skinning and shift higher-order skeletal joints or bone locations as needed so that the resulting LBS provides high-quality animation, thereby reducing or eliminating the need for blendshapes.

[0215] The output of the joint breakdown system can include updates to the higher-order skeleton (which may have been used to initialize the system), including new linear blend skinning weights or joint or bone positions (either or both of which can be optimized).

[0216] For example, standard LBS is typically unable to capture fine details such as crow's feet (e.g., the fine wrinkles and clusters of fine lines that form around the outer corners of the eyes), while embodiments of the present system and method are able to do so. In standard animation techniques for obtaining crow's feet, a human animator moves the joints to what they subjectively feel is the best location, and then manually paints the vertex assignments for those higher-order skeletal joints to match the specific shape of the crow's feet, resulting in an iterative process that takes significant time to perform and is not suitable for real-time applications such as AR or games.

[0217] An embodiment of the joint decomposition system can convert target blend shapes into LBS vertex mappings, allowing the rigging system to quickly and efficiently simulate blend shapes (see, for example, examples in Figures 12 and 13).

[0218] A method is provided for algorithmically deriving efficient mesh weights and joint positions for use with a linear blend skinning interpolation engine. An embodiment of the present technique differs from standard techniques in which LBS parameters and blend shapes are input into a radial basis function (RBF) interpolation engine to determine the blend shapes to be applied to deform a mesh for a new avatar pose. Standard techniques can be difficult on an interpolation engine because there are many calculations involved when using blend shapes. In contrast to standard techniques, an embodiment herein inputs LBS parameters (e.g., weight maps, rotations, translations) into an RBF interpolation engine to determine how to deform an avatar's mesh without using blend shapes. Potential advantages of such an embodiment include that fewer calculations need to be performed by the interpolation engine, and therefore such a technique is not only faster but also computationally easier to implement, which enables real-time performance.

[0219] An embodiment of the joint decomposition system can take as input a skin cluster (e.g., a mapping of mesh vertices to joints), a high-order skeleton for initializing a constrained optimization algorithm, and a target shape to be matched during optimization (e.g., by reducing or minimizing error).

[0220] Advantages of the joint decomposition process include high-quality LBS skinning parameters as output that generalize well and are suitable for real-time animation applications such as AR or games. The LBS output may be considered to generalize well because the output may be reused in an RBF interpolation engine and reused for different avatars or characters. This is possible because RBF typically utilizes a set of known or fixed inputs, outputs, and training poses. The algorithm weighted paint described herein may use the same high-order skeleton as input to initialize the optimization and match the target poses, such that all LBS values ​​are calculated based on the same overall structure and format of the joints and vertices, thereby enabling the use of the LBS output in an RBF interpolation engine.

[0221] As described above, the joint resolution system and method can utilize human understandable constraints imposed on the optimization algorithm. For example, the upper arm can move approximately 180 degrees in a straight line up or down about the shoulder joint when raising or lowering the arm. This range of motion can be added as a constraint to the constrained optimization algorithm so that the joint resolution converges to a biometrically correct solution instead of converging in an anatomically impossible way.

[0222] Some conventional systems create a higher-order skeleton, either from scratch or from a lower-order skeleton, which can be computationally difficult. Thus, embodiments of the joint decomposition system use the higher-order skeleton as input to allow a constrained optimization algorithm to move or adjust or optimize the higher-order skeleton instead of starting from a blank slate. The use of the higher-order skeleton as input enables generalization since the higher-order skeleton can be used across different avatars, and thus there will be the same number of joints for each different avatar, etc. Since all humans have the same underlying higher-order skeleton, this is an acceptable standardization to make. (Exemplary Process for Joint Degradation)

[0223] Figure 14 illustrates an example process 1400 for joint disassembly. The example process 1400 may be performed by one or more computing systems, including the remote processing module 270, the local processing and data module 260, the avatar processing and rendering system 690 of Figure 6B, the remote computing system 920 of Figure 9A, other suitable computing devices, and combinations of these and other computing devices.

[0224] At block 1404, process 1400 accesses a target mesh for a target pose of the avatar. The target mesh may comprise any of the training datasets described herein (e.g., scanned ground truth, expert sculpts, simulations, or biometrically derived models). The target mesh may comprise one or more of a sequence of poses for the avatar. The target mesh may share topology with a base neutral mesh used for animation of the avatar, which is transformed into the target pose used to skin the avatar.

[0225] At block 1408, process 1400 accesses an initialization mesh that may be used to seed the constrained optimization process. The initialization mesh may comprise a base skeleton (e.g., in a neutral or resting pose) or a higher-order skeleton (e.g., in a pose that generally approximates the target pose). The initialization mesh may comprise a skin cluster, which assigns mesh vertices to the joints (or bones) of the skeleton. As described herein, it may be advantageous to initialize the constrained optimization process with a higher-order skeleton, which may lead to faster and more efficient convergence of the optimization. The initialization mesh may optionally include an initial weight map, rotation, or translation parameters that are close to the target pose, which may also improve the convergence of process 1400. The initial weight map may be created using any suitable method, including LBS or a user-defined method as described above.

[0226] In block 1412, the process accesses or updates the constraints to be applied to the constrained optimization process. Various constraints have been described in detail above (e.g., joint position or orientation locks, joint distance or range of motion constraints, segmentation sets, etc.). By applying suitable constraints, the optimization provides a practical threshold and reduces or avoids popping from overfitting. Also, as described above, the constraints can comprise anatomically or biometrically motivated human understandable constraints (e.g., constraints on shoulder or knee range of motion). Human understandable constraints can advantageously allow the constrained optimization process to converge to a biometrically correct solution instead of converging in an anatomically impossible way.

[0227] At block 1416, process 1400 performs a constrained optimization to determine skinning parameters for a target pose of the avatar according to the constraints accessed at block 1412. The constrained optimization can be initialized with the initialization mesh accessed at block 1408. As described herein with reference to equations (1) and (2), the output of the constrained optimization can be, for example, a weight map w for all joints of the skeleton. ij , rotation R j , and translation T j The result may be a refined or optimized set of skinning parameters, such as: 1) the set of skinning parameters, e.g., 2) the set of skinning parameters, 3) the set of skinning parameters, 4) the set of skinning parameters, 5) the set of skinning parameters, 6) the set of skinning parameters, 7) the set of skinning parameters, 8) the set of skinning parameters, 9) the set of skinning parameters, 10) the set of skinning parameters, 11) the set of skinning parameters, 12) the set of skinning parameters, 13) the set of skinning parameters, 14) the set of skinning parameters, 15) the set of skinning parameters, 16) the set of skinning parameters, 17) the set of skinning parameters, 18) the set of skinning parameters, 19) the set of skinning parameters, 20) the set of skinning parameters, 21) the set of skinning parameters, 22) the set of skinning parameters, 30) the set of skinning parameters, 40) the set of skinning parameters, 50) the set of skinning parameters, 6) the set of skinning parameters, 7) the set of skinning parameters, 8) the set of skinning parameters, 9) the set of skinning parameters, 10) the set of skinning parameters, 11) the set of skinning parameters, 12) the set of skinning parameters, 13) the set of skinning parameters, 14) the set of skinning parameters, 15) the set of skinning parameters, 16) the set of skinning parameters, 17) the set of skinning parameters, 18) the set of skinning parameters, 19) the set of skinning parameters, 21) the set of skinning parameters, 22) the set of skinning parameters, 30) the set of skinning parameters, 31) the set of skinning parameters, 32) the set of skinning parameters, 33) the set of skinning parameters, 34) the set

[0228] At block 1420, a convergence test is applied to the constrained optimization process. For example, convergence occurs when the error E falls below an error threshold or when the increment v to the transformed position iA convergence may be found if ' falls below a threshold (e.g., the mesh position changes little between iterations) or a maximum number of iterations has been reached. If convergence does not occur, process 1400 may return to block 1412 and the constraints may be updated. For example, the distance threshold or segmentation set may change as the joint decomposition progresses. In some embodiments, user input may be received at block 1412 to help improve convergence, and the use of human understandable constraints may be advantageous, since a human animator may determine the progress of convergence and apply, for example, biometrically appropriate constraints at the appropriate stage of the optimization process. Thus, the system may be highly configurable to better define the problem space in which the joint decomposition will operate.

[0229] If the constrained optimization has converged, the process 1400 moves to block 1424, where the results of the optimization are output. For example, the output can be stored in a non-transitory computer storage device, communicated (e.g., via a network 990 described with reference to FIG. 9A ) to an augmented reality display device 930 b, 930 c for animation of the virtual avatar, etc. For linear blend skinning, the output is a weight map w for all joints of the skeleton. ij , rotation R j , and translation T j In some embodiments, a radial basis function (RBF) formula for the skinning parameters may be calculated and output in block 1424. (Additional Aspects)

[0230] 1. A system comprising: a non-transitory computer storage configured to store a target mesh representing a pose of a virtual avatar, the target mesh comprising a plurality of target mesh vertices; an initialized skeleton comprising a plurality of joints; skinning parameters representing transformations of the skeleton to the pose of the virtual avatar; and a set of constraints associated with the skinning parameters or the joints; and a hardware processor in communication with the non-transitory computer storage, the hardware processor programmed to access the target mesh, the initialized skeleton, and the set of constraints, perform a constrained optimization process, and determine skinning parameters that provide a fit to the target mesh, the constrained optimization process according to the set of constraints, and output the skinning parameters.

[0231] 2. The system of aspect 1, wherein the target mesh is determined from a photographic scan of a human or animal subject in a certain pose.

[0232] 3. The system of any of the preceding aspects, wherein the target mesh is determined from an expert sculpt, a computational simulation of the pose, or a biometrically derived model.

[0233] 4. The system of any of the preceding aspects, wherein the initialization scaffold comprises a higher-order scaffold.

[0234] 5. The system of any of the preceding aspects, wherein the initialized skeleton comprises a low-order skeleton in a base or neutral pose.

[0235] 6. The system of any of the preceding aspects, wherein the initialized skeleton further comprises a skin cluster mapping of vertices to joints of the skeleton.

[0236] 7. The system of any of the preceding aspects, wherein the skinning parameters comprise a weighting map that measures the amount of influence of the joints on the vertices.

[0237] 8. The system of any of the preceding aspects, wherein the skinning parameters comprise six degree of freedom parameters associated with the movement of each joint.

[0238] 9. The system of aspect 8, wherein the six degree of freedom parameters include rotation and translation.

[0239] 10. The system of any of the preceding aspects, wherein the hardware processor is further programmed to calculate a radial basis function (RBF) formula associated with the skinning parameters.

[0240] 11. The system of any of the preceding aspects, wherein the constrained optimization process is configured to blend results from a first domain into a second domain and is programmed to implement a falloff.

[0241] 12. The system of any of the preceding aspects, wherein the constrained optimization process comprises linear or quadratic programming.

[0242] 13. The system of any of the preceding aspects, wherein the constrained optimization process comprises a nonlinear solver that utilizes first or second derivative estimates.

[0243] 14. The system of any of the preceding aspects, wherein the set of constraints comprises one or more of: a non-negativity constraint on weights in the weight map; a normalization constraint on the total influence of all joints on a vertex; a maximum number of non-zero weights for any vertex; a constraint that rotation parameters associated with the joint movements represent three-dimensional rotations; or a constraint that rotation parameters associated with the joint movements comprise an orthogonal matrix.

[0244] 15. The system of any of the preceding aspects, wherein the set of constraints comprises one or more of: a constraint that restricts child joints to be at a fixed distance from a parent joint; a constraint that restricts nearby vertices of the mesh to have similar weights in the weight map; a smoothness constraint that represents an anatomical property of a joint; a geometric constraint that represents a joint's length, lateral size, or range of motion (ROM); a geometric constraint that locks a subset of joints from being updated during the constrained optimization process; a constraint that joint positions may only be translated along certain axes; a constraint that locks joint properties during the constrained optimization process; a constraint that sets a weight for a vertex to zero if the joint is located beyond a threshold distance from the vertex; or a constraint that associates a set of one or more vertices with a set of one or more joints;

[0245] 16. The system of any of the preceding aspects, wherein the set of constraints comprises constraints incorporating a segmentation set that relate a set of one or more vertices to a set of one or more joints, and the constrained optimization process is programmed to act only on joints or vertices within the segmentation set.

[0246] 17. The system of any of the preceding aspects, wherein the hardware processor is programmed to dynamically update the set of constraints during the constrained optimization process.

[0247] 18. A method, comprising: under control of a hardware processor in communication with a non-transitory storage device, accessing a target mesh representing a target pose for a virtual avatar; accessing an initialization mesh for use in matching the target mesh; accessing a set of constraints; performing a constrained optimization to determine skinning parameters representing the target pose, the constrained optimization being performed according to the set of constraints, the constrained optimization being initialized with the initialization mesh; and outputting the skinning parameters.

[0248] 19. The method of aspect 18, wherein the target mesh is determined from a photographic scan of a human or animal subject assuming a target pose, an expertly made sculpture, a computational simulation of the target pose, or a biometrically derived model of the target pose.

[0249] 20. The method of aspect 18 or aspect 19, wherein the initialization mesh comprises a high-order skeleton.

[0250] 21. The method of any one of aspects 18-20, wherein the initialization mesh comprises a skin cluster mapping of vertices of the mesh to joints of the skeleton.

[0251] 22. The method of any one of aspects 18-21, wherein the skinning parameters comprise a weight map that measures the amount of influence of the joints on the vertices.

[0252] 23. The method of any one of aspects 18-22, wherein the skinning parameters comprise six degree of freedom parameters associated with the movement of each joint of the skeleton.

[0253] 24. The method of aspect 23, wherein the six degree of freedom parameters comprise rotation and translation.

[0254] 25. The method of any one of aspects 18-24, further comprising the step of calculating a radial basis function (RBF) equation associated with the skinning parameters.

[0255] 26. The method of any one of aspects 18-25, wherein performing the constrained optimization includes iterating an error term between the transformed mesh and the target mesh until convergence.

[0256] 27. The method of any one of aspects 18-26, further comprising dynamically updating the set of constraints while performing the constrained optimization.

[0257] 28. The method of aspect 27, wherein dynamically updating the set of constraints includes receiving user input regarding the set of constraints.

[0258] 29. The method of any one of aspects 18-28, wherein the set of constraints comprises one or more of a non-negativity constraint on weights in the weight map, a normalization constraint on the total influence of all joints on a vertex, a maximum number of non-zero weights for any vertex, a constraint that rotation parameters associated with the joint translations represent three-dimensional rotations, or a constraint that rotation parameters associated with the joint translations comprise orthogonal matrices.

[0259] 30. The method of any one of aspects 18-29, wherein the set of constraints comprises one or more of: a constraint restricting child joints to be at a fixed distance from a parent joint; a constraint restricting nearby vertices of the mesh to have similar weights in the weight map; a smoothness constraint representing an anatomical property of a joint; a geometric constraint representing the length, lateral size, or range of motion (ROM) of a joint; a geometric constraint locking a subset of joints from being updated while performing the constrained optimization; a constraint that joint positions may only be translated along certain axes; a constraint locking joint properties while performing the constrained optimization; a constraint setting a weight for a vertex to zero if the joint is located beyond a threshold distance from the vertex; a constraint associating a set of one or more vertices with a set of one or more joints; incorporating a segmentation set; and a constraint.

[0260] 31. The method of any one of aspects 18-30, wherein the set of constraints incorporates a segmentation set that associates one or more sets of vertices with one or more sets of joints, and performing the constrained optimization includes acting only on joints or vertices in the segmentation set.

[0261] 32. A method for automatically painting weights for linear blend skinning and determining bone locations, comprising the step of solving an error minimization equation based on an initial set of linear blend skinning weights, a high-order skeleton, and a target pose under the control of a hardware processor in communication with a non-transient storage device, wherein the equation obeys human-understandable constraints.

[0262] 33. The method of aspect 32, wherein the first human-understandable constraint comprises a limited posture space for limb rotation based on anatomical constraints.

[0263] 34. The method of aspect 33, wherein limb rotation is constrained to not move through the torso of the digital character, with the arms.

[0264] 35. The method of any one of aspects 32-34, wherein the second human-understandable constraint includes preventing a portion of the mesh from changing.

[0265] 36. A method comprising algorithmically deriving skin weights and joint positions using an algorithm under control of a hardware processor in communication with a non-transitory storage device, the inputs to the algorithm being an initial set of linear blend skinning weights, a high-order skeleton, and a target pose, the algorithm subject to a set of constraints based on one or more human understandable constraints.

[0266] 37. The method of aspect 36, wherein the first human-understandable constraint comprises a limited posture space for limb rotation based on anatomical constraints.

[0267] 38. The method of aspect 37, wherein limb rotation is constrained so as not to move through the torso of the digital character, with the arms.

[0268] 39. The method of any one of aspects 36-38, wherein the second human-understandable constraint includes preventing a portion of the mesh from changing.

[0269] 40. A system comprising a non-transitory storage device and a hardware processor programmed to implement the method of any one of aspects 18-39. Other considerations

[0270] Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying figures may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, and may be fully or partially automated thereby. For example, the computing systems may include general-purpose computers (e.g., servers) or special-purpose computers programmed with specific computer instructions, special-purpose circuits, etc. Code modules may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some implementations, certain operations and methods may be performed by circuitry specific to a given function.

[0271] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that application-specific hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved, or to provide results in substantially real-time. For example, an animation or video may contain many frames, each frame may have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time. As another example, calculating weight maps, rotation, and translation parameters for a skinning system by solving a constrained optimization problem for these parameters is highly computationally intensive (see, for example, example process 1400 described with reference to FIG. 14).

[0272] The code modules or any type of data may be stored on any type of non-transitory computer readable medium, such as physical computer storage devices, including hard drives, solid state memory, random access memory (RAM), read only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations of the same, and / or the like. The methods and modules (or data) may also be transmitted (e.g., as part of a carrier wave or other analog or digital propagated signal) generated on a variety of computer readable transmission media, including wireless-based and wired / cable-based media, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device, or communicated via a computer readable transmission medium.

[0273] Any process, block, state, step, or functionality in the flow diagrams described herein and / or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. Various processes, blocks, states, steps, or functionality can be combined, rearranged, added, deleted, modified, or otherwise altered from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith can be performed in other sequences as appropriate, e.g., serially, in parallel, or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems may generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

[0274] The process, method, and system may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0275] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in the present disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Thus, the claims are not intended to be limited to the implementations shown herein, but should be accorded the widest scope consistent with the disclosure, principles, and novel features disclosed herein.

[0276] Certain features described herein in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable subcombination. Furthermore, although features may be described above as acting in a combination and may even be initially claimed as such, one or more features from the claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination. No single feature or group of features is required or essential to every embodiment.

[0277] In particular, conditional statements used herein, such as "can," "could," "might," "may," "eg," and the like, are generally intended to convey that certain embodiments include certain features, elements, or steps, while other embodiments do not, unless specifically stated otherwise or understood otherwise within the context as used. Thus, such conditional statements are generally not intended to suggest that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are to be included or performed in any particular embodiment, with or without authorial input or prompting. The terms "comprising," "including," "having," and the like, are synonymous and used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not its exclusive sense), so that, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, the articles "a," "an," and "the," as used in this application and the appended claims, unless otherwise specified, should be interpreted to mean "one or more" or "at least one."

[0278] As used herein, a phrase referring to a list of items "at least one of" refers to any combination of those items, including single elements. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Transitive phrases such as "at least one of X, Y, and Z" are generally understood differently in the context in which they are used to convey that an item, term, etc. may be at least one of X, Y, or Z, unless specifically stated otherwise. Thus, such transitive phrases are generally not intended to suggest that an embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0279] Similarly, although operations may be depicted in the figures in a particular order, it should be appreciated that such operations need not be performed in the particular order depicted, or in sequential order, or that all of the depicted operations need not be performed to achieve desirable results. Additionally, the figures may diagrammatically depict one or more exemplary processes in the form of a flow chart. However, other operations not depicted may be incorporated within the diagrammatically depicted exemplary methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or during any of the depicted operations. Additionally, operations may be rearranged or reordered in other implementations. In some circumstances, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results.

Claims

1. A system for automatically painting weights and determining bone locations for linear blend skinning, comprising: The system includes a hardware processor in communication with a non-transitory storage device; The hardware processor includes: deriving skinning parameters that provide a match between a plurality of initialization mesh vertices of an initialization mesh corresponding to the initial pose and a plurality of target mesh vertices of a target mesh corresponding to the target pose by solving an error minimization equation based on an initialization skeleton at an initial pose and a target pose, the initialization skeleton comprising a low-order skeleton at a base or neutral pose and a high-order skeleton representing clothing of a virtual avatar as a series of joints in a hierarchical chain, the error minimization equation being subject to a set of constraints, and deriving the skinning parameters includes using the error minimization equation to determine that a difference between 1) positions of the plurality of initialization mesh vertices adjusted as a result of transforming the initialization skeleton to the target pose and 2) positions of a corresponding set of one or more mesh vertices satisfies an error threshold; outputting the skinning parameters; Controlling the higher-order skeleton based on the lower-order skeleton; displaying the virtual avatar in the initial pose; identifying a sound source within an environment of the virtual avatar; after the virtual avatar is displayed in the initial pose, displaying the virtual avatar in the target pose based on the skinning parameters; displaying the virtual avatar in the target pose and, after the sound source disappears, displaying the virtual avatar in the initial pose. A system that is programmed to:

2. The system described in claim 1, wherein the skinning parameters include one or more weights, one or more rotations, and one or more translations associated with transforming one or more joints of the initialized skeleton to different poses.

3. The system described in claim 1, wherein one or more joints of the initialized skeleton are initially transformed using an initial linear-blend skinning weight set, and then transformed by a refined linear-blend skinning weight set determined by solving the error minimization equation.

4. The system of claim 3, wherein the hardware processor is programmed to iteratively solve the error minimization equation.

5. The system of claim 1, wherein the set of constraints includes a first constraint that limits child joints to be a fixed distance from a parent joint, a second constraint that joints may only translate along specific axes, and a third constraint that sets weighting for a vertex to zero if the joint is located beyond a threshold distance from the vertex of the initialized skeleton.

6. The system described in claim 5, wherein the set of constraints includes a constraint that restricted postures regarding limb rotation are based on anatomical constraints.

7. The system of claim 6, wherein the limb rotation includes arms that are constrained from moving through the torso of the digital character.

8. The system of claim 5, wherein the set of constraints further comprises a constraint incorporating a segmentation set that associates a set of one or more initialization mesh vertices with a set of one or more joints of the initialization skeleton.

9. The system of claim 5, wherein the set of constraints further includes a constraint that locks a subset of joints from being updated when solving the error minimization equation.

10. The system of claim 1, wherein the set of constraints includes non-rigid constraints.

11. A method for automatically painting weights and determining bone locations for linear blend skinning, comprising: The method includes, under control of a hardware processor in communication with a non-transitory storage device, deriving skinning parameters that provide a match between a plurality of initialization mesh vertices of an initialization mesh corresponding to the initial pose and a plurality of target mesh vertices of a target mesh corresponding to the target pose by solving an error minimization equation based on an initialization skeleton at an initial pose and a target pose, the initialization skeleton comprising a low-order skeleton at a base or neutral pose and a high-order skeleton representing clothing of a virtual avatar as a series of joints in a hierarchical chain, the error minimization equation being subject to a set of constraints, and deriving the skinning parameters includes using the error minimization equation to determine that a difference between 1) positions of the plurality of initialization mesh vertices adjusted as a result of transforming the initialization skeleton to the target pose and 2) positions of a corresponding set of one or more mesh vertices satisfies an error threshold; outputting the skinning parameters; Controlling the higher-order skeleton based on the lower-order skeleton; displaying the virtual avatar in the initial pose; identifying a sound source within an environment of the virtual avatar; after the virtual avatar is displayed in the initial pose, displaying the virtual avatar in the target pose based on the skinning parameters; displaying the virtual avatar in the target pose and, after the sound source disappears, displaying the virtual avatar in the initial pose. A method comprising:

12. The method described in claim 11, wherein the skinning parameters include one or more weights, one or more rotations, and one or more translations associated with transforming one or more joints of the initialized skeleton to different poses.

13. The method described in claim 11, wherein one or more joints of the initialized skeleton are initially transformed using an initial linear-blend skinning weight set, and then transformed by a refined linear-blend skinning weight set determined by solving the error minimization equation.

14. The method of claim 13, wherein the method includes iteratively solving the error minimization equation.

15. The method described in claim 11, wherein the set of constraints includes a first constraint that limits child joints to be a fixed distance from a parent joint, a second constraint that joints may only translate along specific axes, and a third constraint that sets weightings for vertices of the initialized skeleton to zero if the joints are located beyond a threshold distance from the vertices.

16. The method described in claim 15, wherein the set of constraints further includes a constraint that the restricted postures regarding limb rotation are based on anatomical constraints.

17. The method of claim 16, wherein the limb rotation comprises arms that are constrained from moving through the torso of the digital character.

18. The method described in claim 15, wherein the set of constraints further comprises a constraint incorporating a segmentation set that associates a set of one or more initialization mesh vertices with a set of one or more joints of the initialization skeleton.

19. The method of claim 11, wherein the set of constraints includes non-rigid constraints.

Images