Projecting radiance fields to mesh surfaces

Abstract

Various implementations described herein relate to methods, systems, and non-transitory computer-readable media to transfer a texture from a source three-dimensional (3D) object to a target 3D object. In some implementations, a method includes obtaining a spatial grid for a set of Gaussians for the source 3D object, wherein the source 3D object has a texture, and wherein each Gaussian of the set of Gaussians occupies a respective subgrid of the spatial grid based on a covariance size of the Gaussian. The method further includes performing target vectorization on the target 3D object to generate a cached projection map for the target 3D object, and performing texture projection for each pixel in the cached projection map using the spatial grid to associate the pixel with a pixel color.

Description

PROJECTING RADIANCE FIELDS TO MESH SURFACESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 637,000, entitled “PROJECTING RADIANCE FIELDS TO MESH SURFACES WITH LOCAL SPACE TEXTURE MAPPING,” filed on April 22, 2024, and U.S. Provisional Application No. 63 / 637,541, entitled “PROJECTING RADIANCE FIELDS TO MESH SURFACES,” filed on April 23, 2024, the contents of both of which are incorporated herein in their entirety.TECHNICAL FIELD

[0002] This disclosure relates generally to computer graphics, and more particularly but not exclusively, relates to methods, systems, and computer readable media to project radiance fields to mesh surfaces with local space texture mapping.BACKGROUND

[0003] Texture transfer between differently shaped avatars is a common rendering problem. Radiance fields produce high fidelity images with high rendering speed to address this problem but are difficult to manipulate. Representing a radiance field may be performed using a variety of computing pipelines. For example, some techniques include per face texture projection and raycasting. These techniques are resource-intensive and time consuming.

[0004] Some implementations were conceived in light of the above.

[0005] The background description provided herein is for the purpose of presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the prior disclosure.SUMMARY

[0006] A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions.One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0007] According to one aspect, a computer-implemented method to transfer texture from a source three-dimensional (3D) object to a target 3D object is provided, the method comprising: obtaining a spatial grid for a set of Gaussians for the source 3D object, wherein the source 3D object has a texture, and wherein each Gaussian of the set of Gaussians occupies a respective subgrid of the spatial grid based on a covariance size of the Gaussian; performing target vectorization on the target 3D object to generate a cached projection map for the target 3D object; and performing texture projection for each pixel in the cached projection map using the spatial grid to associate the pixel with a pixel color.

[0008] Various implementations of the computer-implemented method are described herein.

[0009] In some implementations, the source 3D object is represented as a 3D mesh, and wherein further comprising, prior to obtaining the spatial grid, converting the 3D mesh into the set of Gaussians.

[0010] In some implementations, converting the 3D mesh into the set of Gaussians is performed via local space texture projection.

[0011] In some implementations, wherein a position and a normal of each Gaussian corresponds to a current pixel local space and the covariance size of each Gaussian is based upon a respective rate of change within positional iterations in a UV-space and a current normal of a source 3D mesh of the source 3D object.

[0012] In some implementations, the spatial grid is a cubic grid, and wherein individual axes of the spatial grid have a size equal to a cube-root of a total target pixel count of the target 3D object.

[0013] In some implementations, each subgrid corresponds to a respective pixel of the target 3D object.

[0014] In some implementations, performing texture projection for each pixel in the cached projection map comprises: finding a subgrid in the spatial grid that corresponds to the pixel;and identifying, based on the spatial grid, particular Gaussians from the set of Gaussians that are aligned with the pixel.

[0015] In some implementations, performing texture projection for each pixel in the cached projection map further comprises performing a ray cast from a position of the pixel to blend colors of the particular Gaussians by alpha transparency values.

[0016] In some implementations, performing the target vectorization comprises performing local space vectorization on a target mesh of the target 3D object to generate the cached projection map, wherein the cached projection map comprises a positional map and a normal map.

[0017] According to another aspect, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium with instructions stored thereon that, responsive to execution by a processing device, causes the processing device to perform operations comprising: obtaining a spatial grid for a set of Gaussians for a source three- dimensional (3D) object, wherein the source 3D object has a texture, and wherein each Gaussian of the set of Gaussians occupies a respective subgrid of the spatial grid based on a covariance size of the Gaussian; performing target vectorization on a target 3D object to generate a cached projection map for the target 3D object; and performing texture projection for each pixel in the cached projection map using the spatial grid to associate the pixel with a pixel color.

[0018] Various implementations of the non-transitory computer-readable medium are described herein.

[0019] In some implementations, the source 3D object is represented as a 3D mesh, and wherein further comprising, prior to obtaining the spatial grid, converting the 3D mesh into the set of Gaussians.

[0020] In some implementations, converting the 3D mesh into the set of Gaussians is performed via local space texture projection.

[0021] In some implementations, a position and a normal of each Gaussian corresponds to a current pixel local space and the covariance size of each Gaussian is based upon a respectiverate of change within positional iterations in a UV-space and a current normal of a source 3D mesh of the source 3D object.

[0022] In some implementations, the spatial grid is a cubic grid, and wherein individual axes of the spatial grid have a size equal to a cube-root of a total target pixel count of the target 3D object.

[0023] In some implementations, each subgrid corresponds to a respective pixel of the target 3D object.

[0024] According to another aspect, a system is disclosed, comprising: a memory with instructions stored thereon; and a processing device, coupled to the memory, the processing device configured to access the memory, wherein the instructions when executed by the processing device cause the processing device to perform operations comprising: obtaining a spatial grid for a set of Gaussians for a source three-dimensional (3D) object, wherein the source 3D object has a texture, and wherein each Gaussian of the set of Gaussians occupies a respective subgrid of the spatial grid based on a covariance size of the Gaussian; performing target vectorization on a target 3D object to generate a cached projection map for the target 3D object; and performing texture projection for each pixel in the cached projection map using the spatial grid to associate the pixel with a pixel color.

[0025] Various implementations of the system are described herein.

[0026] In some implementations, the source 3D object is represented as a 3D mesh, and wherein further comprising, prior to obtaining the spatial grid, converting the 3D mesh into the set of Gaussians.

[0027] In some implementations, converting the 3D mesh into the set of Gaussians is performed via local space texture projection.

[0028] hi some implementations, a position and a normal of each Gaussian corresponds to a current pixel local space and the covariance size of each Gaussian is based upon a respective rate of change within positional iterations in a UV-space and a current normal of a source 3D mesh of the source 3D object.

[0029] In some implementations, the spatial grid is a cubic grid, and wherein individual axes of the spatial grid have a size equal to a cube-root of a total target pixel count of the target 3D object.

[0030] According to yet another aspect, portions, features, and implementation details of the systems, methods, and non-transitory computer-readable media may be combined to form additional aspects, including some aspects which omit and / or modify some or portions of individual components or features, include additional components or features, and / or other modifications, and all such modifications are within the scope of this disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0031] FIG. 1 is an example of transferring makeup on a mesh onto another mesh, in accordance with some implementations.

[0032] FIG. 2 is an example of a cage and a corresponding original mesh, in accordance with some implementations.

[0033] FIG. 3 is an example of a cage with makeup texture and a corresponding cage UV structure, in accordance with some implementations.

[0034] FIG. 4 is a diagram illustrating source preconditioning, in accordance with some implementations.

[0035] FIG. 5 is a diagram illustrating target vectorization, in accordance with some implementations.

[0036] FIG. 6 is a diagram illustrating how source preconditioning and target vectorization provide information for texture projection, in accordance with some implementations.

[0037] FIG. 7 is a diagram illustrating texture projection, in accordance with some implementations.

[0038] FIG. 8 is a diagram illustrating ways texture projection results are used, in accordance with some implementations.

[0039] FIG. 9 is a diagram illustrating example results of a projection to a target texture, in accordance with some implementations.

[0040] FIG. 10 is a diagram illustrating example results of a UV mapping to a target mesh, in accordance with some implementations.

[0041] FIG. 11 is a flowchart of a method to project radiance fields to a mesh surface, in accordance with some implementations.

[0042] FIG. 12A is a flowchart of a method to perform source preconditioning based on a source mesh, in accordance with some implementations.

[0043] FIG. 12B is a flowchart of a method to perform source preconditioning based on a set of Gaussians, in accordance with some implementations.

[0044] FIG. 13 is a flowchart of a method to perform target vectorization, in accordance with some implementations.

[0045] FIG. 14 is flowchart of a method to map a target texture to a target mesh, in accordance with some implementations.

[0046] FIG. 15 is a diagram of an example system architecture that provides a way to project radiance fields to mesh surfaces, in accordance with some implementations.

[0047] FIG. 16 is a block diagram that illustrates an example computing device which may be used to implement one or more features described herein, in accordance with some implementations.DETAILED DESCRIPTION

[0048] In the following detailed description, reference is made to the accompanying drawings, which form apart hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative implementations described in the detailed description, drawings, and claims are not meant to be limiting. Other implementations may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. Aspects of the present disclosure, as generally described herein, and illustrated in the Figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

[0049] References in the specification to “one implementation,” “an implementation,” “an example implementation,” etc. indicate that the implementation described may include a particular feature, structure, or characteristic, but every implementation may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same implementation. Further, when a particular feature, structure, or characteristic is described in connection with an implementation, such feature, structure, or characteristic may be affected in connection with other implementations whether or not explicitly described.

[0050] The present disclosure is directed towards, inter alia, projecting radiance fields to target mesh surfaces by representing the source field with three-dimensional (3D)-Gaussian Splatting (3DGS) techniques. The techniques involve a texture projection pipeline (e.g., that can be used for three-dimensional (3D) avatars) that includes source preconditioning, target vectorization, and texture projection. The pipeline can be used to perform projections from a source to target(s) that have different shapes. The pipeline is computationally efficient and can be used to perform projections in real-time (or near real-time) applications.

[0051] Radiance fields generate high fidelity images with high rendering speed but are difficult to manipulate. Various implementations effectively perform avatar texture transfer across different appearances by combining benefits from using radiance fields and mesh surfaces. Implementations represent the source as a radiance field using Gaussians for a 3D Gaussian Splatter (3DGS), then project the Gaussians onto a target mesh.

[0052] In the context of 3D Gaussian Splatting, “Gaussians” refer to 3D Gaussian functions used to represent and render 3D scenes. These Gaussians are essentially ellipsoids with parameters like position, size, color, opacity, and so on that are adjusted through iterative optimization to match original images of a given scene. Each Gaussian may be a 3D bellshaped function, similar to a probability distribution, but is used to represent a point in 3D space. The Gaussians may be visualized as tiny, translucent ellipsoids in 3D space.

[0053] Each ellipsoid has several parameters. One parameter is a position, provided as 3D coordinates that specify where the ellipsoid is situated in 3D space. Another parameter is a size, provided as a covariance matrix that specifies the shape and orientation of the ellipsoid, such as how the ellipsoid is stretched, squashed, and / or deformed. Another parameter is a color, e.g.,represented as Red-Green-Blue (RGB) values or in any suitable format / color space. Another parameter is opacity, which may be a value specifying the transparency of the ellipsoid.

[0054] In iterative optimization, the parameters of the Gaussians are iteratively adjusted during a process to match the original images of the scene. This fine-tuning ensures that the rendered scene closely resembles the captured images.

[0055] For rendering, 3DGS uses a rasterization technique to project the Gaussians onto a 2D surface, considering the distance of the Gaussians from the viewer for depth. Each pixel's color is calculated based on the blended colors of the overlapping Gaussians.

[0056] 3DGS techniques use an adaptive density control, which means that Gaussians can be added, removed, or split to model the scene's complexity more accurately. For example, more Gaussians are used where there are more details (e.g., trees in a scene), and fewer where there are fewer details (e.g., the sky).

[0057] Texture transfer between differently shaped avatars (or other types of objects that may be present in a virtual environment) is a common rendering problem. Various implementations described herein represent a source radiance field as a 3D Gaussian Splatter (3DGS), incorporating local space projection to cache and speed up locality projection, before directly projecting the Gaussians on the target texture, blending them for color and lighting. The techniques efficiently complete realistic avatar texture transfer across different appearances quickly.

[0058] For example, some implementations of the present pipeline may complete a sample projection task in 1.12 seconds. By contrast, per face texture projection may require 31 seconds to perform a similar sample projection task. Raycasting may require 4.1 minutes to perform a similar sample projection task. By comparison to implementations described herein, per face texture projection and raycasting involve a significantly longer computation time (with similar computational resources).

[0059] The present pipeline can perform efficient texture transfer across different appearances. Such efficient texture transfer is particularly applicable for real-time and near real-time applications, or other applications where minimizing execution time is applicable. Such a projection task may provide for effective projection of surface features onto a 3D avatar, with high-quality results and manageable execution time.

[0060] For example, the surface features may include projection of makeup, tattoos, or other features onto a face of an avatar. These are non- limiting examples of surface features. Some implementations may project makeup or tattoos onto other portions of an avatar. Some implementations may project other surface features onto a face of an avatar or another portion of an avatar, such as another portion of the skin of an avatar.

[0061] Representing a radiance field with 3D-Gaussian Splatting (3DGS) achieves high fidelity images from captured media with high rendering speed. In various implementations, avatar source texture may be represented in 3DGS, before projecting the texture onto a target mesh by harmonizing the Gaussians for color and lighting. The 3DGS techniques as described herein are also usable to reconstruct avatars from videos and to perform scene reconstruction from multiple images.

[0062] The implementations discussed herein provide several technical advantages. For example, the operation speed is improved dramatically when projecting surface features onto an avatar (or other 3D object) with low memory usage. For example, by using a grid setup to cache pixel locality information at creation time rather than performing global searches per pixel for neighboring Gaussians at projection time converts a polynomial search to a constant lookup based on using the cached data to accelerate the retrieval.

[0063] Using techniques as described herein to perform target vectorization generates cached projection maps. The cached projection maps, in conjunction with the grids, accelerate a ray cast, providing for rapid projection of each pixel on the target texture. Then, the target texture consolidates the projected pixels and uses them to perform UV mapping to overlay the source surface features onto the target mesh. Such overlaying may occur in an accelerated manner with good graphical quality. Because the target texture has a corresponding size to that of the result texture, there is no potentially wasteful processing related to additional raycasting and UV mapping.

[0064] FIG. 1 is an example 100 of transferring makeup on a mesh onto another mesh, in accordance with some implementations. When rendering an avatar in a three-dimensional (3D) virtual environment, an avatar may have various forms of makeup on the face of the avatar.

[0065] For example, makeup may include base makeup (e.g., primer, foundation, concealer, powder, setting spray, etc.), eye makeup (e.g., eyeshadow, eyeliner, mascara,eyebrow products, false eyelashes, etc.), cheek makeup (e.g., blush, bronzer, highlighter, etc.), and / or lip makeup (e.g., lipstick, lip gloss, lip liner, lip stain / tint, etc.).

[0066] Other forms of makeup may also be relevant such as theatrical and / or screen costume makeup. Additionally, techniques used to transfer makeup may be used to transfer other exterior features of the skin of an avatar, whether on the avatar’s face or elsewhere. For example, makeup may be found on other portions of the avatar’s surface. Alternatively, tattoos or other skin art may also be transferred using the techniques presented in the present disclosure.

[0067] FIG. 1 illustrates various makeup 110 on a face of an example avatar. For example, makeup 110 includes eye makeup (e.g., eye shadow, eyeliner), cheek makeup (e.g., blush), lip makeup (e.g., lipstick), and a facial tattoo (e.g., a butterfly tattoo on a right cheek) of the example avatar.

[0068] FIG. 1 further illustrates other example avatars onto which the makeup 110 may be projected. Avatar 120 is a neck and head of a female human (without hair). Avatar 130 is a shoulder and head of a blocky character. Avatar 140 is a neck, shoulder, and head of a demonic monster. Some implementations provide ways to transfer makeup 110 onto corresponding parts of example avatars, such as avatar 120, avatar 130, and avatar 140.

[0069] FIG. 2 is an example 200 of a cage and a corresponding original mesh, in accordance with some implementations. For example, FIG. 2 illustrates a cage 210 and an original mesh 220. For example, each avatar head according to various implementations has a cage, such as cage 210. The cage’s UV maps are congruent to each other. Here, UV mapping is the 3D modeling process of projecting a 3D model's surface to a 2D image for texture mapping. The letters U and V denote the axes of the 2D texture because X, Y, and Z are already used to denote the axes of the 3D object in model space, while W (in addition to XYZ) is used in calculating quaternion rotations, a common operation in computer graphics. They are the same size and shape, so that they may overlap or be superimposed onto each other.

[0070] Then, the makeup may be mapped to the cage. The cage is then projected onto a target mesh. For example, the cage 210 is a low polygon count representation of the current avatar’s head. Because of the congruence, as discussed, the cage projection is able to be performed effectively and easily.

[0071] FIG. 3 is an example 300 of a cage with makeup texture and a corresponding cage UV structure, in accordance with some implementations. In the example of FIG. 3, a cage with a makeup texture 310 is shown. The cage’s UV structure 320 is also shown.

[0072] Cages manipulated by techniques may have the same UV structure, such that they can share the same makeup texture. The shared UV structure is helpful when performing the mapping of the cage to the mesh. The mapping includes source preconditioning, target vectorization, and texture projection, as discussed further herein. The shared UV structure may be helpful when swapping a makeup texture with another makeup texture.

[0073] FIG. 4 is a diagram 400 illustrating source preconditioning, in accordance with some implementations. FIG. 4 illustrates source preconditioning 410. For example, a textured mesh 412 may be obtained. Textured mesh 412 shows a finalized version of the mesh and textures that are used as the basis for the mapping.

[0074] Source preconditioning begins with a source 414 that is (if necessary) transformed into a source three-dimensional Gaussian Splatter (3DGS) 424, which may include a plurality of Gaussians (i.e., 3D ellipsoids) used to model source 414. For example, source 414 may be provided as a source mesh 420 or a source 3DGS 424. If source 414 includes data as a source mesh 420, the source preconditioning 410 takes option 416. If source 414 is a 3DGS 424 initially, source preconditioning takes option 418.

[0075] For example, there may be source mesh 420. Operation 422 takes source mesh 420 (after following option 416) and converts source mesh 420 into a corresponding source 3DGS 424 (e.g., by local space texture projection caching). Alternatively, the source 414 may already be a source 3DGS 424 and no conversion operation is necessary (after following option 418). Once ready (either after conversion or as provided initially), the source 3DGS 424 is inserted into a spatial grid 428 at operation 426.

[0076] For source preconditioning 410, with respect to a source mesh 420, it is important for the techniques described herein to work in a 3DGS space. Therefore, it is possible to convert the source mesh 420 into a corresponding source 3DGS 424 at operation 422. The generated source 3DGS 424 may be inserted into a spatial grid 428 with 0(1) lookup access (i.e., constant time lookup). Such lookup access is fast and efficient, in that a lookup takes a specific amount of time regardless of the size of the spatial grid 428.

[0077] This representation permits the reconstruction of complex environments (e.g., avatars or 3D objects) in extremely high detail. For example, a source mesh 420 may be generated using a 3D computer graphics application that permits creation of assets for interactive 3D applications.

[0078] If a source mesh 420 is provided, the source mesh 420 may be represented as a source 3DGS 424 via local space texture projection. Such projection may be used in a conversion operation 422. For example, the 3DGS may be the local space texture’s individual pixel (where each pixel on the local space texture can be considered as a small 3D ellipsoid with a Gaussian distribution of density, meaning that it is brighter in the center and fades out to the edges).

[0079] The Gaussians in the source 3DGS 424 may be defined to have a covariance of each depending on its rate of change within the positional iteration in the UV-space and the current normal of the corresponding source mesh 420. To convert the source mesh 420 into a corresponding source 3DGS 424, some implementations generate a Gaussian per pixel based on a texture of the source mesh 420.

[0080] The position and normal of the generated Gaussian may correspond to the local space of the current pixel. The covariance spread may be determined by using a rate of change of the positional iteration from the next corresponding pixel when computed in the local space. The color corresponds to the current pixel color. Some implementations may then densify the source 3DGS 424 to have a uniform surface density and to increase the overlap between Gaussians in the source 3DGS 424.

[0081] Various implementations may construct the spatial grid 428 to contain about one target pixel per subgrid. Accordingly, the grid size is roughly a cube-root of a total target pixel count. The Gaussians are added into subgrids of the spatial grid 428 that collide with their covariance volume.

[0082] This grid setup converts a search task that would otherwise be a polynomial search task to a constant search task at projection time by caching pixel locality information during creation. Such caching may eliminate performance of a global search for neighboring Gaussians for each pixel during projection.

[0083] For example, the source 3DGS 424 represents a corresponding source mesh 420 or a larger scene as a large group of Gaussians. Each Gaussian has a mean in three-dimensional space and a 3x3 positive-definite covariance matrix defining the Gaussian. The covariance matrix of each Gaussian may be parameterized by using a scaling vector with three components and a quaternion with four components encoding the rotation of the Gaussian.

[0084] In addition, each Gaussian may have a view-dependent radiance represented by an opacity in the range [0,1] and a set of spherical harmonics coordinates defining the colors emitted in corresponding directions. To render an image from a given view, a rasterizer splats the three-dimensional (3D) Gaussians into two-dimensional (2D) Gaussians parallel to the image plane. The rasterizer then blends the splats depending on the opacity and depths of splats. Such rendering is very fast, providing an advantage of 3DGS over volumetric rendering, e.g., neural radiance fields (NeRFs).

[0085] Gaussian splatting may be considered to be an approximation of traditional volumetric rendering of radiance fields using a density function. In Gaussian splatting, the density function is computed as a sum of Gaussian values weighted by alpha-blending coefficients at any 3D point in three-dimensional space.

[0086] FIG. 5 is a diagram 500 illustrating target vectorization, in accordance with some implementations. FIG. 5 illustrates target vectorization 510. For example, FIG. 5 illustrates an example of a target mesh 520. The target mesh 520 is subjected to operation 530, which includes local space vectorization. This operation 530 of local space vectorization generates cached projection maps 540. For example, the cached projection maps 540 may include a positional projection map 550 and a normal projection map 560. Such maps are used later in the pipeline.

[0087] A positional projection map 550 refers to a map projection that aims to accurately represent the relative positions of features on a surface. A positional projection map 550 is a type of map projection designed to preserve spatial relationships and distances between points, although no map projection can completely avoid distortion.

[0088] A normal projection map 560 refers to a map projection's orientation, specifically when the central axis of the developable surface (like a cylinder or cone) aligns with an axis of rotation. This means that north and south are straight up and down on the map. In essence, anormal projection map 560 is a common and intuitive way to display a map, aligning the normal projection map 560 with the natural orientation of the mesh.

[0089] FIG. 6 is a diagram 600 illustrating how source preconditioning and target vectorization provide information for texture projection, in accordance with some implementations. FIG. 6 illustrates source preconditioning performed at block 610 and target vectorization performed at block 620. The source preconditioning performed at block 610 and the target vectorization performed at block 620 are performed for corresponding pixels in the cached projection map at block 630. These results are provided to be used in texture projection at block 640.

[0090] Additional aspects of texture projection performed at block 640 are provided in the discussion of FIG. 7. For example, the source preconditioning performed at block 610 may proceed as illustrated in FIG. 4, such as by taking Gaussians in a source 3DGS (received as an input or converted from a source mesh) and inserting the Gaussians into a spatial grid. The target vectorization performed at block 620 may proceed as illustrated in FIG. 5, such as by taking a target mesh and using local space vectorization to generate cached projection maps, such as a positional projection map and a normal projection map.

[0091] FIG. 6 illustrates that the source preconditioning performed at block 610 and the target vectorization performed at block 620 are performed at block 630 for each of the corresponding pixels in the cached projection map. After block 630, the information generated in the source preconditioning performed at block 610 and in the target vectorization performed at block 620 is provided for texture projection performed at block 640. Additional details of texture projection are presented in FIG. 7.

[0092] FIG. 7 is a diagram 700 illustrating texture projection 710, in accordance with some implementations. Texture projection 710 may begin with operation 720. In operation 720, a grid cell in which a target pixel is located is selected. For example, FIG. 7 illustrates grid 722. Grid 722 includes selected cell 724, and target pixel 726 is located in selected cell 724. Operation 720 may be followed by operation 730. At operation 730, Gaussians 732 are selected as being aligned by a normal ray 734. Operation 730 may be followed by operation 740. At operation 740, intersecting Gaussians 742 are summed for a pixel color for the target pixel in a ray tracing operation based on ray 744.

[0093] For example, to obtain a projection result, for the pixels of a target mesh texture, there may be a one-to-one mapping with the two generated local space textures. For each pixel, there is a corresponding target position and normal. This information (the target position and the normal) may be used to lookup information from the spatial grid to gather the 3DGS around the vicinity of the pixel. In this process, Gaussians in the 3DGS that are aligned by the normal are selected. The radiance transfer may then be computed by summing the Gaussians 3DGS that are used to blend into the target texture.

[0094] FIG. 8 is a diagram 800 illustrating ways texture projection results are used, in accordance with some implementations. FIG. 8 illustrates that, at block 810, texture projection is performed. For example, additional details of texture projection are discussed in FIG. 7. Block 810 may be followed by block 820.

[0095] At block 820, results of projection to a target texture are assembled for the UV mapping. Examples of the results of the projection to the target texture that are assembled in block 820 are illustrated in FIG. 9. Block 820 may be followed by block 830.

[0096] At block 830, the results of the projection to the target texture are followed by UV mapping to a target mesh at block 830. Examples of the UV mapping to a target mesh as provided in block 830 are illustrated in FIG. 10.

[0097] FIG. 9 is a diagram 900 illustrating example results of a projection to a target texture, in accordance with some implementations. For example, FIG. 9 illustrates a face 910, a face 920, and a face 930. Each face has a different shape. FIG. 9 illustrates that face 910, face 920, and face 930 each have similar makeup. The makeup on face 910, face 920, and 930 are overlaid using techniques provided herein. For example, the eye and eyebrow makeup may be applied differently based on eye shape. The overlaying of makeup (or other textures) may be different for different textures, even when the source is the same.

[0098] FIG. 10 is a diagram 1000 illustrating example results of a UV mapping to a target mesh, in accordance with some implementations. This includes a target mesh 1010, a target mesh 1020, and a target mesh 1030. The faces are mapped from target textures as provided in FIG. 9 to target meshes in FIG. 10 using UV mapping. As noted above, a UV Map is a type of vertex map that stores vertical and horizontal positions on a 2D texture. The letters U (Horizontal) and V (Vertical) denote the axes of the 2D texture because X, Y and Z are used todenote the axes of the 3D space. FIG. 10 shows that similar textures (as shown in FIG. 9) may look quite different when applied to generate actual UV mappings to target meshes.

[0099] For example, target mesh 1010 has a target texture projection mapped from face910 using UV mapping. Target mesh 1020 has a target texture projection mapped from face920 using UV mapping. Target mesh 1030 has a target texture projection mapped from face930 using UV mapping. It may be seen in FIG. 10 that while target mesh 1010, target mesh1020, and target mesh 1030 vary considerably in shape, the techniques presented herein provide an effective way to transfer makeup on these target meshes with results that preserve the texture when projected on arbitrary target meshes.

[0100] FIG. 11 is a flowchart of a method 1100 to project radiance fields to a mesh surface, in accordance with some implementations. The method 1100 may be a method to transfer texture from a source 3D object to a target 3D object. Method 1100 may begin at block 1102.

[0101] At block 1102, a spatial grid is obtained. In some implementations, the spatial grid is obtained for a set of Gaussians for the source 3D object. The source 3D object may have a texture. Each Gaussian of the set of Gaussians occupies a respective subgrid of the spatial grid based on a covariance size of the Gaussian.

[0102] In some implementations, a position and a normal of each Gaussian corresponds to a current pixel local space and the covariance size of each Gaussian is based upon a respective rate of change within positional iterations in a UV-space and a current normal of a source mesh of the 3D object.

[0103] In some implementations, the spatial grid may be a cubic grid. Individual axes of the spatial grid have a size equal to or approximately equal to (e.g., within a threshold difference of) a cube-root of a total target pixel count of the target 3D object. In some implementations, each subgrid may correspond to a respective pixel of the target 3D object. Block 1102 may be followed by block 1104.

[0104] At block 1104, target vectorization is performed. The target vectorization may be performed on the target 3D object to generated a cached projection map for the target 3D object. In some implementations, performing the target vectorization comprises performing local space vectorization on a target mesh of the target 3D object to generate the cached projection map, wherein the cached projection map comprises a positional map and a normal map.Additional aspects of target vectorization are discussed at FIG. 5. Block 1104 may be followed by block 1106.

[0105] At block 1106, texture projection is performed. The texture projection may be performed for each pixel in the cached projection map using the spatial grid to associate the pixel with a pixel color. For example, texture projection for each pixel in the cached projection map includes finding a subgrid in the spatial grid that corresponds to the pixel.

[0106] Texture projection also includes identifying, based on the spatial grid, particular Gaussians from the set of Gaussians that are aligned with the pixel. In some implementations, performing texture projection for each pixel in the cached projection map further comprises performing a ray cast from a position of the pixel to blend colors of the particular Gaussians by alpha transparency values.

[0107] Method 1100 may include other operations in addition to or instead of those discussed above. The operations may be performed in the order presented above, or in alternative orders. The operations may be performed in serial and / or in parallel in various implementations.

[0108] FIG. 12A is a flowchart of a method 1200a to perform source preconditioning based on a source mesh, in accordance with some implementations. Method 1200a may begin at block 1202. Method 1200a may begin at block 1202 by receiving a source 3D mesh. Additional aspects of source preconditioning are discussed at FIG. 4.

[0109] For example, at block 1202, a source 3D mesh is received. Block 1202 may be followed by block 1204. At block 1204, the source 3D mesh is converted into a set of Gaussians (which may be used for a 3DGS). In some implementations, the source 3D object is represented as a 3D mesh. Prior to obtaining the spatial grid, the 3D mesh is converted into the set of Gaussians. In some implementations, converting the 3D mesh into the set of Gaussians is performed via local space texture projection. Block 1204 may be followed by block 1206.

[0110] At block 1206, the Gaussians are inserted into the spatial grid. Each Gaussian of the set of Gaussians occupies a respective subgrid of the spatial grid based on a covariance size of the Gaussian. In some implementations, a position and a normal of each Gaussian corresponds to a current pixel local space and the covariance size of each Gaussian is based upon arespective rate of change within positional iterations in a UV-space and a current normal of a source mesh of the 3D object.

[0111] In some implementations, the spatial grid may be a cubic grid. Individual axes of the spatial grid have a size equal to or approximately equal to (e.g., within a threshold difference of) a cube-root of a total target pixel count of the target 3D object. In some implementations, each subgrid may correspond to a respective pixel of the target 3D object.

[0112] Method 1200a may include other operations in addition to or instead of those discussed above. The operations may be performed in the order presented above, or in alternative orders. The operations may be performed in serial and / or in parallel in various implementations.

[0113] FIG. 12B is a flowchart of a method 1200b to perform source preconditioning based on a set of Gaussians, in accordance with some implementations. Method 1200b may begin at block 1202. Method 1200b may begin at block 1208 by receiving a set of Gaussians. Additional aspects of source preconditioning are discussed at FIG. 4. At block 1208, because the input is already in the form of a set of Gaussians, no conversion is necessary. Block 1208 may be followed by block 1210.

[0114] At block 1210, the Gaussians are inserted into the spatial grid. Each Gaussian of the set of Gaussians occupies a respective subgrid of the spatial grid based on a covariance size of the Gaussian. In some implementations, a position and a normal of each Gaussian corresponds to a current pixel local space and the covariance size of each Gaussian is based upon a respective rate of change within positional iterations in a UV-space and a current normal of a source mesh of the 3D object.

[0115] In some implementations, the spatial grid may be a cubic grid. Individual axes of the spatial grid have a size equal to or approximately equal to (e.g., within a threshold difference of) a cube-root of a total target pixel count of the target 3D object. In some implementations, each subgrid may correspond to a respective pixel of the target 3D object.

[0116] Method 1200b may include other operations in addition to or instead of those discussed above. The operations may be performed in the order presented above, or in alternative orders. The operations may be performed in serial and / or in parallel in various implementations.

[0117] FIG. 13 is a flowchart of a method 1300 to perform target vectorization, in accordance with some implementations. Method 1300 may begin at block 1302. Additional aspects of target vectorization are discussed at FIG. 5.

[0118] At block 1302, a texture mesh is received. The texture mesh may represent an object onto which the radiance field is to be projected. Block 1302 may be followed by block 1304.

[0119] At block 1304, local space vectorization is performed. Local space vectorization may include rasterizing triangles in the target mesh to their UV-space and converting the mesh into cached projection maps. The cached projection maps may include a positional map and a normal map.

[0120] Local space vectorization means representing data (like position, normal, or color) relative to the object's own coordinate system, rather than a global or world coordinate system. This technique is useful for calculations that are calculated relative to an object's orientation and position.

[0121] Benefits of local space vectorization may include simplifying calculations for objects that move or rotate and improving performance in certain rendering scenarios. A target mesh is the mesh that is being used as a basis for comparison, transformation, or optimization. A target mesh may be used when mapping texture. Block 1304 may be followed by block 1306.

[0122] At block 1306, a positional projection map is generated. As discussed above, a positional projection map refers to a map projection that aims to accurately represent the relative positions of features on a surface. A positional projection map is a type of map projection designed to preserve spatial relationships and distances between points, although no map projection can completely avoid distortion. Block 1306 may be followed by block 1308.

[0123] At block 1308, a normal projection map is generated. As discussed above, a normal projection map refers to a map projection's orientation, specifically when the central axis of the developable surface (like a cylinder or cone) aligns with an axis of rotation. This means that north and south are straight up and down on the map. In essence, a normal projection map is a common and intuitive way to display a map, aligning the map with the natural orientation of the mesh. Block 1308 may be followed by block 1310.

[0124] At block 1310, a target texture is mapped to the target mesh. There may be a one- to-one match between the cached projection maps and the target texture. Additional details of the mapping are presented in FIG. 14.

[0125] Method 1300 may include other operations in addition to or instead of those discussed above. The operations may be performed in the order presented above, or in alternative orders. The operations may be performed in serial and / or in parallel in various implementations.

[0126] FIG. 14 is flowchart of a method 1400 to map a target texture to a target mesh, in accordance with some implementations. Method 1400 may begin at block 1402.

[0127] At block 1402, results are sent to a target texture. For each pixel in the cached projection map, the subgrid is indexed based on the pixel position. Gaussians are then filtered for Gaussians in the indexed subgrid that are aligned with the pixel with a dot product between each Gaussian and the normal. Block 1402 may be followed by block 1404.

[0128] At block 1404, a target texture is mapped to a target mesh. For example, some implementations perform a ray cast from the target position and normal. The colors of the Gaussians intersecting the ray are blended by their alpha values, and are weighted by their angle to the normal, forming the color of the target texture pixel.

[0129] Method 1400 may include other operations in addition to or instead of those discussed above. The operations may be performed in the order presented above, or in alternative orders. The operations may be performed in serial and / or in parallel in various implementations.

[0130] FIG. 15 is a diagram of an example system architecture that provides a way to project radiance fields to mesh surfaces, in accordance with some implementations. FIG. 15 and the other figures use like reference numerals to identify similar elements. A letter after a reference numeral, such as “1510," indicates that the text refers specifically to the element having that particular reference numeral. A reference numeral in the text without a following letter, such as "1510," refers to any or all of the elements in the figures bearing that reference numeral (e.g., "1510" in the text refers to reference numerals “1510a," “1510b," and / or “1510n” in the figures).

[0131] The system architecture 1500 (also referred to as “system” herein) includes online virtual experience server 1502, data store 1520, client devices 1510a, 1510b, and 1510n (generally referred to as “client device(s) 1510” herein), and developer devices 1530a and 1530n (generally referred to as “developer device(s) 1530” herein). Virtual experience server 1502, data store 1520, client devices 1510, and developer devices 1530 are coupled via network 1522. In some implementations, client devices(s) 1510 and developer device(s) 1530 may refer to the same or same type of device.

[0132] Online virtual experience server 1502 can include, among other things, a virtual experience engine 1504, one or more virtual experiences 1506, and graphics engine 1508. In some implementations, the graphics engine 1508 may be a system, application, or module that permits the online virtual experience server 1502 to provide graphics and animation capability. In some implementations, the graphics engine 1508 and / or virtual experience engine 1504 may perform one or more of the operations described below in connection with the flowcharts shown in FIGS. 11-14. A client device 1510 can include a virtual experience application 1512, and input / output (I / O) interfaces 1514 (e.g., input / output devices). The input / output devices can include one or more of a microphone, speakers, headphones, display device, mouse, keyboard, game controller, touchscreen, virtual reality consoles, etc.

[0133] A developer device 1530 can include a virtual experience application 1532, and input / output (I / O) interfaces 1534 (e.g., input / output devices). The input / output devices can include one or more of a microphone, speakers, headphones, display device, mouse, keyboard, game controller, touchscreen, virtual reality consoles, etc.

[0134] System architecture 1500 is provided for illustration. In different implementations, the system architecture 1500 may include the same, fewer, more, or different elements configured in the same or different manner as that shown in FIG. 15.

[0135] In some implementations, network 1522 may include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), a wired network (e.g., Ethernet network), a wireless network (e.g., an 802.11 network, a Wi-Fi® network, or wireless LAN (WLAN)), a cellular network (e.g., a 5G network, a Long Term Evolution (LTE) network, etc.), routers, hubs, switches, server computers, or a combination thereof.

[0136] In some implementations, the data store 1520 may be a non-transitory computer readable memory (e.g., random access memory), a cache, a drive (e.g., a hard drive), a flash drive, a database system, or another type of component or device capable of storing data. The data store 1520 may also include multiple storage components (e.g., multiple drives or multiple databases) that may also span multiple computing devices (e.g., multiple server computers). In some implementations, data store 1520 may include cloud-based storage.

[0137] In some implementations, the online virtual experience server 1502 can include a server having one or more computing devices (e.g., a cloud computing system, a rackmount server, a server computer, cluster of physical servers, etc.). In some implementations, the online virtual experience server 1502 may be an independent system, may include multiple servers, or be part of another system or server.

[0138] In some implementations, the online virtual experience server 1502 may include one or more computing devices (such as a rackmount server, a router computer, a server computer, a personal computer, a mainframe computer, a laptop computer, a tablet computer, a desktop computer, etc.), data stores (e.g., hard disks, memories, databases), networks, software components, and / or hardware components that may be used to perform operations on the online virtual experience server 1502 and to provide a user with access to online virtual experience server 1502. The online virtual experience server 1502 may also include a website (e.g., a web page) or application back-end software that may be used to provide a user with access to content provided by online virtual experience server 1502. For example, users may access online virtual experience server 1502 using the virtual experience application 1512 on client devices 1510.

[0139] In some implementations, virtual experience session data are generated via online virtual experience server 1502, virtual experience application 1512, and / or virtual experience application 1532, and are stored in data store 1520. With permission from virtual experience participants, virtual experience session data may include associated metadata, e.g., virtual experience identifier(s); device data associated with the participant(s); demographic information of the participant(s); virtual experience session identifier(s); chat transcripts; session start time, session end time, and session duration for each participant; relative locations of participant avatar(s) within a virtual experience environment; purchase(s) within the virtual experience by one or more participants(s); accessories utilized by participants; etc.

[0140] In some implementations, online virtual experience server 1502 may be a type of social network providing connections between users or a type of user-generated content system that allows users (e.g., end-users or consumers) to communicate with other users on the online virtual experience server 1502, where the communication may include voice chat (e.g., synchronous and / or asynchronous voice communication), video chat (e.g., synchronous and / or asynchronous video communication), or text chat (e.g., 1:1 and / or N:N synchronous and / or asynchronous text-based communication). A record of some or all user communications may be stored in data store 1520 or within virtual experiences 1506. The data store 1520 may be utilized to store chat transcripts (text, audio, images, etc.) exchanged between participants, with appropriate permissions from the players and in compliance with applicable regulations.

[0141] In some implementations, the chat transcripts are generated via virtual experience application 1512 and / or virtual experience application 1532 or and are stored in data store 1520. The chat transcripts may include the chat content and associated metadata, e.g., text content of chat with each message having a corresponding sender and recipient(s); message formatting (e.g., bold, italics, loud, etc.); message timestamps; relative locations of participant avatar(s) within a virtual experience environment, accessories utilized by virtual experience participants, etc. In some implementations, the chat transcripts may include multilingual content, and messages in different languages from different sessions of a virtual experience may be stored in data store 1520.

[0142] In some implementations, chat transcripts may be stored in the form of conversations between participants based on the timestamps. In some implementations, the chat transcripts may be stored based on the originator of the message(s).

[0143] In some implementations of the disclosure, a “user” may be represented as a single individual. Other implementations of the disclosure encompass a “user” (e.g., creating user) being an entity controlled by a set of users or an automated source. For example, a set of individual users federated as a community or group in a user-generated content system may be considered a “user.”

[0144] In some implementations, online virtual experience server 1502 may be a virtual gaming server. For example, the gaming server may provide single-player or multiplayer games to a community of users that may access as “system” herein) includes online virtual experience server 1502, data store 1520, client or interact with virtual experiences using clientdevices 1510 via network 1522. In some implementations, virtual experiences (including virtual realms or worlds, virtual games, other computer- simulated environments) may be two- dimensional (2D) virtual experiences, three-dimensional (3D) virtual experiences (e.g., 3D user-generated virtual experiences), virtual reality (VR) experiences, or augmented reality (AR) experiences, for example. In some implementations, users may participate in interactions (such as gameplay) with other users. In some implementations, a virtual experience may be experienced in real-time with other users of the virtual experience.

[0145] In some implementations, virtual experience engagement may refer to the interaction of one or more participants using client devices (e.g., 1510) within a virtual experience (e.g., 1506) or the presentation of the interaction on a display or other output device (e.g., 1514) of a client device 1510. For example, virtual experience engagement may include interactions with one or more participants within a virtual experience or the presentation of the interactions on a display of a client device.

[0146] In some implementations, a virtual experience 1506 can include an electronic file that can be executed or loaded using software, firmware or hardware configured to present the virtual experience content (e.g., digital media item) to an entity. In some implementations, a virtual experience application 1512 may be executed and a virtual experience 1506 rendered in connection with a virtual experience engine 1504. In some implementations, a virtual experience 1506 may have a common set of rules or common goal, and the environment of a virtual experience 1506 shares the common set of rules or common goal. In some implementations, different virtual experiences may have different rules or goals from one another.

[0147] In some implementations, virtual experiences may have one or more environments (also referred to as “virtual experience environments” or “virtual environments” herein) where multiple environments may be linked. An example of an environment may be a three- dimensional (3D) environment. The one or more environments of a virtual experience 1506 may be collectively referred to as a “world” or “virtual experience world” or “gaming world” or “virtual world” or “universe” herein. An example of a world may be a 3D world of a virtual experience 1506. For example, a user may build a virtual environment that is linked to another virtual environment created by another user. A character of the virtual experience may cross the virtual border to enter the adjacent virtual environment.

[0148] It may be noted that 3D environments or 3D worlds use graphics that use a three- dimensional representation of geometric data representative of virtual experience content (or at least present virtual experience content to appear as 3D content whether or not 3D representation of geometric data is used). 2D environments or 2D worlds use graphics that use two-dimensional representation of geometric data representative of virtual experience content.

[0149] In some implementations, the online virtual experience server 1502 can host one or more virtual experiences 1506 and can permit users to interact with the virtual experiences 1506 using a virtual experience application 1512 of client devices 1510. Users of the online virtual experience server 1502 may play, create, interact with, or build virtual experiences 1506, communicate with other users, and / or create and build objects (e.g., also referred to as “item(s)” or “virtual experience objects” or “virtual experience item(s)” herein) of virtual experiences 1506.

[0150] For example, in generating user-generated virtual items, users may create characters, decoration for the characters, one or more virtual environments for an interactive virtual experience, or build structures used in a virtual experience 1506, among others. In some implementations, users may buy, sell, or trade virtual experience objects, such as in-platform currency (e.g., virtual currency), with other users of the online virtual experience server 1502. In some implementations, online virtual experience server 1502 may transmit virtual experience content to virtual experience applications (e.g., 1512). In some implementations, virtual experience content (also referred to as “content” herein) may refer to any data or software instructions (e.g., virtual experience objects, virtual experience, user information, video, images, commands, media item, etc.) associated with online virtual experience server 1502 or virtual experience applications. In some implementations, virtual experience objects (e.g., also referred to as “item(s)” or “objects” or “virtual objects” or “virtual experience item(s)” herein) may refer to objects that are used, created, shared or otherwise depicted in virtual experience 1506 of the online virtual experience server 1502 or virtual experience applications 1512 of the client devices 1510. For example, virtual experience objects may include a part, model, character, accessories, tools, weapons, clothing, buildings, vehicles, currency, flora, fauna, components of the aforementioned (e.g., windows of a building), and so forth.

[0151] It may be noted that the online virtual experience server 1502 hosting virtual experiences 1506, is provided for purposes of illustration. In some implementations, onlinevirtual experience server 1502 may host one or more media items that can include communication messages from one user to one or more other users. With user permission and express user consent, the online virtual experience server 1502 may analyze chat transcripts data to improve the virtual experience platform. Media items can include, but are not limited to, digital video, digital movies, digital photos, digital music, audio content, melodies, website content, social media updates, electronic books, electronic magazines, digital newspapers, digital audio books, electronic journals, web blogs, real simple syndication (RSS) feeds, electronic comic books, software applications, etc. In some implementations, a media item may be an electronic file that can be executed or loaded using software, firmware or hardware configured to present the digital media item to an entity.

[0152] In some implementations, a virtual experience 1506 may be associated with a particular user or a particular group of users (e.g., a private virtual experience), or made widely available to users with access to the online virtual experience server 1502 (e.g., a public virtual experience). In some implementations, where online virtual experience server 1502 associates one or more virtual experiences 1506 with a specific user or group of users, online virtual experience server 1502 may associate the specific user(s) with a virtual experience 1506 using user account information (e.g., a user account identifier such as username and password).

[0153] In some implementations, online virtual experience server 1502 or client devices 1510 may include a virtual experience engine 1504 or virtual experience application 1512. In some implementations, virtual experience engine 1504 may be used for the development or execution of virtual experiences 1506. For example, virtual experience engine 1504 may include a rendering engine (“renderer”) for 2D, 3D, VR, or AR graphics, a physics engine, a collision detection engine (and collision response), sound engine, scripting functionality, animation engine, artificial intelligence engine, networking functionality, streaming functionality, memory management functionality, threading functionality, scene graph functionality, or video support for cinematics, among other features. The components of the virtual experience engine 1504 may generate commands that help compute and render the virtual experience (e.g., rendering commands, collision commands, physics commands, etc.) In some implementations, virtual experience applications 1512 of client devices 1510, respectively, may work independently, in collaboration with virtual experience engine 1504 of online virtual experience server 1502, or a combination of both.

[0154] In some implementations, both the online virtual experience server 1502 and client devices 1510 may execute a virtual experience engine / application (1504 and 1512, respectively). The online virtual experience server 1502 using virtual experience engine 1504 may perform some or all the virtual experience engine functions (e.g., generate physics commands, rendering commands, etc.), or offload some or all the virtual experience engine functions to virtual experience engine 1504 of client device 1510. In some implementations, each virtual experience 1 06 may have a different ratio between the virtual experience engine functions that are performed on the online virtual experience server 1502 and the virtual experience engine functions that are performed on the client devices 1510. For example, the virtual experience engine 1504 of the online virtual experience server 1502 may be used to generate physics commands in cases where there is a collision between at least two virtual experience objects, while the additional virtual experience engine functionality (e.g., generate rendering commands) may be offloaded to the client device 1510. In some implementations, the ratio of virtual experience engine functions performed on the online virtual experience server 1502 and client device 1510 may be changed (e.g., dynamically) based on virtual experience engagement conditions. For example, if the number of users engaging in a particular virtual experience 1506 exceeds a threshold number, the online virtual experience server 1502 may perform one or more virtual experience engine functions that were previously performed by the client devices 1 10.

[0155] For example, users may be playing a virtual experience 1506 on client devices 1510, and may send control instructions (e.g., user inputs, such as right, left, up, down, user election, or character position and velocity information, etc.) to the online virtual experience server 1502. Subsequent to receiving control instructions from the client devices 1510, the online virtual experience server 1502 may send experience instructions (e.g., position and velocity information of the characters participating in the group experience or commands, such as rendering commands, collision commands, etc.) to the client devices 1510 based on control instructions. For instance, the online virtual experience server 1502 may perform one or more logical operations (e.g., using virtual experience engine 1504) on the control instructions to generate experience instruction) s) for the client devices 1510. In other instances, online virtual experience server 1502 may pass one or more or the control instructions from one client device 1510 to other client devices (e.g. , from client device 1510a to client device 1510b) participating in the virtual experience 1506. The client devices 1510 may use the experience instructions and render the virtual experience for presentation on the displays of client devices 1510.

[0156] In some implementations, the control instructions may refer to instructions that are indicative of actions of a user’s character within the virtual experience. For example, control instructions may include user input to control action within the experience, such as right, left, up, down, user selection, gyroscope position and orientation data, force sensor data, etc. The control instructions may include character position and velocity information. In some implementations, the control instructions are sent directly to the online virtual experience server 1502. In other implementations, the control instructions may be sent from a client device 1510 to another client device (e.g., from client device 1510b to client device 1510n), where the other client device generates experience instructions using the local virtual experience engine 1504. The control instructions may include instructions to play a voice communication message or other sounds from another user on an audio device (e.g., speakers, headphones, etc.), for example voice communications or other sounds generated using the audio spatialization techniques as described herein.

[0157] In some implementations, experience instructions may refer to instructions that enable a client device 1510 to render a virtual experience, such as a multiparticipant virtual experience. The experience instructions may include one or more of user input (e.g., control instructions), character position and velocity information, or commands (e.g., physics commands, rendering commands, collision commands, etc.).

[0158] In some implementations, characters (or virtual experience objects generally) are constructed from components, one or more of which may be selected by the user, that automatically join together to aid the user in editing.

[0159] In some implementations, a character is implemented as a 3D model and includes a surface representation used to draw the character (also known as a skin or mesh) and a hierarchical set of interconnected bones (also known as a skeleton or rig). The rig may be utilized to animate the character and to simulate motion and action by the character. The 3D model may be represented as a data structure, and one or more parameters of the data structure may be modified to change various properties of the character, e.g., dimensions (height, width, girth, etc.); body type; movement style; number / type of body parts; proportion (e.g., shoulder and hip ratio); head size; etc. is provided as illustration. In some implementations, any number of client devices 1510 may be used.

[0160] One or more characters (also referred to as an “avatar” or “model” herein) may be associated with a user where the user may control the character to facilitate a user’ s interaction with the virtual experiences 1506.

[0161] In some implementations, a character may include components such as body parts (e.g., hair, arms, legs, etc.) and accessories (e.g., t-shirt, glasses, decorative images, tools, etc.). In some implementations, body parts of characters that are customizable include head type, body part types (arms, legs, torso, and hands), face types, hair types, and skin types, among others. In some implementations, the accessories that are customizable include clothing (e.g., shirts, pants, hats, shoes, glasses, etc.), weapons, or other tools.

[0162] In some implementations, for some asset types, e.g., shirts, pants, etc. the online virtual experience platform may provide users access to simplified 3D virtual object models that are represented by a mesh of a low polygon count, e.g., between about 20 and about 30 polygons.

[0163] In some implementations, the user may also control the scale (e.g., height, width, or depth) of a character or the scale of components of a character. In some implementations, the user may control the proportions of a character (e.g., blocky, anatomical, etc.). It may be noted that is some implementations, a character may not include a character virtual experience object (e.g., body parts, etc.) but the user may control the character (without the character virtual experience object) to facilitate the user’s interaction with the virtual experience (e.g., a puzzle game where there is no rendered character game object, but the user still controls a character to control in-game action).

[0164] In some implementations, a component, such as a body part, may be a primitive geometrical shape such as a block, a cylinder, a sphere, etc., or some other primitive shape such as a wedge, a torus, a tube, a channel, etc. In some implementations, a creator module may publish a user's character for view or use by other users of the online virtual experience server 1502. In some implementations, creating, modifying, or customizing characters, other virtual experience objects, virtual experiences 1506, or virtual experience environments may be performed by a user using a I / O interface (e.g., developer interface) and with or without scripting (or with or without an application programming interface (API)). It may be noted that for purposes of illustration, characters are described as having a humanoid form. It may furtherbe noted that characters may have any form such as a vehicle, animal, inanimate object, or other creative form.

[0165] In some implementations, the online virtual experience server 1502 may store characters created by users in the data store 1520. In some implementations, the online virtual experience server 1502 maintains a character catalog and virtual experience catalog that may be presented to users. In some implementations, the virtual experience catalog includes images of virtual experiences stored on the online virtual experience server 1502. In addition, a user may select a character (e.g., a character created by the user or other user) from the character catalog to participate in the chosen virtual experience. The character catalog includes images of characters stored on the online virtual experience server 1502. In some implementations, one or more of the characters in the character catalog may have been created or customized by the user. In some implementations, the chosen character may have character settings defining one or more of the components of the character.

[0166] In some implementations, a user’s character (e.g., avatar) can include a configuration of components, where the configuration and appearance of components and more generally the appearance of the character may be defined by character settings. In some implementations, the character settings of a user’s character may at least in part be chosen by the user. In other implementations, a user may choose a character with default character settings or character setting chosen by other users. For example, a user may choose a default character from a character catalog that has predefined character settings, and the user may further customize the default character by changing some of the character settings (e.g., adding a shirt with a customized logo). The character settings may be associated with a particular character by the online virtual experience server 1502.

[0167] In some implementations, the client device(s) 1510 may each include computing devices such as personal computers (PCs), mobile devices (e.g., laptops, mobile phones, smart phones, tablet computers, or netbook computers), network-connected televisions, gaming consoles, etc. In some implementations, a client device 1510 may also be referred to as a “user device.” In some implementations, one or more client devices 1510 may connect to the online virtual experience server 1502 at any given moment. It may be noted that the number of client devices 1510 is provided as illustration. In some implementations, any number of client devices 1510 may be used.

[0168] In some implementations, each client device 1510 may include an instance of the virtual experience application 1512, respectively. In one implementation, the virtual experience application 1512 may permit users to use and interact with online virtual experience server 1502, such as control a virtual character in a virtual experience hosted by online virtual experience server 1502, or view or upload content, such as virtual experiences 1506, images, video items, web pages, documents, and so forth. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, present, or navigate content (e.g., virtual character in a virtual environment, etc.) served by a web server. In another example, the virtual experience application may be a native application (e.g., a mobile application, app, virtual experience program, or a gaming program) that is installed and executes local to client device 1510 and allows users to interact with online virtual experience server 1502. The virtual experience application may render, display, or present the content (e.g., a web page, a media viewer) to a user. In an implementation, the virtual experience application may also include an embedded media player (e.g., a Flash® or HTML5 player) that is embedded in a web page.

[0169] According to aspects of the disclosure, the virtual experience application may be an online virtual experience server application for users to build, create, edit, upload content to the online virtual experience server 1502 as well as interact with online virtual experience server 1502 (e.g., engage in virtual experiences 1506 hosted by online virtual experience server 1502). As such, the virtual experience application may be provided to the client device(s) 1510 by the online virtual experience server 1502. In another example, the virtual experience application may be an application that is downloaded from a server.

[0170] In some implementations, each developer device 1530 may include an instance of the virtual experience application 1532, respectively. In one implementation, the virtual experience application 1532 may permit a developer user(s) to use and interact with online virtual experience server 1502, such as control a virtual character in a virtual experience hosted by online virtual experience server 1502, or view or upload content, such as virtual experiences 1506, images, video items, web pages, documents, and so forth. In one example, the virtual experience application may be a web application (e.g., an application that operates in conjunction with a web browser) that can access, retrieve, present, or navigate content (e.g., virtual character in a virtual environment, etc.) served by a web server. In another example, the virtual experience application may be a native application (e.g., a mobile application, app,virtual experience program, or a gaming program) that is installed and executes local to developer device 1530 and allows users to interact with online virtual experience server 1502. The virtual experience application may render, display, or present the content (e.g., a web page, a media viewer) to a user. In an implementation, the virtual experience application may also include an embedded media player (e.g., a Flash® or HTML5 player) that is embedded in a web page.

[0171] According to aspects of the disclosure, the virtual experience application 1532 may be an online virtual experience server application for users to build, create, edit, upload content to the online virtual experience server 1502 as well as interact with online virtual experience server 1502 (e.g., provide and / or engage in virtual experiences 1506 hosted by online virtual experience server 1502). As such, the virtual experience application may be provided to the developer device(s) 1530 by the online virtual experience server 1502. In another example, the virtual experience application 1532 may be an application that is downloaded from a server. Virtual experience application 1532 may be configured to interact with online virtual experience server 1502 and obtain access to user credentials, user currency, etc. for one or more virtual experiences 1506 developed, hosted, or provided by a virtual experience developer.

[0172] In some implementations, a user may login to online virtual experience server 1502 via the virtual experience application. The user may access a user account by providing user account information (e.g., username and password) where the user account is associated with one or more characters available to participate in one or more virtual experiences 1506 of online virtual experience server 1502. In some implementations, with appropriate credentials, a virtual experience developer may obtain access to virtual experience virtual objects, such as inplatform currency (e.g., virtual currency), avatars, special powers, accessories, that are owned by or associated with other users.

[0173] In general, functions described in one implementation as being performed by the online virtual experience server 1502 can also be performed by the client device(s) 1510, or a server, in other implementations if appropriate. In addition, the functionality attributed to a particular component can be performed by different or multiple components operating together. The online virtual experience server 1502 can also be accessed as a service provided to other systems or devices through suitable application programming interfaces (APIs), and thus is not limited to use in websites.

[0174] FIG. 16 is a block diagram that illustrates an example computing device 1600 which may be used to implement one or more features described herein, in accordance with some implementations. In one example, computing device 1600 may be used to implement a computer device (e.g., 1502 and / or 1510 of FIG. 15), and perform appropriate method implementations described herein. Computing device 1600 can be any suitable computer system, server, or other electronic or hardware device. For example, the computing device 1600 can be a mainframe computer, desktop computer, workstation, portable computer, or electronic device (portable device, mobile device, cell phone, smartphone, tablet computer, television, TV set top box, personal digital assistant (PDA), media player, game device, wearable device, etc.). In some implementations, computing device 1600 includes a processor 1602, a memory 1604, input / output (I / O) interfaces 1606, and audio / video input / output devices 1614.

[0175] Processor 1602 can be one or more processors and / or processing circuits to execute program code and control basic operations of the computing device 1600. A “processor” includes any suitable hardware and / or software system, mechanism or component that processes data, signals or other information. A processor may include a system with a general- purpose central processing unit (CPU), multiple processing units, dedicated circuitry for achieving functionality, or other systems. Processing need not be limited to a particular geographic location or have temporal limitations. For example, a processor may perform its functions in “real-time,” “offline,” in a “batch mode,” etc. Portions of processing may be performed at different times and at different locations, by different (or the same) processing systems. A computer may be any processor in communication with a memory.

[0176] Memory 1604 is typically provided in computing device 1600 for access by the processor 1602, and may be any suitable processor-readable storage medium, e.g., random access memory (RAM), read-only memory (ROM), Electrical Erasable Read-only Memory (EEPROM), Flash memory, etc., suitable for storing instructions for execution by the processor, and located separate from processor 1602 and / or integrated therewith. Memory 1604 can store software operating on the computing device 1600 by the processor 1602, including an operating system 1608, a virtual experience application 1610, a radiance field projection application 1612, and other applications (not shown). In some implementations, virtual experience application 1610 and / or radiance field projection application 1612 can include instructions that enable processor 1602 to perform the functions (or control the functions of) described herein, e.g., some or all of the methods described with respect to FIGS. 11-14.

[0177] For example, virtual experience application 1610 can include a radiance field projection application 1612, which as described herein can manage radiance field projection operations within an online virtual experience server (e.g., 1502). Elements of software in memory 1604 can alternatively be stored on any other suitable storage location or computer- readable medium. In addition, memory 1604 (and / or other connected storage device(s)) can store instructions and data used in the features described herein. Memory 1604 and any other type of storage (magnetic disk, optical disk, magnetic tape, or other tangible media) can be considered "storage" or "storage devices."

[0178] I / O interface(s) 1606 can provide functions to enable interfacing the computing device 1600 with other systems and devices. For example, network communication devices, storage devices (e.g., memory and / or data store 1520), and input / output devices can communicate via I / O interface(s) 1606. In some implementations, the I / O interface(s) 1606 can connect to interface devices including input devices (keyboard, pointing device, touchscreen, microphone, camera, scanner, etc.) and / or output devices (display device, speaker devices, printer, motor, etc.).

[0179] The audio / video input / output devices 1614 can include a user input device (e.g., a mouse, etc.) that can be used to receive user input, a display device (e.g., screen, monitor, etc.) and / or a combined input and display device, that can be used to provide graphical and / or visual output.

[0180] For ease of illustration, FIG. 16 shows one block for each of processor 1602, memory 1604, I / O interface(s) 1606, and software blocks of operating system 1608, virtual experience application 1610, and radiance field projection application 1612. These blocks may represent one or more processors or processing circuitries, operating systems, memories, I / O interfaces, applications, and / or software engines. In other implementations, computing device 1600 may not have all of the components shown and / or may have other elements including other types of elements instead of, or in addition to, those shown herein. While the online virtual experience server 1502 is described as performing operations as described in some implementations herein, any suitable component or combination of components of online virtual experience server 1502 or similar system, or any suitable processor or processors associated with such a system, may perform the operations described.

[0181] A user device can also implement and / or be used with features described herein. Example user devices can be computer devices including some similar components as the computing device 1600, e.g., processor(s) 1602, memory 1604, and I / O interface(s) 1606. An operating system, software and applications suitable for the client device can be provided in memory and used by the processor. The I / O interface for a client device can be connected to network communication devices, as well as to input and output devices, e.g., a microphone for capturing sound, a camera for capturing images or video, a mouse for capturing user input, a gesture device for recognizing a user gesture, a touchscreen to detect user input, audio speaker devices for outputting sound, a display device for outputting images or video, or other output devices. A display device within the audio / video input / output devices 1614, for example, can be connected to (or included in) the computing device 1600 to display images pre- and postprocessing as described herein, where such display device can include any suitable display device, e.g., an LCD, LED, or plasma display screen, CRT, television, monitor, touchscreen, 3-D display screen, projector, or other visual display device. Some implementations can provide an audio output device, e.g., voice output or synthesis that speaks text.

[0182] One or more methods described herein (e.g., methods 1100, 1200, 1300, and 1400) can be implemented by computer program instructions or code, which can be executed on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuitry), and can be stored on a computer program product including a non-transitory computer readable medium (e.g., storage medium), e.g., a magnetic, optical, electromagnetic, or semiconductor storage medium, including semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), flash memory, a rigid magnetic disk, an optical disk, a solid-state memory drive, etc. The program instructions can also be contained in, and provided as, an electronic signal, for example in the form of software as a service (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.), or in a combination of hardware and software. Example hardware can be programmable processors (e.g., Field-Programmable Gate Array (FPGA), Complex Programmable Logic Device), general purpose processors, graphics processors, Application Specific Integrated Circuits (ASICs), and the like. One or more methods can be performed as part of or component of an application running on the system, or as an application or software running in conjunction with other applications and operating systems.

[0183] One or more methods described herein can be run in a standalone program that can be run on any type of computing device, a program run on a web browser, a mobile application (“app”) run on a mobile computing device (e.g., cell phone, smart phone, tablet computer, wearable device (wristwatch, armband, jewelry, headwear, goggles, glasses, etc.), laptop computer, etc.). In one example, a client / server architecture can be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives from the server the final output data for output (e.g., for display). In another example, all computations can be performed within the mobile app (and / or other apps) on the mobile computing device. In another example, computations can be split between the mobile computing device and one or more server devices.

[0184] Although the description has been described with respect to particular implementations thereof, these particular implementations are merely illustrative, and not restrictive. Concepts illustrated in the examples may be applied to other examples and implementations.

[0185] The functional blocks, operations, features, methods, devices, and systems described in the present disclosure may be integrated or divided into different combinations of systems, devices, and functional blocks as would be known to those skilled in the art. Any suitable programming language and programming techniques may be used to implement the routines of particular implementations. Different programming techniques may be employed, e.g., procedural or object-oriented. The routines may execute on a single processing device or multiple processors. Although the steps, operations, or computations may be presented in a specific order, the order may be changed in different particular implementations. In some implementations, multiple steps or operations shown as sequential in this specification may be performed at the same time.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method to transfer texture from a source three- dimensional (3D) object to a target 3D object, the method comprising: obtaining a spatial grid for a set of Gaussians for the source 3D object, wherein the source 3D object has a texture, and wherein each Gaussian of the set of Gaussians occupies a respective subgrid of the spatial grid based on a covariance size of the Gaussian; performing target vectorization on the target 3D object to generate a cached projection map for the target 3D object; and performing texture projection for each pixel in the cached projection map using the spatial grid to associate the pixel with a pixel color.

2. The computer-implemented method of claim 1 , wherein the source 3D object is represented as a 3D mesh, and wherein further comprising, prior to obtaining the spatial grid, converting the 3D mesh into the set of Gaussians.

3. The computer-implemented method of claim 2, wherein converting the 3D mesh into the set of Gaussians is performed via local space texture projection.

4. The computer-implemented method of claim 1 , wherein a position and a normal of each Gaussian corresponds to a current pixel local space and the covariance size of each Gaussian is based upon a respective rate of change within positional iterations in a UV- space and a current normal of a source 3D mesh of the source 3D object.

5. The computer-implemented method of claim 1 , wherein the spatial grid is a cubic grid, and wherein individual axes of the spatial grid have a size equal to a cube-root of a total target pixel count of the target 3D object.

6. The computer-implemented method of claim 5, wherein each subgrid corresponds to a respective pixel of the target 3D object.

7. The computer-implemented method of claim 1 , wherein performing texture projection for each pixel in the cached projection map comprises:finding a subgrid in the spatial grid that corresponds to the pixel; and identifying, based on the spatial grid, particular Gaussians from the set ofGaussians that are aligned with the pixel.

8. The computer-implemented method of claim 7, wherein performing texture projection for each pixel in the cached projection map further comprises performing a ray cast from a position of the pixel to blend colors of the particular Gaussians by alpha transparency values.

9. The computer-implemented method of claim 1 , wherein performing the target vectorization comprises performing local space vectorization on a target mesh of the target 3D object to generate the cached projection map, wherein the cached projection map comprises a positional map and a normal map.

10. A non-transitory computer-readable medium with instructions stored thereon that, responsive to execution by a processing device, causes the processing device to perform operations comprising: obtaining a spatial grid for a set of Gaussians for a source three-dimensional (3D) object, wherein the source 3D object has a texture, and wherein each Gaussian of the set of Gaussians occupies a respective subgrid of the spatial grid based on a covariance size of the Gaussian; performing target vectorization on a target 3D object to generate a cached projection map for the target 3D object; and performing texture projection for each pixel in the cached projection map using the spatial grid to associate the pixel with a pixel color.

11. The non-transitory computer-readable medium of claim 10, wherein the source 3D object is represented as a 3D mesh, and wherein further comprising, prior to obtaining the spatial grid, converting the 3D mesh into the set of Gaussians.

12. The non-transitory computer-readable medium of claim 11 , wherein converting the 3D mesh into the set of Gaussians is performed via local space texture projection.

13. The non-transitory computer-readable medium of claim 10, wherein a position and a normal of each Gaussian corresponds to a current pixel local space and the covariance size of each Gaussian is based upon a respective rate of change within positional iterations in a UV-space and a current normal of a source 3D mesh of the source 3D object.

14. The non-transitory computer-readable medium of claim 10, wherein the spatial grid is a cubic grid, and wherein individual axes of the spatial grid have a size equal to a cube-root of a total target pixel count of the target 3D object.

15. The non-transitory computer-readable medium of claim 14, wherein each subgrid corresponds to a respective pixel of the target 3D object.

16. A system comprising: a memory with instructions stored thereon; and a processing device, coupled to the memory, the processing device configured to access the memory and execute the instructions, wherein the instructions cause the processing device to perform operations comprising: obtaining a spatial grid for a set of Gaussians for a source three-dimensional (3D) object, wherein the source 3D object has a texture, and wherein each Gaussian of the set of Gaussians occupies a respective subgrid of the spatial grid based on a covariance size of the Gaussian; performing target vectorization on a target 3D object to generate a cached projection map for the target 3D object; and performing texture projection for each pixel in the cached projection map using the spatial grid to associate the pixel with a pixel color.

17. The system of claim 16, wherein the source 3D object is represented as a 3D mesh, and wherein further comprising, prior to obtaining the spatial grid, converting the 3D mesh into the set of Gaussians.

18. The system of claim 17, wherein converting the 3D mesh into the set of Gaussians is performed via local space texture projection.

19. The system of claim 16, wherein a position and a normal of each Gaussian corresponds to a current pixel local space and the covariance size of each Gaussian is based upon a respective rate of change within positional iterations in a UV-space and a current normal of a source 3D mesh of the source 3D object.

20. The system of claim 16, wherein the spatial grid is a cubic grid, and wherein individual axes of the spatial grid have a size equal to a cube-root of a total target pixel count of the target 3D object.

Images