Texture targets in scene description

Abstract

One implementation proposes providing signaling to inform applications about representing textures as a linear combination of textures. Given at least one texture target with associated target weights, the proposed representation format is reconstructed. This representation facilitates the transfer of parametric meshes with parametric textures.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application claims the benefit of European Application No. 23305963.3, filed on 16 June 2023, which is incorporated herein by reference in its entirety.

[0002] This embodiment relates, in general terms, to texture encoding / decoding for 3D scene representation, and more specifically, to a format used to represent textures as a linear combination of texture images in an MPEG-I scene description. [Background technology]

[0003] Three-dimensional scenes and models can be rendered to the user in real time. Several formats, such as Khronos / glTF (Graphics Language Transmission Format), its extensions defined in the MPEG scene description format, or Apple / USDZ, are possible methods for representing rendered 3D content. Such formats support the geometry, appearance, scene graph hierarchy, and animation of 3D models. They are intended to be streamlined, interoperable formats for delivering 3D assets while minimizing file size and runtime processing for applications. In such an environment, improved texture representation is desirable to provide greater flexibility. [Overview of the project]

[0004] The shortcomings and disadvantages of the prior art are resolved and addressed by the general embodiments described herein.

[0005] One implementation proposes providing the encoding / decoding of parametric textures for 3D scene representation. The newly proposed format allows textures to be encoded as a linear combination of texture images in MPEG-I scene descriptions. The term "texture target" is introduced to indicate the texture component in this combination. The new proposed format, called "MPEG_texture_targets," enables the transfer of meshes with parametric textures. Furthermore, the proposed format is fully compatible with existing texture encodings.

[0006] According to a first aspect, a method is provided. The method includes: obtaining from a description of a 3D scene at least one format used to represent a texture as a linear combination of textures; obtaining from the description at least one texture target having associated target weights; and reconstructing a texture based on the at least one format and the at least one texture target having associated target weights.

[0007] A method is provided according to a second aspect of the method, which includes: obtaining from a description of a 3D scene at least one format used to represent a texture as a linear combination of textures; obtaining from the description at least one texture target having associated target weights; and encoding a texture based on the at least one format and the at least one texture target having associated target weights.

[0008] One or more embodiments also provide a computer program that, when executed by one or more processors, includes instructions causing one or more processors to carry out a method according to any of the embodiments described herein. One or more of these embodiments also provide a computer-readable storage medium storing instructions for processing a scene description according to the method described herein.

[0009] One or more embodiments also provide a computer-readable storage medium storing scene descriptions generated according to the methods described above. One or more embodiments also provide methods and apparatus for transmitting or receiving scene descriptions generated according to the methods described herein. [Brief explanation of the drawing]

[0010] [Figure 1] This document illustrates the exemplary architecture of 3D content creation tools and a state-of-the-art graphics processing engine. [Figure 2] This shows an example of the syntax for a data stream that encodes a 3D scene description. [Figure 3] An illustrative graph of a 3D scene description is shown. [Figure 4] This specification shows the relationships between top-level sequences in either a glTF asset or an MPEG-I scene description (SD) that can be implemented as described herein. [Figure 5a] This shows various steps of the same animation of a texture according to one embodiment. [Figure 5b] This shows various steps of the same animation of a texture according to one embodiment. [Figure 5c] This shows various steps of the same animation of a texture according to one embodiment. [Figure 6] The texture image used in the animation of the texture according to one embodiment is shown. [Figure 7] An exemplary embodiment of a method for analyzing TextureInfo is shown. [Figure 8] This shows various textures of a parametric mesh to which the texture image processing according to one embodiment can be applied. [Modes for carrying out the invention]

[0011] Figure 1 shows an exemplary architecture of a 3D processing engine 130 that may be configured to carry out the methods described herein. Devices according to the architecture of Figure 1 are linked with other devices via their bus 131 and / or via I / O interface 136.

[0012] Device 130 comprises the following elements, which are linked to each other by a data and address bus 131. - For example, a microprocessor 132 (or CPU), which is a DSP (or Digital Signal Processor), -ROM (or Read Only Memory) 133 -RAM (or Random Access Memory) 134, -Storage interface 135, - An I / O interface 136 for receiving data to be sent from the application, and - Power source (not shown in Figure 1), e.g., battery.

[0013] For example, the power supply is external to the device. In each of the above memories, the word “register” as used herein may correspond to a small area (a few bits) or a very large area (e.g., an entire program or a large amount of received or decoded data). ROM 133 includes at least a program and parameters. ROM 133 may store algorithms and instructions for implementing the technology according to the principles described herein. When switched on, CPU 132 uploads the program into RAM and executes the corresponding instructions.

[0014] The RAM 134 includes a program that is executed by the CPU 132 and uploaded after the device 130 is switched on, includes input data in the register, includes intermediate data of different states of the method in the register, and includes other variables used to execute the method in the register.

[0015] The device 130 is linked, for example, via the bus 131, to a set of sensors 137 and a set of rendering devices 138. The sensors 137 may be, for example, a camera, a microphone, a temperature sensor, an inertial measurement device, GPS, a humidity measurement sensor, an IR or UV light sensor, or a wind sensor. The rendering device 138 may be, for example, a display, a speaker, a vibrator, heat, a fan, etc.

[0016] According to an example, the device 130 is configured to implement a method according to the principles described herein and belongs to a set including the following. - A mobile device, - A communication device, - A game device, - A tablet (or tablet computer), - A laptop, - A still camera, - A video camera.

[0017] In 3D applications, a scene description is used to combine an explicit and easily parseable description of the scene structure with several binary representations of media content. Figure 2 shows an example of the syntax of a data stream encoding a 3D scene description. Figure 2 also shows an exemplary structure 210 of a 3D scene description. The structure resides within a container that organizes the stream into independent syntax elements. This structure may include a header section 220, which is a set of data common to all syntax elements in the stream. For example, the header section contains some metadata about the syntax elements, describing their respective properties and roles. The structure also includes a payload containing elements of syntax 230 and elements of syntax 240. The syntax elements 230 contain data representing media content items described in the scene graph nodes related to the scene elements. Images, meshes, and other raw data may be compressed according to a compression method. Alternatively, images, meshes, and other raw data may be embedded directly into the scene description data 240 without compression. The elements of syntax 240 are part of the payload of the data stream and contain data that encodes the scene description as described herein in accordance with the principles described herein.

[0018] Figure 3 shows an exemplary graph 310 of a 3D scene description. In this example, the scene graph may include descriptions of real-world objects, e.g., a "plane horizontal plane" (which could be a table or a road), and descriptions of 3D objects 312, e.g., the animation of a car. The scene description is organized as an array of nodes 310. Nodes can link to child nodes to form a scene structure 311. Nodes can carry descriptions of real-world objects (e.g., semantic descriptions) or descriptions of 3D objects. In the example in Figure 3, node 301 describes a camera located within the 3D volume of the scene. Node 302 describes a car and includes an index of the car's representation, e.g., an index in an array of 3D meshes. Node 303 is a child of node 302 and includes a description of one of the car's wheels. Similarly, it includes an index to the wheel's 3D mesh. Since the scale, position, and orientation of objects are described in the scene nodes, the same 3D mesh may be used for several objects in the 3D scene.

[0019] Figure 4 illustrates a current solution for representing textures in MPEG-I Scene Description (SD). In particular, Figure 4 shows the glTF file structure. The entry point is a “scene” node (430), which contains one or more “node” nodes (435) that can be, for example, a “camera” (410) or a “mesh” (440). The MPEG-I Scene Description (SD) format allows for the storage of texture images (498) or links (498) in a glTF file. The representation of a texture (490) uses an image (498) and a texture sampler (495) within the glTF file. The first image (498) defines an image loaded from a file or an image contained in the glTF file (with a buffer view (450)). The second texture sampler (495) refers to a sampler that allows for the definition of how the image data is used. For example, the MPEG-I Scene Description (SD) format allows for the definition of sampling and wrapping strategies. Furthermore, extensions can be applied to KHR_texture_transforms, which specify more complex transformations to apply to texture images, such as rotating or rescaling the image at the glTF texture level. Then, material assets (470) within the glTF file reference texture can be used to define how the texture is used in rendering, such as changing color, normals, etc.

[0020] Today, this approach is limited because it does not offer flexibility when combining different texture images or when attempting to animate textures. Texture merging can be performed before the creation of the glTF file in a so-called "baking" process. During these preprocessing steps, the merge is calculated, and only the resulting texture is stored in the glTF file. If the final scene is to be transmitted and rendered, all the merging process is lost, so no further changes can be made in the rendering of the final scene. Texture animation can also be performed by "baking." The texture image for each frame of the animation can be pre-calculated and then transmitted together with the glTF file. However, those skilled in the art will recognize that such processing results in a very large amount of image data at a rapid pace, and there is still no method to modify these texture images.

[0021] Therefore, it is desirable to improve the signaling / encoding of texture assets, such as combining different texture images or animating textures over time.

[0022] MPEG_Texture_Targets extension in MPEG-I scene description The formats proposed by any of the embodiments disclosed below are advantageously compliant with the glTF format, and with recent MPEG-I SD efforts to extend glTF using MPEG-I SD extensions, or with any other 3D scene description format. Those skilled in the art will recognize that the meaning and use of the disclosed embodiments are common and can be coded in other formats (XML, USD, ...).

[0023] In current versions of MPEG-I, or glTF, and their available extensions, the merging of different texture images or the animation of textures must be predefined and calculated, which is inconvenient, for example, when adapting textures to a parametric mesh. Therefore, one embodiment proposes introducing the definition of a weighted sum of texture images into the glTF file. texture=clamp(textureBase+Σ i weight i ×textureTarget i ) (1)

[0024] Using this newly proposed definition, a weighted sum of images (such as a 2D array of vectors) is processed. According to certain modifications, the values ​​obtained from the weighted sum are clamped to ensure they fall within a valid color range. If the range is defined as [0,1], the clamping operation converts any negative values ​​to zero and any values ​​greater than 1 to 1. Thus, the proposed encoding allows for the creation of new materials based on weighted combinations of texture images.

[0025] In equation (1) above, textureBase and textureTargets are textures, accessors, or images defined in the glTF file using current standards and extensions (for texture, accessor, or image assets, respectively).

[0026] Furthermore, the set of texture bases and targets considered in the merging process is assumed to have the same layout (same width, height, and vector size / channel count), regardless of the process used to construct them. This includes extensions like KHR_texture_transform, which allow for further modification of the layout.

[0027] The glTF specification uses common engineering and graphics terminology, such as images, buffers, and textures, to identify and describe a particular glTF configuration and its attributes, states, and behaviors. This specification defines textures as objects that combine images and their samplers (disclosed in section 5.29 “Texture” of the glTF 2.0 specification (https: / / registry.khronos.org / glTF / specs / 2.0 / glTF-2.0.pdf)). These textures allow for the definition of sampling and wrapping strategies.

[0028] [Table 1]

[0029] The "source" property of a texture refers to the base texture. According to a particular embodiment, it may correspond to the base texture in equation (1).

[0030] Furthermore, glTF files allow textures to be referenced using the TextureInfo property. This is used in several places within material assets, such as "baseColorTexture," "normalTexture," or "metallicRoughnessTexture." This specification defines four properties for TextureInfo: "index," "texCoord," "extensions," and "extras," which are disclosed in Section 5.30 "TextureInfo" of the glTF 2.0 specification (https: / / registry.khronos.org / glTF / specs / 2.0 / glTF-2.0.pdf) and are reproduced below.

[0031] [Table 2]

[0032] The "index" property of TextureInfo refers to the base texture. According to a particular embodiment, it may correspond to the base texture in equation (1).

[0033] According to one embodiment, a new "MPEG_texture_targets" property is defined in the "extensions" attribute, respecting the definition of the TextureInfo property. According to another embodiment, a new "MPEG_texture_targets" property may be defined in the "extensions" attribute of the texture property. Those skilled in the art will recognize that this new "MPEG_texture_targets" property is optional.

[0034] The "MPEG_texture_targets" property contains an array of two properties.

[0035] [Table 3]

[0036] The "targets" property is an array of TextureTargets. This contains all the information needed to reference and ultimately update the texture. Each item at index i in this array is a textureTarget in expression (1). i It corresponds to.

[0037] The "weights" property is an array of floating-point numbers that weight each texture target. Each item at index i in this array is the weight in equation (1). i It corresponds to.

[0038] According to certain characteristics, the sizes of the arrays "targets" and "weights" are the same.

[0039] According to another specific feature, the TextureTarget property defines the texture data for a single texture target.

[0040] [Table 4]

[0041] There are three possible sources.

[0042] If "type" is equal to "texture", the texture data is loaded using the "texture" property, which is a standard TextureInfo property.

[0043] If "type" is equal to "image", the texture data is loaded using the "index" property, which is a number that references one of the images in the glTF file.

[0044] If "type" is equal to "accessor", the texture data is loaded using the "index" property, which is a number that references one of the accessors in the glTF file.

[0045] Regardless of the source type, the layout of the data contained in the referenced accessor must be compatible with the base texture. For example, if the base texture is a 512x512 RGB image, the "texture" source must also be a 512x512 RGB image, and the "accessor" source must be an array of 262144 (=512x512) 3D vectors.

[0046] Advantageously, the disclosed format offers high compatibility with existing implementations, as certain software that does not support texture target extensions can ignore the targets and use the standard-defined base texture. Therefore, assuming that the base texture is the primary contributor in most texture combinations and the targets only slightly alter the final result, imperfect but relevant rendering is still possible. Those skilled in the art will understand that if an item in "targets" has a property "texture" of type TextureInfo, the values ​​in its "extensions" property can be defined recursively, and a specific operation is iterated over for each texture target. For example, each item in the "targets" array can use the KHR_texture_transform extension to transform the texture target before combining them in a weighted sum.

[0047] Example of a glTF schema To clarify, these examples only show the material-related portion of the glTF file. We assume we have a scene with one (or more) meshes that use this material. Therefore, the following will only describe the material loading procedure.

[0048] According to the first exemplary embodiment, the following glTF can define a material having a combination of three texture images for the base color.

[0049] [Table 5]

[0050] [Table 6]

[0051] The material loader first parses the "images" list and loads the three image files "textureBase.jpg", "textureTarget0.jpg", and "textureTarget1.jpg". Next, it parses the "textures" list and assigns each texture to one of the images. Texture 0 corresponds to the image in the "textureBase.jpg" file. Texture 1 corresponds to the image in the file "textureTarget0.jpg". Texture 2 corresponds to the image in the file "textureTarget1.jpg".

[0052] In the next step, the material loader parses the "materials" list and finds a single material with the "pbrMetallicRoughness" attribute. This attribute defines only the base color texture within the "baseColorTexture" attribute. This is a TextureInfo property, and the "index" attribute defines the base texture to be used. Since "index" is equal to 0, it corresponds to texture 0, and therefore to the image in the "texture_base.jpg" file.

[0053] Next, if the material loader supports the proposed "MPEG_texture_targets" extension, it parses the "MPEG_texture_targets" property in "materials / pbrMetallicRoughness / baseColorTexture / extensions". The "targets" list contains two texture targets. Texture target 0 references texture 1 with a weight of -0.1. Texture target 1 references texture 2 with a weight of 0.4.

[0054] As a result, the base color texture of Material 0 is the sum of the following: baseColorTexture=clamp(textureBase.jpg-0.1×textureTarget0.jpg+0.4×textureTarget1.jpg

[0055] The resulting texture is a standard 2D array of RGB vectors that can be processed by the current glTF decoder and renderer.

[0056] According to a second exemplary embodiment, glTF can define a material that creates an animation with a texture. Figures 5a, 5b, and 5c illustrate various steps of the same animation of a texture resulting from processing a texture image as shown in Figure 6, according to a particular embodiment. In this second example, a material can be created that changes over time, starting with a glowing rectangle as shown in Figure 5a. This material can be applied to any surface, for example, a mesh having the shape of a cube or a screen. In the second step, we want the glowing number "2" to slowly appear (for example, the number fades from black to completely white). Figure 5b shows the animated texture with the number "2" fully visible. For that purpose, the texture of the number "2" is added to the rectangle texture in Figure 5a. We then want the number to slowly disappear and fade back to black, and we may further want to repeat the same fade-in / fade-out with the number "1" and the message "GO!". Figure 5c shows the animated texture with the number "1" and the subsequent "GO!" fully visible. The animation ends with a final fade of the "GO!" message, returning to the initial state of Figure 5a, which has an empty rectangle.

[0057] Figure 6 shows a texture image used for the animation of a texture according to one embodiment. Advantageously, the principle described herein allows this animation to be created using a single texture image. This single texture image is a texture atlas, as it can contain four textures within a single image. The glTF file then defines which part of the image to use in each case. The material can be encoded using the following glTF content.

[0058] [Table 7]

[0059] [Table 8]

[0060] [Table 9]

[0061] The glTF loader first decodes the "images" and "textures" properties to read the texture images. In this example, since there is a single texture, it has a single texture image with index 0.

[0062] Next, we analyze the "materials" property to find a single emissive texture. "emissiveTexture" is a TextureInfo, and its "index" property, which has a value of 0, points to the texture image. Its "extensions" property has two items, namely "KHR_texture_transform" and "MPEG_texture_targets".

[0063] In a preferred variation, the other extension, in this example, "KHR_texture_transform," is decoded first. The "offset" and "scale" properties define a texture transformation that crops the top-left portion of the texture atlas corresponding to the glowing rectangle of the sky. Next, the loader decodes the "MPEG_texture_targets" extension. There are three targets, each referencing the same texture ("index" is 0). They all also have "KHR_texture_transform" content describing the crop. The first target crops the top-right portion (number "2"), the second target the bottom-left portion (number "1"), and the last target the bottom-right portion (message "GO!"). The "weights" property defines the amount of each texture target to add.

[0064] The items in the "animations" list define a single animation that updates the material. This animation has one channel in the "channels" list. This refers to the sampler defined in the "samplers" list and points to the "weights" property of the "MPEG_texture_targets" of the luminous texture using the "KHR_animation_pointer" extension. The sampler refers to the time value (in "input") and weight value (in "output"). For clarity, the accessor / buffer view / buffer that holds these values ​​is not shown, but its contents are presented here.

[0065] The value that "input" refers to is as follows: [0,1,2,3,4,5,6]

[0066] These correspond to the main steps (in seconds) of the animation. The values ​​that "output" refers to are as follows: [[0,0,0],[1,0,0],[0,0,0],[0,1,0],[0,0,0],[0,0,1],[0,0,0]]

[0067] Each value of the weight combination (such as w0, w1, w2 = [0, 1, 0]) corresponds to one of the texture targets. That is, the texture target with the number "2", the texture target with the number "1", and the texture target with the message "GO!". The final texture is always the base texture (the one with a rectangle), to which the weighted texture targets are added. texture = texture rectangle + w0 × texture number2 + w1 × texture number1 + w2 × texture messageGO

[0068] Taking into account that linear interpolation is performed between each step, these result in the following changes. Time 0 seconds: All weights are zero. Only the base texture (with a rectangle) is used. Time 1 second: The weights are [1, 0, 0]. The texture with the number "2" is fully added to the base texture. Time 2 seconds: All weights are zero. Only the base texture (with a rectangle) is used. Time 3 seconds: The weights are [0, 1, 0]. The texture with the number "1" is fully added to the base texture. Time 4 seconds: All weights are zero. Only the base texture (with a rectangle) is used. Time 5 seconds: The weights are [0, 0, 1]. The texture with the message "GO!" is fully added to the base texture. Time 6 seconds: All weights are zero.

[0069] Those skilled in the art will understand the data compactness required to generate this texture animation according to at least one embodiment. Thanks to the proposed extension, only four textures (grouped into a single atlas image) are needed to create the animation. In contrast, without a texture target and with a frame rate of 60 images per second, a total of 60 × 6 - 3 = 357 textures would have to be baked.

[0070] According to a third exemplary embodiment, glTF can define a material whose texture source is obtained from an accessor.

[0071] [Table 10]

[0072] [Table 11]

[0073] [Table 12]

[0074] The glTF loader first decodes the "images" and "textures" properties to read the texture images. In this example, there is a single texture with index 0. This texture is a 256x256 RGB image.

[0075] Next, the "buffers" property, the "bufferViews" property, and the "accessors" property are decoded. As a result, 65536 (=256) has a 32-bit floating-point value. * Two arrays of 256 3D vectors are obtained. Accessor 0: A first array of 65536 3D vectors, each having a 32-bit floating-point value. Accessor 1: A second array of 65536 3D vectors, each having a 32-bit floating-point value.

[0076] Next, if the material loader supports the proposed "MPEG_texture_targets" extension, it parses the "MPEG_texture_targets" property in "materials / pbrMetallicRoughness / baseColorTexture / extensions". The "targets" list contains two texture targets. Texture target 0 references accessor 0 with a weight of -0.1. Texture target 1 references accessor 1 with a weight of 0.4.

[0077] As a result, the base color texture of Material 0 is the sum of the following: baseColorTexture=clamp(textureBase.png-0.1×accessor0+0.4×accessor1)

[0078] The resulting texture is a standard 2D array of RGB vectors that can be processed by current glTF decoders and renderers. Conveniently, embodiments based on the "accessor" property allow for the processing of texture data with 32-bit floating-point values. Since standard textures are defined from 8-bit integers converted to floating-point, embodiments based on the "texture" property are limited to 8-bit precision.

[0079] Processing model Figure 7 shows an exemplary embodiment of how TextureInfo is parsed. The proposed extension data is parsed whenever the TextureInfo property occurs within the glTF file. Most glTF parsers already provide a hook mechanism that can be registered. In this case, the main glTF parser is asked to call the extension code whenever it finds the TextureInfo property. As shown in Figure 7, the process takes in the TextureInfo data (710) and outputs the texture (780).

[0080] In the first step (720), the method first parses the data in the default way defined without any extensions. This step (720) initializes the main texture. Next, it tests the texture's extensions property (730). If the "extensions" property does not exist, the process terminates and outputs the main texture (780). If extensions exist, it first applies any non-MPEG_texture_targets extensions such as KHR_texture_transform (740, 750). This applies to all current extensions, but may not apply to future extensions. In these latter cases, it is up to the designer to choose the order of extensions. If no order is defined, it is assumed that MPEG_texture_targets extensions always come last. Each extension updates the main texture. Its layout and values ​​may change, but it remains a texture (e.g., a 2D array of vectors).

[0081] If the MPEG_texture_targets extension is defined in the TextureInfo data (760), the process (770) parses each item in the "targets" property. Since each item in this array is TextureInfo, the process iterates through all of the current processes shown in Figure 7. This means that each texture target (771, 772) can be transformed or can be the result of a combination in a particular way. From the perspective of the process of the present invention, this is not a problem as the process always outputs a texture. The only assumption is that each texture target has the same layout as the main texture. Next, each texture target is multiplied by the corresponding weight in the "weights" array (773) and added to the main texture (774). Once all targets have been processed, the main texture is returned (780).

[0082] These and other aspects, features, and advantages of the general embodiment will become apparent from the following detailed advantages of the exemplary embodiment, which will be read in conjunction with the accompanying drawings.

[0083] An exemplary embodiment relates to texture animation. Using current glTF standards and extensions, textures can be animated with pre-calculation of all steps. All of these steps are then stored in a glTF file, and the texture changes frame by frame, leading to a large memory footprint that may not be suitable for small hardware. Advantageously, if the animation is represented or approximated as a linear combination of textures, as revealed, for example, in non-restrictive Example 2, the proposed extension dramatically reduces the size of the scene data. In this approach, only the required textures and the weights and animation data that reproduce the texture animation are stored.

[0084] Another exemplary embodiment concerns baking lighting effects. A common technique for rendering scenes with complex lighting effects is to use texture baking. This involves pre-calculating the light that scene objects receive and storing it in textures. The renderer then no longer needs to calculate the light, and only the textures need to be applied to obtain a high-quality result. This is a common approach in video games, allowing them to display renderings that consoles cannot calculate in real time. This technique works well when there is little to no dynamism in the lighting of a scene. If there are many changes, such as unstable lighting (campfire, candle, etc.), explosions, object collapse, many textures must be baked periodically to reflect those changes. In the worst case, textures must be baked frame by frame. This results in very large glTF files, and the texture data no longer fits on small hardware. Advantageously, the proposed extension makes it possible to approximate texture changes by combining them. For example, unstable lighting can be approximated by a dozen or so textures, each corresponding to direction and hue. Then, for each frame, a combination of these textures is chosen. This approach can be repeated whenever the true texture can be approximated by a linear combination of texture targets. The resulting glTF file is small, and the scene can be adapted to small hardware.

[0085] Another exemplary embodiment concerns the handling of numerous similar objects or characters. Indeed, some scenes can contain many similar objects or characters. For example, a forest might contain hundreds of trees, or a battlefield might contain many soldiers. Without the proposed extension, if one wishes to have unique objects or characters, one would have to pre-calculate and store different textures for each of them. As in the case mentioned earlier, this would quickly lead to very large glTF files, the data of which may not fit on small hardware. Fortunately, the proposed extension makes it possible to use a limited number of texture targets and to create unique combinations for each object or character in the scene. In this approach, each new item requires only a small texture target weight.

[0086] Figure 8 shows various textures for a parametric mesh to which texture image processing according to one embodiment can be applied. There are many methods for creating combinations of these textures. Here, we present an example of creating textures unique to the face of each character in a scene using the FLAME face model. The FLAME face model is disclosed in "FLAME, Learning a model of facial shape and expression from 4D scans" by Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero (SIGGRAPH ASIA 2017, Bangkok, Thailand). The FLAME face model defines a parametric face rig. Face meshes can be created by combining many morph targets. These meshes can be textured by combining a base texture with 200 texture targets having 512x512 floating-point values. Combinations 810, 820 represent a specific facial identity (skin color, eye brown, etc.), as shown in Figure 8. Combinations of morph targets can be stored in a glTF file using standard MPEG-I SD functionality. An advantage is that the proposed extension allows texture merging to be stored within the same glTF file. As a result, scenes with thousands of characters can be created with a smaller memory footprint.

[0087] Furthermore, such glTF files can be transmitted using many methods (drivers, networks, etc.), and users who receive them can render faces and modify joins as needed. For example, artists in CGI production can receive character drafts and refine them in their own software (Blender, Maya, Unity, etc.) (e.g., change join weights).

[0088] Furthermore, extensions like KHR_animation_pointer can be used to animate texture blending, for example, to render characters whose facial skin changes over time (new hair, tattoos, scars, etc.).

[0089] Various numerical values ​​are used in this application. Certain values ​​are for illustrative purposes only, and the embodiments described are not limited to these specific values.

[0090] Various methods are described herein, each of which includes one or more steps or actions to achieve the described method. Unless a particular order of steps or actions is required for the normal operation of the method, the order and / or use of any particular steps and / or actions may be modified or combined. Furthermore, terms such as “first,” “second,” etc., may be used in various embodiments to modify elements, components, steps, actions, etc., such as “first decryption” and “second decryption.” The use of such terms does not imply a modified order of actions unless specifically required. Thus, in this example, the first decryption does not need to be performed before the second decryption, and may occur, for example, before, during, or over the overlap period with the second decryption.

[0091] The implementations and embodiments described herein may be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even when considered only in the context of a single implementation (for example, only as a method), the implementations of the features considered may also be implemented in other forms (for example, apparatus or programs). Apparatus may be implemented, for example, with appropriate hardware, software, and firmware. The method may be implemented, for example, in an apparatus, which generally refers to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device, e.g., a processor. Processors also include communication devices, e.g., computers, mobile phones, portable / personal digital assistants ("personal digital assistants, PDAs"), and other devices that facilitate the transmission of information between end users.

[0092] The terms "one embodiment" or "one embodiment," or "one implementation" or "one implementation," and any other variations thereof, mean that the specific features, structures, characteristics, etc., described in relation to the embodiments are included in at least one embodiment. Therefore, the appearance of the phrases "in one embodiment" or "in one embodiment," or "in one implementation" or "in one implementation," and any other variations, found in various places throughout this application, do not necessarily all refer to the same embodiment.

[0093] Additionally, this application may refer to "determining" various types of information. Determining information may include, for example, one or more of the following: estimating information, calculating information, predicting information, or retrieving information from memory.

[0094] Furthermore, this application may also refer to “accessing” various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0095] Additionally, this application may refer to “receiving” various types of information. Receiving is intended to be a broad term, similar to “accessing.” Receiving information may include, for example, accessing information or retrieving information (for example, from memory). Furthermore, “receiving” typically involves, in some way, during operation, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0096] For example, in the cases of "A / B", "A and / or B", and "at least one of A and B", the use of any of the following " / ", "and / or", and "at least one of" should be understood as being intended to cover the selection of only the first option (A), only the second option (B), or both options (A and B). As further examples, in the cases of "A, B, and / or C" and "at least one of A, B, and C", such phrasing is intended to cover the selection of only the first option (A), only the second option (B), only the third option (C), only the first and second options (A and B), only the first and third options (A and C), only the second and third options (B and C), or all three options (A, B, and C). This may be extended to the number of items listed, as is obvious to anyone with ordinary knowledge of this and related technologies.

[0097] As will be apparent to those with ordinary skill, the implementations can generate a wide variety of signals, for example, that are formatted to carry information that can be stored or transmitted. This information may include, for example, instructions for performing a method, or data generated by one of the implementations described. For example, a signal may be formatted to carry a bitstream of the embodiment described. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted over a wide variety of different wired or wireless links, as is known. The signal may be stored in a processor-readable medium.

Claims

1. From the description of the 3D scene, obtain at least one format used to represent textures as a linear combination of textures, From the above description, obtain at least one texture target having associated target weights, Reconstructing the texture based on the at least one format and the at least one texture target having the associated target weights, Methods that include...

2. In describing an augmented reality scene, at least one format used to represent textures as a linear combination of textures is specified, To indicate at least one texture target having associated target weights, Encoding the texture based on the at least one format and the at least one texture target having the associated target weights, Methods that include...

3. The method according to claim 1 or 2, wherein the at least one format includes an array of texture targets.

4. The method according to any one of claims 1 to 3, wherein the at least one format includes an array of target weights.

5. Obtaining the base texture, The linear combination of textures is obtained as a weighted sum of the result of multiplying the at least one texture target by the associated target weight and the base texture, The method according to any one of claims 1 to 4, further comprising:

6. The method according to claim 5, further comprising clamping the weighted sum to obtain a linear combination of textures.

7. The method according to any one of claims 1 to 6, wherein the texture target is obtained from the at least one format used to represent the texture as a linear combination of texture targets in a second description of a 3D scene.

8. It comprises one or more processors and at least one memory connected to the one or more processors, The one or more processors described above are: From the description of the 3D scene, obtain at least one format used to represent textures as a linear combination of textures. From the above description, obtain at least one texture target having associated target weights, The texture is reconstructed based on the at least one format and the at least one texture target having the associated target weights. A device configured in such a way.

9. It comprises one or more processors and at least one memory connected to the one or more processors, The one or more processors described above are: In describing an augmented reality scene, at least one format is shown that is used to represent textures as a linear combination of textures. It shows at least one texture target having associated target weights, The texture is encoded based on the at least one format and the at least one texture target having the associated target weights. A device configured in such a way.

10. The apparatus according to claim 8 or 9, wherein the at least one format includes an array of texture targets.

11. The apparatus according to claim 8 or 9, wherein the at least one format includes an array of target weights.

12. The one or more processors described above are: Get the base texture, A linear combination of textures is obtained as a weighted sum of the at least one texture target multiplied by the associated target weight and the base texture. The apparatus according to claim 8 or 9, configured as described above.

13. The apparatus according to claim 12, wherein one or more processors are configured to clamp the weighted sum to obtain a linear combination of textures.

14. The apparatus according to any one of claims 8 to 13, wherein the texture target is obtained from the at least one format used to represent the texture as a linear combination of texture targets in a second description of a 3D scene.

15. A non-temporary computer-readable medium that, when executed by a computer, includes an instruction causing the computer to perform the method according to any one of claims 1 to 7.

Images