Method and apparatus for encoding / decoding 3D meshes - Patents.com

Abstract

A method and apparatus for encoding or decoding a 3D object is provided. For at least one face of a mesh representing the 3D object, including a vertex position and a first texture coordinate in a first texture map associated with the vertex position, a second texture coordinate in a second texture map is obtained from the decoded vertex position of the at least one face and the decoded topology of the mesh. A second texture map is obtained and encoded from the first texture map based on the first texture coordinate and the second texture coordinate.

Description

[Technical field]

[0001] The present embodiments relate generally to methods and apparatus for encoding and decoding of 3D meshes, and more particularly to encoding and decoding of 3D objects represented as meshes. [Background technology]

[0002] Free viewpoint video can be implemented by capturing an animated model using a set of spatially dispatched physical capture devices (video, infrared, ...). The captured animated sequence can then be encoded and transmitted to a terminal for playback from any virtual viewpoint with six degrees of freedom (6dof). There are various approaches to encoding the animated model. For example, the animated model can be represented as an image / video, a point cloud, or a textured mesh.

[0003] In image / video-based approaches, a set of additional metadata is added to the video stream and stored, and warping or any other reprojection is performed to generate an image from a virtual viewpoint during playback. This solution requires large bandwidth and introduces many artifacts.

[0004] In point cloud approaches, an animated 3D point cloud is reconstructed from a set of input animated images, thus leading to a more compact 3D model representation. The animated point cloud can then be projected onto a plane of a volume enclosing the animated point cloud, and the projected points (also known as patches) are encoded into a set of 2D coded video streams for distribution (e.g., using HEVC, AVC, VVC...). However, the nature of the model is very limited in terms of spatial extension, and some artifacts may appear, such as holes on the surface for close-up views.

[0005] In the textured mesh approach, an animated textured mesh is reconstructed from a set of input animated images. Meshes are characterized by the fact that they can have very low shape definition and a photometric texture atlas can be encoded in a standard video stream. Textured mesh encoding relies on texture coordinates (UV) to perform the mapping of texture images onto the faces / triangles of the mesh. Summary of the Invention

[0006] According to one embodiment, there is provided a method for encoding at least one 3D object represented using a mesh or encoding at least one 3D mesh, the method comprising: for at least one face of the mesh, the face including a vertex position and a first texture coordinate in a first texture map associated with the vertex position, obtaining second texture coordinates in a second texture map from a decoded vertex position of the at least one face and a decoded topology of the mesh; obtaining a second texture map from the first texture map based on the first texture coordinates and the second texture coordinates; and encoding the second texture map.

[0007] According to another embodiment, there is provided an apparatus for encoding at least one 3D object represented using a mesh or encoding at least one 3D mesh, comprising one or more processors configured to: obtain, for at least one face of the mesh including a vertex position and a first texture coordinate in a first texture map associated with the vertex position, second texture coordinates in a second texture map from decoded vertex positions of the at least one face and a decoded topology of the mesh, obtain a second texture map from the first texture map based on the first texture coordinates and the second texture coordinates, and encode the second texture map.

[0008] According to another embodiment, there is provided a method for decoding at least one 3D object represented using a mesh or for decoding at least one 3D mesh, the method including: decoding a topology of the mesh and at least one face of the mesh, the at least one face including vertex positions, and obtaining texture coordinates for vertices of the at least one face based on the decoded topology and the decoded vertex positions.

[0009] According to another embodiment, there is provided an apparatus for decoding at least one 3D object represented using a mesh or encoding at least one 3D mesh, comprising one or more processors configured to: decode a topology of the mesh and at least one face of the mesh, the at least one face including vertex positions, and obtain texture coordinates for vertices of the at least one face based on the decoded topology and the decoded vertex positions.

[0010] According to another embodiment, a bitstream is provided, the bitstream comprising coded video data representing a topology of a mesh, at least one face of the mesh, the at least one face including vertex positions; - coded data representing instructions to a decoder to obtain texture coordinates for vertices of at least one face based on the decoded topology and the decoded vertex positions.

[0011] One or more embodiments also provide a computer program comprising instructions that, when executed by one or more processors, cause the one or more processors to perform any one of the encoding or decoding methods according to any of the embodiments described above. One or more of the embodiments also provide a computer-readable storage medium having stored thereon instructions for encoding or decoding a 3D object according to the methods described herein. One or more embodiments also provide a computer-readable storage medium having stored thereon a bitstream generated according to the methods described herein. One or more embodiments also provide a method and apparatus for transmitting or receiving a bitstream generated according to the methods described herein. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates a block diagram of a system in which aspects of the present embodiments may be implemented. [Diagram 2] FIG. 2 shows a block diagram of one embodiment of a video encoder. [Diagram 3] FIG. 3 shows a block diagram of one embodiment of a video decoder. [Figure 4] FIG. 4 illustrates an example of a method for encoding a 3D object, according to one embodiment. [Diagram 5] FIG. 5 illustrates an example of a method for decoding a 3D object according to one embodiment. [Figure 6] FIG. 6 illustrates an example of a method for decoding a 3D object according to another embodiment. [Figure 7] FIG. 7 illustrates an example of a method for decoding a 3D object according to another embodiment. [Figure 8] FIG. 8 shows an example of an original texture map and a reprojected texture map, according to one embodiment. [Figure 9] FIG. 9 illustrates an example of a method for reprojecting a texture map, according to one embodiment. [Figure 10]FIG. 10 illustrates an example of a method for reprojecting a triangle, according to one embodiment. [Figure 11] FIG. 11 illustrates an example of a method for fetching source and destination triangles according to one embodiment. [Figure 12] FIG. 12 illustrates an example of a method for reprojecting pixels, according to one embodiment. [Figure 13] FIG. 13 shows two remote devices communicating over a communications network, in accordance with an example of the present principles. [Figure 14] FIG. 14 shows the syntax of a signal, in accordance with an example of the present principles. [Figure 15] FIG. 15 illustrates an embodiment of a method for transmitting a signal according to any one of the above-mentioned embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] FIG. 1 illustrates a block diagram of an example of a system in which various aspects and embodiments may be implemented. System 100 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 100 may be embodied, alone or in combination, in a single integrated circuit, multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 100 are distributed across multiple ICs and / or separate components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 100 is configured to perform one or more of the aspects described herein.

[0014] System 100 includes at least one processor 110 configured to execute instructions loaded therein, for example, to implement various aspects described herein. Processor 110 may include embedded memory, input / output interfaces, and various other circuits as known in the art. System 100 includes at least one memory 120 (e.g., volatile and / or non-volatile memory devices). System 100 includes storage device 140, which may include non-volatile and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. Storage device 140 may include, by way of non-limiting example, an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0015] The system 100 includes an encoder / decoder module 130 configured to process data to provide, for example, encoded video / 3D objects or decoded video / 3D objects, which may include its own processor and memory. The encoder / decoder module 130 represents a module that may be included within a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding and decoding module. Additionally, the encoder / decoder module 130 may be implemented as a separate element of the system 100 or may be incorporated within the processor 110 as a combination of hardware and software, as is known to those skilled in the art.

[0016] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described herein may be stored in the storage device 140 and then loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input videos / 3D objects, decoded videos / 3D objects or portions of decoded videos / 3D objects, bitstreams, matrices, variables, and intermediate or final results from the processing of expressions, equations, operations, and arithmetic logic.

[0017] In some embodiments, memory internal to the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory, such as RAM, is used as working memory for coding and decoding operations, such as MPEG-2, HEVC, or VVC.

[0018] Input to the elements of system 100 may be provided through various input devices, as indicated at block 105. Such input devices may include, but are not limited to, (i) an RF section that receives, for example, an RF signal transmitted over the air by a broadcast station, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0019] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band that may be referred to (for example) as a channel in a particular embodiment, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs these various functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and refiltering RF signals transmitted over a wired (e.g., cable) medium to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0020] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices over USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 110, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 110, as desired. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the encoder / decoder 130, which operates in combination with the processor 110 and memory and storage elements that process the data streams as desired for presentation on an output device.

[0021] The various elements of system 100 may be provided within an integrated housing in which the various elements may be interconnected and transmit data between each other using suitable connection arrangements 115, e.g., internal buses known in the art, including I2C buses, wiring, and printed circuit boards.

[0022] System 100 includes a communication interface 150 that enables communication with other devices over a communication channel 190. Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 190. Communication interface 150 may include, but is not limited to, a modem or a network card, and communication channel 190 may be implemented in a wired and / or wireless medium, for example.

[0023] Data is streamed to the system 100 in various embodiments using a Wi-Fi network such as IEEE 802.11. The Wi-Fi signal in these embodiments is received via a communication channel 190 and communication interface 150 adapted for Wi-Fi communication. The communication channel 190 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, a set-top box delivering data through an HDMI connection of the input block 105 is used to provide the streamed data to the system 100. In yet other embodiments, an RF connection of the input block 105 is used to provide the streamed data to the system 100.

[0024] System 100 may provide output signals to various output devices, including display 165, speakers 175, and other peripheral devices 185. Other peripheral devices 185, in various example embodiments, include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 100. In various embodiments, control signals are communicated between system 100 and display 165, speakers 175, or other peripheral devices 185 using signaling such as AV.Link, CEC, or other communication protocols that allow inter-device control with or without user intervention. Output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, output devices may be connected to system 100 via communication interface 150 using communication channel 190. Display 165 and speakers 175 may be integrated into a single unit with other components of system 100, for example, in an electronic device such as a television. In various embodiments, the display interface 160 includes a display driver, such as a timing controller (T Con) chip.

[0025] Display 165 and speakers 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which display 165 and speakers 175 are external components, output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0026] 2 illustrates an example video encoder 200, such as a High Efficiency Video Coding (HEVC) encoder, that may be used to encode one or more attributes of an animated mesh, according to one embodiment. FIG. 2 may also illustrate an encoder that improves on the HEVC standard, or that employs technology similar to HEVC, such as the Versatile Video Coding (VVC) encoder under development by the Joint Video Exploration Team (JVET).

[0027] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "encoded" and "coded" may be used interchangeably, the terms "pixel" or "sample" may be used interchangeably, and the terms "image", "picture", and "frame" may be used interchangeably. Usually, but not necessarily, the term "reconstructed" is used on the encoder side, while the term "decoded" is used on the decoder side.

[0028] Before being encoded, a video sequence may undergo pre-encoding processing (201), such as applying a color transformation to the input color picture (e.g., from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0029] In the encoder 200, a picture is coded by the encoder elements as described below. The picture to be coded is divided (202) into units, e.g., CUs, and processed. Each unit is coded, e.g., using either intra mode or inter mode. When a unit is coded in intra mode, it performs intra prediction (260). In inter mode, motion estimation (275) and motion compensation (270) are performed. The encoder decides (205) whether to use intra mode or inter mode to code the unit, and indicates the intra / inter decision, e.g., by a prediction mode flag. The encoder may also mix (263) intra and inter prediction results, or mix results from different intra / inter prediction methods.

[0030] The prediction residual is calculated, for example, by subtracting the predicted block from the original image block (210). The motion refinement module (272) uses already available reference pictures to refine the motion field of the block without referring to the original block. The motion field for a region can be considered as a collection of motion vectors for all pixels that comprise the region. If the motion vectors are subblock-based, the motion field can also be represented as a collection of all subblock motion vectors in the region (all pixels in a subblock have the same motion vector, and the motion vectors can be different for each subblock). If a single motion vector is used for a region, the motion field for the region can also be represented by a single motion vector (the same motion vector for all pixels in the region).

[0031] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying a transform or quantization process.

[0032] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the predicted block are combined (255) to reconstruct an image block. An in-loop filter (265) is applied to the reconstructed picture to perform, for example, deblocking / Sample Adaptive Offset (SAO) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).

[0033] 3 illustrates a block diagram of an exemplary video decoder 300 that may be used to decode one or more attributes of an animated mesh according to one embodiment. In the decoder 300, a bitstream is decoded by a decoder element, as described below. The video decoder 300 generally performs a decoding path that is the inverse of the encoding path described in FIG. 2. The encoder 200 also generally performs video decoding as part of encoding the video data.

[0034] In particular, the decoder's input includes a video bitstream, which may be generated by video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. The decoder may then partition the picture according to the decoded picture partition information (335). The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residual. The decoded prediction residual and the predicted block are combined (355) to reconstruct an image block.

[0035] A prediction block can be obtained (370) from intra prediction (360) or motion compensated prediction (i.e., inter prediction) (375). The decoder may mix (373) intra and inter prediction results, or mix results from multiple intra / inter prediction methods. Before motion compensation, the motion field may be improved (372) by using already available reference pictures. An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380).

[0036] The decoded picture may further undergo post-decoding processing (385), such as an inverse color conversion (e.g., from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding processing (201). The post-decoding processing may use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0037] The present application provides various embodiments for encoding / decoding at least one 3D object or animated 3D object, i.e. a 3D object that evolves over time. According to one embodiment, the 3D object is represented as a point cloud or a 3D mesh. The following embodiments are described for the case of a 3D object represented as a 3D mesh. In some variants, the 3D mesh may be derived from the point cloud of the 3D object.

[0038] A mesh includes at least the following characteristics: a list of vertex positions, a topology defining the connections between the vertices, e.g. a list of faces, and optionally photometric data, such as texture maps or color values, associated with the vertices. The faces defined by the connected vertices can be triangles or any other possible form. For simplest encoding, the photometric data is often projected onto a texture map so that the texture map can be encoded as a video image.

[0039] When coding a mesh with a texture map, it is also necessary to encode the texture UV coordinates that are used to map the image texture onto the mesh surface when rendering a textured 3D object. Storing these UV coordinates can consume a lot of storage space and due to the stronger variability of UV coordinates they can be more complicated to encode than vertex positions. Therefore, providing a solution that allows to encode a mesh without its UV coordinates but still allows to decode the mesh with its UV coordinates is a strong added value for a mesh codec.

[0040] According to the principles presented herein, a method for encoding / decoding 3D objects is provided, in which encoding of texture UV coordinates is avoided when compressing textured meshes. According to one embodiment, a new texture UV atlas is generated based on the compressed-decompressed version of the mesh and by reprojecting the original texture map onto the new texture map using the new UV parameterization. The mesh can then be encoded without UV coordinates and only with metadata in the bitstream, finally in addition to the reprojected texture map. In some embodiments, the metadata can signal activation of UV generation, UV generation mode, and some parameters related to the UV generation mode. During decoding, both the model and the texture map are decoded, and then the UV coordinates corresponding to the decoded texture atlas are determined on the decoder side. Thanks to this approach, the texture coordinates do not need to be encoded in the bitstream. The cost of coding texture UV coordinates is reduced from its original payload (several megabytes) to very few bytes (e.g., 2-16 bytes), if there is metadata, from its original payload (several megabytes).

[0041] According to one embodiment, a method 400 for encoding a 3D object is described with reference to Fig. 4. The 3D object is represented using a mesh that includes faces defined by vertices, which are connected according to the topology of the mesh. Each vertex of the mesh includes a 3D position of the vertex (x,y,z) and a first texture coordinate (u,v) that indicates the location of texture information in a first texture map for the vertex.

[0042] At 401, the topology and vertex positions of the mesh are coded in the bitstream. At 402, second texture coordinates are obtained for the vertices based on the decoded topology and decoded positions of the mesh. The second texture coordinates indicate the location of texture information in the second texture map for the vertex with which the second texture coordinates are associated. At 03, a second texture map is obtained from the first texture map based on the first texture coordinates and the second texture coordinates. At 404, the second texture map is coded in the bitstream.

[0043] 5 illustrates an example of a method 500 for decoding a 3D object according to one embodiment. The 3D object is decoded from a received bitstream that includes at least coded data representing a topology of a mesh representing the 3D object, coded data representing vertex positions.

[0044] At 501, a topology and vertex positions are decoded from a bitstream. At 502, texture coordinates associated with each vertex are determined based on the decoded topology and the decoded vertex positions. The texture coordinates indicate a location of texture information in a texture map for the vertex with which the texture information is associated. In a variation, the method 500 also includes decoding a texture map from the bitstream and rendering the 3D object using the texture map and the texture coordinates to apply the texture map to faces of a mesh.

[0045] FIG. 6 illustrates an example of a method 600 for encoding a 3D object according to an embodiment. The 3D object is represented using a mesh that includes faces defined by vertices, which are connected according to the topology of the mesh. Each vertex of the mesh includes the 3D position of the vertex (x,y,z) and a first texture coordinate (original U,V) that indicates the location of texture information in the first texture map (original texture map patch atlas) for the vertex. Here, the original mesh UV atlas generator is assumed to be unknown. In other words, in order to be as comprehensive as possible, the method used to obtain the first texture coordinate associated with the vertex of the mesh is assumed to be unknown.

[0046] In 601, the vertex positions and the topology of the mesh are encoded. The topology is losslessly encoded, while the vertex positions may be losslessly or lossily coded. Any method for encoding the topology and vertex positions can be used. For example, the EdgeBreaker algorithm as defined in J. Rossignac, "3D compression made simple: Edgebreaker with ZipandWrap on a corner-table" in Proceedings International Conference on Shape Modeling and Applications, Genova, Italy, 2001 can be used. The original UVs are not encoded. The coded topology and positions are added in the bitstream. In 602, the coded topology and coded positions are decoded. In 603, a new UV atlas (i.e., a set of UV coordinates) is determined from the decoded mesh (topology and positions). The new UV atlas can be determined using any UV atlas generator.

[0047] For example, some methods that can be used to generate a UV atlas are defined in: [1]: Microsoft UV atlas. https: / / github.com / microsoft / UVAtlas, [2]: 2018-OptCuts-joint optimization of surface cuts and parameterization. Minchen Li and Danny M. Kaufman and Vladimir G. Kim and Justin Solomon 0001 and Alla Sheffer. ACM Trans. Graph., 247:1-247:13, or [3]: 2002-Least squares conformal maps for automatic texture atlas generation. Bruno Levy and Sylvain Petitjean and Nicolas Ray and Jerome Maillot. ACM Trans. Graph., 362-371. These are just examples, and other methods for UV atlas generation can also be used.

[0048] In some variants, the choice of the method for generating a new UV atlas depends on the trade-off between computational complexity and atlas quality. In this way, a new texture UV is obtained for the decoded topology and position. In 604, the original texture map is reprojected from the original UV atlas (first texture coordinate) to the new UV atlas (second texture coordinate) to obtain a new texture map. Then, in 605, the new texture map is encoded in the bitstream. In some variants, if several methods for generating a UV atlas are possible in the decoder, metadata can also be encoded to signal which UV atlas generation method is used and any necessary parameters of the method, so that the decoder can generate a UV atlas in the same manner as the encoder side.

[0049] FIG. 7 shows an example of a method 700 for decoding a 3D object according to another embodiment. In 701, the position and topology of a mesh and a texture map are decoded from the bitstream. According to the principles provided herein, the bitstream does not include UV texture coordinates associated with the vertices of the mesh. In 702, the decoded mesh including the decoded position and topology is used to determine the UV texture coordinates associated with each decoded vertex of the mesh. The UV texture coordinates are generated according to the same method used in the encoder. In some variants, metadata indicating the UV atlas generation method and any useful parameters are decoded from the bitstream. In this way, the complete decoded mesh and the decoded texture map are obtained. In 703, the 3D object is rendered.

[0050] The encoding and decoding algorithms used to compress and decompress meshes without texture coordinates can be of any kind. However, if the topology changes, e.g. if the topology is lossy coded, a mapping between the original mesh triangles and the coded / decoded triangles must be provided, thus allowing to determine the reprojection of the texture map at the coding stage. This mapping can be a table that associates a destination triangle index with a source triangle index for each triangle of the coded / decoded mesh.

[0051] Texture map reprojection relies on the respective UV coordinates of the source and destination meshes to generate a new texture map that matches the newly generated UV atlas (of the destination mesh). Figure 8 shows an example of an original texture map on the left, and an example of a texture map reprojected using the newly generated UV atlas on the right.

[0052] 9 illustrates an example of a method 900 for reprojecting a texture map, according to one embodiment. For easier understanding, some variables are described below.

[0053] Notation: In the following, * The " character represents standard multiplication.

[0054] In the following, the "." character represents a dereference of a value in a value field (e.g., vec.x accesses the x field of the vector vec).

[0055] Input parameters: srcModel: The source mesh with the original UVs (first texture coordinates). dstModel: The destination mesh with the new UVs (secondary texture coordinates). inputMap: 2D array of colors, input texture maps outputMap: 2D array of colors, the reprojected texture map useFaceMapping: A Boolean value that is set to true if the source and destination meshes are not identically indexed. modelFaceMapping: A table relating destination triangle indexes to source triangle indexes

[0056] variable: srcV1, srcV2, srcV3: source triangle vertices; each vertex contains a real-valued UV 2D vector and a position 3D vector. dstV1, dstV2, dstV3: source triangle vertices; each vertex contains a real-valued UV 2D vector and a position 3D vector, scrTriIdx: source triangle index: integer uvMin: A vector of two components, each of which is a real value. uvMax: A vector of two components, each of which is a real value. intUvMin: A two-component vector, where each component is an integer value. intUvMax: A two-component vector, where each component is an integer value. dstUV: A vector of two components, each of which is real-valued. srcUV: A two-component vector, each component of which is real-valued. bary: a vector of three components, each of which is real-valued srcCol: A color (can be any expression, but must match one of the colors stored in the inputMap and outputMap).

[0057] The method 900 loops over all triangles of the decoded mesh and reprojects each triangle from the original texture map to a new texture map (902). To that end, a variable triIdx, which indicates the index of the current triangle of the decoded mesh, is initialized to 0 in 901. The current triangle is reprojected in 902. The method for reprojecting a triangle is described below. In 903, it is checked whether all triangles of the decoded mesh have been reprojected. If not, the process proceeds to the next triangle in 904. If not, the process ends.

[0058] 10 illustrates an example of a method 1000 for reprojecting a triangle, according to one embodiment. At 1001, the source (i.e., original) triangle and the destination (i.e., decoded) triangle are fetched, i.e., their vertices are retrieved from memory. At 1002, the bounding box of the decoded triangle in a new texture map is determined.

[0059] An example of source code for determining the bounding box of a decoded triangle is provided below:

[0060] [Table 1]

[0061] At 1003, the decoded pixels are projected, i.e. the pixels inside the bounding box of the decoded triangle are parsed and each pixel belonging to the decoded triangle is assigned a color value obtained based on the source texture map.

[0062] FIG. 11 shows an example of a method 1100 for fetching a decoded triangle and a corresponding source triangle according to one embodiment. At 1101, the vertices of the decoded triangle are obtained. At 1102, it is checked whether the source mesh and the decoded mesh are identically indexed. If so, the index of the current triangle of the decoded mesh is the same as the corresponding triangle in the source mesh. Thus, at 1103, the vertices of the source triangle are retrieved. If not, at 1104, the index for the source triangle is obtained from the modelFaceMapping table using the index of the current triangle. The table modelFaceMapping relates the decoded triangle index to the index of the triangle in the source mesh. This table can be determined, for example, when encoding the topology of the source mesh.

[0063] Since the modelFaceMapping table is only needed when acquiring a new texture map, which is then encoded in the bitstream, there is no need to encode the modelFaceMapping table.

[0064] Then, at 1104, the vertex of the source triangle indexed by the obtained index is retrieved.

[0065] 12 illustrates an example of a method 1200 for reprojecting a pixel, according to one embodiment. Method 1200 may be used, for example, in 1003 of method 1000. In 1201, normalized UV coordinates of a current parsed pixel of a decoded triangle bounding box determined in 1002 are obtained. To that end, dstUV is initialized with the uv coordinates corresponding to the center of a pixel with coordinates (i,j) as dstUV={(0.5+i) / inputMap.width,(0.5+j) / inputMap.height}.

[0066] In 1202 it is checked whether the current pixel is inside the decoded triangle. The function getBarycentric is used, which returns true if dstUV is inside the triangle consisting of (dstV1.uv,dstV2.uv,dstV3.uv), and false otherwise. The boolean output is set to the variable inside=getBarycentric(dstUV,dstV1.uv,dstV2.uv,dstV3.uv,bary). This function calculates the barycentric coordinates (u,v,w)~res(x,y,z) for point p for triangle (a,b,c). An example source code for this function is given below:

[0067] [Table 2]

[0068] If the current pixel is inside the decoded triangle, the original UV texture coordinates for the current pixel are determined at 1203. For example, an interpolation using the original UV texture coordinates associated with the source triangle that corresponds to the decoded triangle is used.

[0069] An example of the source code of a function to calculate the position of a point from a triangle (v0,v1,v2) and barycentric coordinates (u,v) is given below: inline void triangleInterpolation(const glm:vec3& v0,const glm:vec3& v1,const glm:vec3& v2,float u,float v,glm:vec3& p) {p=v0 * (1.0fuv)+v1 * u+v2 * v;}

[0070] At 1204 it is checked which method is used to determine the color of the current pixel. If bilinear interpolation is used, at 1205 the color is obtained from the source texture map using bilinear interpolation and the UV texture coordinates determined at 1203.

[0071] If not, then in 1207 a color is obtained from the source texture map using the closest pixel and the UV texture coordinates determined in 1203. In 1206, the determined color is assigned to the current pixel in the new texture map.

[0072] In the methods above, several methods have been used to determine the color for a pixel in a source texture map. Other methods are possible and / or only one method may be available.

[0073] In the following, we provide an example of a syntax that may be used in any one of the embodiments for encoding / decoding 3D objects presented above.

[0074] For example, UV_generation information indicating the activation and / or method used for reprojection may be signaled as follows:

[0075] [Table 3]

[0076] If the UV_generation algorithm is [1], the following parameters related to the method of reprojection can be signaled, and other parameters not described in this specification can also be signaled.

[0077] [Table 4]

[0078] If the UV_generation algorithm is [2], the following parameters about the method can also be signaled, as well as other parameters not described here:

[0079] [Table 5]

[0080] Described above are embodiments of a method for encoding / decoding at least one 3D object represented as an animated textured mesh. These embodiments allow for more economical encoding of texture coordinates at a low cost of metadata (i.e. fewer bytes of metadata per potential megabyte of UV coordinates). The above embodiments can be applied to static and dynamic mesh lossy encoding.

[0081] It may be noted that reprojection introduces small distortions that prevent the use of solutions for lossless coding.

[0082] According to an example of the present principles shown in FIG. 13, in a transmission context between two remote devices A and B over a communication network NET, device A comprises a processor associated with memory RAM and ROM configured to implement a method for encoding a 3D object according to an embodiment as described in relation to FIGS. 1 to 12, and device B comprises a processor associated with memory RAM and ROM configured to implement a method for decoding a 3D object according to an embodiment as described in relation to FIGS. 1 to 12.

[0083] According to one embodiment, the network is a broadcast network adapted to broadcast / transmit signals from device A to decoding devices including device B.

[0084] The signal intended to be transmitted by device A carries at least one bitstream generated by a method for encoding a 3D object according to any one of the embodiments described above. According to one embodiment, the bitstream includes coded video data representing a topology of a 3D object, a mesh representing at least one face of the mesh, the at least one face including vertex positions, and coded data representing instructions to a decoder to obtain texture coordinates for vertices of the at least one face based on the decoded topology and the decoded vertex positions. In some embodiments, the bitstream also includes coded data representing metadata related to a method used to generate texture coordinates and / or texture data associated with the mesh, such as a texture map.

[0085] 14 shows an example of the syntax of such a signal transmitted over a packet-based transmission protocol: Each transmitted packet P includes a header H and a payload PAYLOAD.

[0086] Fig. 15 shows an embodiment of a method (1500) for transmitting a signal according to any one of the above-mentioned embodiments. Such a method includes accessing data (1501) that includes such a signal and transmitting the accessed data (1502) over a communication channel that may be implemented, for example, in a wired and / or wireless medium. According to an embodiment, the method may be performed by the device 100 shown in Fig. 1 or the device A from Fig. 13.

[0087] Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as "first", "second" and the like may be used in various embodiments to modify elements, components, steps, operations, etc., e.g., "first decoding" and "second decoding". The use of such terms does not imply any ordering on the modified operations, unless specifically required. Thus, in this embodiment, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or during an overlapping time with the second decoding.

[0088] Moreover, the aspects are not limited to VVC or HEVC, but may be applied, for example, to other standards and recommendations, and extensions of any such standards and recommendations.Unless otherwise specified or technically precluded, the aspects described in this application may be used individually or in combination.

[0089] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes and the described aspects are not limited to these specific values.

[0090] Various implementations involve decoding. As used herein, "decoding" may encompass all or a portion of the processes performed on a received encoded sequence to generate a final output suitable for, for example, a display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as, for example, entropy decoding, inverse quantization, inverse transform, and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0091] Various implementations involve encoding. Similar to the discussion above regarding "decoding," "encoding" as used in this application may encompass all or a portion of the processes performed on an input video sequence to generate, for example, an encoded bitstream.

[0092] The implementations and aspects described herein may be embodied in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only a single implementation form (e.g., discussed only as a method), the implementation of the discussed features may also be embodied in other forms (e.g., an apparatus or a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The method may be implemented in an apparatus such as a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include, for example, communication devices such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0093] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "an embodiment" or "in one implementation" or "in an implementation," as well as other variations thereof, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0094] Additionally, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0095] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0096] Additionally, the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" typically involves some form of activity, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0097] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A and B and C). This may be expanded as many times as the number of items listed, as would be apparent to one of ordinary skill in the art and related arts.

[0098] Also, as used herein, the term "signaling" specifically means to indicate something to a corresponding decoder. For example, in a particular embodiment, the encoder signals a quantization matrix for dequantization. Thus, in an embodiment, the same parameters are used at both the encoder and decoder sides. Thus, for example, the encoder can transmit a particular parameter to the decoder so that the decoder can use the same particular parameter (explicit signaling). In contrast, if the decoder already has the particular parameter as well as other parameters, a non-transmitting signaling (implicit signaling) can be used to simply allow the decoder to know and select the particular parameter. By avoiding the transmission of any actual function, bit savings are realized in various embodiments. It will be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used in various embodiments to signal information to a corresponding decoder. The above relates to the verb form of the word "signal", which may also be used as a noun in this specification.

[0099] As will be apparent to one of ordinary skill in the art, implementations may generate a variety of signals formatted to carry information that may be, for example, stored or transmitted. Information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The signal that it carries may be, for example, analog or digital information. The signal may be transmitted by a variety of different wired or wireless links, as is known. The signal may be stored in a processor-readable medium.

Claims

1. For at least one face of a mesh representing a 3D object, the mesh including vertex positions and first texture coordinates in a first texture map associated with the vertex positions, generating second texture coordinates in a second texture map from the decoded vertex positions of the at least one face and the decoded topology of the mesh; obtaining the second texture map from the first texture map based on the first texture coordinates and the second texture coordinates; encoding the second texture map; encoding an indication of a method used to generate the second texture coordinates; A method comprising:

2. The method of claim 1 , further comprising: encoding the topology of the mesh and the at least one face of the mesh to provide a coded mesh.

3. The method of claim 1 or claim 2, wherein generating the second texture coordinates comprises generating the second texture coordinates using the decoded vertex positions.

4. 3. The method of claim 1, wherein deriving a second texture map from the first texture map based on the first texture coordinates and the second texture coordinates comprises reprojecting the first texture map onto the second texture map using the first texture coordinates and the second texture coordinates.

5. The method of claim 1 or claim 2, further comprising encoding metadata associated with obtaining the second texture map.

6. 3. The method of claim 1, wherein reprojecting the first texture map onto the second texture map using the first texture coordinates and the second texture coordinates comprises identifying, for at least one decoded face of the coded mesh, a corresponding face in the mesh before encoding.

7. 1. An apparatus comprising one or more processors, The one or more processors: For at least one face of a mesh representing a 3D object, the mesh including vertex positions and first texture coordinates in a first texture map associated with the vertex positions, generating second texture coordinates in a second texture map from the decoded vertex positions of the at least one face and the decoded topology of the mesh; obtaining the second texture map from the first texture map based on the first texture coordinates and the second texture coordinates; encoding the second texture map; encoding an indication of a method used to generate the second texture coordinates; 20. An apparatus configured to:

8. 8. The apparatus of claim 7, wherein the one or more processors are further configured to encode the topology of the mesh and the at least one face of the mesh to provide a coded mesh.

9. The apparatus of claim 7 or claim 8, wherein generating the second texture coordinates comprises generating the second texture coordinates using the decoded vertex positions.

10. 9. The apparatus of claim 7 or claim 8, wherein obtaining a second texture map from the first texture map based on the first texture coordinates and the second texture coordinates comprises reprojecting the first texture map onto the second texture map using the first texture coordinates and the second texture coordinates.

11. The apparatus of claim 7 or claim 8, wherein the one or more processors are further configured to encode metadata associated with obtaining the second texture map.

12. 9. The apparatus of claim 7 or claim 8, wherein reprojecting the first texture map onto the second texture map using the first texture coordinates and the second texture coordinates comprises identifying, for at least one decoded face of the coded mesh, a corresponding face in the mesh before encoding.

13. decoding a topology of a mesh representing a 3D object and at least one face of the mesh, the at least one face including vertex positions; decoding an indication of the method used to generate texture coordinates; generating the texture coordinates for the vertices of the at least one face based on the decoded topology and the decoded vertex positions and based on the indication; A method comprising:

14. decoding a texture map representing texture data associated with the mesh; Rendering the 3D object using at least the generated texture coordinates and the decoded texture map; The method of claim 13 further comprising:

15. The method of claim 13 or 14, wherein the topology and vertex positions are decoded from the bitstream.

16. 15. The method of claim 13 or 14, further comprising decoding an indication to obtain texture coordinates for vertices of the at least one face based on the decoded topology and the decoded vertex positions.

17. 1. An apparatus comprising one or more processors, The one or more processors: decoding a topology of a mesh representing a 3D object and at least one face of the mesh, the at least one face including vertex positions; decoding an indication of the method used to generate texture coordinates; generating the texture coordinates for the vertices of the at least one face based on the decoded topology and the decoded vertex positions and based on the indication; 2. A device configured to:

18. The one or more processors: decoding a texture map representing texture data associated with the mesh; Rendering the 3D object using at least the generated texture coordinates and the decoded texture map; 20. The apparatus of claim 17, further configured to perform:

19. A computer-readable storage medium having stored thereon instructions for causing one or more processors to perform the method of claim 1 or claim 13.

20. A device, An apparatus according to claim 17 or claim 18; at least one of (i) an antenna configured to receive a signal, the signal including data representing at least a portion of a 3D object; (ii) a band limiter configured to limit the received signal to a band of frequencies including the data representing at least a portion of the 3D object; or (iii) a display configured to display at least a portion of the 3D object; A device with.

Images