Adaptive lifting wavelet transform for 3D mesh displacement

Abstract

A decoder decodes, from a bitstream, a first wavelet coefficient representing a first displacement of a first vertex of a three-dimensional (3D) mesh. The decoder applies a reverse lifting wavelet transform to the first wavelet coefficients to determine the first displacements. From the first wavelet coefficients and for a wavelet coefficient of the first wavelet coefficients, the decoder determines a second wavelet coefficient corresponding to a second vertex of the first vertices, the second wavelet coefficients forming an edge associated with a vertex corresponding to the wavelet coefficient. The decoder determines update weights for updating the second wavelet coefficients based on a level of detail (LOD) of the vertex. The second wavelet coefficients are updated based on the wavelet coefficients and the update weights. Based on the wavelet coefficient and a displacement predictor determined from an updated second wavelet coefficient, the decoder converts the wavelet coefficient of the vertex into a displacement of the first displacements.

Description

Cross Reference to Related Applications

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 527,330, filed July 17, 2023, and U.S. Provisional Application No. 63 / 538,069, filed September 12, 2023, all of which are hereby incorporated by reference in their entirety. BRIEF DESCRIPTION OF DRAWINGS

[0002] Examples of several of the various implementations of the present disclosure are described herein with reference to the accompanying drawings.

[0003] Figure 1 An example mesh encoding / decoding system in which implementations of the present disclosure can be implemented is illustrated.

[0004] Figure 2A A block diagram illustrating an example encoder for intra-encoding a 3D mesh, in accordance with some implementations, is illustrated.

[0005] Figure 2B A block diagram illustrating an example encoder for inter-encoding a 3D mesh, in accordance with some implementations, is illustrated.

[0006] Figure 3 A diagram illustrating an example decoder is illustrated.

[0007] Figure 4 A diagram illustrating an example process for generating displacements for an input mesh (e.g., an input 3D mesh frame) to be encoded, in accordance with some implementations, is illustrated.

[0008] Figure 5 An example process for approximating and encoding geometry of a 3D mesh, in accordance with some implementations, is illustrated.

[0009] Figure 6 Examples of vertices of a subdivided mesh (e.g., a subdivided base mesh) corresponding to multiple levels of detail (LoDs), in accordance with some implementations, are illustrated.

[0010] Figure 7A An example of an image filled with displacements (e.g., displacement fields or vectors) using a packing method, in accordance with some implementations, is illustrated.

[0011] Figure 7B An example of a displacement image with marked LODs, in accordance with some implementations, is illustrated.

[0012] Figure 8A An example of a lifting scheme for representing displacement information of a 3D mesh as wavelet coefficients, in accordance with some implementations, is illustrated.

[0013] Figure 8BAn example of a lifting scheme for representing displacement information of a 3D mesh as wavelet coefficients is illustrated, according to some embodiments, in which update weights can be adaptively determined based at least on the LOD corresponding to the lifting operation in which the update weights are used.

[0014] Figure 9A An example forward lifting scheme for transforming displacement of a 3D mesh into wavelet coefficients is illustrated, according to some embodiments.

[0015] Figure 9B An example of an inverse lifting scheme for transforming wavelet coefficients into displacement of a 3D mesh is illustrated, according to some embodiments.

[0016] Figure 10 is a diagram illustrating an example of iteratively performing an inverse lifting scheme for each of the LODs for vertices in a 3D mesh, according to some embodiments.

[0017] Figure 11 A flowchart of a method for performing a forward lifting scheme is illustrated, according to some embodiments.

[0018] Figure 12 A flowchart of a method for performing an inverse lifting scheme is illustrated, according to some embodiments.

[0019] Figure 13 A more detailed process for performing a forward lifting scheme is shown, according to some embodiments.

[0020] Figure 14 A more detailed process for performing an inverse lifting scheme is shown, according to some embodiments.

[0021] Figure 15 A flowchart of a method for performing an inverse lifting scheme for decoding displacement of a 3D mesh is illustrated, according to some embodiments.

[0022] Figure 16 A block diagram of an example computer system in which embodiments of the disclosure can be implemented is illustrated. DETAILED DESCRIPTION

[0023] In the following description, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure, including the structural, system, and method embodiments, can be practiced without these specific details. The description and representation are used by those skilled in the art to most effectively convey the substance of their work to others skilled in the art. In other instances, well-known methods, procedures, components, and circuitry are not described in detail to avoid unnecessarily obscuring aspects of the present disclosure.

[0024] References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described can include a particular feature, structure, or characteristic, but every embodiment can not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of those in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0025] Also note that various embodiments can be described as a process depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged. A process is terminated when its operations are completed, but could be terminated before, upon exception, or in accordance with a termination contract. A process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

[0026] The term “computer-readable medium” includes, but is not limited to portable or fixed storage devices, optical storage devices, and various other mediums capable of storing, containing or carrying instruction(s) and / or data. A computer-readable medium can include a non-transitory medium in which data can be stored and that does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of a non-transitory medium can include, but are not limited to, a magnetic disk, a magnetic tape, or a compact disk (CD) or a digital versatile disk (DVD), a flash memory, a memory or memory device. A computer-readable medium can store code and / or machine- executable instructions that can represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0027] Furthermore, embodiments can be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks (e.g., a computer-program product) can be stored in a computer-readable or machine-readable medium. A processor(s) can perform the necessary tasks.

[0028] Traditional visual data uses a series of pixels to describe an object or scene, each including a location in two dimensions (x and y) and one or more optional attributes, such as color. Volumetric visual data adds another dimension of location to this traditional visual data. Volumetric visual data uses a series of points to describe an object or scene, each including a location in three dimensions (x, y, and z) and one or more optional attributes, such as color. Compared to traditional visual data, volumetric visual data can provide a more immersive way to experience visual data. For example, an object or scene described by volumetric visual data can be viewed from any (or multiple) angles, whereas traditional visual data can generally only be viewed from the angle at which it was captured or rendered. Volumetric visual data can be used in many applications, including augmented reality (AR), virtual reality (VR), and mixed reality (MR). Volumetric visual data can be in the form of a volumetric frame describing an object or scene captured at a particular instance in time, or a sequence of volumetric frames (referred to as a volumetric sequence or volumetric video) describing an object or scene captured at multiple different instances in time.

[0029] One format for storing volumetric visual data is a three-dimensional (3D) mesh (referred to as a mesh or mesh frame hereinafter). A mesh frame (or mesh) includes a collection of points (also referred to as vertices) in three-dimensional (3D) space. Each vertex in the mesh includes geometry information indicating a location of the vertex in 3D space. For example, the geometry information can indicate the location of the vertex in 3D space using three Cartesian coordinates (x, y, and z). In addition, the mesh can include geometry information indicating a plurality of triangles. Each triangle includes three vertices connected by three edges and a face. One or more types of attribute information can be stored for each face (of a triangle). The attribute information can indicate characteristics of a visual appearance of the face. For example, the attribute information can indicate a texture (e.g., color) of the face, a material type of the face, transparency information of the face, reflectivity information of the face, a normal vector of a surface of the face, a velocity at the face, an acceleration at the face, a timestamp indicating when the face (and / or vertex) was captured, or a modality (e.g., running, walking, or flying) indicating how the face (and / or vertex) was captured. In another example, a face (or vertex) can include light field data in the form of multi-view dependent texture information. The light field data can be another type of optional attribute information.

[0030] Triangles (e.g., represented by vertices and edges) in a mesh can describe an object or a scene. For example, triangles in a mesh can describe an external surface and / or internal structure of an object or a scene. The object or scene can be synthetically generated by a computer or can be generated from a capture of a real-world object or scene. Geometric information of a real-world object or scene can be obtained through 3D scanning and / or photogrammetry. 3D scanning can include laser scanning, structured light scanning, and / or modulated light scanning. 3D scanning can obtain geometric information by moving one or more laser heads, structured light cameras, and / or modulated light cameras relative to an object or scene being scanned. Photogrammetry can obtain geometric information by triangulating the same features or points in 2D photos that are spatially displaced. Mesh data can be in the form of a mesh frame describing an object or scene captured at a particular instance in time, or in the form of a sequence of mesh frames (referred to as a mesh sequence or mesh video) describing an object or scene captured at a plurality of different instances in time.

[0031] In addition to one or more types of attribute information, the data size of a mesh frame or sequence can be too large for storage and / or transmission in many applications. For example, a single mesh frame can include thousands or tens of thousands or hundreds of thousands of triangles, where each triangle (e.g., vertex and / or edge) includes geometric information and one or more optional types of attribute information. The geometric information for each vertex can include three Cartesian coordinates (x, y, and z) each represented, for example, using a total of 8 bits or 24 bits. The attribute information for each point can include a texture corresponding to three color components (e.g., R, G, and B color components) each represented, for example, using a total of 8 bits or 24 bits. Thus, in this example, a single vertex includes 48 bits of information with 24 bits of geometric information and 24 bits of texture. Encoding can be used to compress the size of a mesh frame or sequence to provide more efficient storage and / or transmission. Decoding can be used to decompress a compressed mesh frame or sequence for display and / or other forms of consumption (e.g., by machine learning-based devices, neural network-based devices, artificial intelligence-based devices, or other forms of consumption by other types of machine-based processing algorithms and / or devices).

[0032] Compression of a mesh can be lossy (e.g., introduce differences relative to original data) for distribution to and visualization by an end user, for example, on AR / VR glasses or any other 3D-capable device. Lossy compression allows for very high compression ratios but incurs a tradeoff between compression and visual quality perceived by the end user. Other frameworks, such as medical or geological applications, can require lossless compression to avoid altering the decompressed mesh.

[0033] The volumetric visual data can be stored after being encoded into a bitstream in a container, such as a file server in a network. An end user can request a particular bitstream according to the user's requirements. The user can also request adaptive streaming of the bitstream, where an algorithm considers the tradeoff between network resource consumption and visual quality perceived by the end user.

[0034] Figure 1 An exemplary mesh coding / decoding system 100 in which embodiments of the present disclosure can be implemented is illustrated. The mesh coding / decoding system 100 includes a source device 102, a transmission medium 104, and a destination device 106. The source device 102 encodes a mesh sequence 108 into a bitstream 110 for more efficient storage and / or transmission. The source device 102 can store the bitstream 110 and / or can transmit the bitstream to the destination device 106 via the transmission medium 104. The destination device 106 decodes the bitstream 110 to display the mesh sequence 108 or for other forms of consumption. The destination device 106 can receive the bitstream 110 from the source device 102 via a storage medium or the transmission medium 104. The source device 102 and the destination device 106 can be any of a variety of different devices, including a cluster of interconnected computer systems (also referred to as a computer cloud or cloud computer) acting as a seamless pool of resources, a server, a desktop computer, a laptop computer, a tablet computer, a smartphone, a wearable device, a television, a camera, a video game console, a set-top box, a video streaming device, an autonomous vehicle, or a head-mounted display. The head-mounted display can allow a user to view a VR, AR, or MR scene and adjust the perspective of the scene based on movement of the user's head. The head-mounted display can be tethered to a processing device (e.g., a server, a desktop computer, a set-top box, or a video game console) or can be completely self-contained.

[0035] To encode the mesh sequence 108 into the bitstream 110, the source device 102 can include a mesh source 112, an encoder 114, and an output interface 116. The mesh source 112 can provide or generate the mesh sequence 108 from a capture of a natural scene and / or a synthetically generated scene. The synthetically generated scene can be a scene that includes computer-generated graphics. The mesh source 112 can include one or more mesh capture devices (e.g., one or more laser scanning devices, structured light scanning devices, modulated light scanning devices, and / or passive scanning devices), a mesh archive that includes previously captured natural scenes and / or synthetically generated scenes, a mesh feed interface to receive captured natural scenes and / or synthetically generated scenes from a mesh content provider, and / or a processor to generate synthetic mesh scenes.

[0036] As Figure 1As shown in the middle, the mesh sequence 108 can include a series of mesh frames 124. A mesh frame describes an object or scene captured at a particular time instance. When the mesh frames 124 of the mesh sequence 108 are presented continuously using constant or variable time, the mesh sequence 108 can achieve the impression of motion. A (3D) mesh frame includes a set of vertices 126 in 3D space and geometry information of the vertices 126. A 3D mesh can include a set of vertices, edges, and faces that define the shape of a polyhedral object. Further, a mesh frame includes a plurality of triangles (e.g., polygonal triangles). For example, a triangle can include vertices 134A-C and edges 136A-C and a face 132. Faces are typically composed of triangles (triangle mesh), quadrilaterals (Quads), or other simple convex polygons (n-gons) as this simplifies rendering, but these faces can also more generally be composed of concave polygons, or even polygons with holes. Each of the vertices 126 can include geometry information indicating a position of the point in 3D space. For example, the geometry information can indicate the position of the point in 3D space using three Cartesian coordinates (x, y, and z). For example, the geometry information can indicate a plurality of triangles each including three of the vertices 126. One or more of the triangles can also include one or more types of attribute information. The attribute information can indicate characteristics of a visual appearance of the point. For example, the attribute information can indicate a texture (e.g., color) of the face, a material type of the face, transparency information of the face, reflectivity information of the face, a normal vector of a surface of the face, a velocity at the face, an acceleration at the face, a timestamp indicating when the face was captured, a modality (e.g., running, walking, or flying) indicating when the face was captured. In another example, one or more of the faces (or triangles) can include light field data in the form of multi-view dependent texture information. The light field data can be another type of optional attribute information. Color attribute information of one or more of the faces can include a luminance value and two chrominance values. The luminance value can represent a brightness (or luminance component Y) of the point. The chrominance values can represent a blue component and a red component (or chrominance components Cb and Cr) of the point, respectively, separated from the brightness. Other color attribute values are possible based on different color schemes (e.g., RGB or monochrome color schemes).

[0037] In some embodiments, a 3D mesh (e.g., one of the mesh frames 124) can be a static or dynamic mesh. In some examples, a 3D mesh can be represented (e.g., defined) by connectivity information, geometry information, and texture information (e.g., texture coordinates and texture connectivity). In some embodiments, the geometry information can represent the locations of the vertices of the 3D mesh in 3D space, and the connectivity information can indicate how the vertices are to be connected together to form the polygons (e.g., triangles) that make up the 3D mesh. Additionally, the texture coordinates indicate the locations of the pixels in a 2D image corresponding to the vertices of the 3D mesh (or a sub-mesh of the 3D mesh). In some examples, the patch information can indicate how the texture coordinates defined with respect to a 2D bounding box are mapped into the 3D space of the 3D bounding box associated with the patch based on how the points are projected onto the projection plane used for the patch. Additionally, the texture connectivity information can indicate how the vertices represented by the texture coordinates are to be connected together to form the polygons of the 3D mesh (or sub-mesh). For example, each texture or attribute patch of a texture image can correspond to a corresponding sub-mesh defined using the texture coordinates and texture connectivity.

[0038] In some embodiments, for each 3D mesh, one or more 2D images can represent the texture or attribute associated with the mesh. For example, the texture information can include geometry information listed as X, Y, and Z coordinates of vertices, and texture coordinates listed as 2D dimensional coordinates corresponding to the vertices. An example texture mesh can include texture connectivity information indicating the mapping between the geometry coordinates and the texture coordinates to form polygons, such as triangles. For example, a first triangle can be formed by three vertices, where a first vertex is defined as a first geometry coordinate (e.g., 64.062500, 1237.739990, 51.757801) corresponding to a first texture coordinate (e.g., 0.0897381, 0.740830). A second vertex of the triangle can be defined as a second geometry coordinate (e.g., 59.570301, 1236.819946, 54.899700) corresponding to a second texture coordinate (e.g., 0.899059, 0.741542). Finally, a third vertex of the triangle can correspond to a third listed geometry coordinate matching a third listed texture coordinate. However, it should be noted that in some cases, the vertices of a polygon, such as a triangle, can map to a set of geometry coordinates and texture coordinates that can have different index positions in the respective lists of geometry coordinates and texture coordinates. For example, a second triangle has a first vertex corresponding to a fourth listed set of geometry coordinates and a seventh listed set of texture coordinates. A second vertex corresponding to a first listed set of geometry coordinates and a first listed set of texture coordinates, and a third vertex corresponding to a third listed set of geometry coordinates and a ninth listed set of texture coordinates.

[0039] The encoder 114 can encode the mesh sequence 108 into the bitstream 110. To encode the mesh sequence 108, the encoder 114 can apply one or more prediction techniques to reduce redundant information in the mesh sequence 108. Redundant information is information that can be predicted at the decoder, and thus can not need to be transmitted to the decoder for accurate decoding of the mesh sequence 108. For example, the encoder 114 can convert attribute information (e.g., texture information) of one or more mesh frames 124 from 3D to 2D, and then apply one or more 2D video encoders or encoding methods to the 2D images. For example, any of a number of different proprietary or standardized 2D video encoders / decoders can be used, including International Telecommunication Union Telecommunication Standardization Sector (ITU-T) H.1263, ITU-T H.1264, and Moving Picture Experts Group (MPEG)-4 Visual (also known as Advanced Video Coding (AVC)), ITU-T H.1265, and MPEG-H Part 2 (also known as High Efficiency Video Coding (HEVC), ITU-T H.1265, and MPEG-I Part 3 (also known as Versatile Video Coding (VVC)), WebM VP8 and VP9 decoders, and AOMedia Video 1 (AV1). The encoder 114 can encode the geometry of the mesh sequence 108 based on Video Dynamic Mesh Coding (V-DMC). V-DMC specifies the encoding bitstream syntax and semantics for transmitting or storing a mesh sequence, and decoder operations for reconstructing a mesh sequence from a bitstream.

[0040] The output interface 116 can be configured to write and / or store the bitstream 110 onto the transmission medium 104 for transmission to the target device 106. Additionally or alternatively, the output interface 116 can be configured to transmit, upload, and / or stream the bitstream 110 to the target device 106 via the transmission medium 104. The output interface 116 can include a wired and / or wireless transmitter configured to transmit, upload, and / or stream the bitstream 110 according to one or more proprietary and / or standardized communication protocols, such as Digital Video Broadcast (DVB) standards, Advanced Television Systems Committee (ATSC) standards, Integrated Services Digital Broadcast (ISDB) standards, Data Over Cable Service Interface Specification (DOCSIS) standards, Third Generation Partnership Project (3GPP) standards, Institute of Electrical and Electronics Engineers (IEEE) standards, Internet Protocol (IP) standards, and Wireless Application Protocol (WAP) standards.

[0041] The transmission medium 104 can comprise a wireless, wired, and / or computer readable medium. For example, the transmission medium 104 can comprise one or more wires, cables, air interfaces, optical fibers, flash memories, and / or suitable combinations thereof. Further or alternatively, the transmission medium 104 can comprise one or more networks (e.g., the Internet) or file servers for storing and / or transmitting encoded video data.

[0042] To decode the bitstream 110 into the mesh sequence 108 for display or other forms of consumption, the destination device 106 can include an input interface 118, a decoder 120, and a mesh display 122. The input interface 118 can be configured to read the bitstream 110 stored on the transmission medium 104 by the source device 102. Additionally or alternatively, the input interface 118 can be configured to receive, download, and / or stream the bitstream 110 from the source device 102 via the transmission medium 104. The input interface 118 can include a wired and / or wireless receiver configured to receive, download, and / or stream the bitstream 110 according to one or more proprietary and / or standardized communication protocols, such as those mentioned above.

[0043] The decoder 120 can decode the mesh sequence 108 from the encoded bitstream 110. To decode the attribute information (e.g., textures) of the mesh sequence 108, the decoder 120 can reconstruct 2D images compressed using one or more 2D video encoders. The decoder 120 can then reconstruct the attribute information of the 3D mesh frames 124 from the reconstructed 2D images. In some examples, the decoder 120 can decode a mesh sequence that approximates the mesh sequence 108 due to, for example, lossy compression of the mesh sequence 108 by the encoder 114 and / or errors introduced into the encoded bitstream 110 during transmission to the destination device 106. Further, the decoder 120 can decode the geometry of the mesh sequence 108 from the encoded bitstream 110, as will be further described below. Subsequently, one or more of the decoded attribute information can be applied to the decoded mesh frames of the mesh sequence 108.

[0044] The mesh display 122 can display the mesh sequence 108 to a user. The mesh display 122 can include a cathode ray tube (CRT) display, a liquid crystal display (LCD), a plasma display, a light emitting diode (LED) display, a 3D display, a holographic display, a head-mounted display, or any other display device suitable for displaying the mesh sequence 108.

[0045] It is noted that the mesh transcoding / decoding system 100 is presented by way of example, and not limitation. While the mesh transcoding / decoding system 100 is illustrated and described as having a specific number of components in a specific configuration, it is understood that the mesh transcoding / decoding system 100 can have any suitable number of components in any suitable configuration. Figure 1In the example of FIG. 1, the trellis coding / decoding system 100 can have other components and / or arrangements. For example, the trellis source 112 can be external to the source device 102. Similarly, the trellis display 122 can be external to the target device 106, or omitted entirely in cases where the trellis sequence is intended for consumption by a machine and / or storage device. In another example, the source device 102 can also include a trellis decoder, and the target device 106 can include a trellis encoder. In such an example, the source device 102 can be configured to further receive an encoded bitstream from the target device 106 to support bidirectional trellis transmission between the devices.

[0046] Figure 2A A block diagram illustrating an example encoder 200A for intra-frame encoding of 3D meshes in accordance with some embodiments is described. For example, an encoder (e.g., the encoder 114) can include the encoder 200A.

[0047] In some examples, a trellis sequence (e.g., the trellis sequence 108) can include a set of trellis frames (e.g., the trellis frame 124) that can be individually encoded and decoded. As will be described below with respect to Figure 4 Further described, a base mesh 252 can be determined (e.g., generated) from a trellis frame (e.g., an input mesh) by an extraction process. In the extraction process, the mesh topology of the trellis frame can be reduced to determine a base mesh (e.g., an extracted mesh or an extracted base mesh). The base mesh 252 can be encoded by the base mesh encoder 204, the geometry information (e.g., vertices) of which can be quantized by the quantizer 202 to generate a base mesh bitstream 254. In some examples, the base mesh encoder 204 can be an existing encoder, such as Draco or Edgebreaker.

[0048] The displacement generator 208 can generate displacements for vertices of the trellis frame based on the base mesh 252, as will be described below with respect to Figure 4 and Figure 5 Further explained, the displacements are determined based on the reconstructed base mesh 256. The reconstructed base mesh 256 can be determined (e.g., output or generated) by the mesh decoder 206, which decodes the encoded base mesh (e.g., in the base mesh bitstream 254) determined (e.g., output or generated) by the mesh encoder 204. The displacement generator 208 can subdivide the reconstructed base mesh 256 using a subdivision scheme (e.g., a subdivision algorithm) to determine a subdivided mesh (e.g., a subdivided base mesh). The displacements 258 can be determined based on fitting the subdivided mesh to the original input mesh surface. For example, the displacements 258 for the vertices in the trellis frame can include displacement information (e.g., displacement vectors) indicating displacements from the corresponding vertices’ positions in the subdivided mesh to the vertices’ positions in the trellis frame.

[0049] The displacements 258 can be transformed by the wavelet transformer 210 to generate wavelet coefficients (e.g., transform coefficients) that represent the displacement information and can be more efficiently encoded (and subsequently decoded). The wavelet coefficients can be quantized by the quantizer 212 and packed (e.g., arranged) by the image packer 214 into pictures (e.g., one or more images or picture frames) to be encoded by the video encoder 216. The multiplexer 218 can combine (e.g., multiplex) the displacement bitstream 260 output by the video encoder 216 with the base mesh bitstream 254 to form the bitstream 266.

[0050] The attribute information 262 of the mesh frames (e.g., colors, textures, etc.) can be encoded separately from the geometry information of the mesh frames described above. In some examples, the attribute information 262 of the mesh frames can be represented (e.g., stored) by an attribute map (e.g., a texture map) that associates each vertex of a mesh frame with corresponding attribute information for the vertex. The attribute transfer 232 can reparameterize the attribute information 262 in the attribute map based on the reconstructed mesh determined (e.g., generated or output) by the mesh reconstruction component 225. The mesh reconstruction component 225 performs an inverse or decoding function and can be the same or similar component as the decoder (e.g., decoder 300 of FIG. 2) in the decoder 226. Figure 3 The inverse quantizer 228 can inverse quantize the reconstructed base mesh 256 to determine (e.g., generate or output) a reconstructed base mesh 268. The video decoder 226, the image unpacker 224, the inverse quantizer 222, and the inverse wavelet transformer 220 can perform functions that are inverse to the functions of the video encoder 216, the image packer 214, the quantizer 212, and the wavelet transformer 210, respectively. Thus, the reconstructed displacements 270 corresponding to the displacements 258 can be generated by applying the video decoder 226, the image unpacker 224, the inverse quantizer 222, and the inverse wavelet transformer 220 in that order. The warped mesh reconstructor 230 can determine a reconstructed mesh corresponding to the input mesh frames based on the reconstructed base mesh 268 and the reconstructed displacements 270. In some examples, the reconstructed mesh can be the same decoded mesh determined by the decoder based on decoding the base mesh bitstream 254 and the displacement bitstream 260.

[0051] The attribute information of the reparameterized attribute graph can be packed in an image (e.g., a 2D image or a picture frame) by packing component 234. Packing component 234 can fill (e.g., pad) portions of the image that do not contain attribute information. In some examples, color space converter 236 can convert a representation of color (e.g., an example of attribute information 262) from a first format to a second format (e.g., from RGB444 to YUV420) to achieve improved rate-distortion (RD) performance when encoding the attribute graph. In examples, color space converter 236 can also perform chroma subsampling to further improve encoding performance. Finally, video encoder 240 encodes the image (e.g., picture frame) representing attribute information 262 of the mesh frame to determine (e.g., generate or output) an attribute bitstream 264 that is multiplexed into bitstream 266 by multiplexer 218. In some examples, video encoder 240 can be an existing 2D video compression encoder, such as a HEVC encoder or a VVC encoder.

[0052] Figure 2B A block diagram illustrating an example encoder 200B for interframe encoding of 3D meshes according to some embodiments is described. For example, an encoder (e.g., encoder 114) can include encoder 200B. As shown in FIG. 2B, encoder 200B includes many of the same components as encoder 200A. In comparison to encoder 200A, encoder 200B does not include mesh encoder 204 and mesh decoder 206 corresponding to a coder for static 3D meshes. Instead, encoder 200B includes motion encoder 242, motion decoder 244, and base mesh reconstructor 246. Motion encoder 242 can determine a motion field (e.g., one or more motion vectors (MVs)) that, when applied to a reconstructed quantized reference base mesh 243, approximates a base mesh 252. Figure 2B

[0053] The determined motion field can be encoded in bitstream 266 as a motion bitstream 272. In some examples, the motion field (e.g., motion vectors in x, y, and z directions) can be entropy encoded as codewords (e.g., for each directional component) produced by a coding scheme, such as an exponential Golomb code, a Rice code, or a combination thereof. In some examples, the codewords can be arithmetically coded, e.g., using CABAC. The prefix portion of the codewords can be context coded, and the suffix portion of the codewords can be bypass coded. In some examples, the sign bits for each directional component of a motion vector can be separately coded.

[0054] In some examples, motion bitstream 272 can also include an indication of the selected reconstructed quantized reference base mesh 243. ​

[0055] In some examples, the motion bitstream 272 can be decoded by the motion decoder 244 and used by the base mesh reconstructor 246 to generate the reconstructed quantized base mesh 256. For example, the base mesh reconstructor 246 can apply the decoded motion field to the reconstructed quantized reference base mesh 243 to determine (e.g., generate) the reconstructed quantized base mesh 256.

[0056] In some examples, the reconstructed quantized reference base mesh m'(j) associated with the reference mesh frame having index j can be used to predict the base mesh m(i) associated with the current frame having index i. The base meshes m(i) and m(j) can include the same: number of vertices, connectivity, texture coordinates, and texture connectivity. The positioning of the vertices can differ between the base mesh m(i) and the base mesh m(j).

[0057] In some examples, the motion field f(i) can be computed by considering the quantized version of m(i) and the reconstructed quantized base mesh m'(j). The base mesh m'(j) can have a different number of vertices than m(j) (e.g., vertices can have been merged or removed). Thus, the encoder can track the transformation applied to m(j) to determine (e.g., generate or obtain) m'(j) and apply it to m(i). This transformation can enable a 1-to-1 correspondence between the vertices of the base mesh m'(j) and the transformed and quantized version of the base mesh m(i) (denoted as m^ (i). The motion field f(i) can be computed by subtracting the quantized position p(i, v) of the vertex v of m^ (i) from the position Pos(j, v) of the vertex v of m'(j): f(i, v) = Pos(i, v) - Pos(j, v). The motion field can be further predicted by using the connectivity information of the base mesh m'(j), and the prediction residual can be entropy coded.

[0058] In some examples, since the motion field compression process can be lossy, a reconstructed motion field denoted as f'(i) can be computed by applying the motion decoder component. The reconstructed quantized base mesh m'(i) can then be computed by adding the motion field to the positioning of the vertices in the base mesh m'(j). To better exploit the temporal correlation in the displacement and attribute map images (e.g., sequence of images / video), inter-frame prediction can be enabled in the video encoder.

[0059] In some embodiments, an encoder (e.g., the encoder 114) can include the encoder 200A and the encoder 200B.

[0060] Figure 3 A diagram showing an example decoder 300 is illustrated. The decoder 300 can be used in conjunction with the encoder 200A and / or the encoder 200B. Figure 2Aand Figure 2B The bit stream 330, which corresponds to bit stream 266 and can be received as a binary file, can be demultiplexed by demultiplexer 302 to separate bit stream 330 into base mesh bit stream 332, displacement bit stream 334, and attribute bit stream 336, which respectively carry base mesh geometry information, displacement geometry information, and attribute information. Attribute bit stream 336 may include one or more attribute graph substreams for each attribute type.

[0061] In some examples, for inter-frame decoding, the bitstream is demultiplexed into separate substreams, including: a motion substream, a displacement substream for localization and possibly for each vertex attribute, zero or more attribute graph substreams, and an atlas substream that contains patch information in the same manner as in V3C / V-PCC.

[0062] In some examples, the underlying grid bitstream 332 can be decoded in intra-frame or inter-frame mode. In intra-frame mode, the static grid decoder 320 can decode the underlying grid bitstream 332 (e.g., to generate a reconstructed underlying grid m'(i)), which is then dequantized by the inverse quantizer 318 to determine (e.g., generate or output) the decoded underlying grid 340 (e.g., reconstructed quantized underlying grid m"(i)). In some examples, the static grid decoder 320 can be coupled with... Figure 2A The corresponding mesh decoder 206.

[0063] In some examples, in inter-frame mode, the base grid bitstream 332 may include motion field information decoded by the motion decoder 324. In some examples, the motion decoder 324 may be coupled with... Figure 2B The motion decoder 324 corresponds to the motion decoder 244. For example, the motion decoder 324 can perform entropy decoding on the base grid bitstream 332 to determine motion field information. In inter-frame mode, the base grid bitstream 332 can indicate the previous base grid (e.g., reference base grid m'(j)) decoded by the static grid decoder 320 and stored (e.g., cached in) the grid buffer 322. The base grid reconstructor 326 can generate a quantized reconstructed base grid m'(i) by applying the decoded motion field (output by the motion decoder 324) to the previously decoded (e.g., reconstructed) base grid m'(j) stored in the grid buffer 322. In some examples, the base grid reconstructor 326 can be associated with... Figure 2B The base mesh reconstructor 246 corresponds to this. The quantized reconstructed base mesh can be dequantized by the inverse quantizer 318 to determine (e.g., generate or output) the decoded base mesh 340 (e.g., the reconstructed base mesh m"(i)). In some examples, the decoded base mesh 340 can be... Figure 2A and Figure 2B The base grid 268 of the reconstruction is the same.

[0064] In some examples, decoder 300 includes video decoder 308, image unpacker 310, inverse quantizer, and inverse wavelet transform 314, which determines (e.g., generates) the decoded shift 338 based on shift bitstream 334. Video decoder 308, image unpacker 310, inverse quantizer, and inverse wavelet transform 314 correspond to video decoder 226, image unpacker 224, inverse quantizer 222, and inverse wavelet transform 220, respectively, and perform the same or similar operations. For example, a picture frame (e.g., an image) received in shift bitstream 334 can be decoded by video decoder 308, the shift information can be unpacked from the decoded image by image unpacker 310, and inverse quantizer 312 can inverse quantize it into determined inverse-quantized wavelet coefficients representing the encoded shift information. Subsequently, the unquantized wavelet coefficients can be inversely transformed by the inverse wavelet transform 314 to determine the decoded displacement d"(i). In other words, the decoded displacement 338 (e.g., the decoded displacement field d"(i)) can be compared with... Figure 2A and Figure 2B The reconstructed displacement is the same as 270.

[0065] The deformable mesh reconstructor 316, corresponding to the deformable mesh reconstructor 230, can determine (e.g., generate or output) the decoded mesh 342 (M"(i)) based on the decoded displacements 338 and the decoded base mesh 340. For example, the deformable mesh reconstructor 316 can combine (e.g., add) the decoded displacements 338 to subdivide the decoded mesh 340 to determine the decoded mesh 342.

[0066] In some examples, decoder 300 includes video decoder 304 that decodes attribute bitstream 336, which includes encoded attribute information represented (e.g., stored) as 2D images (or picture frames), into determined attribute information 344 (e.g., decoded or reconstructed attribute information). In some examples, video decoder 304 may be an existing 2D video compression decoder, such as an HEVC decoder or a VVC decoder. Decoder 300 may include reconstructible attribute information... Figure 2A and Figure 2B The color space converter 236 in the color space converter 306 performs color format conversion.

[0067] Figure 4 Figure 400 illustrates an example process (e.g., a preprocessing operation) for generating a displacement 414 of an input mesh 430 (e.g., an input 3D mesh frame) to be encoded, according to some embodiments. In some examples, the displacement 414 may be... Figure 2A and Figure 2B The displacement 258 shown corresponds to the displacement shown.

[0068] In Figure 400, the mesh extractor 402 determines (e.g., generates or outputs) an initial base mesh 432 based on (e.g., using) an input mesh 430. In some examples, the initial base mesh 432 can be determined (e.g., generated) based on the input mesh 432 through an extraction process. During the extraction process, the mesh topology of the mesh frame can be reduced to determine the initial base mesh (which may be referred to as the extracted mesh or the extracted base mesh). Figure 5 The lieutenant general explained that the extraction process may involve a downsampling process to remove vertices from the input grid 432, such that a small fraction (e.g., 6% or less) of the vertices in the input grid 430 can remain in the initial base grid 432.

[0069] Mesh subdivision 404 applies a subdivision scheme to generate an initial subdivided mesh 434. (The remaining text appears to be unrelated and possibly machine-generated gibberish.) Figure 5 In more detail, the subdivision scheme may involve upsampling the initial base mesh 432 to add more vertices to the 3D mesh based on the topology and shape of the original mesh to generate the initial subdivision mesh 434.

[0070] The fitting component 406 can fit the initially subdivided mesh to determine a deformed mesh 436 that more closely approximates the surface of the input mesh 430. (As for...) Figure 5 In more detail, fitting can be performed by moving the vertices of the initial subdivided mesh 434 toward the surface of the input mesh 430 so that the subdivided mesh 434 can be used to approximate the input mesh 430. In some specific implementations, fitting is performed by moving each vertex of the initial subdivided mesh 434 along the normal direction of the vertices until the vertices intersect the surface of the input mesh 430. The resulting mesh is a deformed mesh 436. The normal direction can be indicated by the vertex normal at the vertex, which can be obtained from the face normal of the triangle formed by the vertices.

[0071] The base mesh generator 408 can perform another fitting process to generate a base mesh 438 based on the initial base mesh 432. For example, the base mesh generator 408 can deform the initial base mesh 432 based on the deformed mesh 436, making the initial base mesh 432 approximate the deformed mesh 436. In some specific implementations, the fitting process can be performed in a manner similar to that of the fitting component 406. For example, the base mesh generator 408 can move each vertex along the normal direction of each vertex in the initial base mesh 432 (e.g., based on the vertex normal at each vertex) until the vertex reaches the surface of the deformed mesh 436. The output of this process is the base mesh 438.

[0072] The base mesh 438 can be output to the mesh reconstruction process 410 to generate the reconstructed base mesh 440. The reconstructed base mesh 440 can be subdivided by the mesh subdivision unit 418, and the subdivided mesh 442 can be input to the displacement generator 420 to generate (e.g., determine or output) the displacement 414, as described below. Figure 5 Further description. In some examples, mesh subdivision 418 may apply the same subdivision scheme as mesh subdivision 404. In these examples, the vertices in the subdivided mesh 442 have a one-to-one correspondence with the vertices in the deformed mesh 436. Thus, displacement generator 420 can generate displacement 414 by calculating the difference between each vertex of the subdivided mesh 442 and the corresponding vertex of the deformed mesh 436. In some specific implementations, this difference can be projected onto the normal direction of the associated vertex, and the resulting vector is displacement 414. In this way, only the sign and magnitude of displacement 414 need to be encoded in the bit stream, thereby improving decoding efficiency. In addition, because the base mesh 438 has been fitted toward the deformed mesh 436, the displacement 414 between the deformed mesh 436 and the subdivided mesh 442 (generated from the reconstructed base mesh 440) will have a small magnitude, which further reduces the payload and improves decoding efficiency.

[0073] In some examples, one advantage of applying the subdivision process is that it allows for more efficient compression while providing a reliable approximation of the original input mesh 430 (e.g., the surface or curve of the original input mesh 430). Compression efficiency is achieved because the base mesh (e.g., the extracted mesh) has fewer vertices compared to the number of vertices in the input mesh 430, thus requiring fewer bits to be encoded and transmitted. Furthermore, once the base mesh has been decoded, the subdivided mesh can be automatically generated by the decoder, requiring no information from the encoder other than the subdivision scheme (e.g., the subdivision algorithm) and the parameters used for subdivision (e.g., the subdivision iteration count). The reconstructed mesh can be determined by decoding the displacement information (e.g., displacement vectors) associated with the vertices of the subdivided mesh (e.g., the subdivided curves / surfaces of the base mesh). Not only does the subdivision process allow for spatial / quality scalability, but the efficient decoding of displacements using wavelet transforms (e.g., wavelet decomposition) further enhances compression performance.

[0074] In some implementations, the mesh reconstruction process 410 includes components for encoding and subsequently decoding the base mesh 438. Figure 4 An example for intra-frame mode is shown, where the mesh reconstruction process 410 may include functions that can be performed separately with... Figure 2AThe quantizer 202, mesh encoder 204, mesh decoder 206, and inverse quantizer 228 operate in the same or similar manner as the quantizer 411, static mesh encoder 412, static mesh decoder 413, and inverse quantizer 416. For inter-frame mode, the mesh reconstruction process 410 may include the quantizer 202, motion encoder 242, motion decoder 244, basic mesh reconstructor 246, and inverse quantizer 228.

[0075] Figure 5 An example process for approximating and encoding the geometry of a 3D mesh, according to some embodiments, is illustrated. For illustrative purposes, the 3D mesh is shown as a 2D curve. The original surface 510 of the 3D mesh (e.g., a mesh frame) includes vertices (e.g., points) and edges connecting adjacent vertices. For example, points 512 and 513 are connected by edges corresponding to surface 514.

[0076] In some examples, an decimation process (e.g., a downsampling process or decimation / downsampling scheme) can be applied to the original surface 510 of the original mesh to generate a downsampled surface 520 of the decimated (or downsampled) mesh. In the context of mesh compression, decimation refers to the process of reducing the number of vertices in a mesh while preserving the overall shape and topology of that mesh. For example, the original mesh surface 510 is decimated into a surface 520 with fewer samples (e.g., vertices and edges) but still retains the main features and shape of the original mesh surface 510. This downsampled surface 520 can correspond to the surface of the base mesh (e.g., the decimated mesh).

[0077] In some examples, after the extraction process, a subdivision process (e.g., a subdivision scheme or subdivision algorithm) can be applied to the downsampled surface 520 to generate an upsampled surface 530 with more samples (e.g., vertices and edges). The upsampled surface 530 can be part of a subdivided mesh (e.g., a subdivided base mesh) generated by subdividing the downsampled surface 520 corresponding to the base mesh.

[0078] Subdivision is a process commonly used after decimation in mesh compression to improve the visual quality of the compressed mesh. The subdivision process involves adding new vertices and faces to the mesh based on the topology and shape of the original mesh. In some examples, the subdivision process begins by taking the reduced mesh generated by the decimation process and iteratively adding new vertices and edges. For example, the subdivision process may include dividing each edge (or face) of the reduced / decimated mesh into shorter edges (or smaller faces) and creating new vertices at the division points. These new vertices are then connected to form new faces (e.g., triangles, quadrilaterals, or other polygons). By applying subdivision after the decimation process, a higher level of compression can be achieved without a significant loss of visual fidelity. Various subdivision schemes can be used, such as, for example, midpoint, Catmull-Clark subdivision, butterfly subdivision, ring subdivision, etc., or combinations thereof.

[0079] For example, Figure 5 An example of a midpoint subdivision scheme is illustrated. In this scheme, each subdivision iteration divides each triangle into four sub-triangles. A new vertex is introduced in the middle of each edge. The subdivision process can be applied independently to geometry and texture coordinates because the connectivity used for geometry and texture coordinates is typically different. The subdivision scheme calculates the newly introduced vertices as follows. v 12 In the first vertex ( v 1) and the second vertex ( v 2) The formed edge ( v 1. v 2) Positioning at the center Pos ( v 12 ): , in Pos ( v 1) and Pos ( v 2) is a vertex v 1 and v Positioning of 2. In some examples, the same process can be used to calculate the texture coordinates of the newly created vertex. For the normal vector, the normalization step can be applied as follows: , N ( v 12 ), N ( v 1) and N ( v 2) These are respectively with the vertex v 12 , v 1 and v 2. Associated normal vectors. ||x || is a vector x Norm 2.

[0080] Using a midpoint subdivision scheme, as shown in the upsampled surface 530, point 531 can be generated as the midpoint of edge 522, which connects points 532 and 533. Point 531 can be added as a new vertex. Edges 534 and 542 are also added to connect the newly added vertex corresponding to point 531. In some examples, the original edge 522 can be replaced by two new edges 534 and 542.

[0081] In some examples, the downsampled surface 520 can be iteratively subdivided to generate the upsampled surface 530. For example, the mesh of the first subdivision, generated by the first iteration of subdivision applied to the downsampled surface 520, can be further subdivided according to the subdivision scheme to generate the mesh of the second subdivision, and so on. In some examples, the number of iterations corresponding to the subdivision level can be predetermined. In other examples, the encoder can indicate the number of iterations to the decoder, which can similarly generate the subdivided mesh, as further described above.

[0082] In some implementations, the subdivided mesh can be deformed toward (e.g., approximated) the original mesh to determine (e.g., acquire or obtain) a prediction of the original mesh with the original surface 510. Points on the subdivided mesh can move along a calculated normal vertex / orientation until they reach the original surface 510 of the original mesh. The distance between the intersection point on the original surface 510 and the subdivided point can be calculated as a displacement (e.g., a displacement vector). For example, point 531 can move toward the original surface 510 along (e.g., represented by edge 542) a calculated normal orientation of the surface. When point 531 intersects with surface 514 of the original surface 510 (of the original / input mesh), a displacement vector 548 can be calculated. The displacement vector 548 applied to point 531 can produce a displacement surface 540 that better approximates the original surface 510. In some examples, displacement information (e.g., displacement vector 548) for vertices of the subdivided mesh (e.g., the upsampled surface 530 of the subdivided mesh) can be... Figure 2A and Figure 2B The shift bit stream 260 shown in the example encoder is encoded and transmitted. It should be noted that, as regarding... Figure 4 As explained, the subdivided mesh corresponding to the upsampled surface can be a subdivided mesh 442 that is compared with the deformed mesh 436 of the original surface 510 representing the input mesh.

[0083] In some implementations, displacement d(i) (e.g., displacement field or displacement vector) can be calculated and / or stored based on local or global coordinates. For example, a global coordinate system is a reference system used to define the positioning and orientation of objects or points in 3D space. The global coordinate system provides a fixed reference frame independent of the object or point being described. The origin of the global coordinate system can be defined as the point where the three axes intersect. Any point in 3D space can be positioned relative to the origin by specifying its location along the three axes using Cartesian coordinates (x, y, z). For example, displacement can be defined in the same Cartesian coordinate system as the input or original mesh.

[0084] In a local coordinate system, mutually perpendicular normal vectors, tangent vectors, and / or binormal vectors can be defined for a local basis in 3D space to represent the orientation and positioning of an object relative to a reference frame in space. In some examples, the displacement field d(i) can be transformed from the normal coordinate system to a local coordinate system, such as one defined by the normals of the subdivided mesh at each vertex (often referred to as vertex normals). The normals at each vertex can be obtained by combining the face normals of the triangles formed by the vertices. In some examples, using a local coordinate system allows for further compression of the tangential component of the displacement compared to the normal component.

[0085] In some implementations, the decoder (e.g.) Figure 3 The decoder 300 can receive and decode a base mesh corresponding to (e.g., having) the downsampled surface 520. Similar to the encoder, the decoder can apply a subdivision scheme to determine a subdivided mesh having an upsampled surface 530 generated from the downsampled surface 520. The decoder can receive and decode displacement information including a displacement vector 548, and determine a decoded mesh (e.g., a reconstructed mesh) based on the subdivided mesh (corresponding to the upsampled surface 530) and the decoded displacement information. For example, the decoder can add the displacement at each vertex to the corresponding vertex's location within the subdivided mesh. The decoder can obtain a reconstructed 3D mesh by combining the obtained / decoded displacements with the vertices of the subdivided mesh.

[0086] Figure 6 Examples of vertices of a subdivided mesh (e.g., a subdivided base mesh) corresponding to multiple levels of detail (LoD) according to some implementation schemes are illustrated. (See above regarding...) Figure 5As described, the subdivision process (e.g., a subdivision scheme) can be an iterative process in which the mesh can be subdivided multiple times, generating a hierarchical data structure containing multiple levels. Each level of the hierarchical data structure can include a different number of data samples (e.g., vertices and edges in the mesh) representing (e.g., forming) different densities / resolutions (e.g., also known as Level of Detail (LoD)). For example, after a first iteration of subdivision, a downsampled surface 520 (of the extracted mesh) can be subdivided into an upsampled surface 530. The upsampled surface 530 can be further subdivided into an upsampled surface 630, and so on. In this case, the vertices of the mesh with the downsampled surface 520 can be considered to be at or associated with LOD0. Vertices generated in the upsampled surface 530 after the first iteration of subdivision (e.g., vertex 632) can be at LOD1. Vertices generated in the upsampled surface 630 after another iteration of subdivision (e.g., vertex 634) can be at LOD2, and so on. In some examples, LOD0 can refer to a vertex generated by decimating an input (e.g., the original) mesh to produce a base mesh with (e.g., having) a downsampled surface 520. For example, a vertex at LOD0 could be... Figure 2A to Figure 2B The basic grid for reconstruction quantization 256 Figure 3 The base mesh for reconstruction / decoding is 340. Figure 4 The vertices of the reconstructed base mesh are 440.

[0087] In some examples, the calculation of displacement in different LODs follows the same principles as described above. Figure 5 The mechanism described is the same. In some examples, the displacement vector 643 can be calculated from the location of vertex 641 in the original surface 510 (of the original mesh) to vertex 642 at LOD0 from the displacement surface 640 of the deformed mesh. Similarly, the displacement vectors 644 and 645 from the corresponding vertices 632 and 634 from LOD1 and LOD2, respectively, can be calculated. Therefore, in some examples, the number of subdivision iterations can correspond to the number of LODs, and one iteration in the iterations can correspond to one LOD in the LODs.

[0088] Figure 7A An example is illustrated where an image 720 (e.g., a picture or picture frame) with a displacement 700 (e.g., a displacement field or vector) is filled using a packing method (e.g., a packing scheme or packing algorithm) according to some implementation schemes. Specifically, as described above regarding... Figure 5 and Figure 6As described, displacements of 700 can be generated and these displacements can be packed into a 2D image. In some examples, the displacement can be a 3D vector containing values ​​for three components for distance. For example, the increment x value represents the offset along the x-axis from point A to point B in a Cartesian coordinate system. In some examples, the displacement vector can be represented by fewer than three components, such as one or two components. For example, when using a local coordinate system to store displacement values, the component with the highest importance can be stored as representing the displacement, and other components can be discarded.

[0089] In some examples, as will be further described below, the displacement values ​​can be transformed into other signal domains for better compression. For example, the displacement can be wavelet transformed and decomposed into and represented as wavelet coefficients (e.g., coefficient values ​​or transform coefficients). In these examples, the displacement 700 packed in image 720 can include the resulting wavelet coefficients (e.g., transform coefficients) that can be compressed more efficiently than the untransformed displacement values. At the decoder side, the decoder can decode the displacement 700 into wavelet coefficients and can apply an inverse wavelet transform process to reconstruct the original displacement values ​​obtained at the encoder.

[0090] In some examples, one or more displacements in displacement 700 can be quantized by the encoder before being packed into displacement image 720. In some examples, one or more displacements can be quantized before, after, or both before and after the wavelet transform. For example, Figure 7A The wavelet transform values ​​8, 4, 1, -1, etc., for quantization at a displacement of 700 are shown. At the decoder side, the decoder can perform inverse quantization to reverse or undo the quantization process performed by the encoder.

[0091] In general, quantization in signal processing can be seen as the process of mapping input values ​​from a larger set to output values ​​in a smaller set. This process is often used in data compression to reduce the amount, precision, or resolution of data into a more compact representation. However, this reduction can lead to information loss and introduce compression artifacts. The choice of quantization parameters (such as the number of quantization levels) is a trade-off between the desired level of precision and the resulting data size. Many different quantization techniques exist, such as selectable / enabled / applied uniform quantization, non-uniform quantization, and adaptive quantization. These techniques can be adopted depending on the specific requirements of the application.

[0092] In some examples, wavelet coefficients (e.g., displacement coefficients representing displacement signals) can be adaptively quantized based on the LoD. As explained above, the mesh can be iteratively subdivided to generate a hierarchical data structure comprising multiple LoDs. In this example, each vertex and its associated displacement belong to the same level of hierarchy within the LOD structure, such as the LOD corresponding to the subdivision iteration in which that vertex was generated. In some examples, vertices at each LOD can be quantized based on quantization parameters corresponding to the LOD, which specify different intensity / precision levels of the signal to be quantized. For example, wavelet coefficients at LOD 3 could have a quantization parameter of, for example, 42, and wavelet coefficients at LOD 0 could have different, smaller quantization parameters of 28 to preserve more detail information in LOD 0.

[0093] In some examples, displacement 700 can be packed onto pixels in a displacement image 720 having width W and height H. In these examples, the size of displacement image 720 (e.g., W multiplied by H) can be greater than or equal to the number of components in displacement 700 to ensure that all displacement information can be packed. In some examples, displacement image 720 can be further divided into smaller regions (e.g., squares) called packing blocks 730. In these examples, the length of packing block 730 can be an integer multiple of 2.

[0094] Displacements 700 (e.g., displacement signals represented by quantized wavelet coefficients) can be packed into packing blocks 730 according to packing order 732. Each packing block 730 can be packed (e.g., arranged or stored) in the displacement image 720 according to packing order 722. Once all displacements 700 are packed, empty pixels in the image 720 can be filled with adjacent pixel values ​​to improve compression. Figure 7A In the example shown, the packing order 722 for packing blocks can be a raster order, and the packing order 732 for displacements within packing block 730 can be, for example, a Z-order. However, it should be understood that other packing schemes can be used for both block and intra-block displacements. In some embodiments, the packing scheme for blocks and / or intra-block displacements can be predetermined. In some embodiments, the packing scheme can be signaled by the encoder in the bitstream of each patch, patch group, tile, image, or image sequence. Accordingly, the signaled packing scheme can be obtained from the bitstream by the decoder.

[0095] In some examples, the packing order 732 can follow a space-filling curve that traverses space in a continuous, non-repeating manner. Some examples of space-filling curve algorithms (e.g., schemes) include Z-order curves, Hilbert curves, Peano curves, Moore curves, Sierpinski curves, Dragon curves, and others. Space-filling curves have been used in image packing techniques to efficiently store and retrieve images in a way that maximizes storage space and minimizes retrieval time. Space-filling curves are well-suited for this task because they provide a one-dimensional representation of a two-dimensional image. A common image packing technique using space-filling curves is called Z-order or Morton order. A Z-order curve is constructed by interleaving the binary representations of the x and y coordinates of each pixel in the image. This creates a one-dimensional representation of the image that can be stored in a linear array. To pack images using a Z-order curve, the image is first divided into small blocks, typically 8×8 or 16×16 pixels in size. Each block is then encoded using a Z-order curve, and each block is stored in a linear array. When it is necessary to retrieve an image, inverse Z-order curves are used to decode the blocks and reassemble them into the original image.

[0096] In some examples, once packaged, the displacement image 720 can be encoded and decoded using a regular 2D video codec.

[0097] Figure 7B An example of a displacement image 720 according to some embodiments is illustrated. As shown, these displacements can be ordered according to the Level of Displacement (LOD) of the displacements 700 filled in the displacement image 720. For example, displacement coefficients (e.g., quantized wavelet coefficients) can be ordered from lowest LOD to highest LOD. In other words, wavelet coefficients representing displacements for vertices at a first LOD can be packed (e.g., arranged and stored in the displacement image 720) according to the first LOD. For example, displacements 700 can be packed from lowest LOD to highest LOD. Higher LODs represent higher density vertices and correspond to more displacements compared to lower LODs. The portions of the displacement image 720 that are not at any LOD can be filled portions.

[0098] In some examples, displacements can be packed in reverse order from highest LOD to lowest LOD. In one example, the encoder can signal whether the displacements are packed from lowest LOD to highest LOD or vice versa.

[0099] In some examples, wavelet transform can be applied to displacement values ​​to generate wavelet coefficients (e.g., displacement coefficients) that can be more easily compressed. Wavelet transform is commonly used in signal processing to decompose a signal into a set of wavelets, which are smaller wave-like functions that allow them to capture local features in the signal. The result of wavelet transform is a set of coefficients representing the contribution of each wavelet at different scales and locations in the signal. This result can be used to detect and locate transient features in a signal and is generally used for signal analysis and data compression, such as image, video, and audio compression.

[0100] Using a 2D image as an example, wavelet transform is used to decompose an image (signal) into two discrete components called approximation / prediction and detail. The decomposed signal is further divided into high-frequency components (details) and low-frequency components (approximation / prediction) by passing it through two filters (a high-pass filter and a low-pass filter). In the 2D image example, two filtering stages (horizontal filtering and vertical filtering) are applied to the image signal. After each filtering stage of the decomposed components, a downsampling step is required to obtain wavelet coefficients, resulting in four sub-signals at each decomposition level. High-frequency components correspond to rapid changes or abrupt transitions in the signal (such as edges or lines in an image). On the other hand, low-frequency components refer to the global characteristics of the signal. Depending on the application, different filtering and compression methods can be implemented. Various types of wavelets exist, such as Haar, Daubechies, Symlets, etc., each with different characteristics such as frequency resolution, temporal positioning, etc.

[0101] In signal processing, lifting schemes are techniques used for both designing wavelets and performing Discrete Wavelet Transform (DWT). Lifting schemes offer several advantages in terms of computational efficiency and flexibility, serving as an alternative to the traditional filter bank implementation of DWT. A lifting scheme uses a series of lifting steps to decompose the signal, allowing the input signal (e.g., displacement for a 3D mesh) to be converted in situ into displacement coefficients. Within a lifting scheme, a series of lifting operations (e.g., lifting steps) can be performed. Each lifting operation involves a prediction step (e.g., prediction operation) and an update step (e.g., update operation). These lifting operations can be applied iteratively to obtain wavelet coefficients.

[0102] In various implementations of 3D mesh decoding, wavelet transforms with lifting (e.g., known as lifting schemes) can be used to transform the displacement for a 3D mesh frame. Specifically, wavelet transforms can "split" the input signal (e.g., the displacement signal) into two signals: an even-sample signal. E Odd-numbered samples (O signal). Even-numbered samples. E It can include two displacement signals associated with two vertices. E 1 and E2. These two vertices are considered to be on the edges of the vertices associated with the input displacement signal. Odd-numbered samples O This can represent the input signal corresponding to that vertex. As explained above, edge information can be determined (e.g., generated or received) based on the subdivision scheme applied to each mesh frame of the 3D mesh.

[0103] Figure 8A Examples of lifting schemes for representing displacement information of a 3D mesh as wavelet coefficients, according to some implementations, are illustrated. The lifting scheme may refer to forward lifting scheme 802A and / or inverse lifting scheme 804A. The lifting scheme includes multiple lifting operations that can be performed iteratively. Each lifting operation may include a prediction operation (e.g., a prediction step) and an update operation (e.g., an update step). An encoder may perform (e.g., apply) forward lifting scheme 802A to determine (e.g., derive, generate, or obtain) wavelet coefficients representing the displacement information. A decoder may perform (e.g., apply) inverse lifting scheme 804A to reverse the operations of the forward lifting scheme, thereby determining (e.g., deriving, generating, or obtaining) the displacement information based on the wavelet coefficients decoded from the bitstream. As explained above, the decoded displacement information may include displacement values ​​(e.g., displacement vectors) corresponding to the vertices of the mesh frame, which the decoder can use to generate a decoded mesh (e.g., a reconstructed mesh).

[0104] In some examples, the forward boosting scheme 802A includes a splitting operation (e.g., a splitting step labeled "split") that splits the signal s j ( j ≥1) Split (e.g., separate) into two signals (e.g., non-overlapping signals): from s evenk The even-numbered sample signal (represented) ) and by s oddk This represents an odd-numbered sample signal. s j This represents a displacement value (e.g., a displacement signal) determined for a vertex of a 3D mesh frame. For example, the displacement value includes a displacement field (e.g., a displacement vector), which can have one, two, or three components, as explained above.

[0105] Forward lift scheme 802A includes, for example, LOD (Level of Detail) N 810, LOD N-1 812, LOD N-2Multiple iterations corresponding to multiple LODs 814 and LOD0816. Each iteration of the forward boosting scheme 802A (e.g., four iterations are shown as four dashed boxes corresponding to LODs 810 to 816) includes a prediction operation (e.g., a prediction step shown as a "P" block / step) that determines (e.g., computes) a prediction for odd-numbered samples based on even-numbered samples. This prediction can be subtracted from the odd-numbered samples (e.g., shown as a circle with a negative sign) to create / generate a prediction error, such as an error signal. d k The forward boosting scheme 802A also includes an update operation (e.g., an update step shown as a "U" block / step) that uses some of the energy removed during subsampling to recalibrate (e.g., for signals corresponding to lower LODs) the low-frequency signal. In the case of classical boosting, this is used to prepare an even-numbered signal for the next prediction operation in the next iteration of the forward boosting scheme 802A. For example, the update operation is based on representing odd-numbered samples. s oddk Error signal of the difference between the predicted odd sample and the corresponding odd sample d k To update (e.g., prepare) even-numbered signals. In some examples, the update operation can be based on the prediction error. d k Added to (e.g., shown as a circle with a positive sign) even-numbered signals s evenk Update the even signal using each even signal in the data. s evenk In some examples, the prediction error can be adjusted by updating the weights. d k As will be discussed below Figure 9A to Figure 9B and Figure 10 The text further describes how even-numbered signals can be updated based on the adjusted prediction error.

[0106] In some implementations, the decoder performs a reverse lifting scheme 804A to reverse the operation of the forward lifting scheme 802A. For example, the forward lifting scheme 802A includes starting from a higher LOD (e.g., LOD 804A). N The promotion operation is performed iteratively from a lower LOD (e.g., LOD0816) to a higher LOD (e.g., LOD0816), while the reverse promotion scheme 804A includes a promotion operation performed iteratively from a lower LOD (e.g., LOD0816) to a higher LOD (e.g., LOD0816). N 810) Iteratively executed promotion operations. Compared to the forward promotion scheme 802A, in each promotion operation of the reverse promotion scheme 804A, the update operation can be performed from even-numbered samples. s evenk Subtract prediction error d kThis is used to update even-numbered samples. In some examples, the prediction error can be adjusted by updating the weights. d k As will be discussed below Figure 9A to Figure 9B and Figure 10 The text further describes how the even-numbered signal can be updated based on the adjusted prediction error. In each boosting operation of the inverse boosting scheme 804A, the prediction operation can, for example, be based on the updated even-numbered signal. s evenk Combinations (e.g., summation or averaging) are used to determine the odd-numbered samples for reconstructed predictions. s oddk Each boosting operation of the inverse boosting scheme 804A reconstructs the predicted odd-numbered samples. s oddk With prediction error d k Combining (e.g., displayed as circles with a positive sign) to determine (e.g., generate or obtain) a displacement signal corresponding to the displacement value determined at the encoder. s oddk In other words, the multiple iterations of the inverse lifting scheme 804A convert the wavelet coefficients, generated by the encoder and representing displacement information, into displacement values ​​that can be used to reconstruct the mesh frame. Furthermore, to recover the splitting operation of the forward lifting scheme 802A, each lifting operation of the inverse lifting scheme 804A includes a merging operation, which updates the even-numbered samples. s evenk With reconstructed odd-numbered samples s oddk To merge (e.g., to sort or combine sequences of signals or values).

[0107] It should be noted that Figure 8A The value in j Corresponding to the number of iterations used in the lifting operation, this number varies depending on the specific requirements of the application. For example, the number of levels in the LOD defined by the mesh extraction process can be used for the lifting operation. In some examples, a midpoint subdivision scheme can be used in the mesh extraction process. In these examples, since each vertex at a higher LOD level is the midpoint of the edge generated by the two vertices at a lower LOD level, the signal associated with that vertex (e.g., displacement value or its wavelet coefficient representation) can be decomposed and represented by two sub-signals (e.g., displacement value or its wavelet coefficient representation) belonging to the corresponding two vertices. For example, a vertex at LOD1 (e.g., LOD of level 1) v It can be two vertices at LOD0 (e.g., LOD of level 0). v 1 and v The midpoint of 2. In this example, a lifting scheme can be used to... vThe associated displacements are subjected to wavelet transform. For the vertex... v (For example, the signal is a displacement signal or its wavelet coefficient representation) corresponding to odd-numbered signals s oddk For odd signals s oddk Determined even-numbered samples s evenk Vertices can be generated from them v vertex v 1 and v 2 (for example, the signal is a displacement signal or its wavelet coefficients) corresponds to this.

[0108] In a boosting scheme, prediction weights and update weights are coefficient values ​​used to modify the input data during the prediction and update steps, respectively. Prediction weights can be scalar values ​​or a set of coefficients defining a linear combination of adjacent signals used for prediction, while update weights determine the contribution of the prediction error to the final updated value. For example, a prediction can be determined based on two even-numbered input samples with a prediction weight equal to half, effectively averaging the signal values ​​of the two even-numbered input samples. Typically, prediction and update weights are chosen to satisfy certain characteristics or conditions to achieve desired properties in the transformed data. For example, in a lossless boosting scheme, weights can be designed to ensure perfect reconstruction of the original signal. In a lossy boosting scheme, weights can be chosen to achieve specific frequency response characteristics or to minimize distortion based on compression or denoising requirements.

[0109] In various specific implementations of 3D mesh decoding, prediction weights and update weights can be determined (e.g., selected) for a lifting scheme applied to the displacements of vertices of a 3D mesh (e.g., each mesh frame in a mesh frame sequence) to balance the accuracy and characteristics produced by the wavelet transform corresponding to the displacement. As explained above, the prediction operation for each iteration of the inverse lifting scheme can depend on (e.g., be influenced by) the updated signal input for the prediction operation. However, the update weights can be chosen to be uniformly applied to the values ​​of the wavelet coefficients corresponding to the displacement (e.g., representing the displacement) (e.g., 1 / 8, 1 / 4, or 1 / 16, etc.). Due to the characteristics and geometry of the mesh frames, the characteristics at each LOD may differ. Therefore, applying the same update weights can reduce the compression of displacements (e.g., displacement signals) for vertices at some LODs.

[0110] Implementations of this disclosure relate to applying adaptive update weights in a lifting scheme applied to the displacement of vertices of a 3D mesh (e.g., mesh frames in a sequence of mesh frames of a 3D mesh). For example, the update weights for a wavelet coefficient can be determined based on the LOD associated with each wavelet coefficient. As explained above, the lifting scheme can include multiple lifting operations corresponding to multiple LODs in the 3D mesh (e.g., mesh frames). For a forward lifting scheme, each iteration of the lifting operation can update (e.g., lift) a sequence of displacement signaling (e.g., displacement values ​​or corresponding wavelet coefficients representing displacement values) from a higher LOD (e.g., denser vertices) to one or more lower LODs (e.g., sparser vertices), and accumulate predictions for vertices at the lowest LOD (e.g., vertices of the base mesh). Similarly, but inversely, for a backward lifting scheme, each iteration of the lifting operation can update (e.g., lift) a sequence of displacement signaling (e.g., displacement values ​​or corresponding wavelet coefficients representing displacement values) from a lower LOD (e.g., sparser vertices) to a higher LOD (e.g., denser vertices). Since the update weights determine the contribution of the prediction error to the final update value, using uniform weights does not account for the impact of different LOD levels, resulting in less accurate prediction signals across different LOD levels. In some examples, lower LOD can be associated with smaller update weights, and higher LOD can be associated with larger update weights.

[0111] In some implementations, the decoder (e.g.) Figure 3 The inverse wavelet transform (314) obtains (e.g., receives or decodes) first wavelet coefficients representing first displacements of first vertices in a 3D mesh at multiple Levels of Depth (LODs) from the bit stream. The decoder applies an inverse lifting wavelet transform to these first wavelet coefficients to determine these first displacements. Applying the inverse lifting wavelet transform involves iteratively applying a lifting operation to second wavelet coefficients from the first wavelet coefficients associated with each of the multiple LODs in order to determine the first displacements.

[0112] For example, applying the inverse lifting wavelet transform to the wavelet coefficients in the first wavelet coefficients involves determining second wavelet coefficients based on the first wavelet coefficients. These second wavelet coefficients: correspond to second vertices in the first vertex that are at one or more LODs below the first LOD of the vertex corresponding to the wavelet coefficient; and are located on vertices or edges associated with vertices. The lifting operation applied to the first wavelet coefficients can include update and prediction operations, as described above regarding... Figure 8AAs explained, as part of the update operation, the decoder can determine update weights for updating the second wavelet coefficients based on the LOD of the vertex. Subsequently, the second wavelet coefficients can be updated based on the wavelet coefficients and the update weights. In some examples, because the update weights can be determined specifically and individually for each LOD, the magnitude of the prediction error due to the geometric characteristics (e.g., contour, curvature, etc.) of the vertices at each LOD can be more appropriately and accurately accounted for. Based on the wavelet coefficients and the displacement predictor determined according to the updated second wavelet coefficients, the decoder converts the wavelet coefficients of the vertex into displacements in the first displacement.

[0113] In some examples, a first indication (e.g., a flag or syntax element) can be signaled in the bitstream to indicate whether to apply adaptive update weights in the boosting scheme. In some examples, the first indication can be signaled for each 3D mesh sequence, each mesh frame, each submesh, each patch, each patch group, each LOD, etc.

[0114] In some examples, a second indication (e.g., a flag or syntax element) can be signaled in the bitstream to indicate the scaling factor to be used for adapting (e.g., updating) the update weights of each LOD. For example, the second indication could indicate an index that indicates (e.g., identifies) the scaling factor. In some examples, the second indication can be signaled based on an indication to enable adaptive update weights. In some examples, the scaling factor can be a fixed value. In some examples, the second indication can be signaled for each 3D mesh sequence, each mesh frame, each submesh, each patch, each patch group, each LOD, etc.

[0115] In some examples, the update weights determined for a displacement can be scaled based on a scaling value (e.g., an adaptive ratio) based on the LOD associated with the displacement. For example, the scaling value could be a power of a scaling factor that has an exponent based on the LOD. For instance, the scaling value could be a scaling factor equal to a power of the difference between the total number of LODs and the LODs. In the example, the update weights could be determined based on the product of the default update weights and the scaling value.

[0116] This article describes these and other implementation schemes.

[0117] Figure 8B An example of a lifting scheme for representing displacement information of a 3D mesh as wavelet coefficients, according to some implementations, is illustrated, wherein the update weights 820 to 826 can be adaptively determined (e.g., set or adjusted) at least based on LODs 810 to 816 corresponding to the lifting operations in which update weights 820 to 826 are used (e.g., applied). This lifting scheme may refer to those corresponding to forward lifting scheme 802A and inverse lifting scheme 804A, respectively (e.g., by...).Figure 2A and / or Figure 2B The forward lifting scheme 802B (executed by the encoder or wavelet transform 210) and / or (e.g. by...) Figure 3 The inverse lifting scheme 804B is executed by the decoder or inverse wavelet transform 314. Similarly, the forward lifting scheme 802B and the inverse lifting scheme 804B include multiple lifting operations corresponding to LODs 810 to 816. In the forward lifting scheme 802B, the lifting operations are iteratively applied (e.g., executed) to the displacement signals of vertices from higher LOD to lower LOD. In the inverse lifting scheme 804B, the lifting operations are iteratively applied (e.g., executed) to the displacement signals of vertices from lower LOD to higher LOD.

[0118] and Figure 8A Compared to the improvement plan described in [the document], Figure 8B The enhancement scheme illustrates update weights 820 to 826 corresponding to the respective LODs 810 to 816. In other words, each LOD can have a corresponding update weight determined at least based on that LOD (e.g., the index of the LOD). In some examples, lower LODs can be associated with smaller (e.g., lower) update weights, and higher LODs can be associated with larger (e.g., higher) update weights.

[0119] In some examples, update weights 820 to 826 can be determined for LOD 810 to 816 based on the index of the relative resolution of the details across LOD 810 to 816. For example, update weights 820 to 826 can be determined based on LOD 810 to 816. N 810 (e.g., LOD) N The index of 810 is used to determine the LOD. N The update weight 820 is applied in the update operation corresponding to the boost operation of 810. Similarly, it can be based on LOD. N-1 The index 812 is used to determine the update weight 822; this can be based on LOD. N-2 The index 814 determines the update weight 824; and the update weight 826 can be determined based on the index LOD 826. Since the forward boosting scheme 802B and the reverse boosting scheme 804B can mutually inversely / equally and independently determine the update weights 820 to 826 based on LODs 810 to 816, the update weights 820 to 826 do not need to be signaled in the bitstream from encoder to decoder to enable adaptive weight updates.

[0120] In some implementations, a first indication (e.g., mode indication, flag, or syntax element) indicating whether adaptive update weights are enabled (e.g., to be applied) in the boosting scheme (e.g., in inverse boosting scheme 804B) can be signaled in the bitstream. If the first indication indicates that adaptive update weights are disabled (e.g., not enabled or not applied), the same update weights can be used in boosting operations corresponding to multiple LODs of vertices of the 3D mesh (e.g., a mesh frame). In this example, each update weight in update weights 820 to 826 can have the same update value independent of the corresponding LODs 810 to 816. In some examples, the encoder can base its performance on whether using uniform update weights (e.g., disabling the first indication of adaptive update weights) or adaptive update weights (e.g., enabling the first indication of adaptive update weights) produces higher compression performance (e.g., in...). Figure 2A and / or Figure 2B (A smaller number of bits are generated in the displacement bit stream 260). The decoder can obtain (e.g., receive and / or decode) a first indication from the bit stream and determine whether the inverse boosting scheme 804B should be applied based on the first indication. In some examples, the first indication can be signaled for each 3D mesh sequence, each mesh frame, each submesh, each patch, each patch group, each LOD, etc.

[0121] In some implementations, when applying adaptive weight updates, it can be based on the scaling factor. S The first update weight is adjusted according to the LOD associated with the update weight. U (For example, the initial / default update weights) are used to determine each update weight in update weights 820 to 826. In some examples, the weights used for LOD can be determined according to the following relationship. i Update weights U i : ,in U It is the initial update weight, and n This could be the number of Levels of Exploration (LOD). Therefore, the weights are updated. U i This can be used to determine the scaling value based on the difference between the total number of LODs and the index indicating the LOD. For example, for a dataset including four LODs (e.g., ... n = 4) of the vertices in the grid frame, with a scaling factor of ½ (e.g. S = ½) and the first update weight 1 / 8 (e.g. U = 1 / 8), then the update weights for each of LOD3, LOD2, and LOD1 can be calculated to be 1 / 8, 1 / 16, and 1 / 32, respectively. Therefore, the update weights for each lower LOD can be determined based on scaling (e.g., multiplying) the previous update weights used for the next higher LOD by this scaling factor.

[0122] In some implementations, a second indication (e.g., a flag or syntax element) can be signaled in the bitstream to indicate a scaling factor used to determine (e.g., derive or compute) the update weights corresponding to the LOD. For example, the second indication could indicate an index of a set of scaling factors to specify a scaling factor among the scaling factors. In some examples, the second indication can be signaled for each 3D mesh sequence, each mesh frame, each submesh, each patch, each patch group, each LOD, etc.

[0123] Figure 9A and Figure 9B The text describes each iteration of the improvement scheme, as described above. Figure 8B More detailed descriptions are available in the text.

[0124] Figure 9A An example forward lifting scheme for transforming the displacement of a 3D mesh (e.g., a mesh frame of a 3D mesh) into wavelet coefficients is illustrated according to some embodiments. As explained above, the forward lifting scheme may include multiple lifting operations that iteratively perform multiple instances corresponding to multiple LODs of the 3D mesh frame. Each lifting operation may correspond to an operation performed in lifting operator 901. For example, lifting operator 901A may be applied to an input signal 942 corresponding to a displacement (e.g., a displacement value determined by an encoder). Splitting operator 940 may determine an odd signal 952 and a corresponding even signal 954 for predicting the odd signal 952. For example, the odd signal 952 may correspond to a displacement value that is the first LOD (e.g., LOD) in the LODs associated with the displacement at a vertex of the 3D mesh. N The splitting operator 940 can determine an even signal 954 comprising two displacements, which are associated with one or more lower LODs (e.g., LOD0 to LOD) on the same edge as the vertex. N-1 Two corresponding vertices of a vertex are paired. For example, these two vertices could be the closest vertices that sandwich a vertex on an edge and come from one or more lower LODs and were used to generate the vertex. In some examples, the splitting operator 940 can determine the edges of a vertex based on the subdivided mesh, and then determine the two vertices on the same edge, such as the two vertices that form the edge.

[0125] Prediction filter 960 (also referred to as a prediction step or prediction operation) can generate a displacement predictor for odd signal 952 based on even signal 954 and, in some examples, on prediction weights. For example, prediction filter 960 can determine the displacement predictor as the average of two even signals represented by even signal 954 for odd signal 952 (e.g., when the prediction weight is half). Prediction filter 960 can convert odd signal 952 (e.g., displacement at the vertex) into wavelet coefficients corresponding to prediction error signal 962. For example, prediction error signal 962 can be determined as the difference between odd signal 952 and the displacement predictor. Therefore, prediction filter 960 can replace odd signal 952 with the difference between odd signal 952 (e.g., the original value) and its prediction. Thus, lift operator 901A can update (e.g., replace) the displacement signal in situ without storing the updated signal separately.

[0126] The update filter 970 can update the even signal 954 (e.g., the displacement signals corresponding to vertices v1 and v2, represented by wavelet coefficients) using the prediction error signal 962 based on the update weights. The even signal 954 can be transformed (e.g., replaced) using the updated prediction signal 972. In some examples, when uniform update weights are applied (e.g., enabled or selected), the update weights can be predetermined values, such as 1 / 2, 1 / 4, 1 / 8, or 1 / 16. In some examples, when uniform update weights are applied, the encoder in the bitstream can signal the value of the update weights to the decoder.

[0127] In some implementations, the update weights can be adjusted (e.g., adaptively) based on the LOD corresponding to the boosting operator in which the update weights are used. (See above regarding...) Figure 8B As explained, the update weights for the even signal 954 can be determined based on the LOD associated with the vertices corresponding to the odd signal 952 (e.g., by the displacement signal corresponding to vertex v (e.g., represented by wavelet coefficients)).

[0128] For example, the first update weight (e.g., " updateWeight The parameter (which can be a default value or a fixed value) can be determined by a scaling value (e.g., " adaptive_ratio Scaling the value. For example, scaling the value (e.g., ""). adaptive_ratio The value can be a scaling factor (e.g., "). UpdateWeightScale The power of the parameter, which has a power related to the total number of LODs (e.g., the count / number of LODs associated with the 3D mesh, " lod_count "parameters") and the index of the LOD corresponding to the lifting operator (e.g., current LOD level, " lod_current The index associated with the parameter. For example, if the scaling factor (e.g., "...") UpdateWeightScaleThe parameter is equal to 1 / 2 and the LOD (e.g., " lod_count The total number of parameters is 4, and the current LOD (e.g., " lod_current The difference between the parameters (3) and 1 is equal to 1 if the first updated weight (e.g., " updateWeight If the parameter is equal to 1 / 8, then the first update weight (e.g., "") used for update operations at the LOD level (e.g., the index indicating LOD2) is 1 / 8. updateWeight The scaling factor (e.g., "parameter") can be adjusted (e.g., updated or adapted) to 1 / 16. Therefore, the greater the difference between the total number of LODs and the current LOD, the more important the scaling factor (e.g., "parameter") becomes. UpdateWeightScale The greater the impact of the scaling factor on the update weights in the update filtering (e.g., update step or update operation), the greater the effect. Since the scaling factor can be less than 1, this means that the larger the difference, the smaller the adjusted update weights will become. Therefore, in some implementations, the update weights for the even signal 954 can be determined based on the index of the LOD indicating the odd signal 952 or the index of that LOD. For example, the update weights can include a scaling value (e.g., a ratio value) determined based on the difference between the total number of LODs and the current LOD indicated by the index.

[0129] Example pseudocode for the update operation in lift operator 902 corresponding to the inverse lift scheme 804B is shown below:

[0130] As shown above, it is used to update the even signal. s evenk The updated weight 820 can be based on " updateWeight "and" adaptive_ratio As mentioned above... Figure 8B and Figure 9B As explained, in the reverse promotion scheme (e.g., with that used for vertices) v The prediction error corresponding to the displacement signal[v] d k It can be adjusted by the scaling value, and then from the even signal / sample 954 s evenk (e.g., representing a vertex) v1 and v2 The corresponding displacement signal is subtracted from each even-numbered signal / sample in signal[v1] and signal[v2]).

[0131] The pseudocode for the lift operator 901 corresponding to the forward lift scheme 802A could be similar. However, as mentioned above... Figure 8A to Figure 8B As explained, signal[v] can be a signal representing the difference between the odd signal 952 and the shift predictor. dk And instead of subtracting the scaling prediction error d , will scale the prediction error d Each even signal is added to even signal 954 to adjust each even signal in even signal 954.

[0132] In some examples, the adaptive update weights can be further adjusted (e.g., increased or decreased) based on the geometric distance between two vertices. For instance, for example, for a sample value with ¼ of its displacement value to be updated with the error signal to prepare it for update weights in LOD 3 for the next prediction step, the update value can be reduced to ½ to 1 / 8 of its value, thereby reducing the impact of the subsequent error signal computed in the next LOD 2 level.

[0133] Therefore, the update filter 970 can replace the even signal 952 with an updated even signal corresponding to the updated prediction signal 972. The updated prediction signal 972 may include a scaled prediction error signal 962 (e.g., in the pseudocode above). d The sum of the value and the corresponding even-numbered signal 954. Therefore, the lift operator 901A can update (e.g., replace) the displacement signal in situ without having to store the updated signal separately.

[0134] like Figure 9A As shown, lifting operations 901A to 901B iterate from signal samples (e.g., displacement signals and corresponding wavelet coefficient representations) in a higher LOD to a lower LOD. In each iteration, lifting operator 901 takes the signal processed at a higher LOD in the previous lifting operation and splits (e.g., separates) that signal into signals corresponding to the lower LOD to generate a predicted and updated signal. Lifting operator 901 is performed iteratively for each lower LOD until the lowest LOD level is processed, at which point all displacement signals (e.g., input signal 942) will have been transformed into wavelet coefficients. For example, a base grid of 900 vertices can be subdivided into an upsampled grid of 57,600 vertices, for example, spanning four LOD levels (e.g., LOD0 including vertices with indices 1 to 900, LOD1 including vertices with indices 901 to 3600, LOD2 including vertices with indices 3601 to 14400, and LOD3 including vertices with indices 14401 to 57600). The associated displacements (e.g., displacement values / signals) have the same order as these vertices. Lifting operators 901A through 901B can iterate from the highest LOD (LOD 3 in this example) in lifting operator 901A. Subsequently, the lifting operation is iteratively performed up to, for example, lifting operator 901B, etc., until all signals across all LODs have been processed.

[0135] Figure 9BExamples of inverse lifting schemes for transforming wavelet coefficients into a 3D mesh displacement, according to some implementation schemes, are illustrated. For example, the inverse lifting operators 900A to 900B of the inverse lifting scheme can... Figure 9A The operation of the lifting scheme described herein is reversed. For example, instead of iterating from a higher LOD to a lower LOD in the forward lifting scheme, the lower LOD in the inverse lifting scheme is processed before the higher LOD. The previously processed wavelet coefficients can be input into the inverse lifting operator 900B from the previous iteration of the inverse lifting scheme (e.g., from the inverse lifting operator 900A) as the reconstructed updated signal 932 and the reconstructed error signal 922.

[0136] For example, for the wavelet coefficients (represented by the reconstructed error signal 922) of a vertex at LOD, a reconstructed updated signal 932 corresponding to two vertices as determined by the forward lifting scheme can be determined. The update filter 930 can determine the reconstructed even signal 914 based on the reconstructed error signal 922 and update weights (e.g., the same update weights applied by the update filter 970 in the lifting operator 901A of the forward lifting scheme). In some embodiments, the decoder can obtain (e.g., receive and / or decode) an indication of whether adaptive update weights are enabled (or disabled). Based on the indication that adaptive update weights are not enabled (or disabled), the update weights can be a uniform value of the operations used to lift all LODs associated with the vertices of the 3D mesh.

[0137] In some implementations, when adaptive update weights are used (e.g., based on an indication that adaptive update weights are enabled or as a default mode), update filter 930 can determine update weights (e.g., adjusted update weights) based on the LOD associated with the vertex corresponding to the reconstructed error signal 922. Furthermore, prediction filter 920 can determine a displacement predictor based on the prediction weights and the updated signal from update filter 930 (e.g., the reconstructed even signal 914). Subsequently, prediction filter 920 can combine (e.g., sum) the displacement predictor and the reconstructed error signal 922 to determine the reconstructed odd signal 912.

[0138] For example, for the reconstructed error signal 922 D1 corresponding to the first vertex, two reconstructed signals 932 E corresponding to the second and third vertices can be determined respectively. u1 and E u2As explained above, similarly performed by the encoder, the decoder can also apply multiple iterative subdivision schemes to the decoded base grid (e.g., the reconstructed base grid) to determine a subdivided grid, which includes vertices at multiple levels of digits (LODs) corresponding to the multiple iterations. For example, each successive iteration of the subdivision scheme can generate vertices at the next lower LOD. Thus, for a first vertex from a first LOD, the decoder can determine a second and third vertex from a lower LOD on the same edge as the first vertex. For example, the second and third vertices can be vertices from LODs lower than the first LOD, which are closest to the first vertex on the same edge as the first vertex. For example, the second and third vertices can form an edge associated with the first vertex. Subsequently, the update filter 930 can be based on the reconstructed signal 932 E. u1 and E u2 The reconstructed even signals 914 E1 and E2 are generated as follows: E1 = E u1 – w u D1 and E2 = E u2 – w u D1, where w u It updates the weights. (As mentioned above...) Figure 9A As explained, the updated weights w can be adaptively determined based on the LOD of the first vertex corresponding to the reconstructed error signal 922. u For example, the updated weight w can be determined based on a scaling value (e.g., a ratio). u The scaling value is determined based on the index of the Level of Detail (LOD). For example, the scaling value can be indicated based on the difference between the total number of LODs and their indices. Therefore, the update filter 930 can replace (e.g., updated from a previous iteration such as the inverse lift operator 900A) the reconstructed updated signal 932 with the reconstructed even signal 914. Thus, the inverse lift operator 900B can update (e.g., replace) the displacement signal in situ without needing to store the updated signal separately.

[0139] The prediction filter 920 can determine the prediction P1 for the displacement signal corresponding to the first vertex as follows: P1 = w p (E1 + E2), where w p It updates the weights. For example, w p It can be set to half, such that prediction P1 represents the average of the two reconstructed signals 914 E1 and E2 after the update operation of update filter 930, for example, as regarding Figure 8BThis is explained by the inverse boosting scheme 804B. Finally, the reconstructed odd signal 912 O1 can be determined based on the prediction P1 and the prediction error P1 (e.g., their sum) as follows: O1 = D1 + P1. Therefore, the prediction filter 920 can replace the reconstructed error signal 922 (e.g., the displacement signal corresponding to or representing the odd signal) with the reconstructed odd signal 912. For example, the prediction filter 920 can determine the reconstructed odd signal 912 based on a linear combination of the reconstructed even signal 914, for example, by averaging the two signals of the reconstructed even signal 914.

[0140] The merging operator 910 can sort the reconstructed odd signal 912 and the reconstructed even signal 914, so as to further process them with the next higher LOD corresponding to the next inverse lifting operator 900.

[0141] Figure 10 Figure 1000 illustrates an example of iteratively performing an inverse lifting scheme for each LOD of vertices in a 3D mesh (e.g., mesh frames in a mesh frame sequence) according to some implementation schemes. For example, the vertices of a mesh frame may include corresponding displacement signals 1030, 1020, 1010, 1022, and 1032 (e.g., displacement values ​​or corresponding wavelet coefficient representations). As explained above, these displacement values ​​can be associated with the LOD of the corresponding vertex. Vertices at each higher LOD can be generated by iteratively applying a subdivision scheme, as per [the relevant context]. Figure 6 This is explained below. For example, displacement signals 1030 and 1032 can correspond to vertices at LOD0, displacement signal 1010 can correspond to vertices at LOD1, and displacement signals 1020 and 1022 can correspond to vertices at LOD2. The displacement signals can be ordered from lower LOD to higher LOD (e.g., as shown in array 1002). For example, the displacement signals can be ordered (e.g., arranged) and packaged in a 2D image, as described above regarding... Figure 7B As described.

[0142] As shown in Figure 1000, the inverse lifting scheme includes multiple iterative lifting operations, and is iterated for each LOD level 1004 until each LOD in the LOD has been processed in the corresponding lifting operation. For each vertex in the LOD, the lifting operation iterates across all displacement signals 1006 (e.g., wavelet coefficient signals / samples) at the vertices of the LOD. For example, for LOD2, the inverse lifting operator 900 can be applied to the displacement signals at all vertices in LOD2. For example, for the odd signal 1012 of the displacement signal 1022 in LOD2, the even signal 1014 corresponding to the displacement signals 1010 and 1032 in lower LODs can be determined and processed.

[0143] As shown in Figure 1000, after all wavelet coefficient signals / samples have been processed by the inverse lifting transform, samples from the next LOD level are processed until all LODs have been processed by the inverse lifting transform scheme.

[0144] Figure 11 A flowchart 1100 illustrates a method for executing a forward lifting scheme according to some implementation schemes. (The above refers to...) Figure 8A and Figure 9A The operation of flowchart 1100 is described in more detail. Encoder (e.g.) Figure 2A to Figure 2B The encoders 200A to 200B receive displacement signals associated with a Level of Detail (LOD) level (e.g., an index indicating the LOD). For example, the displacement signal could be an "odd" number of samples, such as... Figure 9A The odd-numbered signal 952. The encoder determines an edge consisting of two vertices for the vertex associated with the received displacement signal. As explained above, an edge can be formed by two vertices used to generate vertices during subdivision iterations during the generation of the subdivision mesh. The encoder (e.g. Figure 9A The prediction filter (960) uses prediction weights and the two displacements (i.e., the two vertices) of the determined edge vertices to determine the displacement prediction for the displacement signal. For example, the two displacements could be... Figure 9A The even-numbered signals 954 correspond to the "even-numbered" samples. The encoder, for example, determines the prediction error of the displacement signal based on the difference between the displacement signal and the displacement prediction (e.g., Figure 9A The prediction error signal 962). Encoder (e.g. Figure 9A The update filter 970 uses adaptive update weights based on LOD level information and the determined prediction error to determine the updated displacements for the two displacements of the two vertices (e.g., Figure 9A The updated prediction signal (972). For example, LOD level information could be an index indicating the LOD level of the vertices associated with the displacement signal. The encoder determines whether an iterative forward lifting operation has been completed for the displacement signal at the LOD level. If not, a forward lifting operation (1102) is performed for the remaining displacement signals at the LOD level. Otherwise, the encoder can determine whether an iterative forward lifting operation has been completed for all LOD levels of the 3D mesh. If there are remaining LOD levels, a forward lifting operation (1102) can be performed for the displacement signal at the next LOD level. Once all LOD levels have been processed, the displacement signal of the 3D mesh has been converted into a wavelet coefficient representation that can be compressed more efficiently.

[0145] Figure 12 A flowchart 1200 illustrates a method for executing a reverse lifting scheme according to some implementation schemes. (The above refers to...) Figure 8B and Figure 9BThe operation of flowchart 1200 is described in more detail. Decoder (e.g.) Figure 3 The decoder 300 receives a displacement signal of the transform associated with the LOD level (e.g., an index indicating the LOD). (See also: Regarding...) Figure 8B and Figure 9B The received displacement signal can be interpreted as representing the prediction error determined at the encoder. For example, the displacement signal can be an "odd" number of samples, such as... Figure 9B The reconstructed error signal 922. The decoder determines an edge consisting of two vertices for the vertex associated with the received displacement signal. As explained above, an edge can be formed by two vertices used to generate vertices during the subdivision iteration during the generation of the subdivision mesh. The decoder (e.g. Figure 9B The update filter 930 uses adaptive update weights based on LOD level information and the received displacement signal to determine the updated displacements for the two displacements of the two vertices (e.g., Figure 9B The reconstructed even signal 914). For example, the two displacements can be... Figure 9B The reconstructed updated signal 932 corresponds to the "even" number of samples. For example, LOD level information could be an index indicating the LOD level of the vertices associated with the transformed displacement signal. Decoder (e.g.) Figure 9B The prediction filter 920 uses prediction weights and two updated shifts (e.g., Figure 9B The reconstructed even-numbered signal (914) is used to determine the displacement prediction for the displacement signal. The decoder updates the displacement signal (e.g., based on the displacement prediction) according to the displacement prediction. Figure 9B The reconstructed error signal 922). For example, the updated displacement signal (e.g., the reconstructed odd signal 912) can represent the reconstructed displacement resulting from adding the received transformed displacement signal to the displacement prediction. The decoder determines whether the iterative inverse lifting operation has been completed for the displacement signal at the LOD level of the vertices. If not, the inverse lifting operation 1202 is performed for the remaining displacement signals at the LOD levels. Otherwise, the decoder determines whether the iterative lifting operation has been completed for all LOD levels of the 3D mesh. If there are remaining LOD levels, the inverse lifting operation 1202 can be performed for the displacement signal at the next LOD level. Once all LOD levels have been processed, the displacement signal of the 3D mesh has been reconstructed based on the wavelet coefficient representation received from the bitstream.

[0146] Figure 13 A more detailed process 1300 for performing a forward lifting scheme according to some embodiments is shown. Process 1300 can be performed by, for example... Figure 2A to Figure 2B The encoders 200A to 200B perform the operation. As shown, the forward lifting scheme includes a wavelet transform performed for each displacement of the 3D mesh at each LOD level of the 3D mesh.

[0147] The encoder can target LOD level N The displacement of the vertices (e.g., "odd" samples) is used to perform a split operation to obtain (e.g., find): by the vertices in lower LOD 0~( N The corresponding edge formed by the two vertices of -1), and the two displacements (e.g., "even" samples) from the input signal corresponding to the two vertices.

[0148] The encoder can perform a prediction operation to replace odd sample values ​​(i.e., shift values) with the difference between the (original) odd sample values ​​and the predicted values ​​used for the odd sample values. For example, as regarding Figure 9A As explained by prediction filter 960, the predicted value can be based on (e.g., equal to) the sum of two even-numbered samples multiplied by a constant prediction weight value α (e.g., α = ½). Therefore, odd-numbered samples can be transformed into a prediction with the prediction error (such as...) Figure 9A The new / updated odd sample corresponding to the prediction error signal 962).

[0149] The encoder can perform update operations to replace even-numbered sample values ​​with updated / new even-numbered sample values. For example, each even-numbered sample can be replaced by an update value, which is the sum of the even-numbered sample value and (the new / updated odd-numbered sample value × the adaptive update weight β). The adaptive update weight β can be based on LOD parameters, such as the LOD level. N The index. For example, for LOD3, β could be 1 / 8, for LOD2, β could be 1 / 16, and so on.

[0150] As mentioned above Figure 8A and Figure 9A The explanation is that LOD displacement signals / samples can be processed for each LOD, with the current LOD level being processed first. N The displacement of all vertices, then at the next LOD level. N- The process iterates over the displacements of the vertices at level 1. Once the displacement signals across all vertices of the LOD have been processed, the input displacements will have been transformed into wavelet coefficients encoded in the bitstream.

[0151] Figure 14 A more detailed process 1400 for performing a reverse lifting scheme, according to some embodiments, is shown. Process 1400 can be performed by a decoder (such as...) Figure 3 The decoder 300 performs the inverse lifting scheme. As shown, the inverse lifting scheme includes performing an inverse wavelet transform for each displacement of the 3D mesh at each LOD level in the LOD levels of the 3D mesh. The decoder receives wavelet coefficients representing the displacements of the vertices of the 3D mesh from the bit stream.

[0152] For those at LOD level NThe wavelet coefficients of the vertices (e.g., "odd" samples), obtained by the decoder (e.g., found): by the wavelet coefficients of the vertices at lower LOD 0~( N-1 The corresponding edge formed by the two vertices of the input wavelet coefficients, and the two displacements corresponding to the two vertices (e.g., “even” samples).

[0153] The decoder can perform update operations to replace even-numbered sample values ​​with updated / new even-numbered sample values. For example, each even-numbered sample can be replaced by an update value based on the even-numbered sample and the odd-numbered sample multiplied by an adaptive update weight β. For instance, the update value could be the difference between the even-numbered sample and (odd-numbered sample × adaptive update weight). The adaptive update weight β can be based on LOD parameters, such as the LOD level. N The index. For example, for LOD 3, β could be 1 / 8, for LOD 2, β could be 1 / 16, and so on.

[0154] The decoder can perform a prediction operation to replace odd sample values ​​(i.e., represented by wavelet coefficients) with the sum of the odd sample values ​​and the predicted values ​​used for the odd sample values. For example, as regarding Figure 9B As explained by prediction filter 920, the predicted value can be based on (e.g., equal to) the sum of two even-numbered samples multiplied by a constant prediction weight value α (e.g., α = ½). Therefore, odd-numbered samples can be transformed into... Figure 9B The reconstructed odd signal 912 corresponds to the new / updated odd sample.

[0155] The decoder can perform a merging operation, making it possible to use LOD (Level of Detail) N The current reverse boosting operation, where updated odd-numbered samples and updated even-numbered samples are sorted for reconstruction to be in the next LOD. N+ The displacement value of the vertex of 1.

[0156] As mentioned above Figure 8B and Figure 9B The explanation is that LOD displacement signals / samples can be processed for each LOD, with the current LOD level being processed first. N The displacements (or wavelet coefficient representations) of all vertices are then applied at the next LOD level. N+ The process iterates over the displacements of the vertices at level 1. Once the displacement signals across all vertices of the LOD have been processed, the input wavelet coefficients are inversely transformed into displacement values ​​for reconstructing the 3D mesh. For example, the vertices of a subdivided mesh can be combined with their corresponding determined displacement values ​​to reconstruct the 3D mesh.

[0157] Figure 15A flowchart illustrating method 1500 for performing an inverse lifting scheme for decoding the displacement of a 3D mesh, according to some embodiments, is provided. In some examples, method 1500 may be performed by a decoder (e.g., Figure 3 The decoder 300) performs the operation. The following descriptions of the various steps can be directed by... Figure 3 The operation performed by the inverse wavelet transform 314, and / or may refer to the operation performed by the inverse wavelet transform 314. Figure 8B , Figure 9B , Figure 10 , Figure 12 and / or Figure 14 The operation of the described process.

[0158] At box 1502, the decoder decodes from the bit stream the first wavelet coefficients representing the first displacement of the first vertex of the first vertex at multiple levels of detail (LOD) of the three-dimensional (3D) mesh.

[0159] At box 1504, the decoder applies an inverse lifting wavelet transform to the first wavelet coefficients to determine (e.g., reconstruct) the first displacement. Applying the inverse lifting wavelet transform can include the operations in boxes 1506 through 1512.

[0160] In some implementations, box 1506 to 1512 can be executed for each wavelet coefficient corresponding to the corresponding vertex of the LOD in the first vertex that is in one of multiple LODs. (See above regarding...) Figure 8B , Figure 9B , Figure 10 , Figure 12 and Figure 14 This explains that an inverse lifting wavelet transform can be performed on the wavelet coefficients of each of the multiple levels of time (LOD). Specifically, an inverse lifting wavelet transform can be performed from a lower LOD to a higher LOD.

[0161] At box 1506, the decoder determines second wavelet coefficients corresponding to the second vertex in the first set of vertices, based on the first wavelet coefficients and the wavelet coefficients within the first set of vertices. These second wavelet coefficients form edges associated with the vertices corresponding to the wavelet coefficients. The second vertex is located at one or more LODs below the first LOD of the vertex corresponding to the wavelet coefficient. In some examples, the second vertex is the two vertices that form the edge.

[0162] In some examples, as explained above, during the subdivision process used to subdivide the base mesh of a 3D mesh, vertices can be generated based on the edges formed by the second vertex. Therefore, vertices can be on edges.

[0163] At box 1508, the decoder determines the update weights for updating the second wavelet coefficients based on the first LOD of the vertex.

[0164] At box 1510, the decoder updates the second wavelet coefficients based on the wavelet coefficients and update weights.

[0165] At box 1512, the decoder converts the wavelet coefficients of the vertex into the displacement in the first displacement based on the wavelet coefficients and the displacement predictor determined according to the updated second wavelet coefficients.

[0166] In some examples, the first wavelet coefficients are ordered according to their LOD (Level of Detail). Referring to box 1506, the decoder can apply the inverse lifting wavelet transform sequentially to wavelet coefficients with lower LODs, preceding those with higher LODs.

[0167] In some examples, the decoder receives from the bitstream a first indication as to whether the updated weights for the inverse lifting wavelet transform are adjusted according to the LOD. In some examples, at box 1510, the second wavelet coefficients are updated based on the wavelet coefficients and the updated weights in response to the first indication.

[0168] In some examples, reference box 1508 can determine update weights based on the index of the vertex's first level of descent (LOD). For instance, a lower LOD can be associated with a lower update weight.

[0169] In some examples, the decoder receives a second indication of a scaling factor used to adapt the updated weights in the inverse lifting wavelet transform. In some examples, the scaling factor can be a ratio between 0 and 1 (e.g., ½, ¼, 1 / 8). (See above regarding...) Figure 8B and Figure 9B As explained, the scaling factor can be determined based on the index of the first LOD. For example, the scaling factor can be determined based on the difference between the total number of LODs and the index of the first LOD.

[0170] In some examples, determining the update weights involves determining the power of the scaling factor, where the exponent of the power is based on the level of distance (LOD).

[0171] In some examples, the second instruction includes an index to the list of scaling factors.

[0172] In some examples, the second indicator further indicates the scaling function applied to the scaling factor.

[0173] In some examples, reference box 1510, each of the second wavelet coefficients in the second wavelet coefficients can be updated based on adjusting / scaling the wavelet coefficients with updated weights. For example, each of the second wavelet coefficients in the second wavelet coefficients can be updated by subtracting the adjusted wavelet coefficient from each of the second wavelet coefficients in the second wavelet coefficients.

[0174] In some examples, the first instruction is signaled for the following: a 3D mesh sequence of a 3D mesh, a 3D mesh, a LOD level, or a submesh of a 3D mesh.

[0175] In some examples, the decoder reconstructs the geometry of the 3D mesh based on a first displacement. In other examples, the decoder decodes the underlying mesh associated with the 3D mesh and iteratively applies a subdivision scheme to the underlying mesh to generate the vertices of the subdivided underlying mesh. Each LOD in the LOD is associated with a subdivision iteration. For example, reconstructing the geometry involves adding the first displacement to the corresponding vertices of the subdivided underlying mesh.

[0176] In some examples, a higher LOD is associated with a higher number of subdivision iterations. In other examples, each LOD is associated with a different update weight.

[0177] In some examples, decoding the first wavelet coefficients includes: decoding an image containing the first wavelet coefficients from a bitstream, and determining the first wavelet coefficients based on the packet order of the wavelet coefficients in the decoded image, as described above. Figure 7A to Figure 7B As described.

[0178] In some examples, the decoder inverse-quantizes the first wavelet coefficients before performing the inverse lifting wavelet transform, such that the inverse lifting wavelet transform is applied to the inverse-quantized first wavelet coefficients.

[0179] In some examples, each wavelet coefficient is inversely quantized using a quantized value based on the LOD associated with each wavelet coefficient in the first wavelet coefficients.

[0180] In some implementations, the encoder can perform the same actions as described above. Figure 15 The described inverse lifting transform is the inverse of the forward lifting transform. For example, an encoder can determine the first displacement of a first vertex in a three-dimensional (3D) mesh at multiple levels of detail (LODs). The encoder can apply a forward lifting wavelet transform to the first displacement to determine wavelet coefficients representing the first displacement. Applying the forward lifting wavelet transform can include displacements of vertices in the first vertex at LODs of multiple LODs: determining a second vertex based on the first vertex, thereby forming edges associated with the vertex. The second vertex is at one or more LODs below the first LOD. Edges may have been used to generate vertices during the subdivision process. Therefore, edges can include vertices. The encoder can determine displacement predictors for the displacement of the vertex based on the second displacement of the second vertex. The displacement of the vertex can be converted by the encoder into wavelet coefficients based on the difference between the displacement and the displacement predictor. The encoder can determine (e.g., similar to) the LOD of the vertex. Figure 15The encoder determines the update weights (of which the update weights are used to update the second displacement). The encoder can update the second displacement based on the wavelet coefficients and the update weights. Subsequently, the encoder can encode the wavelet coefficients in the bit stream.

[0181] The embodiments of this disclosure can be implemented in hardware using analog and / or digital circuitry, in software, by executing instructions by one or more general-purpose or special-purpose processors, or as a combination of hardware and software. Therefore, embodiments of this disclosure can be implemented in a computer system or other processing system environment. An example of such a computer system 1600 is... Figure 16 As shown in the figures above. The blocks depicted in the figures above (such as...) Figure 1 The blocks in the flowcharts described in this disclosure can be executed on one or more computer systems 1600. Furthermore, each step in the flowcharts described in this disclosure can be implemented on one or more computer systems 1600. When more than one computer system 1600 is used to implement embodiments of this disclosure, the computer systems 1600 can be interconnected by one or more networks to form a cluster of computer systems that can act as a single, seamless pool of resources. The interconnected computer systems 1600 can form a computer "cloud".

[0182] Computer system 1600 includes one or more processors, such as processor 1604. Processor 1604 may be, for example, a dedicated processor, a general-purpose processor, a microprocessor, or a digital signal processor. Processor 1604 may be connected to communication infrastructure 1602 (e.g., a bus or network). Computer system 1600 may also include main memory 1606, such as random access memory (RAM), and may also include secondary memory 1608.

[0183] Secondary storage 1608 may include, for example, a hard disk drive 1610 and / or a removable storage drive 1612, representing a magnetic tape drive, an optical disc drive, etc. The removable storage drive 1612 can read from and / or write to the removable storage unit 1616 in a well-known manner. The removable storage unit 1616 represents a magnetic tape, optical disc, etc., read from and written to by the removable storage drive 1612. Those skilled in the art will understand that the removable storage unit 1616 includes a computer-usable storage medium in which computer software and / or data are stored.

[0184] In alternative implementations, secondary storage 1608 may include other similar means for allowing computer programs or other instructions to be loaded into computer system 1600. Such means may include, for example, removable storage unit 1618 and interface 1614. Examples of such means may include program boxes and box interfaces (such as those found in video game devices), removable storage chips (such as EPROM or PROM) and associated sockets, thumb drives and USB ports, and other removable storage units 1618 and interfaces 1614 that allow software and data to be transferred from removable storage unit 1618 to computer system 1600.

[0185] Computer system 1600 may also include communication interface 1620. Communication interface 1620 allows software and data to be transferred between computer system 1600 and external devices. Examples of communication interface 1620 may include modems, network interfaces (such as Ethernet cards), communication ports, etc. Software and data transmitted via communication interface 1620 are in the form of signals, which may be electronic signals, electromagnetic signals, optical signals, or other signals that can be received by communication interface 1620. These signals are provided to communication interface 1620 via communication path 1622. Communication path 1622 carries signals and may be implemented using wires or cables, optical fibers, telephone lines, cellular telephone links, RF links, and other communication channels.

[0186] The computer system 1600 may also include one or more sensors 1624. Sensors 1624 can measure or detect one or more physical quantities and convert the measured or detected physical quantities into electrical signals in digital and / or analog form. For example, sensor 1624 may include an eye-tracking sensor to track a user's eye movements. Based on the user's eye movements, the display of the point cloud can be updated. In another example, sensor 1624 may include a head-tracking sensor to track a user's head movements. Based on the user's head movements, the display of the point cloud can be updated. In yet another example, sensor 1624 may include a camera sensor for taking photographs and / or a 3D scanning device, such as a laser scanning device, a structured light scanning device, and / or a modulated light scanning device. The 3D scanning device can acquire geometric information by moving one or more laser heads, structured light, and / or modulated light cameras relative to the scanned object or scene. The geometric information can be used to construct a point cloud.

[0187] As used herein, the terms "computer program medium" and "computer-readable medium" are used to refer to tangible storage media, such as removable storage units 1616 and 1618 or a hard disk installed in hard disk drive 1610. These computer program products are means for providing software to computer system 1600. The computer program (also referred to as computer control logic) may be stored in main memory 1606 and / or secondary storage 1608. The computer program may also be received via communication interface 1620. When executed, this computer program enables computer system 1600 to implement the disclosures discussed herein. Specifically, when executed, the computer program enables processor 1604 to implement the processes of this disclosure, such as any of the methods described herein. Thus, this computer program represents the controller of computer system 1600.

[0188] In another embodiment, the features of this disclosure can be implemented in hardware using, for example, hardware components such as application-specific integrated circuits (ASICs) and gate arrays. Implementing a hardware state machine to perform the functions described herein will also be apparent to those skilled in the art.

Claims

1. A method comprising: decoding, from a bitstream, first wavelet coefficients representative of a first displacement of a first vertex of a three-dimensional (3D) mesh; and applying an inverse lifting wavelet transform to the first wavelet coefficients to determine the first displacement, wherein the applying comprises: determining, from the first wavelet coefficients and for a wavelet coefficient of the first wavelet coefficients, a second wavelet coefficient corresponding to a second vertex of the first vertices, the second wavelet coefficient forming an edge associated with a vertex corresponding to the wavelet coefficient, wherein the second vertex is at one or more levels of detail (LODs) lower than the vertex; determining an update weight for updating the second wavelet coefficient based on the LODs; updating the second wavelet coefficient based on the wavelet coefficient and the update weight; and converting the wavelet coefficient of the vertex to a displacement of the first displacement based on the wavelet coefficient and a displacement predictor determined from the updated second wavelet coefficient.

2. The method of claim 1, wherein the first vertex is at a plurality of LODs including the LOD and the one or more LODs.

3. The method of claim 2, wherein each of the plurality of LODs is associated with a different update weight.

4. The method of claim 2, wherein a lower LOD of the plurality of LODs is associated with a lower update weight than an update weight of a higher LOD of the plurality of LODs.

5. The method of claim 2, wherein the first wavelet coefficients are ordered according to the plurality of LODs, and wherein the inverse lifting wavelet transform is applied to wavelet coefficients of vertices having lower LODs before wavelet coefficients of vertices having higher LODs.

6. The method of claim 1, wherein determining the update weight comprises determining a scaling value based on an index of the LOD.

7. The method of claim 6, wherein the scaling value is a ratio.

8. The method of claim 7, wherein the ratio is between 0 and 1.

9. The method of claim 6, wherein the update weight is determined based on a power of a scaling factor, wherein an exponent of the power is based on the LOD.

10. The method of claim 6, wherein the scaling value is further determined based on a difference between a total number of the plurality of LODs and the index of the LOD.

11. The method of claim 1, wherein the second vertex comprises two vertices and the second wavelet coefficients comprise two wavelet coefficients of the two vertices.

12. The method of claim 1, wherein updating the second wavelet coefficients comprises: subtracting the wavelet coefficient scaled by the update weight from each of the second wavelet coefficients.

13. The method of claim 1, further comprising receiving, from the bitstream, a first indication of whether update weights of the inverse lifting wavelet transform are adapted according to LODs.

14. The method of claim 13, wherein updating the second wavelet coefficients is based on the first indication.

15. The method of claim 13, wherein the first indication is obtained for a 3D mesh sequence including the 3D mesh, the 3D mesh, the LOD level, or a sub-mesh of the 3D mesh.

16. The method of claim 1, further comprising receiving a second indication of a scaling factor for adapting an update weight in the inverse lifting wavelet transform from the bitstream.

17. The method of claim 16, wherein determining the update weight comprises determining a scaling value based on a power of the scaling factor, wherein an exponent of the power is based on the LOD.

18. The method of claim 16, wherein the second indication comprises an index to a list of scaling factors.

19. The method of claim 1, further comprising reconstructing a geometry of the 3D mesh based on the first displacement.

20. The method of claim 19, further comprising: decoding a base mesh associated with the 3D mesh; and iteratively applying a subdivision scheme to the base mesh to generate vertices of a subdivided base mesh, wherein each of the LODs is associated with a subdivision iteration of the subdivision scheme.

21. The method of claim 20, wherein reconstructing the geometry comprises adding the first displacement to corresponding vertices of the subdivided base mesh.

22. The method of claim 20, wherein a higher LOD is associated with a higher iteration of the subdivision iterations.

23. The method of claim 1, wherein decoding the first wavelet coefficients comprises: decoding an image including the first wavelet coefficients from the bitstream; and determining the first wavelet coefficients from the decoded image according to a packing order of wavelet coefficients in the decoded image.

24. The method of claim 23, further comprising inverse quantizing the first wavelet coefficients, and wherein the inverse lifting wavelet transform is applied to the inverse quantized first wavelet coefficients.

25. The method of claim 24, wherein each of the first wavelet coefficients is inverse quantized using a quantization value based on a respective LOD associated with the wavelet coefficient.

26. A method comprising: determining a first displacement of a first vertex of a three-dimensional, 3D, mesh; and applying a forward lifting wavelet transform to the first displacement to determine a wavelet coefficient representing the first displacement, wherein applying the forward lifting wavelet transform comprises: determining a second displacement corresponding to a second vertex of the first vertex from the first displacement and for a displacement in the first displacement, the second displacement forming an edge associated with a vertex corresponding to the displacement, wherein the second vertex is at one or more levels of detail, LODs, lower than the vertex; converting the displacement of the vertex to a wavelet coefficient based on a difference between the displacement and a displacement predictor determined from the second displacement; and determining an update weight for updating the second displacement based on the LOD; and updating the second displacement based on the wavelet coefficient and the update weight.

27. The method of claim 26, wherein the first vertex is at a plurality of LODs including the LOD and the one or more LODs.

28. The method of claim 27, wherein the first displacement is ordered according to the plurality of LODs, and wherein the forward lifting wavelet transform is applied to displacements of vertices having a higher LOD before displacements of vertices having a lower LOD.

29. The method of claim 26, further comprising encoding the wavelet coefficients in a bitstream.

30. The method of claim 26, wherein determining the update weight comprises determining a scaling value based on an index of the LOD.

31. The method of claim 30, wherein the scaling value is a ratio.

32. The method of claim 31, wherein the ratio is between 0 and 1.

33. The method of claim 30, wherein the update weight is determined based on a power of a scaling factor, wherein an exponent of the power is based on the LOD.

34. The method of claim 30, wherein the scaling value is further determined based on a difference between a total number of the plurality of LODs and the index of the LOD.

35. The method of claim 26, wherein updating the second wavelet coefficients comprises: adding the wavelet coefficient scaled by the update weight to each of the second displacements.

36. An encoder comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the encoder to perform the method of any of claims 26-35.

37. An decoder comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the decoder to perform the method of any of claims 1-25.

38. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a device, cause the device to perform the method of any of claims 1-35. ​ ​

Images