Re-gridding for efficient compression
By re-meshing and wavelet transformation of 3D mesh, combined with grid extraction and fitting subdividing surface technology, the problem of low compression efficiency of 3D mesh in the existing technology is solved, and efficient grid coding and decoding is achieved Effect.
Patent Information
- Application Number
- CN202380026673.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-02-08
- Filing Date
- 2023-03-03
- Publication Date
- 2024-11-15
AI Technical Summary
The prior art is difficult to efficiently compress the connectivity information of 3D grids, especially in static and dynamic grids. Traditional video encoding technology cannot effectively encode geometric structures and attribute information, resulting in low compression efficiency.
A grid compression method based on image/video is adopted to re-mesh the 3D grid through the pre-processing stage to generate the basic grid and displacement field, combining grid extraction, fitting subdivided surfaces and wavelet transformation techniques. Optimize the encoding process to improve compression efficiency.
It realizes efficient compression of 3D mesh, reduces the number of bits required for encoding, improves compression efficiency and decoded quality, and adapts to different resolution and quality requirements.
Smart Images

Figure CN118974772A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This patent application claims the benefit of priority to the following U.S. Provisional Patent Applications, which are hereby incorporated by reference in their entirety: U.S. Provisional Application No. 63 / 269,211, filed on March 11, 2022, entitled “Image / Video Based Mesh Compression”; U.S. Provisional Application No. 63 / 269,213, filed on March 11, 2022, entitled “Remeshing for Efficient Compression”; U.S. Provisional Application No. 63 / 269,214, filed on March 11, 2022, entitled “Attribute Transfer for Efficient Dynamic Mesh Coding”; U.S. Provisional Application No. 63 / 269,215, filed on March 11, 2022, entitled “Motion Compression for Efficient Dynamic Mesh Coding”; U.S. Provisional Application No. 63 / 269,217, filed on March 11, 2022, entitled “Attribute Transfer for Efficient Dynamic Mesh Coding”; U.S. Provisional Application No. 63 / 269,218, filed on March 11, 2022, entitled “Adaptive Tessellation for Efficient Dynamic Mesh Encoding, Decoding, Processing, and Rendering”; and U.S. Provisional Application No. 63 / 368,793, filed on July 19, 2022, entitled “VDMC support in the V3C framework”. Background Art
[0003] Video-based solutions such as V3C have been successfully developed to efficiently compress 3D volumetric data such as point clouds (i.e., V3C / V-PCC) or 3DoF+ content (V3C / MIV). The V3C standard enables the compression of 3D data such as static and dynamic point clouds by combining existing video coding techniques and metadata using well-defined syntax structures and processing steps. Video coding techniques are used to compress 3D projection data on a 2D plane, such as geometry and properties, while metadata includes information on how to extract and reconstruct a 3D representation from those 2D projections. Figure 1 A block diagram of a V-PCC TMC2 encoder is shown. Summary of the invention
[0004] Disclosed herein are methods and apparatus for compressing static and dynamic meshes based on images / videos. Specifically, disclosed herein are preprocessing remeshing techniques that can improve the compression efficiency of 3D meshes. A remeshing preprocessor for remeshing a 3D texture mesh M(i) to generate a base mesh m(i) and a displacement field d(i) for input into a mesh encoder may include: a mesh extraction module, the mesh extraction module including processing hardware, the processing hardware reducing the number of vertices or faces of the input mesh M(i) or a mesh derived therefrom while substantially retaining the shape of the input mesh M(i), thereby generating an extracted mesh dm(i) and a projected mesh P(i); and a fitting subdivision surface module, the fitting subdivision surface module including processing hardware, the processing hardware processing the input mesh M(i), the extracted mesh dm(i) or a mesh derived therefrom, and the projected mesh P(i) to generate a base mesh m(i) and the displacement field d(i) for input into a mesh encoder.
[0005] The remeshing preprocessor may also include a duplicate vertex removal module, the duplicate vertex removal module including processing hardware, the processing hardware removing duplicate vertices from the input mesh M(i) to produce a mesh UM(i) with uniform vertices, the mesh UM(i) with uniform vertices being input into the mesh extraction module in place of the input mesh M(i). The remeshing preprocessor may also include a duplicate triangle removal module, the duplicate triangle removal module including processing hardware, the processing hardware receiving the extracted mesh dm(i) as input and processing it to remove triangles that reference the same vertex. The remeshing preprocessor may also include a small connected component removal module, the small connected component removal module including processing hardware, the processing hardware detecting and removing connected components having a number of vertices, a number of triangles, or an area below a determined threshold from the extracted mesh dm(i) or a mesh derived therefrom. The remeshing preprocessor may also include an atlas parameterization module, the atlas parameterization module including processing hardware, the processing hardware reducing the number of tiles of the extracted mesh dm(i) or a mesh derived therefrom.
[0006] The fitting subdivision surface module may also include a mesh subdivision module, which includes processing hardware, which receives the extracted mesh dm(i) or a mesh derived therefrom and subdivides its polygons to produce a subdivided mesh S(i). The fitting subdivision surface module may also include at least one mesh deformation module, which includes processing hardware, which receives the subdivided mesh S(i) and two or more of the following: a mesh derived from the subdivided mesh, the projected mesh P(i), the input mesh M(i), and generates a deformed mesh by moving the vertices of the subdivided mesh S(i) to match the shape of the input mesh M(i). The fitting subdivision surface module may also include a base mesh optimization module, which includes processing hardware, which receives the deformed mesh and the extracted mesh pm(i), and generates a base mesh m(i) by updating the position of pm(i) to minimize the distance between the subdivided version of pm(i) and the deformed mesh. The fitting subdivision surface module may further include a displacement calculation module, which receives the deformed mesh and the base mesh m(i) as input and calculates a displacement field d(i) as a difference between the deformed mesh and the base mesh m(i).
[0007] The at least one mesh deformation module may include: an initial mesh deformation module, the initial mesh deformation module including processing hardware, the processing hardware, for each initial 3D position Pos(v) corresponding to a vertex v of the subdivided mesh S(i), finding a nearest point H(v) on the surface of the projected mesh P(i) such that an angle between a normal N(v) and a normal of H(v) is below a user-defined threshold, and moving the vertex v to the nearest point H(v) to produce an initial deformed mesh F0(i); and an iterative mesh deformation module, the iterative mesh deformation module including processing hardware, the processing hardware, for each vertex v of the deformed mesh having a position Pos(v), finding a nearest point H(v) on the input mesh M(i) such that an angle between a normal vector associated with Pos(v) and H(v) is below a user-defined threshold, and moving the vertex v to a new position determined by:
[0008] Pos(v)+<H(v)-Pos(v),N(v)> *N(v)
[0009] in<H(v)-Pos(v),N(v)> is the dot product of two 3D vectors H(v)-Pos(v) and N(v), and where N(v) is the normal vector at Pos(v), resulting in a new deformed mesh, which becomes the final deformed mesh F(i) after a determined number of iterations.
[0010] A method for extracting an input mesh M(i) or a mesh UM(i) with uniform vertices derived therefrom by repeated vertex deletion to produce an extracted mesh dm(i) may include: reducing the number of vertices / faces of the input mesh M(i) while substantially retaining the shape of the original mesh; and tracking the mapping between the input mesh M(i) by projecting the deleted vertices onto the extracted mesh dm(i).
[0011] A method for deforming a subdivided mesh S(i) may include: for each initial 3D position Pos(v) corresponding to a vertex v of the subdivided mesh S(i), finding a nearest point H(v) on the surface of a projected mesh P(i) such that an angle between a normal N(v) and a normal of H(v) is below a user-defined threshold; moving the vertex v to the nearest point H(v) to produce a deformed mesh, which is initially an initial deformed mesh F0(i); and iteratively, for each vertex v of the deformed mesh having a position Pos(v), finding a nearest point H(v) on an input mesh M(i) such that an angle between normal vectors associated with Pos(v) and H(v) is below a user-defined threshold; and moving the vertex v to a new position determined by:
[0012] Pos(v)+<H(v)-Pos(v),N(v)> *N(v)
[0013] in<H(v)-Pos(v),N(v)> is the dot product of two 3D vectors H(v)-Pos(v) and N(v), and where N(v) is the normal vector at Pos(v), resulting in a new deformed mesh, which becomes the final deformed mesh F(i) after a determined number of iterations.
[0014] A method for time-consistent remeshing may include reusing a base mesh pm(j) associated with a reference mesh M(j) for a base mesh pm(i) associated with a mesh M(i), wherein pm(i) and pm(j) have the same connectivity. The method may also include applying a fit subdivision surface module if the input meshes M(i) and M(j) are time-coherent, the fit subdivision surface module comprising processing hardware that processes the input mesh M(i), a decimated mesh dm(j) or a mesh derived therefrom, and a projected mesh P(j) to produce a base mesh m(i) and a displacement field d(i) for input into a mesh encoder. Alternatively, if the input meshes M(i) and M(j) are not time-coherent, the method may further include: generating a mesh M'(j) which is a deformed version of M(j) having the same shape as M(i); and applying a fitting subdivision surface module, the fitting subdivision surface module comprising processing hardware that processes the mesh M'(j), the extracted mesh dm(j) or a mesh derived therefrom, and the projected mesh P(j) to produce a base mesh m(i) and the displacement field d(i) for input into a mesh encoder. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] This patent or patent application file contains at least one drawing executed in color. Copies of this patent or patent application publication with one or more color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
[0016] Figure 1 An exemplary V-PCC encoder block diagram is illustrated.
[0017] Figure 2 An example of a textured mesh is illustrated.
[0018] Figure 3 illustrates an example of a textured mesh stored in OBJ format.
[0019] Figure 4 A high-level block diagram illustrating the grid encoding process.
[0020] Figure 5 A high-level block diagram illustrating the trellis decoding process.
[0021] Figure 6The resampling process for a 2D curve is illustrated.
[0022] Figure 7 Tessellation and displacement for a 2D curve are illustrated.
[0023] Figure 8 The original mesh, the decimated mesh, and the deformed mesh are illustrated.
[0024] Fig. 9 The original (wireframe) mesh and the deformed (flat shaded) mesh are illustrated.
[0025] Fig.10 An intra encoder / encoding process is illustrated.
[0026] Fig.11 A midpoint subdivision scheme is illustrated.
[0027] Fig.12 A positive lifting transformation is illustrated.
[0028] Fig.13 The inverse lifting transform is illustrated.
[0029] Fig.14A An algorithm for calculating the local coordinate system is illustrated.
[0030] Fig. 14B An algorithm for quantizing wavelet coefficients is illustrated.
[0031] Fig.15 An algorithm for packing wavelet coefficients into a 2D image is illustrated.
[0032] Fig.16 An algorithm for computing the Morton order is illustrated.
[0033] Fig.17 An inter-frame encoder / encoding process is illustrated.
[0034] Fig.18 An intra decoder / decoding process is illustrated.
[0035] Fig.19 An inter-frame decoder / decoding process is illustrated.
[0036] Fig. 20 A block diagram of a remeshing system is illustrated.
[0037] Fig.21 An example of mesh extraction with tracking is illustrated.
[0038] Fig. 22 A mesh parameterization with a reduced number of tiles is illustrated.
[0039] Fig.23 An example of property transfer after remeshing is illustrated.
[0040] Fig.24 The attribute transfer process is illustrated.
[0041] Fig.25 An exemplary implementation of the attribute transfer process is illustrated.
[0042] Fig.26 Illustrate discontinuities on boundary edges.
[0043] Fig. 27 A process for seam edge discontinuity mitigation is illustrated.
[0044] Fig.28 An example of attribute filling is illustrated.
[0045] Fig.29 A block diagram of the proposed motion compression system is illustrated.
[0046] Fig.30 An example of CABAC-based encoding of prediction indexes and prediction attributes is illustrated.
[0047] Fig.31 An exemplary V3C extended V-grid bitstream system block diagram is illustrated.
[0048] Fig.32 The v-grid decoder framework block diagram is illustrated.
[0049] Fig.33 An exemplary input grid illustrating the grid normalization process.
[0050] Fig.34 An exemplary output of the grid normalization process is illustrated.
[0051] Fig.35 An exemplary subdivision of regions in a grid based on information from their corresponding tiles is illustrated.
[0052] Fig.36 An example of a simple base mesh is illustrated.
[0053] Fig.37 An example of an interpolation grid is illustrated.
[0054] Fig.38 The brightness plane of the geometric structure image is illustrated.
[0055] Fig.39 An example of a geometric structure image is illustrated.
[0056] Fig.40 An example of vertex indices in a subpart associated with a shard is illustrated.
[0057] Fig.41Demonstrates adjusting the global mesh resolution by changing the subdivision iteration count.
[0058] Fig.42 Rules for adaptively subdividing triangles are illustrated.
[0059] Fig.43 An implementation of a system for implementing video dynamic trellis coding (v-DMC) is illustrated. DETAILED DESCRIPTION
[0060] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of the drawings in the drawings of the present disclosure represent structures and devices in block diagram form for simplicity. For the sake of clarity, not all features of actual implementations are described in the present disclosure. In addition, the language used in the present disclosure is selected for readability and instructional purposes and has not been selected to describe or limit the disclosed subject matter. Instead, the attached claims are intended for this purpose.
[0061] Various embodiments of the disclosed concepts are illustrated by way of example, and are not limited to individual drawings, in which similar reference numerals indicate similar elements. For simplicity and clear illustration, reference numerals are repeated in different drawings to indicate corresponding or similar elements, where appropriate. In addition, many specific details are shown in order to provide a full understanding of the specific implementation described herein. In other cases, methods, procedures, and components are not described in detail to avoid blurring the related functions described. Mentioning "one", "a kind of" or "another" embodiment in this disclosure is not necessarily the same or different embodiments, and this means at least one. A given drawing may be used to illustrate more than one embodiment or more than one type of features of the present disclosure, and all elements in the drawings may not be required for a given embodiment or type. When provided in a given drawing, the reference numeral refers to the same element throughout several drawings, but it may not be repeated in each drawing. Unless otherwise specified, the drawings are not drawn to scale, and the proportions of certain components may be enlarged to better illustrate the details and features of the present disclosure.
[0062] Part 1: Image / Video Based Mesh Compression
[0063] Static / dynamic meshes can be represented as a set of 3D meshes M(0), M(1), M(2), ..., M(n). Each mesh M(i) can be defined by connectivity C(i), geometry G(i), texture coordinates T(i) and texture connectivity CT(i). Each mesh M(i) can be associated with one or more 2D images A(i,0), A(i,1), ..., A(i,D-1), also called attribute maps, describing a set of attributes associated with the mesh surface. An example of an attribute would be texture information (see Figures 2 to 3 ). A set of vertex attributes can also be associated with the vertices of the mesh, such as color, normal, transparency, etc.
[0064] While the geometry and attribute information can be mapped again to 2D images and compressed efficiently using video coding techniques, the connectivity information cannot be encoded efficiently using similar schemes. A dedicated decoding solution optimized for such information is needed. In the following section, we present an efficient framework for static / dynamic mesh compression.
[0065] Figure 4 and Figure 5 A high-level block diagram of the encoding process 400 and the decoding process 500 of the present invention are shown, respectively. The encoding process includes a preprocessor 403 that receives a static or dynamic grid M(i) and an attribute graph A(i). The preprocessor produces a base grid m(i) and a displacement d(i) that can be provided to an encoder 402, which produces a compressed bitstream b(i) therefrom. The encoder 402 can also receive the attribute graph A(i) directly. The feedback loop 401 allows the encoder 402 to guide the preprocessor 403 and to change its parameters to achieve the best possible compromise for encoding the bitstream b(i) according to various criteria, including but not limited to:
[0066] Rate distortion,
[0067] Encoding / decoding complexity,
[0068] Random access,
[0069] Reconstruction complexity,
[0070] Terminal capabilities,
[0071] Encoding / decoding power consumption, and / or
[0072] Network bandwidth and latency.
[0073] On the decoder side ( Figure 5), a compressed bitstream b(i) is received by a decoder 502, which decodes the bitstream to produce metadata (i) associated with the bitstream and the decoded grid, a decoded grid m'(i), a decoded displacement d'(i), and a decoded attribute map A'(i). Each of these outputs of the decoder 502 can be provided to a post-processor 503, which can perform various post-processing steps, such as adaptive pattern mosaicking. The post-processor 503 can produce a post-processed grid M"(i) and a post-processed attribute map A"(i), both of which correspond to the input grid M(i) and input attribute map A(i) provided to the encoder. (As will be appreciated, the output is different from the input due to the lossy nature of compression due to quantization and other encoding effects.) The application 501 consuming the content can provide feedback 501a to the decoder 502 to guide the decoding process, and provide feedback 501b to the post-processor 503. As just one example, based on the position of the dynamic mesh relative to the camera view volume, the decoder 502 and post-processor 503 may adaptively adjust the resolution / accuracy of the generated mesh M"(i) and / or its associated attribute map A"(i).
[0074] Preprocessing
[0075] Figure 6 An exemplary preprocessing scheme that may be applied by the preprocessor 403 is illustrated. To keep the illustration simple and clear, the illustrated example uses the case of 2D curves, but the same concept can be applied to an input static or dynamic 3D mesh M(i)=(C(i), G(i), T(i), TC(i)) to produce the above referenced Figure 4 The base mesh m(i) and displacement field d(i) are discussed. Figure 6 In , an input 2D curve 601 (represented by a 2D polyline), referred to as the "original" curve, is first downsampled to generate a base curve / polyline 602, referred to as the "decimated" curve. A subdivision scheme (such as those described in reference [A1]) (identified below) may be applied to the decimated polyline 602 to generate a "subdivided" curve 603. As an example, in Figure 6 In , a subdivision scheme using an iterative interpolation scheme can be applied. This can include inserting new points in the middle of each edge of the polyline at each iteration. Figure 6 In the illustrated example, two subdivision iterations are applied.
[0076] The solution of the present invention is independent of the subdivision scheme chosen and can be combined with any subdivision scheme, such as the one described in reference [A1]. The subdivided polyline can then be deformed to better approximate the original curve. More precisely, a displacement vector (given by Figure 6The shape of the displacement curve is close enough to the shape of the original curve. Figure 7 . ) One advantage of the subdivided curve (mesh) 603 may be that it may have a subdivision structure that allows for more efficient compression while still ensuring a reliable approximation of the original curve (mesh). Improved compression efficiency may be obtained due to various characteristics, including but not necessarily limited to the following:
[0077] • Decimation / basis curves may have a small number of vertices and may require fewer bits to encode / send.
[0078] • Subdivision curves may be automatically generated by the decoder once the base / decimation curves are decoded (ie, may not require any information other than the subdivision scheme type and subdivision iteration count to be encoded / sent).
[0079] • Displacement curves can be generated by decoding the displacement vectors associated with the subdivision curve vertices. In addition to allowing spatial / quality scalability, the subdivision structure also enables efficient wavelet decomposition (reference [A2]), which can provide high compression performance (ie, rate-distortion performance).
[0080] When the same concept is applied to the input mesh M(i), a mesh extraction technique (such as the mesh extraction technique described in reference [A3]) can be used to generate the extracted / base mesh. A subdivision scheme (such as those described in reference [A4]) can be applied to generate the subdivided mesh. The displacement field d(i) can be calculated by any method. An example is described below in Section 2. Figure 8 An example of resampling applied to an original mesh 801 having 40K triangles, resulting in a 1K triangle decimated / base mesh 802 and a 150K deformed mesh 803 is shown. Fig. 9 The original mesh 901 (in wireframe form) is compared to the deformed mesh 902 (flat shaded).
[0081] The resampling process can compute a new parameterized atlas that can be better suited for compression. In the case of dynamic meshes, this can be achieved by using a temporally consistent remeshing process that can produce the same subdivision structure shared by the current mesh M'(i) and the reference mesh M'(j). An example of such a remeshing process is described below in Section 2. Such a coherent temporal remeshing process makes it possible to skip the encoding of the base mesh m(i) and reuse the base mesh m(j) associated with the reference frame M(j). This can also achieve better temporal prediction for both attribute and geometry information. More precisely, as described in more detail below, a motion field f(i) that describes how to move the vertices of m(j) to match the position of m(i) can be computed and encoded.
[0082] Coding - Intra-frame coding
[0083] Fig.10 A block diagram of the intra-coding process is shown.
[0084] Basic Grid Coding
[0085] The base grid m(i) associated with the current frame may be first quantized 1001 (eg, using uniform quantization) and then encoded using a static grid encoder 1002. (See below for Fig.17 Inter-frame coding using a moving grid encoder is described. ) The methods and apparatus herein are independent of which grid codec is used, i.e., any of a variety of grid codecs may be used with the techniques described herein. For example, grid codecs such as those described in references [A5], [A6], [A7], or [A8] may also be used. The grid codec used may be explicitly specified in the bitstream by encoding the grid codec ID, or may be implicitly defined / fixed by the specification and / or application. Since the quantization step and / or the grid compression module may be lossy, a reconstructed quantized version of m(i) (denoted as m'(i)) may be calculated by the grid decoder 1003 within the intra-frame encoder. If the grid information is losslessly encoded and the quantization step is skipped (either or both of which may be true in some embodiments), then m(i) will exactly match m'(i).
[0086] Displacement Encoding
[0087] Depending on the application and the target bitrate / visual quality, the encoder may optionally encode a set of displacement vectors associated with the subdivided mesh vertices, which is referred to as a displacement field d(i). A technique for computing the displacement field d(i) is described below in Section 2. The displacement field d(i) may then be updated by a displacement updater 1004 using the reconstructed quantized base mesh m'(i) to generate an updated displacement field d'(i) that takes into account the differences between the reconstructed base mesh m'(i) and the original base mesh m(i). A wavelet transform 1005 (as described below) may then be applied to d'(i) by utilizing the subdivided surface mesh structure (as described below) to generate a set of wavelet coefficients e(i). The wavelet coefficients e(i) may then be quantized 1006 (producing quantized wavelet coefficients e'(i)), packed into a 2D image / video by an image packer 1007, and compressed using an image / video encoder 1008. The encoding of the wavelet coefficients may be lossless or lossy. A reconstructed version of the wavelet coefficients can be obtained by applying image unpacking 1009 and inverse quantization 1010 to the reconstructed wavelet coefficient video generated during the video encoding process. The reconstructed displacement d"(i) can then be calculated by applying the inverse wavelet transform 1011 to the reconstructed wavelet coefficients. The reconstructed base mesh m"(i) can be obtained by applying inverse quantization 1012 to the reconstructed quantized base mesh m'(i). The reconstructed deformed mesh DM(i) can be obtained by subdividing m"(i) and applying the reconstructed displacement d"(i) to its vertices by the reconstruction block 1013.
[0088] Segmentation scheme
[0089] Various subdivision schemes can be used with the techniques of this article. Suitable subdivision schemes may include, but are not limited to, those described in reference [A4]. One possible solution is a midpoint subdivision scheme, which divides the subdivision area by bisects at each subdivision iteration. Fig.11 Each triangle is subdivided into four sub-triangles for each edge of the triangle illustrated in . For example, starting from two triangles with an initial condition s0 of two triangles 1101 and 1102, the first iteration s1 produces four sub-triangles 1101a to 1101d for triangle 1101 and four sub-triangles 1102a to 1102d for triangle 1102. Each sub-triangle can be further divided in a subsequent iteration s2. A new vertex 1103 can be introduced in the middle of each edge in iteration s1, a new vertex 1104 can be introduced in the middle of each edge in iteration s2, and so on. Since the connectivity of the geometry and texture coordinates can be different, the subdivision process can be applied independently to the geometry and texture coordinates. The subdivision scheme calculates the position Pos(v) of the newly introduced vertex (v1, v2) at the center of the edge as shown below:
[0090] Pos(v)=0.5*(Pos(v1)+Pos(v2)),
[0091] Where Pos(v1) and Pos(v2) are the positions of vertices v1 and v2. The same process can be used to calculate the texture coordinates of newly created vertices. For the normal vector, an additional normalization step can be applied as follows:
[0092] N(v)=(N(v1)+N(v2)) / ||N(v1)+N(v2)||
[0093] Where: N(v), N(v1), and N(v2) are the normal vectors associated with vertices v, v1, and v2 respectively, and ||x|| is the norm 2 of vector x.
[0094] The subdivision scheme behavior can be adaptively changed (e.g., to preserve sharp edges) based on implicit and explicit criteria such as:
[0095] Full / edge / vertex attribute information associated with the base mesh and explicitly encoded as mesh attributes by the mesh codec.
[0096] Analyze the base mesh or mesh from a previous iteration to decide how to update subdivision behavior.
[0097] Wavelet Transform
[0098] Various wavelet transforms may be applied, including but not limited to those described in reference [A2]. An example of a low complexity wavelet transform may be obtained by using Fig.12 and Fig.13 The pseudo code of the lifting scheme illustrated in FIG is implemented. These figures illustrate only one exemplary implementation of a low complexity wavelet transform using the lifting scheme. Other implementations are possible and contemplated. The scheme has two parameters:
[0099] prediction weights, which control the prediction step, and
[0100] Update weights, which control the update step.
[0101] One possible choice for the prediction weight is 1 / 2. The update weight can be chosen to be 1 / 8. Note that this scheme allows skipping the update process by setting skip update to true.
[0102] Local and Canonical Coordinate Systems for Displacements
[0103] The displacement field d(i) may be defined in the same Cartesian coordinate system as the input mesh. In some cases, a possible optimization may be to transform d(i) from this canonical coordinate system to a local coordinate system, which may be defined by the normals at each vertex of the subdivided mesh. Fig.14A The pseudo code in shows an exemplary way to calculate such a local coordinate system. Other specific implementations and algorithms are possible and contemplated. The normal vector associated with the subdivided mesh can be calculated as follows:
[0104] • The normal vectors associated with the base mesh can be directly decoded or calculated based on the quantized geometry.
[0105] · Calculate the normal vectors associated with the vertices introduced during the subdivision process as described above. One potential advantage of the local coordinate system of the displacement is the possibility to quantize the tangential component of the displacement to a greater extent than the normal component. In many cases, the normal component of the displacement can have a more significant impact on the quality of the reconstructed mesh than the two tangential components.
[0106] The decision to use a canonical or local coordinate system can be made at the sequence, frame, tile group, or tile level. This decision can be made as follows:
[0107] Explicitly by encoding additional attributes associated with the base mesh vertices, edges, or faces, or
[0108] Implicitly derived by analyzing the underlying mesh connectivity / geometry / attribute information
[0109] (e.g. using a canonical coordinate system on the grid boundaries).
[0110] Wavelet coefficient quantization
[0111] Various strategies can be used to quantize the displacement wavelet coefficients. Fig. 14B An exemplary solution is illustrated. Other techniques are possible and contemplated. The idea includes using a uniform quantizer with a quiet zone, and adjusting the quantization step so that high frequency coefficients are quantized to a greater extent. Discrete quantization parameters can be used instead of directly defining the quantization step. More complex adaptive quantization schemes can be applied, such as:
[0112] · Grid quantization (as described in reference [A16]).
[0113] Optimize the quantization parameters of three components at once to minimize the distance between the reconstructed mesh and the original mesh.
[0114] · The quantization adaptive rounding scheme described in [A17].
[0115] Packing wavelet coefficients
[0116] Various strategies can be employed to pack the wavelet coefficients into a 2D image. Fig.15 One such strategy is illustrated and may be performed as follows:
[0117] First, it iterates through the coefficients from low frequency to high frequency.
[0118] • Then, for each coefficient, it determines the index of the NxM pixel block (eg, N=M=16) where it should be stored in the raster order of the block.
[0119] Finally, the position within the NxM pixel block can be calculated by using the Morton order (see reference [A9]) to maximize locality (see Fig.16 ).
[0120] Fig.15 The example is only an exemplary implementation, and other packaging schemes / strategies are possible and contemplated. In a particular embodiment, the values of N and M can be selected to be powers of 2, which can avoid Fig.15 and Fig.16 The division in the scheme described in . Fig.16 This is only one exemplary implementation of Morton order computation, and other implementations are possible and contemplated.
[0121] Attribute transfer
[0122] The attribute transfer module can calculate a new attribute map based on the input mesh M(i) and the input texture map A(i). The new attribute map can be better suited for the reconstructed deformed mesh MD(i). A more detailed description is provided below in Section 3.
[0123] Displacement Video Coding
[0124] The techniques described herein are independent of which video encoder or standard is used, which means that a wide variety of video codecs are applicable. When decoding the shifted wavelet coefficients, a lossless approach can be used because quantization can be applied in a separate module. Another approach can rely on a video encoder to compress the coefficients in a lossy manner and apply quantization in the original domain or in the transformed domain.
[0125] Color space conversion and chroma subsampling
[0126] As in the case of conventional 2D image / video coding, applying color space conversion and chroma subsampling may optionally be applied to achieve better rate-distortion performance (e.g., converting RGB 4:4:4 to YUV 4:2:0). When applying such color space conversion and chroma subsampling processes, it may be beneficial to consider surface discontinuities in the texture domain (e.g., only consider samples belonging to the same tile and potentially exclude empty areas).
[0127] Interframe Coding
[0128] Fig.17 A block diagram of an inter-frame encoding process (i.e., an encoding process in which the encoding depends on a temporally separated (e.g., previous) version of a mesh) is shown. In a non-limiting example, a reconstructed quantized reference base mesh m'(j) may be used to predict a current frame base mesh m(i). The pre-processing module described above may be configured such that m(i) and m(j) share the same number of vertices, connectivity, texture coordinates, and texture connectivity. Thus, only the positions of the vertices differ between m(i) and m(j).
[0129] The motion field f(i) (which corresponds to the displacement of vertices between m(i) and m(j)) can be calculated by the motion encoder 1701 taking into account the quantized 1702 version of m(i) and the reconstructed quantized base mesh m'(j). Since m'(j) can have a different number of vertices than m(j) (e.g., vertices can be merged / deleted), the mesh encoder can keep track of the transformation applied from m(j) to m'(j). The mesh encoder can then apply the same transformation to m(i) to ensure a 1-to-1 correspondence between m'(j) and the transformed and quantized version of m(i) (denoted as m*(i)). The motion field f(i) can be calculated by the motion encoder 1701 by subtracting the quantized position p(i,v) of vertex v of m*(i) from the position p(j,v) of vertex v of m'(j):
[0130] f(i,v)=p(i,v)-p(j,v)
[0131] The connectivity information of m'(j) can then be used to further predict the motion field, and the result is then entropy coded (eg, using context adaptive binary arithmetic coding). More details on motion field compression are provided in Section 4 below.
[0132] Since the motion field compression process may be lossy, a reconstructed motion field (denoted as f'(i)) may be calculated by applying the motion decoder module 1703. The reconstructed quantized basis grid m'(i) may then be calculated 1704 by adding the motion field to the position of m'(j). The rest of the encoding process is similar to that described above with reference to Fig.10The intra-frame encoding process includes corresponding elements.
[0133] decoding
[0134] Intra-frame decoding
[0135] Fig.18 A block diagram of the intra decoding process is shown. First, the bitstream b(i) is demultiplexed 1801 into three or more separate substreams: (1) a mesh substream, (2) a displacement substream for positions and potentially additional substreams for each vertex attribute, and (3) a property map substream for each property map. In an alternative embodiment, an atlas substream containing tile information can also be included in V3C / V-PCC in the same manner.
[0136] The grid substream can be fed to a static grid decoder 1802 corresponding to the grid encoder used to encode the substream to generate a reconstructed quantized base grid m'(i). The decoded base grid m"(i) can then be obtained by applying inverse quantization 1803 to m'(i). The techniques described herein can be used with any suitable grid codec. For example, a grid codec such as those described in references [A5], [A6], [A7] or [A8] can be used. The grid codec used can be explicitly specified in the bitstream, or can be implicitly defined / fixed by the specification and / or application.
[0137] The displacement substream can be decoded by a video / image decoder 1804 corresponding to the video / image encoder used to encode the substream. The generated image / video can then be unpacked 1805, and inverse quantization 1806 can be applied to the wavelet coefficients generated by the unpacking. Any video codec / standard can be used with the techniques described herein. For example, image / video codecs such as HEVC / H.265, AVC / H.264, AV1, AV2, JPEG, JPEG2000 can be utilized. The use of such video codecs can allow the trellis encoding and decoding techniques described herein to utilize sophisticated encoding and decoding algorithms, thereby providing high performance and high power efficiency, and the sophisticated encoding and decoding algorithms are implemented in hardware on a wide variety of platforms.
[0138] In an alternative embodiment, the displacements may be decoded by a dedicated displacement data decoder. A dictionary-based decoder such as ZIP or a motion decoder for decoding mesh motion information may, for example, be used as a dedicated displacement data decoder. The decoded displacements d"(i) may then be generated by applying an inverse wavelet transform 1807 to the unquantized wavelet coefficients. The final decoded mesh M"(i) may be generated by applying a reconstruction process 1808 to the decoded base mesh m"(i) and adding the decoded displacement field d"(i).
[0139] The attribute substream can be directly decoded by a video / image decoder 1809 corresponding to the video / image encoder used to encode the substream. The decoded attribute map A" (i) can be generated directly as an output of the decoder and / or using an appropriate color format / color space conversion 1810. As with the displacement substream, any video codec / standard can be used with the techniques described herein, including (but not limited to) image / video codecs such as HEVC / H.265, AVC / H.264, AV1, AV2, JPEG, JPEG2000. Alternatively, the attribute substream can be decoded by a non-image / video decoder (e.g., using a dictionary-based decoder such as ZIP). Multiple substreams can be decoded, each of which is associated with a different attribute map. In some embodiments, each substream can use a different codec.
[0140] Interframe decoding
[0141] Fig.19 A block diagram of the inter-frame decoding process is shown. First, the bitstream can be demultiplexed 1901 into three separate substreams: (1) motion substream, (2) displacement substream, and (3) attribute substream. In some embodiments, an atlas substream containing tile information can also be included in V3C / V-PCC in the same manner.
[0142] The motion substream can be decoded by applying a motion decoder 1902 corresponding to the motion encoder used to encode the substream. As described herein, a variety of motion codecs / standards can be used to decode the motion information. For example, any of the motion decoding schemes described below in Section 4 can be used. The decoded motion information can then optionally be added to the decoded reference quantization base grid m'(j) (in the reconstruction block 1903) to generate a reconstructed quantization base grid m'(i). In other words, the grid that has been decoded at instance j can be used (together with the motion information) to predict the grid at instance i. Thereafter, the decoded base grid m"(i) can be generated by applying inverse quantization 1904 to m'(i).
[0143] The displacement and attribute substreams can be decoded in the same manner as described above for the intra-frame decoding process. The decoded grid M" (i) is also reconstructed in the same manner. The inverse quantization and reconstruction process is not standardized and can be implemented in various ways and / or combined with the rendering process.
[0144] Post-processing
[0145] Additional post-processing modules may also be applied to improve the visual / objective quality of the decoded mesh and attribute maps and / or adapt the resolution / quality of the decoded mesh and attribute maps to the viewpoint or terminal capabilities. Some examples of post-processing are provided below:
[0146] Color format / space conversion;
[0147] Use tile information and occupancy maps to guide chroma upsampling;
[0148] Geometric smoothing filtering (see reference [A10]);
[0149] Attribute smoothing filtering (see references [A11], [A12]);
[0150] Image / video smoothing filter / filtering algorithm;
[0151] Adaptive pattern tessellation (see references [A13], [A14], [A15], [A16]).
[0152] Other extensions
[0153] In some embodiments and / or applications, it may be advantageous to subdivide the grid into a set of tiles (i.e., sub-portions) and to selectively group the tiles into a set of tile groups / blocks. In some cases, each tile or tile group may be compressed using different parameters (such as subdivision, quantization, wavelet transform, coordinate system, etc.). In such cases, it may be desirable to encode the tile information into a separate substream (similar to V3C / V-PCC). Such techniques may be advantageous for handling cracks at tile boundaries, providing:
[0154] Lossless decoding for boundary vertices
[0155] Make sure positions / vertex attributes match after displacement;
[0156] Use a local coordinate system; and
[0157] · Selectively disable quantization of wavelet coefficients.
[0158] The encoder / decoder arrangement as described herein may also support different levels of scalability. For example, temporal scalability, which may be achieved through temporal subsampling and frame reordering. Similarly, different mechanisms may be used for geometry / vertex attribute data and attribute graph data to achieve quality and spatial scalability. As an example, geometry scalability may be achieved by utilizing a subdivision structure such that the mesh resolution may be changed from one level of detail to the next. The displacement information may then be stored as two or more image / video substreams, for example:
[0159] Base layer / Detail level 0: separate video substream for low-frequency coefficients;
[0160] Refinement level 0: separate video substream for the next coefficient band;
[0161] ·…
[0162] • Refinement level N-1: separate video substream for the highest coefficient band.
[0163] In this example, detail level m may be generated by combining detail level m-1 with refinement level m-1. Furthermore, the property map may be encoded in a scalable manner by utilizing scalable video coding techniques such as those used in HEVC / H.265, AVC / H.264, VVC, AV1, or any other method that supports quality / spatial scalability of 2D images or videos.
[0164] ROI encoding can be provided by configuring the encoding process described above to encode the region of interest (ROI) with higher resolution and / or higher quality for geometry, vertex attributes and / or attribute map data. Such configuration can be used to provide higher visual quality content under tight bandwidth and complexity constraints. As an example, when encoding a mesh representing a person, a higher quality can be used for the face compared to the rest of the body. Priority / importance / space / bounding box information can be associated with slices, slice groups, blocks, NAL units and / or sub-bitstreams in a manner that allows the decoder to adaptively decode subgroups of the grid based on the view volume, power budget or terminal capabilities. It should be noted that any combination of such decoding units can be used together to implement such functionality. For example, NAL units and sub-bitstreams can be used together.
[0165] Temporal and spatial random access may also be provided. Temporal random access may be achieved by introducing IRAPs (Intra-frame Random Access Points) in different substreams (e.g., atlas, video, grid, motion, and displacement substreams). Spatial random access may be supported by defining and using blocks, sub-pictures, slice groups, and / or slices, or any combination of these decoding units. Metadata describing the layout and relationship between different units may also need to be generated and included in the bitstream to help the decoder determine the units that need to be decoded.
[0166] Lossless geometry / vertex attribute decoding may be supported by disabling one or more of the following blocks: remeshing; subdivision (e.g., setting the subdivision level to 0 so that the base mesh is the same as the input mesh); base mesh quantization; displacement substream computation. Alternatively, a simplified version of the base mesh (e.g., a quantized, low-quality version) may be encoded along with a set of displacements so that the decoder can retrieve a higher-quality version, up to and including the original mesh information.
[0167] Lossless attribute map coding may be supported by configuring the video encoder to compress the attribute map in a lossless manner (eg, using a lossless transform, PCM mode).
[0168] To maintain high quality texture coordinates, one option may be to transmit a separate displacement substream for texture coordinates. A motion substream for texture coordinates may also be employed.
[0169] Full vertex attributes can also be compressed in the same manner as the geometry information. For example, a mesh codec can be used to encode vertex attributes associated with base mesh vertices. Wavelet-based encoding can be used for attributes associated in a high-resolution mesh, which can then be stored / sent as a separate vertex attribute substream. An equivalent process applied on the decoder side can then recover / decompress the vertex attribute information.
[0170] Support for polygonal / quadrilateral meshes can be achieved by using a mesh codec capable of encoding polygonal / quadrilateral meshes and / or by choosing a subdivision scheme suitable for non-triangular meshes, such as Catmull-Clark or Doo-Sabin (see reference [A4]).
[0171] In the arrangement described above, the mesh encoder explicitly specifies the texture coordinates of the base mesh and encodes them in the bitstream. An alternative approach may be to use implicit texture coordinates derived from the position by projection (in the same way as in V-PCC or MIV) or by considering any other model (e.g., B-spline surface, polynomial function, etc.). In this case, the texture coordinates Tex cood may be defined by:
[0172] Tex coord = f(position),
[0173] where the function f may be a projection onto a predefined set of planes as in V-PCC (or any other suitable function).
[0174] References to previous sections related to "Image / Video Based Mesh Compression", each of which is incorporated by reference in its entirety:
[0175] [A1]https: / / www.cs.utexas.edu / users / fussell / courses / cs384g-fall2011 / lectures / lecture17-Subdivision_curves.pdf
[0176] [A2]http: / / www.mat.unimi.it / users / naldi / lifting.pdf
[0177] [A3]https: / / www.cs.cmu.edu / ~garland / Papers / quadrics.pdf
[0178] [A4]https: / / en.wikipedia.org / wiki / Subdivision_surface
[0179] [A5]https: / / github.com / rbsheth / Open3DGC
[0180] [A6]https: / / google.github.io / draco /
[0181] [A7]http: / / mcl.usc.edu / wp-content / uploads / 2014 / 01 / 200503-Technologies-for-3D-triangular-mesh-compression-a-survey.pdf
[0182] [A8]https: / / perso.liris.cnrs.fr / glavoue / travaux / revue / CSUR2015.pdf
[0183] [A9]https: / / en.wikipedia.org / wiki / Z-order_curve
[0184] [A10]https: / / graphics.stanford.edu / courses / cs468-12-spring / LectureSlides / 06_smoothing.pdf
[0185] [A11]https: / / cragl.cs.gmu.edu / seamless /
[0186] [A12]https: / / www.sebastiansylvan.com / post / LeastSquaresTextureSeams /
[0187] [A13]https: / / developer.nvidia.com / gpugems / gpugems2 / part-i-geometric-complexity / chapter-7-adaptive-tessellation-subdivision-surfaces
[0188] [A14]https: / / niessnerlab.org / papers / 2015 / 0dynamic / schaefer2015dynamic.pdf
[0189] [A15]https: / / giv.cpsc.ucalgary.ca / publication / c5 /
[0190] [A16]https: / / projet.liris.cnrs.fr / imagine / pub / proceedings / ICME-2007 / pdfs / 0000468.pdf
[0191] [A17] GJ Sullivan: “Adaptive Quantization Encoding Technique Using an Equal Expected-value Rule”, Joint Video Task Force, JVT-N011, Hong Kong, China (January 2005); https: / / www.google.com / url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwitlP7t46P2AhXBJkQIHRQhDp8QFnoECAcQAQ&url=https%3A%2F%2Fwww.itu.int%2Fwftp3%2Fav-arch%2Fjvt-site%2F2005_01_HongKong%2FJVT-N011.doc&usg=AOvVaw0BvZVSEpKnTznCpBKWlvBn
[0192] Part 2: Re-meshing for efficient compression
[0193] As mentioned above, static / dynamic meshes can be represented as a set of 3D meshes M(0), M(1), M(2), ..., M(n). Each mesh M(i) can be defined by connectivity C(i), geometry G(i), texture coordinates T(i) and texture connectivity CT(i). Each mesh M(i) can be associated with one or more 2D images A(i,0), A(i,1), ..., A(i,D-1), which are also called attribute maps, describing a set of attributes associated with the mesh surface. An example of an attribute would be texture information (see Figure 2 ). A set of vertex attributes can also be associated with the vertices of the mesh, such as color, normal, transparency, etc.
[0194] While the geometry and attribute information can be mapped again to 2D images and compressed efficiently using video coding techniques, the connectivity information cannot be encoded efficiently using similar schemes. A dedicated decoding solution optimized for such information is needed. In the following section, we present an efficient framework for static / dynamic mesh compression.
[0195] Discussed in more detail in Part 1 above Figure 4 and Figure 5 High-level block diagrams of the encoding and decoding processes of the present invention are shown, respectively. Note that the feedback loop during the encoding process allows the encoder to guide the preprocessing step and change its parameters to achieve the best possible compromise according to various criteria, including but not limited to:
[0196] Rate distortion,
[0197] Encoding / decoding complexity,
[0198] Random access,
[0199] Reconstruction complexity,
[0200] Terminal capabilities,
[0201] Encoding / decoding power consumption, and / or
[0202] Network bandwidth and latency.
[0203] On the decoder side, the application consuming the content can provide feedback to guide both the decoding and post-processing blocks. As just one example, based on the position of the dynamic mesh relative to the camera view volume, the decoder and post-processing blocks can adaptively adjust the resolution / accuracy of the generated mesh and / or its associated property map.
[0204] Preprocessing
[0205] In addition, as mentioned above, Figure 6 The preprocessing scheme of the present invention is illustrated in the case of 2D curves. The same concept can be applied to an input static or dynamic 3D mesh M(i) = (C(i), G(i), T(i), TC(i)) to generate a base mesh m(i) and a displacement field d(i). Figure 6 In , an input 2D curve (represented by a 2D polyline), referred to as the "original" curve, is first downsampled to generate a base curve / polyline, referred to as the "decimated" curve. A subdivision scheme (such as those described in reference [B1]) (identified below) may be applied to the decimated polyline to generate a "subdivided" curve. As an example, in Figure 6 In , a subdivision scheme using an iterative interpolation scheme can be applied. This can include inserting new points in the middle of each edge of the polyline at each iteration. Figure 6 In the illustrated example, two subdivision iterations are applied.
[0206] The solution of the invention is independent of the chosen subdivision scheme and can be combined with any subdivision scheme, such as the one described in reference [B1]. The subdivided polyline can then be deformed to better approximate the original curve. More precisely, a displacement vector can be calculated for each vertex of the subdivided mesh so that the shape of the displacement curve is sufficiently close to the shape of the original curve. (See Figure 7 One advantage of a subdivided curve may be that it may have a subdivision structure that allows for more efficient compression while still ensuring a reliable approximation of the original curve. Improved compression efficiency may be achieved due to various characteristics, including but not necessarily limited to the following:
[0207] • Decimation / basis curves may have a small number of vertices and may require fewer bits to encode / send.
[0208] • Subdivision curves may be automatically generated by the decoder once the base / decimation curves are decoded (ie, may not require any information other than the subdivision scheme type and subdivision iteration count to be encoded / sent).
[0209] • Displacement curves can be generated by decoding the displacement vectors associated with the subdivision curve vertices. In addition to allowing spatial / quality scalability, the subdivision structure also enables efficient wavelet decomposition (reference [B2]), which can provide high compression performance (ie, rate-distortion performance).
[0210] When the same concept is applied to the input mesh M(i), a mesh extraction technique (such as the mesh extraction technique described in reference [B3]) can be used to generate the extracted / base mesh. A subdivision scheme (such as those described in reference [B4]) can be applied to generate the subdivided mesh. The displacement field d(i) can be calculated by any method. Examples are described in more detail elsewhere in this document. Also as discussed above, Figure 8 An example of resampling applied to an original mesh with 40K triangles is shown, which produces a 1K triangle decimation / base mesh and a 150K deformed mesh. Also as discussed above, Fig. 9 Compare the original mesh (in wireframe form) to the deformed mesh (flat shaded).
[0211] It should be noted that the resampling process can compute a new parameterized atlas that may be better suited for compression. In the case of a dynamic mesh, this can be achieved by using a temporally consistent re-meshing process that can produce the same subdivision structure shared by the current mesh M'(i) and the reference mesh M'(j). This is described in more detail below. Such a coherent temporal re-meshing process makes it possible to skip the encoding of the base mesh m(i) and reuse the base mesh m(j) associated with the reference frame M(j). This can also achieve better temporal prediction for both attribute and geometry information. More precisely, as described in more detail below, a motion field f(i) that describes how to move the vertices of m(j) to match the position of m(i) can be calculated and encoded.
[0212] 3D remeshing
[0213] Fig. 20A block diagram of the remeshing system of the present invention is shown. The input mesh M(i) can be an irregular mesh. The output can be a base mesh m(i) with a set of displacements d(i) associated with a subdivided version of m(i). The various blocks of the system are described below. Each of these blocks can be implemented using a data processing system that includes dedicated hardware or hardware with appropriate software and / or firmware, such as CPU hardware, GPU hardware, FPGA hardware, DSP hardware, ASIC, etc.
[0214] Duplicate Vertex Removal
[0215] The duplicate vertex removal block 2001 is intended to merge duplicate vertices (i.e., vertices with the same position) or vertices with close 3D positions (e.g., vertices with a distance between them less than a user-defined threshold). The duplicate vertex removal process can be accelerated by utilizing data structures such as hash tables, kd trees, octrees, etc. By removing duplicate vertices, cracks can be avoided during subsequent processing stages (including mesh extraction stages). In addition, duplicate vertex removal can also improve decoding efficiency and encoding / decoding complexity by eliminating calculations using or based on redundant data.
[0216] Grid extraction
[0217] The mesh extraction block 2002 may employ techniques such as those described in references [B3] or [B4], for example to simplify the mesh by reducing the number of vertices / faces while substantially preserving the shape of the original mesh. Fig.21 An original grid 2101, a decimated grid 2102, a projected grid 2103, and a projected grid overlaid on top of the decimated grid 2104 are illustrated. Substantially preserving the shape of the original grid may include preserving the shape of the input grid sufficiently to achieve desired encoder and / or decoder performance while achieving a desired level of accuracy or fidelity in the resulting grid representation. This may vary from one application to another depending on the capabilities of available encoder and decoder equipment, the capabilities of displays or other output equipment, and / or the requirements of a particular application.
[0218] The illustrated mesh extraction block may apply a mesh extraction algorithm that extends those described in references [B3] or [B4] (or any other suitable extraction algorithm) by also keeping track of the mapping between the full-resolution input mesh and the extracted mesh. More specifically, at each iteration of the extraction process, the mesh extraction block 2002 may project the deleted vertices onto the extracted version of the mesh. Alternatively, the mesh extraction block may project the deleted points to the nearest corresponding points in the simplified mesh. "Nearest" may mean nearest based on the shortest L2 distance in 3D space. Fig.21Examples of the original (2101) grid, the extracted (2102) grid, and the projected (2103) grid are shown. Other criteria for defining the projection process can be used. For example, in addition to the L2 distance in 3D space, other distance metrics in 3D space (e.g., L1, Lp, L_inf, etc.) can also be used. Alternatively, the distance in a lower dimensional space can be used by projection on a 2D local plane. This can use orthogonal and / or non-orthogonal projections. For a given use case, other projection processes can also be used appropriately. The simplification algorithm can also be modified to prevent extraction operations that cause flipped triangles in the extracted grid and / or the projected grid. This optional additional requirement can help to produce a better mapping between the extracted grid and the projected grid.
[0219] Duplicate triangle removal
[0220] The duplicate triangle removal module 2003 can detect and remove duplicate triangles (i.e., triangles that reference the same vertex) in the decimated mesh dm(i). This can improve compression efficiency and encoding / decoding complexity. However, for some embodiments, duplicate triangle removal may be optional.
[0221] Small connected component removal
[0222] The small connected component removal module 2004 can detect and remove connected components. In this sense, connected components mean groups of vertices that are connected to each other but not to the rest of the mesh. Connected components targeted for deletion may include components with a number of triangles or vertices below a user-defined threshold (e.g., 8) and / or an area below a user-defined threshold (e.g., 0.1% of the original mesh area). Such small connected components are expensive to encode and have limited impact on the final visual quality of the model.
[0223] The connected component deletion criteria can be selected to be fixed for the entire mesh, or to be adaptive based on local surface properties or user-provided information describing the importance or significance of sub-portions of the mesh. For example, for a mesh including a representation of a person, a strict deletion criterion can be applied to the region depicting the head (thereby deleting fewer small connected surfaces), while a relaxed deletion criterion can be applied to the region depicting the body (thereby deleting more small connected surfaces). Additionally or alternatively, the small connected component deletion thresholds can be tuned based on rate / distortion criteria, complexity criteria, power consumption criteria, resulting bit rate, etc. In at least some embodiments, these thresholds can be provided by or derived from feedback from an encoder module (e.g., Figure 4 exemplified in ).
[0224] Atlas parameterization
[0225] The parameterization information associated with the input mesh M(i) may be suboptimal because it may define a large number of small patches (see Fig. 22 ), making it difficult to extract, remesh and compress. Instead of trying to preserve the initial parameterization during the simplification process, the atlas parameterization module 2005 can optionally recalculate the initial parameterization using techniques applied to the decimated mesh dm(i) or the decimated mesh cm(i) from which duplicate triangles and / or small connected components have been removed (such as those described in references [B6], [B7]). Fig. 22 As shown, the parameterized decimated grid 2202 has only nine tiles, compared to the original grid 2201 which has more than 100 tiles.
[0226] Mesh Subdivision
[0227] The remeshing system described herein may employ a mesh subdivision module 2006 that implements various mesh subdivision techniques, such as those described in references [B9], [B10]. The remeshing techniques described herein may be used in conjunction with these or any other subdivision techniques. For triangular meshes, mid-edge interpolation, loop, butterfly, and Catmull-Clark subdivision techniques are the most popular. These methods offer various tradeoffs in computational complexity, generality (e.g., applicability to triangular meshes versus triangular / quadrilateral or polygonal meshes), and approximation and smoothness of the generated surfaces, which may affect the rate-distortion performance of the encoder module.
[0228] Initial mesh deformation
[0229] The initial mesh deformation module 2007 can move the vertices of the subdivided mesh S(i) so that it has a shape close to the input mesh M(i). The quality of this approximation can directly affect the rate-distortion performance of the encoder. An algorithm of the present invention can be performed as follows: (1) For each vertex v of the subdivided mesh S(i), let Pos(v) indicate its initial 3D position and let N(v) indicate its normal vector. (2) For each initial 3D position Pos(v), find the nearest point H(v) on the surface of the projected mesh P(i) so that the angle between the normal N(v) and the normal of H(v) is below a user-defined threshold. Various distances can be used, including but not limited to L1, L2, Lp, Linf. The threshold can be fixed for the entire mesh, or can be adaptive based on local surface attributes and / or user-provided information describing the importance or significance of a sub-part of the mesh (e.g., face vs. body). Additionally or alternatively, the threshold can be based on a rate-distortion criterion or other criteria (e.g., complexity, power consumption, bit rate, etc.) provided as feedback from the encoder module (e.g., Figure 4 ).
[0230] H(v) can be identified by the index of the triangle it belongs to (tindex(v)) and its barycentric coordinates (a,b,c) relative to that triangle. Because the projected mesh P(i) and the mesh UM(i) have a 1-to-1 mapping between their vertices (i.e., they have the same connectivity), we can compute a point H'(v) that lies on UM(i) by using the barycentric coordinates (a,b,c) of the triangle with index tendex(v) relative to the mesh UM(i).
[0231] Iterative mesh deformation
[0232] The iterative mesh deformation module 2008 may have the deformed mesh F0(i) as input and may generate the final deformed mesh F(i) therefrom. The iterative mesh deformation module 2008 may iteratively apply an algorithm including the following:
[0233] Recalculate the normal vectors associated with the mesh vertices (see reference [B11]).
[0234] For each vertex v of the deformed mesh with position Pos(v), find its closest point H(v) on the input mesh M(i) such that the angle between the normal vectors associated with Pos(v) and H(v) is below a user-defined threshold.
[0235] o As described above, the closest point may refer to the point with the smallest distance, where various distances are used, such as L1, L2, Lp, Linf, etc.
[0236] o As also described above, the threshold may be fixed for the entire mesh, adaptive based on local surface properties and / or user-provided information describing the importance or saliency of a sub-portion of the mesh (e.g., face vs. body) and / or based on a rate-distortion criterion or any other criterion provided as feedback from the encoder module (e.g., complexity, power consumption, bit rate) (see Figure 3 ).
[0237] Move vertex v to a new position determined by:
[0238] Pos(v)+<H(v)-Pos(v),N(v)> *N(v)
[0239] in<H(v)-Pos(v),N(v)> is the dot product of two 3D vectors H(v)-Pos(v) and N(v), and where N(v) is the normal vector at Pos(v).
[0240] Optionally, check that no triangles were flipped (ie, no normal vectors were inverted) in the previous step, otherwise do not move the vertex v and mark it as an unprocessed vertex. This step can help ensure better remeshing results.
[0241] Optionally, apply a mesh smoothing filter algorithm (such as those described in references
[10] ,
[11] ) to the unprocessed vertices while taking into account the updated positions of other vertices.
[0242] Optionally, apply a mesh smoothing filter algorithm (such as those described in references
[10] ,
[11] ) to all vertices and adjust parameters to reduce the smoothing filter strength based on the fit iteration index and other criteria. The smoothing filter can be applied to vertex positions and / or displacement vectors relative to the initial mesh.
[0243] The number of deformation iterations (ie, the number of iterations through the algorithm described above) may be a parameter provided by the user or automatically determined based on a convergence criterion (eg, the displacement applied in the last iteration falls below a user defined threshold).
[0244] Basic mesh optimization
[0245] The basic grid optimization module 2009 can use the final subdivision deformed grid F(i) and the extraction grid pm(i) as input. If iterative grid deformation is omitted, the final deformed grid F(i) can be replaced by the initial deformed grid F0(i). Then, the basic grid optimization module 2009 can update the position of pm(i) to minimize the distance between the subdivision version of pm(i) and F(i) (or F0(i)). In some embodiments, this can be realized by solving a sparse linear system. A possible method for efficiently solving this type of sparse linear system is the conjugate gradient method (see, for example, reference [B15]). Other techniques can also be used.
[0246] Calculate displacement
[0247] The displacement calculation module 2010 can calculate the displacement d(i) by taking the difference between the position of the subdivided version of F(i) (or F0(i)) and pm(i) to exploit the correlation between the two grids and produce a more compressible representation. The resulting displacement field d(i) can then be fed as input (along with the base grid m(i)) to the encoder module, as described above in Part 1.
[0248] Time-consistent remeshing
[0249] The remeshing procedure described above processes each frame M(i) independently. While this is optimal for intra-frame decoding, temporally consistent remeshing can allow for better temporal prediction for both mesh and image data. For temporally consistent remeshing, one concept is to reuse the base mesh pm(j) associated with the reference frame M(j) for a base mesh pm(i) with the same connectivity. By ensuring that there is a 1-to-1 mapping between pm(i) and pm(j), and that pm(i) and pm(j) have the same number of vertices, the same number of triangles (or polygons), texture coordinates, and texture coordinate triangles (or polygons), the difference between pm(i) and pm(j) is only the position of their vertices. Therefore, there are two different cases: (1) the input meshes M(i) and M(j) are themselves temporally coherent, or (2) the input meshes M(i) and M(j) are not temporally coherent.
[0250] In the first case, i.e., if the input meshes M(i) and M(j) are time-coherent, only the subdivision surface fitting module can be applied. In other words, there is no need to simplify or pre-filter repeated vertices and connected components. In this case, the input of the fitting subdivision surface module 2011 (composed of the components 2006 to 2010 discussed above) can be the input mesh M(i), the projection mesh P(j) (from the reference frame) and the extraction mesh pm(j) (also from the reference frame), instead of M(i), P(i), pm(i).
[0251] In the second case, i.e., if the input meshes M(i) and M(j) are not time-coherent, a deformed version of M(j) (denoted M'(j)) having the same shape as M(i) may be generated. M'(j) may be generated using techniques such as those described in references [B12], [B13], [B14] (or similar techniques). Then, one may proceed as described above, only applying the Fit Subdivision Surface module 2011, providing M'(j), P(j), pm(j) instead of M(i), P(i), and pm(i) as input.
[0252] References to previous sections related to "Re-meshing for Efficient Compression", each of which is incorporated by reference in its entirety:
[0253] [B1]https: / / github.com / rbsheth / Open3DGC
[0254] [B2]https: / / google.github.io / draco /
[0255] [B3]https: / / www.cs.cmu.edu / ~garland / Papers / quadrics.pdf
[0256] [B4]http: / / jerrytalton.net / research / t-ssmsa-04 / paper.pdf
[0257] [B5]https: / / graphics.stanford.edu / courses / cs468-10-fall / LectureSlides / 08_Simplification.pdf
[0258] [B6]https: / / graphics.stanford.edu / courses / cs468-05-fall / Papers / param-survey.pdf
[0259] [B7]https: / / www.semanticscholar.org / paper / Iso-charts%3A-stretch-driven-mesh-parameterization-Zhou-Snyder / 27b260713ad9802923aec06963cd5f2a41c4e20a
[0260] [B8]https: / / members.loria.fr / Bruno.Levy / papers / LSCM_SIGGRAPH_2002.pdf
[0261] [B9]https: / / en.wikipedia.org / wiki / Subdivision_surface
[0262] [B10]https: / / graphics.pixar.com / opensubdiv / docs / intro.html
[0263] [B11]https: / / cs.nyu.edu / ~perlin / courses / fall2002 / meshnormals.html
[0264] [B10]https: / / graphics.stanford.edu / courses / cs468-12-spring / LectureSlides / 06_smoothing.pdf
[0265] [B11]https: / / www.medien.ifi.lmu.de / lehre / ws2122 / gp / slides / gp-ws2122-3-smooth.pdf
[0266] [B12]https: / / lgg.epfl.ch / publications / 2008 / sgp2008GCO.pdf
[0267] [B13]https: / / arxiv.org / abs / 2004.04322
[0268] [B14]https: / / people.inf.ethz.ch / ~sumnerb / research / embdef / Sumner2007EDF.pdf
[0269] [B15]http: / / math.stmarys-ca.edu / wp-content / uploads / 2017 / 07 / Mike-Rambo.pdf
[0270] Part 3: Attribute transfer for efficient dynamic mesh decoding
[0271] As described above, static / dynamic meshes can be represented as a set of 3D meshes M(0), M(1), M(2), ..., M(n). Each mesh M(i) can be defined by connectivity C(i), geometry G(i), texture coordinates T(i) and texture connectivity CT(i). Each mesh M(i) can be associated with one or more 2D images A(i,0), A(i,1), ..., A(i,D-1) (also called attribute maps). The attribute map describes a set of attributes associated with the mesh surface. An example of an attribute would be texture information (see Figure 2 , Figure 3 , Fig.23 ). A set of vertex attributes can also be associated with the vertices of the mesh, such as color, normal, transparency, etc.
[0272] When decoding a dynamic mesh, a preprocessing stage such as described above may be applied to produce a more compression-friendly version of the input mesh. Such preprocessing may involve re-meshing (i.e., resampling) and re-parameterization (i.e., computing a new atlas of the parameterization with a smaller number of tiles and low parameterization distortion). Once the mesh is resampled, attribute transfer may be performed. Attribute transfer may include computing a new attribute map coherent with the re-meshed and / or re-parameterized mesh. For example, Fig.23An example 2300 of attribute transfer after remeshing is illustrated. First, an original input mesh 2302 and associated tiles 2304 are obtained. The original input mesh 2302 can be remeshed / resampled via the pre-processing stage described herein, thereby producing a remeshed mesh 2306. As illustrated by the updated tiles 2308 associated with the remeshed mesh 2306, a new atlas associated with the remeshed mesh 2306 can be calculated, which can provide a smaller number of tiles and low parameterization distortion. Therefore, once the original mesh 2302 is remeshed / resampled, a new attribute map coherent with the remeshed mesh 2306 can be calculated via attribute transfer from the original texture map 2310 to the updated texture map 2312 associated with the remeshed mesh 2306.
[0273] See below Figure 24 to Figure 27 A detailed discussion of the attribute transfer process is described. Fig.24 The attribute transfer process 2400 is illustrated. Fig.25 An exemplary implementation of an attribute transfer process 2500 is provided. For clarity, these figures will be discussed together.
[0274] Steering Fig.24 , the process 2400 may be run for each pixel A(i,j) of the attribute map A to be generated. First, the texture coordinates (u,v) of each pixel (i,j) of the attribute map A(i,j) to be generated are calculated (block 2401). Fig.25 , a pixel A(i,j) 2502 of a property map A to be generated (e.g., a texture map 2312 associated with the re-meshed mesh 2306) is associated with a coordinate (u,v) 2504 in a texture domain 2506, which includes an updated patch 2308 associated with the re-meshed mesh 2306.
[0275] Next, it is determined whether the point P(u,v) in texture space belongs to any triangle of the re-meshed mesh (block 2402). Fig.25 , determine whether the point P(u,v) associated with the coordinates (u,v) 2504 is associated with at least one update slice in the update slice 2308.
[0276] If P(u,v) does not belong to any triangle (block 2403, no), the pixel is marked as an empty pixel (block 2404), and the empty pixel is Fig. 27 The empty pixel filling process described in process 2700 and below is used to fill the empty pixel.
[0277] Otherwise, if P(u,v) belongs to the triangle (T) defined by the three vertices (A,B,C) (block 2403, yes), the pixel is marked as filled (block 2405). The barycentric coordinates (α, β, γ) of the point P(u,v) of the triangle T in the parameter space are calculated (block 2406).
[0278] The 3D point M(x,y,z) associated with the texture coordinate P(u,v) is calculated using the barycentric coordinates (α,β,γ) and the 3D position associated with the triangle T in 3D space (block 2407). Fig.25 , a 3D point M(x, y, z) of the 3D domain of the re-meshed mesh 2306 is identified.
[0279] Next, the nearest 3D point M'(x', y', z') located on the triangle T' of the original mesh is identified (block 2408). The barycentric coordinates (α', β', γ') of M' are calculated based on T' in 3D space (block 2409). For example, Fig.25 As illustrated in , a 3D point M′(x, y, z) is identified in the 3D domain of the original mesh 2302 based on a point M(x, y, z) of the remeshed mesh 2306 .
[0280] The point P'(u',v') associated with M' is calculated using the barycentric coordinates (α', β', γ') with the 2D parameter coordinates associated with the three vertices of T' (block 2410). Fig.25 As illustrated, a point P'(u',v') is identified based on a point M'(x,y,z) and a triangle (T') of M'.
[0281] The texture coordinates (u', v') of the point P'(u', v') are calculated to sample the original texture map and calculate the attribute value A'(i', j') of the input attribute map (block 2411). Bilinear interpolation, nearest neighbor interpolation, tile-aware bilinear interpolation, tile-aware nearest neighbor interpolation, and / or other interpolation methods may be used to calculate these coordinates. The attribute value A'(i', j') may then be assigned to the pixel (i, j) of the attribute map A(i, j), causing an attribute transfer to the generated attribute map (block 2412). For example, the generated texture map 2312 illustrates pixels filled with values from the original texture map 2310 after the process 2400 is completed for each pixel of the generated texture map 2312.
[0282] When implemented, Fig.24 and Fig.25 The process described in can generate discontinuities at parameterized seams, such as Fig.262601 compared to the right image 2602. In practice, edges located on parameterized seams may correspond to tile boundaries. Due to the cut operation used to flatten the mesh, each seam edge in 3D space is mapped to two edges in texture space. Since the algorithm described in the previous section calculates the color of each pixel in the texture domain independently without considering the seams, inconsistent colors may result on the edges. This can be further exacerbated by the bilinear interpolation used during the rendering process. Potential solutions to address such issues are described in [C1] and [C2].
[0283] However, references [C1] and [C2] may be complex solutions that utilize significant processing resources and / or processing time. Fig. 27 A relatively low-complexity alternative to [C1] and [C2] is provided in Fig.26 The remedy for the discontinuity is illustrated on the right side of . Fig. 27 In the seam edge discontinuity process 2700, for example, Fig.24 For each empty pixel A(i,j) adjacent to the filled pixel A(k,l), determine the triangle T used to fill A(k,l) (box 2701).
[0284] Next, apply the above reference Fig.24 The process 2400 (block 2702) described above takes into account the barycentric coordinates calculated for A(i, j) with respect to triangle T. In other words, when pixel A(i, j) is empty, the process 2400 can use the triangle T used to fill the adjacent pixels when determining the attribute value to transfer for A(i, j).
[0285] Although pixel A(i,j) is empty, this still causes attribute transfer for A(i,j), thereby reducing the seam edge discontinuity. Therefore, pixel A(i,j) can be marked as filled, and the index of triangle T can be stored (block 2703).
[0286] Fig. 27 The process may be applied once or multiple times. The process may be controlled via parameters that can be provided by the user and / or automatically calculated (e.g., based on processing resource availability, grid characteristics, etc.). Fig. 27 It can be understood that by focusing on empty pixels, Fig. 27 The process does not change in Fig.24 The value of the pixel calculated during the attribute transfer process. Fig. 27 The process of filling and Fig. 27 The process fills in adjacent pixels to support consistent colors on the seam edges by exploiting property consistency in 3D space.
[0287] In Application Fig.24 The property transfer process 2400 and Fig. 27 After the seam edge mitigation process 2700, only a subset of the attribute map pixels may be filled. For example, Fig.28 The occupancy map 2801 indicates whether a pixel is empty or full. A padding algorithm can be used to fill the remaining empty pixels with color so that the attributes are smoother and / or easier to compress. In particular, a push-pull algorithm such as that described in reference [C3] can be applied to the initial attribute map 2802 to identify an initial padding solution for filling the empty pixels. The initial solution can be refined by applying an iterative algorithm, which is described in the applicant's co-pending U.S. patent application No. 16 / 586,872, entitled "Point Cloud Compression Image Padding", filed on September 27, 2019, which is incorporated herein by reference. The technology includes using padding to fill blank spaces in an image frame, wherein the pixel values used for padding are determined based on adjacent pixel values so that the padding is smoothed and filtered in the image frame. The resulting padding can be combined with the initial attribute map 2802 to produce a filled attribute map 2803.
[0288] References to the previous section related to "Attribute Transfer for Efficient Dynamic Mesh Coding", each of which is incorporated by reference in its entirety:
[0289] [C1]https: / / www.sebastiansylvan.com / post / LeastSquaresTextureSeams /
[0290] [C2]https: / / cragl.cs.gmu.edu / seamless /
[0291] [C3]https: / / citeseerx.ist.psu.edu / viewdoc / download? doi=10.1.1.219.7566&rep=rep1&type=pdf
[0292] Part 4: Motion Compression for Efficient Dynamic Mesh Decoding
[0293] As mentioned above, static / dynamic meshes can be represented as a set of 3D meshes M(0), M(1), M(2), ..., M(n). Each mesh M(i) can be defined by connectivity C(i), geometry G(i), texture coordinates T(i) and texture connectivity CT(i). Each mesh M(i) can be associated with one or more 2D images A(i,0), A(i,1), ..., A(i,D-1), which are also called attribute maps, describing a set of attributes associated with the mesh surface. An example of an attribute would be texture information (see Figures 2 to 3 ). A set of vertex attributes can also be associated with the vertices of the mesh, such as color, normal, transparency, etc.
[0294] Dynamic meshes can exhibit high temporal correlation because they can correspond to smooth motion and / or smooth changes in attribute characteristics. When decoding dynamic meshes as described above in Section 1, a video encoder can efficiently exploit attribute temporal correlation to provide more efficient compression. Disclosed herein are techniques for compressing motion data associated with such representations (i.e., geometry and vertex attributes that change from one frame to another). An input mesh (e.g., M(i)) can be subdivided into a set of tiles P(i,j), where i is a frame index and j is a tile index. The input data can have a temporally consistent structure that can ensure that at least a subset of tiles in a current frame has the same connectivity as a corresponding tile in a reference frame. If the input data does not have such a temporally consistent structure, a preprocessing step can be applied as described above in Section 2, which includes applying temporally consistent remeshing.
[0295] In either case, once a temporally consistent sequence of meshes is received, P(i,j) may be tile j of the current frame i, and P(k,l) may be the corresponding tile (tile l) of the reference frame k. Due to the temporal consistency described above, P(i,j) and P(k,l) may have the same connectivity (i.e., the same number of vertices and faces). P(i,j) and P(k,l) may therefore differ only in corresponding positions or vertex attributes. As described in more detail below, this may be accomplished by applying quantization ( Fig.29 , 2901), space-time prediction ( Fig.29 , 2902) and entropy decoding ( Fig.29 , 2903) to compress these different positions and / or vertex attributes.
[0296] Quantification
[0297] In some embodiments, the quantization module ( Fig.29, 2901) uniform quantization is applied to vertex positions and / or vertex attributes. Using uniform quantization, the same number of quantization bits (quantization levels) may be applied to all vertices or attributes. Alternatively, in some embodiments, an adaptive quantization scheme may be employed. An adaptive quantization scheme may use coarser quantization (fewer bits / level) for some regions and finer quantization (more bits / level) for other regions. Such an adaptive quantization scheme may adaptively change the quantization step size based on a provided input used (e.g., a user identification of a region of interest (ROI), such as a face of a mesh depicting a person). Additionally or alternatively, an adaptive quantization scheme may adaptively change the quantization step size based on an analysis of a dynamic mesh, for example, as described in reference [D1]. The analysis may be performed online or offline. In any case, care may be required to avoid introducing cracks at tile boundaries, for example due to different quantization levels on either side of the boundary.
[0298] Space-time prediction
[0299] Prediction Module( Fig.29 , 2902) can utilize either or both of the shared connectivity information (spatial information) and the reference frame P(k, l) (temporal information) to efficiently predict the geometry and / or vertex attributes associated with the vertices of the tile P(i, j). A variety of "predictors" can be implemented by the prediction module. For a given implementation, these predictors can be used alone or in appropriate combinations. In the foregoing description of predictors, the following symbols are used.
[0300] Pos(i,j,v) is the vertex position of vertex v in the current slice P(i,j);
[0301] ·Pos(i,j,v0),…,Pos(i,j,v n-1 ) are the neighboring vertices v0, ..., v that have been encoded or decoded and can be used for prediction. n-1 The position of (the adjacent vertices of vertex v);
[0302] Pos(k,l,v) is the position of vertex v in the reference patch P(k,l); and
[0303] ·Pos(k,l,v0),...,Pos(k,l,v n-1 ) are adjacent vertices v0, ..., v n-1 Position in the reference slice P(k,l).
[0304] With the above notation in mind, the prediction module 2902 may implement different prediction factors as described below. As a few non-limiting examples:
[0305] The Δ temporal predictor can use temporal information (instead of spatial information) to generate the residual ρ(i,j,v) (defining the difference between the current frame and the reference frame) as follows:
[0306] ρ(i,j,v)=Pos(i,j,v)-Pos(k,l,v)
[0307] The average spatial predictor can use spatial information (instead of temporal information) to generate the residual ρ(i,j,v) as follows:
[0308]
[0309] The average predictor can use temporal and spatial information to generate the residual ρ(i,j,v) as follows:
[0310]
[0311] The spatial parallelogram predictor can use the spatial information to predict the residuals based on parallelograms as follows:
[0312] ρ(i,j,v)=Pos(i,j,v)-Π(i,j,v)
[0313] in:
[0314] Π(i,j,v)=Pos(i,j,v a )+Pos(i,j,v b )-Pos(i,j,v c )
[0315] The predictor for the spatio-temporal parallelogram prediction can use both spatial and temporal information to predict the residual based on the parallelogram as follows:
[0316] ρ(i,j,v)=Pos(i,j,v)-(Π(i,j,v)+Pos(k,l,v)-Π(k,l,v))
[0317] Geometry-guided predictors described in applicant’s co-pending U.S. Provisional Patent Applications 63 / 197,288, entitled “Compression of Attribute Values Comprising Unit Vectors” and 63 / 197,287, entitled “Attribute Value Compression for a Three-Dimensional Mesh Using Geometry Information to Guide Prediction”, both filed on June 4, 2021.
[0318] • Other predictors, such as those described in references [D6], [D7], [D8], [D9], [D10], [D11], [D12].
[0319] In some embodiments, the encoder may evaluate multiple different predictors and select a predictor that produces a rate-distortion performance that is the best trade-off between the number of bits used to encode the motion information and the distortion impact of the encoded grid compared to the original grid. For any predictor used, an index to the predictor (i.e., an identification of the predictor used) along with the prediction residual may be entropy encoded as described below for transmission to the decoder.
[0320] Entropy decoding
[0321] As described above, by applying an entropy encoder (2903, Fig.29 ) to efficiently decode the predictor index and the prediction residual. Examples of suitable entropy encoders may include, but are not limited to, a context-adaptive binary arithmetic coder (CABAC) (see, e.g., reference [D2]), a Huffman encoder (see, e.g., references [D3] and [D4]) combined with a general code (see, e.g., reference [D5]), and the like. Fig.30 An example of employing a CABAC encoder together with an Exponential Golomb code to compress both the predictor index and the prediction residual is shown. Fig.30 The example of is only one possible implementation of such an algorithm, and other implementations and / or other algorithms are possible and contemplated.
[0322] Other extensions
[0323] In at least some embodiments, motion coding of mesh data can be extended in other ways. As an example, the MPEG FAMC (Frame-based Animation Mesh Compression) standard can be used. See, for example, references [D15], [D16]. In at least some embodiments, wavelet-based decoding schemes can be used, such as the decoding schemes described in references [D13], [D14]. In at least some embodiments, principal component analysis (PCA)-based decoding (as described in reference [D17]) can be used.
[0324] References for previous sections related to "Motion Compression for Efficient Dynamic Trellis Coding", each of which is incorporated by reference in its entirety:
[0325] [D1]https: / / www.sciencedirect.com / topics / computer-science / adaptive-quantization
[0326] [D2]https: / / en.wikipedia.org / wiki / Context-adaptive_binary_arithmetic_coding
[0327] [D3]https: / / en.wikipedia.org / wiki / Huffman_coding
[0328] [D4]https: / / en.wikipedia.org / wiki / Asymmetric_numeral_systems[D5]https: / / en.wikipedia.org / wiki / Universal_code_(data_compression)
[0329] [D6] L.Ibarria et J.Rossignac.Dynapack: space-time compression of the3D animations of triangle meshes with fixed connectivity. In EurographicsSymposium on Computer Animation, pages 126–133, San Diego, E′tats-Unis, 2003.
[0330] [D7] N. Stefanoski and J. Ostermann. Connectivity-guided predictive compression of dynamic 3D meshes. In IEEE International Conference on Image Processing, pp. 2973–2976, Atlanta, United States, 2006.
[0331] [D8] J.-H. Yang, C.-S. Kim, and S.-U. Lee. Compression of 3-D triangle mesh sequences based on vertex-wise motion vector prediction. IEEE Transactions on Circuits and Systems for Video Technology, 12(12): 1178–1184, 2002.
[0332] [D9] N. Stefanoski, P. Klie, X. Liu, and J. Ostermann. Scalable linear predictive coding of time-consistent 3D mesh sequences. In The True Vision - Capture, Transmission and Display of 3D Video, pp. 1–4, Kos Island, Greece, 2007.
[0333] [D10] N. Stefanoski, X. Liu, P. Klie, and J. Ostermann. Layered predictive coding of time-consistent dynamic 3D meshes using a non-linear predictor. In IEEE International Conference on Image Processing, pp. 109–112, San Antonio, United States, 2007.
[0334] [D11]V. Libor and S. Vaclav. Coddyac: Connectivity driven dynamic mesh compression. In 3DTV International Conference: True Vision - Capture, Transmission and Display of 3D Video, Kos Island, Greece, 2007.
[0335] [D12]M. Sattler, R. Sarlette, and R. Klein. Simple and efficient compression of animation sequences. In Eurographics Symposium on Computer Animation, pp. 209–217, Los Angeles, United States, 2005.
[0336] [D13]I. Guskov and A. Khodakovsky. Wavelet compression of parametrically coherent mesh sequences. In Eurographics Symposium on Computer Animation, pp. 183–192, Grenoble, France, 2004.
[0337] [D14]J. W. Cho, M. S. Kim, S. Valette, H. Y. Jung, and R. Prost. 3D dynamic mesh compression using wavelet - based multiresolution analysis. In IEEE International Conference on Image Processing, pp. 529–532, Atlanta, United States, 2006.
[0338] [D15]K. Mamou, T. Zaharia, F. Preteux, A skinning approach for dynamic 3D mesh com - pression, Computer Animation and Virtual Worlds, Vol. 17(3 - 4), July 2006, pp. 337 - 346.
[0339] [D16] K.Mamou, N.Stefanoski, H.Kirchhoffer, K.Muller, T.Zaharia, F.Preteux, D.Marpe, J.Ostermann, The new MPEG-4 / FAMC standard for animated 3D meshcompression, 3DTV Conference (3DTV-CON 2008), Istanbul, Turkey, May 2008.
[0340] [D17]K.Mamou,T.Zaharia,F.Preteux,A.Kamoun,F.Payan,M.Antonini.Twooptimizations of the MPEG-4FAMC standard for enhanced compression of animated3D meshes.IEEE International Conference on Image Processing(2008)
[0341] [D18]https: / / www.researchgate.net / publication / 224359352_Two_Optimizations_of_the_MPEG-4_FAMC_standard_for_Enhanced_Compression_of_Animated_3D_Meshes / link / 0912f50b3802603f34000000 / download
[0342] Part 5: V-trellis bitstream structure including syntax elements and decoding process with reconstruction
[0343] In order to better support Video Dynamic Grid Coding (V-DMC) in the context of the V3C specification, new syntax elements can be introduced to handle grid information. Unlike V-PCC, V-DMC can be seen as a scalable coding solution, where an initial representation of the grid is provided by a so-called base grid. This representation is then enhanced by including additional information through the V3C framework. One of the enhancements introduced here is the inclusion of the base grid information in a new substream, the base grid data substream. Similar to the substreams of atlas and video decoding, this substream is a timed sequence of decoded grid information. For more information on this substream, we refer to reference [D5].
[0344] Furthermore, in V-DMC, the coded geometry data are actually transformed and quantized data, and their transformations can be inverted before being used in the reconstruction process. Specifically, after decoding the geometry video, the decoded data can also be processed by a so-called "displacement" decoder. This decoder performs a dequantization process followed by an inverse transform process as specified by instructions in the atlas data substream, which include information about the quantization and the transform method used when encoding the geometry information.
[0345] Similar to the geometry information, additional processing may be performed on the base mesh information after it is decoded. More specifically, after the base mesh data is decoded, the resulting mesh may be subdivided through a mesh subdivision process. This process requires information that may be indicated / included in the atlas data substream, such as the subdivision method to be used, etc. The subdivided / resampled mesh is then refined by adding the displacements from the geometry displacement decoder. Additional information from the atlas data substream may be used to perform this final process. For example, subpart ids may be used to pair the displacements from the displacement decoder with vertices in the resampled mesh.
[0346] To help understand the concepts introduced in V-DMC, we first introduce some basic terms and definitions:
[0347] The base grid is the output of the base grid substream decoder. Each base grid is compared to V-
[0348] The expected output of the DMC decoder may have a smaller number of vertices.
[0349] The resampled base mesh is the output of the mesh subdivision process. Each base mesh is compared to V-
[0350] The expected output of the DMC decoder may have the same number of vertices.
[0351] The displacement video is the output of the displacement decoder. The input to this process is the decoded geometry video and information from the atlas about how to interpret / process this video. The displacement video contains the displacement values to be added to the corresponding vertices.
[0352] The face group Id is one of the attribute types assigned to each triangular face of the resampled base mesh. The face group Id can be compared with the id of the subpart in the tile to determine the corresponding face group of the tile. If the face group Id is not transmitted by the base mesh substream decoder, it is derived from information in the atlas data substream.
[0353] The submesh id is one of the attribute types assigned to each vertex of the resampled base mesh. The submesh id can be compared with the segment id to determine the corresponding vertex of the segment. If it is not transmitted by the base mesh substream decoder, it is derived from information in the atlas data substream.
[0354] like Fig.31 As illustrated in , 3D texture static and / or dynamic grids can be encoded by V-grid encoder 3101 into V-grid bitstream 3102 for subsequent decoding by V-grid decoder 3103. As discussed in more detail below, the V-grid bitstream structure can be an extension of V3C for efficient processing. Details on encoding of 3D texture static and / or dynamic grids are discussed elsewhere in this document (e.g., Part 1).
[0355] Fig.32 An implementation of a V-trellis decoder framework 3200 is illustrated. When a v-trellis bitstream is provided to a decoder, the decoder demultiplexes (3201) the bitstream into V3C parameter sets 3202, trellis sub-bitstreams 3203, geometry sub-bitstreams 3204, attribute sub-bitstreams 3205, and atlas data sub-bitstreams 3206, which may be incorporated into the V-trellis bitstream via the encoding process described above in Part 1. The parameter sets 3202 may be decoded via the parameter set decoder 3203, and the atlas data sub-bitstream 3206 may be decoded via the atlas data sub-bitstream decoder 3208.
[0356] Using the information provided by the parameter set 3202 and the atlas data sub-bitstream 3206, other sub-bitstreams can be converted into appropriate forms through a normalization process. For example, the grid sub-bitstream 3203 can be decoded by the grid sub-bitstream decoder 3209 into a base grid, which can be normalized via a grid subdivision / grid normalization process 3210. The geometry sub-bitstream 3204 can be decoded by the video decoder 3211 into a geometry image. The geometry image can be normalized via a displacement decoder / geometry normalization process 3212 to produce a displacement value. The attribute sub-bitstream 3205 can be decoded by the video decoder 3213, and the decoded output can be normalized by an attribute normalization process 3214 to produce an attribute image. (Video decoder 3211 may be the same as video decoder 3209 or a different video decoder selected for a particular implementation.) Output mesh geometry, texture coordinates, and connectivity are computed by a mesh position refinement process (3215), which combines the output of mesh normalization process 3210 and the output of geometry normalization process 3212 to derive a resulting mesh (e.g., described by mesh geometry, texture coordinates, and connectivity). Fig.18 and Fig.19 In contrast, the geometry normalization process includes inverse quantization and inverse wavelet transform. The attribute normalization process may also include color space conversion. The mesh normalization process may include some form of reconstruction of the deformed mesh as described above.
[0357] As described above, the compressed base mesh can be signaled in a new substream, which is named the base mesh data substream (e.g., it has a unit type V3C_MD). As with other v3c units, the unit type and its associated v3c parameter set id and atlas id are signaled in the v3c_unit_header(). The proposed format of the mesh data substream is further discussed in reference [D5]. In order to facilitate signaling the compressed base mesh, the encoded bitstream can be encoded in an extended V3C manner. Therefore, described below are examples of enumerated syntax elements that can be specifically used for 3D texture static and / or dynamic mesh decoding and existing syntax elements that can be configured for use with 3D texture static and / or dynamic mesh decoding.
[0358] V3C parameter set extension elements
[0359] Beginning the discussion with modifications to the V3C parameter set 3202 to support V-DMC, additional parameters and / or modified use of parameters in the VLC parameter set 3202 may be implemented. The following are examples of such additional parameters and / or modified use of existing parameters in the V3C parameter set 3202 to be supported.
[0360] V3C unit type, V3C-MD - The identifier V3C_MD (tentatively vuh_unit_type=5) is assigned to indicate a grid sub-bitstream. Through this identifier, v3c_unit_header() and v3c_unit_payload() include the procedures for the grid sub-bitstream as follows:
[0361]
[0362]
[0363] vuh_mesh_data_sample_stream_flag indicates that the mesh sub-bitstream has a sample stream format as defined herein. When flagged as 0, the mesh sub-bitstream is fully decoded using external methods.
[0364] vuh_mesh_data_motion_field_present_flag indicates that the mesh sub-bitstream contains data that can be used for inter prediction between mesh data in the mesh sub-bitstream. In some embodiments, vuh_mesh_data_motion_field_present_flag indicates that the mesh sub-bitstream requires more than one decoder engine to decode the contained information.
[0365] In some implementations, an extension may be introduced in the v3c_parameter_set syntax structure to facilitate handling of dynamic trellis decoding coding (V-DMC). The following discussion focuses on several new parameters that may be introduced in the extension to handle V-DMC.
[0366] v3c_vmesh_extension in V3C parameter set 3202 is an extension that provides several new parameters to V3C parameter set 3202 to enable V-DMC. To signal the base mesh information in V3C parameter set 3202, the extension flag vps_extension_present_flag may be set to 1 and (vps_extension_8bits>>N)&1 may be 1. Here, "N" is less than 8 and may be decided when finalizing version 2 of 23090-5 (reference [E2]). In the following example, N is set to 4. V3C_vmesh_extension may be signaled as follows:
[0367]
[0368] vps_vmesh_extension_length_minus1 indicates the size of vps_v3c_vmesh_extension.
[0369] In some embodiments, the following parameters may be added via V3C parameter set 3202 extension:
[0370]
[0371] vps_geometry_frame_width and vps_geometry_frame_height indicate the nominal width and height of the geometry video bitstream. vps_disp_frame_width and vps_disp_frame_height override the geometry video bitstream. vps_frame_width and vps_frame_height correspond only to the width and height of the non-geometry video bitstream.
[0372] vps_atlas_data_substream_present_flag indicates the presence of atlas sub-bitstream 3206 in the bitstream. If this flag is false, atlas substream 3206 should not be present in the bitstream. If such a bitstream is present, it should be ignored. In some embodiments, this flag is not signaled, but is always set to 1 for the v-trellis codec.
[0373] vps_mesh_substream_present_flag indicates the presence of a mesh sub-bitstream in the bitstream. In some embodiments, this flag is not signaled but is always set to 1 for the v-mesh codec.
[0374] A basemesh_information may be added to the V3C parameter set to signal the information of the mesh sub-bitstream, extending v3c_parameter_set to add this element. basemesh_information may include the following:
[0375]
[0376] mi_datatype_count indicates the number of different data types in the mesh bitstream. It is set to 1 when vuh_mesh_data_motion_field_present_flag is false or when vuh_mesh_data_sample_stream_flag is false.
[0377] mi_type_id[atlasID][i] indicates the data type. For example, it can be decoded to indicate mesh data or motion field.
[0378] mi_codec_id[atlasID][i] indicates the codec used to decode the associated data type. Current technology does not limit the format of this element. The value can be a 4cc code or a number explicitly defined in the v-trellis codec. For example, mi_codec_id[atlasID][0]=DRACO indicates that data of data type=0 is decoded by the trellis codec, and DRACO.mi_codec_id[atlasID][0]=INTERNAL indicates that data of data type=0 is decoded by the decoder defined in the v-trellis codec.
[0379] mi_basemesh_geometry_3d_bit_depth_minus1 indicates the nominal bit depth of the locations of the mesh conveyed in the mesh sub-bitstream.
[0380] mi_basemesh_geometry_MSB_align_flag indicates how to convert decoded base mesh positions to samples at the nominal geometry bit depth.
[0381] mi_basemesh_meshattribute_count indicates the number of attributes of a mesh (such as color, normal or reflectivity) conveyed in the mesh sub-bitstream.
[0382] mi_basemesh_meshattribute_type_id indicates the attribute type of the mesh conveyed in the mesh sub-bitstream, such as color, normal, reflectivity, etc.
[0383] mi_basemesh_meshattribute_bit_depth_minus1 indicates the bit depth of the base mesh attributes.
[0384] mi_basemesh_meshattribute_MSB_align_flag indicates how to convert the decoded base mesh attributes to samples at the nominal bit depth.
[0385] In some embodiments, the following parameters may be added via V3C parameter set 3202 extension:
[0386]
[0387]
[0388] vps_ext_atlas_data_substream_present_flag indicates the presence of an atlas data substream in the bitstream. If this flag is false, the atlas substream shall not be present in the bitstream. If such a bitstream is present, it shall be ignored.
[0389] vps_ext_mesh_data_substream_present_flag indicates the presence of a mesh data substream in the bitstream. If this flag is false, the base mesh substream shall not be present in the bitstream. If such a bitstream is present, it shall be ignored.
[0390] vps_ext_mesh_data_facegroup_id_attribute_present_flag equal to 1 indicates that one of the attribute types present in the base mesh data stream is FaceGroupId.
[0391] vps_ext_mesh_data_submesh_id_attribute_present_flag equal to 1 indicates that one of the attribute types of the base mesh data stream is a Submesh Id.
[0392] vps_ext_mesh_data_attribute_count indicates the number of total attributes in the base mesh, including attributes signaled by the base mesh data substream and attributes signaled in the video substream (using ai_attribute_count). When vps_ext_mesh_data_facegroup_id_attribute_present_flag is equal to 1, it shall be greater than or equal to ai_attribute_count+1. When vps_ext_mesh_data_submesh_id_attribute_present_flag is equal to 1, it shall be greater than or equal to ai_attribute_count+1. This may be constrained by the profile / level.
[0393] Attribute types that are signaled by the base mesh substream but not by the video substream are signaled as the vps_ext_mesh_attribute_type data type.
[0394] When vps_ext_mesh_data_facegroup_id_attribute_present_flag is equal to 1, one of the vps_ext_mesh_attribute_type may be facegroup_id.
[0395] When vps_ext_mesh_data_gap_id_attribute_present_flag is equal to 1, one of the vps_ext_mesh_attribute_type may be a submesh_id.
[0396] vps_ext_mesh_data_substream_codec_id indicates the identifier of the codec used to compress the base mesh data. This codec may be identified by a profile of the component codec map SEI message or by a construct outside of this document.
[0397] vps_ext_attribute_frame_width[i] and vps_ext_attribute_frame_height[i] indicate the height and correspondence length of video data corresponding to the i-th attribute among attributes signaled in the video substream.
[0398] Grid sub-bitstream
[0399] As described above, a trellis data bitstream may be added to the encoded bitstream to facilitate V-DMC. A discussion of this bitstream is provided below.
[0400] The mesh sub-bitstream 3202 contains data for generating the base mesh to be fed to the mesh subdivision / mesh normalization process 3210. It mainly contains one or more parameter sets and one or more mesh frame layers (consisting of data units). Each data unit has a data type and size in its header. Based on the information type, the corresponding decoding engine indicated in basemesh_information() is used.
[0401] In some embodiments, information related to data type id (mi_type_id), data codec id (mi_codec_id) may be signaled in the trellis sub-bitstream as part of the trellis sequence parameter set. In addition, in some embodiments, data type and data codec id may be signaled per data unit.
[0402] When vuh_mesh_data_sample_stream_flag is true, the sub-bitstream has a sample stream format. sample_stream_nal_header() may include two values, ssnh_unit_size_precision_bytes_minus1 and ssnh_reserved_zero_5bits, as defined in Appendix D.2.1 of reference [E1], and sample_stream_mesh_nal_unit() may be the same as defined in Appendix D.2.2 of reference [E1] or in the v-mesh codec. An example of this case is as follows:
[0403]
[0404]
[0405] mesh_nal_unit() has a header and rbsp_byte. The header may be the same as 8.3.5.1nal_unit() in reference [E1] or as defined in the v-mesh codec. An example of this is as follows:
[0406]
[0407]
[0408]
[0409] mesh_nal_unit_type indicates the NAL type of the current mesh_nal_unit. It may be assigned a reserved value of the Nal unit type code (eg, Table 4 in reference [E1]) or may be defined in the v-mesh.
[0410] mesh_unit_data_type indicates the data type of mesh_nal_unit. For example, when the data unit is a sequence parameter set, mesh_unit_data_type = MESH_MSPS. When mesh_nal_unit_type indicates that the nalu type of the current mesh is a sequence parameter set, mesh_unit_data_type should be MESH_MSPS. In some embodiments, when the data unit is a decoded mesh data that can be decoded with a specified mesh codec such as Draco, it is not signaled in the case of mesh_unit_data_type = MESH_BODY, and when the data unit contains a motion vector between two meshes that can be decoded by a specified entropy codec, it is not signaled in the case of mesh_unit_data_type = MESH_MOTION. This information type must be associated with one of the mi_type_ids signaled in basemesh_information. The specified codec is determined based on mesh_unit_data_type. In some embodiments, mesh_nal_unit_header may only signal mesh_nal_unit_type. In some implementations, mesh_unit_data_type may be signaled in mesh_frame_header() instead of mesh_nal_unit_header().
[0411] A trellis sequence parameter set (MSPS) contains information about the trellis data bitstream. An example of a trellis sequence parameter set rbsp is provided below.
[0412]
[0413]
[0414] mesh_ref_list_struct is equivalent to ref_list_struct in reference [E1].
[0415] In some embodiments, some of the information signaled in basemesh_information(), such as mi_geometry_MSB_align_flag, mi_meshattribute_count, mi_meshattribute_type_id, mi_meshattribute_bit_depth_minus1, mi_meshattribute_MSB_align_flag, may be signaled in mesh_sequence_parameter_set_rbsp().
[0416] The mesh frame layer unit RSBP is signaled when mesh_unit_data_type does not indicate that the data unit is mesh_sequence_parameter_set.
[0417]
[0418] Signal mesh_frame_header on a frame-by-frame basis.
[0419] mfh_mesh_sequence_parameter_set_id indicates the id of the mesh sequence parameter set used for this mesh data unit.
[0420] mfh_frame_type indicates whether the data may require another grid data to generate the corresponding grid. For example, mfh_frame_type may be I_FRAME indicating that no other grid is required to generate the grid. mfh_frame_type may be P_FRAME or SKIP_FRAME indicating that other grids are required to generate the grid corresponding to the data unit.
[0421] In some implementations, mfh_frame_type is not signaled but derived from mesh_unit_data_type.When mesh_unit_data_type is MESH_BODY, mfh_frame_type is set to I_FRAME, and when mesh_unit_data_type is MESH_MOTION, mfh_frame_type is set to P_FRAME.
[0422] mfh_mesh_frm_order_cnt_lsb indicates the frame index of the mesh data.
[0423] mfh_num_of_reference_frame indicates the number of reference frames used for this frame.
[0424] In some implementations, mfh_num_of_reference_frame is not signaled but is set to 1.
[0425] mfh_frm_diff represents the difference between the current frame index and the reference frame index when mfh_frame_type is not I_FRAME.
[0426]
[0427] In some embodiments, the reference list structure described below may be used instead of signaling mfh_frm_diff. This implementation is identical to that in the V3C specification reference [E1].
[0428]
[0429]
[0430] mesh_frame_data has data blocks that can be decoded using the specified codec. For example, when mesh_unit_data_type=MESH_BODY and the codec specified for this data type is Draco, a data block of size ssnu_mesh_nal_unit_size–size of mesh_nal_unit_header–size of mesh_frame_header is fed to the codec and a mesh is generated. When mesh_unit_data_type=MESH_MOTION and the codec specified for this type is INTERNAL, a block of size ssnu_mesh_nal_unit_size–size of mesh_nal_unit_header–size of mesh_frame_header is decoded using the decoding process provided by the v-mesh codec.
[0431]
[0432] In some embodiments, mesh_subbitstream may be split into more than two independent sub-bitstreams. Each sub-bitstream contains only one type of mesh_nuit_data_type. Subsequently, basemesh_information() is repeated as many times as the number of sub-bitstreams that have corresponding updated information.
[0433] In some embodiments, to signal the attribute index of submesh_id and facegroup_id, the index of the element may be explicitly signaled after the current tag as exemplified below.
[0434]
[0435]
[0436] In some embodiments, vps_ext_mesh_attribute_type, facegroupId may appear only once. vps_ext_mesh_attribute_type, submeshId appears only once in the vps_ext_mesh_data_attribute_count[j]-attribute.
[0437] In some implementations, vps_ext_mesh_data_submesh_id_attribute_present_flag is not signaled, but the output of the base mesh substream decoder is a sequence of submesh frames instead of a sequence of mesh frames.
[0438] Atlas parameter set sequence extension
[0439] To support V-DMC, an atlas data sub-bitstream 3206 sequence extension may also be provided. A discussion of some possible parameters of the extension is provided below.
[0440] asps_vmesh_extension in the atlas sequence parameter set, in order to signal information related to the v-mesh in the atlas sequence parameter set, the extension flag vps_extension_present_flag should be set to 1 and (asps_extension_7bits>>N)&1 should be 1. Here, "N" is less than 7 and can be decided when V3C version 2 is finalized (reference [D2]). In the following example, the Nth bit is temporarily named asps_vmesh_extension_present_flag, and N is set to 6. asps_vmesh_extension can be signaled as follows:
[0441]
[0442]
[0443] In some embodiments, the parameters of the atlas sub-bitstream 3206 sequence extension may include:
[0444]
[0445]
[0446] asps_vmc_ext_atlas_width_displacement and asps_vmc_ext_atlas_height_displacement indicate the width and height of the atlas.
[0447] In some embodiments, asps_vmc_ext_atlas_width_displacement and asps_vmc_ext_atlas_height_displacement are not signaled, but instead are set to asps_frame_width and asps_frame_height for use.
[0448] asps_vmc_ext_prevent_geometry_video_conversion_flag prevents the output of the geometry video stream from being converted. When this flag is true, the output from Annex B in [E1] is used as is without any conversion process. When this flag is true, the size of the geometry video shall be the same as the nominal video size indicated in the bitstream.
[0449] asps_vmc_ext_prevent_attribute_video_conversion_flag prevents the output of the attribute video stream from being converted. When this flag is true, the output from Annex B in reference [E1] is used as is without any conversion process. When this flag is true, the size of the attribute video shall be the same as the nominal video size indicated in the bitstream.
[0450] In some embodiments, asps_vmc_ext_prevent_geometry_video_conversion_flag and asps_vmc_ext_prevent_attribute_video_conversion_flag may be located in a V3C parameter set.
[0451] asps_vmc_ext_geometry_3d_bitdepth indicates the bit depth of the output mesh's location.
[0452] asps_vmc_ext_coordinate_2d_bitdepth indicates the bit depth of the texture coordinates of the output mesh.
[0453] Many subdivision methods may be performed. In some embodiments, asps_vmc_ext_subdivision_method and asps_vmc_ext_subdivision_iteration_count_minus1 signal information about the subdivision method.
[0454] asps_vmc_ext_subdivision_method is given to the mesh normalization process and indicates the method used to increase the number of vertices in the base mesh. In some embodiments, when this parameter is set to 0 (or other predetermined values), the midpoint subdivision method is used. In some embodiments, when this parameter is 0, the base mesh is not modified / normalized by the mesh normalization process. In some embodiments, asps_vmc_ext_subdivision_method can indicate any resampling method to be applied to resample the vertices in the base mesh. In this case, asps_vmc_ext_subdivision_iteration_count_minus1 is possible.
[0455] asps_vmc_ext_subdivision_iteration_count_minus1 indicates the number of iterations required by the subdivision method. In some embodiments, when asps_vmc_ext_subdivision_method is 0, it is not signaled but set to 0.
[0456] asps_vmc_ext_displacement_coordinate_system indicates the coordinate system applied during the conversion process from geometry image to displacement as described above.
[0457] asps_vmc_ext_transform_index indicates a method for converting pixel values from the geometry image into displacements. For example, when set to 0, this may indicate that it is NONE and that the transform is not applied to the pixel values from the output geometry image, but rather that the values are added directly to the output of the mesh normalization process. In some embodiments, when set to 1, the transform is set to linear lifting. In such embodiments, the necessary parameters for this method may be signaled as vmc_lifting_transform_parameters.
[0458] In some embodiments, if the method is not the wavelet transform described above, the relevant variables may be signaled in the SEI message.
[0459] asps_vmc_ext_segment_mapping_method indicates how to map the segment ID to each vertex. When asps_vmc_ext_segment_mapping_method is set to 0, this may indicate that the decoded base mesh includes attributes for such ids. When asps_vmc_ext_segment_mapping_method is set to 1, this may indicate that the submesh ID is derived from the tile information in the tile. Each tile in the atlas data substream corresponds to a submesh. Otherwise, the base mesh is segmented by the method defined by the syntax element asps_vmc_ext_segment_mapping_method.
[0460] asps_vmc_ext_patch_mapping_method indicates how to map sub-portions of a base mesh or sub-mesh to tiles. When asps_vmc_ext_patch_mapping_method is equal to 0, all triangles in the segment indicated by mdu_segment_id are associated with the current tile. In this case, only one tile is associated with the segment. When asps_vmc_ext_segment_mapping_method is equal to 1, asps_vmc_ext_patch_mapping_method cannot be 0.
[0461] In some embodiments, all triangles in the segment indicated in the atlas tile header are associated with the current tile. In this case, the tile has only one tile.
[0462] When asps_vmc_ext_patch_mapping_method is equal to 1, the index corresponding to the sub-portion of the tile is explicitly signaled in the grid tile data unit.
[0463] In some implementations, when asps_vmc_ext_patch_mapping_method is equal to 2, the indices of the triangle faces corresponding to the patch are explicitly signaled in the mesh patch data unit.
[0464] In some embodiments, when asps_vmc_ext_patch_mapping_method is equal to 2, the indices of vertices corresponding to the tiles are explicitly signaled in the mesh tile data unit.
[0465] In other cases, the vertices (or triangle faces) in the segment indicated by mdu_segment_id are further segmented into sub-parts by the method indicated by asps_vmc_ext_patch_mapping_method. In this case, the i-th sub-part determined by the corresponding method is mapped to the i-th tile. Each mesh tile corresponds to only one sub-part of the base mesh.
[0466] In some embodiments, when asps_vmc_ext_patch_mapping_method>2, multiple subparts may be mapped to a shard. In this case, the i-th subpart to the (i+mdu_num_subparts[patchIndex]-1)-th subpart correspond to the patchIndex-th shard. i is the cumulative mdu_num_subparts up to patchIndex-1.
[0467] asps_vmc_ext_t_junction_removing_method indicates the method used to remove T-junctions created by different subdivision methods or by subdivision iterations of two triangles that share an edge.
[0468] asps_vmc_ext_multilocated_vertex_merge_method indicates the method used to merge multiple geometry locations of a vertex caused when the vertex is shared by two different tiles.
[0469] asps_vmc_ext_num_attribute indicates the total number of attributes carried by the corresponding mesh. Its value should be less than or equal to vps_ext_mesh_data_attribute_count.
[0470] asps_vmc_ext_attribute_type is the type of the i-th attribute and it should be one of ai_attribute_type_id or vps_ext_mesh_attribute_type.
[0471] asps_vmc_ext_direct_atrribute_projection_enabled_flag indicates that the 2d position where the attribute is projected is explicitly signaled in the grid tile data unit. Therefore, the projection id and orientation index used in reference [D4] can also be signaled.
[0472] In some implementations, asps_vmc_ext_transform_index is not signaled but is always set to the wavelet transform described above.
[0473] To perform the transformation, in some embodiments, the transformation parameters may be provided as follows:
[0474]
[0475] As illustrated in the figure, in the current implementation, asps_vmc_extension_transform_parameters includes the following parameters:
[0476] asps_vmc_ext_lifting_skip_update,
[0477] asps_vmc_ext_lifting_quantization_parameters_x、
[0478] asps_vmc_ext_lifting_quantization_parameters_y、
[0479] asps_vmc_ext_lifting_quantization_parameters_z、
[0480] asps_vmc_ext_log2_lifting_lod_inverseScale_x,
[0481] asps_vmc_ext_log2_lifting_lod_inverseScale_y,
[0482] asps_vmc_ext_log2_lifting_lod_inverseScale_z,
[0483] asps_vmc_ext_log2_lifting_update_weight,
[0484] asps_vmc_ext_log2_lifting_prediction_weight, which are transformation related variables.
[0485] In some embodiments, asps_vmc_extension_transform_parameters may be signaled persistently.In some embodiments, asps_vmc_extension_transform_parameters may be signaled only when asps_vmc_ext_transform_index indicates that the method used for conversion is wavelet transform as described above.
[0486] asps_vmc_ext_num_attribute indicates the number of attributes to which the transformation is applied in addition to the geometry image.
[0487] asps_vmc_ext_attribute_type indicates the attribute type to which the following transformation applies.
[0488] asps_vmc_ext_attribute_transform_index indicates the transformation applied to the attribute type asps_vmc_ext_attribute_type.
[0489] asps_vmc_extension_transform_parameters_present_flag indicates that asps_vmc_extension_transform_parameters() is signaled for the attribute type. If not, the flag is false and the values are copied from the previously signaled attribute type. In some embodiments, the attribute type asps_vmc_extension_transform_parameters() copied from can be explicitly signaled.
[0490] In some embodiments, all syntax elements except asps_vmc_ext_geometry_3d_bitdepth and asps_vmc_ext_coordinate_2d_bitdepth may be signaled in a V3C parameter set or v3c unit header. In some embodiments, all syntax elements except asps_vmc_ext_geometry_3d_bitdepth and asps_vmc_ext_coordinate_2d_bitdepth may be signaled in an atlas_frame_parameter_set.
[0491] Some syntax elements in the atlas sequence parameter set may be overridden by the same syntax elements in the atlas frame parameter set. When the first bit in afps_extension_8bits is 1, afps_vmc_extension() is signaled.
[0492] In some embodiments, the parameters of the atlas sub-bitstream 3206 sequence extension may include:
[0493]
[0494] Lifting transformation parameters can include:
[0495]
[0496] In some embodiments, asps_vmc_ext_attribute_transform_index, asps_vmc_extension_transform_parameters_present_flag, vmc_lifting_transform_parameter, and asps_vmc_ext_direct_atrribute_projection_enabled_flag may be signaled only for attributes signaled through the video stream.
[0497] Atlas parameter set 3206 frame extension
[0498] To support V-DMC, atlas data sub-bitstream 3206 frame extension (afps) may also be provided.
[0499] A discussion of some possible parameters for the extension is provided below.
[0500]
[0501]
[0502]
[0503]
[0504] afps_vmc_ext_direct_attribute_projection_enabled indicates that direct attribute projection can be used.
[0505] afps_vmc_ext_overriden_flag indicates any additional information to be signaled to override syntax elements in ASPS.
[0506] afps_vmc_ext_subdivision_enable_flag indicates to use afps_vmc_ext_subdivision_method and afps_vmc_ext_subdivision_iteration_count_minus1 instead of asps_vmc_ext_subdivision_method and asps_vmc_ext_subdivision_iteration_count_minus1.
[0507] afps_vmc_ext_displacement_coordinate_system_enable_flag indicates the use of afps_vmc_ext_displacement_coordinate_system instead of asps_vmc_ext_displacement_coordinate_system.
[0508] afps_vmc_ext_transform_index_enable_flag indicates the use of afps_vmc_ext_transform_index instead of asps_vmc_ext_transform_index.
[0509] afps_vmc_ext_transform_parameters_enable_flag indicates that afps_vmc_extension_transform_parameters() is signaled to be used instead of asps_vmc_extension_transform_parameters().
[0510] afps_vmc_ext_num_attribute_enable_flag indicates that the afps_vmc_ext_num_attribute attribute uses the override parameters.
[0511] afps_vmc_ext_attribute_type indicates the attribute type.
[0512] afps_vmc_ext_attribute_transform_index_enable_flag indicates the use of afps_vmc_ext_attribute_transform_index instead of asps_vmc_ext_attribute_transform_index of the corresponding attribute type.
[0513] afps_vmc_ext_attribute_transform_parameters_enable_flag indicates that afps_vmc_extension_transform_parameters() is signaled to be used instead of asps_vmc_extension_transform_parameters() for the corresponding attribute type.
[0514] In some embodiments, all parameters may be always signaled without an enabled flag.
[0515]
[0516]
[0517] The afps_vmc_ext_transform_lifting_quantization_parameters_enable_flag indicates to signal the use of afps_vmc_ext_transform_lifting_quantization_parameters_x, afps_vmc_ext_transform_lifting_quantization_parameters_y, and afps_vmc_ext_transform_lifting_quantization_parameters_z instead of asps_vmc_ext_transform_lifting_quantization_parameters_x, asps_vmc_ext_transform_lifting_quantization_parameters_y, and asps_vmc_ext_transform_lifting_quantization_parameters_z.
[0518] The afps_vmc_ext_transform_log2_lifting_lod_inverseScale_enable_flag indicates to signal the use of afps_vmc_ext_transform_log2_lifting_lod_inverseScale_x, afps_vmc_ext_transform_log2_lifting_lod_inverseScale_y, and afps_vmc_ext_transform_log2_lifting_lod_inverseScale_z instead of asps_vmc_ext_transform_log2_lifting_lod_inverseScale_x, asps_vmc_ext_transform_log2_lifting_lod_inverseScale_y, and asps_vmc_ext_transform_log2_lifting_lod_inverseScale_z.
[0519] afps_vmc_ext_transform_log2_lifting_update_weight_enable_flag indicates signaling afps_vmc_ext_transform_log2_lifting_update_weight to be used instead of asps_vmc_ext_transform_log2_lifting_update_weight.
[0520] afps_vmc_ext_transform_log2_lifting_prediction_weight_enable_flag indicates signaling afps_vmc_ext_transform_log2_lifting_prediction_weight to be used instead of asps_vmc_ext_transform_log2_lifting_prediction_weight.
[0521] Use afps_vmc_ext_transform_lifting_skip_update instead of asps_vmc_ext_transform_lifting_skip_update.
[0522] In some embodiments, all parameters may be always signaled without an enabled flag.
[0523] In some embodiments, the atlas data sub-bitstream 3206 frame extension may include:
[0524]
[0525]
[0526] afps_vmc_ext_single_segment_in_frame_flag indicates that there is only one segment for the atlas frame.
[0527] afps_vmc_ext_single_attribute_tile_in_frame_flag indicates that there is only one tile for each attribute signaled in the video stream.
[0528] In some implementations, afps_vmc_ext_single_attribute_tile_in_frame_flag is signaled only when afti_single_tile_in_atlas_frame_flag is not true. When afti_single_tile_in_atlas_frame_flag is true, afps_vmc_ext_single_attribute_tile_in_frame_flag is inferred to be true.
[0529] In some embodiments, the tile mapping method may be signaled in the afps_vmc_extension.
[0530] In some embodiments, signaling a shard mapping method overrides a flag, and only when the flag is true is the shard mapping method signaled. In this case, the shard mapping method is used instead of asps_vmc_ext_patch_mapping_method.
[0531] When afps_vmc_ext_overriden_flag in afps_vmc_extension() is true, the subdivision method, displacement coordinate system, transform index, transform parameter, and attribute transform parameter may be signaled again, and this information may override the information signaled in asps_vmc_extension().
[0532] In some implementations, afps_vmc_ext_displacement_coordinate_system_enable_flag is not signaled, but afps_vmc_ext_displacement_coordinate_system is always signaled.
[0533] The tile information of the attribute signaled by the video substream may be provided in afps_ext_vmc_attribute_tile_information() as follows:
[0534]
[0535] Shard Data Unit
[0536] Like V-PCC slice data units, mesh slice data units are signaled in the atlas data sub-bitstream 3206. Mesh intra-slice data units, mesh inter-slice data units, mesh merged slice data units, and mesh skipped slice data units may be used. A discussion of these is provided below.
[0537] ath_type=SKIP_PLUS_TILE indicates that all but a few tiles are copied from the tile's reference tile. The tile indices that are not copied are explicitly signaled in atlas_tile_data_unit. The atlas_tile_data_unit also specifies whether any new tiles are added.
[0538] In some embodiments, SKIP_PLUS_TILE only copies tiles for mesh information, and does not copy tiles of tile type P_SKIP, P_MERGE, P_INTRA, P_INTER, P_RAW, P_EOM, I_INTRA, I_RAW, or I_EOM.
[0539] In some embodiments, SKIP_PLUS_TILE only copies tiles for mesh information, but does not copy tiles whose information is related to explicit geometry or attribute locations, temporarily named RAW_MESH.
[0540]
[0541]
[0542]
[0543] atdu_patch_data_present_flag indicates the presence of signaled patch data (patch_information_data()) in this tile.
[0544] atdu_num_deleted_patchgroups indicates the number of shard groups that were not copied from the reference chunk. Each shard has a group index, and shards with the same group index are considered to be in the same group.
[0545] atdu_deleted_patchgroup_idx indicates the index of the patch group that was not copied. When atdu_num_deleted_patchgroups is not 0, the list is derived from the patch index in the patch group.
[0546] atdu_num_deleted_patches indicates the number of patches that were not copied from the reference patch. When atdu_num_deleted_patchgroups is not 0, atdu_num_deleted_patches needs to be derived by counting the number of patches in each patch group.
[0547] atdu_deleted_patch_idx indicates the patch index that was not replicated.
[0548] In some embodiments, these indexes can be recalculated by reordering the tiles. For example, when a MESH_RAW is located between two non-MESH_RAW mesh tiles, the MESH_RAW tile can be deleted for index calculation.
[0549] Looking at the fragments in the reference chunk, if the fragment index is one of the fragment indexes in atdu_deleted_patchgroup_idx, the fragment is copied via skip_patch_data_unit().
[0550] If atdu_patch_data_present_flag is true (which indicates that more tile information is present in the tile), patch_information_data() is signaled, and when atdu_num_deleted_patches is not 0, the tile index of the subsequent tile starts with RefAtduTotalNumPatches[tileID]-atdu_num_deleted_patches.
[0551] atdu_patchgroup_index indicates the patch group index of the following atdu_group_information. PATCHGROUP_END indicates that there is no more atdu_group_information.
[0552] In some implementations, atdu_patchgroup_index is not signaled but is always set to PATCHGROUP_END.
[0553] atdu_patchgroup_information has common information that can be shared by all shards with the same shard group index.
[0554]
[0555]
[0556] atdu_patchgroup_overriden_flag indicates any additional information to be signaled to override syntax elements in the corresponding AFPS and / or in the corresponding ASPS.
[0557] atdu_patchgroup_subdivision_enable_flag indicates the use of atdu_patchgroup_subdivision_method and atdu_patchgroup_subdivision_iteration_count_minus1 instead of asps_vmc_ext_subdivision_method and asps_vmc_ext_subdivision_iteration_count_minus1 or afps_vmc_ext_subdivision_method and afps_vmc_ext_subdivision_iteration_count_minus1.
[0558] atdu_patchgroup_displacement_coordinate_system_enable_flag indicates the use of atdu_patchgroup_displacement_coordinate_system instead of asps_vmc_ext_displacement_coordinate_system.
[0559] atdu_patchgroup_transform_index_enable_flag indicates to use atdu_patchgroup_transform_index instead of asps_vmc_ext_transform_index or afps_vmc_ext_transform_index.
[0560] atdu_patchgroup_transform_parameters_enable_flag indicates that atdu_patchgroup_transform_parameters() is signaled to be used instead of asps_vmc_extension_transform_parameters() or afps_vmc_extension_transform_parameters().
[0561] atdu_patchgroup_num_attribute_enable_flag indicates that the atdu_patchgroup_num_attribute attribute uses the override parameter.
[0562] atdu_patchgroup_attribute_type indicates the attribute type.
[0563] atdu_patchgroup_attribute_transform_index_enable_flag indicates the use of atdu_patchgroup_attribute_transform_index instead of asps_vmc_ext_attribute_transform_index or afps_vmc_ext_attribute_transform_index of the corresponding attribute type.
[0564] atdu_patchgroup_attribute_transform_parameters_enable_flag indicates that atdu_transform_parameters() is signaled to be used instead of asps_vmc_extension_transform_parameters() or afps_vmc_extension_transform_parameters() for the corresponding attribute type.
[0565] In some embodiments, all parameters may be always signaled without an enabled flag.
[0566]
[0567]
[0568] atdu_patchgroup_transform_lifting_quantization_parameters_enable_flag indicates that atdu_patchgroup_transform_lifting_quantization_parameters_x, atdu_patchgroup_transform_lifting_quantization_parameters_y, and atdu_patchgroup_transform_lifting_quantization_parameters_z are signaled to be used instead of the corresponding syntax elements signaled in the ASPS and / or in the AFPS.
[0569] atdu_patchgroup_transform_log2_lifting_lod_inverseScale_enable_flag indicates that atdu_patchgroup_transform_log2_lifting_lod_inverseScale_x, atdu_patchgroup_transform_log2_lifting_lod_inverseScale_y, and atdu_patchgroup_transform_log2_lifting_lod_inverseScale_z are signaled to be used instead of the corresponding syntax elements signaled in the ASPS and / or in the AFPS.
[0570] atdu_patchgroup_transform_log2_lifting_update_weight_enable_flag indicates that atdu_patchgroup_transform_log2_lifting_update_weight is signaled to be used instead of the corresponding syntax element signaled in the ASPS and / or in the AFPS.
[0571] atdu_patchgroup_transform_log2_lifting_prediction_weight_enable_flag indicates that atdu_patchgroup_transform_log2_lifting_prediction_weight is signaled to be used instead of the corresponding syntax element signaled in the ASPS and / or in the AFPS.
[0572] Use atdu_patchgroup_transform_lifting_skip_update instead of asps_vmc_ext_transform_lifting_skip_update or afps_vmc_ext_transform_lifting_skip_update. In some embodiments, all parameters may always be signaled without enabling the flag.
[0573] The slicing mode, P_MESH, M_MESH, RAW_MESH, and I_MESH indicate that the slicing mode contains the information of the v-mesh.
[0574] In one embodiment, the implementation of signaling a mesh shard data unit may include:
[0575]
[0576] In some embodiments, specific implementations of signaling a mesh shard data unit may include:
[0577]
[0578]
[0579] Data Units in Grid Shards
[0580] The mesh_intra_data_unit has information about the connected geometry video, texture video, and base mesh. These values are given to the mesh normalization process, geometry normalization process, and attribute normalization process. In one embodiment, the mesh intra-data unit can be implemented as follows:
[0581]
[0582]
[0583]
[0584]
[0585] mdu_patchgroup_index indicates the group index of the patch.
[0586] mdu_patch_parameters_enable_flag indicates whether certain parameters are copied from atdu_pathgroup_information. In some implementations, mdu_patch_parameters_enable_flag is not signaled but is always set to true.
[0587] mdu_geometry_2d_pos_x and mdu_geometry_2d_pos_y indicate the upper left corner of the corresponding area in the geometry video frame.
[0588] mdu_geometry_2d_size_x_minus1 and mdu_geometry_2d_size_y_minus1 indicate the size of the corresponding area in the geometry video frame.
[0589] mdu_attributes_2d_pos_x and mdu_attributes_2d_pos_y indicate the upper left corner of the corresponding area in the attribute video frame.
[0590] mdu_attributes_2d_size_x_minus1 and mdu_attributes_2d_size_y_minus1 indicate the size of the corresponding region in the attribute video frame. In some embodiments, mdu_attributes_2d_pos_x, mdu_attributes_2d_pos_y, mdu_attributes_2d_size_x_minus1 and mdu_attributes_2d_size_y_minus1 may be signaled only if vmc_ext_direct_attribute_projection_enabled is true.
[0591] mdu_3d_offset_u, mdu_3d_offset_v, and mdu_3d_offset_d indicate the offset of the corresponding 3D space. In some embodiments, these three values may be signaled in a SEI message.
[0592] mdu_3d_range_d specifies the nominal maximum value of the offset expected to be present in the reconstructed bit-depth slice geometry samples. In some embodiments, mdu_3d_range_d may be signaled in a SEI message.
[0593] mdu_vertex_count_minus1 indicates the number of vertices corresponding to this tile in the normalized mesh.
[0594] mdu_triangle_count_minus1 indicates the number of triangles corresponding to the slice in the normalized grid. In some embodiments, mdu_vertex_count_minus1 and / or mdu_triangle_count_minus1 can be the number in the base grid. In some embodiments, mdu_vertex_count_minus1 and / or mdu_triangle_count_minus1 can be signaled in the SEI message. In some embodiments, mdu_vertex_count_minus1 and mdu_triangle_count_minus1 can be derived from mdu_vertex_index_list with or without mdu_subdivision_iteration_count.
[0595] mdu_head_vertex_index indicates the index of the first vertex corresponding to the tile in the normalized mesh. It is the minimum vertex index among the vertices corresponding to the tile.
[0596] mdu_num_sequential_vertex_index indicates the number of vertices whose indexes are consecutive from mdu_head_vertex_index.
[0597] mdu_vertex_index_diff indicates the difference between the vertex indices in mdu_vertex_index_list.
[0598] mdu_vertex_index_list lists the vertex indices corresponding to the slices. It can be derived from mdu_head_vertex_index, mdu_num_sequential_vertex_index, and mdu_vertex_index_diff. For example, the list can be set as follows:
[0599]
[0600] In some embodiments, mdu_head_vertex_index can be the index of the first vertex corresponding to the slice in the base grid. mdu_vertex_index_list is a list of vertex indexes corresponding to the slice in the base grid. In this case, when mdu_vertex_count_minus1 indicates the number of corresponding vertices in the normalized grid, mdu_vertex_index_list can be increased to the size of (mdu_vertex_count_minus1+1) during the mesh normalization process. In some embodiments, the total size of mdu_vertex_index_list can be derived during the mesh normalization process.
[0601] In some embodiments, the tile with index p corresponds to the pth connected component of the base mesh. The mdu_vertex_index_list is derived from this connected component without signaling mdu_head_vertex_index, mdu_num_sequential_vertex_index, or mdu_vertex_index_diff.
[0602] In some embodiments, the tile with index p corresponds to the mdu_cc_indexth connected component of the base mesh. mdu_cc_index is signaled, and mdu_vertex_index_list is derived from this connected component without signaling mdu_head_vertex_index, mdu_num_sequential_vertex_index, or mdu_vertex_index_diff.
[0603] In some embodiments, the index (p) used to find the corresponding connected component of the base mesh can be derived by not including non-mesh related tiles.
[0604] In some embodiments, all parameters for finding connected components of a mesh may be delivered in a v-mesh extension in an atlas sequence parameter set.
[0605] In some embodiments, the tile with index p corresponds to the pth connected component of the normalized mesh. The mdu_vertex_index_list is derived from this connected component without signaling mdu_head_vertex_index, mdu_num_sequential_vertex_index, or mdu_vertex_index_diff.
[0606] Similar to reference [E1], mdu_projection_id indicates the value of the projection mode and the value of the index of the normal of the projection plane of the tile.
[0607] Similar to reference [E1], mdu_orientation_index indicates the slice orientation index.
[0608] mdu_lod_enabled_flag indicates that LOD parameters exist for the current slice p.
[0609] Similar to reference [E1], mdu_lod_scale_x_minus1 and mdu_lod_scale_y_idc indicate the scaling factors for the x and y coordinates.
[0610] In some implementations, mdu_projection_id, mdu_orientation_index, mdu_lod_enabled_flag, mdu_lod_scale_x_minus1, and mdu_lod_scale_y_idc may be signaled in atdu_patchgroup_information and overwritten in the mesh_intra_data_unit.
[0611] mdu_subdivision_enable_flag indicates the use of mdu_subdivision_method and mdu_subdivision_iteration_count_minus1 instead of the corresponding syntax elements signaled in ASPS, in AFPS and / or atdu_patchgroup_information.
[0612] mdu_displacement_coordinate_system_enable_flag indicates that mdu_displacement_coordinate_system is signaled to be used instead of the corresponding syntax element signaled in ASPS, in AFPS and / or atdu_patchgroup_information.
[0613] mdu_transform_index_enable_flag indicates that mdu_transform_index is signaled to be used instead of the corresponding syntax element signaled in ASPS, in AFPS and / or atdu_patchgroup_information.
[0614] mdu_transform_parameters_enable_flag indicates that mdu_transform_parameters() is signaled to be used instead of the corresponding syntax elements signaled in ASPS, in AFPS and / or atdu_patchgroup_information.
[0615] mdu_num_attribute_enable_flag indicates that the mdu_num_attribute attribute uses overridden parameters.
[0616] mdu_attribute_type indicates the attribute type.
[0617] mdu_attribute_transform_index_enable_flag indicates that mdu_attribute_transform_index is signaled to be used instead of the corresponding syntax element signaled in ASPS, in AFPS and / or atdu_patchgroup_information.
[0618] mdu_attribute_transform_parameters_enable_flag indicates that mdu_transform_parameters() is signaled to be used instead of the corresponding syntax elements signaled in ASPS, in AFPS and / or atdu_patchgroup_information.
[0619] In some embodiments, without enabling the flag, all parameters may always be signaled. The following is an embodiment of a specific implementation of mdu_transform_parameters().
[0620]
[0621]
[0622] mdu_transform_lifting_quantization_parameters_enable_flag indicates that mdu_transform_lifting_quantization_parameters_x, mdu_transform_lifting_quantization_parameters_y, and mdu_transform_lifting_quantization_parameters_z are signaled to be used instead of the corresponding syntax elements signaled in ASPS, in AFPS, and / or atdu_patchgroup_information.
[0623] mdu_transform_log2_lifting_lod_inverseScale_enable_flag indicates that mdu_transform_log2_lifting_lod_inverseScale_x, mdu_transform_log2_lifting_lod_inverseScale_y, and mdu_transform_log2_lifting_lod_inverseScale_z are signaled to be used instead of the corresponding syntax elements signaled in ASPS, in AFPS, and / or in atdu_patchgroup_information.
[0624] mdu_transform_log2_lifting_update_weight_enable_flag indicates that mdu_transform_log2_lifting_update_weight is signaled to be used instead of the corresponding syntax element signaled in ASPS, in AFPS and / or atdu_patchgroup_information.
[0625] mdu_transform_log2_lifting_prediction_weight_enable_flag indicates that mdu_transform_log2_lifting_prediction_weight is signaled to be used instead of the corresponding syntax element signaled in ASPS, in AFPS and / or atdu_patchgroup_information.
[0626] mdu_transform_lifting_skip_update is used instead of the corresponding syntax elements signaled in ASPS, in AFPS and / or atdu_patchgroup_information.
[0627] In some embodiments, all parameters may be always signaled without an enabled flag.
[0628] An embodiment of a specific implementation of data units within a grid shard is provided below.
[0629]
[0630]
[0631]
[0632] In the current implementation, mdu_segment_id indicates the segment ID associated with the current fragment.
[0633] When asps_vmc_ext_segment_mapping_method is equal to 0, the associated segment (a set of connected vertices) is the union of vertices with submeshId equal to mdu_segment_id and associated information (connectivity and / or attributes). In this case, vps_ext_mesh_data_submesh_id_attribute_present_flag should be true and one of the vps_ext_mesh_attribute_types is submeshId.
[0634] When asps_vmc_ext_segment_mapping_method is equal to 1, the associated segment is derived from the shard information. The segment is the union of the vertices of the shards mapped into one block.
[0635] Otherwise, the associated segment is the mdu_segment_idth segment determined by asps_vmc_ext_segment_mapping_method.
[0636] In some embodiments, when the output of the base grid sub-stream decoder is a sequence of sub-grid frames, one output of the base grid sub-stream decoder is mapped to one mdu_segment_id.
[0637] mdu_vertex_count_minus1 and mdu_triangle_count_minus1 indicate the number of vertices and triangles associated with the current fragment.
[0638] When asps_vmc_ext_patch_mapping_method is equal to 0, all triangles in the segment indicated by mdu_segment_id are associated with the current tile. In this case, only one tile is associated with the segment. When asps_vmc_ext_segment_mapping_method is equal to 1, asps_vmc_ext_patch_mapping_method cannot be equal to 0.
[0639] In some embodiments, all triangles in more than one fragment indicated in the atlas tile header are associated with the current tile. In this case, the tile has only one tile.
[0640] The syntax elements mdu_num_subparts and mdu_subpart_id are signaled when asps_vmc_ext_patch_mapping_method is equal to 1. The related triangle faces are the union of the triangle faces with face group id equal to mdu_subpart_id.
[0641] In some cases, the associated triangular faces are the union of the triangular faces in the mdu_subpart_idth subpart as determined by asps_vmc_ext_patch_mapping_method.
[0642] When mdu_patch_parameters_enable_flag is true, the tessellation method, displacement coordinate system, transform index, transform parameters, and attribute transform parameters may be signaled again, and this information overwrites the corresponding information signaled in asps_vmc_extension().
[0643] In some implementations, mdu_displacement_coordinate_system_enable_flag is not signaled, but mdu_displacement_coordinate_system is always signaled.
[0644] In some embodiments, the segment id may be signaled in the atlas_tile_header, and the segment id for all tiles in the atlas tile is the same as the segment id. An example of this implementation is provided below.
[0645]
[0646] In some embodiments, a tile may be associated with more than one grid segment. In this case, each slice belonging to the tile has a segment_id.
[0647] In some embodiments, in a mesh slice data unit, the mdu_segment_id may indicate the order of appearance rather than the segment_id itself.
[0648]
[0649] In some embodiments, the tile data unit may not contain 2d_pos_x, 2d_pos_y, 2d_size_x_minus1 or 2d_size_y_minus1. The 2D position and 2D size are signaled once in the atlas tile header. In this case, all tiles in an atlas tile have the same fragment id.
[0650]
[0651] Inter-shard data unit
[0652] Turning now to the discussion of inter-shard data units, embodiments of specific implementations of mesh inter-shard data units are provided below.
[0653]
[0654]
[0655] The slice mode P_MESH indicates that the slice is predicted from the reference slice. Syntax elements that are not signaled are copied from the reference slice.
[0656] midu_ref_index indicates the reference index of the reference partition.
[0657] midu_patch_index indicates the reference index of the reference patch.
[0658] In some embodiments, midu_patch_index indicates a recomputed patch index derived without non-mesh related patches and / or RAW_MESH patches.
[0659] midu_geometry_2d_pos_x and midu_geometry_2d_pos_y indicate the upper left corner of the corresponding area in the geometry video frame.
[0660] midu_geometry_2d_delta_size_x and midu_geometry_2d_delta_size_y indicate the size difference between the corresponding region and the region corresponding to the reference tile in the geometry video frame.
[0661] midu_attributes_2d_pos_x and midu_attributes_2d_pos_y indicate the upper left corner of the corresponding area in the attribute video frame.
[0662] midu_attributes_2d_size_x_minus1 and midu_attributes_2d_size_y_minus1 indicate the size difference between the corresponding region and the region corresponding to the reference slice in the attribute video frame.
[0663] In some implementations, midu_attributes_2d_pos_x, midu_attributes_2d_pos_x, midu_attributes_2d_delta_size_x, and mdu_attributes_2d_delta_size_y may be signaled only if vmc_ext_direct_attribute_projection_enabled is true.
[0664] midu_3d_offset_u, midu_3d_offset_v, and midu_3d_offset_d indicate offsets of the corresponding 3D space.
[0665] In some implementations, these three values may be signaled in a SEI message.
[0666] midu_3d_range_d specifies the nominal maximum value of the offset expected to be present in the reconstructed bit-depth tile geometry samples.
[0667] In some implementations, mdu_3d_range_d may be signaled in a SEI message.
[0668] Another embodiment of a specific implementation of an inter-grid data unit is provided below.
[0669]
[0670] Grid Merge Shard Data Unit
[0671] The tile mode M_MESH indicates that the tiles are copied from the reference frame, but some of the information is overwritten. An example of a mesh merge data unit is provided below.
[0672]
[0673] Grid Skip
[0674] The mesh skip sharding mode indicates that data units should be skipped. An embodiment of a specific implementation of this mode is provided below.
[0675] mesh_skip_data_unit(tileID,patchIdx){ Descriptors }
[0676] Original grid data unit
[0677] The original grid slice mode indicates that the data unit contains explicit information about the location and attributes. The following provides an embodiment of the specific implementation of the grid skip / grid original slice data unit.
[0678]
[0679] rmdu_patch_in_auxiliary_video_flag indicates whether the geometry and attribute data associated with the patch are coded in the auxiliary video sub-bitstream.
[0680] rmdu_geometry_2d_pos_x and rmdu_geometry_2d_pos_y indicate the upper left corner of the corresponding area in the geometry video frame.
[0681] rmdu_geometry_2d_size_x_minus1 and rmdu_geometry_2d_size_y_minus1 indicate the size of the corresponding area in the geometry video frame.
[0682] rmdu_attributes_2d_pos_x and rmdu_attributes_2d_pos_y indicate the upper left corner of the corresponding area in the attribute video frame.
[0683] rmdu_attributes_2d_size_x_minus1 and rmdu_attributes_2d_size_y_minus1 indicate the size of the corresponding area in the attribute video frame.
[0684] In some implementations, rmdu_attributes_2d_pos_x, rmdu_attributes_2d_pos_y, rmdu_attributes_2d_size_x_minus1, and rmdu_attributes_2d_size_y_minus1 may be signaled only if vmc_ext_direct_attribute_projection_enabled is true.
[0685] rmdu_3d_offset_u, rmdu_3d_offset_v, and rmdu_3d_offset_d indicate the offset of the corresponding 3D space.
[0686] In some implementations, these three values may be signaled in a SEI message.
[0687] rmud_head_vertex_index indicates the index of the first vertex corresponding to the tile. In some embodiments, this value is not signaled, but the output of the tile is appended to the end of the position list of the corresponding mesh.
[0688] rmdu_vertex_count_minus1 indicates the number of vertices corresponding to this fragment.
[0689] rmdu_triangle_count_minus1 indicates the number of triangles corresponding to this tile.
[0690] In some embodiments, a mesh_raw_patch_data_unit may be divided into two tile data units: a mesh_raw_geometry_patch_data_unit and a mesh_raw_attribute_patch_data_unit.
[0691] Another embodiment of a specific implementation of the original mesh slicing mode using "mrdu" instead of "rmdu" is provided below.
[0692]
[0693]
[0694] Normalization
[0695] Having discussed the demultiplexed sub-bitstreams and bitstream syntax elements of the encoded bitstream representing 3D texture static and / or dynamic meshes, the discussion now turns to the decoder normalization process. As described above, the decoder normalization process includes mesh normalization, geometry normalization, and attribute normalization.
[0696] Starting with mesh normalization, as described above, the base mesh obtained by decoding the mesh sub-bitstream 3203 demultiplexed from the coded bitstream can be normalized via the mesh normalization process 3210. In the mesh normalization process 3210, the output of the mesh sub-bitstream 3203 is processed to be added to the output of the geometry normalization process 3212. The input to this process is the output mesh of the mesh sub-bitstream 3203, the vmc_ext_subdivision_method and vmc_ext_subdivision_iteration_count from the v-mesh extension in the atlas sequence parameters, and the tile information. The output is meshes, and the total number of vertices in these meshes is the same as the total number of displacements generated from the geometry normalization process 3212, unless otherwise specified by the v-mesh codec.
[0697] When vmc_ext_subdivision_method is 0 or (mdu_patch_subdivisionmethod_enable_flag is true and mdu_subdivision_method_index is 0), no additional process is applied to the corresponding area of the base mesh. The corresponding area of the normalized mesh is the same as that of the base mesh.
[0698] When vmc_ext_subdivision_method is not 0 or (mdu_patch_subdivisionmethod_enable_flag is true and mdu_subdivision_method_index is not 0), the corresponding area of the mesh is filled with vertices by the method indicated by vmc_ext_subdivision_method or mdu_subdivision_method_index in the manner described above.
[0699] Fig.33 and Fig.34 Examples of the input to this normalization process and its output are shown. Fig.33 An input grid 3300 is illustrated as being provided as input to the grid normalization process 3210 .
[0700] Fig.34 The output 3400 of the grid normalization process 3210 is illustrated. Fig.34 As shown, the number of vertices is increased via a mesh normalization process. This results in a more refined 3D mesh, as illustrated in output 3400.
[0701] Based on the tile information associated with the regions in the grid, different subdivision methods can be applied. For example, Fig.35An example 3500 is illustrated, in which a left portion 3501, a right portion 3502, and a head portion 3503 each correspond to a different shard (e.g., shard 0, shard 1, and shard 2, respectively). Each portion can be subdivided differently (e.g., as indicated by the corresponding shard information), thereby generating a different number of vertices and thus causing refinement. For example, in the example 3500 in which only the subdivision iteration count of each of the three portions (e.g., the left portion 3501, the right portion 3502, and the head portion 3503) is changed, different numbers of vertices and refinement for different portions of the 3D mesh are provided. As illustrated, shard 2 can specify that fewer subdivision iterations are desired than shard 0 and / or shard 1, thereby generating a head portion 3503 with a lower degree of refinement having fewer vertices. In addition, shard 1 can indicate that more detailed refinement is permitted, thereby causing a relatively higher number of subdivision iterations. This causes more vertices to be generated in the right portion 3502, thereby causing relatively more refinement in that portion of the 3D mesh.
[0702] Having discussed the inputs and outputs of the mesh normalization process 3210, the discussion now turns to an example of the subdivision process described herein. Fig.36 and Fig.37 A simplified example of the current process is provided. First, from Fig.36 Initially, assume that v0, v1, and v3 are connected (cc0), v0, v1, and v2 are connected (cc1), and v2, v1, and v4 are connected (cc2). The connected components correspond to shards, shards[0], shards[1], and shards[2], respectively.
[0703] In this example, vmc_ext_subdivision_method = 1 and vmc_ext_subdivision_iteration_count = 2. In addition:
[0704] 1) For patch [0], mdu_patch_subdivisionmethod_enable_flag is true, and mdu_subdivision_method_index is 1 and mdu_subdivision_iteration_count is 1. Then, Fig.37 As illustrated in tile[0] of , fill the region corresponding to the tile (the triangle constructed by v0, v1, and v3) with vertices by using the method with index 1 and iteration count 1. mdu_vertex_index_list is set to {v0,v1,v3,v5,v6,v7}. The order of vertex indices can be determined by the subdivision method, and the order of the corresponding displacements is consistent with that order.
[0705] 2) For fragmentation [1], mdu_patch_subdivisionmethod_enable_flag can be false. Therefore, the subdivision method is set to vmc_ext_subdivision_method, and mdu_subdivision_iteration_count is set to vmc_ext_subdivision_iteration_count. Fig.37 As shown, the area corresponding to the tile (the triangle constructed by v0, v1 and v2) is filled with vertices accordingly. mdu_point_index_list is set to {v0,v1,v2,v5,v7,v8,v9,v10,v11,v12,v13,v14,v15,v16,v17}. The order of vertex indices can be determined by the subdivision method, and the order of the corresponding displacements is consistent with this order.
[0706] 3) For fragmentation [2], mdu_patch_subdivisionmethod_enable_flag is true and mdu_subdivision_method_index is 0. Therefore, Fig.37 As illustrated, the region corresponding to the tile (the triangle constructed by v2, v1, and v4) remains the same as the input. mdu_vertex_index_list is set to {v2, v1, v4}. The order of vertex indices can be determined by the subdivision method, and the order of the corresponding displacements is consistent with this order.
[0707] Thus, the mesh normalization process 3210 can cause different numbers of subdivisions, and thus produce dynamically adjustable numbers of vertices for different tiles within the mesh. In this way, the mesh normalization process can provide significant flexibility in determining the level of refinement associated with each tile (or subgroup of tiles) of the 3D mesh.
[0708] Having discussed the mesh normalization process 3210, the discussion now turns to the geometry normalization process 3212. As described above, the geometry image decoded from the geometry sub-bitstream 3204 and demultiplexed from the encoded bitstream can be normalized via the geometry normalization process 3212. In the geometry normalization process 3212, the output of the geometry video sub-bitstream 3204 is processed to be added to the output of the mesh normalization process 3210. The inputs to this process 3212 are the output geometry image of the geometry video sub-bitstream 3204, the v-grid extension in the atlas data sub-bitstream 3206, and the tile information (e.g., from the parameter set 3202). The pixel values in the corresponding areas in the geometry image can be converted by the method described above in Part 1. The converted values (i.e., displacement values) are added to the corresponding vertices as indicated in the tile. If mdu_transform_index is NONE, the pixel value of the nth pixel in the region is added to the position of the vertex with the nth index in mdu_vertex_index_list without any transformation. Otherwise, the displacement generated from the nth pixel value is added to the position of the vertex whose index is the nth index in mdu_vertex_index_list.
[0709] Fig.38 A brightness plane 3800 of a geometric structure image is illustrated, and Fig.39 An example of a geometric structure image is illustrated. In this example, the size of the image is 256x48. The figure shows only one of the 3 planes. The first few pixel triplets of the plane are described. Even in this case, the attribute image can be 2048x2048.
[0710]
[0711] For each shard (for example, Fig.38 The pixel values in the geometry image of tile 0 in ) are converted into displacements and added to the corresponding areas in the base mesh (in this example, Fig.35 The vertices in slice 0) in .
[0712] Assume that, for slice [0], (mdu_geometry_2d_pos_x, mdu_geometry_2d_pos_y) is (X0, Y0) and (mdu_geometry_2d_size_x_minus1, mdu_geometry_2d_size_y_minus1) is (sizeX0-1, sizeY0-1):
[0713] 1. For tile [0], convert the pixel value I(X0,Y0) at (X0,Y0) to D(X0,Y0) as described above in Part 1. Add D(X0,Y0) to vertex [mdu_vertex_index_list[0]], the position of vertex [v0]. Also, convert the pixel value I(X0+1,Y0) at (X0+1,Y0) to D(X0+1,Y0). Add D(X0+1,Y0) to the position of vertex [v1].
[0714] 2.I(x,y) indicates the 3-tuple value at (x,y). It can indicate 3 numbers from 3 different planes.
[0715] 3. In some embodiments, the three values of the 3-tuple may be distributed on a plane. For example, when pixel_value_y indicates a pixel value of a specific plane such as a luminance plane, I(x,y)[0]=pixel_value_y(x,y), I(x,y)[1]=pixel_value_y(x+m,y+n), I(x,y)[2]=pixel_value_y(x+k,y+l).
[0716] 4. The generalized procedure for each shard can be as follows:
[0717] i.pos(i) indicates the position of the i-th vertex in the output mesh of the v-mesh bitstream.
[0718] ii. mdu_vertex_count_minus1+1 indicates the total number of vertices corresponding to the current tile in the normalized mesh.
[0719] iii.geometryVideoBlockSize indicates 1< <asps_log2_patch_packing_block_size。
[0720] iv.patchWidthInBlocks is set to (mdu_geometry_2d_size_x_minus1+1)) / geometryVideoBlockSize.
[0721]
[0722]
[0723] The process of Convert(I(xv,yv)) and apply(D,pos) is described above in Part 1.
[0724] 5. If the tile is RAW_MESH, the pixel values in the area corresponding to the tile in the geometry image are directly interpreted as the location of the mesh. The process can be described as follows:
[0725] a.patchWidthInBlocks is set to (rmdu_geometry_2d_size_x_minus1+1)) / geometryVideoBlockSize.
[0726] b. And other values are set as described above.
[0727]
[0728] 6. In some embodiments, the process can be described as follows:
[0729]
[0730] If the tile is RAW_MESH, the pixel values in the area of the attribute image corresponding to the tile are directly interpreted as the positions of the mesh. The process can be described as follows:
[0731] patchWidthInBlocks is set to (rmdu_attribute_2d_size_x_minus1+1)) / geometryVideoBlockSize, and other values are as described above.
[0732]
[0733] 1) In other embodiments, the process can be described as follows:
[0734]
[0735] decoding
[0736] The position of the mesh is reconstructed by adding the i-th displacement in the area corresponding to the current tile data unit in the displacement video to the i-th vertex in the sub-portion associated with the current tile data unit in the resampled base mesh.
[0737] The position of displacement(i) is counted from the geometry_2d_pos_x and geometry_2d_pos_y of the current tile (corresponding to the upper left corner of the region).
[0738] Create a list of vertices from the triangle faces (with the same face group id) associated with the current tile. Non-overlapping vertex indices are saved into the list based on the order they appear.
[0739] To illustrate this, Fig.40 Provides an example of vertex indices in the subpart associated with a shard. View Fig.40 For example, if the patch (mesh_intra_patch_data_unit[0]) has subpart_id 0, then the triangular faces with fi(facegroupId) 0 are associated with the patch, and these triangular faces are f 1 / 2 / 4, f 2 / 4 / 5, and f 0 / 1 / 2. Fig.40 As illustrated in the example of , in some embodiments, the correlation between the triangular faces and the face group Ids can be indicated via a correlation ordering between the list of triangular faces and the list of associated face group Ids. Fig.40 , a sorted list of triangular faces (e.g., f 1 / 2 / 4, f 2 / 4 / 5, f 2 / 5 / 3, and f 0 / 1 / 2) and a corresponding sorted list of face group Ids (e.g., f1 0, fi 0, fi 1, and fi0) indicating each of f 1 / 2 / 4, f2 / 4 / 5, and f0 / 1 / 2 are associated with face group Id 0 and triangular face f 2 / 5 / 3 is associated with face group Id 1.
[0740] Then, the associated vertices are sorted as 1, 2, 4, 5, 0. Therefore, the first displacement is added to vertex 1 (x1, y1, z1) and the last displacement is added to vertex 0 (x0, y0, z0). For slice [1], the associated vertices are sorted as 2, 5, 3.
[0741] In some embodiments, non-overlapping vertex indices are stored in a list in order of size. In the same example above, the associated vertices are ordered 0, 1, 2, 4, 5. The first displacement is added to vertex 0 (x0, y0, z0) and the last displacement is added to vertex 5 (x5, y5, z5). For tile [1], the associated vertices are ordered 2, 5, 3.
[0742] When a vertex shared by two tiles (v2 and v5 in this example) results in different geometry positions, they can be merged by the method indicated by asps_vmc_ext_multilocated_vertex_merge_method.
[0743] In some embodiments, the displacements of vertices shared by multiple tiles are signaled only once in the geometry image. For example, the displacements corresponding to vertices 2 and 5 in tile [1] are not present in the displacement image, so no additions are made to vertices 2 and 5. The number of displacements in the region corresponding to tile 0 is 5, and the number for tile 1 is 1.
[0744] In some embodiments, displacements of vertices shared by multiple tiles are ignored after the first occurrence. For example, the displacements corresponding to vertex 2 and vertex 5 in tile [1] are still present in the displacement image, but they are not added to these vertices. The number of displacements in the region corresponding to tile 0 is 5, and the number for tile 1 is 3.
[0745] In some embodiments, the displacement added to the i-th vertex in the segment indicated by the current tile data unit is determined by the geometry position in the tile associated with the current tile data unit. The position of displacement(i) is counted from ath_geometry_2d_pos_x and ath_geometry_2d_pos_y in the atlas tile header.
[0746] Creates a list of vertices from the triangle faces (with the same face group id) associated with the current tile. Non-overlapping vertex indices are stored in the list based on the order in which they appear. Fig.40 In the example of , the associated vertices are ordered as 1, 2, 4, 5, 0, 3. Then, the first displacement (at ath_geometry_2d_pos_x, ath_geometry_2d_pos_y) is added to vertex 1 (x1, y1, z1), and the last displacement is added to vertex 3 (x3, y3, z3). In some embodiments, non-overlapping vertex indices are stored in a list in the order of their size.
[0747] References for previous sections related to "V-TRUS BITSTREAM STRUCTURE INCLUDING SYNTAX ELEMENTS AND DECODING PROCEDURE WITH RECONSTRUCTION", each of which is incorporated by reference in its entirety:
[0748] [E1]ISO / IEC 23090-5ISO / IEC Information technology—CodedRepresentation of Immersive Media—Part 5: Visual Volumetric Video-basedCoding (V3C) and Video-based Point Cloud Compression (V-PCC).
[0749] [E2] K. Mammou, J. Kim, A. Tourapis, D. Podborski, K. Kolarov, “[V-CG] Apple’s Dynamic Mesh Coding CfP Response,” ISO / IEC JTC1 / SC29 / WG7 / m59281, online, April 2022.
[0750] [E3] A. Tourapis, J. Kim, D. Podborski, K. Mammou, “Base mesh data substream format for VDMC,” ISO / IEC JTC1 / SC29 / WG7 / m60362, online, July 2022.
[0751] Part 6: Adaptive Pattern Tessellation for Efficient Dynamic Mesh Encoding, Decoding, Processing, and Rendering
[0752] As mentioned above, static / dynamic meshes can be represented as a set of 3D meshes M(0), M(1), M(2), ..., M(n). Each mesh M(i) can be defined by connectivity C(i), geometry G(i), texture coordinates T(i) and texture connectivity CT(i). Each mesh M(i) can be associated with one or more 2D images A(i,0), A(i,1), ..., A(i,D-1), which are also called attribute maps, describing a set of attributes associated with the mesh surface. An example of an attribute would be texture information (see Figures 2 to 3 ). A set of vertex attributes can also be associated with the vertices of the mesh, such as color, normal, transparency, etc.
[0753] While the geometry and attribute information can be mapped again to 2D images and compressed efficiently using video coding techniques, the connectivity information cannot be encoded efficiently using similar schemes. A dedicated decoding solution optimized for such information is needed. In the following section, we present an efficient framework for static / dynamic mesh compression.
[0754] Figure 4 and Figure 5 High-level block diagrams of the encoding and decoding processes of the present invention are shown, respectively. Note that the feedback loop during the encoding process allows the encoder to guide the preprocessing step and change its parameters to achieve the best possible compromise according to various criteria, including but not limited to:
[0755] Rate distortion,
[0756] Encoding / decoding complexity,
[0757] Random access,
[0758] Reconstruction complexity,
[0759] Terminal capabilities,
[0760] Encoding / decoding power consumption, and / or
[0761] Network bandwidth and latency.
[0762] On the decoder side, the application consuming the content can provide feedback to guide both the decoding and post-processing blocks. As just one example, based on the position of the dynamic mesh relative to the camera view volume, the decoder and post-processing blocks can adaptively adjust the resolution / accuracy of the generated mesh and / or its associated property map.
[0763] Post-processing
[0764] Additional post-processing modules may also be applied to improve the visual / objective quality of the decoded mesh and attribute maps and / or adapt the resolution / quality of the decoded mesh and attribute maps to the viewpoint or terminal capabilities. An example of such post-processing includes adaptive pattern mosaicking, as described in references [EA], [EB], [EC], [ED].
[0765] The dynamic mesh compression scheme described in Section 1 teaches, among other things, a tessellation structure that achieves high rate-distortion compression performance. While optimization and control of compression performance can help enable a wide variety of applications (e.g., augmented reality / virtual reality (AR / VR), 3D mapping, autonomous driving, etc.), other functionality such as scalable decoding and rendering can also be used to enable broad deployment over a variety of networks (e.g., with different bandwidth and latency characteristics and constraints) and on a variety of platforms (e.g., with different processing / rendering capabilities and power constraints). Described below is an adaptive pattern tessellation scheme that can adjust the resolution of a dynamic mesh (e.g., the number of vertices / faces, the resolution of the attribute graph, etc.) to adapt to network conditions and / or the capabilities and constraints of the consumer device / platform.
[0766] Adaptive pattern tessellation as post-processing
[0767] Discussed above Figure 5 The interaction between: (1) Adaptive Pattern Mosaic Post-Processor Module 503; (2) Decoder 502; and (3) Application Module 501 is shown. More specifically, Adaptive Pattern Mosaic Module 503 may take as input:
[0768] Metadata: Metadata describing various information about the mesh structure (i). For example, this may include tile / tile group information, subdivision scheme, subdivision iteration count, bounding box, chunking, etc.
[0769] The decoded base mesh m'(i), which may (but need not) have full vertex / face / edge attributes describing saliency and importance / priority information;
[0770] A set of displacements d'(i) associated with the subdivided mesh vertices; and
[0771] • Optionally, one or more property graphs A'(i) describing information associated with the mesh surface.
[0772] These inputs may be calculated or otherwise determined as described above.
[0773] The application module 501 may provide control parameters to guide both the decoding module 502 and the adaptive pattern mosaic module 503. Such control parameters may include:
[0774] Current and / or future (potentially predicted) 3D camera position and view volume;
[0775] Available processing and rendering capabilities, such as the capabilities of the application and / or the device on which it runs;
[0776] · power consumption constraints of the application and / or the device on which it runs; and
[0777] • Region of Interest (ROI) information that identifies one or more portions of the mesh as regions of interest where more detail may be desired compared to other areas of the mesh.
[0778] The tessellation module 503 may utilize the subdivision structure described above, along with information provided by the decoder 502 and / or the application 501, to generate a mesh M"(i) to be rendered or processed by the application 501. (One example of processing by the application 501 may include collision detection, although any of a variety of operations on the mesh M"(i) are contemplated.) Exemplary strategies for utilizing the subdivision structure may include adjusting the global mesh resolution by changing the subdivision iteration count (see Fig.41 ). For example, if the model is far from the camera or if the terminal has limited rendering capabilities, the adaptive pattern mosaic module 503 can generate a base mesh 4201. As the object approaches the camera, the adaptive pattern mosaic module 503 can then progressively switch to higher resolution meshes, such as 4102, 4103, and 4104. These higher resolution meshes 4102 to 4104 correspond to subdivision iterations performed on the base mesh. For example, mesh 4102 can correspond to one subdivision of base mesh 4101. Mesh 4103 can correspond to another subdivision of mesh 4102, i.e., two subdivisions of base mesh 4101. Mesh 4104 can correspond to another subdivision of mesh 4103, i.e., three subdivisions of base mesh 4101.
[0779] Alternative strategies for utilizing the subdivision structure may include locally adjusting the mesh resolution in certain areas based on various criteria. Various methods of local mesh adjustment are possible, including those described in references [E1], [E2], [E3], [E4], [E5], [E6], [E7], [E8], [E9]. An example of local mesh resolution as a simple and efficient solution may be performed as follows:
[0780] Analyze local properties of the mesh, such as:
[0781] o displacements associated with the vertices of the mesh;
[0782] o explicitly encoded attributes associated with a base or subdivided grid that describe saliency and importance / priority information;
[0783] o Implicitly derived saliency and importance / priority information obtained by analyzing the mesh and attribute information. Examples may include surface curvature, gradients of vertex attributes or attribute maps, edge lengths, etc.;
[0784] Based on the analyzed local characteristics, determine for each edge of the mesh whether it should be subdivided. For example, if the displacement associated with the vertices of the edge is below a user-defined or automatically derived threshold, then a decision may be made not to subdivide the edge. Otherwise, (if there is a relatively large displacement associated with the edge / vertex) a decision may be made to subdivide it;
[0785] For each triangle (or other polygon), apply a subdivision scheme based on the number of edges to be subdivided. Fig.42 An exemplary subdivision scheme (for a triangle) is illustrated.
[0786] Repeat the above steps N times, where N is the number of subdivision iteration counts used to generate the final output mesh.
[0787] Fig.42 (See Fig.11 ) illustrates a technique for subdividing a triangle based on the number of edges to be subdivided according to the algorithm described above. In 4201a, each edge is to be subdivided so that the triangle is divided into four triangles as shown. In (b), two edges of the triangle are to be subdivided (the edges other than the base) so that the triangle is divided into three triangles as shown. In (c), only one edge is to be subdivided so that the triangle is divided into two triangles as shown. In (d), no edges are to be subdivided, which means that the original triangle is retained. This is just one possible method, and other subdivision methods can be applied, such as in the case of higher order polygons, etc.
[0788] Adaptive pattern mosaic for progressive / scalable coding
[0789] Adaptive pattern tessellation can also be achieved during the pre-processing and encoding stages by adaptively adjusting the subdivision scheme based on various criteria. Such criteria may include:
[0790] • Processing can be done on both the encoder side and the decoder side. In general, if more processing is available, a more detailed mesh (e.g., a mesh with more vertices, edges, a higher resolution property map, etc.) can be provided, and vice versa.
[0791] Rendering capabilities of the consuming terminal. Relatedly, if the consuming terminal has limited rendering capabilities, a less detailed mesh (e.g., a mesh with fewer vertices, edges, a lower resolution property map, etc.) may be provided, and vice versa. In some cases, the scalable nature of the encoding process may allow different layers of mesh information to be provided, with more capable terminals consuming multiple layers to provide higher levels of detail, while less capable terminals consume fewer layers or even only the base layer to provide lower levels of detail.
[0792] Power consumption constraints on the encoder and decoder sides. Similar to processing power, power consumption limits (such as battery-powered mobile devices) can be used to limit the ability of consumer devices to process, render, and / or display higher resolution meshes, even if computing resources are otherwise available. In this case, the pattern mosaic can be tailored to the power consumption constraints, which can be viewed as acting as constraints on the computing limits of one or more consumer devices.
[0793] Region of Interest (ROI) information. As mentioned above, for some applications, certain areas of the mesh may be more important than others. As an example, the face region may be of more interest than the body region of a mesh representing a person. In this case, the region of interest may be considered to guide the segmentation on the pre-processor / encoder side. The region of interest may be given explicitly by the consuming application or may be implicitly inferred from information about the mesh.
[0794] Saliency and importance / priority information provided by the user or obtained by analyzing mesh and attribute data (such as surface curvature, gradients of vertex attributes or attribute maps, edge lengths). Similar to ROI information, other forms of saliency and importance / priority information can be used to inform the encoder / pre-processor side tessellation process.
[0795] For any combination of the foregoing, the behavior of the tessellation scheme may be adjusted in the same manner as described above with respect to decoder / post-processor side tessellation.In the case of the encoder / pre-processor side, displacement and vertex attribute information may be encoded based on the adaptively tessellated mesh.
[0796] In some embodiments, the decimation stage described in Section 2 may be updated to take into account the criteria described above when generating the base mesh. For example, a higher resolution may be generated in a ROI provided by the user or by analyzing property map information associated with the region.
[0797] References for previous sections related to "Adaptive Pattern Tessellation for Efficient Dynamic Mesh Encoding, Decoding, Processing, and Rendering", each of which is incorporated by reference in its entirety:
[0798] [FA]https: / / developer.nvidia.com / gpugems / gpugems2 / part-i-geometric-complexity / chapter-7-adaptive-tessellation-subdivision-surfaces
[0799] [FB]https: / / niessnerlab.org / papers / 2015 / 0dynamic / schaefer2015dynamic.pdf
[0800] [FC]https: / / giv.cpsc.ucalgary.ca / publication / c5 /
[0801] [FD]https: / / projet.liris.cnrs.fr / imagine / pub / proceedings / ICME-2007 / pdfs / 0000468.pdf
[0802] [F1]https: / / www.researchgate.net / publication / 221434740_Incremental_Adaptive_Loop_Subdivision
[0803] [F2]https: / / www.researchgate.net / publication / 2554610_Adaptive_Subdivision_Schemes_for_Triangular_Meshes / link / 546e58c30cf2b5fc176074c3 / download
[0804] [F3]http: / / diglib.eg.org / bitstream / handle / 10.2312 / osg20031418 / 05settgast.pdf
[0805] [F4]http: / / www.graphics.stanford.edu / ~niessner / brainerd2016efficient.html
[0806] [F5]https: / / www.researchgate.net / publication / 220954613_Near-Optimum_Adaptive_Tessellation_of_General_Catmull-Clark_Subdivision_Surfaces / link / 00b7d53ae32d0c726a000000 / download
[0807] [F6]https: / / www.cs.cmu.edu / afs / cs / academic / class / 15869-f11 / www / readings / fisher09_diagsplit.pdf
[0808] [F6]http: / / www.graphics.stanford.edu / ~niessner / papers / 2015 / 0dynamic / schaefer2015dynamic.pdf
[0809] [F7]https: / / www.cise.ufl.edu / research / SurfLab / papers / 05adapsub.pdf
[0810] [F8]https: / / anjulpatney.com / docs / papers / 2009_Patney_PVT.pdf
[0811] [F9]http: / / research.michael-schwarz.com / publ / files / cudatess-eg09.pdf
[0812] Summary
[0813] The exemplary embodiments of the grid encoder and decoder that adopt video / image encoder / decoder for displacement and attribute are described above. Although many specific features and various embodiments have been described, it should be understood that unless otherwise indicated to be mutually exclusive, various features and embodiments can be combined into various arrangements in specific implementations. Therefore, the various embodiments described above are only provided by way of example, and should not be interpreted as limiting the scope of the present disclosure. Without departing from the scope of the present disclosure and without departing from the scope of the claims, various modifications and changes can be made to the principles and embodiments of this article.
[0814] In view of the foregoing and to help illustrate a machine that can be used to implement the processes described herein, Fig.43 4 shows an electronic device 4300 including an electronic display 4302. As described in more detail below, the electronic device 4300 may be any suitable electronic device, such as a computer, a mobile phone, a portable media device, a tablet computer, a television, a virtual reality headset, a vehicle dashboard, etc. Therefore, it should be noted that Fig.43 This is merely one example of a particular implementation and is intended to illustrate the types of components that may be present in electronic device 4300 .
[0815] The electronic device 4300 includes an electronic display 4302, one or more input devices 4304, one or more input / output (I / O) ports 4306, a processor core complex 4308 having one or more processing circuit systems or processing circuit system cores, a local memory 4310, a main memory storage device 4312, a network interface 4314, and a power source 4316 (e.g., a power supply). Fig.43 The various components described may include hardware elements (e.g., circuit systems), software elements (e.g., tangible, non-transitory computer-readable media storing executable instructions), or a combination of both hardware and software elements. It should be noted that the various depicted components may be combined into fewer components or separated into additional components. For example, local memory 4310 and main memory storage device 22 may be included in a single component.
[0816] The processor core complex 4308 is operably coupled to the local memory 4310 and the main memory storage device 4312. Thus, the processor core complex 4308 can execute instructions stored in the local memory 4310 or the main memory storage device 4312 to perform operations, such as generating and / or sending image data for display on the electronic display 4302. As such, the processor core complex 4308 may include one or more general purpose microprocessors, one or more application specific integrated circuits (ASICs), one or more field programmable logic arrays (FPGAs), or any combination thereof.
[0817] In addition to program instructions, local memory 4310 or main memory storage device 4312 can store data to be processed by processor core complex 4308. Therefore, local memory 4310 and / or main memory storage device 4312 can include one or more tangible non-transitory computer-readable media. For example, local memory 4310 may include random access memory (RAM), and main memory storage device 4312 may include read-only memory (ROM), rewritable non-volatile memory such as flash memory, hard drive, optical disk, etc.
[0818] The network interface 4314 can transmit data with another electronic device or network. For example, the network interface 4314 (e.g., a radio frequency system) can enable the electronic device 4300 to be communicatively coupled to a personal area network (PAN) (e.g., a Bluetooth network), a local area network (LAN) (e.g., an 802.11x Wi-Fi network) and / or a wide area network (WAN) (e.g., 4G), long term evolution (LTE) or 5G cellular network. The power supply 4316 can provide power to one or more components in the electronic device 4300 (such as the processor core complex 4308 or the electronic display 4302). Therefore, the power supply 4316 can include any suitable energy source, such as a rechargeable lithium polymer (Li-poly) battery or an alternating current (AC) power converter. The I / O port 4306 can enable the electronic device 4300 to interact with other electronic devices. For example, when a portable storage device is connected, the I / O port 4306 can enable the processor core complex 4308 to transmit data between the portable storage device.
[0819] Input device 4304 may enable user interaction with electronic device 4300, for example, by receiving user input via buttons, keyboard, mouse, touch pad, etc. Input device 4304 may include a touch sensing component in electronic display 4302. The touch sensing component may receive user input by detecting the incidence or location of an object touching the surface of electronic display 4302.
[0820] In some embodiments, the pixel or image data may be generated by an image source, such as a processor core complex 4308, a graphics processing unit (GPU), or an image sensor. Additionally, in some embodiments, the image data may be received from another electronic device 4300, for example, via a network interface 4314 and / or an I / O port 4306. Similarly, the electronic display 4302 may display frames based on the pixel or image data generated by the processor core complex 4308, or the electronic display 4302 may display frames based on the pixel or image data received via a network interface 4314, an input device, or an I / O port 4306.
[0821] Entities implementing this technology should take care to ensure compliance with established privacy policies and / or privacy practices to the extent that any sensitive information is used in a particular implementation. In particular, such entities will be expected to implement and consistently apply privacy practices that are generally recognized as meeting or exceeding industry or government requirements for maintaining user privacy. Implementers should inform users of where they expect personally identifiable information to be sent, and allow users to "opt-in" or "opt-out" of participation.
[0822] Once the data is no longer needed, risk can be minimized by limiting data collection and data deletion. In addition, and when applicable, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at a city level rather than an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods such as differential privacy, where appropriate. Robust encryption can also be used to reduce the likelihood that communications between inductively coupled devices can be spoofed.
Claims
1. A remeshing preprocessor for remeshing a 3D texture mesh M(i) to generate a base mesh m(i) and a displacement field d(i) for input into a mesh encoder, wherein the preprocessor include: a mesh decimation module comprising processing hardware that reduces the number of vertices or faces of an input mesh M(i) or a mesh derived from the input mesh M(i) while substantially preserving the shape of the input mesh M(i), thereby generating a decimated mesh dm(i) and a projected mesh P(i); and A fitting subdivision surface module, the fitting subdivision surface module comprising processing hardware, the processing hardware processing the input mesh M(i), the extracted mesh dm(i) or a mesh derived from the extracted mesh dm(i), and the projected mesh P(i) to generate a base mesh m(i) and a displacement field d(i) for input into a mesh encoder.
2. The remeshing preprocessor according to claim 1 further comprises a duplicate vertex removal module, wherein the duplicate vertex removal module comprises processing hardware, wherein the processing hardware removes duplicate vertices from the input mesh M(i) to generate a mesh UM(i) with uniform vertices, wherein the mesh UM(i) with uniform vertices replaces the input mesh M(i) and is input into the mesh extraction module.
3. The remeshing preprocessor of claim 1 , further comprising a duplicate triangle removal module, the duplicate triangle removal module comprising processing hardware that receives the decimated mesh dm(i) as input and processes it to remove triangles that reference the same vertices.
4. The remeshing preprocessor according to claim 1 further comprises a small connected component removal module, wherein the small connected component removal module comprises processing hardware, wherein the processing hardware detects and removes connected components having a number of vertices, a number of triangles or an area below a determined threshold from the extracted mesh dm(i) or a mesh derived from the extracted mesh dm(i).
5. The remeshing preprocessor of claim 1 , further comprising an atlas parameterization module comprising processing hardware for reducing the number of tiles of the decimated mesh dm(i) or a mesh derived from the decimated mesh dm(i).
6. The remeshing preprocessor according to claim 1, wherein the remeshing preprocessor further include: a duplicate vertex removal module, the duplicate vertex removal module comprising processing hardware, the processing hardware removing duplicate vertices from the input mesh M(i) to generate a mesh UM(i) with uniform vertices, the mesh UM(i) with uniform vertices being input to the mesh extraction module in place of the input mesh M(i); a duplicate triangle removal module, the duplicate triangle removal module comprising processing hardware that receives as input the decimated mesh dm(i) and processes it to remove triangles that reference the same vertex; a small connected component removal module, the small connected component removal module comprising processing hardware for detecting and removing connected components having a number of vertices, a number of triangles or an area below a determined threshold from the extracted mesh dm(i) or a mesh derived from the extracted mesh dm(i); and An atlas parameterization module comprising processing hardware for reducing the number of tiles of the decimated mesh dm(i) or a mesh derived from the decimated mesh dm(i).
7. A remeshing preprocessor according to claim 1, wherein the fitting subdivision surface module also includes a mesh subdivision module, the mesh subdivision module includes processing hardware, the processing hardware receives the extracted mesh dm(i) or a mesh derived from the extracted mesh dm(i) and subdivides its polygons to produce a subdivided mesh S(i).
8. A remeshing preprocessor according to claim 7, wherein the fit subdivision surface module further comprises at least one mesh deformation module, the at least one mesh deformation module comprising processing hardware, the processing hardware receiving the subdivision mesh S(i) and two or more of the following: a mesh derived from the subdivision mesh, the projected mesh P(i), the input mesh M(i), and generating a deformed mesh by moving vertices of the subdivision mesh S(i) to match the shape of the input mesh M(i).
9. A remeshing preprocessor according to claim 8, wherein the fit subdivision surface module also includes a base mesh optimization module, the base mesh optimization module includes processing hardware, the processing hardware receives the deformed mesh and the extracted mesh pm(i), and generates a base mesh m(i) by updating the position of pm(i) to minimize the distance between the subdivided version of pm(i) and the deformed mesh.
10. The remeshing preprocessor of claim 9, wherein the fit subdivision surface module further comprises a displacement calculation module, the displacement calculation module receiving the deformed mesh and the base mesh m(i) as input and calculating a displacement field d(i) as a difference between the deformed mesh and the base mesh m(i).
11. The remeshing preprocessor of claim 10, wherein the at least one mesh deformation module include: an initial mesh deformation module, the initial mesh deformation module comprising processing hardware, the processing hardware finding, for each initial 3D position Pos(v) corresponding to a vertex v of the subdivided mesh S(i), a closest point H(v) on the surface of the projected mesh P(i) such that an angle between a normal N(v) and a normal of H(v) is below a user-defined threshold, and moving the vertex v to the closest point H(v) to produce an initial deformed mesh F0(i); and An iterative mesh deformation module comprising processing hardware that finds, for each vertex v of the deformed mesh having position Pos(v), a closest point H(v) on the input mesh M(i) such that the angle between the normal vectors associated with Pos(v) and H(v) is below a user-defined threshold, and moves the vertex v to a new position determined by: Pos(v)+<H(v)-Pos(v),N(v)> *N(v) in<H(v)-Pos(v),N(v)> is the dot product of two 3D vectors H(v)-Pos(v) and N(v), and where N(v) is the normal vector at Pos(v), thereby producing a new deformed mesh, which becomes the final deformed mesh F(i) after a determined number of iterations.
12. The remeshing preprocessor of claim 9, wherein the at least one mesh deformation module include: an initial mesh deformation module, the initial mesh deformation module comprising processing hardware, the processing hardware finding, for each initial 3D position Pos(v) corresponding to a vertex v of the subdivided mesh S(i), a closest point H(v) on the surface of the projected mesh P(i) such that an angle between a normal N(v) and a normal of H(v) is below a user-defined threshold, and moving the vertex v to the closest point H(v) to produce an initial deformed mesh F0(i); and An iterative mesh deformation module comprising processing hardware that finds, for each vertex v of the deformed mesh having position Pos(v), a closest point H(v) on the input mesh M(i) such that the angle between the normal vectors associated with Pos(v) and H(v) is below a user-defined threshold, and moves the vertex v to a new position determined by: Pos(v)+<H(v)-Pos(v),N(v)> *N(v) in<H(v)-Pos(v),N(v)> is the dot product of two 3D vectors H(v)-Pos(v) and N(v), and where N(v) is the normal vector at Pos(v), thereby producing a new deformed mesh, which becomes the final deformed mesh F(i) after a determined number of iterations.
13. The remeshing preprocessor of claim 8, wherein the at least one mesh deformation module include: an initial mesh deformation module, the initial mesh deformation module comprising processing hardware, the processing hardware finding, for each initial 3D position Pos(v) corresponding to a vertex v of the subdivided mesh S(i), a closest point H(v) on the surface of the projected mesh P(i) such that an angle between a normal N(v) and a normal of H(v) is below a user-defined threshold, and moving the vertex v to the closest point H(v) to produce an initial deformed mesh F0(i); and An iterative mesh deformation module comprising processing hardware that finds, for each vertex v of the deformed mesh having position Pos(v), a closest point H(v) on the input mesh M(i) such that the angle between the normal vectors associated with Pos(v) and H(v) is below a user-defined threshold, and moves the vertex v to a new position determined by: Pos(v)+<H(v)-Pos(v),N(v)> *N(v) in<H(v)-Pos(v),N(v)> is the dot product of two 3D vectors H(v)-Pos(v) and N(v), and where N(v) is the normal vector at Pos(v), thereby producing a new deformed mesh, which becomes the final deformed mesh F(i) after a determined number of iterations.
14. A method for extracting an input mesh M(i) or a mesh UM(i) with uniform vertices derived from the input mesh M(i) by removing duplicate vertices to generate an extracted mesh dm(i), the method include: reducing the number of vertices / faces of the input mesh M(i) while substantially preserving the shape of the original mesh; as well as The mapping between the input mesh M(i) is tracked by projecting the deleted vertices onto the decimated mesh dm(i).
15. A method for deforming a subdivided grid S(i), the method include: For each initial 3D position Pos(v) corresponding to a vertex v of the subdivided mesh S(i), find the closest point H(v) on the surface of the projected mesh P(i) such that the angle between the normal N(v) and the normal of H(v) is below a user-defined threshold; Move the vertex v to the closest point H(v) to generate a deformed mesh, which is initially the initial deformed mesh F0(i); as well as Iteratively, for each vertex v of the deformed mesh with position Pos(v), find the closest point H(v) on the input mesh M(i) such that the angle between the normal vectors associated with Pos(v) and H(v) is below a user defined threshold; and Move the vertex v to a new position determined by: Pos(v)+<H(v)-Pos(v),N(v)> *N(v) in<H(v)-Pos(v),N(v)> is the dot product of two 3D vectors H(v)-Pos(v) and N(v), and where N(v) is the normal vector at Pos(v), thereby producing a new deformed mesh, which becomes the final deformed mesh F(i) after a determined number of iterations.
16. A method of time-consistent remeshing, the method comprising reusing a base mesh pm(j) associated with a reference mesh M(j) as a base mesh pm(i) associated with a mesh M(i), wherein pm(i) and pm(j) have the same connectivity.
17. The method according to claim 16, further comprising: include: if the input meshes M(i) and M(j) are temporally coherent, applying a fit subdivision surface module, the fit subdivision surface module comprising processing hardware that processes the input mesh M(i), a decimated mesh dm(j) or a mesh derived from the decimated mesh dm(j), and a projected mesh P(j) to produce a base mesh m(i) and a displacement field d(i) for input into a mesh encoder; or If the input grids M(i) and M(j) are not time-coherent: generating a mesh M'(j) which is a deformed version of M(j) having the same shape as M(i); as well as A fitting subdivision surface module is applied, which includes processing hardware, which processes the mesh M'(j), the extracted mesh dm(j) or a mesh derived from the extracted mesh dm(j), and the projected mesh P(j) to produce a base mesh m(i) and a displacement field d(i) for input into a mesh encoder.
Citation Information
Cited By
Dynamic grid vertex displacement coding
CN118176719A
Method, apparatus, medium for video encoding
CN118176719B
