V-PCC-based dynamic texture mesh coding without occupancy graph

Through the V-PCC encoder, the dynamic texture grid compression problem in 3D grid video encoding is solved, and efficient coding and low bandwidth transmission is achieved, reducing artifact problems .

CN118923118APending Publication Date: 2024-11-08INTERDIGITAL CE PATENT HOLDINGS SAS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380027821.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-06-07
Filing Date
2023-03-02
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

When the prior art performs video encoding of 3D meshes, it is difficult to effectively compress dynamic texture meshes, and traditional solutions require occupancy graph encoding, resulting in bandwidth height and artifact problems.

Method used

A new solution based on V-PCC encoder is proposed. By encoding the boundary segments of the face sheet and the adjacent face sheet list, the coding of the occupancy graph is avoided, and the fast implicit re-meshing and filtering method is adopted to directly on the face. The space between the face sheets is filled during the reconstruction process.

Benefits of technology

It realizes efficient coding of dynamic texture mesh, reduces bandwidth requirements, reduces artifact problems, and improves compression efficiency and rendering quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118923118A_ABST
    Figure CN118923118A_ABST
Patent Text Reader

Abstract

A scheme that extends a video point cloud compression codec scheme to store grid information and provide good compression efficiency and good objective and subjective quality. To reduce the amount of data that needs to be transmitted, the scheme removes an occupancy map for knowing which portions of the patch are occupied. These occupancy maps produce a large number of artifacts on the boundaries of patches due to the sub-resolution of the occupancy maps used to transmit occupancy information. In contrast, the scheme transmits the three-dimensional boundaries of the patches (a list of three-dimensional points constituting the patch boundaries) and uses this information to identify which portions of the two-dimensional patches are occupied and join the three-dimensional reconstructed patches according to the real three-dimensional boundaries of the patches.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The technical field of the present disclosure is video compression, and more specifically video encoding of a 3D mesh consisting of points in 3D space with relevant properties (e.g., connectivity, texture associated with the mesh, 2D coordinates of the points in the texture image). Background Art

[0002] One of the prior art approaches for achieving good compression efficiency when encoding point clouds (Test Model Category 2 of MPEG 3DG / PCC, called TMC2) consists in projecting multiple geometric and texture / attribute information onto the same location (pixel) of the 2D image; i.e., encoding multiple layers of information for each input point cloud. Typically, two layers are considered. Summary of the invention

[0003] At least one embodiment of the present invention is generally directed to a method or apparatus in the context of compressing images and videos of 3D meshes using correlation properties.

[0004] According to a first aspect, a method is provided. The method comprises the following steps: decoding one or more video bitstreams to obtain a depth map and an attribute map; further decoding a bitstream containing a boundary point list to obtain a boundary point line segment; generating a plurality of adjacent patch lists based on syntax information; obtaining a boundary point line segment of a patch according to the boundary points and the adjacent patch list encoded in the bitstream; generating an occupancy map of the patch using the boundary point line segment of the patch; constructing a mesh of the patch based on the occupancy map, the depth map and the attribute map; and filling an inter-patch space between a reconstructed patch and the boundary point line segment to obtain a reconstructed model.

[0005] According to a second aspect, a method is provided. The method comprises the following steps: correcting the geometry and topology of a scaled three-dimensional mesh model; grouping the triangles of the mesh into connected components; refining the connected components; extracting patch boundary segments of a patch; creating an occupancy map of the patch based on the patch boundary segments; rasterizing the mesh of the connected components of the patch to create a depth map and a depth video frame of the patch; reconstructing the three-dimensional mesh of the patch based on the depth map and the occupancy map; filling the inter-patch space based on the patch boundary segments of the patch and the boundary edges of the reconstructed mesh; extracting all vertices of the mesh based on the reconstructed mesh to create a reconstructed point cloud; coloring the points of the reconstructed point cloud; and creating an attributed video frame using the colored reconstructed point cloud.

[0006] According to another aspect, a device is provided. The device includes a processor. The processor can be configured to implement the general aspects by executing any of the methods described.

[0007] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of illustrative embodiments which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A V-PCC encoder is shown.

[0009] Figure 2 A V-PCC decoder is shown.

[0010] Figure 3 Shows an example of patch boundary segments.

[0011] FIG. 4 shows an example of two-dimensional triangulation of a patch boundary.

[0012] FIG. 5 shows examples of inter-patch space filling: (a) the case of reconstructed meshing and (b) the case of triangulated meshing.

[0013] Figure 6 Another example of a V-PCC decoder is shown.

[0014] Figure 7 An example of a free viewpoint video is shown.

[0015] Figure 8 An example of a reconstructed 3D animation mesh sequence is shown.

[0016] Fig. 9 An example of a wireframe view of a reconstructed animated texture mesh (top) and (b) the associated texture atlas is shown.

[0017] Fig.10 An example of a trellis coding scheme for extended V-PCC is shown.

[0018] Fig.11 An example of a trellis decoding scheme for extended V-PCC is shown.

[0019] Fig.12 An example of an encoding flow chart is shown.

[0020] Fig.13 An example of a decoding flow chart is shown.

[0021] Fig.14 The V-PCC encoder architecture is shown.

[0022] Fig.15 The V-PCC decoder architecture is shown.

[0023] Fig.16 An improved V-PCC encoder is shown.

[0024] Fig.17 An improved V-PCC decoder is shown.

[0025] Fig.18 Shows an example of patch boundary segments.

[0026] 19 shows an example of boundary segment extraction: (a) source model, (b) oriented edges between patches, (c) concatenated edges, and (d) boundary segments constructed by concatenating edges.

[0027] Fig. 20 Shows an example of a neighbor patch list.

[0028] Fig.21 Shows an example of constructing a patch occupancy map.

[0029] Fig. 22 Examples of intersection points on boundary line segments are shown.

[0030] Fig.23 Shows an example of an occupancy map and intersection points computed on a 3D patch.

[0031] Fig.24 An example of points used to reconstruct an occupancy map for a grid square is shown.

[0032] Fig.25 An example of intersection points used to construct a square grid is shown.

[0033] Fig.26 An example of the Douglas-Peucker algorithm is shown.

[0034] FIG. 27 shows examples of boundary segment simplification: (a) original case, (b) case of threshold=1, (c) case of threshold=2, and (d) case of threshold=4.

[0035] FIG28 shows an example of aliasing: (a) original, (b) reconstructed mesh.

[0036] FIG. 29 shows an example of a 2D depth map projection of a mesh edge.

[0037] FIG. 30 shows another example of 2D depth map projection of mesh edges.

[0038] FIG31 shows an example of a reconstructed model utilizing depth map scaling: (a) original, (b) reconstructed mesh using scaled depth.

[0039] Fig.32 The V3C / V-PCC trellis coding process is shown.

[0040] FIG. 33 shows an example of blurred boundaries due to degenerate triangles.

[0041] Fig.34 The V3D / V-PCC grid decoding process is shown.

[0042] Fig.35 An embodiment of a method for performing the described aspects is shown.

[0043] Fig.36 Another embodiment of a method for performing the described aspect is shown.

[0044] Fig.37 A processor-based system for implementing the described aspects is shown.

[0045] Fig.38 Another processor-based system for implementing the described embodiments is shown.

[0046] FIG. 39 shows an example of a reconstructed mesh with one occupied point.

[0047] FIG. 40 shows an example of a reconstructed mesh with two occupied points.

[0048] FIG. 41 shows an example of the reconstruction process in a case where two occupied points are not adjacent.

[0049] FIG. 42 shows an example of a reconstructed mesh with three occupied points.

[0050] FIG. 43 shows an example of a reconstructed mesh with four occupied points.

[0051] Fig.44 An example of reconstructing a mesh patch is shown.

[0052] Fig.45 The V3C / V-PCC trellis coding process is shown.

[0053] Fig.46 An example of an updated V3C / V-PCC trellis coding process is shown.

[0054] Fig.47 The V3C / V-PCC trellis decoding process is shown.

[0055] Fig.48 An updated V3C / V-PCC trellis decoding process is shown. DETAILED DESCRIPTION

[0056] One or more embodiments rely on concepts introduced in other work by the PCC Ad hoc Group entitled "Grid Coding using V-PCC" and proposed as EE2.6.

[0057] The following sections briefly introduce the background concepts:

[0058] -Point cloud codec: V-PCC

[0059] -Grid format: Simple format

[0060] V-PCC Overview - Point Cloud Compression (Meshless)

[0061] One of the prior art solutions for achieving good compression efficiency when encoding point clouds (Test Model Category 2 of MPEG3DG / PCC, called TMC2) consists in projecting multiple geometry and texture / attribute information to the same location (pixel) of a 2D image; i.e., encoding multiple layers of information for each input point cloud. Typically, two layers are considered. This means that for each input point cloud, multiple 2D geometry and / or 2D texture / attribute images are generated. In the case of TMC2, for each input point cloud, two depth (for geometry) and color (for texture, also known as attributes) images are encoded.

[0062] One embodiment proposes to extend the V-PCC (MPEG-I Part 5) codec of the Ad hoc Group on Point Cloud Compression of MPEG 3D Graphics (3DG) to be able to solve the mesh compression problem and to be able to solve the dynamic texture mesh compression problem. It complements the idea of ​​proposing a coding scheme for encoding 3D meshes without occupancy maps, and proposes new methods for constructing occupancy maps of patches, reconstructing patches, and filling the space between patches.

[0063] The embodiment proposes a new scheme that builds an occupancy map of a patch, reconstructs the patch, and fills the inter-patch space based on the patch boundary segments of the 3D point list defining the patch boundary and the patch information (patch parameters, depth map, attribute map).

[0064] Methods for constructing occupancy maps and filling spaces between patches:

[0065] - triangulate the area defined by the boundary segments of the patch to obtain an occupancy map; and

[0066] -Triangulate the area defined by the boundary segments of the patch and the boundary edges of the reconstructed patch for inter-patch filling.

[0067] These processes are complex and require extracting boundary edges from the reconstructed mesh. To make these processes more efficient and allow for parallel reconstruction of patches, they have been updated to allow both patch reconstruction and patch boundary filling to be processed in a single pass.

[0068] One embodiment relates to a V-PCC codec, the structure of which is as follows Figure 1 (Encoder and proposed embodiment) and Figure 2 (Decoder) shows:

[0069] Volumetric Video

[0070] “With recent advances in capture technology and consumer virtual / augmented reality hardware such as Microsoft HoloLens, interest in free viewpoint video (FVV) has skyrocketed. As real-time view tracking becomes more accurate and ubiquitous, new types of immersive viewing experiences become widely available, which require equally immersive content.” Cited from Microsoft (Collet et al., 2015).

[0071] A patch is represented by a parameter set that includes a list of boundary line segments of 3D points that define the boundary of the patch. Figure 3 Shows an example of boundary segments on a mesh.

[0072] The captured animation sequence can then be replayed from any virtual perspective with six degrees of freedom (6dof). To provide such functionality, there are solutions such as images / videos, point clouds, and textured meshes.

[0073] Image / video based solutions would store a set of video streams along with additional metadata and perform warping or any other reprojection to generate images from a virtual perspective at playback time. This solution requires a lot of bandwidth and introduces many artifacts.

[0074] Point cloud solutions will reconstruct an animated 3D point cloud from a set of input animated images, resulting in a more compact representation of the 3D model. The animated point cloud can then be projected onto a plane that encapsulates the volume of the animated point cloud, and the projected points (a.k.a. patches) are encoded into a set of 2D coded video streams (e.g. using HEVC, AVC, VVC, etc.) for transmission. This is the solution developed in the MPEG V-PCC (ISO / IEC JTC1 / SC29 WG11, w19332, V-PCC Codec Description, April 2020) standard, which gives very good results. However, the nature of the model is very limited in terms of spatial resolution and some artifacts may appear, such as holes in the surface in close-up views.

[0075] The boundary segments of a patch are used to:

[0076] Construct an occupancy map of the patch, which represents the area actually covered by the patch;

[0077] Extend the 3D reconstructed patch to the actual location of the boundary segment, i.e. the true 3D position of the patch set by the encoder.

[0078] These procedures will be updated through the examples described in the rest of this document, including:

[0079] Reconstruct occupancy map

[0080] Reconstruct the 3D mesh of the patch

[0081] Filling spaces between patches

[0082] In our previous scheme, these two processes are performed by triangulating 2D polygons extracted from 3D boundary segments and the boundaries of the reconstructed patch. Figure 4 shows an example of constructing an occupancy map of a patch by triangulating 2D polygons. The obtained mesh is rasterized into a 2D map to construct a 2D occupancy map.

[0083] Using the same triangulation process used to create the occupancy map, the inter-patch spaces are filled by triangulating 2D polygons constructed from boundary segments and the boundary edges of the reconstructed patches. The result of this process is shown in Figure 5.

[0084] An updated embodiment describing occupancy map construction is described.

[0085] The improved patch reconstruction process is described in the following section.

[0086] The filling process of the inter-patch spaces presented in the previous scheme is removed, as it is now performed directly during the patch reconstruction process.

[0087] The texture mesh scheme will reconstruct an animated texture mesh from a set of input animated images (see Figure 8 )(Collet et al., 2015)(Carranza, Theobalt, Magnor, & Seidel, 2003). This reconstruction usually goes through an intermediate representation as a voxel or point cloud. Fig. 9 Describes the type of quality that can be achieved with this reconstructed mesh. The advantage of the mesh is that the geometry definition can be very low and the photometric texture atlas can be encoded in a standard video stream (see Fig. 9, bottom). Point cloud solutions require “complex” and “lossy” implicit or explicit projections (ISO / IEC JTC1 / SC29 WG11, w19332, V-PCC Codec Description, April 2020) to obtain a planar representation compatible with video-based coding schemes. In contrast, texture mesh encoding relies on texture coordinates (“uvs”) to map the texture image to the triangles of the mesh. While video-based encoding of texture atlases (e.g., using HEVC) can achieve very high compression rates with minimal loss, we will see that encoding the remaining parts describing the topology (face lists) and vertex attributes (positions, uvs, normals, etc.) using video coding remains challenging. It is also important to note that the relative size of the original image atlas with respect to the original geometry (topology plus attributes) varies depending on the reconstruction parameters and the target application. However, the original size of the geometry is usually smaller than the photometric data (texture atlas). Nevertheless, it still makes sense to efficiently encode the geometry to reduce the global load. Furthermore, encoding the geometry using a video encoder such as HEVC will achieve to some extent a pure video-based compression that is compatible with most existing HEVC implementations (including hardware implementations).

[0088] Both textured mesh and point cloud solutions are related, and in some specific cases even image / video solutions. The author usually chooses the approach (mesh or point cloud) based on the nature of the model. With point clouds, sparse elements like hair or leaves will render better. With meshes, surfaces like skin and clothing will render better. Therefore, both solutions are suitable for optimization. Also note that these solutions can be combined to represent different parts of the model.

[0089] Problem to be solved

[0090] One or more embodiments propose a new scheme for encoding dynamic textured meshes using a V-PCC encoder (projection-based). We propose a complete chain for projecting meshes into patches encoded using a video-based scheme using V-PCC. We also propose a solution that prevents the need to encode an occupancy map as required by the standard V-PCC chain. Instead, we propose edge contour encoding. Some other solutions are also proposed, such as fast implicit remeshing (which prevents the encoding of topology) and some filtering methods for enhancing the reconstructed mesh.

[0091] Prior work

[0092] In the Exploratory Experiments (EE) track in V-PCC, a solution was proposed that combines V-PCC with the TFAN codec as the vertex connectivity codec.

[0093] The proposed encoding and decoding architectures are as follows: Figure 7 and Fig.11 shown.

[0094] Basically, on the encoder side, the input grid is decomposed (demultiplexed) into two groups:

[0095] -Vertex coordinates and vertex attributes, which are fed into the V-PCC encoder

[0096] - Vertex connectivity, which is encoded by the TFAN encoder

[0097] Since the TFAN encoder generates data with an order different from that of the input grid, a reordering process should be applied according to the V-PCC output order.

[0098] In the first version, it was proposed to reorder on both sides and transmit this reordering table ( Fig.10 and Fig.11 ). The second version then proposes to reorder the input point cloud of V-PCC according to the output TFAN ordering, so that there is no need to transmit the reordering information. The geometry and attributes are then packed into V-PCC RAW patches according to the traversal order of the TFAN connections. As defined in the V3C / V-PCC specification (w19579_ISO_IEC_FDIS_23090-5, Section H.11.4, Reconstruction of RAW Patches), RAW patches directly encode the geometric coordinates and attribute values ​​of 3D RAW points in the three components of the geometry frame and the attribute frame.

[0099] The resulting codes are multiplexed into the so-called extended V-PCC bitstream.

[0100] It should be noted that when the mesh is considered sparse (and the attributes are contained in separate image files), this scheme proposes a pre-processing: downsizing, converting the texture image to vertex colors, and then voxelizing it to 10 bits, followed by cleaning the non-manifold and degenerate faces generated by the voxelization process.

[0101] On the decoder side, the double operation of the encoder is performed in order to reconstruct the grid.

[0102] Although the proposal utilizes and combines two existing codecs, the following shortcomings are pointed out:

[0103] -Interface modifications to both codecs are required to account for point cloud reordering

[0104] - said reordering may introduce latency on the encoder side (depending on how the vertices are ordered, it may need to wait for the entire mesh to be processed for the reordering)

[0105] - One of the two codecs (TFAN) is over 10 years old and has reportedly never been implemented or considered by the industry

[0106] - Intrusive modification of the syntax of the V-PCC bitstream is required to integrate TFAN data

[0107] - There may be texture loss, since attributes are associated to each point (vertex) - The density of textures may be greater than the vertex / point density.

[0108] - This solution only works for lossless encoding.

[0109] This proposal is presented below (in Fig.12 and Fig.13 Flowchart of the encoder and decoder in .

[0110] On the encoder side, module 10 takes the input mesh as input, outputs connectivity information to the TFAN encoder of module 20, outputs vertex / point coordinates and attributes to vertex reordering module 30, whose purpose is to align the positions of vertices / points according to the TFAN and V-PCC point encoding order, and then processes the reordered vertices using the V-PCC encoder module 40. Finally, the outputs of the TFAN and V-PCC encoders are encapsulated in the output bitstream, which may be compatible with an extended version of the V-PCC encoder.

[0111] On the decoder side, module 60 parses the V-PCC mesh-extended bitstream, allowing module 70 to decode the connectivity encoded by the TFAN decoder, while module 80 decodes the attributes and coordinates of the relevant vertices. Finally, module 90 combines all the decoded information to create the output mesh.

[0112] Typically, the above techniques allow lossless compression of a mesh object by a ratio of approximately 7 (eg, 150 bpv --> 23 bpv) relative to the original (uncompressed) mesh model.

[0113] Detailed description of the solution

[0114] The main intentions behind the general aspects described in this article are twofold:

[0115] - Use only the V-PCC codec (but extend it to handle meshes),

[0116] - Minimize changes to current V-PCC encoders and decoders.

[0117] The current version of V-PCC encodes the information representing a point cloud (i.e., attributes, occupancy maps, and geometric information) in a so-called atlas (“set”).

[0118] In the proposed embodiment, modules 3600 and 4300 are removed and new processes are added.

[0119] The V-PCC encoder and decoder schemes have been modified as follows:

[0120] - Modify patch generation using rasterization of the input mesh

[0121] - Occupancy maps are not created and transmitted in the V-PCC bitstream

[0122] -Transmit the 3D boundary of the patch and the adjacency information between patches.

[0123] This solution proposes to store the following in the bitstream instead of sending a 2D occupancy map video in the bitstream:

[0124] - Patch boundary line segments (a list of 3D points that define the boundary line segments between two patches). These point lists are oriented and define a list of edges that are shared between the two patches.

[0125] - For each patch, a list of neighboring patches (indices of patches that share border segments with the current patch).

[0126] Notes:

[0127] - A boundary segment between two patches is the same for both corresponding patches and is stored only once. The boundary segment is indexed by a pair of values ​​[n; m] corresponding to the indices of the two patches n and m.

[0128] - A border can be between two patches, but it can also be between a patch and empty space if the patch has a border that is not connected to any other part of the mesh. In this case, the index of the second patch used to define the two patches that share the current border segment is set to negative one (-1), and the border segment index is [n;-1].

[0129] - Boundary segment points are oriented, and if the current patch corresponds to the first index of the pair [n; m], then the boundary segment points shall all be used in direct clockwise order (or optionally in counter-clockwise order if applying equally to all boundaries), and if the current patch index corresponds to the second value of the pair [n; m], then the boundary segment points shall be used in reverse order.

[0130] - Considering the previous description, a complete list of boundary points of a patch can be obtained by sequentially connecting all boundary line segments that appear at the current patch index.

[0131] Fig.18 Shows an example of boundary segments on a mesh.

[0132] Extraction of Patch Boundary Segments

[0133] According to the segmentation process, each triangle of the mesh is assigned to a patch, and each triangle has a patch index value indicating which patch it belongs to.

[0134] The following describes the algorithm for calculating the boundary point segments of each patch and the list of adjacent patches.

[0135] For each triangle T with patch index pi(T):

[0136] - For each neighbor triangle N of T

[0137] o If the patch indices of N and T are different (pi(T)!= pi(N))

[0138] · If two vertices (v0, v1) of T also exist in triangle N: T and N; or

[0139] · If two vertices (v0, v1) of T do not exist in any neighbor triangle N,

[0140] o If the edge (v0, v1) does not exist yet, create the edge in the list of edges between patches (min(pi(T), pi(N)), max(pi(T), pi(N))).

[0141] For the list of edges between every two patches p0 and p1:

[0142] - Find M segments of consecutive points

[0143] - Store the list of M points in the boundary segment list of the patch (p0, p1).

[0144] This process is allowed because the triangles and edges have been oriented clockwise in the first stage of the process.

[0145] After this process, we have Q lists of oriented points (Pi, Pj) that contain the patch boundary points between patches Pi and Pj. In the pair value (Pi, Pj), Pi < Pj, and if the boundary of patch Pi is not linked to any other patch, then Pj can be equal to -1.

[0146] These lists of boundary points can be used to extract all the boundary patches of a patch Pk by concatenating in order all the point lists in which the index Pk exists in the pair values (Pi, Pj). If Pj is equal to Pk, the point list should be reversed to obtain the clockwise-ordered points corresponding to that patch.

[0147] The set of pair values (Pi, Pj) can also be used to extract the list of adjacent patches. The following example shows the list of adjacent patches:

[0148] Transfer boundary points, line segments and adjacent face lists

[0149] The boundary points, line segments and neighbor patch lists are used by the encoder and decoder and should be transmitted in the V-PCC bitstream.

[0150] Boundary point segment coding

[0151] The order of the points should be preserved and the decoder should reconstruct the list of points for each adjacent patch (i, j).

[0152] Connect the point order of the boundary segments into a list according to the patch order defined in ascending order in the adjacent patch list.

[0153] To allow us to detect where a list of consecutive line segments ends, the last point of each such list is repeated.

[0154] The list of boundary points can then be encoded into a quantized point cloud, for example using Draco or using any other encoder that will preserve the order of the points and their multiplicity. The encoder can be lossless or lossy.

[0155] The corresponding bitstream is stored in a V3C unit in the V-PCC bitstream. The bitstream is stored in a V3C_OVD type V3C unit corresponding to occupied video data, but another V3C unit may be used or defined specifically for this.

[0156] The decoding process takes the corresponding V3C unit, decodes the data using an appropriate decoder (e.g. Draco), and obtains a list of boundary points containing the coded points in the same order as the repeated points.

[0157] According to the neighbor patch list in ascending order, the points of the list are added to the corresponding boundary list points (Pi, Pj). Each detected duplicate point is used to know that the next point is the starting point of a new line segment. In this case, the next pair of values ​​(Pi, Pj) is taken from the neighbor patch list in ascending order.

[0158] This process is lossless, and the decoded list of boundary points is identical to the encoded list of boundary points.

[0159] Encode neighbor patch list

[0160] The decoder side needs the neighbor patch list to reconstruct the boundary point segments and reconstruct the patches, and this data should be transmitted.

[0161] This list is arranged by patch and stores the indices of neighboring patches. If the current frame is represented by n patches, then for each patch with index I in [0; n-1], we have a list of neighboring patches: {j, k, l, m, o, ..., [-1], [-1]}. These lists have some specific properties:

[0162] -The indices stored in the list are always higher than i and lower than n.

[0163] - The stored indices are in ascending order.

[0164] -The last element of the list can be multiples of -1.

[0165] To encode this list, we can create an intermediate list containing the incremented values ​​and encode that incremented list:

[0166] -For each example in the neighbor patch list {j, k, l, m, o, -1, -1}

[0167] -Replace -1 with the number of patches n → {j, k, l, m, o, n, n}

[0168] - Encodes the delta relative to the previous element (i+1 for the first element). →{j–i+1, kj, lk, ml, om, no, 0}

[0169] On the decoder side, the reverse process can be performed to reconstruct the list:

[0170] - For the decoded delta list {d1, d2, d3, d4, d5, d6, d7}

[0171] - Add the previous element to each element {d1+i+1,d1+i+1+d2,d1+i+1+d2+d3...}

[0172] - Replace the elements equal to n with -1 {d1+i+1, d1+i+1+d2, d1+i+1+d2+d3…, -1, -1}

[0173] After this process, we need to encode the incremental value of the neighbor patch list in the V3C bitstream.

[0174] We should update the V3C grammar defined in w19579_ISO_IEC_FDIS_23090-5 with the following grammar elements:

[0175] -8.3.7.3 Syntax of patch data unit

[0176] -8.3.7.5 Syntax of Merged Patch Data Units

[0177] -8.3.7.6 Inter-panel data unit syntax

[0178] 8.3.7.3 Syntax of patch data unit

[0179]

[0180]

[0181] 8.3.7.5 Syntax of Merged Patch Data Units

[0182]

[0183]

[0184] 8.3.7.6 Inter-patch data unit syntax

[0185]

[0186] Occupancy map reconstruction

[0187] Based on the patch boundary segments and the neighboring patch list, the occupancy map of the patch can be constructed by calculating the intersection points between the patch boundary segments and the horizontal lines in the occupancy map.

[0188] For each horizontal line of the occupancy map, we can calculate the intersection points with the patch boundary line segments. The list of intersection points can be obtained based on the value of u (see Fig.21 ) are arranged in ascending order, and these points can be recorded as {p 0 ,p 1 , p 2 , ..., p n}.

[0189] The greedy algorithm can process the point list, for each point p i , invert the value of the flag that marks whether the intersection point after this point is inside or outside the patch. If the intersection point [p i , p i+1 ] is inside the patch, then the occupancy map pixels in this intersection are set to true.

[0190] Fig.21 An example of this process is shown.

[0191] Furthermore, in addition to the occupancy information stored in the occupancy map, the proposed algorithm for constructing the occupancy map also stores for each occupied point the intersection points between the boundary line segments and the horizontal / vertical lines {B 0 , B 1 ,...}. Each boundary point of the occupancy map stores the nearest intersection point on the boundary line segment in the u and v directions.

[0192] The intersection point is also added to the 0 , B 0 , B 1 ,...B n , S 1 , B n+1 , Bn+2 ,...B n+m , S 2 ...}, where S i is the point of the patch boundary segment, B i are the intersection points. This list is named S′ and stored in memory for later use. The S′ list is not encoded in the bitstream like the patch boundary segment list shown in [1], but will be reconstructed by the decoding process, just like the occupancy Figure 1 Sample.

[0193] Fig. 22 An example of this process is shown.

[0194] Fig.23 An example of this process on a real patch is shown.

[0195] 3D mesh reconstruction of patches

[0196] Based on the occupancy map, decoded geometry video, boundary segments, and boundary segment intersections (stored in S′), we can reconstruct the 3D mesh corresponding to each patch and directly fill the inter-patch space to ensure that the reconstructed patch covers all the space defined by the boundary segments. In this case, there is no need for an additional process of filling the inter-patch space (as proposed in the first method) because the inter-patch space is already covered by the reconstructed patch.

[0197] Therefore, the reconstruction process proposed in the first approach has been updated to directly fill the inter-patch space.

[0198] To allow GPU shaders to perform parallel reconstruction of patches, each occupied pixel (u, v) of the occupancy map can be reconstructed in parallel, and then the grid corresponding to the square defined by the four points (u, v), (u+1, v), (u+1, v+1), and (u, v+1) can be reconstructed, where these points are denoted as: 0 、p 1 、p 2 and p 3 ,like Fig.24 as shown in the example.

[0199] Each point (u, v) in the occupancy graph has a list of intersection points B i . Just consider the square [p 0 , p 1 , p 2 , p 3 ]], such as Fig.25 shown.

[0200] Depending on the number of occupied points on the four corners of the square, specialized reconstructions are performed using specific gridding patterns, as described in the following subsections.

[0201] In order to limit the number of cases to be considered and the complexity of the reconstruction process, the boundary segments are simplified during the encoding process to ensure that there is only one point S in each 3D voxel of size 1. i After this simplification, we know that each square (p 0 ,p 1 , p 2 , p 3 ) can only have one point S i .

[0202] An occupied point

[0203] If only one point is occupied, such as p shown in Figure 39 3 , we know that the occupied point is on the adjacent edge [p 0 ;p 3 ] and [p 3 ;p 2 ] has two intersection points B 0 and B 1 , we use B 0 、p 3 and B 1 Construct a list of boundary points of the corresponding reconstructed grid. In order to know whether to 0 and B 1 We can study the list S′ and add all the points of this directed list to B 0 and B 1 Between, such as Fig.11 b and Fig.11 c. The resulting list is well oriented with no intersections and can be easily triangulated.

[0204] Note: If Fig.11 c, the reconstructed grid can be located in the square [p 0 , p 1 , p 2 , p 3 ]. This is true only if exactly one point in the square is occupied.

[0205] Two occupied points

[0206] If two points are occupied, such as p3 and p0 shown in Figure 40, the corresponding space can be remeshed according to the following process.

[0207] If two points each have only one intersection point B i , we can construct list B 0 、p 0 、p 3 , B 1 and B in S′ 0 and B 1to triangulate this polygon ( Fig.40a and Fig.40b ).

[0208] If Fig.40c If the two points shown each have two intersections, each occupied point should be reconstructed independently and the reconstruction procedure described in the "One Occupied Point" section should be used.

[0209] If Fig.41a The two occupied points shown are not adjacent (p 0 and p 2 or p 1 and p 3 ), each occupied point has two intersection points, but the boundary line segment can surround these two points ( Fig.41d ), or the two points can be separated by two boundary segments ( Fig.41e ). In these cases, the intersection point B should be investigated in S′ i to assess which situation we are in.

[0210] To distinguish these two cases, we can calculate the four subsegments of S′:

[0211] Contains the first occupied point p in S′ 0 The two intersection points B 0 and B 1 The points between Fig.41b );

[0212] Contains the second occupied point p in S′ 1 The two intersection points B 2 and B 3 The points between Fig.41b );

[0213] Contains the first occupied point p in S′ 0 The second intersection point B 1 and the second occupied point p 1 The first intersection point B 2 The points between Fig.41c );

[0214] Contains the second occupied point p in S′ 1 The second intersection point B 3 and the second first point p 0 The first intersection point B 0 The points between Fig.41c ).

[0215] The number of points in a sub-segment is used to know whether the point is inside or outside the boundary segment, and to perform a specific reconstruction process accordingly. In the following formula, we use the symbol |...| to represent the number of points in a sub-segment.

[0216] if Each occupied point should be reconstructed independently and the reconstruction process described in the "One occupied point" section should be used ( Fig.41e ).

[0217] if Then we can create list B 0 、p 0 , B 3 , B 2 、p 2 , B 1 , and B in S′ 0 and B 1 The point between S′ and B 3 and B 2 points between them to triangulate ( Fig.41d ).

[0218] Three occupied points

[0219] If three points are occupied, such as p in Figure 42 0 、p 1 and p 3 , the following process can be performed to remesh the corresponding space.

[0220] If two points each have a boundary point, and the other point has no boundary point ( Fig.42a and b), we can use point B with directional order 0 、p 1 、p 0 、p 3 , B 1 Construct a list of boundary points of the reconstructed grid and use B in S′ 0 and B 1 The points between complete the list. This list can be triangulated to build a mesh.

[0221] If one point has two intersections, and the other two points have only one intersection each, we can reconstruct the mesh using the process described previously ( Fig.42c ).

[0222] If the three points each have two boundary points, we can reconstruct the mesh using the process described earlier for an occupied point ( Fig.42d ).

[0223] Four occupied points

[0224] If all four points of the square are occupied, we can study the number of intersections at each point to reconstruct the grid.

[0225] If there is no boundary point at any point, then the square has no intersection with the boundary line segment and the square should be completely triangulated. According to the modulus of the u and v coordinates, the two triangles representing the square are (p 0 , p1, p 3 ) and (p 1 , p 2 , p 3 ) or (p 0 , p 1 , p 2 ) and (p 0 , p 2 , p 3 ) to ensure that the entire patch will be meshed by diamond-shaped oriented triangles ( Fig.43a and Figure 43b ).

[0226] If there are only two occupied points, each with an intersection, we can use point B with an oriented order 0 、p 1 、p 0 、p 3 、p 2 , B 1 and B in S′ 0 and B 1 The points between and form a list of boundary points for the reconstructed mesh. This list can be triangulated to construct the mesh ( Fig.43c ).

[0227] The remaining cases can be reconstructed using the reconstruction process based on one, two, or three occupied points described in the previous section.

[0228] If a point has two intersections, then the point is isolated and the procedure described in the section on an occupied point should be used ( Figure 43d , Figure 43f and Figure 43g ).

[0229] If two adjacent points each have only one intersection, the procedure described in the section on two occupied points should be used ( Figure 43e and Figure 43f ).

[0230] If, of the three adjacent points, the first and third have an intersection, but the second does not, then the procedure described in the section on three occupied points should be used ( Figure 43d ).

[0231] Fig.44 An example of applying the previously described process to a real patch is shown.

[0232] Possible evolution

[0233] The process described below can be applied to any type of mesh using floating point coordinates where the positions of the points of the boundary segments are not aligned with the 2D grid of the patch. If the points of the input mesh are quantized on a regular grid, we can align the 2D grid of the patch with the 3D grid used to quantize the input mesh, in which case the patch (S i ) will be located on the 2D square (B j ) and the reconstruction process described above will be simpler.

[0234] An evolution of the proposed algorithm could be to j ) stores the index of the line segment that created this intersection. This information can be used to know if two intersections come from the same line segment of the boundary segment, and in this case directly triangulate the space without building and studying the list of intersections S′ between them. Fig.39a In the figure, the two intersection points B 0 and B 1 From the same line segment, can be directly triangulated. In contrast, Fig.39b Midpoint B 0 and B 1 do not come from the same line segment, in which case B should be constructed 0 and B 1 List of points S′ between them to add points S during triangulation 0 .

[0235] Coding process

[0236] Based on the previous description, the encoding process is summarized as follows.

[0237] Fig.45 The main stages of the encoding and segmentation process are shown, where the mesh model is converted into patch information and 2D video that can be transmitted in the V-PCC bitstream.

[0238] The new proposed process updates the following key points:

[0239] · Fig.45 i: Occupancy rasterization is now described in the Occupancy Map Reconstruction section. (Note: Rasterization of depth maps has not changed)

[0240] · Fig.45 j: Geometry reconstruction.

[0241] And remove the following processes:

[0242] · Fig.45 k: Extract the boundaries between patches

[0243] Figure 451: Filling the space between patches

[0244] Fig.46 An updated encoding scheme is shown.

[0245] Decoding process

[0246] The newly proposed process also simplifies the decoding process.

[0247] New process updated Fig.47 The following points are described in:

[0248] Occupancy rasterization is now partially replaced by occupancy map reconstruction

[0249] Geometry reconstruction.

[0250] And remove the following processes:

[0251] Extract the boundaries between patches

[0252] Filling spaces between patches

[0253] Fig.48 An updated decoding scheme is shown.

[0254] Depth map filtering based on boundary segments

[0255] The patch boundary points and segments are losslessly encoded, so we can be sure that the positions of these points are correct.

[0256] In lossy mode, the geometry map is compressed using the video encoder and some noise is present on the depth values. Due to the depth filling process and the values ​​of neighboring patches stored in the depth map, we can observe more noise on the patch borders or patch areas close to the patch borders.

[0257] At the decoding end, knowing the exact location of the boundary points, we can correct the depth values ​​of the depth map pixels close to the edge of the boundary segment.

[0258] Based on the patch parameters, we can project the boundary segments in the depth map and use the calculated depth to correct the pixel values ​​of the depth map during rasterization at the boundary edges.

[0259] After this process, the pixel values ​​of the depth map (D) corresponding to the patch boundaries are accurate, and it makes sense to propagate this information into the patch to correct the near-boundary pixel values.

[0260] To do this, a second depth map (R) is used to store the exact depth values ​​of the patch calculated using the 3D border point positions. The border points are rasterized in this second depth map based on the patch parameters and the values ​​of other pixels are set using mipmap dilation.

[0261] This process is not limited to this implementation:

[0262] For each pixel (u, v) of the depth map D:

[0263] -If the distance d to the boundary segment is less than the threshold (t)

[0264] o then update the pixel value of D according to the following formula:

[0265]

[0266] Simplify patch boundary segments

[0267] The coordinates of the boundary points should be Draco encoded as described in the previous section. For low bit rate experiments, the size of the Draco bitstream can be significant, so it is valuable to reduce the size of this data.

[0268] To reduce the size of the Draco bitstream, we need to reduce the number of points used to describe the patch boundaries.

[0269] Once we know the points of the boundary segments, we can use several algorithms to clean the boundary points. In our implementation, we use the Douglas-Peucker algorithm to remove the points of the less representative segments.

[0270] Fig.26 An example of such simplification for a simple 2D polyline is shown.

[0271] Based on this process and according to a threshold parameter, we can simplify the 2D line segments of the boundary points. Figure 27 shows an example of simplified boundary line segments.

[0272] The table below shows the gain in the number of edges when using such simplifications.

[0273] Number of edges Reduction relative to the original original 2910 100.00% Threshold = 1 2555 87.80% Threshold = 2 2054 70.58% Threshold = 4 1213 41.68% Threshold = 8 456 15.67% Threshold = 16 233 8.01% Threshold = 32 166 5.70%

[0274] Another procedure that can be used to simplify boundary segments is based on Dijkstra's algorithm to compute the minimum path between two vertices on the mesh. Based on some parameters, the list of points (L) of each boundary segment is simplified based on some very simple rules: keep the first and the last point, remove one point from N, keep the extreme points, ...). This shortened list of points is called S. After the first stage, for each point in S, we add in the final list F the shortest path between the current point (s(i)) and the next point in S (s(i+1)) computed using Dijkstra's algorithm.

[0275] This process creates smoother boundaries with fewer points in the boundary segments.

[0276] Scale the depth of the patch

[0277] In V3C / V-PCC, the depth values ​​stored in the depth map are integer values ​​in the range [0, 2N-1], where N is the bit depth of the video, and in the reconstruction process, the depth values ​​are used to reconstruct the geometry. The normal coordinates of the reconstructed points are also integer values. This method is suitable for encoding point clouds that have been quantized on a discrete grid, where all points use integer coordinates, but there are some problems when encoding the depth values ​​of the center points of discrete edges.

[0278] If the two vertices v1 and v2 of an edge are quantized, the coordinates of these points are integer values ​​(x1, y1, z1) and (x2, y2, z2) respectively. The projection of the normal coordinates (e.g., Z coordinate) into the depth map will provide depth values ​​for these two points: z1 and z2, which are integer values.

[0279] For example, the projection of the edge (v1, v2) to the center v3 ((x2-x1) / 2, (y2-y1) / 2, (z2-z1) / 2) will provide a depth value: (z2-z1) / 2, which is not an integer value. In order to be encoded in a depth map, this value should be truncated and only the integer part of the depth should be kept.

[0280] Thereafter, the reconstructed mesh will suffer from aliasing. Figure 28 shows an example of this problem.

[0281] Figure 29 shows a 2D example of this problem. On the left, the edges of the original mesh are in blue. The two depth values ​​for the two vertices are correctly set in the depth map, and the corresponding points are well reconstructed on the right. But for the middle point, the depth value is truncated and the reconstructed line segment is incorrect.

[0282] To limit the impact of this problem, we recommend scaling the depth values ​​stored in the depth map according to the following parameters:

[0283] - Video bit depth used to store depth values

[0284] - The maximum depth of the current patch.

[0285] The depth value can be scaled linearly according to the following formula:

[0286]

[0287] Where N is the bit depth of the video and MaxDepth is the depth value. This value can be calculated based on all the depth values ​​of the patch, but can also be sent in the bitstream to allow for more accurate reconstruction. Figure 30 shows an example of this process in 2D.

[0288] FIG. 31 shows an example of a depth scaled 3D reconstructed patch.

[0289] To allow this process, the V3C / V-PCC syntax should be updated to indicate that the depth value of the current patch needs to be scaled before the reconstruction process. The updated syntax will affect the V3C syntax defined in w19579_ISO_IEC_FDIS_23090-5 and the following syntax elements:

[0290] -8.3.7.3 Syntax of patch data unit

[0291] -8.3.7.5 Syntax of Merged Patch Data Units

[0292] -8.3.7.6 Inter-panel data unit syntax

[0293] A) Encode a flag to indicate whether the depth should be scaled:

[0294] 8.3.7.3 Syntax of patch data unit

[0295]

[0296]

[0297] NOTE 1: The same line can be added in 8.3.7.5 Merged patch data unit syntax and 8.3.7.6 Inter-patch data unit syntax to encode mpdu_scale_depth_flag and ipdu_scale_depth_flag respectively.

[0298] NOTE 2: This new flag may not be encoded in the bitstream but copied from the reference slice for inter and merged slices. In this case, Section 9.2.5.5.1 "General decoding process for slice data units coded in inter prediction mode" in the w19579_ISO_IEC_FDIS_23090-5 document should be updated to explain that this value is copied from the reference slice to the inter slice, or Section 9.2.5.4 "Decoding process for slice data units coded in merge prediction mode" for merged slices should be updated accordingly. The copy function can be:

[0299] TilePatchScaleDepthValue[tileID][p]=refPatchScaleDepthValue

[0300] B) Encodes the maximum depth value that will be used to scale the depth values.

[0301] 8.3.7.3 Syntax of patch data unit

[0302]

[0303]

[0304] NOTE: The same lines can be added in 8.3.7.5 Merged Patch Data Unit Syntax and 8.3.7.6 Inter-Panel Data Unit Syntax to encode mpdu_scale_depth_value and ipdu_scale_depth_value respectively.

[0305] NOTE 2: This new value may not be encoded in the bitstream but copied from the reference slice for inter and merged slices. In this case, Section 9.2.5.5.1 "General decoding process for slice data units coded in inter prediction mode" in the w19579_ISO_IEC_FDIS_23090-5 document should be updated to explain the process of copying this value from the reference slice to the inter slice, or Section 9.2.5.4 "Decoding process for slice data units coded in merge prediction mode" for merged slices should be updated accordingly. The copy function can be:

[0306] TilePatchScaleDepthValue[tileID][p]=refPatchScaleDepthValue

[0307] Note 3: In addition to Note 2, the delta values ​​can also be stored in the bitstream in merged slices and inter-slices, in which case the syntax should be updated as follows:

[0308] 8.3.7.5 Merge patch data unit syntax

[0309]

[0310]

[0311] 8.3.7.6 Inter-patch data unit syntax

[0312]

[0313]

[0314] And Section 9.2.5.5.1 "General decoding process for patch data units encoded in inter prediction mode" in the w19579_ISO_IEC_FDIS_23090-5 document or Section 9.2.5.4 "Decoding process for patch data units encoded in merge prediction mode" for merged patches should be updated accordingly to define the replication function as:

[0315] TilePatchScaleDepthValue[tileID][p]=refPatchScaleDepthValue+IpduScaleDepthDelta[tileID][p]

[0316] TilePatchScaleDepthValue[tileID][p]=refPatchScaleDepthValue+mpduScaleDepthDelta[tileID][p]

[0317] Coding process

[0318] Based on the above description, the encoding process is summarized as follows: The V-PCC encoding process has been updated to use the process described above. Fig.16 The modified V-PCC encoder architecture is shown in Figure 2. This section describes the encoding process in more detail, in particular the new segmentation process that processes meshes instead of point clouds.

[0319] The segmentation process of the V-PCC encoder has been updated to take into account the mesh format, and in particular the surface topology information given by this new format, which is not available in the original point cloud format.

[0320] Fig.32 The main stages of the encoding and segmentation process are shown, where the mesh model is converted into patch information and 2D video that can be transmitted in the V-PCC bitstream. Fig.32 Each process in the diagram is described in more detail below.

[0321] Scale Position

[0322] The Mesh V-PCC encoder loads a 3D mesh model as input. It scales the bounding box of the input sequence to the range [0, 2N-1], where N is the geometry quantization bit depth set by the user ( Fig.32 a).

[0323] Preprocessing

[0324] Before the encoding and segmentation process, preprocessing is performed on the source model ( Fig.32 b) To correct the geometry and topology of the mesh to facilitate subsequent processes. Some triangles of the input model can be:

[0325] - empty (no area: two of the three points are the same or the three points are aligned),

[0326] - Poor connectivity, as shown in the example in Figure 33.d,

[0327] - Incorrect orientation relative to neighboring triangles, due to quantization errors the orientation of some triangles can change.

[0328] Some vertices that define a model may appear duplicated in the topological representation due to uv texture coordinates, normal values ​​or color values. To make subsequent processes more efficient, all duplicated vertices are merged (in terms of position coordinates) to reduce the number of vertices that should be used.

[0329] As shown in Figure 33, a mesh correction step is required to ensure that the shared patch boundary always contains the same line segments on both sides. In fact, by construction or as a result of quantization, some faces may have zero area. If the degenerate triangles with 3 co-linear vertices are removed, the boundary will consist of two line segments on one side and one line segment on the other side. The process detects empty triangles (labeled ABC on Figure 33), removes these triangles, and correctly connects the vertex C located on the edge AB to the adjacent triangle ABC' by splitting the adjacent triangle ABC' into two new triangles ACC' and CBC'.

[0330] The preprocessing described above ( Fig.32 .b) Correcting the previously mentioned problems, we obtain a mesh with the following characteristics:

[0331] - All triangles

[0332] · Not empty · Connected to all its adjacent triangles (shares a complete edge with its adjacent triangles)

[0333] Correct orientation

[0334] - All vertices with the same position are unique (i.e. recorded only once)

[0335] Initial connected component creation

[0336] In order to create a set of 2D patches that can be well encoded in a V-PCC bitstream, we need to group the triangles of the mesh into a set of connected components (CC, a set of connected triangles) that will be rasterized to create the 2D patches.

[0337] The connected components created need to have certain properties to ensure that all parts of the mesh model are well described in the patch and that the patch is not too complex to encode. The properties are:

[0338] - All triangles of a CC should have similar orientation in terms of triangle normals.

[0339] - All triangles stored in one CC should have maximum 2D projected area in the same projected 2D space (i.e. all triangles are well described in the projection plane chosen by the CC).

[0340] -Triangles of connected components at different depths should not be rasterized to the same 2D position (i.e., there should be no wrinkles on the CC surface and the projected areas of all triangles in the CC should not overlap).

[0341] - The depth range at which a triangle is projected according to the patch projection parameters should be in the range [0, N-1], where N is the bit depth of the geometry video.

[0342] - CC should be as large as possible to limit the number of CCs used to describe the mesh and thus ensure good compression efficiency.

[0343] To group triangles, we describe each triangle with a vector S of size numberOfProjection, where numberOfProjection is the number of projection planes used to describe the mesh (6 by default: {-X, +X, -Y, +Y, -Z, +Z}, more if 45° projection or extended projection planes mode is used. These modes can be activated with the following flags: ptc_45degree_no_projection_patch_constraint_flag and asps_extended_projection_enabled_flag.) Fig.32 c).

[0344] For each triangle j and each projection plane i in [0, numberOfProjection-1], we store in S the normalized signed area of ​​the 2D triangle projected onto the corresponding plane i:

[0345]

[0346] Where i is the projection plane index and j is the triangle index. The sign of the 2D area is set by comparing the projection plane direction to the 3D triangle direction. This can be done using the normal of the current 3D triangle. j and the normal of the projection plane used i It is calculated by the dot product between .

[0347] Each triangle is represented by a vector S that describes how well the triangle can be represented by the projection plane.

[0348] To limit the number of CCs created, the S vector of each triangle is averaged over its neighboring triangles ( Fig.32 d). To do this, use a N-V -1] 3where N is the geometry bit depth and V is the square size of the cell (input parameter: voxelDimensionRefineSegmentation).

[0349] For each 3D cell, we calculate the average value of S for all triangles intersecting the cell, denoted as cellS. The value of cellS is averaged over the neighboring cells (all cells whose distance is less than the input parameter searchSizeRefineSegmentation). For all triangles in the cell, we update S with cellS. This process can be performed multiple times depending on the input parameter iterationCountRefineSegmentation.

[0350] For each orientation i, we set the score of that orientation i to the normalized score of a virtual triangle parallel to the corresponding projection plane. These scores are denoted as ScoreProjPlane(i).

[0351] Based on the average score S, we can group each triangle j into each CC of index I with the following characteristics:

[0352] -The minimum distance between ScoreProjPlane(i) and S(j);

[0353] - Can be rasterized:

[0354] - All depth values ​​are in [0, 2 N -1]; and

[0355] - There are no depth values ​​that are too large (i.e. the absolute depth distance should be less than a threshold) compared to the depth values ​​of projected triangles in the neighboring area (the distance at all positions is less than a threshold: maxDepthVariationInNeighborhood).

[0356] During this process, if a CC is found to be too small (the number of triangles is lower than minNumberOfTriangleByCC), the triangles of this CC are deregistered and allowed to be attached to other CCs.

[0357] This process creates a preliminary connected component segmentation ( Fig.32 e), which will be refined in the following process.

[0358] Connected component thinning

[0359] To refine CCs, based on the preliminary CC segmentation, isolated triangles (not attached to the current CC by an edge) or triangles not attached to any CC are evaluated to see if they can be added to an existing CC.

[0360] Remove every isolated triangle from a CC that is not attached to any other triangle in that CC by an edge (two adjacent triangles should share two vertices) Fig.32 f).

[0361] For each triangle not represented in a CC, we check for each neighboring CC whether the triangle can be rasterized and connect the triangle to the largest CC among all possible CCs ( Fig.32 g).

[0362] Note: Connected components CC are now referred to as "patches" in the following sections

[0363] Boundary patch line segment extraction

[0364] The patch boundary segment extraction of the patch is described in Section 6.1 ( Fig.32 h).

[0365] Occupancy and depth map rasterization

[0366] An occupancy map for a patch can be created from the patch boundary segments, as described in Section 6.3 ( Fig.32 i).

[0367] Using the 3D triangles and occupancy map for each patch, we can rasterize the patch’s triangles in all the areas defined by the occupancy map and store these values ​​in the patch’s depth map.

[0368] According to the process defined in the previous section, the depth values ​​of the patch may or may not be scaled.

[0369] Geometry reconstruction

[0370] The 3D mesh of the patch can be reconstructed from the depth map and the occupancy map, as described in Section 6.4 ( Fig.32 .j).

[0371] Filling spaces between patches

[0372] The inter-patch space can be filled based on the boundary patch segments and the boundary edges of the reconstructed mesh of the patch, as described in Section 6.5 ( Fig.32 .k and Fig.32 .l).

[0373] Property box creation

[0374] Based on the reconstructed mesh (faces + inter-face filling), we can extract all the vertices of the mesh to create a reconstructed point cloud ( Fig.32 .m), denoted as RecPC.

[0375] This reconstructed point cloud has no color, and we need to color these points. Based on the source grid model, we can create a dense colored source point cloud by sampling and quantizing the source grid. This process will create a source colored point cloud, denoted as SourcePC.

[0376] Just like in V3C / V-PCC point cloud encoding, we can use a color transfer process that colors the reconstructed point cloud according to the source point cloud color. The color of the RecPC point is obtained from the closest corresponding point in SourcePC.

[0377] Each point in the point cloud has frame coordinates (u, v) that define the coordinates of the pixel in the depth map and attribute map that will be used to set the pixel value of the attribute frame for each reconstructed color point ( Fig.32 .n).

[0378] Decoding process

[0379] Based on the foregoing description, the decoding process is outlined below.

[0380] Parse the V3C / V-PCC bitstream to obtain the stored data:

[0381] - Patch metadata, including:

[0382] Patch parameters

[0383] Neighboring patch list

[0384] -Boundary point compression list bitstream

[0385] -Geometry video bitstream

[0386] -Attributes video bitstream

[0387] Decode the video bitstream to get the depth map and attribute map.

[0388] Decode a bitstream containing a list of boundary points to obtain boundary point segments.

[0389] The adjacent patch list is reconstructed patch by patch based on the pdu_delta_adjacent_patches, mpdu_delta_adjacent_patches and ipdu_delta_adjacent_patches information.

[0390] The neighboring patch list is used to reconstruct the patch boundary points and segments.

[0391] The patch boundary point segments are used to construct the occupancy map of the patch.

[0392] A mesh of patches is constructed based on the occupancy map, depth map, and attribute map.

[0393] Filling of the inter-patch space is performed between the reconstructed patches and the boundary point segments to obtain a reconstructed model.

[0394] Fig.34 These processes are shown.

[0395] The general aspects described in this article apply directly to the V-MESH coding draft. CfP was published in October 2021. These aspects have the potential to be adopted by the V-MESH standard as part of the specification.

[0396] Fig.35 An embodiment of a method 3500 under the general aspects described herein is shown. The method starts at a start block 3501, and control continues to a block 3510 to decode one or more video bitstreams to obtain a depth map and an attribute map. Control continues from block 3510 to a block 3520 to further decode a bitstream containing a boundary point list to obtain a boundary point line segment. Control continues from block 3520 to a block 3530 to generate a plurality of neighboring patch lists based on syntax information.

[0397] Control continues from block 3530 to block 3540 to obtain a patch boundary point line segment based on the boundary point and the neighboring patch list encoded in the bitstream;

[0398] Control continues from block 3540 to block 3550 to generate an occupancy map for the patch using the patch boundary point segments;

[0399] Control continues from block 3550 to block 3560 to construct a mesh of the patch based on the occupancy, depth, and attribute maps; and,

[0400] Control continues from block 3560 to block 3570 to fill the inter-patch space between the reconstructed patches and the boundary point segments to obtain a reconstructed model.

[0401] Fig.36 Another embodiment of a method 3600 according to the general aspects described herein is shown. The method begins at start block 3601 and control continues to block 3610 to correct the geometry and topology of the scaled three-dimensional mesh model.

[0402] Control continues from block 3610 to block 3620 to group the triangles of the mesh into connected components. Control continues from block 3620 to block 3630 to refine the connected components. Control continues from block 3630 to block 3640 to extract patch boundary line segments of the patch. Control continues from block 3640 to block 3650 to create an occupancy map of the patch based on the patch boundary line segments.

[0403] Control continues from block 3650 to block 3660 to rasterize the mesh of connected components of the patch to create a depth map and a depth video frame for the patch. Control continues from block 3660 to block 3670 to reconstruct a three-dimensional mesh of the patch based on the depth map and the occupancy map.

[0404] Control continues from block 3670 to block 3680 to fill the inter-patch space according to the patch boundary line segments and the boundary edges of the reconstructed mesh of the patches.

[0405] Control continues from block 3680 to block 3690 to extract all vertices of the mesh based on the reconstructed mesh to create a reconstructed point cloud. Control continues from block 3690 to block 3692 to color the points of the reconstructed point cloud. Control continues from block 3692 to block 3697 to create an attribute video frame using the colored reconstructed point cloud.

[0406] Fig.37 An embodiment of an apparatus 3700 for segmenting, compressing, analyzing, interpolating, representing, and understanding point cloud signals is shown. The apparatus includes a processor 3710 and may be interconnected to a memory 3720 via at least one port. The processor 3710 and the memory 3720 may also have one or more additional interconnections to external connections.

[0407] Processor 3710 is also configured to insert or receive information in a bitstream and perform segmentation, compression, analysis, interpolation, representation, and understanding of point cloud signals using any of the described aspects.

[0408] The embodiments described herein include various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specific and are usually described in a manner that may sound restrictive, at least to show the various features. However, this is for clarity of description and does not limit the application or scope of these aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with aspects described in previous files.

[0409] The aspects described and contemplated in this application may be implemented in a variety of different forms. Fig.35 , 36 and 37 provide some embodiments, but other embodiments are contemplated, and Fig.35 , 36The discussion of 37 does not limit the breadth of the embodiments. At least one of these aspects generally relates to segmentation, compression, analysis, interpolation, representation, and understanding of point cloud signals. These and other aspects can be implemented as methods; apparatus; computer-readable storage media storing instructions for segmenting, compressing, analyzing, interpolating, representing, and understanding point cloud signals according to any of the methods described; and / or computer-readable storage media storing a bitstream generated according to any of the methods described.

[0410] Various methods are described herein, and each method includes one or more steps or actions for implementing the method. Unless a specific order of steps or actions is required for the correct operation of the method, the order and / or use of specific steps and / or actions can be modified or combined.

[0411] Various numerical values ​​are used in this application. Specific values ​​are only for exemplary purposes, and the described aspects are not limited to these specific values.

[0412] Fig.38 A block diagram of an example of a system implementing various aspects and embodiments is shown. System 3800 may be embodied as a device including various components described below, and is configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, a variety of electronic devices, such as personal computers, notebook computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 1000 may be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components, either individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed over multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices, such as by a communication bus or by a dedicated input and / or output port. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0413] The system 1000 includes at least one processor 3710 configured to execute instructions loaded therein to implement, for example, the various aspects described in this document. The processor 3710 may include embedded memory, input-output interfaces, and various other circuits known in the art. The system 3700 includes at least one memory 3720 (e.g., a volatile memory device and / or a non-volatile memory device). The system 3700 may include a storage device, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. As non-limiting examples, the storage device may include an internal storage device, an additional storage device (including removable and non-removable storage devices), and / or a network accessible storage device.

[0414] Program code loaded onto the processor 3710 to perform various aspects described in this document may be stored in a storage device and subsequently loaded onto the memory 3720 for execution by the processor 3710. According to various embodiments, one or more of the processor 3710, memory 3720, or storage devices may store one or more of the various items during execution of the processes described in this document.

[0415] In some embodiments, memory internal to processor 3710 and / or memory 3720 is used to store instructions and provide working memory for the required processing. However, in other embodiments, memory external to the processing device (e.g., the processing device can be processor 3710 or an external device) is used for one or more of these functions. The external memory can be memory 3720 and / or a storage device, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, an operating system for a television.

[0416] The embodiments may be implemented by computer software or hardware or a combination of hardware and software implemented by the processor 3710. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 3720 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory as non-limiting examples. The processor 3710 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture as non-limiting examples.

[0417] When a diagram is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0418] The embodiments and aspects described herein may be implemented, for example, in a method or process, a device, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of embodiment (e.g., discussed only as a method), the embodiments of the features discussed may also be implemented in other forms (e.g., a device or program). The device may be implemented, for example, in appropriate hardware, software, and firmware. The method may be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0419] Reference to "an embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof means that the specific features, structures, characteristics, etc. associated with the embodiment are included in at least one embodiment. Therefore, the phrases "in an embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations thereof appearing in multiple places in the present application do not necessarily refer to the same embodiment.

[0420] Furthermore, the present application may involve “determining” a variety of information. Determining information may include one or more of the following, such as estimating information, calculating information, predicting information, or retrieving information from memory.

[0421] Furthermore, the present application may involve "accessing" various information. Accessing information may include one or more of the following, such as receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0422] Furthermore, the present application may involve "receiving" a variety of information. Like "accessing," receiving is a broad term. Receiving information may include one or more of the following, such as accessing information or retrieving information (e.g., from storage). Furthermore, "receiving" generally involves in some way during an operation such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0423] It should be understood that the use of any of the following " / ", "and / or", and "at least one", such as in the case of "A / B", "A and / or B", and "at least one of A and B", is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to as many items as listed as will be apparent to one of ordinary skill in this and related arts.

[0424] In addition, as used herein, the term "signal" refers, among other things, to indicating something to a corresponding decoder. For example, in some embodiments, the encoder sends a specific one of a plurality of transforms, coding modes or flags with a signal. Thus, in one embodiment, the same transform, parameter or mode is used on the encoder side and the decoder side. Thus, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling (implicit signaling) can be used without transmission to allow only the decoder to know and select specific parameters. Bit savings are achieved in various embodiments by avoiding the transmission of any actual function. It should be understood that signaling can be done in a variety of ways. For example, in various embodiments, one or more grammatical elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signal" is mentioned above, the word "signal" can also be used as a noun here.

[0425] As will be appreciated by those of ordinary skill in the art, embodiments may generate a variety of formatted signals to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for executing a method or data generated by one of the embodiments. For example, a signal may be formatted to carry a bitstream of the embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0426] We have described multiple embodiments, across a variety of claim categories and types. Features of these embodiments may be provided individually or in any combination. In addition, embodiments may include one or more of the following features, devices, or aspects, provided individually or in any combination, across a variety of claim categories and types:

[0427] Transfer the 3D boundary of the patch (a list of 3D points that make up the boundary of the patch)

[0428] Use boundary information to identify which parts of the 2D patch are occupied

[0429] Connect 3D reconstructed patches according to their true 3D boundaries

[0430] • Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of said syntax elements or variants thereof.

[0431] A television, set-top box, mobile phone, tablet computer or other electronic device that performs the conversion method according to any of the embodiments described above.

[0432] A television, set-top box, mobile phone, tablet or other electronic device that performs a transformation method according to any of the embodiments described and displays (e.g. using a monitor, screen or other type of display) the resulting image.

[0433] A television, set-top box, mobile phone, tablet or other electronic device that selects a channel, limits bandwidth or tunes (e.g. using a tuner) to receive a signal comprising an encoded image and performs a conversion method according to any of the described embodiments.

[0434] A television, set-top box, mobile phone, tablet or other electronic device that wirelessly receives (e.g., using an antenna) a signal including the encoded image and performs the transformation method.

Claims

1. A method for decoding a mesh model, include: decoding one or more video bitstreams to obtain a depth map and an attribute map; further decoding the bit stream containing the boundary point list to obtain boundary point line segments; Generate multiple adjacent face lists based on grammatical information; Obtaining a boundary point line segment of a patch according to the boundary point and the adjacent patch list encoded in the bitstream; Using boundary point segments of the patch to generate an occupancy map of the patch; constructing a mesh of the patch based on the occupancy map, the depth map, and the attribute map; as well as The inter-patch space between the reconstructed patch and the boundary point line segments is filled to obtain a reconstructed model.

2. An apparatus for decoding a grid model, comprising a memory and a processor, wherein the processor is configured to execute: decoding one or more video bitstreams to obtain a depth map and an attribute map; further decoding the bit stream containing the boundary point list to obtain boundary point line segments; Generate multiple adjacent face lists based on grammatical information; Obtaining a boundary point line segment of a patch according to the boundary point and the adjacent patch list encoded in the bitstream; Using boundary point segments of the patch to generate an occupancy map of the patch; A mesh of the patch is constructed based on the occupancy map, the depth map and the attribute map; and an inter-patch space between a reconstructed patch and the boundary point line segments is filled to obtain a reconstructed model.

3. A method for encoding a mesh model, include: Correcting the geometry and topology of the scaled three-dimensional mesh model; Group the triangles of the mesh into connected components; Refining the connected components; Extracting the face boundary line segments of the face; Creating an occupancy map of the patch based on the patch boundary segments; rasterizing a mesh of connected components of the patch to create a depth map and a depth video frame for the patch; reconstructing a three-dimensional mesh of the patch based on the depth map and the occupancy map; filling inter-patch spaces based on the patch boundary segments of the patches and boundary edges of the reconstructed mesh; extracting all vertices of the mesh based on the reconstructed mesh to create a reconstructed point cloud; Color the points of the reconstructed point cloud; as well as Use the colorized reconstructed point cloud to create attributed video frames.

4. An apparatus for encoding a grid model, comprising a memory and a processor, the processor being configured to execute: Correcting the geometry and topology of the scaled three-dimensional mesh model; Group the triangles of the mesh into connected components; Refining the connected components; Extracting the face boundary line segments of the face; Creating an occupancy map of the patch based on the patch boundary segments; rasterizing a mesh of connected components of the patch to create a depth map and a depth video frame for the patch; reconstructing a three-dimensional mesh of the patch based on the depth map and the occupancy map; filling inter-patch spaces based on the patch boundary segments of the patches and boundary edges of the reconstructed mesh; extracting all vertices of the mesh based on the reconstructed mesh to create a reconstructed point cloud; Coloring the points of the reconstructed point cloud; as well as Use the colorized reconstructed point cloud to create attributed video frames.

5. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein the generating of the occupancy map include: Calculating the intersection point with the boundary line segment of the face patch; as well as Indicates the intersection points that lie within the patch.

6. The method according to claim 1 or 3, or the apparatus according to claim 2 or 4, wherein said constructing a mesh of said patch and filling the space between patches include: reconstructing occupied pixels of the map in parallel; as well as Directly fill the space between patches.