Dynamic mesh coding with topology simplification

Abstract

In some embodiments, a mesh decoder reconstructs geometry information of a dynamic mesh from an encoded mesh stream of the dynamic mesh. The reconstructed geometry information includes data specifying vertices of the dynamic mesh. The decoder also reconstructs connectivity information of the dynamic mesh from the encoded mesh stream. The reconstructed connectivity information includes data specifying faces of the dynamic mesh. The decoder further refines the reconstructed connectivity information based on the reconstructed geometry information to generate refined connectivity information by dividing a face specified by the reconstructed connectivity information into at least two faces based on vertices specified in the reconstructed geometry information. The decoder reconstructs the dynamic mesh based on the reconstructed geometry information and the refined connectivity information, and causes the reconstructed dynamic mesh to be presented for display.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 266,176, filed on December 29, 2021, entitled “Topology Coding Method and Apparatus for Dynamic Mesh Coding,” U.S. Provisional Application No. 63 / 295,138, filed on December 30, 2021, entitled “Attribute Adjustment Method and Apparatus for Dynamic Mesh Coding with Simplified Topology,” and U.S. Provisional Application No. 63 / 266,175, filed on December 29, 2021, entitled “Attribute Reconstruction Filter for Dynamic Mesh Coding with Topology Simplification,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to computer-implemented methods and systems for dynamic mesh coding. Specifically, the present disclosure relates to dynamic mesh coding with topology simplification. Background Art

[0004] 3D graphics technology is integrated into a variety of applications, such as entertainment applications, engineering applications, manufacturing applications, and architectural applications. In these various applications, 3D graphics can be used to generate 3D models with amazing detail and complexity. Given the detail and complexity of 3D models, the data sets associated with the 3D models can be very large. In addition, these very large data sets can be transmitted, for example, over the Internet. Therefore, the transmission of large data sets, such as large data sets associated with detailed and complex 3D models, can become a bottleneck in various applications. As shown in this example, the development of 3D graphics technology has provided improved practicality for various applications, but it has also brought technical challenges. Therefore, improvements to 3D graphics technology represent improvements to the various technical applications that use 3D graphics technology. Therefore, technical improvements are needed to solve these technical problems and other technical problems related to 3D graphics technology. Summary of the Invention

[0005] Some embodiments relate to topology-simplified dynamic mesh encoding. In one example, a computer-implemented method for decoding an encoded mesh codestream representing a dynamic mesh of three-dimensional (3D) content includes: reconstructing geometric information of the dynamic mesh based on a geometry component codestream in the encoded mesh codestream, the reconstructed geometric information including data specifying a plurality of vertices of the dynamic mesh; reconstructing connectivity information of the dynamic mesh based on a connectivity component codestream in the encoded mesh codestream, the reconstructed connectivity information including data specifying a plurality of faces of the dynamic mesh; refining the reconstructed connectivity information based on the reconstructed geometric information to generate refined connectivity information by partitioning one of the plurality of faces specified by the reconstructed connectivity information into at least two faces based on one of the plurality of vertices specified in the reconstructed geometric information; reconstructing the dynamic mesh based on the reconstructed geometric information and the refined connectivity information; and rendering the reconstructed dynamic mesh for display.

[0006] In another example, a non-transitory computer-readable medium having program code stored thereon, the program code executable by one or more processing devices to perform operations. The operations include: reconstructing geometric information of a dynamic mesh based on a geometry component codestream in an encoded mesh codestream of the dynamic mesh, the reconstructed geometric information including data specifying a plurality of vertices of the dynamic mesh; reconstructing connectivity information of the dynamic mesh based on a connectivity component codestream in the encoded mesh codestream, the reconstructed connectivity information including data specifying a plurality of faces of the dynamic mesh; refining the reconstructed connectivity information based on the reconstructed geometric information to generate refined connectivity information by dividing one of the plurality of faces specified by the reconstructed connectivity information into at least two faces based on one of the plurality of vertices specified in the reconstructed geometric information; reconstructing the dynamic mesh based on the reconstructed geometric information and the refined connectivity information; and rendering the reconstructed dynamic mesh for display.

[0007] In another example, a system includes a processing device and a non-transitory computer-readable medium communicatively coupled to the processing device. The processing device is configured to execute program code stored in the non-transitory computer-readable medium to perform operations. The operations include: reconstructing geometric information of a dynamic mesh based on a geometry component codestream in an encoded mesh codestream of the dynamic mesh, the reconstructed geometric information including data specifying a plurality of vertices of the dynamic mesh; reconstructing connectivity information of the dynamic mesh based on a connectivity component codestream in the encoded mesh codestream, the reconstructed connectivity information including data specifying a plurality of faces of the dynamic mesh; refining the reconstructed connectivity information based on the reconstructed geometric information to generate refined connectivity information by dividing one of the plurality of faces specified by the reconstructed connectivity information into at least two faces based on one of the plurality of vertices specified in the reconstructed geometric information; reconstructing the dynamic mesh based on the reconstructed geometric information and the refined connectivity information; and rendering the reconstructed dynamic mesh for display.

[0008] These illustrative embodiments are not mentioned to limit or define the present disclosure, but rather to provide examples to aid understanding. Additional embodiments are discussed in the detailed description, and further description is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The features, embodiments, and advantages of the present disclosure may be better understood when the following detailed description is read with reference to the accompanying drawings.

[0010] Figure 1 An example encoder system for trellis coding according to various embodiments of the present disclosure is shown.

[0011] Figure 2 An example decoder system for trellis decoding according to various embodiments of the present disclosure is shown.

[0012] Figure 3 An example mesh frame associated with a color-per-vertex approach according to various embodiments of the present disclosure is shown.

[0013] Figure 4 Examples of a mesh frame and its underlying definition data associated with a color-per-vertex approach and corresponding 3D content are shown according to various embodiments of the present disclosure.

[0014] Figure 5 An example mesh frame associated with a 3D encoding method using vertex mapping according to various embodiments of the present disclosure is shown.

[0015] Figure 6Examples of data defining a mesh frame, corresponding 3D content, and corresponding property maps associated with a 3D encoding method using property mapping according to various embodiments of the present disclosure are shown.

[0016] Figure 7 Examples related to determining a surface direction in various 3D encoding methods according to various embodiments of the present disclosure are shown.

[0017] Figure 8 An example of an encoder system for trellis coding with connectivity reduction according to various embodiments of the present disclosure is shown.

[0018] Figure 9 An example of facet degradation in a mesh caused by lossy encoding according to various embodiments of the present disclosure is shown.

[0019] Figure 10 An example of connectivity simplification according to some embodiments of the present disclosure is provided.

[0020] Figure 11 An example decoder system for trellis decoding with connectivity refinement is shown in accordance with various embodiments of the present disclosure.

[0021] Figure 12 An example of performing a connectivity refinement process according to some embodiments of the present disclosure is shown.

[0022] Figure 13 Another example of performing a connectivity refinement process according to some embodiments of the present disclosure is shown.

[0023] Figure 14 An example of a mesh coding process with connectivity simplification according to some embodiments of the present disclosure is described.

[0024] Figure 15 An example of a process for decoding an encoded trellis code stream subjected to connectivity refinement according to some embodiments of the present disclosure is described.

[0025] Figure 16 An example of an encoder system for trellis coding for connectivity simplification and attribute adjustment as well as mapping adjustment according to various embodiments of the present disclosure is shown.

[0026] Figure 17 An example of affine transformation according to various embodiments of the present disclosure is shown.

[0027] Figure 18 An example decoder system for trellis decoding with connectivity refinement and map refinement is shown in accordance with various embodiments of the present disclosure.

[0028] Figure 19An example of a mesh coding process that performs connectivity simplification and attribute adjustment as well as mapping adjustment according to some embodiments of the present disclosure is described.

[0029] Figure 20 An example of a process for decoding an encoded trellis code stream subjected to connectivity refinement and map refinement according to some embodiments of the present disclosure is described.

[0030] Figure 21 An example of an encoder system for trellis coding for connectivity simplification and attribute adjustment as well as mapping adjustment according to various embodiments of the present disclosure is shown.

[0031] Figure 22 An example decoder system for mesh decoding with connectivity refinement, map refinement, and attribute refinement is shown, according to various embodiments of the present disclosure.

[0032] Figure 23 An example of a mesh coding process that performs connectivity simplification and mapping adjustment but no attribute adjustment is described according to some embodiments of the present disclosure.

[0033] Figure 24 An example of a process for decoding an encoded mesh code stream subjected to connectivity refinement, map refinement, and attribute refinement according to some embodiments of the present disclosure is described.

[0034] Figure 25 An example of a computing system is described that can be used to implement some embodiments of the present disclosure. DETAILED DESCRIPTION

[0035] Various embodiments provide for encoding dynamic meshes with simplified topology to improve coding efficiency. Due to the lossy nature of mesh coding, some information may be distorted during the encoding process, such as geometric information describing the coordinates of the mesh's vertices. Thus, the reconstructed position of a decoded vertex of the mesh may be different from the original position of the vertex. As a result, adjacent faces may degenerate into simplified geometric shapes. This degradation can be used to simplify the topological information contained in the connectivity data without introducing too much further distortion, thereby reducing the amount of data to be encoded and improving coding efficiency. Various embodiments described herein relate to encoding and decoding of dynamic meshes with simplified connectivity information.

[0036] The following non-limiting examples are provided to introduce some embodiments. In one embodiment, a mesh encoder accesses a dynamic mesh to be encoded. The dynamic mesh can be represented as an uncompressed mesh frame sequence comprising mesh frames. Each mesh frame includes at least one mesh tile or mesh slice, which includes data describing 3D content (e.g., a 3D object) in a digital representation as a collection of geometry information, connectivity information, attribute information, and attribute mapping information. The encoder can extract attribute components (including color information), geometry components (including a list of vertex coordinates), connectivity components (including a list of faces with corresponding vertex indices and texture indices), and mapping components (including a list of projected vertex attribute coordinate information) from the uncompressed mesh frame sequence.

[0037] The encoder encodes the geometry components and further decodes the encoded geometry components to generate reconstructed geometry information. The encoder identifies vertices in the reconstructed geometry information that may be removed from the connectivity components due to degradation caused by coding distortion. The encoder further identifies adjacent faces that can be merged if the vertices are removed. The encoder updates the connectivity components of the dynamic mesh to simplify the connectivity information, thereby replacing these faces with merged faces. The encoder encodes the updated connectivity components to generate a connectivity component codestream. The encoder further weights the encoding of other components and generates a coded mesh codestream by including the encoded components.

[0038] By merging faces and removing vertices, the connectivity component contains less data, so the encoded connectivity component requires fewer bits to represent than the original connectivity component. As a result, the size of the entire encoded mesh codestream is also smaller than the size of the mesh codestream encoded without connectivity simplification. In addition, because the vertices to be removed are selected in a way that produces less distortion (for example, the vertices with the smallest distance to the edge of the face are selected for removal), the visual artifacts introduced by the simplification process are minimized. In this way, the encoded mesh codestream can be decoded using a conventional mesh decoder without knowing that the encoded mesh codestream was generated by simplifying the connectivity information.

[0039] In another embodiment, the connectivity-simplified encoded mesh code stream can be decoded using a mesh decoder with connectivity refinement to further reduce the distortion and visual artifacts introduced by the connectivity simplification on the encoding side. In order to perform connectivity refinement, the mesh decoder can be configured with a connectivity refinement module to restore the faces that have been merged during the connectivity simplification. Based on the reconstructed geometric information, the decoder identifies the vertices located within the face specified by the reconstructed connectivity information and divides the face into two or more faces based on the positions of the vertices to generate refined connectivity information. The refined connectivity information is used to reconstruct the dynamic mesh and other decoded components of the mesh. Due to the refinement process, the connectivity information used to reconstruct the mesh is closer to the connectivity information before the connectivity simplification when encoded. As a result, the reconstructed mesh has less distortion and fewer visual artifacts than the mesh decoded by a conventional mesh decoder.

[0040] In another embodiment, in addition to connectivity simplification, the mesh encoder can further reproject the attribute image based on the distortion introduced by the geometric information encoding and reconstruction. The distortion can be described by the transformation applied to a point that belongs to the edge between two adjacent triangles that are degenerated and merged during connectivity simplification. Therefore, the reprojection process can be implemented as the same transformation as the transformation applied to the face corresponding to the degenerated triangle face in the attribute image. Similarly, the mapping information can also be updated to be consistent with the connectivity information. Updating the attribute image and mapping information in this way reduces visual artifacts caused by geometric information reconstruction errors.

[0041] In yet another embodiment, instead of reconstructing the distortion and updating the attribute image based on geometric information at the encoder side, the attribute image is refined at the decoder side. A process similar to the previous embodiment can be used to estimate the transformation, and the inverse transform of the estimated transformation can be applied to the decoded attribute image before the attribute image is used to reconstruct the dynamic mesh. Compared to the previous embodiment in which the attribute image is updated at the encoder, this embodiment allows the mesh to be encoded faster with less computational resource consumption. This burden is transferred to the decoder, which has looser timing requirements and more powerful computing power than the encoder. Therefore, this embodiment is more suitable for applications such as live broadcasting.

[0042] The descriptions of various embodiments provided herein may include one or more of the terms listed below. For purposes of illustration and not limitation of the present disclosure, exemplary descriptions of terms are provided herein.

[0043] Mesh: A collection of vertices, edges, and faces that can define the shape / topology of a polyhedral object. Faces can include triangles (e.g., a triangle mesh).

[0044] Mesh slice: A collection of vertices, edges, and faces that defines the shape / topology of a polygon. A mesh frame is composed of several mesh slices.

[0045] Dynamic mesh: A mesh with at least one of various possible components (e.g., connectivity, geometry, mapping, vertex attributes, and attribute mapping) that changes over time.

[0046] Animated Mesh: A dynamic mesh with constant connectivity.

[0047] Connectivity: A set of vertex indices that describes how mesh vertices are connected to create a 3D surface (e.g., geometry and all attributes can share the same unique connectivity information).

[0048] Geometry: A set of vertex 3D (e.g., x, y, z) coordinates that describe the positions associated with mesh vertices. The coordinates representing positions (e.g., x, y, z) can have limited precision and dynamic range.

[0049] Mapping: describes how to map a mesh surface to a flat 2D area. This mapping can be described by a set of UV parameters / texture (e.g., mapping) coordinates and connectivity information associated with the mesh vertices.

[0050] Vertex attribute: A scalar vector attribute value associated with a mesh vertex.

[0051] Attribute Map: Attributes associated with a mesh surface, stored in the form of a 2D image / video. The mapping between the video (e.g., parameter space) and the surface can be defined by the mapping information.

[0052] Vertex: A position (e.g., in 3D space) along with other information such as color, normal, and texture coordinates.

[0053] Edge: The connection between two vertices.

[0054] Face: A closed set of edges, where a triangular face has three edges bounded by three vertices. The orientation of a face can be determined using a "right-handed" coordinate system.

[0055] Orientation of a face: This is defined by the order of the vertices in the face. Some transpositions are allowed, while other transpositions of the vertices in a face can result in different orientations.

[0056] Surface: A collection of faces that separates a three-dimensional object from its environment.

[0057] Connectivity Coding Unit (CCU): A connectivity coding sample of a square unit of size N×N, which carries connectivity information.

[0058] Connectivity Coding Sample: The coding element of connectivity information, calculated as the element difference between the current surface and the predicted surface.

[0059] Block: Represents a mesh segment as a collection of connectivity coded samples, represented as three attribute channels. A block can consist of multiple CCUs.

[0060] bits per point (bpp): The amount of information, in bits, that can be used to describe a point in the grid.

[0061] 3D content, such as 3D graphics, can be represented as a mesh (e.g., 3D mesh content). The mesh can include vertices, edges, and faces that describe the shape or topology of the 3D content. The mesh can be divided into blocks (e.g., segments, tiles). For each block, vertex information associated with each face can be sorted in order (e.g., descending order). When the vertex information associated with each face is sorted in order, the faces are sorted in order (e.g., ascending order). The sorted faces in each block can be packed into a two-dimensional (2D) frame. Sorting the vertex information can ensure an increasing order of vertex indices, which facilitates improved mesh processing. Connectivity information components in the 3D mesh content can be converted from one-dimensional (1D) connectivity components (e.g., lists, face lists) to a 2D connectivity image (e.g., a connectivity encoding sample array). By converting the connectivity information in the 3D mesh content into a 2D connectivity image, video coding processing can be applied to the 2D connectivity image (e.g., as a video connectivity frame). In this way, by utilizing a video coding solution, 3D mesh content can be efficiently compressed and decompressed. According to these methods, encoded 3D mesh content can be efficiently decoded. Connectivity components can be extracted from the encoded dynamic mesh code stream and decoded into frames (e.g., images). Connectivity code samples corresponding to pixels in the frames are extracted. The 3D mesh content can be reconstructed based on the extracted connectivity information.

[0062] The coded bitstream of a dynamic grid is represented as a collection of components, consisting of a grid bitstream header and a data payload. The grid bitstream header may include a sequence parameter set, a picture parameter set, adaptation parameters, tile information parameters, and supplemental enhancement information. The grid bitstream payload may include a coded atlas information component (auxiliary information required to transform the local coordinate system of a block into the global coordinate system of the grid frame), a coded attribute information component, a coded geometry (position) information component, a coded mapping information component, and a coded connectivity information component.

[0063] Figure 1 An example encoder system 100 for trellis coding according to various embodiments of the present disclosure is shown. Figure 1 As shown, an uncompressed grid frame sequence 102 can be input to an encoder system 100, and the example encoder system 100 can generate an encoded grid frame sequence 124 based on the uncompressed grid frame sequence 102. Generally, a grid frame sequence includes multiple grid frames. A grid frame is a data format that describes 3D content (e.g., a 3D object) in a digital representation as a collection of geometric information, connectivity information, attribute information, and attribute mapping information. Each grid frame is characterized by a presentation time and a duration. A grid frame sequence (e.g., a sequence of multiple grid frames) forms a dynamic grid video.

[0064] like Figure 1 As shown, the encoder system 100 can generate encoded grid sequence / picture header information 106 based on the uncompressed grid frame sequence 102. The encoded grid sequence / image header information 106 may include image header information, such as a sequence parameter set (SPS), a picture parameter set (PPS), a slice header (SH), and supplemental enhancement information (SEI). The grid code stream header 132 may include the encoded grid sequence / picture header information 106. The uncompressed grid frame sequence 102 may be input to the grid segmentation module 104. The grid segmentation module 104 segments the uncompressed grid frame sequence 102 into block data and segmented grid data. The grid code stream payload 130 may include block data and segmented grid data. The grid code stream header 132 and the grid code stream payload 130 may be multiplexed together by the multiplexer 122 to generate the encoded grid frame sequence 124.

[0065] The encoder system 100 may include a block segmentation information module 108 that generates block segmentation information (e.g., atlas information) based on the block data. Based on the segmented mesh data, the encoder system 100 may generate an uncompressed attribute component using an attribute image synthesis module 110, an uncompressed geometry component using a geometry image synthesis module 112, an uncompressed connectivity component using a connectivity image synthesis module 114, and an uncompressed map component using a map image synthesis module 116. Figure 1 As shown, the connectivity image synthesis module 114 and the mapping image synthesis module 116 may also use the block segmentation information generated by the block segmentation information module 108 when generating the corresponding components. As an example of the generated information, the block segmentation information may include binary atlas information. The attribute image component may include RGB component information and YUV component information (e.g., RGB 4:4:4, YUV 4:2:0). The geometry component may include 3D coordinates of vertex information in a canonical or local coordinate system (e.g., XYZ 4:4:4, XYZ 4:2:0). The connectivity component may include vertex index and texture vertex information (e.g., dv0, dv1, dv2 4:4:4). The mapping component may include texture vertex information (e.g., UV 4:4:X). These generated components may be represented as images.

[0066] The block segmentation information may be provided to a binary entropy encoder 118 to generate an atlas component. The binary entropy encoder 118 may be a lossless encoder, allowing the encoded information to be recovered without any distortion. The uncompressed attribute component, generated by the attribute image synthesis module 110 and represented as an image, may be provided to a video encoder 120a to generate an encoded attribute component. The video encoder 120a may be a lossy encoder, where the encoded information may not be fully recoverable at the decoder. Similarly, the geometry component, represented as an image, may be provided to a video encoder 120b to generate an encoded geometry component. The video encoder 120b may also be a lossy encoder. The connectivity image component, represented as an image, may be provided to a video encoder 120c to generate an encoded connectivity component. The video encoder 120c may be a lossless encoder. The map component, represented as an image, may be provided to a video encoder 120d to generate an encoded map component. The video encoder 120d may be a lossless encoder. Video encoders 120a through 120d can be any video or image encoder capable of compressing information in a video sequence or image to reduce the video size, such as an H.264 video encoder, an H.265 video encoder, an H.266 video encoder, a JPEG image encoder, and the like. Video encoders 120a through 120d can use the same type of video encoder or different types of video encoders. The grid codestream payload 130 can include an atlas component, an attribute component, a geometry component, a connectivity component, and a mapping component. The grid codestream payload and the grid codestream header are multiplexed together by a multiplexer 122 to generate an encoded grid frame sequence 124.

[0067] Figure 2 An example decoder system 200 for trellis decoding according to various embodiments of the present disclosure is shown. Figure 2 As shown, the encoded grid frame sequence 224, for example, Figure 1 The encoded grid frame sequence 124 generated by the encoder system 100 is input to the decoder system 200. The encoded grid frame sequence 224 can be a code stream. The example decoder system 200 can generate a reconstructed grid frame sequence 202 based on the encoded grid frame sequence 224.

[0068] like Figure 2As shown, decoder system 200 demultiplexes the encoded trellis frame sequence 224 using demultiplexer 222 to identify various components of the encoded information, including the encoded trellis sequence / picture / slice header information 206 and the encoded block partitioning information, which can be decoded using entropy decoder 218. The demultiplexed information also includes an encoded geometry component, an encoded connectivity image component, an encoded map component, and an encoded attribute component. The identified various encoded components can be decoded using video decoders 220a through 220d, which correspond to the respective video encoders used to encode the information indicated in the encoded trellis sequence header 106, such as video encoders 120a through 120d. Similar to video encoders 120a through 120d, video decoders 220a through 220d can be any video decoder or image decoder.

[0069] Corresponding processing modules may also be used to process the decoded video data, such as the attribute image decoding module 210, the geometry image decoding module 212, the connectivity image decoding module 214, and the mapping image decoding module 216. These decoding modules convert the decoded video data into corresponding data formats. For example, for geometry data, the decoded images in the video can be reformatted back into canonical XYZ 3D coordinates to generate geometry data. Similarly, the decoded connectivity video / image can be reformatted into connectivity encoded samples dv0, dv1, dv2 to generate decoded connectivity data; the decoded mapping video / image can be reformatted into uv coordinates to generate decoded mapping data; and the decoded attribute video / image can be used to generate RGB or YUV attribute data for the mesh.

[0070] The geometry reconstruction module 232 reconstructs geometry information based on the decoded 3D coordinates. The connectivity reconstruction module 234 reconstructs topology (e.g., surfaces) based on the decoded connectivity data. The mapping reconstruction module 236 reconstructs attribute maps based on the decoded mapping data. Using the reconstructed geometry information, surfaces, mapping data, attribute data, and the decoded mesh sequence / picture header information 206, the mesh reconstruction module 226 reconstructs the mesh to generate a reconstructed mesh frame sequence 202.

[0071] Figures 3 to 7 Examples of information associated with encoding and decoding a mesh according to various embodiments of the present disclosure are shown. In various methods for encoding 3D content, geometry information, attribute information, and connectivity information are encoded in a mesh frame. For example, in the per-vertex color method, attribute information is stored together with geometry information, and connectivity information and associated vertex indices are stored in the mesh frame. Figure 3An example mesh frame 300 associated with a per-vertex color approach according to various embodiments of the present disclosure is shown. Figure 3 As shown, the geometry and attribute information 302 can be stored in the mesh frame as an ordered list of vertex coordinate information. Each vertex coordinate is stored together with the corresponding geometry and attribute information. The connectivity information 304 can be stored in the mesh frame as an ordered list of face information, each face including a corresponding vertex index and texture index.

[0072] Figure 4 An example 400 of a mesh frame 402, its underlying definition data 406 associated with a per-vertex color method, and corresponding 3D content 404 is shown, according to various embodiments of the present disclosure. As shown in mesh frame 402 and defined in corresponding data 406, geometric coordinates with associated attribute information and connectivity information are stored in the mesh frame. The geometric and attribute information are stored as an ordered list of vertex geometric coordinates with associated attribute information, while the connectivity information is stored as an ordered list of face information with corresponding vertex indices. The geometric and attribute information shown in mesh frame 402 includes four vertices. The positions of the vertices are represented by X, Y, and Z coordinates, and the color attributes are represented by a_1, a_2, and a_3 values ​​representing R, G, and B color values. The connectivity information shown in mesh frame 402 includes three faces. Each face includes three vertex indices listed in the geometric and attribute information to form a triangular face. By using the vertex index of each corresponding face to point to the geometric and attribute information stored for each vertex coordinate, 3D content 404 (e.g., a 3D triangle) can be decoded based on mesh frame 402.

[0073] Figure 5 An example uncompressed mesh frame 500 associated with a 3D encoding method using texture mapping according to various embodiments of the present disclosure is shown. Figure 5 As shown, geometric information 502 can be stored in the mesh frame as an ordered list of vertex coordinate information. Each vertex coordinate is stored together with the corresponding geometric information. Attribute information 504 can be stored in the mesh frame as an ordered list of projected vertex attribute coordinate information, separate from the geometric information 502. The projected vertex attribute coordinate information is stored as 2D coordinate information together with the corresponding attribute information. Connectivity information 506 can be stored in the mesh frame as an ordered list of face information, each face including a corresponding vertex index and texture index. In some examples, the mesh frame is formatted according to the wavefront OBJ file format.

[0074] Figure 6 An example 600 of data 602 defining a grid frame, corresponding 3D content 604, and a corresponding property map 606 associated with a 3D encoding method using property mapping is shown according to various embodiments of the present disclosure. Figure 6 As shown, the geometric information, mapping information (e.g., attribute information), and connectivity information are stored in a mesh frame generated based on the information described in data 602. The geometric information contained in the mesh frame includes four vertices. The positions of the vertices are represented by X coordinates, Y coordinates, and Z coordinates. The mapping information in the mesh frame includes five texture vertices. The positions of the texture vertices are represented by U coordinates and V coordinates. The connectivity information in the mesh frame includes three faces. Each face contains three pairs of vertex indices and texture vertex coordinates. Figure 6 As shown, 3D content 604 (e.g., an object formed by triangles in 3D space) and an attribute map 606 can be decoded based on a mesh frame using vertex indices and texture vertex coordinate pairs for each face. Attribute information associated with the attribute map 606 can be applied to the 3D content 604 to apply the attribute information to the 3D content 604. In this example, the coordinates are normalized to a scale of -1.0 to +1.0 for each axis in the geometry and to a scale of 0.0 to +1.0 for the attribute map. The coordinates in the mesh encoder are first converted from floating point values ​​to a fixed point representation with a given bit depth and then compressed by the mesh encoder.

[0075] Figure 7 7 shows an example 700 associated with determining face directions in various 3D encoding methods according to various embodiments of the present disclosure. Figure 7 As shown, a right-handed coordinate system can be used to determine the direction of the face. Each face shown in example 700 includes three vertices, which form three edges. Each face is described by three vertices. In the manifold mesh 702, each edge belongs to at most two different faces. In the non-manifold mesh 704, an edge can belong to two or more different faces. In both the manifold mesh 702 and the non-manifold mesh 704, a right-handed coordinate system can be applied to determine the direction of the face, which can also be referred to as the normal vector direction or the face normal direction.

[0076] Figure 8 An example of an encoder system 800 for trellis coding with connectivity reduction is shown, according to various embodiments of the present disclosure. Figure 8 The encoder system 800 shown includes a Figure 1 In addition, the encoder system 800 includes a geometry reconstruction module 802, a reconstruction geometry analysis module 804, and a connectivity image simplification module 806. The connectivity image simplification module 806 is used to simplify connectivity information before encoding to improve encoding efficiency.

[0077] For example, the geometry reconstruction module 802 can decode the encoded geometry components generated by the video encoder 120b back into a geometry component image. In examples where the video encoder 120b is a lossy encoder, the reconstructed geometry component image will differ from the geometry component image due to prediction and quantization. Consequently, the reconstructed positions of decoded mesh vertices may differ from their original positions. Consequently, adjacent faces may degenerate into simplified geometric shapes. Figure 9 An example of topological degradation is shown in Figure 9 , original mesh 902 includes five triangular faces: A, B, C, D, and E. After reconstruction, the positions of the reconstructed vertices in reconstructed mesh 904 are shifted compared to the positions of the vertices in original mesh 902. In this example, the geometric primitives in original mesh 902 degenerate into planar or nearly planar surfaces. This planar surface can be represented using two faces: face 908A (approximately the union of faces A and B) and face 908B (approximately the union of faces C, D, and E); or face 910A (approximately the union of faces A and C) and face 910B (approximately the union of faces B, D, and E). Reconstruction error 906 indicates that the positions of the vertices in reconstructed mesh 904 are shifted compared to the original mesh 902.

[0078] Back to Figure 8 Based on the decoded / reconstructed geometric component image, the reconstruction geometry analysis module 804 can analyze the geometric information to determine vertices that can be removed from the connectivity component image, thereby simplifying the connectivity component image. To determine the vertices to be removed, the four vertices of a pair of adjacent faces or a pair of adjacent triangles of the mesh can be analyzed. In some examples, the two adjacent triangles have the same orientation. The reconstruction geometry analysis module 804 can use three of the four vertices to form a triangle, where the fourth vertex is within the triangle. The distance from the fourth vertex to each of the three sides of the triangle can be determined. The minimum of the three distances is determined. If the minimum distance is below a predetermined threshold, the vertex can be selected as the vertex to be removed. In some examples, the threshold can be adaptively defined by the encoder system 800 or passed to the encoder system 800 as user input. The resulting triangle can be used as the triangle to be encoded. In other words, the pair of adjacent triangles can be merged into a single triangle. The process can be repeated for another pair of adjacent triangles.

[0079] The reconstruction geometry analysis module 804 provides the analysis results to the connectivity image simplification module 806, which can simplify the connectivity component image. The analysis results can include vertices selected to be removed and, in some examples, triangles to be merged. The connectivity image simplification module 806 can remove the selected vertices from the connectivity component image and update the connectivity component image to replace the data of adjacent triangle pairs with corresponding merged triangles. The simplified connectivity component image can be provided to the video encoder 120c for encoding, as described above with respect to Figure 1 The multiplexer 122 can then be used with respect to Figure 1 The coded information is multiplexed in a similar manner to that described to generate the coded grid frame sequence 810. Due to the connectivity simplification, the coded grid frame sequence 810 is different from the coded grid frame sequence 124 generated using the encoder system 100 and has a smaller size than the coded grid frame sequence 124. Although Figure 8 It is shown that three independent modules are involved in connectivity simplification, namely, a geometry reconstruction module 802, a reconstruction geometry analysis module 804, and a connectivity image simplification module 806, but any number of modules can be used to implement connectivity simplification.

[0080] Figure 10 An example of connectivity simplification according to some embodiments of the present disclosure is provided. In this example, subgraph (A) shows the original geometric information, which specifies four vertices P, V1, V2, and V3. Subgraph (B) shows the original geometric information and the original connectivity information, which define two adjacent triangles based on these four vertices: triangle P_V2_V3 and triangle P_V1_V2. These two triangles have the same counterclockwise direction. Subgraph (C) shows the reconstructed geometric information and connectivity information. The reconstructed geometric information includes vertices P', V1', V2', and V3', and due to the lossy compression of the video encoder, the coordinates of these vertices may be different from the coordinates of their respective original vertices. Among the four reconstructed vertices, V1', V2', and V3' can form a triangle 1008, wherein the fourth reconstructed vertex P' is within the triangle.

[0081] The distance from P' to each of the three sides of triangle 1008 is calculated. In some examples, the distance is measured as the length of a line perpendicular to the side from the vertex to the side. For example, the distance from P' to side V1'V3' can be determined as the length of line 1010 perpendicular to side V1'V3' and connecting P'. Alternatively or additionally, the distance can be defined as the length of a line created by extending the line connecting the vertex opposite the side and P' until the line intersects the side. In this example, the distance from P' to side V1'V3' can be determined as the length of line 1012 (i.e., line P'P") created by extending line V2'P' until it intersects side V1'V3' at point P". The distance from P' to the other two sides, V1'V2' and V2'V3', can be calculated in a similar manner. The minimum of the three distances can be selected and compared to a predetermined threshold to determine whether point P' can be removed or excluded from the encoding. If the minimum distance is less than a predetermined threshold, the face associated with point P' can be marked as removed or excluded from the connectivity encoding. In this case, the two triangles P_V2_V3 and P_V1_V2 can be merged into one triangle V1'V2'V3, which also maintains the counterclockwise orientation of the original two triangles. The face associated with point P' within the merged triangle is excluded from the connectivity component image. With this simplification, the reconstructed connectivity information includes only one triangle, and the reconstructed geometric information includes the three vertices of the triangle and one vertex P' within the triangle, as shown in subfigure (D). In some examples, only triangles with the same orientation can be merged.

[0082] The encoded grid frame sequence 810 can be stored or transmitted and used as described above with respect to Figure 2 The decoder system 200 is used to decode the coded trellis frame sequence 810 with simplified connectivity. In other words, the decoder system can decode the coded trellis frame sequence 810 with simplified connectivity without considering that the connectivity information has been simplified.

[0083] In another embodiment, the decoder system can be configured to reconstruct the original connectivity information based on the simplified connectivity information. This can be performed using vertices decoded from geometric information that was not simplified during encoding. Thus, the decoding process can include a demultiplexer of the decoder extracting corresponding components from an encoded mesh code stream, the encoded mesh code stream including a geometry component code stream, an attribute component code stream, a connectivity component code stream, and a mapping component code stream. Each component of the encoded mesh code stream is then processed by a decoder for the corresponding component. For example, the attribute component can be decoded by a video decoder indicated in the mesh code stream header. Similarly, other components can also be decoded by the corresponding video decoder indicated in the mesh code stream header. Based on the extracted components, a reconstruction process is applied to generate a reconstructed mesh frame.

[0084] In some examples, the decoding process includes the following steps: obtaining an encoded mesh codestream from a network or storage source, demultiplexing components of the encoded codestream, and decoding a geometry component codestream, an attribute component codestream, a connectivity component codestream, and a mapping component codestream in parallel or otherwise independently. The decoding process also includes refining the decoded connectivity information to reconstruct or restore the original connectivity information that was simplified by subdividing the faces or triangle edges containing vertices therein. The refined and decoded components can be used to reconstruct the mesh frame. By refining the decoded connectivity information, distortion and visual artifacts in the decoded mesh frame introduced by the connectivity simplification can be reduced.

[0085] Figure 11 An example decoder system 1100 for trellis decoding with connectivity refinement is shown, according to various embodiments of the present disclosure. Figure 11 The decoder system 1100 shown includes Figure 2 Furthermore, the decoder system 1100 includes a connectivity refinement module 1102 to restore connectivity information that was simplified at the encoder before the connectivity information is sent to the mesh reconstruction module 226 to reconstruct the mesh frame sequence 1104 .

[0086] In some examples, the connectivity refinement module 1102 obtains the geometry information decoded by the geometry reconstruction module 232 from the geometry component codestream. The vertex coordinates (X, Y, Z) are reconstructed using the decoded geometry components and the atlas components. The connectivity refinement module 1102 also obtains the connectivity information decoded by the connectivity reconstruction module 234 from the decoded connectivity component codestream. The connectivity refinement module 1102 can mark the vertices in the geometry component of the decoded connectivity information that belong to the face list as visited, and mark the vertices that do not belong to the face list as unvisited.

[0087] For each unvisited vertex P′, the connectivity refinement module 1102 finds the face or triangle in which P′ is located. For example, Connectivity refinement module 1102 projects vertex P′ to the nearest edge of the triangle (e.g., ) to find the projected vertex P". In some examples, the nearest edge is identified by the minimum Euclidean distance between the vertex and the edge. The projected vertex P" is defined as the vertex from the triangle The relative vertex V′2 is projected onto the triangle The nearest edge The connectivity refinement module 1102 subdivides the triangle V′1V′2V′3 into two triangles, both of which have the same Same direction. If the triangle If there is more than one point in the triangle, the connectivity refinement module 1102 can recursively iterate the process starting from the point with the largest Euclidean distance from the edge. The updated triangle is then used for further subdivision. The refined connectivity information can be used to reconstruct the mesh frame together with other decoded components.

[0088] Figure 12 An example of performing connectivity refinement process on the decoding side according to some embodiments of the present disclosure is shown. Figure 12 In , the decoded geometry consists of four points: P', V1', V2', and V3', as shown in sub-figure (A). Without connectivity simplification, these four points should form two faces: and Due to the simplification during encoding, the two faces have been merged, as shown in sub-figure (B), which shows the decoded geometry and connectivity information. In sub-figure (B), V1′, V2′, and V3′ form a face / triangle, and point P′ is inside the face. Since the connectivity information was simplified at the encoder, the isolated point P′ inside the face indicates that the decoded triangle is a merged triangle from at least two triangles. To recover these triangles, in this example, triangle The nearest edge to P' in P' is projected onto the nearest edge Thus, the projected vertex P is obtained. Projection can be performed by extending the ray cast from the opposite vertex V2′ of the edge to find the intersection with the edge, as shown in Figure 12 Then the triangle V′1V′2V′3 is divided into two triangles V1′P″V2′ and P″V2′V3′, each of which has the same Therefore, the decoded geometry information and refined connectivity information shown in sub-figure (D) include two triangles instead of a merged triangle.

[0089] Figure 13 Another example of a process for performing connectivity refinement according to some embodiments of the present disclosure is shown. Figure 13 In the decoded triangle, there is more than one point. As shown in sub-figure (A), the decoded geometry information includes five points: V1′, V2′, V3′, P1′, and P2′. As shown in sub-figure (B), the decoded connectivity information indicates that a face is encoded as Where P1′ and P2′ are within the plane. Figure 13 As shown in the subgraph (C), the nearest edge of P1′ is marked as The nearest edge of P2′ is marked as Between P1′ and P2′, P1′ is farther away from the nearest edge, so refinement is performed on P1′ first. As shown in sub-graph (D), the projected vertex P1″ of P1′ is the ray and the edge Therefore, the triangle Divided into two triangles and Each triangle and triangle have the same counterclockwise direction.

[0090] If any triangle contains a point inside, the triangle will be further divided. In this example, the triangle There is P2′ inside, so it will be further analyzed for segmentation. As shown in sub-graph (E), the projection vertex P2″ of P2′ is the ray and the nearest edge of P′2 The intersection of triangles. It can be further divided into triangles and triangle Both have the same Sub-figure (F) shows the reconstructed topology formed by the decoded geometry information and the refined connectivity information, where there are three reconstructed triangles instead of the simplified one triangle before refinement.

[0091] Now refer to Figure 14 , Figure 14 An example of a mesh coding process 1400 for connectivity simplification according to some embodiments of the present disclosure is described. One or more computing devices implement the process by executing appropriate program code. Figure 14 For example, Figure 8 The encoder system 800 can be implemented by executing the corresponding program code Figure 14 For illustrative purposes, process 1400 is described with reference to some examples depicted in the accompanying drawings. However, other implementations are possible.

[0092] At block 1402, process 1400 includes accessing a dynamic mesh to be encoded. As described above, a dynamic mesh can be represented as an uncompressed mesh frame sequence comprising mesh frames. A mesh frame is a data format that describes 3D content (e.g., a 3D object) in a digital representation as a collection of geometric information, connectivity information, attribute information, and attribute mapping information. Each mesh frame is characterized by a presentation time and a duration. A mesh frame sequence (e.g., a sequence of multiple mesh frames) forms a dynamic mesh video. The uncompressed mesh frame sequence can be segmented into segmented mesh data. Based on the segmented mesh data, the encoder system 800 can generate attribute component images, geometric component images, connectivity component images, and mapping component images.

[0093] At block 1404, process 1400 includes encoding the geometry component image using a video encoder to generate a geometry component code stream, as described with respect to FIG. Figure 1 and Figure 8 Detailed description. At block 1406, process 1400 includes decoding the encoded geometry component image to generate a reconstructed geometry component image. Decoding is performed using a video decoder corresponding to the video encoder used to encode the geometry component image. As described above, in an example where the video encoder (e.g., video encoder 120b) is a lossy encoder, the reconstructed geometry component image will differ from the geometry component image due to prediction and quantization. Therefore, the reconstructed positions of the decoded mesh vertices may differ from the original positions of the mesh vertices. As a result, adjacent faces may degenerate into simplified geometric shapes.

[0094] At block 1408, process 1400 includes determining vertices in the reconstructed geometric component image that belong to faces to be removed from the connectivity component image of the dynamic mesh. To determine the faces to be removed, four vertices of a pair of adjacent faces or a pair of triangles of the mesh can be analyzed. In some examples, two adjacent triangles have the same orientation. Three of the four vertices can be used to form a triangle, with the fourth vertex within the triangle. The distance from the fourth vertex to each of the three sides of the triangle can be determined. The minimum of the three distances is determined. If the minimum distance is below a predetermined threshold, the two adjacent faces can be selected as faces to be removed and replaced with a single face. The generated triangle can be used as the triangle to be encoded to replace the two adjacent triangles (i.e., the two adjacent triangles are merged into the generated triangle). This step can be repeated to identify multiple vertices to be removed and multiple pairs of adjacent triangles to be merged.

[0095] At block 1410, process 1400 includes updating the connectivity component image of the dynamic mesh to simplify the connectivity information. For example, the simplification may include removing the faces identified in block 1408 from the connectivity component image and updating the connectivity component image to replace the data of adjacent triangle pairs with corresponding merged triangles. At block 1412, process 1400 includes encoding the updated connectivity component image to generate a connectivity component code stream. As described above with respect to Figure 1 and Figure 8 As discussed in detail, encoding can include generating a connectivity component bitstream using a video encoder. At block 1414, process 1400 includes generating a coded mesh bitstream by including at least a geometry component bitstream and a connectivity component bitstream. For example, the coded mesh bitstream can be generated by multiplexing a mesh bitstream payload, the coded mesh bitstream including the geometry component bitstream and the connectivity component bitstream, and other bitstreams having a mesh bitstream header, such as an attribute component bitstream, a mapping component bitstream, and the like.

[0096] Now refer to Figure 15 , Figure 15 An example of a process 1500 for decoding a coded trellis code stream with connectivity refinement according to some embodiments of the present disclosure is described. One or more computing devices implement the process 1500 by executing appropriate program code. Figure 15 For example, Figure 11 The decoder system 1100 can be implemented by executing the corresponding program code Figure 11 For illustrative purposes, process 1500 is described with reference to some examples depicted in the accompanying drawings. However, other implementations are possible.

[0097] At block 1502, process 1500 includes accessing an encoded mesh code stream of a dynamic mesh for decoding. The encoded mesh code stream is encoded using the connectivity simplification described above. The encoded mesh code stream may include a geometry component code stream, an attribute component code stream, a connectivity component code stream, and a mapping component code stream.

[0098] At block 1504, process 1500 includes reconstructing geometric information of the dynamic mesh based on the geometric component code stream in the coded mesh code stream. The reconstructed geometric information includes data specifying the vertices of the dynamic mesh. Figure 2 and Figure 11 As discussed in detail, the geometric information of the dynamic mesh can be reconstructed based on the geometric component code stream by applying a video decoder to the geometric component code stream to generate a reconstructed geometric component image, which is then used to reconstruct the ordered list of vertex coordinates in the geometric information.

[0099] At block 1506, process 1500 includes reconstructing connectivity information for the dynamic mesh based on the connectivity component codestream in the encoded mesh codestream. The reconstructed connectivity information includes data specifying faces (e.g., triangles) of the dynamic mesh. Similar to the geometric information, the connectivity information for the dynamic mesh can be reconstructed based on the connectivity component codestream by applying a video decoder to the connectivity component codestream to generate a reconstructed connectivity component image. The generated reconstructed connectivity component image is then used to reconstruct an ordered list of face information with corresponding vertex indices and, in some cases, texture indices. Other components of the encoded mesh codestream, such as the attribute component codestream and the mapping component codestream, can also be decoded and used to reconstruct the corresponding information.

[0100] At block 1508, process 1500 includes refining the reconstructed connectivity information based on the reconstructed geometric information to generate refined connectivity information. As discussed in detail above, the refinement process divides a face specified by the reconstructed connectivity information into two faces based on a vertex specified in the reconstructed geometric information, rather than being listed as a vertex of any face specified by the reconstructed connectivity information. For example, a vertex located within a face of the reconstructed connectivity information can be identified, and its projected vertex on an edge of the face can be determined. The face can then be divided into two refined faces, each having the projected vertex as one of its vertices, and each having the same orientation as the originally reconstructed face.

[0101] At block 1510, process 1500 includes reconstructing a dynamic mesh based on the reconstructed geometry information and the refined connectivity information, as well as other information including attribute information, mapping information, etc. At block 1512, process 1500 includes rendering the reconstructed dynamic mesh for display. For example, the reconstructed dynamic mesh may be transmitted to a device or module configured to render a 3D object represented by the reconstructed dynamic mesh to generate a rendered image or video for display.

[0102] Figure 16 An example of an encoder system 1600 for trellis coding with connectivity simplification and attribute adjustment and mapping adjustment is shown according to various embodiments of the present disclosure. Figure 16 The encoder system 1600 shown includes Figure 8 In addition, the encoder system 1600 includes a mapping topology update module 1602 and an attribute image update module 1604 to update the mapping topology and the attribute image, respectively, and also includes connectivity simplification to reduce distortion caused by reconstruction errors of geometric information. Figure 10As shown, due to lossy compression of geometric information, reconstructed vertices may have coordinates that differ from their original coordinates. In this example, vertex P becomes P' after encoding and reconstruction. To reduce visual artifacts caused by such reconstruction errors, attribute information and mapping information should be adjusted accordingly. For example, attribute image update module 1604 can update the attribute component image to reflect the distortion caused by geometric encoding and reconstruction. To this end, attribute image update module 1604 can employ an attribute reprojection process.

[0103] The distortion associated with the attribute image introduced by geometric compression errors can be described as a transformation such as an affine transformation. This transformation is applied to points belonging to an edge between two adjacent triangles that are degenerated and merged during connectivity simplification, e.g. Figure 10 Therefore, the reprojection process can be achieved by applying the same transformation to the corresponding attribute samples of each degenerate triangular face in the attribute image. The boundary of the triangular face is defined by the vertex mapping coordinates (U, V) and connectivity information (f_idx_1, f_idx_2, f_idx_3), as shown in Figure 5 As shown. The transformation T() can be represented by a 4×4 matrix M, where

[0104]

[0105] The transformation T() can be an affine transformation. The affine transformation T() transforms a point (or vector) p into a point (or vector) p'. The transformation T() from point p to point p' is obtained by performing a matrix-vector multiplication Mp:

[0106] p′=Mp (2)

[0107]

[0108] An example of a matrix M implementing a shear transformation T() is

[0109]

[0110] For example, the shear transformation parameter S h It can be expressed from Figure 10 The point P' and the vector of point P are shown.

[0111] Another example of a rotation transformation T() implemented by the matrix M is

[0112]

[0113] Figure 17Examples of affine transformations are shown. Subfigure (A) shows an example of a shearing transformation in a two-dimensional case, and subfigure (B) shows an example of a rotation transformation in a two-dimensional case. Attribute image update module 1604 can select the format of the affine transformation (e.g., shearing transformation, rotation transformation, or other transformation) and determine the parameters of the transformation and the connectivity simplification based on the distortion associated with the lossy compression applied to the geometric components.

[0114] Alternatively or additionally, an interpolation transform filter, such as a bilinear interpolation transform filter, a Lanczos interpolation transform filter, can be used to reproject the surfaces of the attribute image corresponding to the degraded surfaces onto a simplified topology. The parameters of the filter, such as the coefficients, position and length of the filter, can be written into the encoded grid code stream or be a component of the encoder, such as a lookup table. For example, the parameters can be written using a supplemental enhancement information message or by indicating a filter index in a lookup table. The updated attribute component image can be provided to the video encoder 120a for encoding, as described above with respect to Figure 1 discussed.

[0115] Similarly, the mapping topology update module 1602 can update the mapping component image based on the connectivity simplification. The mapping topology update module 1602 can use the analysis results generated by the reconstruction geometry analysis module 804, which can include vertices selected for removal and, in some examples, triangles to be merged. The mapping component coordinates (U, V) can be updated based on the changes in the attribute component image. The mapping coordinates associated with vertex P in the original attribute image are updated to the transformed mapping coordinates associated with vertex P'.

[0116] The updated mapped component images may be provided to the video encoder 120d for encoding, as described above with respect to Figure 1 The multiplexer 122 can then be used in a manner similar to that discussed with respect to Figure 1 The described approach multiplexes the encoded information to generate encoded grid frame sequence 1610. Due to the attribute updates and mapping updates performed in addition to the connectivity simplification, encoded grid frame sequence 1610 differs from encoded grid frame sequence 124 generated using encoder system 100 and encoded grid frame sequence 810 generated using encoder system 800.

[0117] Figure 18 An example decoder system 1800 for trellis decoding with connectivity refinement and map refinement is shown, according to various embodiments of the present disclosure. Figure 18 The decoder system 1800 shown includes Figure 11Furthermore, the decoder system 1800 includes a map refinement module 1802 to recover map information that was updated at the encoder before the map information is sent to the mesh reconstruction module 226 to reconstruct the mesh frame sequence 1804.

[0118] In some examples, the map refinement module 1802 can adjust the position of vertex P' in the attribute component image according to equation (1). For example, the map refinement module 1802 can estimate the transformation defined in equation (1), where P is the reconstructed point position and P' is the projected point position. Based on the estimated transformation, the transformation defined in equation (6) is applied to the U coordinate and V coordinate of the reconstructed map component image. The refined map information can be used to reconstruct the mesh frame together with other decoded components.

[0119]

[0120] Now refer to Figure 19 , Figure 19 An example of a process 1900 for performing mesh coding for connectivity simplification and attribute and mapping adjustments according to some embodiments of the present disclosure is described. One or more computing devices implement the process by executing appropriate program code. Figure 19 For example, Figure 16 The encoder system 1600 can be implemented by executing the corresponding program code Figure 19 For illustrative purposes, process 1900 is described with reference to some examples depicted in the accompanying drawings. However, other implementations are possible.

[0121] At block 1902, process 1900 includes accessing a dynamic mesh to be encoded. As described above, the dynamic mesh can be represented as an uncompressed mesh frame sequence comprising mesh frames. A mesh frame is a data format that describes 3D content (e.g., a 3D object) in a digital representation as a collection of geometric information, connectivity information, attribute information, and attribute mapping information. Each mesh frame is characterized by a presentation time and a duration. A mesh frame sequence (e.g., a sequence of multiple mesh frames) forms a dynamic mesh video. The uncompressed mesh frame sequence can be segmented into segmented mesh data. Based on the segmented mesh data, encoder system 1600 can generate attribute component images, geometric component images, connectivity component images, and mapping component images.

[0122] At block 1904, process 1900 includes encoding the geometry component image using a video encoder to generate a geometry component code stream, as described with respect to FIG. Figure 1Detailed description. At block 1906, process 1900 includes decoding the encoded geometry component image to generate a reconstructed geometry component image. The decoding is performed using a video decoder corresponding to the video encoder used to encode the geometry component image. As described above, in an example where the video encoder (e.g., video encoder 120b) is a lossy encoder, the reconstructed geometry component image will be different from the geometry component image due to prediction and quantization. Therefore, the reconstructed positions of the decoded mesh vertices may be different from the original positions of the mesh vertices. As a result, adjacent faces may degenerate into a simplified topological structure.

[0123] At block 1908, process 1900 includes determining faces containing vertices in the reconstructed geometry component image to be removed from the connectivity component image of the dynamic mesh. At block 1910, process 1900 includes updating the connectivity component image of the dynamic mesh to simplify the connectivity information. Blocks 1908 and 1910 are similar to those described in reference to FIG. Figure 14 Blocks 1408 and 1410 are described.

[0124] At block 1912, process 1900 includes updating the attribute component image and the mapping component image of the dynamic mesh. Figure 16 As discussed in detail, the encoder system 1600 can estimate the transformation applied to the vertices in the degraded geometric information through lossy compression and apply the same transformation to the corresponding faces in the attribute image. The mapping component coordinates (U, V) can be updated based on the changes in the attribute component image. The mapping coordinates associated with vertex P in the original attribute image are updated to the transformed mapping coordinates associated with vertex P'.

[0125] At block 1914, process 1900 includes encoding the updated connectivity component image to generate a connectivity component code stream, encoding the updated attribute component image to generate an attribute component code stream, and encoding the updated mapping component image to generate a mapping component code stream. Figure 1 As discussed in detail, encoding can include using a video encoder to generate corresponding component streams. At block 1916, process 1900 includes generating a coded trellis stream comprising at least a geometry component stream, a connectivity component stream, an attribute component stream, and a mapping component stream. For example, the coded trellis stream can be generated by multiplexing a trellis stream payload, the coded trellis stream comprising the coded component streams with a trellis stream header.

[0126] Now refer to Figure 20 , Figure 20 An example of a process 2000 for decoding a coded grid code stream with connectivity refinement and mapping refinement according to some embodiments of the present disclosure is described. One or more computing devices implement the process by executing appropriate program code. Figure 20For example, Figure 18 The decoder system 1800 can be implemented by executing the corresponding program code Figure 20 For illustrative purposes, process 2000 is described with reference to some examples depicted in the accompanying drawings. However, other implementations are possible.

[0127] At block 2002, process 2000 includes accessing a coded grid code stream of a dynamic grid for decoding. Figure 19 The described connectivity simplification, attribute update and mapping update are used to encode the coded mesh code stream. The coded mesh code stream may include a geometry component code stream, an attribute component code stream, a connectivity component code stream and a mapping component code stream.

[0128] At block 2004, process 2000 includes reconstructing geometric information of the dynamic mesh based on the geometric component code stream in the encoded mesh code stream. The reconstructed geometric information includes data specifying the vertices of the dynamic mesh. Figure 2 As discussed in detail, the geometric information of the dynamic mesh can be reconstructed based on the geometric component code stream by applying a video decoder to the geometric component code stream to generate a reconstructed geometric component image, which is then used to reconstruct the ordered list of vertex coordinates in the geometric information.

[0129] At block 2006, process 2000 includes reconstructing connectivity information of the dynamic mesh based on the connectivity component code stream in the coded mesh code stream. At block 2008, process 2000 includes refining the reconstructed connectivity information based on the reconstructed geometric information to generate refined connectivity information. Blocks 2006 and 2008 are similar to those described in reference to FIG. Figure 15 Blocks 1506 and 1508 are described.

[0130] At block 2010, process 2000 includes reconstructing mapping information based on the mapped component code streams and refining the reconstructed mapping information. Figure 2 As discussed in detail, the mapping information of the dynamic mesh can be reconstructed based on the mapping component code stream by applying a video decoder to the mapping component code stream to generate a reconstructed mapping component image, and the reconstructed geometric component image is then used to reconstruct an ordered list of projected vertex attribute coordinate information. According to equation (1), the reconstructed mapping information can be refined by the mapping refinement module 1802 by adjusting the position of the vertex P' in the attribute component image. For example, the mapping refinement module 1802 can update the mapping component image to reflect the distortion caused by the topological structure simplification. The mapping refinement module 1802 can estimate the transformation defined in equation (1), where P is the reconstructed point position and P' is the projected point position. Based on the estimated transformation, the transformation defined in equation (6) is applied to the U coordinate and V coordinate of the reconstructed mapping component image.

[0131] At block 2012, process 2000 includes reconstructing a dynamic mesh based on the reconstructed geometry information, the refined connectivity information, the refined mapping information, and other information including decoded attribute information. At block 2014, process 2000 includes rendering the reconstructed dynamic mesh for display. For example, the reconstructed dynamic mesh may be transmitted to a device or module configured to render a 3D object represented by the reconstructed dynamic mesh to generate a rendered image or video for display.

[0132] Figure 21 An example of an encoder system 2100 for trellis coding with connectivity simplification and attribute adjustment and mapping adjustment is shown according to various embodiments of the present disclosure. Figure 21 The encoder system 2100 shown includes Figure 16 The components or modules in are similar to various components or modules. Figure 16 In contrast, the encoder system 2100 does not include the attribute image update module 1604 for updating the attribute image. In other words, the attribute component image is encoded by the video encoder 120a. The multiplexer 122 can be similar to the one described with respect to Figure 1 The described manner multiplexes the encoded information to generate the encoded grid frame sequence 2110. Since no attribute updates are performed at the encoder along with the connectivity simplification and mapping updates, the encoded grid frame sequence 2110 differs from the encoded grid frame sequence 124 generated using the encoder system 100, the encoded grid frame sequence 810 generated using the encoder system 800, and the encoded grid frame sequence 1610 generated using the encoder system 1600. Figure 16 Not updating the attribute images on the encoder side allows the encoder system 2100 to generate the encoded grid frame sequence 2110 more quickly than the illustrated encoder system 1600 .

[0133] Figure 22 An example decoder system 2200 is shown for lattice decoding with connectivity refinement, map refinement, and attribute refinement according to various embodiments of the present disclosure. Figure 22 The decoder system 2200 shown includes Figure 18 In addition, the decoder system 2200 includes an attribute refinement module 2202 that refines and transforms the decoded attribute image to reconstruct a grid frame sequence 2204 based on the distortion of the geometric information before sending the attribute image to the grid reconstruction module 226.

[0134] In some examples, the attribute refinement module 2202 estimates the transformation to be applied to each degenerate triangular face in a manner similar to the estimation of the transformation performed by the attribute image update module 1604. For example, the attribute refinement module 2202 can estimate the affine transformation T() according to equations (1) to (5), where P is the original reconstructed point position and P' is the projected point position. The inverse of the estimated affine transformation can be applied to the decoded attribute image. For example, if the estimated transformation is a shear transformation with parameters S h If the shearing transform is , the inverse shearing transform can be applied to the decoded attribute image according to the following formula:

[0135]

[0136] Here, [x'y'z'] represents the reconstructed mapping component coordinates associated with the reconstructed vertex P' of the decoded attribute image, and [x"y"z"] represents the refined mapping component coordinates associated with the projected vertex P" of the refined attribute image.

[0137] In addition to the attribute refinement filter in equation (6), one-dimensional or two-dimensional interpolation transform filters such as bilinear interpolation transform filters and lanczos interpolation transform filters can also be used to refine the surfaces of the attribute component image corresponding to the degenerate surface on the simplified topology. The parameters of the filter, such as the coefficients, position and size of the filter, can be written into the coded grid code stream. For example, the parameters can be written using a supplementary enhancement information message or by indicating the filter index in the lookup table. The refined attribute component image can be used to refine the image as described above. Figure 2 The other decoding components discussed together reconstruct the trellis frame.

[0138] Now refer to Figure 23 , Figure 23 An example of a process 2300 for mesh coding with connectivity simplification and mapping adjustment but without attribute adjustment according to some embodiments of the present disclosure is described. One or more computing devices implement the process by executing appropriate program code. Figure 23 For example, Figure 21 The encoder system 2100 can be implemented by executing the corresponding program code Figure 23 For illustrative purposes, process 2300 is described with reference to some examples depicted in the accompanying drawings. However, other implementations are possible.

[0139] At block 2302, process 2300 includes accessing a dynamic mesh to be encoded. As described above, the dynamic mesh can be represented as an uncompressed mesh frame sequence comprising mesh frames. A mesh frame is a data format that describes 3D content (e.g., a 3D object) in a digital representation as a collection of geometric information, connectivity information, attribute information, and attribute mapping information. Each mesh frame is characterized by a presentation time and a duration. A mesh frame sequence (e.g., a sequence of multiple mesh frames) forms a dynamic mesh video. The uncompressed mesh frame sequence can be segmented into segmented mesh data. Based on the segmented mesh data, the encoder system 2100 can generate attribute component images, geometric component images, connectivity component images, and mapping component images.

[0140] At block 2304, process 2300 includes encoding the geometry component image using a video encoder to generate a geometry component code stream, as described with respect to FIG. Figure 1 Detailed description. At block 2306, process 2300 includes decoding the encoded geometry component image to generate a reconstructed geometry component image. Decoding is performed using a video decoder corresponding to the video encoder used to encode the geometry component image. As described above, in an example where the video encoder (e.g., video encoder 120b) is a lossy encoder, the reconstructed geometry component image will differ from the geometry component image due to prediction and quantization. Therefore, the reconstructed positions of the decoded mesh vertices may differ from the original positions of the mesh vertices. As a result, adjacent faces may degenerate into simplified geometric shapes.

[0141] At block 2308, process 2300 includes determining faces containing vertices in the reconstructed geometry component image to be removed from the connectivity component image of the dynamic mesh. At block 2310, process 2300 includes updating the connectivity component image of the dynamic mesh to simplify the connectivity information. Blocks 2308 and 2310 are similar to those described in reference to FIG. Figure 14 Blocks 1408 and 1410 are described.

[0142] At block 2312, process 2300 includes updating the mapped component images of the dynamic mesh. Figure 21 As discussed in detail, the encoder system 2100 can update the mapped component images by adjusting the mapped coordinates associated with vertices used to simplify connectivity.

[0143] At block 2314, process 2300 includes encoding the updated connectivity component image to generate a connectivity component code stream, and encoding the updated mapping component image to generate a mapping component code stream. Figure 1As discussed in detail, encoding can include using a video encoder to generate corresponding component streams. At block 2316, process 2300 includes generating a coded trellis stream comprising at least a geometry component stream, a connectivity component stream, and a mapping component stream. For example, the coded trellis stream can be generated by multiplexing a trellis stream payload, the coded trellis stream comprising the coded component streams with a trellis stream header.

[0144] Now refer to Figure 24 , Figure 24 An example of a process 2400 for decoding a coded mesh code stream subjected to connectivity refinement, map refinement, and attribute refinement according to some embodiments of the present disclosure is described. One or more computing devices implement the process 2400 by executing appropriate program code. Figure 24 For example, Figure 22 The decoder system 2200 in the embodiment can be implemented by executing the corresponding program code Figure 24 For illustrative purposes, process 2400 is described with reference to some examples depicted in the accompanying drawings. However, other implementations are possible.

[0145] At block 2402, process 2400 includes accessing an encoded grid code stream of a dynamic grid for decoding. Figure 23 The described connectivity simplification and mapping update are used to encode the coded mesh code stream without using attribute update. The coded mesh code stream may include a geometry component code stream, an attribute component code stream, a connectivity component code stream, and a mapping component code stream.

[0146] At block 2404, process 2400 includes reconstructing geometric information of the dynamic mesh based on the geometric component code stream in the coded mesh code stream. The reconstructed geometric information includes data specifying the vertices of the dynamic mesh. Figure 2 As discussed in detail, the geometric information of the dynamic mesh can be reconstructed based on the geometric component code stream by applying a video decoder to the geometric component code stream to generate a reconstructed geometric component image, which is then used to reconstruct the ordered list of vertex coordinates in the geometric information.

[0147] At block 2406, process 2400 includes reconstructing connectivity information of the dynamic mesh based on the connectivity component code stream in the coded mesh code stream. At block 2408, process 2400 includes refining the reconstructed connectivity information based on the reconstructed geometric information to generate refined connectivity information. Blocks 2406 and 2408 are similar to those described in reference to FIG. Figure 15 Blocks 1506 and 1508 are described.

[0148] At block 2410, process 2400 includes reconstructing an attribute image based on the attribute component code stream and refining the reconstructed attribute image. Figure 2 As discussed in detail, a video decoder can be applied to the attribute component bitstream to reconstruct the attribute component image of the dynamic mesh from the attribute component bitstream to generate a reconstructed attribute image. The reconstructed attribute image can be refined by applying an inverse transform to the faces corresponding to the degenerate faces in the connectivity information. The transformation can be estimated based on the decoded vertices in the geometry information and the reconstructed projected vertices in the geometry information.

[0149] At block 2412, process 2400 includes reconstructing a dynamic mesh based on the reconstructed geometric information, the refined connectivity information, the refined attribute image, and other information including the decoded attribute information. At block 2414, process 2400 includes rendering the reconstructed dynamic mesh for display. For example, the reconstructed dynamic mesh may be transmitted to a device or module configured to render the 3D object represented by the reconstructed dynamic mesh to generate a rendered image or video for display.

[0150] Computing System Example

[0151] Any suitable computing system may be used to perform the operations described herein. For example, Figure 25 An example of a computing device 2500 is depicted that may implement the following trellis encoder and trellis decoder: Figure 1 Grid encoder 100, Figure 2 Grid decoder 200, Figure 8 Grid encoder 800, Figure 11 Grid decoder 1100, Figure 16 Grid encoder 1600, Figure 18 Grid decoder 1800, Figure 21 Grid encoder 2100, Figure 22 trellis decoder 2200. In some embodiments, the computing device 2500 may include a processor 2512 that is communicatively coupled to the memory 2514 and executes computer-executable program code and / or accesses information stored in the memory 2514. The processor 2512 may include a microprocessor, an application-specific integrated circuit (ASIC), a state machine, or other processing device. The processor 2512 may include any of a plurality of processing devices, including one. Such a processor may include, or may communicate with, a computer-readable medium storing instructions that, when executed by the processor 2512, causes the processor to perform the operations described herein.

[0152] Memory 2514 may include any suitable non-transitory computer-readable medium. Computer-readable media may include any electronic, optical, magnetic, or other storage device capable of providing computer-readable instructions or other program code to the processor. Non-limiting examples of computer-readable media include disks, memory chips, ROM, RAM, ASICs, configured processors, optical storage, magnetic tape or other magnetic storage, or any other medium from which a computer processor can read instructions. Instructions may include processor-specific instructions generated by a compiler and / or interpreter from code written in any suitable computer programming language, including, for example, C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, and ActionScript.

[0153] The computing device 2500 may also include a bus 2516. The bus 2516 may be communicatively coupled to one or more components of the computing device 2500. The computing device 2500 may also include a plurality of external or internal devices, such as input devices or output devices. For example, the computing device 2500 is shown as having an input / output ("I / O") interface 2518, which may receive input from one or more input devices 2520 or provide output to one or more output devices 2522. One or more input devices 2520 and one or more output devices 2522 may be communicatively coupled to the I / O interface 2518. The communicative coupling may be achieved in any suitable manner (e.g., via a connection on a printed circuit board, via a connection via a cable, via wireless transmission, etc.). Non-limiting examples of the input device 2520 include a touch screen (e.g., one or more cameras for imaging a touch area or a pressure sensor for detecting pressure changes caused by a touch), a mouse, a keyboard, or any other device that may be used to generate input events in response to physical actions of a user of the computing device. Non-limiting examples of output device 2522 include an LCD screen, an external monitor, speakers, or any other device that can be used to display or otherwise present output generated by the computing device.

[0154] The computing device 2500 may execute the configuration of the processor 2512 to execute the above Figures 1 to 24 The program code may include one or more operations described. Figure 1 Grid encoder 100, Figure 2 Grid decoder 200, Figure 8 Grid encoder 800, Figure 11 Grid decoder 1100, Figure 16 Grid encoder 1600, Figure 18Grid decoder 1800, Figure 21 Grid encoder 2100, Figure 22 The trellis decoder 2200. The program code may reside in the memory 2514 or any suitable computer readable medium and may be executed by the processor 2512 or any other suitable processor.

[0155] The computing device 2500 may also include at least one network interface device 2524. The network interface device 2524 may include any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks 2528. Non-limiting examples of the network interface device 2524 include an Ethernet network adapter, a modem, etc. The computing device 2500 may send messages via the network interface device 2524 in the form of electrical or optical signals.

[0156] General considerations

[0157] Numerous specific details are set forth herein to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill in the art have not been described in detail in order to avoid obscuring the claimed subject matter.

[0158] Unless otherwise expressly stated, it should be understood that throughout the discussion of this specification, terms such as "processing," "computing," "calculating," "determining," and "identifying" are used to refer to the actions or processes of a computing device, such as one or more computers or one or more similar electronic computing devices, that operate on or transform data represented as physical electronic or magnetic quantities in the memory, registers, or other information storage devices, transmission devices, or display devices of a computing platform.

[0159] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide a result conditioned on one or more inputs. Suitable computing devices include general-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system, ranging from general-purpose computing devices to specialized computing devices that implement one or more embodiments of the present subject matter. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software used when programming or configuring a computing device.

[0160] Embodiments of the methods disclosed herein may be performed in the operation of such a computing device. The order of the blocks presented in the above examples may be different—for example, the blocks may be reordered, combined, and / or decomposed into sub-blocks. Some blocks or processes may be performed in parallel.

[0161] The use of "adapted to" or "configured to" herein is intended to be open and inclusive, and does not exclude devices adapted or configured to perform additional tasks or steps. Furthermore, the use of "based on" is intended to be open and inclusive, as a process, step, calculation, or other action that is "based on" one or more stated conditions or values ​​may in fact be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for ease of explanation only and are not intended to be limiting.

[0162] Although the subject matter has been described in detail with respect to its specific embodiments, it should be understood that those skilled in the art, after understanding the foregoing, can easily make modifications, changes and equivalents to these embodiments. Therefore, it should be understood that the present disclosure is presented for the purpose of illustration and not limitation, and does not exclude the inclusion of modifications, changes and / or additions to the subject matter that are obvious to those of ordinary skill in the art.

Claims

1. A computer-implemented method for decoding an encoded mesh bitstream representing a dynamic mesh of three-dimensional (3D) content, the method comprising: reconstructing geometric information of the dynamic mesh according to a geometric component code stream in the encoded mesh code stream, wherein the reconstructed geometric information includes data specifying a plurality of vertices of the dynamic mesh; reconstructing connectivity information of the dynamic mesh according to the connectivity component code stream in the encoded mesh code stream, wherein the reconstructed connectivity information includes data specifying a plurality of faces of the dynamic mesh; Refining the reconstructed connectivity information based on the reconstructed geometric information to generate refined connectivity information by dividing one of the plurality of faces specified by the reconstructed connectivity information into at least two faces based on one of the plurality of vertices specified in the reconstructed geometric information; reconstructing the dynamic mesh based on the reconstructed geometric information and the refined connectivity information; as well as The reconstructed dynamic mesh is caused to be rendered for display.

2. The computer-implemented method of claim 1 , wherein: The refining of the reconstructed connectivity information based on the reconstructed geometric information includes: identifying a vertex among the plurality of vertices that is within a face among the plurality of faces; Determining a projection vertex of the one vertex on the edge of the one face; and The one surface is divided into two refined surfaces, each refined surface having the projected vertex as a vertex.

3. The computer-implemented method of claim 2, wherein: Each of the two refinement faces has the same direction as the one face.

4. The computer-implemented method of claim 2, wherein: The determining of the projection vertex of the one vertex includes: determining a nearest edge for the one vertex among the multiple edges of the one face, wherein a distance between the one vertex and the nearest edge is smaller than another distance between the one vertex and another edge among the multiple edges of the one face; and The projected vertex is determined as an intersection of a line with the nearest edge, the line passing through the one vertex and an opposite vertex of the one face that does not belong to the nearest edge.

5. The computer-implemented method of claim 1 , wherein: The refining of the reconstructed connectivity information based on the reconstructed geometric information includes: identifying two or more vertices of the plurality of vertices that are within one of the plurality of faces; selecting a vertex for processing among the two or more vertices based on a distance between each of the two or more vertices and a corresponding nearest edge; Determining a projected vertex of the selected vertex on an edge of the one face; and The one surface is divided into two refined surfaces, and the projected vertex is used as a vertex of each refined surface.

6. The computer-implemented method of claim 5, wherein: The selecting a vertex for processing from the two or more vertices comprises: For each of the two or more vertices, determining a nearest edge among the multiple edges of the one face, wherein a distance between the vertex and the nearest edge is less than another distance between the vertex and another edge among the multiple edges of the one face; and The vertex for processing is selected as a vertex among the two or more vertices having the largest distance to the corresponding nearest edge.

7. The computer-implemented method of claim 1 , wherein: The reconstructing the geometric information of the dynamic mesh according to the geometric component code stream includes: decoding the geometric component code stream through a video decoder.

8. A non-transitory computer-readable medium having program code stored thereon, the program code being executable by one or more processing devices to perform operations comprising: Reconstructing geometric information of the dynamic mesh according to a geometric component code stream in an encoded mesh code stream of the dynamic mesh, wherein the reconstructed geometric information includes data specifying a plurality of vertices of the dynamic mesh; reconstructing connectivity information of the dynamic mesh according to the connectivity component code stream in the encoded mesh code stream, wherein the reconstructed connectivity information includes data specifying a plurality of faces of the dynamic mesh; Refining the reconstructed connectivity information based on the reconstructed geometric information to generate refined connectivity information by dividing one of the plurality of faces specified by the reconstructed connectivity information into at least two faces based on one of the plurality of vertices specified in the reconstructed geometric information; reconstructing the dynamic mesh based on the reconstructed geometric information and the refined connectivity information; as well as The reconstructed dynamic mesh is caused to be rendered for display.

9. The non-transitory computer-readable medium of claim 8, wherein: The refining of the reconstructed connectivity information based on the reconstructed geometric information includes: identifying a vertex among the plurality of vertices that is within a face among the plurality of faces; Determining a projection vertex of the one vertex on the edge of the one face; and The one surface is divided into two refined surfaces, and the projected vertex is used as a vertex of each refined surface.

10. The non-transitory computer-readable medium of claim 9, wherein: Each of the two refinement faces has the same direction as the one face.

11. The non-transitory computer-readable medium of claim 9, wherein: The determining of the projection vertex of the one vertex includes: determining a nearest edge for the one vertex among the multiple edges of the one face, wherein a distance between the one vertex and the nearest edge is smaller than another distance between the one vertex and another edge among the multiple edges of the one face; and The projected vertex is determined as an intersection of a line with the nearest edge, the line passing through the one vertex and an opposite vertex of the one face that does not belong to the nearest edge.

12. The non-transitory computer-readable medium of claim 8, wherein: The refining of the reconstructed connectivity information based on the reconstructed geometric information includes: identifying two or more vertices of the plurality of vertices that are within one of the plurality of faces; selecting a vertex for processing among the two or more vertices based on a distance between each of the two or more vertices and a corresponding nearest edge; Determining a projected vertex of the selected vertex on an edge of the one face; and The one surface is divided into two refined surfaces, and the projected vertex is used as a vertex of each refined surface.

13. The non-transitory computer-readable medium of claim 12, wherein: The selecting a vertex for processing from the two or more vertices comprises: For each of the two or more vertices, determining a nearest edge among the multiple edges of the one face, wherein a distance between the vertex and the nearest edge is less than another distance between the vertex and another edge among the multiple edges of the one face; and The vertex for processing is selected as a vertex among the two or more vertices having the largest distance to the corresponding nearest edge.

14. The non-transitory computer-readable medium of claim 8, wherein: The reconstructing the geometric information of the dynamic mesh according to the geometric component code stream includes: decoding the geometric component code stream through a video decoder.

15. A system comprising: processing equipment, and A non-transitory computer-readable medium communicatively coupled to the processing device, wherein the processing device is configured to execute program code stored in the non-transitory computer-readable medium to perform operations comprising: Reconstructing geometric information of the dynamic mesh according to a geometric component code stream in an encoded mesh code stream of the dynamic mesh, wherein the reconstructed geometric information includes data specifying a plurality of vertices of the dynamic mesh; reconstructing connectivity information of the dynamic mesh according to the connectivity component code stream in the encoded mesh code stream, wherein the reconstructed connectivity information includes data specifying a plurality of faces of the dynamic mesh; Refining the reconstructed connectivity information based on the reconstructed geometric information to generate refined connectivity information by dividing one of the plurality of faces specified by the reconstructed connectivity information into at least two faces based on one of the plurality of vertices specified in the reconstructed geometric information; reconstructing the dynamic mesh based on the reconstructed geometric information and the refined connectivity information; and The reconstructed dynamic mesh is caused to be rendered for display.

16. The system according to claim 15, wherein: The refining of the reconstructed connectivity information based on the reconstructed geometric information includes: identifying a vertex among the plurality of vertices that is within a face among the plurality of faces; Determining a projection vertex of the one vertex on the edge of the one face; and The one surface is divided into two refined surfaces, and the projected vertex is used as a vertex of each refined surface.

17. The system according to claim 16, wherein: Each of the two refinement faces has the same direction as the one face.

18. The system according to claim 16, wherein: The determining of the projection vertex of the one vertex includes: determining a nearest edge for the one vertex among the multiple edges of the one face, wherein a distance between the one vertex and the nearest edge is smaller than another distance between the one vertex and another edge among the multiple edges of the one face; and The projected vertex is determined as an intersection of a line with the nearest edge, the line passing through the one vertex and an opposite vertex of the one face that does not belong to the nearest edge.

19. The system of claim 15, wherein: The refining of the reconstructed connectivity information based on the reconstructed geometric information includes: identifying two or more vertices of the plurality of vertices that are within one of the plurality of faces; selecting a vertex for processing among the two or more vertices based on a distance between each of the two or more vertices and a corresponding nearest edge; Determining a projected vertex of the selected vertex on an edge of the one face; and The one surface is divided into two refined surfaces, and the projected vertex is used as a vertex of each refined surface.

20. The system of claim 19, wherein: The selecting a vertex for processing from the two or more vertices comprises: For each vertex of the two or more vertices, determining a nearest edge among the multiple edges of the one face, wherein a distance between the vertex and the nearest edge is less than another distance between the vertex and another edge of the multiple edges of the one face; and The vertex for processing is selected as a vertex among the two or more vertices having the largest distance to the corresponding nearest edge.

Images