Inter prediction for dynamic grid coding

Abstract

A system includes an encoder configured to compress and encode data of a three-dimensional grid. To compress the three-dimensional mesh, the encoder predicts, for a current frame of the three-dimensional mesh, a vertex value of the current frame using location information from one or more previous frames or using a plurality of vertex values from a single frame. Predictors and residuals used to determine the current frame may be signaled in a bitstream to a decoder to decompress the three-dimensional mesh.

Description

BACKGROUND

[0001] CLAIM OF PRIORITY

[0002] This application claims priority benefit of U.S. Provisional Application Serial No. 63 / 636,583, entitled “Multi-Hypothesis Inter-Prediction for Dynamic Mesh Coding,” filed April 19, 2024, and claims priority benefit of U.S. Provisional Application Serial No. 63 / 636,589, entitled “Parameter Signaling for Attribute and Geometry Encoding Parameters for Dynamic Meshes,” filed April 19, 2024, both of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0003] The present disclosure generally relates to compression and decompression of three-dimensional meshes with associated textures or attributes.

[0004] RELATED TECHNOLOGY

[0005] Various types of sensors, such as light detection and ranging (LIDAR) systems, 3D cameras, 3D scanners, and the like, can capture data indicative of the position of points in three-dimensional space (e.g., position in the X, Y, and Z planes). Additionally, such systems can capture attribute information in addition to spatial information about respective points, such as color information (e.g., RGB values), texture attributes, intensity attributes, reflectivity attributes, motion-related attributes, modal attributes, or various other attributes. In some cases, additional attributes can be assigned to respective points, such as a timestamp of when the point was captured. Points captured by such sensors can constitute a “point cloud” comprising a set of points each having associated spatial information and one or more associated attributes. In some cases, a point cloud can include thousands of points, hundreds of thousands of points, millions of points, or even more. Additionally, in some cases, a point cloud can be generated in software, e.g., as opposed to being captured by one or more sensors. In either case, such point clouds can include a large amount of data, and storing and transmitting these point clouds can be costly and time-consuming. Moreover, three-dimensional visual content can also be captured in other ways, such as via 2D images of a scene captured from multiple viewing positions relative to the scene.

[0006] Such three-dimensional visual content can be characterized by a three-dimensional mesh that includes a plurality of polygons having connected vertices that model a surface of the three-dimensional visual content, such as a surface of a point cloud. Further, when modeled as a three-dimensional mesh, texture or attribute values of points of the three-dimensional visual content can be overlaid on the mesh to characterize attributes or textures of the three-dimensional visual content.

[0007] Additionally, a three-dimensional mesh can be generated, e.g., in software, without first being modeled as a point cloud or other type of three-dimensional visual content. For example, software can directly generate a three-dimensional mesh and apply texture or attribute values to characterize an object. SUMMARY

[0008] In some embodiments, a system includes one or more sensors configured to capture points that characterize an object in a view of the sensor and to capture texture or attribute values associated with the points of the object. The system further includes one or more computing devices that store program instructions that, when executed, cause the one or more computing devices to generate a three-dimensional mesh that models the points of the object using vertices of polygons that define the three-dimensional mesh and connections between the vertices. Additionally, in some embodiments, a three-dimensional mesh can be generated without first being captured by one or more sensors. For example, a computer graphics program can generate a three-dimensional mesh with associated textures or associated attribute values to characterize an object in a scene without having to generate a point cloud that characterizes the object.

[0009] In some embodiments, an encoder / decoder system includes one or more computing devices that store program instructions that, when executed by the one or more computing devices, further cause the one or more computing devices to compress / decompress versions of a three-dimensional mesh using inter-prediction. In inter-prediction, one or more frames of the three-dimensional mesh are encoded / decoded by predicting their content from previously decoded frames, where each frame represents the three-dimensional mesh at a particular frame index. Note that in some embodiments, inter-prediction can be performed only on a portion of the mesh, such as a sub-mesh, and the references used in inter-prediction to reference previously decoded frames can also only reference a portion of the mesh, such as a sub-mesh, in the previously decoded frames. Thus, in some embodiments, different prediction techniques can be used for different sub-meshes of the same overall mesh, and the references to previously decoded frames can also vary at the sub-mesh level. More generally, in inter-prediction, one or more vertices are used to predict a vertex. For example, vertices from one or more reference frames can be used to predict a vertex value of a current frame, and / or multiple different vertices in a given reference frame can be used to predict a vertex value of a current frame. In some embodiments, multi-hypothesis inter-prediction can be used, which uses more than one reference vertex’s position information to predict a vertex position of a vertex whose position is being predicted via the multi-hypothesis inter-prediction. For example, similarly located vertices in two or more previously decoded frames can be used to predict a vertex position of a vertex in another frame that is being decoded. In other words, a displacement vector to be applied at a tessellation location of a base mesh (e.g., which when applied results in vertex information that defines a vertex position) can be predicted using inter-prediction or multi-hypothesis inter-prediction. As another example, vertex information associated with two or more vertices in a single reference frame can be used to predict a vertex position of a vertex in another frame that is being decoded. Likewise, various combinations of vertices in multiple previously decoded frames can be used as predictors of a function of a vertex position of a vertex in a frame that is being decoded.

[0010] In some embodiments, a compressed bitstream can be used to signal vertex positions and other attributes of a three-dimensional mesh in a frame being decoded / reconstructed, which utilizes information provided in previously decoded frames that has been using inter-frame prediction. Inter-frame prediction allows for compression by utilizing temporal redundancies between previously decoded frames of a three-dimensional mesh and a frame that is currently being decoded. For example, instead of independently encoding each frame, inter-frame prediction can be used to predict the content of a given frame, such as a frame that is currently being decoded, based on previously encoded information associated with other frames that has been decoded at the decoder and is available for use by the decoder in performing inter-frame prediction with respect to the given frame. This reduces the amount of information that needs to be transmitted to the decoder by utilizing common relationships that exist across frames, e.g., one or more previously decoded frames that are used as references in decoding the frame that is currently being decoded. Additionally, the bitstream can signal differences (e.g., residuals) between predicted values, such as predicted vertex values, of a predicted frame and corresponding values, such as vertex values, of an original frame (e.g., a frame of a dynamic mesh that is being encoded / compressed), and in such cases, it is not necessary to signal an entire set of geometry and / or attribute information of a three-dimensional mesh of a given frame (e.g., a frame that is currently being decoded using inter-frame prediction) because the information for that frame is signaled in a manner that utilizes information from a reference frame.

[0011] In some embodiments, inter-frame prediction techniques used in compression of a three-dimensional mesh use multiple vertices from multiple frames at different frame indices, or multiple vertices from a single frame at a single frame index. In some embodiments, a single vertex can be selected from each of multiple frames at different frame indices, and multiple vertices are selected from another frame, to predict geometry information of a current frame. In some embodiments, a vertex of a given frame can be determined by averaging values of vertices having the same index position in multiple different reference frames. In another example, multiple vertices having different index positions in a single reference frame can be used to predict vertex values of vertices of a given frame, such as a frame that is currently being encoded / decoded. Inter-frame prediction techniques can also be used to encode vertex connectivity of vertices of a given frame, such as a frame that is currently being encoded / decoded, where the vertex connectivity is determined based on vertex connectivity used in one or more reference frames. In some embodiments, one or more predictors (e.g., input values to a prediction process, such as a combination of specified values or previously decoded data (e.g., vertex values)) can themselves be generated using other predictors that are used as predictors for another frame having another frame index value. For example, in inter-frame prediction, multiple predictors can be combined together in various functions that use the different predictors differently. In addition, predictor weights can be applied to each predictor differently, for example. As an example, in a first given frame, vertex values from two previously decoded frames can be used as predictors in a function that predicts a set of vertex values for the first given frame. However, for a second given frame, the previously used predictors (e.g., vertex values from two previously decoded frames) can be used with an additional predictor, such as a vertex value predicted for the first given frame in a function that predicts a set of vertex values for the second given frame. Similarly, in some embodiments, a residual for a second given frame vertex value can be predicted using a residual for a vertex value signaled for another frame, such as the first given frame or a set of previously signaled frames. In some embodiments, inter-frame prediction techniques include using a function that takes vertex values from reference frames as input. For example, a function used to predict vertex values for a given frame can be a function that indicates different weights (e.g., predictors) to be applied to different vertex values from one or more previously decoded frames. In some embodiments, different weights can be based on a temporal distance between the given frame and the previously decoded frames. Note that the previously decoded frames need not be sequential frames, and in some embodiments, can even be frames that occur later in time than the frame whose vertex values are currently being predicted.

[0012] The indices of the reference frame can be signaled in the bitstream. Further, the indices of the vertices used when referencing the reference frame can be different from the indices of the vertices signaled when signaling the encoded representation of the reference frame. For example, consider a reference frame including vertices A, B, and C for simplicity. An encoded version of the reference frame can place vertex A in the 1st index position, vertex B in the second index position, and vertex C in the third index position. However, when the reference frame is referenced as a previously decoded frame, different indices can be used. For example, the second indices used to reference the vertices of the reference frame can place vertex B in the 1st index position, vertex A in the second index position, and vertex C in the third index position. Reordering the reference index vertex positions in this way can allow the more frequently referenced vertex values to be placed in lower index positions (e.g., at the top of the list), which can improve compression efficiency with respect to signaling prediction factors (e.g., index values of vertices of a previously decoded frame used as prediction factors to predict vertex values of another frame). The reference frame indices can be signaled per frame or per vertex. For example, the same reference frame index order can be used for all prediction factors of the current frame being encoded / decoded, or alternatively, multiple indices can be saved in memory (e.g., multiple orderings of the vertices of the reference frame), and for a given vertex, the reference index to be used to locate the prediction factor and the position within that index can be signaled. Further, the reference indices can be signaled in the sequence parameter set and the frame parameter set, with the frame parameter set indicating differences from the definitions indicated in the sequence parameter set, e.g., differences that only apply to a particular frame. In some embodiments, instead of directly signaling such items, they can be signaled as difference values (or relative sums). For example, the difference between the index of the current vertex and the index of the reference vertex can be signaled (e.g., if the reference vertex position is 3 and the vertex position that uses that reference vertex as a prediction factor is position 2, the reference vertex position can be signaled as +1 (e.g., the difference between the position in the current frame index and the reference frame index)). In some embodiments, it is assumed that the reference vertex index position is always less than the current vertex index, so the difference is always subtracted from the current vertex index.

[0013] In some embodiments, the selection of predictors and / or the manner in which the predictors are used to predict the function of vertex values can be signaled at least in part at a level higher than an individual vertex, such as at a group of vertex level, where the group of vertices uses similar information related to inter-prediction. In some embodiments, a reference index to be used in a reference predictor can be signaled per group. For example, in the case that an additional reorder index of a given reference frame is to be used, such reorder index can be signaled to apply to a group of frames or a group of vertices. In some embodiments, a difference value of a current vertex and a reference vertex can be signaled per group, e.g., an offset or difference to be applied to an index position of a vertex being predicted to locate a vertex value of a predictor in a reference frame can be signaled in a manner such that the same offset is used to predict multiple vertex values of a group of vertices. Further, a type of function to be used (e.g., a function that accepts predictors as input), such as bi-prediction or uni-prediction, can be signaled per group of vertices.

[0014] In some embodiments, multiple attributes of a three-dimensional mesh from one or more frames (e.g., texture coordinates or texture connectivity) can be used to determine attribute values of a given frame being encoded (or decoded). For example, texture coordinates of a current frame can be determined by taking an average of texture coordinates from multiple different previously decoded frames having the same index position (or multiple texture coordinates in different index positions of a single frame). In some embodiments, mesh connectivity, texture coordinates, or texture connectivity of a current frame can be determined based on a pre-established rule, where the pre-established rule can indicate that the current frame is to use mesh connectivity, texture coordinates, or texture connectivity of a frame that is spatially or temporally closest to the current frame. Because a current mesh can have a correlation with more than one previously decoded frame (or with more than one component of a previously decoded frame), using multiple hypothesis inter-prediction to exploit such correlations can result in improved prediction that allows for a significantly reduced amount of data to be used to convey three-dimensional mesh data while maintaining reconstruction fidelity.

[0015] In some embodiments, the mesh in a time point frame can be segmented into multiple "sub-meshes," and the respective sub-meshes can be signaled in at least partially independent ways. For example, a given sub-mesh of a time point frame can be signaled using inter-frame prediction techniques, while another sub-mesh of the same mesh of the same time point frame can be signaled using a different prediction technique, such as intra-frame prediction. In some cases, there can be a different number of sub-meshes in the respective sub-bitstreams for a given time point frame. For example, if inter-frame prediction is being used for sub-mesh "A" of time point frame 2, but intra-frame prediction is being used for sub-mesh "B" of time point frame 2, the base mesh can be signaled for sub-mesh "B" but not for sub-mesh "A," because sub-mesh "A" can predict the vertex positions relative to the reconstructed sub-mesh "A" of time point frame 1 without needing to signal additional base mesh for sub-mesh "A" at time point frame 2. However, the atlas sub-bitstream and / or the displacement sub-bitstream can include entries for both sub-meshes of both time point frames. In some embodiments, to keep the respective sub-bitstreams aligned, empty sub-meshes of a given sub-bitstream can be generated in the reconstruction. As another example, a sub-mesh that is referenced in the atlas sub-bitstream but not referenced in the base mesh sub-bitstream can be removed from the atlas sub-bitstream. In this case, when inter-frame prediction is used, the predicted time point version of the sub-mesh can reuse (and / or predict) the atlas information from the reference frame, such that the atlas information in the atlas sub-bitstream for that sub-mesh of that time point frame can be removed from the atlas sub-bitstream.

[0016] In some embodiments, a video encoder is used to signal residual information for adjusting predicted vertex information, where vertex residuals are grouped into patches, and the patches are packed into two-dimensional (2D) video image frames. An atlas sub-bitstream maps vertices to subdivision positions and / or positions in three-dimensional (3D) space. However, when patches are used, it can also be necessary to signal the number of vertices that end the residual information in each respective packed patch. Thus, a nominal vertex count can be signaled per patch. BRIEF DESCRIPTION OF DRAWINGS

[0017] FIG. 1 An example input information for defining a three-dimensional mesh is illustrated, according to some embodiments.

[0018] FIG. 2 An alternative example of input information for defining a three-dimensional mesh is illustrated, according to some embodiments, where the input information is formatted according to an object format.

[0019] FIG. 3 Multiple reference frame inter-frame prediction of geometry information for a three-dimensional mesh is illustrated, according to some embodiments.

[0020] FIG. 4 Single reference frame inter prediction of geometry information for a three-dimensional mesh is illustrated in accordance with some embodiments.

[0021] FIG. 5 Inter prediction of geometry information for a three-dimensional mesh using more than two reference frames is illustrated in accordance with some embodiments.

[0022] FIG. 6 Non-sequential / unordered multi-reference frame inter prediction of geometry information for a three-dimensional mesh is illustrated in accordance with some embodiments.

[0023] FIG. 7 Inter prediction process for texture coordinates of a three-dimensional mesh is illustrated in accordance with some embodiments.

[0024] FIG. 8 Inter prediction process for connectivity information of a three-dimensional mesh is illustrated in accordance with some embodiments.

[0025] FIG. 9 Adjusting the offset of a three-dimensional mesh relative to an origin for encoding compared to the origin of the actual three-dimensional space in which the three-dimensional mesh resides, for example to save bit-depth, is illustrated in accordance with some embodiments.

[0026] FIG. 10 Different types of prediction-related information that can be signaled for mesh attributes are illustrated in accordance with some embodiments.

[0027] FIG. 11A-11B Example signaling and positioning of tile and slice bounding boxes is illustrated in accordance with some embodiments.

[0028] FIG. 12 Example signaling and positioning of slices is illustrated in accordance with some embodiments, where more than one is signaled for a slice.

[0029] FIG. 13 Example mapping between texture coordinates and image pixel coordinates is illustrated in accordance with some embodiments.

[0030] FIG. 14 Example intra encoder is illustrated in accordance with some embodiments.

[0031] FIG. 15 Example intra decoder for decoding a three-dimensional mesh is illustrated in accordance with some embodiments.

[0032] FIG. 16 Example inter encoder is illustrated in accordance with some embodiments.

[0033] FIG. 17An example inter-frame decoder for decoding a three-dimensional mesh is illustrated in accordance with some embodiments.

[0034] FIG. 18 An example multi-hypothesis inter-frame encoder is illustrated in accordance with some embodiments.

[0035] FIG. 19 An example multi-hypothesis decoder for decoding a three-dimensional mesh is illustrated in accordance with some embodiments.

[0036] FIG. 20 is a flowchart illustrating an example process for reconstructing a dynamic mesh using inter-frame prediction in accordance with some embodiments.

[0037] FIG. 21 is a flowchart illustrating an example process for reconstructing a dynamic mesh in accordance with some embodiments, in which at least some sub-meshes have base meshes omitted from a base mesh sub-bitstream for at least some time point frames.

[0038] FIG. 22 is a flowchart illustrating an example process for reconstructing a dynamic mesh in accordance with some embodiments, in which a vertex count is signaled per tile.

[0039] FIG. 23 An example computer system that can implement an encoder or decoder is illustrated in accordance with some embodiments.

[0040] This specification includes references to “one embodiment” or “an embodiment.” Occurrences of the phrases “in one embodiment” or “in an embodiment” do not necessarily all refer to the same embodiment. Particular features, structures, or characteristics can be combined in any suitable way in one or more embodiments.

[0041] “Comprise,” as well as variations such as “comprises” or “comprising,” is open-ended. As used in the attached claims, this term does not foreclose addition of

[0042] “Configured to.” Various units, circuits, or other components can be described or claimed as “configured to” perform a task or tasks. In such contexts, “configured to” is used to connote structure by indicating that a unit / circuit / component includes structure (e.g., circuitry) that performs the task or tasks during operation. As such, the unit / circuit / component can be said to be configured to perform the task or tasks even when the specified unit / circuit / component is not currently operational (e.g., is not on). Units / circuits / components used with “configured to” language include hardware-only implementations, implementations including software-defined

[0043] “First,” “second,” etc. As used herein, these terms are used as labels in place of referring back to prior nouns and do not connote any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit can be described herein as performing write operations of a “first” value and a “second” value. The terms “first” and “second” do not necessarily connote that the first value must be written before the second value.

[0044] “Based on.” As used herein, this term is used to describe one or more factors that affect a determination. This term does not foreclose additional factors that can affect a determination. That is, a determination can be solely based on the factors specified, or based only partially on the factors specified and / or based on additional factors not specified. Consider the phrase “determine A based on B.” While in this case, B is a factor that affects the determination of A, such a phrase does not foreclose the determination of A from also being based on C. In other instances, A can be determined based solely on B. DETAILED DESCRIPTION

[0045] As data acquisition and display technology has become more advanced, the ability to capture volumetric content including thousands or millions of points in two-dimensional (2D) or three-dimensional (3D) space via LIDAR systems, for example, has increased. Additionally, the development of advanced display technology, such as virtual reality or augmented reality systems, has increased the potential uses of volumetric content. However, volumetric content files are often very large, and storing and sending these volumetric content files can be costly and time consuming. For example, communication of volumetric content over private networks or public networks, such as the Internet, can require a substantial amount of time and / or network resources, such that some uses of volumetric content, such as real-time uses, can be limited. Additionally, the storage requirements of volumetric content files can consume a significant amount of storage capacity of devices storing the volumetric content files, which can also limit potential applications using volumetric content data.

[0046] In some embodiments, encoders can be used to generate compressed volumetric content to reduce the costs and time associated with storing and sending large volumetric content files. In some embodiments, a system can include an encoder that compresses attribute information and / or spatial information of volumetric content such that the volumetric content file can be stored and transmitted faster than non-compressed volumetric content, and in a manner that the volumetric content file can occupy less storage space than non-compressed volumetric content.

[0047] In some embodiments, additionally or alternatively, such encoders and decoders or other encoders and decoders described herein can be adapted to encode three degrees of freedom plus (3DOF+) scenes, visual volumetric content such as MPEG V3C scenes, immersive video scenes such as MPEG MIV, and the like.

[0048] In some embodiments, a static or dynamic mesh to be compressed and / or encoded can include a set of 3D meshes M(0), M(l), M(2)... M(n-l), where "n" is the number of time point meshes in the set of 3D meshes. Each mesh M(i) at frame index "i" (also shown as mesh [i]) can be defined by connectivity information C(i), geometry information G(i), texture coordinates T(i), and texture connectivity CT(i). For each mesh M(i), one or more 2D images A(i) describing textures or attributes associated with the mesh can be included. For example, FIG. 1 An example static or dynamic mesh M(i) is illustrated that includes connectivity information C(i), geometry information G(i), texture images A(i), texture connectivity information TC(i), and texture coordinate information T(i). In some embodiments, the geometry information G(i) can include information about vertices, each vertex including (vertex[0], vertex[l], vertex[2]). FIG. 2Illustrate an example of a textured mesh stored in object (OBJ) format.

[0049] For example, FIG. 2 The example texture mesh stored in the object format shown includes geometric information listed as the X, Y, and Z coordinates of the vertices and texture coordinates listed as the two-dimensional (2D) coordinates of the vertices, where the 2D coordinates identify the pixel location of the pixel storing the texture information for a given vertex. The example texture mesh stored in the object format also includes texture connectivity information indicating the mapping between geometric coordinates and texture coordinates to form polygons such as triangles. For example, a first triangle is formed by three vertices, where the first vertex (1 / 1) is defined as first geometric coordinates (e.g., 64.062500, 1237.739990, 51.757801), which corresponds to first texture coordinates (e.g., 0.0897381, 0.740830). The second vertex (2 / 2) of the triangle is defined as second geometric coordinates (e.g., 59.570301, 1236.819946, 54.899700), which corresponds to second texture coordinates (e.g., 0.899059, 0.741542). Finally, the third vertex of the triangle corresponds to the third listed geometric coordinate that matches the third listed texture coordinate. However, it should be noted that in some instances, the vertices of a polygon, such as a triangle, may be mapped to a set of geometric coordinates and texture coordinates that may have different index positions in the corresponding lists of geometric coordinates and texture coordinates. For example, the second triangle has a first vertex that corresponds to the fourth listed set of geometric coordinates and the seventh listed set of texture coordinates. The second vertex corresponds to the first listed set of geometric coordinates and the first listed set of texture coordinates, and the third vertex corresponds to the third listed set of geometric coordinates and the ninth listed set of texture coordinates.

[0050] In some embodiments, the geometric information G(i) may represent the positions of the vertices of the mesh in 3D space, and the connectivity C(i) may indicate how the vertices are connected together to form the polygons that make up the mesh M(i). Additionally, the texture coordinates T(i) may indicate the positions of pixels in the 2D image corresponding to the vertices of the corresponding sub-mesh. The attribute patch information may indicate how texture coordinates defined relative to a 2D bounding box are mapped into the three-dimensional space of a 3D bounding box associated with the attribute patch based on how the points are projected onto the projection plane of the attribute patch. Additionally, the texture connectivity information TC(i) may indicate how the vertices represented by the texture coordinates T(i) are connected together to form the polygons of the sub-mesh. For example, each texture or attribute patch of the texture image A(i) may correspond to a corresponding sub-mesh defined using the texture coordinates T(i) and the texture connectivity TC(i).

[0051] FIG. 3Inter-prediction of geometry information for a three-dimensional mesh is illustrated in accordance with some embodiments.

[0052] In some embodiments, a 3D mesh can be encoded / decoded using inter-prediction to predict geometry information G(i) for a current (or another) time point frame of the 3D mesh. In some embodiments, vertex values of the 3D mesh can be determined using different vertex values from multiple different previously decoded time point frames of the 3D mesh. For example, geometry information for a given frame (e.g., a frame being encoded or decoded) can be predicted using multiple reference frames (e.g., previously decoded frames, such as a reference frame at index position 0 and a reference frame at index position 1, where index position 0 and index position 1 represent instances of the dynamic mesh at different time instants, likewise within a given reference frame, each of the vertices can also be ordered in one or more indices, such as indices of vertex values). The “geometry” G(i) of the current frame (e.g., vertices) at the current frame can be predicted using a function that relates the current vertex to vertices from reference frame 0 and reference frame 1.

[0053] Reference frame 0 (e.g., at frame index position 0) can correspond to a set of vertices [(vertex[0][0],vertex[0][1],vertex[0][2]),(vertex[1][0],vertex[1][1],vertex[1][2]),(vertex[2][0],vertex[2][1],vertex[2][2])……(vertex[n][0],vertex[n][1],vertex[n][2])] and reference frame 1 can correspond to a set of vertices [(vertex[0][0],vertex[0][1],vertex[0][2]),(vertex[1][0],vertex[1][1],vertex[1][2]),(vertex[2][0],vertex[2][1],vertex[2][2])……(vertex[n][0],vertex[n][1],vertex[n][2])]. Note that in this example, the first index value in the brackets indicates the index position of the vertex in reference frame 0, and the second value in the brackets indicates the component value of that vertex, such as X, Y, and Z components. The vertices of the current frame that are predicted using multiple reference frame inter-prediction can be described using a set of functions:

[0054] mesh[i].vertex[v][0] = F(mesh[refer0].vertex[refV0][0], mesh[refer1].vertex[refV1][0]) + residual[0]

[0055] mesh[i].vertex[v][1] = F(mesh[refer0].vertex[refV0][1], mesh[refer1].vertex[refV1][1]) + residual[1]

[0056] mesh[i].vertex[v][2] = F(mesh[refer0].vertex[refV0][2], mesh[refer1].vertex[refV1][2]) + residual[2]

[0057] The variable “i” indicates a frame index, “v” indicates a vertex index, and vertex[0], vertex[1], and vertex[2] indicate vertex positions for x, y, and z coordinates of the 3D mesh. The residual can be a difference between the original mesh and the encoded and reconstructed version of the mesh. The “refer0” / “refer1” indicates a reference frame index for the reference mesh frame 0 and 1, and the “refV0” / “refV1” indicates a vertex index for the reference mesh frame 0 and the reference mesh frame 1, respectively. The function F() can include various types of functions, such as a linear or non-linear function that determines a component vertex value based on a combination of component vertex values read from the reference frames. In some embodiments, the function F() can be a mean average function or a weighted average function, where the weights for the weighted average function can be based on a temporal distance between the current mesh and the reference meshes (e.g., a temporal distance between the current frame and the respective reference frames). Further, although two reference frames are depicted for the vertices from the two reference frames being used, more than two previously decoded reference frames can be used. FIG. 3

[0058] In some embodiments, the type of inter-prediction (e.g., single reference frame inter-prediction, single reference frame multi-hypothesis inter-prediction, multi-reference frame multi-hypothesis inter-prediction, etc.) can be signaled in the bitstream for the 3D mesh. Further, in some embodiments, the reference mesh set index and the reference mesh index can be signaled in the bitstream. In some embodiments, when two reference meshes are used for inter-prediction, one reference frame can be selected per reference mesh list, or the two reference meshes can be selected from one reference mesh set. In other words, more than one index can be used to order the reference frames. Further, within a given reference frame, more than one index can be used to order the vertex positions.

[0059] ​In some embodiments, instead of signaling a reference list, a predefined combination can be signaled that includes both the prediction information and the reference information to be used by the function for determining the prediction information. For example, the combination can include information about the prediction type and the reference mesh used in the prediction. When multiple combinations or groups of combinations are available, their indices can be signaled in the bitstream, per mesh or per mesh group or per sequence. For example, for the nth mesh frame, a first reference mesh list can be conceptually constructed as {{uni-prediction, mesh[n-1]},{bi-prediction, mesh[n-2], mesh[n-3]}}, and a second reference list can be conceptually constructed as {{uni-prediction, mesh[n-3]},{uni-prediction, mesh[n-2]}}. Then, for each mesh, an indication of which reference list to use and which combination to use can be signaled in the bitstream. In some embodiments, the prediction type can be derived from the number of reference meshes in the combination. In some embodiments, the prediction type can also indicate intra prediction.

[0060] FIG. 4 Single reference frame inter prediction for geometry information of a three-dimensional mesh is illustrated in accordance with some embodiments.

[0061] In some embodiments, a 3D mesh can be encoded / decoded using inter prediction, where multiple different vertices from a single frame of the 3D mesh are used to predict vertex values at another frame (e.g., the frame that is currently being encoded or decoded) of the 3D mesh.

[0062] Similar to FIG. 3 A vertex (or other information) of a current frame can be predicted. However, instead of using multiple vertices from multiple reference frames, the vertex value can be predicted using multiple vertices from a single reference frame. For example, the geometry information G(i) of a current frame can be predicted using a function that relates the current vertex to two different vertices from the same reference frame 0.

[0063] For example, the reference frame 0 can include or correspond to a set of vertices [(vertex[0][0], vertex[0][1], vertex[0][2]), (vertex[1][0], vertex[1][1], vertex[1][2]), (vertex[2][0], vertex[2][1], vertex[2][2])... (vertex[n][0], vertex[n][1], vertex[n][2])]. Using single reference frame multi-vertex inter prediction, a vertex of a current frame can be described using a set of functions:

[0064] mesh[i].vertex[v][0] = F(mesh[refer0].vertex[refV0][0], mesh[refer0].vertex[refV1][0]) + residual[0]

[0065] mesh[i].vertex[v][1] = F(mesh[refer0].vertex[refV0][1], mesh[refer0].vertex[refV1][1]) + residual[1]

[0066] mesh[i].vertex[v][2] = F(mesh[refer0].vertex[refV0][2], mesh[refer0].vertex[refV1][2]) + residual[2]

[0067] As discussed in FIG. 3 , the function F() can include various types of functions, such as linear or non-linear combinations, including mean average functions or weighted average functions based on temporal distance weighting. In addition, more than two vertices from the same frame can be used to predict a current vertex. In some embodiments, a combination of multi-reference frame inter-prediction and single reference frame multi-vertex inter-prediction can be used. For example, a vertex of a current frame can be predicted using multiple vertices from reference frame 0 as well as a vertex from reference frame 1.

[0068] FIG. 5 An example of using more than two reference frames for inter-prediction of geometry information for a three-dimensional mesh according to some embodiments is illustrated.

[0069] In some embodiments, when inter-prediction is used, one or more simulated reference frames can be generated during the decoding process. For example, a new reference frame referMesh can be generated based on two reference frames indicated by the signaled reference indices refer0, referl as follows:

[0070] referMesh.vertex[v][0] = F'(mesh[refer0].vertex[refV0][0], mesh[referl].vertex[refVl][0])

[0071] referMesh.vertex[v][1] = F'(mesh[refer0].vertex[refV0][1], mesh[referl].vertex[refVl][1])

[0072] referMesh.vertex[v][2] = F'(mesh[refer0].vertex[refV0][2], mesh[referl].vertex[refVl][2]).

[0073] In some embodiments, the simulated reference mesh can be generated using the top N reference frames in the reference list.

[0074] The simulated reference mesh can be used as one of the multiple reference frames to predict the current mesh. For example, the geometry information G(i) (e.g., vertices) of the current frame at frame index “i” can be predicted using a function that relates the current vertices to the vertices from reference frame 1 and itself can be based on multiple reference frames, referMesh. More than two reference frames can be used to predict the current frame. The vertices of the current frame at frame index “i” can be described using a set of functions:

[0075] mesh[i].vertex[v][0] = F(mesh[referl].vertex[refV0][0], referMesh.vertex[refV][0]) + residual[0]

[0076] mesh[i].vertex[v][l] = F(mesh[referl].vertex[refV0][l], referMesh.vertex[refV][l]) + residual[l]

[0077] mesh[i].vertex[v][2] = F(mesh[referl].vertex[refV0][2], referMesh.vertex[refV][2]) + residual[2]

[0078] where refer0 indicates the frame index of the first reference frame, and refVl and refV indicate the vertex indices of the first reference mesh frame 1 and the second reference mesh frame, respectively.

[0079] In some embodiments, more than two reference frames can be explicitly indicated. Furthermore, instead of signaling two reference indices, e.g., refer0 and referl above, more than two reference indices such as K indices denoted as referIndex[0], referIndex[l]... referIndex[K-l] can be signaled. The vertices of the current frame at frame index “i” can be described using a set of functions:

[0080] mesh[i].vertex[v][j] = F"(mesh[referIndex[0]].vertex[refV[0]][j],... mesh[referIndex[K-1]].vertex[refV[k-1]][j]) + residual[j]

[0081] where j is 0..2 and refV[k] indicates a vertex index of mesh[referIndex[k]].

[0082] FIG. 6 Non-sequential / unordered reference frame inter prediction for geometry information of a three-dimensional mesh is illustrated according to some embodiments.

[0083] In some embodiments, a 3D mesh can be encoded / decoded using inter prediction, where vertex positions from multiple non-sequential / unordered frames of the 3D mesh are used to predict geometry information G(i) at a current (or another) frame. As discussed in FIG. 4 As discussed in the Background section, geometry information of a current frame can be predicted using multiple reference frames (e.g., reference frame X and reference frame Y, where X and Y are not sequential). G(i) of the current frame can be predicted using a function that relates a current vertex to vertices from reference frame X and reference frame Y. In some embodiments, the reference frames X and Y used to predict the current frame can be non-sequential and / or unordered. For example, there can be multiple reference frames between reference frame X and reference frame Y, such that reference frame Y is not the next reference frame in a sequence of reference frames that can be used to determine vertices of the current frame. Similarly, the current frame can not be the frame to be rendered after reference frame Y.

[0084] For example, reference frame X can correspond to a set of vertices [(vertex[0][0], vertex[0][1], vertex[0][2]), (vertex[1][0], vertex[1][1], vertex[1][2]), (vertex[2][0], vertex[2][1], vertex[2][2])... (vertex[n][0], vertex[n][1], vertex[n][2])] and reference frame Y can correspond to a set of vertices [(vertex[0][0], vertex[0][1], vertex[0][2]), (vertex[1][0], vertex[1][1], vertex[1][2]), (vertex[2][0], vertex[2][1], vertex[2][2])... (vertex[n][0], vertex[n][1], vertex[n][2])]. Reference frame X and reference frame Y can be separated by a number of reference frames and can not be limited to consecutive frames. Vertices of a current frame using inter-frame prediction of a number of reference frames can be described using a set of functions:

[0085] mesh[i].vertex[v][0] = F(mesh[referX].vertex[refVX][0], mesh[referY].vertex[refVY][0]) + residual[0]

[0086] mesh[i].vertex[v][1] = F(mesh[referX].vertex[refVX][1], mesh[referY].vertex[refVY][1]) + residual[1]

[0087] mesh[i].vertex[v][2] = F(mesh[referX].vertex[refVX][2], mesh[referY].vertex[refVY][2]) + residual[2]

[0088] A current frame can similarly have a number of other frames to be rendered between previous reference frame Y.

[0089] FIG. 7 Inter-frame prediction techniques for texture coordinates of three-dimensional meshes are illustrated according to some embodiments.

[0090] In some embodiments, the 3D mesh may be encoded / decoded using inter-frame prediction, where multiple different texture coordinates (e.g., texture coordinates T(i)) from multiple different frames are used to predict the texture coordinates T(i) of the current (or another) frame of the 3D mesh. FIG. 5 As discussed in the inter-frame prediction for determining vertex values, the texture coordinates of the current frame can be predicted using multiple reference frames (e.g., reference frame 1 and another reference frame, or even a simulated reference frame). Reference frame 1 may correspond to a set of texture coordinates [(Vt 1u ,Vt 1v ),(Vt 2u ,Vt 2v )……(Vt nu ,Vt 1n )], and the mesh vertices for the assumed reference frame can be described by a set of functions:

[0091] referMesh.texCoords[t][0]=H'(mesh[refer0].texCoords[refT0][0],mesh[refer1].texCoords[refT1][0])

[0092] referMesh.texCoords[t][1]=H'(mesh[refer0].texCoords[refT0][1],mesh[refer1].texCoords[refT1][1])

[0093] The texture coordinates of the current frame can be predicted using inter-frame prediction using reference frame 1 and another reference frame (such as an analog reference frame), where the analog reference frame is predicted using reference frame 0 and reference frame 1. The vertices of the current frame can be described using a set of functions:

[0094] mesh[i].texCoords[v][0]=H(mesh[refer1].texCoords[refT1[t][0]], mesh[referMesh].texCoords[refT[t][0]])

[0095] mesh[i].texCoords[v][1]=H(mesh[refer1].texCoords[refT1[t][1]], mesh[referMesh].texCoords[refT[t][1]])

[0096] refer0 / refer1 indicate reference frame indices, and refT1 / refT indicate texture coordinate indices corresponding to reference mesh frame 1 and hypothetical reference mesh frame, respectively. Similar to the function F() described in FIG. 3

[0097] Although FIG. 7 While the current frame is depicted as relying on two sequential previously encoded frames, the reference frames used to predict the current (or another frame) frame can be non-sequential and can be at various frame indices, as discussed in FIG. 6 FIG. 4 Additionally, as discussed in

[0098] In some embodiments, a combination of multi-reference frame texture coordinate inter prediction and single-reference frame multi-texture coordinate inter prediction can be used. For example, the texture coordinates of the current frame can be predicted using multiple texture coordinates from reference frame 0 as well as texture coordinates from reference frame 1.

[0099] FIG. 8 Inter prediction techniques for connectivity information of three-dimensional meshes according to some embodiments are illustrated.

[0100] In some embodiments, connectivity information (e.g., mesh vertex connectivity C(i) and / or texture coordinate connectivity TC(i)) can be determined using one or more reference frames. In some embodiments, vertex connectivity information and / or texture connectivity information can be determined based on using an indication of connectivity information and / or texture connectivity from one or more previous frames. For example, connectivity information (e.g., vertex connectivity) of a current frame can be determined based on an indication of reference frame connectivity that is selected to be applied to the current frame signaled in a bitstream.

[0101] For example, the connectivity of the geometry of reference frame 1 can be described by a set of indices (e.g., indicating that a first triangle is formed by vertices with indices 1, 2, and 3, a second triangle is formed by vertices with indices 4, 1, and 3, and a third triangle is formed by vertices with indices V​​i , V j , and V k form a set of indices). Similarly, the connectivity of the texture coordinates of reference frame 1 can be described by a set of indices (e.g., indicating that a first triangle is formed by vertices having indices 1, 2, and 3, a second triangle is formed by vertices having indices 7, 1, and 9, and a third triangle is formed by vertices having indices V l , V m , and V n form a set of indices). The connectivity of the geometry (and the connectivity of the texture coordinates) of reference frame 4 can be described by a different set of indices.

[0102] In some embodiments, one or more indications can be signaled in the bitstream that indicate the reference frame whose connectivity information is to be used. For example, FIG. 8 illustrates that for frame 1, the vertex connectivity from reference frame 1 should be used, and for the current frame, the texture connectivity from reference frame 4 should be used. The signals indicating “referl” / “refer4” indicate that the reference mesh frames 1 and 4 can be provided in the bitstream. In some embodiments, a multi-vertex inter-frame encoder can apply multi-hypothesis inter prediction as described above. For example, instead of using a single reference base mesh, a multi-hypothesis encoder can use multiple frames to perform inter prediction (as discussed above in FIG. 3

[0103] FIG. 9 illustrates adjusting the offset of a three-dimensional mesh relative to an origin point for encoding compared to an origin point for the actual three-dimensional space in which the three-dimensional mesh resides, for example to save bit-depth, in accordance with some embodiments.

[0104] In some embodiments, a 3D mesh can be partitioned into multiple parts (e.g., sub-meshes), and a 3D offset value can be signaled to indicate a translation (and / or rotation) of the corresponding sub-mesh from the origin (0, 0, 0) of the 3D space of each part. For example, as shown in FIG. 16 three different parts of the mesh (e.g., sub-meshes) to reposition these parts to the origin, saving bit-depth.

[0105] ​In some embodiments, when a flag in atlas sequence parameter set (ASPS) or atlas frame parameter set (AFPS) is indicated, the offset can be signaled. In some embodiments, offsetU, offsetV, and offsetD can be signaled using a descriptor ue(v), or depending on when the inverse translation process is applied, they can be signaled in fixed length based on the bit depth of the output mesh of the base mesh sub-bitstream or based on the bit depth of the output mesh of the variable dynamic mesh compression (V-DMC) bitstream.

[0106] In some embodiments, the offset can be derived from the center of the mesh instead of the origin of the 3D space. In some embodiments, in addition to the offset, the size of the bounding box of the reconstructed sub-mesh of the patch can also be indicated.

[0107] Reconstructing meshes from reference meshes with different numbers of vertices

[0108] In some embodiments, when performing the above inter-prediction techniques, when the reference vertex index is inferred to be the same as the current vertex index, and the reference mesh total is a value smaller than the index value of the current mesh, the predictor can be obtained from only the valid reference vertices. If both reference mesh references are invalid, the predictor can be set to 0 or the same as the previously reconstructed vertex, or set to the average of the reconstructed vertices. In some embodiments, when the reference vertex index of one of the reference meshes is invalid, then the predictor can use the last valid vertex value in the reference mesh. The same approach can be applied to the texture coordinates and connectivity when copying them from the reference.

[0109] Prediction by vertex group

[0110] In some embodiments, it can be more efficient to predict only a limited portion of the mesh from the reference meshes, where other portions of the mesh are intra-predicted. To accommodate this, a vertex can be signaled to belong to a group (e.g., sub-mesh), and then the vertices in a group can be predicted by the prediction mode of the group. For each group of vertices, the prediction mode, intra-prediction or other multi-hypothesis prediction such as uni-prediction, bi-prediction can be signaled, where the signaled prediction mode applies to all vertices in the group. Furthermore, for each group, inter-prediction related information such as reference mesh information, reference mesh list index, reference mesh index, predictor derivation method, and weight of the predictor can be signaled for a group of vertices.

[0111] When the reference vertex index is explicitly signaled or the difference between the current vertex and the reference vertex index is signaled, it can be signaled per group. The reference vertex index of a consecutive vertex index in a group can be a consecutive vertex index from the signaled or computed reference vertex index. For example, when the reference vertex index obtained for a group is refG, the reference vertex index of the i-th vertex in the group can be refG+i.

[0112] In some embodiments, the size of a group N can be signaled or can be predefined for the whole sequence or for a frame or for a sub-grid or a grid. In some embodiments, the size of a group (e.g., a sub-grid) can be determined based on the number of vertices in the grid. In some embodiments, the size of a group can be signaled per group (e.g., sub-grid) at the beginning of each group (e.g., sub-grid). In some embodiments, a list containing information indicating the size of each group (e.g., sub-grid) can be signaled at the beginning of the grid. In some embodiments, the number of groups can be predefined and the prediction mode of each group can be predetermined.

[0113] Reference mesh reordering

[0114] In some embodiments, the vertices in the reference time point grid and their corresponding components can be reordered for better prediction. The reordering method can be signaled per grid, per frame, per sequence, or can be predefined. The reordering method of the multi-time point reference grid can be the same or can be different. An example of reordering can include ordering the vertices based on their positions.

[0115] Residual vector prediction

[0116] In some embodiments, a residual can be added to the predictor to reconstruct the current vertex value. The residual can be predicted from other residuals from neighboring vertices in the current grid. For example, the residual of a current frame can be described by the following set:

[0117] residual[v][0] = N(residual[neigh0][0],..., residual[neighn-1][0]) + refine[v][0]

[0118] residual[v][1] = N(residual[neigh0][1],..., residual[neighn-1][1]) + refine[v][1]

[0119] residual[v][0] = N(residual[neigh0][0],..., residual[neighn-l][0]) + refine[v][0] residual[v][l] = N(residual[neigh0][l],..., residual[neighn-l][l]) + refine[v][l] residual[v][2] = N(residual[neigh0][2],..., residual[neighn-l][2]) + refine[v][2]

[0120] neigh0... neighn-l can indicate neighboring vertex indices of the current vertex with vertex index v. The number of neighboring vertices can be finite. N() can be any type of function, such as a linear or non-linear combination among the neighboring vertices. For example, N() can include a mean average function or a weighted average function similar to F() or H() discussed above in FIG. 3 and FIG. 7 In some embodiments, the weights for the weighted average can be signaled. In some embodiments, the weights can be determined from the distance to the neighboring vertices and the predictor for the current vertex, or can be pre-established according to one or more rules. In some embodiments, the weights can be pre-established per grid or per sequence or per group of vertices.

[0121] In some embodiments, the residual for a vertex can be set to be the same as the residual for a reference vertex in the reference grid. In some embodiments, the residual for a vertex can be set to be the same as the residual for another vertex in the current grid. The index of the reference vertex in the current grid can be signaled or derived.

[0122] Prediction accuracy

[0123] In some embodiments, when determining the predictor (such as using the functions F(), H(), N() described above to determine the value of the predictor), the values can be rounded to the nearest integer value. In other cases, the values can be rounded down to the nearest and smaller integer value. In some embodiments, the values can be kept at a certain sub-precision (such as 1 / 2, 1 / 4, or 1 / 8) until the reconstruction of the grid is completed, and then the sub-precision positions of the grid can be refined to integer precision based on a signaled syntax related to the bit depth of the reconstructed grid. In some embodiments, a clipping process can be applied during prediction and / or during residual calculation and / or during the final reconstruction.

[0124] Inter- and intra-predicted meshes

[0125] In some embodiments, some groups of the mesh (e.g., sub-meshes) can be inter-predicted, and some groups (e.g., other sub-meshes of the mesh) can be intra-predicted. In this case, intra-coding data (e.g., static mesh codec coding data) and inter-prediction information such as reference indices and residuals can be signaled for one sub-mesh. For example, for every N[g] vertices of a group, the prediction type, intra-prediction or inter-prediction including the weight of the prediction factor, and inter-prediction related information can be signaled, where N[g] indicates the number of vertices in the gth group. After reconstructing each part, the reconstructed meshes can be connected. The connectivity information can then be updated accordingly.

[0126] In some embodiments, the mesh component of each part (inter-predicted and intra-predicted parts, such as inter-predicted or intra-predicted sub-meshes) can include virtual values for one or more parts whose information is signaled in other parts. For example, if an X value is intra-predicted and Y and Z values are inter-predicted, a virtual value can be used for the X value in inter-prediction. As another example, a first part of a mesh has a first group of N vertices that can be intra-predicted, and a second part of the mesh has the next M vertices that can be inter-predicted, where the reconstructed mesh of the first part can only have N vertices, and the second part only has M vertices. In some embodiments, the reconstructed sub-mesh corresponding to the first part can have (N+M) vertices, but only N vertices can have valid geometry values. The values of the M vertices can be replaced by values reconstructed from the second part. In some embodiments, the reconstruction method for connectivity among groups of vertices can be signaled.

[0127] In some embodiments, the prediction type (e.g., intra-prediction and / or inter-prediction) can be indicated per triangle, instead of per vertex. Similarly, in some embodiments, the motion vector can also be signaled per triangle. A vertex can consist of triangles that are predicted by the indicated method.

[0128] In some embodiments, inter-prediction can have multiple modes that indicate to signal prediction related information for attributes. For example, one of the inter-prediction modes can indicate to signal only the residual of the vertex position (the difference between the position of the vertex of the reference and the current vertex position). Another mode can indicate that the residual of the texture coordinate is also signaled.

[0129] When the residual of the texture coordinate is signaled, the motion can be signaled per group of texture coordinates. The groups can be formed by the connectivity information. Furthermore, a linear transformation can be applied to the texture coordinates to reduce the residual. The linear transformation can include shifting (e.g., translation), rotation (e.g., 90 degrees, 180 degrees, 270 degrees), transposition, and combinations.

[0130] For example, FIG. 10 Different types of prediction-related information signaled for mesh attributes are illustrated according to some embodiments. As shown in FIG. 10 A portion of the mesh can be indicated to be coded with a static mesh codec (e.g., intra-coded), another portion of the mesh can be indicated to have only the residual of signaled vertices (which means connectivity and texture coordinates are the same as the reference), and a third portion of the mesh can be indicated to have the residual of also signaled texture coordinates, but connectivity is the same as the reference frame.

[0131] In addition, in FIG. 10 , texture maps of the current frame and the reference frame illustrate different texture coordinate transformations. In some embodiments, one region of the texture image shows that the corresponding texture coordinates of the current frame have no change compared to the reference frame. Another region of the texture image shows that the corresponding texture coordinates of the current frame can have a shift (translation) compared to the reference frame. A third region of the texture image shows that the corresponding texture coordinates of the current frame can be shifted and rotated from the reference frame. For each set of texture coordinates (e.g., connectivity component), it can be signaled that the texture coordinates are shifted or rotated. Then, the difference between the shifted and / or rotated texture coordinates and the reference texture coordinates can be signaled.

[0132] When the texture coordinates are signaled together with their connectivity component information, the residual can be signaled in a separate (e.g., different) bitstream. For example, in V-DMC, a 3D mesh compression system being developed in MPEG, the residual can be signaled in the video sub-bitstream, and the rest of the information such as translation motion, rotation motion, and bounding box information (minimum and maximum values of the texture coordinates in a group (connectivity component)) can be signaled in the atlas sub-bitstream. In another embodiment, the texture coordinates can be intra-coded on a per-connectivity component basis. The bounding box information of the connectivity component of the texture coordinates can be signaled, and the difference between the texture coordinates and the minimum value of the connectivity component can be signaled. In this case, in the context of V-DMC, the bounding box information and the difference can be signaled in different sub-bitstreams.

[0133] In some embodiments, the bounding box information of the connectivity component of the vertex positions can be signaled, and the difference between the vertex positions and the minimum value of the connectivity component can be signaled. In this case, in the context of V-DMC, the bounding box information and the difference can be signaled in different sub-bitstreams.

[0134] In another implementation, the bounding box information of the connected component of the vertex position can be signaled for intra prediction, where the difference between the vertex position and the minimum value of the connected component can be signaled. Similarly, the bounding box information and the difference can be signaled in different sub-bitstreams.

[0135] V-DMC, inter-prediction of displacements

[0136] In some implementations, the base mesh in a time point frame can be a simplified version of the desired reconstructed mesh. In addition, the delivered refinement data (e.g., “displacement”) can be added to the simplified version. In some implementations, the information of the simplified base mesh sequence is signaled in a sub-bitstream named base mesh sub-bitstream. The information of the displacement sequence can be signaled in a video sub-bitstream. The displacement of a frame can be predicted from the displacement in another frame, and the residual, e.g., the difference between the predicted displacement and the original displacement, can be signaled. The displacement values can be composed of displacement data or residual data before using any compression method such as video codec or arithmetic codec. For this inter-frame prediction, the above-mentioned inter-frame prediction process can be used.

[0137] V-DMC, number of sub-meshes in different sub-bitstreams

[0138] In some implementations, the mesh in a time point frame can be partitioned into multiple “sub-meshes”, and then the sub-meshes can be coded independently. The sub-meshes in one frame can have different prediction types, such as intra prediction and inter prediction. The above-mentioned concepts can be applied by sub-mesh in the present system.

[0139] In some embodiments, the bitstream can be composed of independent sub-bitstreams, such as a base mesh sub-bitstream and an atlas data sub-bitstream. In such embodiments, the number of sub-meshes in a time point frame or their IDs can be represented differently in different sub-bitstreams for that time point frame. For example, in the base mesh sub-bitstream, it can not be necessary to signal a new base mesh for each sub-mesh of each time point frame. Thus, some time point frames can include fewer signaled base meshes for the respective sub-meshes than in other time point frames. As a simplified example, consider a dynamic mesh that is partitioned into 5 sub-meshes. As an example, for a first time point, 5 base meshes can be signaled in the base mesh sub-bitstream, and 5 sets of corresponding atlas information can be signaled in the atlas sub-bitstream. However, for a next time point frame, some of the sub-meshes can use inter-frame prediction such that no new base meshes need to be signaled for the next time point frame. In this case, fewer than 5 (e.g., 3) base meshes can be signaled in the base mesh sub-bitstream for the next time point frame (e.g., 3 sub-meshes are predicted using intra-frame prediction, and the other two sub-meshes are predicted using inter-frame prediction). However, in the atlas sub-bitstream, an additional 5 sets of atlas information will be signaled for the next time point frame.

[0140] Limiting the number of sub-meshes in different sub-bitstreams

[0141] In some embodiments, the difference in the number of sub-meshes in the respective sub-bitstreams for a given time point frame can be prevented by requirements such as the number of sub-meshes in all sub-bitstreams should be the same and the available sub-mesh IDs are the same. In some embodiments, the requirement can be that the number of sub-meshes in the atlas data sub-bitstream is always less than or equal to the number of sub-meshes in the base mesh sub-bitstream. In some embodiments, the available sub-mesh IDs in the atlas data sub-bitstream can be a subset of the available sub-mesh IDs in the base mesh sub-bitstream.

[0142] Allowing different numbers of sub-meshes in different sub-bitstreams

[0143] In some embodiments, if different numbers of sub-meshes and / or sub-mesh IDs of sub-meshes are allowed across the multiple sub-bitstreams of a frame, such differences can be handled in the reconstruction phase. For example, if a tile in the atlas data sub-bitstream indicates that it corresponds to a sub-mesh with a sub-mesh ID, but the sub-mesh with that sub-mesh ID does not exist in the base mesh sub-bitstream, the reconstruction process can output an empty sub-mesh.

[0144] Generating sub-meshes after decoding the base mesh sub-bitstream.

[0145] In some embodiments, when the number of sub-grids is different or the sub-grid Ids do not match, the encoder / decoder can indicate to generate the sub-grids from the reconstructed base grids from the base grid sub-bitstream. The bitstream can convey information about the number of base grids that need to be generated and / or the method to generate the base grids, such as connected components, and can provide a method to generate the sub-grid Ids.

[0146] In some embodiments, there can be multiple methods to generate the base grids. In some embodiments, a predefined method or method type can be signaled. In some embodiments, these methods can be signaled in a message. The number of base grids generated from one sub-grid base grid can be signaled or derived based on the number of sub-grids in the base grid sub-bitstream and / or the number of sub-grids in the atlas data sub-bitstream. The sub-grid Ids of the newly generated base grids of the other sub-grids can be explicitly signaled or can be derived based on the sub-grid Ids of the sub-grids in the base grid sub-bitstream and the sub-grid Ids of the sub-grids in the atlas data sub-bitstream.

[0147] Signaling multiple locations for a given tile or slice

[0148] In some embodiments, tiles or slices can be signaled differently for different uses within a given time frame. For example, a slice signaled for intra prediction of a current time frame in a current time frame can be different from another version of the same slice signaled in the same video frame at the current time, but this version will be used as a reference frame for predicting values of a subsequent time point frame. As another example, attribute values of a given slice packed into an attribute video image frame (e.g., an attribute slice) can be different from geometry slices (e.g., displacement values included in the slice and packed into a 2D image frame representing geometry information). In some cases using inter prediction, geometry slices can be different from attribute slices, e.g., because some geometry slices are predicted using inter prediction, while the corresponding attribute slices of the same time point frame are signaled or predicted differently, e.g., by using a different prediction technique.

[0149] In some embodiments, tiles of a video frame can be used to locate slices packed into the video frame. For example, multiple groups of slices can be included in one tile. In the atlas sub-bitstream, atlas information for locating tiles in a video frame, such as the locations of the corners of a tile and the size of the tile, can be signaled in the atlas sub-bitstream. Additionally, information for locating slices can be signaled in the atlas sub-bitstream, but can be formatted with reference to the tiles. For example, the starting locations of the corners of a tile’s bounding box can be referenced relative to a reference point of the tile (rather than a reference point of the larger video frame).

[0150] In some embodiments, syntax elements of the atlas sub-bitstream are used to locate the areas corresponding to tiles and patches in the 2D image frames used for geometry information and in other 2D image frames used for attribute information.

[0151] For example, the size and position of the larger box (e.g., tile box) is derived using the atlas frame tile information, the atlas frame attribute tile information, and the atlas tile ID. Then, the smaller box (e.g., patch bounding box) (e.g., size and position) can be derived using the patch information included in the atlas sub-bitstream that references the derivation of the larger tile box. For example, this is shown in FIG. 11A-11B

[0152] However, as mentioned above, in some cases, a given patch or tile can need to correspond to different parts of a video image frame. For example, the area covered by an attribute patch can be different from the area covered by a corresponding geometry patch, where the attribute patch and the geometry patch correspond to the same patch when reconstructed as a 3D version of a dynamic mesh to which the texture is applied. In this case, multiple image positions and sizes can be signaled for that same patch.

[0153] For example, the number of grid data units of a mapped area can be signaled as follows: mdu num mapped area [tileID] [patchIdx] and mdu num mapped area attribute [tileID] [patchIdx] [i]. This can be signaled in the patch, the atlas frame parameter set (AFPS), or the atlas frame sequence parameter set (ASPS). In some embodiments, only the number of mapped areas for attributes (or geometry shapes) can be signaled, and the other numbers (not signaled) are assumed to be the same as the signaled number. In some embodiments, if not signaled, the number of mapped areas per patch can be defaulted to “1”, so that only patches deviating from having 1 mapped area have a signaled number of mapped areas.

[0154] Below is an example grid data unit syntax as described above:

[0155]

[0156]

[0157] For example, FIG. 12 An example is shown where two datasets are signaled for geometry video and two datasets are signaled for attribute video. However, in the atlas tile layer, there is one patch that is different from FIG. 11A-11B

[0158] ​​In some embodiments, (1) atlas tile IDs, (2) atlas frame tile information syntax, and (3) atlas frame attribute tile information syntax are used to derive regions corresponding to tiles. In order for multiple regions of a video image frame to correspond to the same tile, position information (e.g., top-left partition index, column offset, and row offset) is signaled in the atlas frame tile information syntax for each tile. For example, the following is an example atlas frame attribute file information syntax:

[0159]

[0160]

[0161] In some embodiments, when multiple regions are signaled for a given tile, an indication (e.g., an index) can be included in the slice to indicate in which tile region the slice bounding box region resides.

[0162] Managing a set size mismatch between image size and texture coordinate size

[0163] Texture coordinates of a mesh indicate where in the image the texture for a given part of the mesh resides (which can be a fractional pixel position). In some cases, texture coordinates are normalized to be between 0 and 1. In order to map pixel positions and texture coordinates, the renderer needs the size of the video image. In V-DMC, the texture coordinates of the output mesh can be normalized based on the nominal texture image.

[0164] In V-DMC, the texture coordinates of all sub-meshes in the base mesh sub-bitstream can be normalized within the nominal size of the image. FIG. 13 This case is depicted on the right side of FIG. 7. For all sub-meshes in the base mesh sub-bitstream, the texture coordinates of the sub-mesh (0, 0) indicate the lower left of the image and (1, 1) indicate the upper right of the image.

[0165] In another embodiment, the texture coordinates of a sub-mesh (0, 0) indicate the lower left of the region indicated in the corresponding slice and (1, 1) indicate the upper right of the region indicated in the corresponding slice, as shown on the upper left side of FIG. 8. FIG. 13

[0166] ​For example, this can be signaled as asve_attribute_subtexture_enabled_flag[attIdx] signaled for the sth attldx attribute indicates this difference. When asve_attribute_subtexture_enabled_flag[attldx] is true, it is assumed that the texture coordinates of the sub-mesh are based on the image size corresponding to the tile and then the texture coordinate adjustment process is invoked to renormalize the texture coordinates based on the frame size of the V-DMC output mesh. For example, for the V-DMC output mesh, the texture coordinate (0,0) of sub-mesh 0 is renormalized to (0,0.5).

[0167] However, in some embodiments, as FIG. 13 shown on the left lower side of FIG. 6, the texture coordinates (0,0) of a sub-mesh indicate the left lower side of the region indicated in the corresponding tile and (1,1) indicate the right upper side of the region indicated in the corresponding tile, as FIG. 13 shown on the left side of FIG. 7.

[0168] When asve_attribute_subtexture_enabled_flag[attldx] is false, it is assumed that the texture coordinates of the sub-mesh are based on the image size corresponding to the tile, which then invokes the texture coordinate adjustment process to renormalize the texture coordinates based on it.

[0169] In another embodiment, each tile can have its own indication equivalent to asve_attribute_subtexture_enabled_flag[attldx] for the texture coordinate adjustment process.

[0170] Reusing signaled tile location information

[0171] For example, the attribute information can be signaled per attribute using the syntax shown as follows:

[0172]

[0173] If multiple attributes have the same tile information, the tile information can be signaled once, for example, as described below. For the sth attldx attribute, the tile information can be copied from the ath attribute. In this case, a is between 0 and afve_num_attributes_info - 1, inclusive.

[0174] In another embodiment, an extended range of afve_reference_attribute_idx can be utilized to copy tile information of a geometry.

[0175] In another embodiment, a set of {tileOffsetX, tileOffsetY, tileWidth and tileHeight} of a reference attribute can be copied instead of a single syntax element.

[0176] In another embodiment, a ratio between tileOffsetX and a nominal frame width of a reference attribute tile, a ratio between tileOffsetY and a nominal frame height of a reference attribute tile, a ratio between tileWidth and a nominal frame width of a reference attribute tile, and a ratio between tileHeight and a nominal frame width of a reference attribute tile can be utilized to derive a set of {tileOffsetX, tileOffsetY, tileWidth and tileHeight} of a current attribute.

[0177]

[0178] In another embodiment, if attribute information in a tile can be shared by multiple attributes, it can be signaled once.

[0179] For iB between afve_num_attributes_info and asve_num_attribute_video - 1, inclusive, the syntax elements can be derived as follows:

[0180] • mdu_attributes_2d_pos_x[tileID][patchIdx][iB] = mdu_attributes_2d_pos_x[tileID][patchIdx][mdu_Reference_attribute_idx[tileID][patchIdx]]

[0181] • mdu_attributes_2d_pos_y[tileID][patchIdx][iB] = mdu_attributes_2d_pos_y[tileID][patchIdx][mdu_Reference_attribute_idx[tileID][patchIdx]]

[0182] • mdu_attributes_2d_size_x_minusl [ tileID ] [ patchld ] [ iB ] = mdu_attributes_2d_size_x_minusl [ tileID ] [ patchld ] [ mdu_Reference_attribute_idx [ tileID ] [ patchld ] ]

[0183] • mdu_attributes_2d_size_y_minusl [ tileID ] [ patchld ] [ iB ] = mdu_attributes_2d_size_y_minusl [ tileID ] [ patchld ] [ mdu_Reference_attribute_idx [ tileID ] [ patchld ] ]

[0184]

[0185] A syntax element afve_num_attributes_info indicating the number of attributes whose information is signaled can be signaled in an AFPS (extension) or in a tile.

[0186] Example of partial signaling of a tile data unit

[0187] In some embodiments, a tile data unit can be used to signal information of volumetric data, where the reconstructed tile will be placed in 3D form. In addition, decoding and reconstruction information can be signaled in the tile data unit. For example, the following is an example tile data unit.

[0188]

[0189]

[0190] In some embodiments, a tile data unit can have one or more flags set to indicate that some of the syntax elements are not currently signaled in the tile data unit. For example, the following is an example syntax with flags set.

[0191]

[0192]

[0193] For example, mdu_vertex_process_information_present_flag[ tileID ][ patchldx ] equal to 1 indicates that some of the syntax elements related to the processing of vertex positions or texture coordinates are present in the patch data unit. mdu_attribute_image_information_present_flag[ tileID ][ patchldx ] equal to 1 indicates that some of the syntax elements related to the corresponding region in the attribute image are present in the patch data unit. For this case, at least one of the flags needs to be equal to 1.

[0194] When mdu_vertex_process_information_present_flag[ tileID ][ patchldx ] is equal to 0, the syntax elements not present in the bitstream are set to their respective default values. For example, if mdu_2d_pos_x[ tileID ][ patchldx ] and mdu_2d_pos_y[ tileID ][ patchldx ] are set to 0, mdu_2d_size_x_minusl [ tileID ][ patchldx ] and mdu_2d_size_y_minusl [ tileID ][ patchldx ] can be set by the width and height of the corresponding tile. And in this case, mdu_parameters_override_flag is set to 0.

[0195] When mdu_attribute_image_information_present_flag[ tileID ][ patchldx ] is equal to 0, asve_attribute_subtexture_enabled_flag[ i ] is inferred to be 0.

[0196] In another embodiment, mdu_vertex_count_minusl and mdu_face_count_minusl are always signaled. For example, the following is an example syntax.

[0197]

[0198]

[0199] In another embodiment, mdu_vertex_process_information_present_flag[ tileID ][ patchIdx ] indicates that there are 4 syntax elements, e.g., mdu_2d_pos_x[ tileID ][ patchIdx ], mdu_2d_pos_y[ tileID ][ patchIdx ], mdu_2d_size_x_minus1[ tileID ][ patchIdx ], and mdu_2d_size_y_minus1[ tileID ][ patchIdx ]. For example:

[0200]

[0201] In another embodiment, mdu_vertex_process_information_present_flag[ tileID ][ patchIdx ] can be controlled by another flag in atlas sequence parameter set (ASPS) that is restricted by the profile indicated in video parameter set (VPS). For example, the flag can be signaled in ASPS extension, and can be set to 0 when using AC coded displacement bitstream. Then, mdu_vertex_process_information_present_flag[ tileID ][ patchIdx ] is signaled based on the flag.

[0202] In another embodiment, instead of signaling mdu_vertex_process_information_present_flag[ tileID ][ patchIdx ], asve_geometry_image_information_present_flag can be used directly.

[0203] The process can be described as follows:

[0204]

[0205]

[0206]

[0207]

[0208] In another embodiment, mdu_attribute_image_information_present_flag[ tileID ][ patchIdx ] can be signaled per attribute.

[0209] When mdu_vertex_process_information_present_flag and mdu_attribute_image_information_present_flag are signaled, there can be multiple patches in one tile that correspond to the same submesh, which means that mdu_submesh_id[tileID][patchIdx] can be the same for multiple patches. In this case, their mdu_vertex_process_information_present_flag and mdu_attribute_image_information_present_flag are mutually exclusive. For example, if mdu_vertex_process_information_present_flag is true for one patch, mdu_vertex_process_information_present_flag is false for the other patch.

[0210] In another embodiment, the vertices reconstructed using the information in the patches can be connected to create one submesh.

[0211] When mdu_vertex_process_information_present_flag and mdu_attribute_image_information_present_flag are signaled, inter prediction tile data units can refer to only those patches that have the same mdu_vertex_process_information_present_flag and mdu_attribute_image_information_present_flag as the current patch.

[0212] In another embodiment, inter prediction patch data units can refer to any patch.

[0213] Example intra encoder

[0214] In some embodiments, the intra-frame encoder 1402 receives the base mesh, displacements, original static / dynamic mesh, and attribute maps for multiple time point frames. The base mesh is provided to a quantization module 1404, where aspects of the base mesh can be (optionally) further quantized. In some embodiments, various mesh encoders can be used to encode the base mesh. Additionally, in some embodiments, the intra-frame encoder 1402 can allow for customization, where different respective mesh encoding schemes can be used to encode the base mesh. For example, the base mesh encoder 1406 can be a selected mesh encoder chosen from a set of possible mesh encoders, such as a DRACO encoder (or another suitable encoder). The encoded base mesh, which has been encoded by the base mesh encoder 1406, is provided to a multiplexer (MUX) 1438 for inclusion in the compressed bitstream. Additionally, the encoded base mesh is provided to a base mesh decoder in order to generate a reconstructed version of the base mesh (as seen by the decoder). This reconstructed version of the base mesh is used to determine the displacements so that they account for any geometric distortion between the original base mesh and the reconstructed version of the base mesh (as seen by the decoder). For example, the base mesh decoder 1408 generates a reconstructed quantized base mesh and provides this reconstructed quantized base mesh to a displacement update module 1410, which also receives the original base mesh and original displacements. The displacement update module 1410 compares the reconstructed quantized base mesh (as seen by the decoder) to the base mesh and adjusts the displacements to account for the difference between the base mesh and the reconstructed quantized base mesh. These updated displacements are provided to a wavelet transform 1412, which applies a wavelet transform to further compress the updated displacements and output wavelet coefficients, which are provided to a quantization module 1414, which generates quantized wavelet coefficients. The quantized wavelet coefficients can then be packed into 2D image frames via an image packing module 1416, where the packed 2D image frames are further video encoded via a video encoding 1418. The encoded video images are also provided to the multiplexer (MUX) 1438 for inclusion in the compressed bitstream. Additionally, in some embodiments, the displacement values (such as indicated in the generated quantized wavelet coefficients, or indicated using other compression schemes) can be at least partially encoded outside of the video sub-bitstream, such as in their own displacement data sub-bitstream, in the base mesh sub-bitstream, or in the atlas data sub-bitstream.

[0215] Additionally, in order to account for any geometric distortion introduced with respect to the original static / dynamic mesh, an attribute transfer process 1430 can be used to modify the attributes to account for the difference between the reconstructed deformed mesh and the original static / dynamic mesh.

[0216] For example, the video encoding 1418 can further perform video decoding (or a complementary video decoding module (not shown) can be used) to generate the reconstructed base mesh (as seen by the decoder) and the reconstructed deformed mesh (as seen by the decoder). The video encoding 1418 can then generate the updated displacements by comparing the reconstructed base mesh (as seen by the decoder) to the reconstructed deformed mesh (as seen by the decoder). FIG. 14The reconstructed packed quantized wavelet coefficients are de-packed via image de-pack module 1420. In addition, inverse quantization can be applied via inverse quantization module 1422, and an inverse wavelet transform 1424 can be applied to generate reconstructed displacements. In some embodiments, other decoding techniques can be used to generate reconstructed displacements, such as decoding the displacements signaled in the atlas data sub-bitstream, the displacement data sub-bitstream, or the base mesh sub-bitstream. In addition, reconstruction of the base mesh 1409 by the static mesh decoder 1408 to generate the reconstructed quantized base mesh can be inverse quantized via inverse quantization module 1428 to generate a reconstructed base mesh. The reconstructed deformed mesh generation module 1426 applies the reconstructed displacements to the reconstructed base mesh to generate a reconstructed deformed mesh. Note that the reconstructed deformed mesh characterizes the reconstructed mesh that the decoder will generate and accounts for any geometric deformations resulting from the losses introduced in the encoding process.

[0217] The attribute transfer module 1430 compares the geometry of the original static / dynamic mesh to the reconstructed deformed mesh and updates the attribute map to account for any geometric deformations, the updated attribute map being output as an updated attribute map. The updated attribute map is then padded, where padding includes 2D images of the attribute images such that the space not used to convey the attribute images has padding applied. In some embodiments, optionally, a color space conversion is applied at color space conversion module 1434. For example, an RGB color space used to characterize the color values of the attribute images can be converted to a YCbCr color space, color space sub-sampling such as 10:2:0, 10:0:0, etc. can also be applied. The updated attribute map that has been padded and optionally color space converted is then video encoded via video encoding module 1436 and provided to multiplexer 1438 for inclusion in the compressed bitstream.

[0218] In some embodiments, the controller 1400 can coordinate the various quantization and inverse quantization steps as well as the video encoding and decoding steps such that the inverse quantization "undoes" the quantization and such that the video decoding "undoes" the video encoding. In addition, the attribute transfer module 1430 can account for the level of quantization being applied based on communications from the controller 1400.

[0219] Example intra decoder

[0220] FIG. 15 An example intra decoder for decoding a three-dimensional mesh is illustrated in accordance with some embodiments.

[0221] The intra decoder 1502 receives a compressed bitstream, such as the compressed bitstream generated by the encoder 1500, and decodes the compressed bitstream to generate a reconstructed mesh. The intra decoder 1502 includes a static mesh decoder 1504, a deformed mesh decoder 1506, and an attribute transfer module 1508. The static mesh decoder 1504 decodes the static mesh data sub-bitstream to generate a reconstructed static mesh. The deformed mesh decoder 1506 decodes the deformed mesh data sub-bitstream to generate a reconstructed deformed mesh. The attribute transfer module 1508 compares the geometry of the original static / dynamic mesh to the reconstructed deformed mesh and updates the attribute map to account for any geometric deformations, the updated attribute map being output as an updated attribute map. The updated attribute map is then padded, where padding includes 2D images of the attribute images such that the space not used to convey the attribute images has padding applied. In some embodiments, optionally, a color space conversion is applied at color space conversion module 1510. For example, an RGB color space used to characterize the color values of the attribute images can be converted to a YCbCr color space, color space sub-sampling such as 10:2:0, 10:0:0, etc. can also be applied. The updated attribute map that has been padded and optionally color space converted is then video encoded via video encoding module 1512 and provided to multiplexer 1514 for inclusion in the compressed bitstream. FIG. 14The intra-frame encoder 1402 generates a compressed bitstream as shown. A demultiplexer (DEMUX) 1504 parses the bitstream into base mesh sub-components, displacement sub-components, and attribute map sub-components. In some embodiments, the displacement sub-components can be signaled in a displacement data sub-bitstream, and can also be at least partially signaled in other sub-bitstreams such as an atlas data sub-bitstream, a base mesh sub-bitstream, or a video sub-bitstream. In this case, a displacement decoder 1522 decodes the displacement sub-bitstream, and / or an atlas decoder 1524 decodes the atlas sub-bitstream.

[0222] A base mesh decoder 1506 decodes the base mesh sub-components to generate a reconstructed quantized base mesh, which is provided to an inverse quantization module 1518, which in turn outputs and provides a decoded base mesh to a reconstructed morphed mesh generator 1520.

[0223] In some embodiments, a portion of the displacement sub-components of the bitstream are provided to a video decode 1508, where video encoded image frames are video decoded and provided to an image unpack 1510. The image unpack 1510 extracts packed displacements from the video decoded image frames and provides them to an inverse quantization 1512, where the displacements are inverse quantized. Additionally, the inverse quantized displacements are provided to an inverse wavelet transform 1514, which outputs decoded displacements. The reconstructed morphed mesh generator 1520 applies the decoded displacements to the decoded base mesh to generate a reconstructed static / dynamic mesh. The decoded displacements can come from any combination of a video sub-bitstream, an atlas data sub-bitstream, a base mesh sub-bitstream, and / or a displacement data sub-bitstream. Additionally, the attribute map sub-components are provided to a video decode 1516, which outputs a decoded attribute map. A reconstructed version of the three-dimensional visual content can then be presented at a device associated with the decoder using the decoded mesh and the decoded attribute map.

[0224] As FIG. 15 shown, the bitstream is demultiplexed into three or more separate substreams:

[0225] • a mesh substream;

[0226] • a displacement substream for positions and possibly for each vertex attribute; and

[0227] • an attribute map substream for each attribute map.

[0228] The grid sub-stream is fed to a grid decoder to generate the reconstructed quantized base grid. Then, the decoded base grid is obtained by applying inverse quantization. The proposed scheme is not subject to which grid codec is used. The grid codec used can be explicitly specified in the bitstream or can be implicitly defined / fixed by the specification or application.

[0229] The displacement sub-stream can be decoded by a video / image decoder. Then, the generated image / video is de-packed and inverse quantization is applied to the wavelet coefficients. In an alternative embodiment, the displacement can be decoded by a dedicated displacement data decoder or atlas decoder. The proposed scheme is not subject to which codec / standard is used. An image / video codec such as [HEVC] [AVC] [AV1] [AV2] [JPEG] [JPEG2000] can be used. A motion decoder for decoding the grid motion information or a dictionary based decoder such as ZIP can be used as a dedicated displacement data decoder, for example. Then, the decoded displacement is generated by applying an inverse wavelet transform to the unquantized wavelet coefficients. The final decoded grid is generated by applying the reconstruction process to the decoded base grid and adding the decoded displacement field.

[0230] The attribute sub-stream is directly decoded by a video decoder and a decoded attribute map is generated as output. The proposed scheme is not subject to which codec / standard is used. An image / video codec such as [HEVC] [AVC] [AV1] [AV2] [JPEG] [JPEG2000] can be used. Alternatively, the attribute sub-stream can be decoded by a non-image / video decoder (e.g., using a dictionary based decoder such as ZIP). Multiple sub-streams can be decoded, each of which is associated with a different attribute map. Each sub-stream can use a different codec.

[0231] Example inter encoder

[0232] FIG. 16 An example inter-frame encoder is illustrated in accordance with some embodiments. Note that a given device can implement both intra-frame and inter-frame encoding, and can select intra-frame or inter-frame encoding for different respective time point frames based on which method provides better compression efficiency, quality, etc.

[0233] In some embodiments, inter-frame encoder 1602 can include similar components as intra-frame encoder 1402, but the inter-frame encoder does not encode the base mesh, but can encode motion vectors that can be applied to the reference mesh to generate the base mesh at the decoder. Further, as described above, in some embodiments, the displacements can not be explicitly signaled for each time point frame, but instead, inter-frame prediction can be used to predict the displacements based on the reference frame. In such embodiments, the residual of the vertex displacements can be grouped into patches, packed into 2D video image frames and video encoded.

[0234] For example, in the case of dynamic meshes, a temporally consistent re-meshing process is used, which can result in the same subdivision structure being shared by the current mesh and the reference mesh. Such coherent temporal re-meshing process makes it possible to skip the encoding of the base mesh and reuse the base mesh associated with the reference frame. This can also enable better temporal prediction for both attribute and geometry information. To be more precise, a motion field can be computed and encoded that describes how to move the vertices of the base mesh to match the positions of the vertices in the base mesh at the current time instant. FIG. 16 Such a process is described in the background. For example, motion encoder 1606 can generate a motion field that describes how to move the vertices of the reference base mesh to match the positions of the base mesh at the current time instant.

[0235] In some embodiments, the base mesh associated with the current frame is first quantized (e.g., using uniform quantization) and encoded by the static mesh encoder. The proposed scheme is not subject to which mesh codec is used. The mesh codec used can be explicitly specified in the bitstream by encoding a mesh codec ID, or can be implicitly defined / fixed by the specification or application.

[0236] Depending on the application and target bit rate / visual quality, optionally, the encoder can encode a set of displacement vectors associated with the vertices of the subdivision mesh, referred to as a displacement field.

[0237] In some embodiments, the motion decoder 1608 performs reconstruction of the base grid 1409 and then uses the reconstructed quantized base grid (e.g., the reconstruction output of the base grid 1409) to update the displacement field (at the update displacement module 1410) to generate an updated displacement field so that it accounts for the differences between the reconstructed base grid and the original base grid. A wavelet transform is then applied at wavelet transform 1412, utilizing the subdivision surface grid structure, and a set of wavelet coefficients is generated. The wavelet coefficients are then quantized at quantization 1414, packed into a 2D image / video (at image packing 1416), and compressed by an image / video encoder (at video encoding 1418). The encoding of the wavelet coefficients can be lossless or lossy. A reconstructed version of the wavelet coefficients is obtained by applying image unpacking and inverse quantization to the reconstructed wavelet coefficient video generated during the video encoding process (e.g., at 1420, 1422, and 1424). The reconstructed displacement is then calculated by applying an inverse wavelet transform to the reconstructed wavelet coefficients. The reconstructed base mesh is obtained by applying inverse quantization to the reconstructed quantized base mesh. The reconstructed deformed mesh is obtained by subdividing the updated base mesh and applying the reconstructed displacements to its vertices (eg, subdivided positions).

[0238] Since the quantization step or / and the mesh compression module may be lossy, a reconstructed quantized version of the base mesh is computed.

[0239] like FIG. 16 As shown, the reconstructed quantized reference base grid is used to predict the current frame base grid.

[0240] Example inter decoder

[0241] FIG. 17 An example inter-frame decoder for decoding a three-dimensional mesh according to some embodiments is illustrated.

[0242] Interframe decoder 1702 includes FIG. 15 The inter-frame decoder 1702 is similar to the intra-frame decoder 1502 shown in FIG. However, instead of receiving a directly encoded base grid, the inter-frame decoder 1702 reconstructs the base grid of the current frame based on the motion vectors of the displacement field relative to the reference frame. For example, the inter-frame decoder 1702 includes a motion field / vector decoder 1704 and a reconstruction of the base grid module 1706.

[0243] In a similar manner to the intra decoder, the inter decoder 1702 splits the bitstream into three separate substreams:

[0244] Movement sub-flow;

[0245] · displacement sub-stream; and

[0246] Attribute subflow.

[0247] The motion sub-stream is decoded by applying the motion decoder 1704. The proposed scheme is not subject to which codec / standard is used to decode the motion information. For example, any motion decoding scheme can be used. Optionally, the decoded motion is then added to the decoded reference quantized base mesh to generate the reconstructed quantized base mesh, i.e., the mesh that has been decoded at index j is available for the prediction of the mesh at frame index i. The decoded base mesh is then generated by applying the inverse quantization.

[0248] Example multi-frame inter-prediction encoder

[0249] The displacement sub-stream and attribute sub-stream are decoded in a similar manner as the intra decoding process described with respect to FIG. 15 The decoded mesh is also reconstructed in a similar manner.

[0250] FIG. 18 An example multi-hypothesis inter-encoder is illustrated in accordance with some embodiments.

[0251] In some embodiments, the multi-hypothesis inter-encoder 1802 can include similar components as the inter-encoder 1602, but can apply multi-hypothesis inter prediction as discussed above in FIG. 1-8 For example, the multi-hypothesis encoder 1806 does not use a single reference base mesh, but can perform multi-hypothesis inter prediction (as described above in FIG. 1-8 ) to generate the base mesh at the decoder. In addition, the multi-hypothesis decoder 1808 generates the reconstructed quantized base mesh based on multiple reference meshes (or multiple vertices from a single reference mesh as described above in FIG. 4 ) and provides the reconstructed quantized base mesh m'(i) to the displacement update module 1410, which also receives the original base mesh and original displacement. In some embodiments, the multi-hypothesis encoder 1806 can provide a displacement multiplexer 1438 for inclusion in the compressed bitstream b(i). In some embodiments, a multi-hypothesis predictor can be provided to the multiplexer 1438 for inclusion in the compressed bitstream. The multi-hypothesis texture coordinate predictor can indicate one or more previous time point frames from which texture coordinate values or texture connectivity information should be obtained, as described in FIG. 7-8

[0252] Example multi-frame inter-prediction decoder

[0253] FIG. 19 An example multi-hypothesis decoder for decoding a three-dimensional mesh is illustrated in accordance with some embodiments.

[0254] The multi-hypothesis decoder 1902 includes similar components as the multi-hypothesis decoder 1702, but can apply multi-hypothesis inter prediction as discussed above in FIG. 17 ​The illustrated inter decoder 1702 similar components. However, rather than reconstructing the base mesh of a given indexed frame based on motion vectors of displacement fields relative to a reference frame, the multi-hypothesis decoder 1902 reconstructs the base mesh of a given indexed frame based on reference meshes from different frames at one or more indices using the multi-hypothesis techniques described in FIG. 1-8

[0255] FIG. 20 is a flowchart illustrating an example process of reconstructing a dynamic mesh using inter prediction, in accordance with some embodiments.

[0256] At block 2002, the decoder receives information for a dynamic mesh (e.g., information that has been compressed using inter prediction). The received information includes displacement values and vertex connectivity information for a first time point frame. For example, this can be signaled using a base mesh and displacements to be applied to the subdivision positions of the base mesh. In addition, the received information includes prediction information to be used to predict displacements and / or connectivity for another time point frame.

[0257] At block 2004, the decoder determines vertex values for a first time point frame version of the dynamic mesh. For example, by using inter prediction or intra prediction as described above with respect to the example intra decoder illustrated in FIG. 15 or FIG. 17

[0258] At block 2006, the decoder predicts vertex values for other (e.g., subsequent) time point frames. This is performed using the vertex values determined for the first time point frame and using the prediction information received at block 2002. In some embodiments, a multi-frame inter prediction process is described in FIG. 19 and FIG. 1-8 In some embodiments, the displacement values for another time point can be inter predicted from the displacements signaled in the first time point frame.

[0259] At block 2008, the decoder applies residual values (e.g., residual values that have been grouped into patches and packed into 2D video image frames) to the vertex values predicted at block 2006. The dynamic mesh version corresponding to the other time point frame is then reconstructed using the vertex values to which the residuals have been applied.

[0260] FIG. 21 is a flowchart illustrating an example process of reconstructing a dynamic mesh, in accordance with some embodiments, in which at least some sub-meshes have base meshes that are omitted from the base mesh sub-bitstream of at least some time point frames.

[0261] ​​At block 2102, the decoder receives information for a dynamic mesh (e.g., information that has been compressed using interframe prediction). The received information includes a base mesh sub-bitstream, a displacement sub-bitstream (e.g., including video encoded images that pack 2D frames with tiles, each tile including information for determining displacement values to apply to base mesh tessellation locations to locate vertices in 3D space), and an atlas sub-bitstream including information for locating tiles from the displacement sub-bitstream and applying them to the correct corresponding tessellation locations of the base mesh of the base mesh sub-bitstream. In some embodiments, the dynamic mesh can be tessellated into sub-meshes, and for each sub-mesh, the base mesh, displacement, and atlas information is signaled in a respective sub-bitstream.

[0262] At block 2104, the decoder identifies a sub-mesh for which atlas information is signaled in the atlas sub-bitstream but for which a corresponding base mesh is not signaled in the base mesh sub-bitstream for at least one time point frame.

[0263] At block 2106, to resolve the mismatch across sub-bitstreams, the decoder inserts an empty base mesh for the sub-mesh into the base mesh sub-bitstream for the at least one time frame point for which the corresponding base mesh for the sub-mesh is missing. This preserves the ordering of time point frames of the respective sub-bitstream.

[0264] Alternatively, at block 2108, the decoder removes the sub-mesh from the atlas sub-bitstream for at least one time point frame to preserve the ordering of time point frames of the respective sub-bitstream. For example, in some cases, the missing sub-mesh can be predicted using interframe prediction such that the base mesh does not need to be signaled for a given time point frame.

[0265] At block 2110, the decoder then determines vertex values for the dynamic mesh for a plurality of time instants, including time instants for which the base mesh is omitted for at least one sub-mesh. The dynamic mesh is reconstructed (e.g., preserving the ordering of time point frames of the respective sub-bitstream) using information from the base mesh sub-bitstream, the displacement sub-bitstream, and the atlas sub-bitstream.

[0266] FIG. 22 is a flowchart illustrating an example process of reconstructing a dynamic mesh in which vertex counts are signaled per tile, according to some embodiments.

[0267] At block 2202, a decoder receives information for a dynamic mesh (e.g., information compressed using inter-frame prediction). The received information includes a base mesh sub-bitstream, a displacement bitstream, and an atlas sub-bitstream. The displacement sub-bitstream is signaled using video-encoded 2D images, wherein corresponding ones of the 2D images include packed tiles, and wherein the packed tiles include residual values ​​used in reconstructing displacement values ​​of corresponding vertices of the dynamic mesh at corresponding points in time. Furthermore, the received information includes information for determining a vertex count for each of the tiles (e.g., enabling independent determination of the vertex counts of corresponding tiles).

[0268] At block 2204 , the decoder determines vertex values ​​for the dynamic mesh using the tiles and the signaled vertex counts of the corresponding tiles.

[0269] Embodiments of the present disclosure may be described in terms of:

[0270] 1. A non-transitory computer-readable storage medium storing program instructions that, when executed using one or more computing devices, cause the one or more computing devices to:

[0271] Receive information about a compressed version of a dynamic mesh, the information comprising:

[0272] Vertex positions and connectivity information of the first frame;

[0273] vertex position and connectivity information for one or more additional frames; and

[0274] multi-frame prediction information for predicting vertex position information for another frame using a plurality of previous frames and an indication of vertex connectivity information to be used for the another frame;

[0275] determining 3-dimensional (3D) vertex values ​​for a first version of the dynamic mesh corresponding to the first frame using the vertex positions and connectivity information of the first frame;

[0276] using the vertex positions and connectivity information of the one or more additional frames to determine 3D vertex values ​​for one or more additional versions of the dynamic mesh corresponding to the one or more additional frames; and

[0277] Determine another version of the 3D top view of the dynamic mesh corresponding to the other frame

[0278] Point value, wherein said determining comprises:

[0279] For one or more vertices of the another version of the dynamic mesh, one or more vertex positions are predicted using previously determined 3D vertex values ​​from at least two different ones of the dynamic meshes corresponding to the first frame and the one or more additional frames.

[0280] 2. The non-transitory computer-readable storage medium of clause 1, wherein the vertex positions of the first frame and / or the vertex positions of the one or more additional frames are determined using inter-frame prediction and / or intra-frame prediction.

[0281] 3. The non-transitory computer-readable storage medium of clause 1, wherein the program instructions, when executed using the one or more computing devices, further cause the one or more computing devices to:

[0282] receive other information of the compressed version of the dynamic mesh, the other information including:

[0283]

[0284] single-reference frame multi-vertex prediction information for predicting vertex position information of a third frame using a plurality of vertices of a single previous frame; and

[0285] determine 3D vertex values of a third version of the dynamic mesh corresponding to the third frame, wherein the determining includes:

[0286]

[0287] for one or more vertices of the third version of the dynamic mesh, predicting one or more vertex positions using at least two different vertices from the single previous frame.

[0288] 4. The non-transitory computer-readable storage medium of clause 1, wherein the program instructions, when executed using the one or more computing devices, further cause the one or more computing devices to:

[0289] determine one or more texture coordinate values of the other version of the dynamic mesh corresponding to the other frame, wherein the determining includes:

[0290] for one or more texture coordinates of the other version of the dynamic mesh, predicting the one or more texture coordinate values using texture coordinate values from at least two different versions of the dynamic mesh corresponding to the first frame and the one or more additional frames.

[0291] 5. The non-transitory computer-readable storage medium of clause 4, wherein predicting the one or more texture coordinate values further includes taking an average of the coordinate values from the at least two different versions of the dynamic mesh corresponding to the first frame and the one or more additional frames.

[0292] 6. The non-transitory computer-readable storage medium of clause 1, wherein:

[0293] ​​The received information of the compressed version of the dynamic mesh includes an indication of one or more previous frames preceding the further frame to be used to determine one or more of:

[0294] vertex connectivity information of the other version of the dynamic mesh corresponding to the other frame,

[0295] texture coordinate values ​​of the other version of the dynamic mesh corresponding to the other frame, or

[0296] texture connectivity information of the other version of the dynamic mesh corresponding to the other frame; and

[0297] The program instructions, when executed using the one or more computing devices, also cause

[0298] the one or more computing devices:

[0299] The vertex connectivity information, the texture coordinate values, or the texture connectivity information of the other version of the dynamic mesh is determined based on the indication of the one or more previous frames.

[0300] 7. The non-transitory computer-readable storage medium of clause 1, wherein the program instructions, when executed using the one or more computing devices, further cause the one or more computing devices to:

[0301] The mesh connectivity information, texture coordinate values ​​or texture connectivity information of the other version of the dynamic mesh is determined based on a pre-established rule, wherein the pre-established rule indicates that the mesh connectivity information, texture coordinate values ​​or texture connectivity of the dynamic mesh corresponding to a previous frame that is spatially or temporally closest to the other frame is to be used as the vertex connectivity information, the texture coordinate values ​​or the texture connectivity information of the other version of the dynamic mesh.

[0302] 8. The non-transitory computer-readable storage medium of clause 1, wherein the information of the compressed version of the dynamic mesh comprises:

[0303] A base grid sub-bitstream for reconstructing a base grid of the dynamic grid; and

[0304] a displacement sub-bitstream comprising information of vectors to be applied at subdivision points of the base mesh to reconstruct the dynamic mesh; and

[0305] The vertex positions of the first frame, the vertex positions of one or more additional frames, and multi-frame prediction information for predicting vertex position information are encoded in the base grid sub-bitstream and / or the displacement sub-bitstream.

[0306] 9. The non-transitory computer-readable storage medium of clause 8, wherein the information of the compressed version of the dynamic mesh comprises:

[0307] atlas sub-bitstream for reconstructing atlas information of the dynamic mesh;

[0308] wherein the program instructions, when executed using the one or more computing devices, further cause the one or more computing devices to:

[0309] determine a plurality of sub-meshes that are signaled in an atlas sub-bitstream but not signaled in a base mesh sub-bitstream; and based on the determination:

[0310] use an empty sub-mesh as part of the base mesh sub-bitstream to correspond to a sub-mesh referenced in the atlas sub-bitstream; or remove the sub-mesh referenced in the atlas sub-bitstream from the atlas sub-bitstream.

[0311] 10. The non-transitory computer-readable storage medium of clause 1, wherein the program instructions, when executed using the one or more computing devices, further cause the one or more computing devices to:

[0312] determine one or more predictors for predicting the one or more vertices of the other version of the dynamic mesh corresponding to the other frame, wherein the determination comprises:

[0313] predicting the one or more predictors using:

[0314] at least two different predictors corresponding to the first frame and the one or more additional frames;

[0315] single reference frame multi-vertex prediction;

[0316] single reference frame inter-frame prediction; or

[0317] intra-frame prediction.

[0318] 11. The non-transitory computer-readable storage medium of clause 1, wherein the program instructions, when executed using the one or more computing devices, further cause the one or more computing devices to:

[0319] determine one or more residuals for correcting predicted one or more vertex positions of the other version of the dynamic mesh to determine the 3D vertex values of the other version of the dynamic mesh, wherein the determination of the one or more residuals comprises:

[0320] predicting the one or more residuals using:

[0321] at least two different residuals corresponding to the first frame and the one or more additional frames;

[0322] single reference frame multi-vertex prediction;

[0323] single reference frame inter-frame prediction; or

[0324] intra-frame prediction.

[0325] 12. The non-transitory computer-readable storage medium of clause 1, wherein the multi-frame prediction information for predicting the vertex position information of the other frame comprises one or more functions for predicting the one or more vertices of the other version of the dynamic mesh, wherein the one or more functions indicate different weights to be applied to previously determined 3D vertex values from the at least two different dynamic meshes of the dynamic mesh corresponding to the first frame and the one or more additional frames, wherein the different weights correspond to temporal distances between the respective frames and the other frame.

[0326] 13. A non-transitory computer-readable storage medium storing program instructions that, when executed using one or more computing devices, cause the one or more computing devices to:

[0327] compress 3D vertex values of a first frame, one or more additional frames, and another frame of a dynamic mesh, wherein the compressing comprises:

[0328] for the other version of the dynamic mesh, use position information from at least two different dynamic meshes in the dynamic mesh corresponding to the first frame and the one or more additional frames to predict one or more 3D vertex values of the other version of the dynamic mesh; and

[0329] determine residual values for the other version of the dynamic mesh based on the predicted one or more 3D vertex values and one or more uncompressed 3D vertex values; and

[0330] signal information of a compressed version of a dynamic mesh, the information comprising:

[0331] vertex position and connectivity information for the first frame;

[0332] vertex position and connectivity information for the one or more additional frames;

[0333] and

[0334] the determined residual values.

[0335] 14. The non-transitory computer-readable storage medium of clause 13, wherein the signaled information further comprises:

[0336] Multi-frame prediction information for predicting vertex position information for another frame using a plurality of previous frames and an indication of vertex connectivity information to be used for the another frame.

[0337] 15. The non-transitory computer-readable storage medium of clause 13, wherein the signaled information further comprises:

[0338] Prediction information for predicting one or more residuals using at least two different residuals corresponding to the first frame and the one or more additional frames.

[0339] 16. The non-transitory computer-readable storage medium of clause 13, wherein the signaled information further comprises:

[0340] Prediction information for predicting one or more predictors using at least two different predictors corresponding to the first frame and the one or more additional frames.

[0341] 17. A method comprising:

[0342] Receive information about a compressed version of a dynamic mesh, the information comprising:

[0343] Vertex positions and connectivity information of the first frame;

[0344] vertex position and connectivity information for one or more additional frames; and

[0345] multi-frame prediction information for predicting vertex position information for another frame using a plurality of previous frames and an indication of vertex connectivity information to be used for the another frame;

[0346] determining 3-dimensional (3D) vertex values ​​for a first version of the dynamic mesh corresponding to the first frame using the vertex positions and connectivity information of the first frame;

[0347] using the vertex positions and connectivity information of the one or more additional frames to determine 3D vertex values ​​for one or more additional versions of the dynamic mesh corresponding to the one or more additional frames; and

[0348] Determine another version of the 3D top view of the dynamic mesh corresponding to the other frame

[0349] Point value, wherein said determining comprises:

[0350] For one or more vertices of the another version of the dynamic mesh, one or more vertex positions are predicted using previously determined 3D vertex values ​​from at least two different ones of the dynamic meshes corresponding to the first frame and the one or more additional frames.

[0351] 18. The method of clause 17, wherein the vertex positions of the first frame and / or the vertex positions of the one or more additional frames are determined using inter- frame prediction and / or intra-frame prediction.

[0352] 19. The method of clause 17, further comprising:

[0353] receiving further information of the compressed version of the dynamic mesh, the further information of the compressed version of the dynamic mesh comprising:

[0354]

[0355] single-reference frame multi-vertex prediction information for predicting vertex position information of a third frame using a plurality of vertices of a single previous frame; and

[0356] determining 3D vertex values of a third version of the dynamic mesh corresponding to the third frame, wherein the determining comprises:

[0357] for one or more vertices of the third version of the dynamic mesh, predicting one or more vertex positions using at least two different vertices from the single previous frame.

[0358] 20. The method of clause 17, further comprising:

[0359] determining one or more texture coordinate values of the other version of the dynamic mesh corresponding to the other frame, wherein the determining comprises:

[0360] for one or more texture coordinates of the other version of the dynamic mesh, predicting the one or more texture coordinate values using texture coordinate values from at least two different versions of the dynamic mesh corresponding to the first frame and the one or more additional frames.

[0361] 21. The method of clause 17, wherein the vertex positions of the first frame and the vertex positions of the one or more additional frames are signaled in the information per triangle of the dynamic mesh.

[0362] 22. The method of clause 17, wherein the received information of a compressed version of the dynamic mesh indicates texture coordinate values of the other version of the dynamic mesh corresponding to the other frame, wherein the texture coordinate values are signaled in the information per triangle of the dynamic mesh.

[0363] 23. The method of clause 22, wherein the information of the compressed version of the dynamic mesh indicates that at least a portion of the texture coordinate values of the other version of the dynamic mesh corresponding to the other frame are translated and / or rotated.​

[0364] 24. The method of clause 22, wherein the texture coordinate values of the other version of the dynamic mesh corresponding to the other frame are signaled in a bitstream that is different from the base mesh sub-bitstream.

[0365] Example computer system

[0366] FIG. 23 An example computer system 2300 that can implement an encoder or a decoder or any other component described herein (e.g., any of the components described above with reference to FIG. 1-22 The computer system 2300 can be configured to perform any or all of the embodiments described above. In different embodiments, the computer system 2300 can be any of a variety of types of devices including, but not limited to: a personal computer system, desktop computer, laptop, notebook, tablet, all-in-one, slate, or netbook computer; a mainframe computer system; a mobile device; a consumer electronic device; a video game console; a handheld computer; a workstation; a network computer; a television; a video recorder; peripheral equipment such as a switch, modem, router; or any other type of computing or electronic device.

[0367] The various embodiments of the point cloud encoder or decoder described herein can be executed on one or more computer systems 2300, which can interact with various other devices. Note that according to various embodiments, any components, actions, or functionality described above with respect to FIG. 1-22 the components described above can be implemented on one or more computers configured to FIG. 23 In the illustrated embodiment, the computer system 2300 includes one or more processors 2310 coupled to a system memory 2320 via an input / output (I / O) interface 2330. The computer system 2300 further includes a network interface 2340 coupled to the I / O interface 2330, and one or more input / output devices 2350, such as cursor control devices 2360, keyboards 2370, and display devices 2380. In some cases, it is contemplated that embodiments can be implemented using a single instance of the computer system 2300 while in other embodiments, multiple such systems, or multiple nodes making up a computer system 2300, can be configured to host different portions or instances of embodiments. For example, in one embodiment, some elements can be implemented via one or more nodes of the computer system 2300 that are distinct from those nodes implementing other elements.

[0368] In various embodiments, computer system 2300 can be a uniprocessor system including one processor 2310, or a multiprocessor system including several processors 2310 (e.g., two, four, eight, or another suitable number). Processors 2310 can be any suitable processors capable of executing instructions and performing the functions as described herein. For example, in various embodiments, processors 2310 can be general- purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of the processors 2310 can typically, but is not required to, implement the same ISA.

[0369] System memory 2320 can be configured to store point cloud compression or point cloud decompression program instructions 2322 and / or sensor data accessible by processor 2310. In various embodiments, system memory 2320 can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile / Flash-type memory, or any other type of memory. In the illustrated embodiment, program instructions 2322 can be configured to implement an image sensor control application in conjunction with any of the above-described functionality. In some embodiments, program instructions and / or data can be received, transmitted or stored on different types of computer-accessible media, or similar media, separate from system memory 2320 or computer system 2300. Although computer system 2300 is described as implementing functionality for the functional blocks of the preceding figures, any of the functionality described herein can be implemented by such a computer system.

[0370] In one embodiment, I / O interface 2330 can be configured to coordinate I / O traffic between processor 2310, system memory 2320, and any peripheral devices in the device, including network interface 2340 or other peripheral interfaces, such as input / output devices 2350. In some embodiments, I / O interface 2330 can perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., system memory 2320) into a format suitable for use by another component (e.g., processor 2310). In some embodiments, I / O interface 2330 can include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example. In some embodiments, the function of I / O interface 2330 can be split into two or more separate components, such as a north bridge and a south bridge, for example. In addition, in some embodiments some or all of the functionality of I / O interface 2330, such as an interface to system memory 2320, can be incorporated directly into processor 2310.

[0371] Network interface 2340 can be configured to allow data to be exchanged between computer system 2300 and other devices attached to a network 2385 (e.g., carrier or agent devices) or between nodes of computer system 2300. In various embodiments, network 2385 can include one or more networks, including, but not limited to, a local area network (LAN), a general wide area network (WAN) (e.g., the Internet), a wireless data network, some other electronic data network, or some combination thereof. In various embodiments, network interface 2340 can support communication via wired or wireless general data networks, such as any suitable type of Ethernet network, for example; via telecommunications / telephony networks such as analog voice networks or digital fiber communications networks; via storage area networks such as Fibre Channel SANs, or via any other suitable type of network and / or protocol.

[0372] In some embodiments, input / output devices 2350 can include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable to input or output data. Multiple input / output devices 2350 can be present in computer system 2300, or can be distributed on various nodes of computer system 2300. In some embodiments, similar input / output devices can be separate from computer system 2300 and can interact with one or more nodes of computer system 2300 through a wired or wireless connection, such as over network interface 2340.

[0373] AsFIG. 23 FIG. 23 As shown, memory 2320 can include program instructions 2322 that can be executable by the processor to implement any of the elements or acts described above. In one embodiment, the program instructions can perform the methods described above. In other embodiments, different elements and data can be included. Note that the data can include any of the data or information described above.

[0374] Those skilled in the art will appreciate that computer system 2300 is merely illustrative and is not intended to limit the scope of embodiments. In particular, the computer system and devices can include any combination of hardware or software that can perform the indicated functions, including computers, network devices, internet appliances, PDAs, wireless telephones, pagers, etc. Computer system 2300 can also be connected to any of a variety of other devices, not shown, such as to an external storage device or other computing devices using a network. Further, computer system 2300 can operate in a networked or distributed environment using logical and / or physical connections to one or more other computers, such as

[0375] Those skilled in the art will appreciate that, while various items are illustrated as being stored in memory or on storage while being used, these items or portions of them can be transferred between memory and other storage devices and can be transmitted or received over a network, for example, using a storage area network (SAN) or other method of distributed storage. Alternatively, some or all of these software components can be executed in memory on another device and communicated to and / or from the example computer system via inter-computer communication techniques. Some or all of the system components or data structures can also be stored (e.g., as instructions or structured data) on computer-accessible media or portable articles, for example, to enable transporting to or from a suitable drive, as described above. In some embodiments, instructions stored on a computer-accessible medium separate from computer system 1800 can be transmitted to computer system 1800 via transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as a network and / or a wireless link. Various embodiments can further include receiving, transmitting, or storing instructions and / or data implemented in accordance with the foregoing on a computer- accessible medium. Generally, a computer-accessible medium can include a non-transitory computer-readable storage medium or memory medium such as magnetic or optical media, e.g., disk or DVD / CD-ROM, volatile or non-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc. In some embodiments, a computer-accessible medium can include transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as a network and / or a wireless link.

[0376] In different embodiments, the methods described herein can be implemented in software, hardware, or a combination thereof. Moreover, the order of the blocks can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. Various modifications and changes can be made as would be obvious to a person skilled in the art having the benefit of this disclosure. The various embodiments described herein are intended to be illustrative and not restrictive. Many variations, modifications, additions, and improvements are possible. Accordingly, plural instances can be provided for components described herein as a single instance. Boundaries between various components, operations and data stores are somewhat arbitrary, and the order of operations of those can be changed. Other allocations of functionality are envisioned. The functionality of one or more components can be implemented in hardware, software, or a combination of both. Finally, the structure of the embodiments is intended to be illustrative only and not restrictive. Several variants, modifications, additions, and improvements can be made to the described embodiments without departing from the scope of the present disclosure.

Claims

1. A non-transitory computer-readable storage medium storing program instructions that, when executed using one or more computing devices, cause the one or more computing devices to: receive information for a compressed version of a dynamic mesh, the information including: vertex displacement and connectivity information for a first time point frame of the dynamic mesh; prediction information for predicting vertex displacement information for another time point frame; determine 3-dimensional (3D) vertex values for a first version of the dynamic mesh corresponding to the first time point frame using the vertex displacement and connectivity information for the first time point frame; and determine 3D vertex values for another version of the dynamic mesh corresponding to the another time point frame, wherein the determining includes: for one or more vertices of the another version of the dynamic mesh, using the prediction information and previously determined 3D vertex values from a version of the dynamic mesh corresponding to a first frame to predict vertex displacement information for the another version corresponding to the another time point frame of the dynamic mesh.

2. The non-transitory computer-readable storage medium of claim 1, wherein the received information further includes: one or more two-dimensional (2D) video encoding frames including residual values to be applied to predicted vertex position information predicted using the prediction information, and wherein the determining further includes: applying the residual values included in the 2D video encoding frames to the predicted vertex displacement information to generate the 3D vertex values for the another version of the dynamic mesh corresponding to the another time point frame.

3. The non-transitory computer-readable storage medium of claim 1 or claim 2, wherein the received information for a compressed version of the dynamic mesh is organized into: a base mesh sub-bitstream for reconstructing a base mesh of the dynamic mesh; and a displacement sub-bitstream including the vertex displacement information to be applied at subdivision points of the base mesh to reconstruct the dynamic mesh.

4. The non-transitory computer-readable storage medium of claim 3, wherein the program instructions, when executed using the one or more computing devices, further cause the one or more computing devices to: identify a plurality of sub-meshes in a atlas sub-bitstream that are signaled but not signaled in the base mesh sub-bitstream; and based on the identifying: use empty sub-meshes as placeholders in the base mesh sub-bitstream, wherein empty base meshes are used for sub-meshes that correspond to being referenced in the atlas sub-bitstream; or remove the sub-meshes that are referenced in the atlas sub-bitstream from the atlas sub-bitstream.

5. The non-transitory computer-readable storage medium of any one of claims 1 to 4, wherein the received information for a compressed version of the dynamic mesh includes: a base mesh sub-bitstream for reconstructing a base mesh of the dynamic mesh; ​ a displacement sub-bitstream comprising displacement information to be applied at subdivision points of the base mesh to reconstruct the dynamic mesh, wherein the displacement information of the displacement sub-bitstream is signaled using tiles comprising displacement values packed into two-dimensional (2D) video image frames of a video encoding; and an atlas sub-bitstream of atlas information for reconstructing the dynamic mesh, wherein the atlas information maps the displacements of the displacement sub-bitstream to subdivision locations of the base mesh, wherein a respective vertex count is signaled for a respective tile of the tiles packed into the 2D video image frames; and wherein to determine the 3D vertex values of the other version of the dynamic mesh corresponding to the other time point frame, the program instructions, when executed by the one or more computing devices, further cause the one or more computing devices to: map displacement information from the displacement sub-bitstream to vertex positions of the reconstructed base mesh signaled in the base mesh sub-bitstream using the vertex counts and the atlas information of the tiles.

6. The non-transitory computer-readable storage medium of claim 5, wherein: a tile data unit is used to signal information for locating a given tile of the tiles in the 2D video image frames, and a flag is used to indicate that one or more parts of the tile data unit are omitted from being signaled, wherein a default value is used in a reconstruction process or a value signaled in a frame parameter set or a sequence parameter set instead of using the omitted one or more parts.

7. A method comprising: receiving information of a compressed version of a dynamic mesh, the information comprising: a base mesh sub-bitstream of a base mesh for reconstructing the dynamic mesh; a displacement sub-bitstream comprising displacement information to be applied at subdivision points of the base mesh to reconstruct the dynamic mesh; and an atlas sub-bitstream of atlas information for reconstructing the dynamic mesh; identifying a plurality of sub-meshes of the sub-meshes that are signaled in the atlas sub-bitstream but not signaled in the base mesh sub-bitstream; and based on the identifying: using an empty sub-mesh as a placeholder in the base mesh sub-bitstream, wherein an empty base mesh is used for a sub-mesh that corresponds to a reference in the atlas sub-bitstream; or removing the sub-meshes that are referenced in the atlas sub-bitstream from the atlas sub-bitstream; determining 3-dimensional (3D) vertex values of the dynamic mesh using the reconstructed base mesh and displacements applied at vertex positions of the reconstructed base mesh, wherein displacement information from the displacement sub-bitstream is mapped to vertex positions of the base mesh signaled in the base mesh sub-bitstream using the atlas information.

8. The method of claim 7, wherein the received information comprises: vertex displacement and connectivity information of a first time point frame of the dynamic mesh; prediction information for predicting vertex displacement information of another time point frame; and wherein said determining the 3D vertex values of the dynamic mesh using the reconstructed base mesh and the displacements applied at the vertex positions of the reconstructed base mesh comprises: determining 3-dimensional (3D) vertex values of a first version of the dynamic mesh corresponding to the first time point frame using the vertex displacements and connectivity information of the first time point frame; and determining 3D vertex values of another version of the dynamic mesh corresponding to the another time point frame, wherein the determining comprises: predicting, for one or more vertices of the another version of the dynamic mesh, vertex displacement information of the another version of the dynamic mesh corresponding to the another time point frame using the prediction information and previously determined 3D vertex values from a version of the dynamic mesh corresponding to a first frame.

9. The method of claim 8, wherein the accepted information further comprises: one or more two-dimensional (2D) video coded frames comprising residual values to be applied to predicted vertex position information predicted using the prediction information, and wherein determining the 3D vertex values of the another version of the dynamic mesh further comprises: applying the residual values comprised in the 2D video coded frames to the predicted vertex displacement information to generate the 3D vertex values of the another version of the dynamic mesh corresponding to the another time point frame.

10. The method of any one of claims 7 to 9, wherein the received information comprises: a base mesh sub-bitstream for reconstructing a base mesh of the dynamic mesh; a displacement sub-bitstream comprising displacement information to be applied at subdivision points of the base mesh to reconstruct the dynamic mesh, wherein the displacement information of the displacement sub-bitstream is signaled using tiles comprising displacement values packed into two-dimensional (2D) video image frames of a video coding; and an atlas sub-bitstream for reconstructing atlas information of the dynamic mesh, wherein the atlas information maps the displacements of the displacement sub-bitstream to subdivision positions of the base mesh, wherein a respective vertex count is signaled for a respective tile in the tiles packed into the 2D video image frames; and wherein said determining the 3D vertex values of the dynamic mesh using the reconstructed base mesh and the displacements applied at the vertex positions of the reconstructed base mesh comprises: mapping displacement information from the displacement sub-bitstream to vertex positions of the reconstructed base mesh signaled in the base mesh sub-bitstream using the vertex counts of the tiles and the atlas information.

11. The method of claim 10, wherein: information for locating a given one of the tiles in the 2D video image frames is signaled using a tile data unit, and one or more parts of the tile data unit are omitted from being signaled, indicated using a flag, wherein a default value is used in a reconstruction process or a value signaled in a frame parameter set or a sequence parameter set, instead of the omitted one or more parts.

12. The method of any one of claims 7 to 11, wherein: the 2D video image frames are packed into a video bitstream, and the base mesh sub-bitstream, the displacement sub-bitstream, and the atlas sub-bitstream are packed into a mesh bitstream, and the mesh bitstream is packed into the video bitstream.

13. The method of any one of claims 7 to 12, wherein: the 2D video image frames are packed into a video bitstream, and the base mesh sub-bitstream, the displacement sub-bitstream, and the atlas sub-bitstream are packed into a mesh bitstream, and the mesh bitstream is packed into the video bitstream.

14. The method of any one of claims 7 to 13, wherein: the 2D video image frames are packed into a video bitstream, and the base mesh sub-bitstream, the displacement sub-bitstream, and the atlas sub-bitstream are packed into a mesh bitstream, and the mesh bitstream is packed into the video bitstream.

15. The method of any one of claims 7 to 14, wherein: the 2D video image frames are packed into a video bitstream, and the base mesh sub-bitstream, the displacement sub-bitstream, and the atlas sub-bitstream are packed into a mesh bitstream, and the mesh bitstream is packed into the video bitstream.

16. The method of any one of claims 7 to 15, wherein: the 2D video image frames are packed into a video bitstream, and the base mesh sub-bitstream, the displacement sub-bitstream, and the atlas sub-bitstream are packed into a mesh bitstream, and the mesh bitstream is packed into the video bitstream.

17. The method of any one of claims 7 to 16, wherein: the 2D video image frames are packed into a video bitstream, and the base mesh sub-bitstream, the displacement sub-bitstream, and the atlas sub-bitstream are packed into a mesh bitstream, and the mesh bitstream is packed into the video bitstream.

18. The method of any one of claims 7 to 17, wherein: the 2D video image frames are packed into a video bitstream, and the base mesh sub-bitstream, the displacement sub-bitstream, and the atlas sub-bitstream are packed into a mesh bitstream, and the mesh bitstream is packed into the video bitstream.

19. The method of any one of claims 7 to 18, wherein: the 2D video image frames are packed into a video bitstream, and the base mesh sub-bitstream, the displacement sub-bitstream, and the atlas sub-bitstream are packed into a mesh bitstream, and the mesh bitstream is packed into the video bitstream.

20. The method of any one of claims 7 to 19, wherein: the 2D video image frames are packed into a video bitstream, and the base mesh sub-bitstream, the displacement sub-bitstream, and the atlas sub-bitstream are packed into a mesh bitstream, and the mesh bitstream is packed into the video bitstream.

12. The method according to any one of claims 7 to 9, wherein the received information of the compressed version of the dynamic grid comprises: Vertex positions and connectivity information of the first frame; vertex positions and connectivity information for one or more additional frames; and multi-frame prediction information for predicting vertex position information for another frame using a plurality of previous frames and an indication of vertex connectivity information to be used for the another frame; and wherein said determining the 3D vertex values ​​of the dynamic mesh using a reconstructed base mesh and the displacements applied at the vertex positions of the reconstructed base mesh comprises: determining 3D vertex values ​​for a first version of the dynamic mesh corresponding to the first frame using the vertex positions and connectivity information of the first frame; using the vertex positions and connectivity information of the one or more additional frames to determine 3D vertex values ​​for one or more additional versions of the dynamic mesh corresponding to the one or more additional frames; and Determining 3D vertex values ​​of another version of the dynamic mesh corresponding to the another frame, wherein the determining comprises: For one or more vertices of the another version of the dynamic mesh, one or more vertex positions are predicted using previously determined 3D vertex values ​​from at least two different ones of the dynamic meshes corresponding to the first frame and the one or more additional frames.

13. The method according to claim 12, further comprising: determining one or more texture coordinate values ​​of the another version of the dynamic mesh corresponding to the another frame, wherein the determining comprises: For one or more texture coordinates of the another version of the dynamic mesh, the one or more texture coordinate values ​​are predicted using texture coordinate values ​​from at least two different versions of the dynamic mesh corresponding to the first frame and the one or more additional frames.

14. A device comprising: a memory storing program instructions; and one or more processors, wherein the program instructions, when executed using the one or more processors, cause the one or more processors to: Receive information about a compressed version of a dynamic mesh, the information comprising: a base grid sub-bitstream for reconstructing a base grid of the dynamic grid; a displacement sub-bitstream comprising displacement information to be applied at subdivision points of the base mesh to reconstruct the dynamic mesh, wherein the displacement information of the displacement sub-bitstream is signaled using slices comprising displacement values ​​packed into two-dimensional (2D) video image frames for video coding; and an atlas sub-bitstream for reconstructing atlas information of the dynamic mesh, wherein the atlas information maps the displacements of the displacement sub-bitstream to subdivision positions of the base grid, wherein corresponding vertex counts are signaled for corresponding ones of the tiles packed into the 2D video image frame; determining 3-dimensional (3D) vertex values of the dynamic mesh using the reconstructed base mesh and the displacement information applied at the vertex positions of the reconstructed base mesh, wherein the vertex count of the tile and the atlas information are used to map displacement information from the displacement sub-bitstream to vertex positions of the reconstructed base mesh signaled in the base mesh sub-bitstream.

15. The device of claim 14, wherein: a tile data unit is used to signal information for locating a given tile of the tiles in the 2D video image frame, and a flag is used to indicate that one or more parts of the tile data unit are omitted from being signaled, wherein a default value or a value signaled in a frame parameter set or a sequence parameter set is used in a reconstruction process instead of the omitted one or more parts.

16. The device of claim 14 or claim 15, wherein the received information includes: vertex displacement and connectivity information for a first time point frame of the dynamic mesh; prediction information for predicting vertex displacement information for another time point frame; and wherein to determine the 3D vertex values of the dynamic mesh, the program instructions, when executed with the one or more processors, further cause the one or more processors to: determine 3-dimensional (3D) vertex values of a first version of the dynamic mesh corresponding to the first time point frame using the vertex displacement and connectivity information for the first time point frame; and determine 3D vertex values of another version of the dynamic mesh corresponding to the another time point frame, wherein the determining includes: predicting, for one or more vertices of the another version of the dynamic mesh, vertex displacement information for the another version of the dynamic mesh corresponding to the another time point frame using the prediction information and previously determined 3D vertex values of the version of the dynamic mesh corresponding to the first frame.

17. The device of claim 16, wherein the received information further includes: one or more two-dimensional (2D) video encoding frames including residual values to be applied to predicted vertex position information predicted using the prediction information, and wherein determining the 3D vertex values of the another version of the dynamic mesh further includes: applying the residual values included in the 2D video encoding frames to the predicted vertex displacement information to generate the 3D vertex values of the another version of the dynamic mesh corresponding to the another time point frame.

18. The device of claim 16, wherein the program instructions, when executed with the one or more processors, further cause the one or more processors to: determine one or more texture coordinate values of the another version of the dynamic mesh corresponding to the another frame, wherein the determining includes: predicting, for one or more texture coordinates of the another version of the dynamic mesh, the one or more texture coordinate values using texture coordinate values from at least two different versions of the dynamic mesh corresponding to the first frame and the one or more additional frames. ​ 19. The device of any one of claims 14-18, wherein the received information of the compressed version of the dynamic mesh is organized into: a base mesh sub-bitstream for reconstructing a base mesh of the dynamic mesh; and a displacement sub-bitstream comprising the vertex displacement information to be applied at subdivision points of the base mesh to reconstruct the dynamic mesh.

20. The device of claim 19, wherein the program instructions, when executed using the one or more processors, further cause the one or more processors to: identify a plurality of sub-meshes in the atlas sub-bitstream that are signaled but not signaled in the base mesh sub-bitstream; and based on the identifying: use empty sub-meshes as placeholders in the base mesh sub-bitstream, wherein empty base meshes are used to correspond to sub-meshes referenced in the atlas sub-bitstream; or remove the sub-meshes referenced in the atlas sub-bitstream from the atlas sub-bitstream.

Images