Signaling of base mesh motion vectors for video-based mesh coding

Abstract

The system includes an encoder configured to compress and encode three-dimensional mesh data. To compress the three-dimensional mesh, the encoder determines the displacements applied to the refinement locations of the base mesh. In addition to signaling displacements, motion vectors of parts of the base mesh (e.g., submeshes) may be signaled. In this way, the similarity of the base mesh structure across multiple moments (or within moments in a time frame) can be utilized to further improve compression efficiency.

Description

Technical Field

[0001] This disclosure generally relates to the compression and restoration of three-dimensional meshes having associated textures or attributes.

Background Art

[0002] (Description of Related Art) Various types of sensors, such as light detection and ranging (LIDAR) systems, 3D cameras, 3D scanners, etc., can capture data indicating the positions of points in three-dimensional space, e.g., locations in the X, Y, and Z planes. Further, such systems can further capture attribute information in addition to the spatial information for each point, such as color information (e.g., RGB values), texture information, intensity attributes, reflectance attributes, motion-related attributes, modality attributes, or various other attributes. In some situations, additional attributes, such as a timestamp when the point was captured, can be assigned to each point. The points captured by such sensors may each constitute a "point cloud" comprising a set of points having associated spatial information and one or more associated attributes. In some situations, the point cloud may include thousands, hundreds of thousands, millions, or even more points. Also, in some situations, the point cloud may be generated by, for example, software rather than being captured by one or more sensors. In any case, such point clouds can contain large amounts of data and can be costly and time-consuming to store and transmit. Also, three-dimensional visual content may be captured in other ways, such as via 2D images of a scene captured from multiple viewing positions with respect to the scene.

[0003] Such three-dimensional visual content can be represented by a three-dimensional mesh including a plurality of polygons having connected vertices that model the surface of the three-dimensional visual content, such as the surface of a point cloud. Further, the texture or attribute values of the points of the three-dimensional visual content can be overlaid on the mesh to represent the attributes or textures of the three-dimensional visual content when modeled as a three-dimensional mesh.

[0004] Additionally, the three-dimensional mesh may be generated, for example, by software, without first being modeled as a point cloud or other type of three-dimensional visual content. For example, software can directly generate a three-dimensional mesh and then apply textures or attribute values ​​to represent an object. [Overview of the Initiative]

[0005] In some embodiments, the system includes one or more sensors configured to capture points representing objects in the sensor's view and capture textures or attribute values ​​associated with the object points. The system also includes one or more computing devices that, when executed, store program instructions causing one or more computing devices to generate a three-dimensional mesh that models the object points using vertices and connections between vertices that define the polygons of the three-dimensional mesh. In some embodiments, the three-dimensional mesh may also be generated without being initially captured by one or more sensors. For example, a computer graphics program can generate a three-dimensional mesh with associated textures or associated attribute values ​​to represent objects in a scene without necessarily generating a point cloud representing the objects.

[0006] In some embodiments, the encoder system, when executed by one or more computing devices, further includes one or more computing devices that store program instructions causing one or more computing devices to determine a plurality of patches for the attributes of a three-dimensional mesh, and corresponding attribute maps that map the attribute patches to the geometry of the mesh.

[0007] The encoder system can further encode the mesh geometry by encoding the base mesh and the vertex displacements relative to the base mesh. The compressed bitstream may contain the compressed base mesh, compressed displacement values, and compressed attribute information. To improve compression efficiency, coding units may be used to encode parts of the mesh. For example, a coding unit may comprise tiles of mesh, each tile comprising independently encoded segments of the mesh. As another example, a coding unit may comprise patches consisting of several submeshes of the mesh, where the submeshes leverage dependencies between them and are therefore not independently encoded. Higher-level coding units, such as groups of patches, can also be used. Different coding parameters may be defined in the bitstream to be applied to different coding units. For example, instead of having to repeatedly signal coding parameters, commonly signaled coding parameters may be applied to members of a given coding unit, such as submeshes of a patch or mesh portions constituting a tile. Some exemplary coding parameters that may be used include, among a variety, entropy coding parameters, intra-frame prediction parameters, inter-frame prediction parameters, and local or sub-mesh indices. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows exemplary input information for defining a three-dimensional mesh according to several embodiments.

[0009] [Figure 2] Several embodiments demonstrate alternative examples of input information for defining a three-dimensional mesh, where the input information is formatted according to an object format.

[0010] [Figure 3]The following are exemplary preprocessors and encoders for encoding a three-dimensional mesh according to several embodiments.

[0011] [Figure 4] More detailed diagrams of exemplary intraframe encoders according to several embodiments are shown.

[0012] [Figure 5] This document describes exemplary intra-frame decoders for decoding a three-dimensional mesh, according to several embodiments.

[0013] [Figure 6] More detailed diagrams of exemplary interframe encoders according to several embodiments are shown.

[0014] [Figure 7] This document illustrates an exemplary interframe decoder for decoding a three-dimensional mesh, according to several embodiments.

[0015] [Figure 8A] This document illustrates the segmentation of a mesh into multiple tiles according to several embodiments. [Figure 8B] This document illustrates the segmentation of a mesh into multiple tiles according to several embodiments.

[0016] [Figure 9] Several embodiments show meshes segmented into two tiles, each containing a submesh.

[0017] [Figure 10] Figures 10A to 10D show adaptive subdivision based on edge subdivision rules for shared edges in several embodiments.

[0018] [Figure 11A]An adaptive subdivision of adjacent patches is shown, where the shared edges are subdivided in a way that ensures that the vertices of adjacent patches are aligned with each other, according to some embodiments. [Figure 11B] An adaptive subdivision of adjacent patches is shown, where the shared edges are subdivided in a way that ensures that the vertices of adjacent patches are aligned with each other, according to some embodiments. [Figure 11C] An adaptive subdivision of adjacent patches is shown, where the shared edges are subdivided in a way that ensures that the vertices of adjacent patches are aligned with each other, according to some embodiments.

[0019] [Figure 12] An exemplary component of a compressed bitstream of a dynamic mesh is shown, according to some embodiments.

[0020] [Figure 13] A flowchart showing a process of reconstructing a dynamic mesh using a base mesh and displacement information, where only a portion of the base mesh corresponding to a subset of all sub-meshes of the base mesh is used, according to some embodiments.

[0021] [Figure 14] Different prediction modes that can be applied to predict various parts (e.g., sub-meshes) of a base mesh, according to some embodiments.

[0022] [Figure 15] A flowchart showing a process for compressing a dynamic mesh using sub-mesh data units, according to some embodiments.

[0023] [Figure 16] An exemplary computer system that can implement an encoder or a decoder is shown, according to some embodiments.

[0024] This specification includes references to “one embodiment” or “an embodiment.” The phrases “in one embodiment” or “in an embodiment” do not necessarily refer to the same embodiment. Certain features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.

[0025] "comprising" is an open-ended term. As used in the attached claims, this term does not exclude additional structures or steps. Consider a claim that reads "an apparatus comprising one or more processor units." Such a claim does not exclude the apparatus comprising additional components (e.g., a network interface unit, a graphics circuit, etc.).

[0026] "configured to." Various units, circuits, or other components may be described or claimed to be "configured to" perform one or more tasks. In such contexts, "configured to" is used when implying a structure by indicating that the unit / circuit / component includes a structure (e.g., a circuit) that performs that one or more tasks during operation. Thus, a unit / circuit / component can be said to be configured to perform tasks even when the specified unit / circuit / component is not currently operational (e.g., not in the on state). Units / circuits / components used with the phrase "configured to" include hardware, e.g., circuits, memory that stores executable program instructions to implement the operation. The statement that a unit / circuit / component is "configured to" perform one or more tasks is explicitly intended to mean that Section 112(f) of the U.S. Patent Act will not be invoked with respect to that unit / circuit / component. Furthermore, “configured” may include a general-purpose structure (e.g., a general-purpose circuit) that is operated by software and / or firmware (e.g., an FPGA or a general-purpose processor running the software) and operates in a manner capable of performing the task(s) in question. “Configured” may also include adapting a manufacturing process (e.g., semiconductor assembly equipment) to assemble a device (e.g., an integrated circuit) adapted to perform or execute one or more tasks.

[0027] "First," "Second," etc. As used herein, these terms are used as indicators of the following noun and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing write operations on "first" and "second" values. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0028] "Based on." As used herein, this term is used to describe one or more factors that influence a determination. This term does not exclude additional factors that may influence the determination; that is, the determination may be based on those factors alone, or at least partially on those factors. Consider the phrase "determine A based on B." In this case, B is a factor that influences the determination of A, but such a phrase does not exclude the fact that the determination of A is also based on C. In other examples, A may be determined based on B alone. [Modes for carrying out the invention]

[0029] As data acquisition and display technologies become more advanced, the ability to capture stereoscopic content, such as thousands or millions of points in 2D or 3D space via LiDAR systems, is increasing. Furthermore, the development of advanced display technologies, such as virtual reality or augmented reality systems, is increasing the potential use of stereoscopic content. However, stereoscopic content files are often very large and can be costly and time-consuming to store and transmit. For example, communicating stereoscopic content over private or public networks such as the internet can require a considerable amount of time and / or network resources, potentially limiting some uses of stereoscopic content, such as real-time use. Additionally, the storage requirements of stereoscopic content files can consume a significant amount of storage capacity on the device storing them, which can also limit potential applications for using stereoscopic content data.

[0030] In some embodiments, the encoder may be used to generate compressed stereoscopic content to reduce the cost and time associated with storing and transmitting large stereoscopic content files. In some embodiments, the system may include an encoder that compresses the attributes and / or spatial information of the stereoscopic content so that the stereoscopic content file can be stored and transmitted faster than uncompressed stereoscopic content and occupy less storage space than uncompressed stereoscopic content.

[0031] In some embodiments, such encoders and decoders, or other encoders and decoders described herein, may be adapted to encode, additionally or alternatively, 3-degrees-of-freedom plus (3DOF+) scenes, visual stereoscopic content such as MPEG V3C scenes, immersive video scenes such as MPEG MIV, and the like.

[0032] In some embodiments, the static or dynamic mesh to be compressed and / or encoded may include a set of 3D meshes M(0), M(1), M(2), ..., M(n). Each mesh M(i) may be defined by connectivity information C(i), geometry information G(i), texture coordinates T(i), and texture connectivity TC(i). For each mesh M(i), one or more 2D images A(i,0), A(i,1)..., A(i,D-1) may be included that describe the texture or attributes associated with the mesh. For example, Figure 1 shows an exemplary static or dynamic mesh M(i) comprising connectivity information C(i), geometry information G(i), texture image A(i), texture connectivity information TC(i), and texture coordinate information T(i). Figure 2 also shows an example of a textured mesh stored in object (OBJ) format.

[0033] For example, an exemplary texture mesh stored in the object format shown in Figure 2 includes geometric information listed as the X, Y, and Z coordinates of the vertices, and texture coordinates listed as the two-dimensional (2D) coordinates of the vertices, where the 2D coordinates identify the pixel location of the pixels that store the texture information of a given vertex. An exemplary texture mesh stored in the object format also includes texture connectivity information that shows the mapping between geometric coordinates and texture coordinates to form polygons such as triangles. For example, a first triangle is formed by three vertices, and the first vertex (1 / 1) is defined as having first geometric coordinates (e.g., 64.062500, 1237.739990, 51.757801) corresponding to first texture coordinates (e.g., 0.0897381, 0.740830). The second vertex (2 / 2) of the triangle is defined as the second geometric coordinates (e.g., 59.570301, 1236.819946, 54.899700) corresponding to the second texture coordinates (e.g., 0.899059, 0.741542). Finally, the third vertex of the triangle corresponds to the third listed geometric coordinates that coincide with the third listed texture coordinates. However, it should be noted that in some cases, the vertices of a polygon, such as a triangle, may map to sets of geometric and texture coordinates that have different index positions in their respective lists of geometric and texture coordinates. For example, the second triangle has a first vertex that corresponds to the fourth listed set of geometric coordinates and the seventh listed set of texture coordinates. The second vertex corresponds to the first listed set of geometric coordinates and the first listed set of texture coordinates, and the third vertex corresponds to the third listed set of geometric coordinates and the ninth listed set of texture coordinates.

[0034] In some embodiments, geometry information G(i) may represent the locations of mesh vertices in 3D space, and connectivity C(i) may indicate how the vertices should be connected to form polygons constituting mesh M(i). Texture coordinates T(i) may indicate the locations of pixels in a 2D image corresponding to the vertices of the corresponding submesh. Attribute patch information may indicate how texture coordinates defined for a 2D bounding box are mapped to the three-dimensional space of a 3D bounding box associated with an attribute patch, based on how the points are projected onto the projection plane of the attribute patch. Texture connectivity information TC(i) may indicate how the vertices represented by texture coordinates T(i) are connected to form polygons of the submesh. For example, each texture or attribute patch in texture image A(i) may correspond to a corresponding submesh defined using texture coordinates T(i) and texture connectivity TC(i).

[0035] In some embodiments, the mesh encoder may perform a patch generation process, and the mesh is subdivided into a set of submeshes. The submeshes may correspond to connected components of texture connectivity, or they may be submeshes distinct from the texture connectivity of the mesh. In some embodiments, the number and size of the submeshes to be determined may be adjusted to balance discontinuity and flexibility for updating the mesh, such as through interpretation. For example, smaller submeshes may allow for finer granularity of updates to modify specific areas of the mesh in subsequent moments, such as using the interpretation process. However, a larger number of submeshes can introduce more discontinuities.

[0036] Figure 3 shows a high-level block diagram of the encoding process in several embodiments. Note that the feedback loop during the encoding process allows the encoder to guide the preprocessing steps and modify its parameters to achieve the best possible compromise according to various criteria such as rate distortion, encoding / decoding complexity, random access, reconstruction complexity, terminal capability, encoder / decoder power consumption, network bandwidth and latency, and / or other factors.

[0037] A mesh, which may be static or dynamic, is received in preprocessing module 302. An attribute map, representing how the attribute image (e.g., texture image) of the static / dynamic mesh is mapped to the mesh, is also received in preprocessing module 302. For example, the attribute map may include the texture coordinates and texture connectivity of the mesh's texture image. Preprocessing module 302 separates the static / dynamic mesh into a base mesh and displacements, where displacements represent how the vertices should be displaced to recreate the original static / dynamic mesh from the base mesh. For example, in some embodiments, vertices included in the original static / dynamic mesh may be omitted from the base mesh (e.g., the base mesh may be a compressed version of the original static / dynamic mesh). As will be described in more detail below, the decoder may predict additional vertices to be added to the base mesh, for example, by subdividing the edges between the remaining vertices included in the base mesh. In such examples, displacements can indicate how the original vertices of the base mesh and any additional vertices added to the subdivision locations should be displaced, and the displacements of the original and added vertices modify the partially reconstructed version of the mesh to better represent the original static / dynamic mesh. For example, Figure 4 shows a detailed intra-frame encoder 402 that can be used to encode the base mesh m(i) and displacement d(i) of the added vertices. For dynamic meshes, an inter-frame encoder such as the one shown in Figure 6 can be used. As seen in Figure 6, instead of signaling a new base mesh for each frame, the base mesh of the current time frame can be compared to a reconstructed quantized reference base mesh m'(ir) (e.g., the reconstructed base mesh as seen by the decoder from the previous time frame), and motion vectors representing how the current base mesh has changed relative to the reference base mesh can be encoded instead of encoding a new base mesh for each frame. Note that motion vectors may not be directly encoded but may be further compressed to utilize the relationships between motion vectors.

[0038] The separated base mesh and displacements separated by the preprocessing module 302 are provided to an encoder 304, which can be an intra-frame encoder as shown in Figure 4 or an inter-frame encoder as shown in Figure 6. An attribute map is also supplied to the encoder 304. In some embodiments, in addition to the separated base mesh and displacements, the original static / dynamic mesh may also be provided to the encoder 304. For example, the encoder 304 may compare a reconstructed version of the static / dynamic mesh (reconstructed by applying a reconstruction process to the base mesh and displacements) to determine geometric distortion. In some embodiments, an attribute transformation process may be performed to adjust the attribute values ​​of the attribute image to account for this slight geometric distortion. In some embodiments, feedback may be provided to the preprocessing 302 to reduce distortion, for example, by changing how the original static / dynamic mesh is decimated to generate the base mesh. In some embodiments, the feedback may be returned to the preprocessing 302, for example, to be used to generate the base mesh for future frames. Note that in some embodiments, the intra-frame encoder and the inter-frame encoder may be combined into a single encoder that includes logic to toggle between intra-frame encoding and inter-frame encoding. The output of encoder 304 is a compressed bitstream representing the original static / dynamic mesh and its associated attributes / textures.

[0039] With respect to mesh decimation, in some embodiments, a portion of the surface of a static / dynamic mesh can be considered as an input 2D curve (represented by a 2D polyline) referred to as the “original” curve. The original curve can first be downsampled to generate a base curve / polyline referred to as the “decimated” curve. Then, a subdivision scheme, such as those described herein, can be applied to the decimated polyline to generate the “subdivision” curve. For example, a subdivision scheme using iterative interpolation may be applied. The subdivision scheme may include inserting a new point in the middle of each edge of the polyline in each iteration. The inserted point represents an additional vertex that can be moved by displacement.

[0040] For example, a subdivided polyline is then deformed to obtain a better approximation of the original curve. More precisely, a displacement vector is calculated for each vertex of the subdivided mesh so that the shape of the displaced curve approximates the shape of the original curve. The advantage of a subdivided curve is that it provides a faithful approximation of the original curve while having a subdivided structure that allows for efficient compression. Compression efficiency is achieved by the following properties: The decimate / base curve has a small number of vertices and a limited number of bits to encode / transmit. When the base / decimated curve is decoded, the subdivided curve is automatically generated by the decoder (for example, the decoder does not need to signal or hardcode any information other than the subdivision method type and subdivision iteration count). The displacement curve is generated by decoding and applying the displacement vectors associated with the subdivided curve vertices. In addition to enabling spatial / quality scalability, the subdivided structure allows for efficient wavelet decomposition, which provides high compressive performance (e.g., with respect to rate-strain performance).

[0041] For example, Figure 4 shows a more detailed diagram of an exemplary intraframe encoder according to several embodiments.

[0042] In some embodiments, the intraframe encoder 402 receives a base mesh m(i), displacement d(i), the original static / dynamic mesh M(i), and attribute map A(i). The base mesh m(i) is provided to the quantization module 404, and the aspects of the base mesh can be (optionally) further quantized. In some embodiments, the base mesh can be encoded using various mesh encoders. Also, in some embodiments, the intraframe encoder 402 can be customizable, and the base mesh can be encoded using different respective mesh encoding schemes. For example, the static mesh encoder 406 may be a selected mesh encoder, chosen from a set of viable mesh encoders such as a DRACO encoder (or another suitable encoder). The encoded base mesh encoded by the static mesh encoder 406 is provided to the multiplexer (MUX) 438 for inclusion in a compressed bitstream b(i). Additionally, the encoded base mesh is provided to the static mesh decoder to generate a reconstructed version of the base mesh (as seen by the decoder). This reconstructed version of the base mesh is used to update the displacement d(i) to take into account any geometric distortions between the original base mesh and the reconstructed version of the base mesh (as seen by the decoder). For example, the static mesh decoder 408 generates a reconstructed quantized base mesh m'(i) and provides the reconstructed quantized base mesh m'(i) to the displacement update module 410, which also receives the original base mesh and the original displacement d(i). The displacement update module 410 compares the reconstructed quantized base mesh m'(i) (as seen by the decoder) with the base mesh m(i) and adjusts the displacement d(i) to account for the difference between the base mesh m(i) and the reconstructed quantized base mesh m'(i).These updated displacements d'(i) are then supplied to a wavelet transform 412 that outputs wavelet coefficients e(i), which are provided to a quantization module 414 that applies a wavelet transform to further compress the updated displacements d'(i) and generate quantized wavelet coefficients e'(i). The quantized wavelet coefficients may then be packed into a 2D image frame via an image packing module 416, and the packed 2D image frame is further video encoded via a video encoder 418. The encoded video image is also supplied to a multiplexer (MUX) 438 for inclusion in a compressed bitstream b(i). In some embodiments, the displacement values ​​(as shown in the generated quantized wavelet coefficients e'(i) or as shown using other compression schemes) may also be encoded at least partially outside of the video subbitstreams, such as their own displacement data subbitstream, base mesh subbitstream, or atlas data subbitstream.

[0043] In addition, to account for any geometric distortions introduced to the original static / dynamic mesh, the attribute transfer process 430 may be used to modify the attributes to account for the differences between the reconstructed deformed mesh DM(i) and the original static / dynamic mesh.

[0044] For example, the video encoder 418 may further perform video decoding (or a complementary video decoding module (not shown in Figure 4) may be used). This generates reconstructed packed quantized wavelet coefficients that are unpacked via the image unpacking module 420. Furthermore, inverse quantization may be applied via the inverse quantization module 422, and the inverse wavelet transform 424 may be applied to generate the reconstructed displacement d'(i). In some embodiments, other decoding techniques may be used to generate the reconstructed displacement d'(i), such as decoding the signaled displacement in the atlas data subbitstream, displacement data subbitstream, or base mesh subbitstream. Also, the reconstructed quantized base mesh m'(i) generated by the static mesh decoder 408 can be inversely quantized via the inverse quantization module 428 to generate the reconstructed base mesh m''(i). The reconstructed deformed mesh generation module 426 applies the reconstructed displacement d'(i) to the reconstructed base mesh m''(i) to generate the reconstructed deformed mesh DM(i). The reconstructed deformed mesh DM(i) represents the reconstructed mesh generated by the decoder, and it should be noted that it takes into account any geometric deformations resulting from the loss introduced during the encoding process.

[0045] The attribute transfer module 430 compares the geometry of the original static / dynamic mesh M(i) with the reconstructed deformed mesh DM(i), updates the attribute map to account for any geometric deformations, and outputs this updated attribute map as the updated attribute map A'(i). The updated attribute map A'(i) is then padded, and the 2D image containing the attribute image is padded so that padding is applied to spaces not used to communicate the attribute image. In some embodiments, color space conversion is optionally applied in the color space conversion module 434. For example, the RGB color space used to represent the color values ​​of the attribute image may be converted to the YCbCr color space, and color space subsampling such as 4:2:0 or 4:0:0 may be applied. The padded and optionally color space converted updated attribute map A'(i) is then video encoded via the video encoding module 436 and provided to the multiplexer 438 for inclusion in the compressed bitstream b(i).

[0046] In some embodiments, the controller 400 can adjust various quantization and dequantization steps, as well as video coding and decoding steps, so that dequantization "undoes" quantization and video decoding "undoes" video coding. The attribute transfer module 430 can also take into account the level of quantization applied based on communication from the controller 400.

[0047] Figure 5 shows exemplary intraframe decoders for decoding a three-dimensional mesh according to several embodiments.

[0048] The intraframe decoder 502 receives a compressed bitstream b(i), such as the compressed bitstream generated by the intraframe encoder 402 shown in Figure 4. The demultiplexer (DEMUX) 504 analyzes the bitstream into base mesh subcomponents, displacement subcomponents, and attribute map subcomponents. In some embodiments, the displacement subcomponents may be signaled in the displacement data subbitstream, or at least partially in other subbitstreams such as the atlas data subbitstream, base mesh subbitstream, or video subbitstream. In such cases, the displacement decoder 522 decodes the displacement subbitstream and / or the atlas decoder 524 decodes the atlas subbitstream.

[0049] The static mesh decoder 506 decodes the base mesh subcomponents to generate a reconstructed quantized base mesh m'(i), which is sent to the inverse quantization module 518, which outputs a decoded base mesh m''(i) and sends it to the reconstructed deformed mesh generator 520.

[0050] In some embodiments, some of the displacement sub-components of the bitstream are provided to video decoding 508, and the video-encoded image frames are video-decoded and provided to image unpacking 510. Image unpacking 510 extracts the packed displacements from the video-decoded image frames and provides them to inverse quantization 512, where the displacements are inversely quantized. The inversely quantized displacements are then provided to inverse wavelet transform 514, which outputs the decoded displacements d'(i). The reconstructed deformation mesh generator 520 applies the decoded displacements d'''(i) to the decoded base mesh m''(i) to generate the decoded static / dynamic mesh M''(i). The decoded displacements can come from any combination of video sub-bitstreams, atlas data sub-bitstreams, base mesh sub-bitstreams, and / or displacement data sub-bitstreams. Additionally, attribute map sub-components are provided to video decoding 516, which outputs the decoded attribute map A''(i). Next, the reconstructed version of the three-dimensional visual content can be rendered on the device associated with the decoder using the decoded mesh M''(i) and the decoded attribute map A''(i).

[0051] As shown in Figure 5, the bitstream consists of three or more separate substreams, i.e., • Mesh subnet, • Position and potentially per-vertex attribute displacement substream, and • The attribute map is demultiplexed into attribute map substreams for each attribute map.

[0052] The mesh substream is fed to the mesh decoder to generate a reconstructed quantized base mesh m'(i). The decoded base mesh m''(i) is then obtained by applying inverse quantization to m'(i). The proposed scheme is ignorant of which mesh codec is used. The mesh codec used may be explicitly specified in the bitstream, or it may be implicitly defined / fixed by the specification or application.

[0053] The displacement substream can be decoded by a video / image decoder. The resulting image / video is then unpacked and inverse quantization is applied to the wavelet coefficients. In an alternative embodiment, the displacement is decoded by a dedicated displacement data decoder or atlas decoder. The proposed scheme does not know which codec / standard is used. Image / video codecs such as [HEVC][AVC][AV1][AV2][JPEG][JPEG2000] can be used. A motion decoder or dictionary-based decoder such as ZIP used to decode mesh motion information can be used, for example, as a dedicated displacement data decoder. The decoded displacement d'(i) is generated by applying the inverse wavelet transform to the unquantized wavelet coefficients. The final decoded mesh is generated by applying a reconstruction process to the decoded base mesh m''(i) and adding the decoded displacement field d''(i).

[0054] The attribute substream is decoded directly by the video decoder, and the decoded attribute map A''(i) is produced as output. The proposed scheme does not know which codec / standard is used. Image / video codecs such as [HEVC][AVC][AV1][AV2][JPEG][JPEG2000] can be used. Alternatively, the attribute substream can be decoded by using a non-image / video decoder (e.g., using a dictionary-based decoder such as ZIP). Multiple substreams can be decoded, each associated with a different attribute map. Each substream can use a different codec.

[0055] Figure 6 shows a more detailed diagram of an exemplary interframe encoder according to several embodiments.

[0056] In some embodiments, the interframe encoder 602 may include components similar to those of the intraframe encoder 402, but instead of encoding a base mesh, the interframe encoder may encode motion vectors that can be applied to a reference mesh, which the decoder then uses to generate the base mesh.

[0057] For example, in the case of dynamic meshes, a time-consistent remeshing process is used, which can generate the same subdivision structure shared by the current mesh M'(i) and the reference mesh M'(j). Such a coherent time remeshing process skips the encoding of the base mesh m(i) and allows the reuse of the base mesh m(j) associated with the reference frame M(j). This can also allow for better time prediction for both attribute and geometric information. More precisely, a motion field f(i) can be computed and encoded that describes how the vertices of m(j) are moved to match the positions of m(i). Such a process is illustrated in Figure 6. For example, the motion encoder 406 can generate a motion field f(i) that describes how the vertices of m(j) are moved to match the positions of m(i).

[0058] In some embodiments, the base mesh m(i) associated with the current frame is first quantized (e.g., using uniform quantization) and encoded by using a static mesh encoder. The proposed scheme is ignorant of which mesh codec is used. The mesh codec used may be explicitly specified in the bitstream by encoding a mesh codec ID, or it may be implicitly defined / fixed by the specification or application.

[0059] Depending on the application and target bitrate / visual quality, the encoder can optionally encode a set of displacement vectors associated with the vertices of a subdivided mesh, referred to as the displacement field d(i).

[0060] Next, the displacement field d(i) is updated (in the displacement update module 410) using the reconstructed quantized base mesh m'(i) (e.g., the output of the reconstruction of base mesh 408) to generate an updated displacement field d'(i) that takes into account the difference between the reconstructed base mesh m'(i) and the original base mesh m(i). By utilizing the refined surface mesh structure, the wavelet transform is applied to d'(i) in wavelet transform 412 to generate a set of wavelet coefficients. The wavelet coefficients are then quantized in quantization 414, packed into a 2D image / video (in image packing 416), and compressed (in video encoding 418) by using an image / video encoder. The encoding of the wavelet coefficients may be reversible or reversible. The reconstructed version of the wavelet coefficients is obtained (e.g., in 420, 422, and 424) by applying image unpacking and inverse quantization to the reconstructed wavelet coefficient video generated during the video encoding process. Next, the reconstructed displacement d'(i) is calculated by applying an inverse wavelet transform to the reconstructed wavelet coefficients. The reconstructed base mesh m''(i) is obtained by applying inverse quantization to the reconstructed quantized base mesh m'(i). The reconstructed deformed mesh DM(i) is obtained by refining m''(i) and applying the reconstructed displacement d''(i) to its vertices.

[0061] Since the quantization step and / or the mesh compression module can be reversible, a reconstructed quantized version of m(i), denoted as m'(i), is computed. If the mesh information is irreversibly encoded and the quantization step is skipped, m(i) will exactly match m'(i).

[0062] As shown in Figure 6, the reconstructed quantized reference base mesh m'(j) is used to predict the current frame base mesh m(i). The preprocessing module 302 described in Figure 3 assumes that m(i) and m(j) are the same. • Number of vertices, Connectivity, • Texture coordinates, and Texture connectivity can be configured to share textures.

[0063] The motion field f(i) is calculated by considering the quantized version of m(i) and the reconstructed quantized base mesh m'(j). Since m'(j) may have a different number of vertices than m(j) (e.g., vertices may be merged / removed), the encoder tracks the transformation applied to m(j) to obtain m'(j) and applies it to m(i) to ensure a one-to-one correspondence between m'(j) and the transformed and quantized version of m(i) denoted as m*(i). The motion field f(i) is calculated by subtracting the quantized position p(i,v) of vertex v in m*(i) from the position p(j,v) of vertex v in m'(j). f(i,v)=p(i,v)-p(j,v)

[0064] Next, the motion field is further predicted and entropy-encoded using the connectivity information of m'(j) (for example, context-adaptive binary arithmetic coding may be used).

[0065] Since the motion field compression process can be reversible, the reconstructed motion field, denoted as f'(i), is computed by applying the motion decoder module 408. Next, the reconstructed quantized base mesh m'(i) is computed by adding the motion field to the positions of m'(j). The remainder of the encoding process is similar to intra-frame encoding.

[0066] Figure 7 shows exemplary interframe decoders for decoding a three-dimensional mesh according to several embodiments.

[0067] The interframe decoder 702 includes components similar to those of the intraframe decoder 502 shown in Figure 5. However, instead of receiving a directly encoded base mesh, the interframe decoder 702 reconstructs the base mesh of the current frame based on the motion vector of the displacement field relative to the reference frame. For example, the interframe decoder 702 includes a motion field / vector decoder 704 and a base mesh reconstruction module 706.

[0068] Similar to the intraframe decoder, the interframe decoder 702 divides the bitstream into three separate substreams, namely, • Motion subnet, • Displacement subnet, and • Separate into attribute substreams.

[0069] The motion subnet is decoded by applying the motion decoder 704. The proposed scheme is independent of which codec / standard is used to decode the motion information. For example, any motion decoding scheme can be used. The decoded motion is then optionally added to the decoded reference quantized base mesh m'(j) to produce a reconstructed quantized base mesh m'(i), i.e., the already decoded mesh in instance j can be used to predict the mesh in instance i. Then, by applying inverse quantization to m'(i), the decoded base mesh m''(i) is produced.

[0070] The displacement substream and attribute substream are decoded in the same manner as the intraframe decoding process described with respect to Figure 5. The decoded mesh M''(i) is also reconstructed in the same manner.

[0071] The dequantization and reconstruction processes are not prescriptive and can be implemented in various ways and / or combined with the rendering process. Controlling the encoding to avoid mesh splitting and cracking.

[0072] In some embodiments, the mesh may be subdivided into sets of patches (e.g., subparts), and the patches may potentially be grouped into groups of patches, such as sets of patches / tiles. In such embodiments, different coding parameters (e.g., subdivision, quantization, wavelet transform, coordinate system, etc.) may be used to compress each patch or group of patches. In some embodiments, irreversible coding may be used for boundary vertices to avoid cracks at patch boundaries. Additionally, quantization of wavelet coefficients for boundary vertices may be disabled, along with the use of a local coordinate system for boundary vertices.

[0073] In some embodiments, scalability may be supported at different levels. For example, temporal scalability may be achieved through temporal subsampling and frame reordering. Quality and spatial scalability may also be achieved by using different mechanisms for geometry / vertex attribute data and attribute map data. Region of interest (ROI) reconstruction may also be supported. For example, the encoding process described in the previous section may be configured to encode ROIs with higher resolution and / or higher quality for geometry, vertex attributes, and / or attribute map data. This is particularly useful for providing content of higher visual quality (e.g., higher quality for faces versus the rest of the body) under tight bandwidth and complexity constraints. Priority / importance / spatial / bounding box information may be associated with patches, patch clusters, tiles, network abstraction layer (NAL) units, and / or sub-bitstreams in a manner that enables the decoder to adaptively decode subsets of the mesh based on viewing frustum, power budget, or terminal capabilities. Note that any combination of such coding units may be used together to achieve such functionality. For example, NAL units and sub-bitstreams may be used together.

[0074] In some embodiments, temporal and / or spatial random access may be supported. Temporal random access can be achieved by introducing IRAPs (Intra Random Access Points) into different substreams (e.g., attribute atlas, video, mesh, motion, and displacement substreams). Spatial random access can be supported through the definition and use of tiles, subpictures, patch groups, and / or patches, or any combination thereof of coding units. To assist the decoder in determining which units need to be decoded, metadata describing the layout and relationships between different units may also be generated and included in the bitstream.

[0075] As mentioned above, it has various functions, namely, • Spatial random access, • Adaptive quality allocation (for example, foveal compression that allocates higher quality to the face compared to the body of a human model), • Access to Regions of Interest (ROI) • Coding unit-level metadata (e.g., object description, bounding box information), • Spatial scalability and quality scalability, and • It can support adaptive streaming and decoding (e.g., streaming / decoding high-priority regions first).

[0076] The disclosed compression schemes enable various coding units (e.g., patches, patch groups, and tiles) to be compressed with different coding parameters (e.g., subdivision scheme, subdivision iteration count, quantization parameters, etc.) which may introduce compression artifacts (e.g., cracks between patch boundaries). In some embodiments, as further described below, efficient strategies (e.g., efficient in terms of computational complexity, compression efficiency, power consumption, etc.) can be used that enable the scheme to handle different coding unit parameters without introducing artifacts. Mesh tiles

[0077] A mesh can be segmented into sets of tiles (e.g., parts / segments), which can be independently encoded and decoded. Vertices / edges shared by two or more tiles are duplicated as shown in Figures 8A / 8B. Note that mesh 800 is divided into two tiles 850 and 852 by duplicating three vertices {V0,V1,V2} and two shared edges {(V0,V1),(V1,V2)}. In other words, when a mesh is divided into two tiles, each of the two tiles contains the vertices and edges that were previously a set of vertices and edges in the combined mesh, and thus the vertices and edges are duplicated within the tile. Submesh

[0078] Each tile can be further segmented into a set of submeshes, and these submeshes can be encoded while taking advantage of the dependencies between them. For example, Figure 9 shows an example of mesh 900 segmented into two tiles (902 and 904) containing three and four submeshes, respectively. For example, tile 902 contains submeshes 0, 1, and 2, and tile 904 contains submeshes 0, 1, 2, and 3.

[0079] The submesh structure is, • For each face of the mesh, explicitly encode a face-specific integer attribute that indicates the index of the submesh to which it belongs, or Connectivity can be defined by either positional connectivity, texture coordinate connectivity, or both, and by implicitly detecting the connected components (CCs) of a mesh by treating each CC as a submesh. Mesh vertices are traversed from neighborhood to neighborhood, which allows for the detection of CCs in a deterministic manner. The index assigned to a CC starts at 0 and is incremented by 1 each time a new CC is detected. patch

[0080] A patch is a set of submeshes. The encoder may explicitly store the index of the submesh to which each patch belongs. In certain embodiments, a submesh may belong to one or more patches (e.g., associating metadata with overlapping parts of a mesh). In other embodiments, a submesh may belong to only a single patch. Vertices located at the boundaries between patches do not overlap. Patches are also encoded using correlations between them, and therefore they cannot be encoded / decoded independently.

[0081] The list of submeshes associated with a patch can be used for various strategies, namely, Entropy coding • Intraframe prediction • Intraframe prediction • It can be encoded using local submesh indices (i.e., smaller ranges). Patch group

[0082] A patch group is a collection of patches. Patch groups are particularly useful for storing parameters shared by those patches, or for enabling unique handles that can be used to associate metadata with those patches.

[0083] In some embodiments, the following may be used to support the use of different encoding parameters for each patch. The relationship between the submesh and the patch may be used so that a patch ID is assigned to each face of the base mesh. Vertices and edges belonging to a single submesh are assigned the patch ID to which the submesh belongs. Vertices and edges located at the boundary of two or more patches are assigned to all corresponding patches. When applying the subdivision method, the decision of whether or not to subdivide an edge is made by considering the subdivision parameters of all patches to which the edge belongs.

[0084] For example, Figures 11A–11C illustrate adaptive subdivision based on shared edges. Based on the subdivision decisions made in Figures 11A–11C (for example, assume there are edges belonging to two patches, Patch0 and Patch1, with Patch0 having a subdivision iteration count of 0 and Patch1 having a subdivision iteration count of 2), the shared edges are subdivided twice (i.e., taking the maximum subdivision count), and the edges are subdivided using the subdivision schemes shown in Figures 10A–10D. For example, based on the subdivision decisions associated with different edges, one of the adaptive subdivision schemes described in Figures 10A–10D is applied. Figures 10A–10D show how a triangle is subdivided when three, two, one, or zero of its edges should be subdivided, respectively. The vertices created after each subdivision iteration are assigned to the patches of their parent edges.

[0085] When applying quantization to wavelet coefficients, the quantization parameters used for a vertex are selected based on the quantization parameters of all patches to which the vertex belongs. For example, suppose a patch belongs to two patches, Patch0 and Patch1. Patch0 has the quantization parameter QP1. Patch1 has the quantization parameter QP2. The wavelet coefficients associated with a vertex are quantized using the quantization parameter QP = min(QP1, QP2). Mesh tiles

[0086] To support mesh tiling, the encoder can divide the mesh into sets of tiles, which are then independently encoded / decoded by applying any mesh codec, or use a mesh codec that natively supports tiled mesh coding.

[0087] In both cases, the set of shared vertices located on the tile boundary is duplicated. To avoid cracks appearing between tiles, the encoder can do the following: ● Encode the splice information showing the mapping between duplicated vertices as follows: ○ Encode each vertex tag to identify duplicated vertices (by encoding vertex attributes with a mesh codec). ○ For each duplicated vertex, encode the index of the vertex it should be merged into. ● Verify that the decrypted position and vertex attributes associated with the duplicated vertices match exactly. ○ For each vertex, identify duplicate vertices and code this information as vertex attributes using a mesh codec. ○ Apply the adaptive subdivision method described in the previous section to ensure consistent subdivision behavior on tile boundaries. ○ The encoder needs to adjust its encoding parameters to maintain the mapping between duplicated vertices and ensure matching values. ■ Reversibly encode the duplicated vertex positions and vertex attribute values. ■ Disable wavelet transform (transformation bypass mode) ■ Perform a search within the encoded parameter space to determine the set of parameters encoded in the bitstream and guarantee matching positions and attribute values. ■ Do nothing The decoder should reconstruct the tag information for each vertex to identify duplicated vertices. If the encoder signals duplicated vertices with different encoding parameters compared to non-duplicated vertices, the decoder should adaptively switch between the two sets of parameters based on the vertex type (e.g., duplicate vs. non-duplicate). The decoder can apply smoothing and auto-stitching as post-processing based on the signaling provided by the encoder or based on the analysis of the decoded mesh. In certain embodiments, a flag is included for each vertex or patch to enable / disable such post-processing. This flag may include, among other things: ■Signed as an SEI message, ■ Encoded with a mesh codec, or ■ Signaling is performed using Atlas subbitstreams.

[0088] In another embodiment, the encoder can duplicate regions of the mesh and store them in multiple tiles. This serves the following purposes, namely, Error recovery capability, • Guard band, and • Can be done for seamless / adaptive streaming.

[0089] Figure 12 shows exemplary components of a dynamic mesh compressed bitstream according to several embodiments.

[0090] In some embodiments, the compressed bitstream may include multiple sub-bitstreams, such as a base mesh sub-bitstream 1220, a displacement / video sub-bitstream 1240, and an atlas sub-bitstream 1260. In some embodiments, the atlas sub-bitstream 1260 includes an atlas 1262 containing information to match the encoded displacement values ​​in video frames 1242-1244 to the corresponding subdivision locations of the base mesh, which can be encoded using sub-mesh data units 1222, 1224, and 1226. Texture coordinates and / or texture connectivity may also be signaled in the atlas sub-bitstream 1260.

[0091] In some embodiments, the compressed bitstream may include information for individually finding and analyzing the submesh data units. For example, SEI messages 1282, 1284, and 1286 may indicate the byte size and vertex count of the respective submesh data units 1222, 1224, and 1226. Alternatively, such information may be encoded in the NAL unit headers of the submesh data units 1222, 1224, and 1226.

[0092] Figure 13 is a flowchart illustrating the process of reconstructing a dynamic mesh using a base mesh and displacement information in several embodiments, where only a portion of the base mesh is used, corresponding to a subset of all submeshes of the base mesh.

[0093] In block 1302, the decoder receives a bitstream of a compressed dynamic mesh, the bitstream comprising submesh data units contained in the base mesh subbitstream of the bitstream and displacement information contained in one or more additional subbitstreams of the bitstream.

[0094] In block 1304, the decoder receives browsing information indicating one or more focal areas for viewing the dynamic mesh. Also in block 1306, the decoder determines, based on the browsing information, which submeshes to be included in the reconstructed version of the dynamic mesh. For example, only a portion of the dynamic mesh may currently be viewed. In such a situation, the observation information may be used to select which submeshes should be used when reconstructing the dynamic mesh.

[0095] In block 1308, the decoder analyzes the base mesh subbitstream to find only the submesh data units necessary for reconstruction. For example, only the submesh data units corresponding to the portion of the dynamic mesh that should be included in the reconstructed version of the dynamic mesh may be extracted from the base mesh subbitstream. In other embodiments, even when all submeshes are to be reconstructed, different prediction techniques may be applied to predict different submeshes of the base mesh. Thus, the ability to analyze the base mesh subbitstream and identify individual submeshes (based on the submesh data units) can be used in such situations.

[0096] In block 1310, at least a portion of the base mesh is reconstructed using submesh information from the submesh data units analyzed from the base mesh subbitstream in block 1308. Furthermore, in block 1312, displacement information is applied to the refinement locations of the portion of the base mesh reconstructed in block 1310. The application of displacement information further reconstructs the dynamic mesh to include both the vertices of the base mesh and additional vertices added to the base mesh's refinement locations (or the portion of the base mesh and the refinement locations corresponding to the reconstructed submesh).

[0097] Figure 14 shows different prediction modes that can be applied to predict various parts (e.g., submeshes) of the base mesh according to several embodiments.

[0098] For example, reconstructing a submesh of a base mesh (as performed in block 1310 of Figure 13) may include performing intra-prediction and / or inter-prediction. For example, in block 1402, the vertices of a given submesh of the base mesh are predicted using intra-prediction based on the vertex values ​​of other vertices at the same moment in the time frame (e.g., of the same submesh or a different submesh at that moment in the time frame). Alternatively, in block 1404, the vertices of a given submesh of the base mesh are predicted using inter-prediction based on the vertex values ​​of the corresponding vertices at other moments in the time frame, such as a preceding moment in the time frame. Also, various inter-prediction techniques may be signaled for different submeshes. For example, in block 1406, an inter-prediction technique that copies motion vectors from a reference frame is signaled. Alternatively, in the case of one or more aspects of motion vector prediction (such as the X, Y, or Z components of the motion vector), a skip flag may be signaled. In such cases, the prediction may be skipped, and the component values ​​may be encoded in the residual values. Also, in block 1410, a flag may be used to indicate that there is no motion vector prediction, in which case the prediction may be a zero-value motion vector, and deviations from "no motion" may be signaled using the residual values. Also, in block 1412, a flag may be encoded to indicate that the adjacent motion vector for adjacent vertices should be used for the prediction. Further details regarding the signaling of prediction types are provided below, following the description of the encoding process in Figure 15.

[0099] Figure 15 is a flowchart showing the process for compressing a dynamic mesh using submesh data units according to several embodiments.

[0100] In block 1502, the encoder accesses the dynamic mesh to be compressed / encoded. In block 1504, the encoder generates a base mesh for the mesh being encoded. Then, in block 1504, submeshes are determined relative to the base mesh. Also, in block 1508, displacement values ​​are determined. Then, in block 1510, the compressed dynamic mesh is signaled, and submesh data units are used to modularly encode each submesh of the base mesh. The submesh data unit includes a bit count that allows individual submesh data units to be analyzed from the base mesh subbitstream, and a vertex count that allows vertex indices to be constructed separately for each submesh.

[0101] In some embodiments, the size of the submesh data, excluding the size of the base mesh subbitstream and submesh header, can be explicitly signaled, as can the number of vertices in the current submesh. Submeshes within the base mesh subbitstream can also be inter-predicted. Reference frames can be listed by a specified syntax element, bmesh_ref_list_struct. In submesh inter-prediction, motion vectors can be derived from the motion vectors of other vertices within the same submesh. This can be explicitly indicated by a per-vertex flag. This functionality can be turned off by a management flag at the beginning of the inter-prediction data unit. Motion vectors can also be derived as 0 when a specified per-vertex flag indicates so (e.g., skipped, or in other words, a vertex in a later submesh may have the same location as the corresponding vertex in the reference submesh, e.g., no motion). In some embodiments, these flags can be signaled as prediction modes instead of flags.

[0102] In the base mesh subbitstream, inter-frame prediction between mesh frames is possible using a reference frame list structure. B.1.1 Base Mesh Submesh Layer RBSP Syntax

[0103] In the base mesh sub-bitstream, the sub-mesh is delivered in NAL units as shown below. [Table 1] NumBytesRbsp is equal to NumBytesInNalUnit - 2, and SubMeshUnitSize is equal to NumBytesRbsp - submesh_header_size. submesh_header_size indicates the size occupied by submesh_header() in bytes. B.1.2 Base Mesh Submesh Data Unit Syntax [Table 2] B.1.3 Base Mesh Intra Submesh Data Unit Syntax

[0104] To use different static mesh codecs based on the profile and signal the 4cc codec, sdu_intra_sub_mesh_unit() can be abstracted.

[0105] [Table 3] sismu_intra_unit(subMeshID,sismu_intra_unit_size)) contains a portion of the mesh data of size sismu_intra_unit_size. The syntax of sismu_intra_unit is defined by the value of bmptl_profile_codec_group_idc or the associated 4CC code, as defined by the component codec mapping SEI message. When sismu_intra_unit_size is signaled, the bit count required to signal it is signaled in the frame parameter set.

[0106] In another embodiment, sismu_intra_unit_size is not signaled but is inferred as unit_size. B.1.4 Base Mesh Inter-Submesh Data Unit Syntax [Table 4] sismu_inter_unit(subMeshID, sismu_inter_vertex_count, sismu_inter_unit_size) contains a portion of the mesh data of size sismu_inter_unit_size. The syntax of sismu_inter_unit is defined by the value of bmptl_profile_codec_group_idc or the associated 4CC code, as defined by the component codec mapping SEI message. When sismu_inter_unit_size is signaled, the bit count required to signal it is signaled in the frame parameter set.

[0107] In another embodiment, sismu_inter_unit_size is not signaled but is inferred as unit_size-(size of sismu_ref_index and sismu_inter_vertex_count). B.1.5 Base Mesh Inter Sub Mesh Data Unit Default Syntax (1)

[0108] If the profile and 4cc codec ID indicate that the motion codec is the default internal motion codec, then sdu_inter_sub_mesh_unit_default() is signaled to sismu_inter_unit(). In one embodiment, vertexCount can be derived as the number of vertices in the reference mesh. [Table 5] B.1.6 Base Mesh Inter-Sub Mesh Data Unit Default Syntax (2)

[0109] When sdu_inter_sub_mesh_unit_default() is used, motion vectors may be signaled based on a flag that is signaled per vertex. sismu_copied_mv_flag may be coded by a fixed length of 1 bit or by context-based arithmetic coding. [Table 6] B.1.7 Base Mesh Inter-Sub Mesh Data Unit Default Syntax (3)

[0110] In another embodiment, a set of sismu_copied_mv_flags can be signaled before the residuals are signaled. [Table 7] B.1.8 Default syntax for base mesh inter-submesh data unit with skipped motion vectors

[0111] When sdu_inter_sub_mesh_unit_default() is used, the motion vector may be derived as a zero vector based on a flag signaled per vertex. sismu_skip_mv_flag may be coded by a fixed length of 1 bit or by context-based arithmetic coding. [Table 8]

[0112] In another embodiment, a set of sismu_skip_mv_flags with size vertexCount may be signaled before the residuals are signaled.

[0113] When sismu_skip_mv_flag and sismu_copied_mv_flag are signaled together, either of the two can come first. B.1.9 Default syntax for base mesh inter-submesh data unit with skip motion vectors and copy motion vectors

[0114] In another embodiment, when both the skip motion vector and the copy motion vector are enabled, the skip motion vector flag does not need to be signaled, but is derived. [Table 9] B.1.10 Default Syntax for Base Mesh Inter-Submesh Data Units Using Motion Vector Grouping

[0115] V-DMC also employs motion vector prediction grouping. The group size bmsps_inter_mesh_motion_group_size_minus1 is signaled by the sequence parameter set. In sdu_inter_sub_mesh_unit_default, (bmsps_inter_mesh_motion_group_size_minus1+1) motion vectors have the same motion vector prediction mode. [Table 10]

[0116] In another embodiment, the (bmsps_inter_mesh_motion_group_size)-motion vector has the same motion vector prediction mode.

[0117] In another embodiment, (bmsps_inter_mesh_motion_group_size) can be signaled per submesh instead of a sequence parameter set. B.1.11 Combination of motion vector grouping and motion vector prediction skipping

[0118] When motion vector grouping is used with skipped and / or copied motion vectors, the skipped and copied motion vectors may be counted toward the number of groups. For example, if the motion vector group size is 16, then 16 motion vectors from the first vertex belong to group 0. In another embodiment, skipped and copied motion vectors are not counted toward the group. For example, if the motion vector group size is 16, then 16 motion vectors from the first vertex, excluding the skipped or copied motion vectors, belong to group 0. B.1.9 can also be combined equivalently with motion vector groups. [Table 11-1] [Table 11-2] B.1.12 Extended Motion Vector Modes

[0119] In another embodiment, when sdu_inter_sub_mesh_unit_default() is used, the motion vector can be calculated based on the mode assigned to each vertex instead of the indication flag. The association of names to sismu_mv_pred_mode can be as follows: [Table 12]

[0120] When sismu_mv_pred_mode is MV_COPIED, the motion vector is not signaled but is derived from other vertices (motion vectors) in the same submode. When sismu_mv_pred_mode is MV_SKIP, the motion vector is not signaled and is derived as a 0 vector. [Table 13]

[0121] The binarization of sismu_mv_pred_mode may change depending on the dominance flag. [Table 14]

[0122] In another embodiment, the binarization of sismu_mv_pred_mode can be performed as follows: [Table 15]

[0123] When they are arithmetic coded, the context used for each bin may differ based on sismu_skip_mv_enabled_flag and sismu_copied_mv_present_flag. B.1.13 Extended motion vector modes (2)

[0124] In another embodiment, before the residuals are signaled, it may be signaled that the motion vector indication is copied (MV_PRED_COPIED) or that the motion vector for a vertex is derived as a zero vector (MV_PRED_SKIP). sismu_mv_skip_type may be as follows: [Table 16]

[0125] In this case, the prediction mode sismu_mv_pred_mode can be 0 or 1. The MvPredMode derived from sismu_mv_pred_mode and sismu_mv_skip_type can be as follows: [Table 17] [Table 18]

[0126] In the base mesh subbitstream, bmesh_profile_toolset_constraints_information() is signaled to indicate restrictions on the tools used for the bitstream. The flag MV_PRED_COPIED, which indicates motion vector copying, can be signaled in bmesh_profile_toolset_constraints_information(). When bmptc_mv_copy_enabled_flag is 0, sismu_copied_mv_present_flag should always be 0 in the compliant bitstream, and the associated function is not called.

[0127] Additionally, bmesh_profile_toolset_constraints_information() may have a flag indicating whether or not skip motion vectors are used. When bmptc_skip_mv_enabled_flag is 0, sismu_skip_mv_enable_flag should always be 0 in the compliant bitstream, and the associated function is not called. [Table 19]

[0128] The bmesh_profile_toolset_constraints_information() function may contain a flag indicating whether motion vector grouping is used. If bmptc_mv_group_enabled_flag is 0, then bmsps_inter_mesh_motion_group_size_minus1 must always be 0 in the adapted bitstream. [Table 20]

[0129] In some embodiments, the motion vector prediction mode MvPredMode is derived from sismu_mv_pred_mode[subMeshID][v] and others as follows: MvPredMode[subMeshID][v]=sismu_mv_pred_mode[subMeshID][v]

[0130] When the above-mentioned indication flag is signaled, MvPreMode can be updated as follows: if(sismu_copied_mv_flag[subMeshID][v]) MvPredMode[subMeshID][v]=MV_COPIED if(sismu_skip_mv_flag[subMeshID][v]) MvPredMode[subMeshID][v]=MV_SKIP

[0131] When ismu_copied_mv_flag is signaled together, one of two things can be executed first.

[0132] Instead of signaling two types of flags for copied motion vectors and skipped motion vectors, only one flag, mv_signalled_flag, is used, and firstVertexIndexDuplicated is used to check whether the motion vector is derived as 0 or is the same as the reference vertex. The derivation is conceptually as follows: if(!sismu_mv_signalled_flag[subMeshID][v]){ if(firstVertexIndexDuplicated(v)==-1) MvPredMode[subMeshID][v]=MV_SKIP others MvPredMode[subMeshID][v]=MV_COPIED }

[0133] When motion vector grouping is used, the motion vector prediction mode of the v-th vertex is the same as that signaled for the group to which the vertex belongs. If sismu_mv_pred_mode[subMeshID][v] is pre-set to -1 for all v, then sismu_mv_pred_mode[subMeshID][v] can be conceptually induced as follows: lastValidIndex=0 for(v=0;v <vertexCount;v++){ if(sismu_mv_pred_mode[subMeshID][v]!=-1)lastValidIndex=v others sismu_mv_pred_mode[subMeshID][v]=sismu_mv_pred_mode[subMeshID][lastValidIndex] }

[0134] Based on MvPredMode, currentSubmeshMotionVectors is derived as follows: If MvPredMode is equal to MV_SKIP, currentSubmeshMotionVectors[v][k]=0 Instead, if MvPredMode is equal to MV_COPIED, currentSubmeshMotionVectors[v][k]=currentSubmeshMotionVectors[vRef][k] Instead, if MvPredMode is equal to MV_PRED_NONE, currentSubmeshMotionVectors[v][k]=VertexMotionVectorResiduals[v][k] Instead, if MvPredMode is equal to MV_PRED_NEIGHB, currentSubmeshMotionVectors[v][k]=VertexMotionVectorResiduals[v][k]+currentSubmeshPredictedMotionVectors[v][k]

[0135] vRef is derived as follows: vRef = firstVertexIndexDuplicated(v) Here firstVertexIndexDuplicated(v){ for(i=0;i <v;i++){ if(referenceSubmeshVertexPositions[i]==referenceSubmeshVertexPositions[v]) return i } return-1 } If vRef=-1, currentSubmeshMotionVectors[v][k] is set to 0.

[0136] If referenceSubmeshVertexPositions[v][k] represents the k-th component of the geometric position of the v-th vertex in the reference submesh, the reconstructed geometric position is derived as follows:

[0137] currentSubmeshVertexPositions[v][k]= referenceSubmeshVertexPositions[v][k]+currentSubmeshMotionVectors[v][k] If vertexCount is greater than the number of vertices in the reference frame, the motion vector currentSubmeshMotionVectors[v] (where v is greater than the number of vertices in the reference frame) is estimated to be 0. Exemplary computer system

[0138] Figure 16 shows an exemplary computer system 1600 that may implement an encoder or decoder, or any other component described herein (for example, any of the components described above with reference to Figures 1 to 15), according to several embodiments. The computer system 1600 may be configured to perform any or all of the embodiments described above. In different embodiments, the computer system 1600 may be any of various types of devices, including but not limited to personal computer systems, desktop computers, laptops, notebooks, tablets, slates, pads, or netbook computers, mainframe computer systems, handheld computers, workstations, network computers, cameras, set-top boxes, mobile devices, consumer devices, video game consoles, handheld video game devices, application servers, storage devices, televisions, video recording devices, peripheral devices such as switches, modems, and routers, or any type of computing device or electronic device in general.

[0139] Various embodiments of point cloud encoders or decoders as described herein may be implemented in one or more computer systems 1600 that can interact with various other devices. Note that any component, action, or functionality described above with respect to Figures 1-15 may be implemented on one or more computers configured as the computer system 1600 of Figure 16, according to various embodiments. In the illustrated embodiment, the computer system 1600 includes one or more processors 1610 coupled to system memory 1620 via an input / output (I / O) interface 1630. The computer system 1600 further includes a network interface 1640 coupled to the I / O interface 1630, and one or more input / output devices 1650, such as a cursor control device 1660, a keyboard 1670, and a display 1680(one or more). In some cases, the embodiment may be implemented using a single instance of the computer system 1600, and in other embodiments, it is intended that multiple such systems, or multiple nodes constituting the computer system 1600, may be configured to host different parts or instances of the embodiment. For example, in one embodiment, some elements may be implemented via one or more nodes of a computer system 1600 that are different from the nodes that implement other elements.

[0140] In various embodiments, the computer system 1600 may be a uniprocessor system including one processor 1610, or a multiprocessor system including several processors 1610 (e.g., two, four, eight, or another suitable number). The processors 1610 may be any suitable processor capable of executing instructions. For example, in various embodiments, the processors 1610 may be general-purpose or embedded processors implementing one of various instruction set architectures (ISAs), such as x86, PowerPC, SPARC, or MIPS ISA, or any other suitable ISA. In a multiprocessor system, each of the processors 1610 may, but not necessarily, implement the same ISA in general.

[0141] The system memory 1620 may be configured to store point cloud compression or point cloud reconstruction program instructions 1622 and / or sensor data accessible by the processor 1610. In various embodiments, the system memory 1620 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory. In the illustrated embodiment, the program instructions 1622 may be configured to implement an image sensor control application incorporating any of the functions described above. In some embodiments, the program instructions and / or data may be received, transmitted, or stored on different types of computer-accessible media, or on similar media separate from the system memory 1620 or the computer system 1600. Although the computer system 1600 is described as implementing the functions of the functional blocks in the previous figure, any of the functions described herein may be implemented via such a computer system.

[0142] In one embodiment, the I / O interface 1630 may be configured to coordinate I / O traffic between the processor 1610, the system memory 1620, and any peripheral devices within the device, including other peripheral interfaces such as the network interface 1640 or input / output device 1650. In some embodiments, the I / O interface 1630 may perform any necessary protocols, timing, or other data conversions to convert data signals from one component (e.g., the system memory 1620) into a format suitable for use by another component (e.g., the processor 1610). In some embodiments, the I / O interface 1630 may include support for devices connected via various types of peripheral buses, such as a PCI bus standard or a variation of the Universal Serial Bus (USB) standard. In some embodiments, the functionality of the I / O interface 1630 may be divided into two or more separate components, such as a northbridge and a southbridge. Also, in some embodiments, some or all of the functionality of the I / O interface 1630, such as the interface to the system memory 1620, may be directly integrated into the processor 1610.

[0143] The network interface 1640 may be configured to allow data to be exchanged between the computer system 1600 and other devices attached to the network 1685 (e.g., carrier or agent devices), or between nodes of the computer system 1600. In various embodiments, the network 1685 may include, but is not limited to, one or more networks, including a local area network (LAN) (e.g., Ethernet or enterprise network), a wide area network (WAN) (e.g., the Internet), a wireless data network, several other electronic data networks, or several combinations thereof. In various embodiments, the network interface 1640 may support communication over common wired or wireless data networks, such as any suitable type of Ethernet network. It may also support communication over telecommunications / telephone networks, such as analog voice networks or digital fiber optic communication networks, over storage area networks, such as Fibre Channel SANs, or over any other suitable type of network and / or protocol.

[0144] In some embodiments, the input / output device 1650 may include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for inputting or accessing data by one or more computer systems 1600. Multiple input / output devices 1650 may reside within the computer system 1600 or be distributed across various nodes of the computer system 1600. In some embodiments, similar input / output devices may be isolated from the computer system 1600 and may interact with one or more nodes of the computer system 1600 via wired or wireless connections, such as a network interface 1640.

[0145] As shown in Figure 16, memory 1620 may include program instructions 1622 that can be processor-executable to implement any of the elements or actions described above. In one embodiment, the program instructions can perform the methods described above. In other embodiments, different elements and data may be included. Note that the data may include any of the data or information described above.

[0146] Those skilled in the art will understand that the computer system 1600 is merely illustrative and not intended to limit the scope of embodiments. In particular, the computer system and devices may include any combination of hardware or software capable of performing the indicated functions, including computers, network devices, internet equipment, PDAs, wireless telephones, pagers, etc. The computer system 1600 may also be connected to other devices not shown, or it may instead operate as a standalone system. In addition, the functions provided by the illustrated components may, in some embodiments, be combined with fewer components or distributed across additional components. Similarly, in some embodiments, some functionality of the illustrated components may not be provided, and / or other additional functionality may be available.

[0147] Those skilled in the art will also understand that while various items are shown to be stored in memory or on storage while in use, these items or parts thereof may be transferred between memory and other storage devices for memory management and data integrity purposes. Alternatively, in other embodiments, some or all of the software components may run in memory on another device and communicate with the illustrated computer system via intercomputer communication. Some or all of the system components or data structures may also be stored as (e.g., instructions or structured data) on a computer-accessible medium or portable article to be read by a suitable drive, as various examples of which have been described above. In some embodiments, instructions stored on a computer-accessible medium separate from the computer system 1600 may be transmitted to the computer system 1600 via a transmission medium or signal, such as an electrical, electromagnetic, or digital signal, transmitted via a communication medium such as a network and / or wireless link. Various embodiments may further include receiving, transmitting, or storing instructions and / or data executed on a computer-accessible medium in accordance with the preceding description. Generally speaking, computer-accessible media may include non-temporary computer-readable storage media or memory media such as magnetic media or optical media, for example, disks or DVD / CD-ROMs, RAM (e.g., SDRAM, DDR, RDRAM, SRAM, etc.), volatile or non-volatile media such as ROM. In some embodiments, computer-accessible media may include transmission media or signals such as electrical signals, electromagnetic signals, or digital signals transmitted over communication media such as networks and / or wireless links.

[0148] The methods described herein may be implemented in software, hardware, or a combination thereof in different embodiments. In addition, the order of the blocks of the method may be changed, and various elements may be added, rearranged, combined, omitted, modified, etc. Various modifications and changes may be made that would be obvious to a person skilled in the art who would benefit from this disclosure. The various embodiments described herein are illustrative and not limiting. Many variations, modifications, additions, and improvements are possible. Thus, for a single component described herein, multiple versions may be provided. The boundaries between the various components, operations, and data stores are arbitrary to some extent, and certain operations are shown in the context of a particular exemplary configuration. Other assignments of functions are conceivable and may be included within the scope of the claims that follow. Finally, structures and functions presented as separate components in exemplary configurations may be realized as combined structures or components.

Claims

1. One or more non-temporary computer-readable storage media for storing program instructions, wherein when the program instructions are executed using one or more processors, the one or more processors... Receiving a bitstream for a dynamic mesh, wherein the bitstream is A base mesh subbitstream containing information about a base mesh, wherein the base mesh subbitstream is signaled using a plurality of submesh data units corresponding to each of the plurality of submeshes included in the base mesh, One or more additional sub-bitstreams containing displacement information for the displacement applied to the subdivision locations of the base mesh, To include, to have them do, The base mesh subbitstream is analyzed, and based on the information signaled in each of the submesh data units, The bit count for information encoding each of the aforementioned submeshes, The vertex count of each of the aforementioned submeshes, To make a decision, Using the bit count and vertex count analyzed from the base mesh subbitstream, at least a portion of the multiple submeshes signaled in the base mesh subbitstream are reconstructed. Apply at least a portion of the displacement information to the subdivided locations of the reconstructed portion of the submesh. One or more non-temporary computer-readable storage media.

2. The program instructions, when executed using the one or more processors, cause the portion of the plurality of submeshes to be reconstructed in an order different from the order in which the corresponding submesh data units are signaled in the basemesh subbitstream, according to one or more non-temporary computer-readable storage media according to claim 1.

3. When the program instruction is executed using one or more processors, it further sends instructions to the one or more processors. The system receives browsing information indicating one or more focus areas of the dynamic mesh, which are the focus for viewing the reconstructed version of the dynamic mesh. To identify a submesh data unit corresponding to a submesh located in one or more of the aforementioned focal areas, Reconstructing at least a portion of the plurality of submeshes signaled in the base mesh subbitstream using the bit count and vertex count analyzed from the base mesh subbitstream, and applying at least a portion of the displacement information to the subdivision locations of the reconstructed portion of the submesh, is performed on the submesh corresponding to the identified submesh data unit without requiring all submeshes signaled in the base mesh subbitstream to be reconstructed. One or more non-temporary computer-readable storage media according to claim 1.

4. To reconstruct at least a portion of the plurality of submeshes signaled in the base mesh subbitstream using the bit count and vertex count analyzed from the base mesh subbitstream, the program instructions, when executed on or across the one or more processors, cause the one or more processors to: Intra-prediction within a point in a time frame to determine the vertex position of the vertices of each of the reconstructed submeshes, or Interpretation using a preceding reference frame to determine the vertex position of the vertices of each of the reconstructed submeshes, One or more non-temporary computer-readable storage media according to claim 1, which cause one or more of the following to be performed.

5. For the aforementioned inter prediction, Copy motion vector indicator, or Skipping one or more aspects of the motion vector prediction indicator, One or more non-temporary computer-readable storage media according to claim 4, wherein one or more of the indicators are signaled for each of the inter-predicted vertices.

6. For the aforementioned inter prediction, Motionless vector prediction indicator, or An indicator that motion vector prediction should be based on motion vectors determined for adjacent vertices. One or more non-temporary computer-readable storage media according to claim 5, wherein one or more of the additional indicators are further signaled for one or more of the inter-predicted vertices.

7. The base mesh subbitstream is The system further comprises a toolset constraint indicator which signals that only a subset of the entire set of available predictors should be used for interprediction for a portion of the base mesh subbitstream, and different binarizations which are used to signal the predictors for the subset when the toolset constraint indicator is signaled. One or more non-temporary computer-readable storage media according to claim 6.

8. The toolset constraint indicator is signaled using one or more invalidation flags for one or more types of motion vector predictors or motion vector copies, one or more non-temporary computer-readable storage media according to claim 7.

9. To reconstruct at least a portion of the plurality of submeshes signaled within the base mesh subbitstream, the program instruction, when executed using one or more processors, further instructs the one or more processors to: For each of the vertices of the reconstructed submesh, determine the motion vector prediction mode used to predict the vertex position of that individual vertex. Apply the signaled residual values ​​to the predicted peak values. One or more non-temporary computer-readable storage media according to claim 1.

10. One or more non-temporary computer-readable storage media according to claim 9, wherein each set of vertices of the portion of the submesh to be reconstructed is grouped, and the prediction information is signaled differently for different groupings.

11. One or more non-temporary computer-readable storage media according to claim 9, wherein different prediction modes are signaled to predict different vertex values ​​for vertices that fall within the same point in a time frame for the dynamic mesh.

12. One or more non-temporary computer-readable storage media according to claim 9, wherein different flag schemes are used by each of the submesh data units to signal the different prediction modes for each different submesh corresponding to each of the submesh data units.

13. The bit count of the information encoding each of the submeshes and the vertex count of each of the submeshes are signaled in a supplemental enhancement information (SEI) message included in the bitstream, one or more non-temporary computer-readable storage media according to claim 1.

14. The bit count of the information encoding each of the submesh and the vertex count of each of the submesh are signaled in the header of a network abstraction layer (NAL) unit included in the base mesh subbitstream, one or more non-temporary computer-readable storage media according to claim 1.

15. The bit count for the information encoding each of the submesh and the vertex count for each of the submesh are signaled in a decoder type-independent format used to reconstruct the dynamic mesh, one or more non-temporary computer-readable storage media according to claim 1.

16. One or more non-temporary computer-readable storage media for storing program instructions, wherein when the program instructions are executed using one or more processors, the one or more processors... Generate a base mesh for the compressed dynamic mesh, Determine the displacement information applied to the subdivided locations of the base mesh. Determine the submesh that represents the portion of the base mesh, A base mesh subbitstream including a plurality of submesh data units corresponding to each of the submeshes of the base mesh, One or more additional subbitstreams containing the displacement information of the displacement applied to the subdivision locations of the base mesh, To signal, A bit count for information encoding each of the submesh data units corresponding to each of the submesh data units, The vertex count of each submesh corresponding to each submesh data unit, This is signaled to the submesh data unit, One or more non-temporary computer-readable storage media.

17. The information contained in the submesh data unit further includes individual frame IDs of the point-in-time frames of the dynamic mesh to which each submesh data unit belongs, one or more non-temporary computer-readable storage media according to claim 16.

18. The bit count of the information encoding each of the submeshes and the vertex count of each of the submeshes are signaled in a supplemental enhancement information (SEI) message included in the bitstream, one or more non-temporary computer-readable storage media according to claim 16.

19. The bit count of the information encoding each of the submeshes and the vertex count of each of the submeshes are signaled in the header of a network abstraction layer (NAL) unit included in the base mesh subbitstream, one or more non-temporary computer-readable storage media according to claim 16.

20. It is a device, The display and Memory for storing program instructions, A system comprising one or more processors, wherein when a program instruction is executed using the one or more processors, the one or more processors, Receiving a bitstream for a dynamic mesh, wherein the bitstream is A base mesh subbitstream containing information about a base mesh, wherein the base mesh subbitstream is signaled using a plurality of submesh data units corresponding to each of the plurality of submeshes included in the base mesh, One or more additional sub-bitstreams containing displacement information for the displacement applied to the subdivision locations of the base mesh, To include, to have them do, The base mesh subbitstream is analyzed, and based on the information signaled in each of the submesh data units, The bit count for information encoding each of the aforementioned submeshes, The vertex count of each of the aforementioned submeshes, To make a decision, Using the bit count and vertex count analyzed from the base mesh subbitstream, at least a portion of the multiple submeshes signaled in the base mesh subbitstream are reconstructed. Apply at least a portion of the displacement information to the subdivided locations of the reconstructed portion of the submesh. A reconstructed version of the dynamic mesh, the reconstructed version of the dynamic mesh comprising the reconstructed submesh of the base mesh and additional vertices added at the subdivision locations, wherein the position of each of the additional vertices is adjusted by applying at least a portion of the displacement information, is displayed on the display of the device. device.

Images