Encoding, signaling and decoding of inter submesh boundaries for video-based dynamic mesh coding

Abstract

An example apparatus includes: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: for each submesh of a plurality of submeshes: determining zero or more traversal paths of one or more primitive boundaries the each submesh shares with one or more neighboring submeshes; and determining one or more corresponding traversal paths of the one or more primitive boundaries in one or more corresponding neighboring submeshes; and signaling pairs of traversal paths of the one or more primitive boundaries between the one or more neighboring submeshes.

Description

ENCODING, SIGNALING AND DECODING OF INTER SUBMESH BOUNDARIES FOR VIDEO-BASED DYNAMIC MESH CODINGTECHNICAL FIELD

[0001] The examples and non-limiting embodiments relate generally to video-based dynamic mesh coding, and more particularly, encoding, signaling and / or decoding of inter submesh boundaries for video-based dynamic mesh coding.BACKGROUND

[0002] It is known to perform encoding and decoding of images and video.SUMMARY

[0003] Example 1 : An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: for each submesh of a plurality of submeshes: determining zero or more traversal paths of one or more primitive boundaries the each submesh shares with one or more neighboring submeshes; and determining one or more corresponding traversal paths of the one or more primitive boundaries in one or more corresponding neighboring submeshes; and signaling pairs of traversal paths of the one or more primitive boundaries between the one or more neighboring submeshes. In an example, the correspondence may mean that each traversal path of a submesh is specific to a traversal path in a specific neighboring submesh.

[0004] Example 2: The apparatus of example 1, wherein a primitive comprise one or more of a vertex, an edge, or a face, and wherein the one or more primitive boundaries comprise one or more of vertex boundaries, edge boundaries, or face boundaries.

[0005] Example 3: The apparatus of any of examples 1 or 2, wherein the apparatus is further caused to perform: segmenting input mesh frame or a base mesh frame of the input mesh into a plurality of submeshes, wherein the plurality of submeshes comprise one or more primitives.

[0006] Example 4: The apparatus of any of examples 1 to 3, wherein the apparatus is furthercaused to perform: storing, for each primitive, a correspondence between an original primitive index and the index of the each primitive in one or more submeshes the each primitive belongs to; storing following, for the each submesh: a list of neighbor submeshes of the each submesh, wherein the neighbor submeshes share a boundary; a list of at least one primitive that belongs to a submesh boundary; list of submeshes the at least one primitive belongs to; and a primitive index of the at least one primitive in the corresponding submeshes.

[0007] Example 5: The apparatus of any of examples 3 or 4, wherein the segmentation is performed based on one or more criteria.

[0008] Example 6: The apparatus of the example 4, wherein the one or more criteria comprises a target number of submeshes, a geometry-based criterion, or a semantic -based criterion.

[0009] Example 7: The apparatus of any of the examples 3 to 6, wherein: when the one more primitive boundaries comprise one or more of the vertex boundaries or the edge boundaries, the segmentation comprises: a non-overlapping segmentation that groups triangles such that a triangle of the input mesh or the base mesh is comprised in exactly one submesh, wherein a set of vertices of the triangle belong to the one or more submeshes, and wherein the set of vertices comprises a set of interior vertices and / or a set of boundary vertices, and wherein the set of boundary vertices that belong to more than one submesh, and wherein the set of interior vertices belong to one submesh; and when the one more primitive boundaries comprise one or more of the face boundaries, the segmentation comprises: an overlapping segmentation that groups one or more triangles such that a triangle comprises a boundary face that belongs to two or more submeshes and a submesh boundary vertex that belongs to the two one or more submeshes.

[0010] Example 8: The apparatus of any of the examples 2 to 7, wherein the one or more vertex boundaries belong to a polyline or piecewise linear curve that connects corresponding vertices with exactly one edge per pair of vertices, and wherein submesh face boundaries are defined for overlapping segmentations and comprises a strip of triangles.

[0011] Example 9: The apparatus of any of the examples 2 to 7, wherein the apparatus is further caused to perform: storing, for the each submesh, a list of faces that belonging to the submesh boundary, original indices of the face, and submesh indices of the face in the each submesh.

[0012] Example 10: The apparatus of any of examples the previous examples, wherein for determining the zero or more traversal paths, the apparatus is further caused to perform: determining a starting oriented primitive, a second primitive, and an end primitive; and iteratively identify a non-yet traversed neighboring boundary primitive of a last traversed boundary primitive, until reaching the end primitive.

[0013] Example 11: The apparatus of any of the previous examples, wherein signaling traversal of the vertex boundary for the each submesh for zippering is done in an atlas or within an supplemental enhancement information (SEI) message.

[0014] Example 12: The apparatus of example 11, wherein signaling within the SEI message comprises signaling following: a number of joint border [k] [p] for indicating a number of submesh joint borders for the zippering instances k and submesh indexes p; and joint borders indices [k] [p] [b] for indicating a bth index corresponding to a submesh that shares a border with a submesh p for a zippering instance k.

[0015] Example 13: The apparatus of any of the examples 1 to 10, wherein signaling traversal of the vertex boundary for the each submesh comprises a zippering border implicit traversal.

[0016] Example 14: The apparatus of example 13, wherein the zippering border implicit traversal comprises signaling a border implicit traversal type for indicating a traversal type that takes values a specified as following: the border implicit traversal type = 0 indicates that the traversal type comprises edge traversal; the border implicit traversal type = 1 indicates that the traversal type comprises triangle strip traversal; and the border implicit traversal type = 2 is reserved.

[0017] Example 15: The apparatus of example 14, wherein the apparatus is further caused to perform: providing following submesh vertices in the edge traversal: a start edge vertex[0] for indicating a first vertex index of a starting edge of a border traversal of submesh (p) with a submesh index (submeshindex) for a zippering instance k; a start edge vertex[l] for indicating a second vertex index of the starting edge of the border traversal of the submesh (p); and an end vertex for indicating an end vertex index of the border traversal of the submesh p.

[0018] Example 16: The apparatus of example 14, wherein the apparatus is caused to perform: providing following submesh faces in the triangle strip traversal: a start triangle for indicating a first triangle index of a starting triangle strip based border traversal of a submesh (p) with a submesh index (submeshindex) for the zippering instance k; a second triangle for indicating a second triangle index of the starting triangle strip based border traversal of the submesh (p); and an end triangle for indicating an end triangle index of the triangle strip based border traversal of the submesh (p).

[0019] Example 17: An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: for a boundary between a first submesh and a second submesh decoding a start primitive and an end primitive; and reproducing for the first submesh and the second submesh a traversal path based on the start primitive and the end primitive.

[0020] Example 18: The apparatus of example 17, wherein a primitive comprises a vertex, an edge, or a face.

[0021] Example 19: The apparatus of any of examples 17 or 18, wherein the apparatus is further caused to perform: determining corresponding pairs of boundary primitives from the first submesh and the second submesh; and fusing, pasting, zippering, or inpainting the first and the second submeshes along corresponding boundaries of the first and second submeshes.

[0022] Example 20: A method comprising: for each submesh of a plurality of submeshes: determining zero or more traversal paths of one or more primitive boundaries the each submesh shares with one or more neighboring submeshes; and determining one or more corresponding traversal paths of the one or more primitive boundaries in one or more corresponding neighboring submeshes; and signaling pairs of traversal paths of the one or more primitive boundaries between the one or more neighboring submeshes.

[0023] Example 21: The method of example 20, wherein a primitive comprise one or more of a vertex, an edge, or a face, and wherein the one or more primitive boundaries comprise one or more of vertex boundaries, edge boundaries, or face boundaries.

[0024] Example 22: The method of any of examples 20 or 21 further comprising: segmenting input mesh frame or a base mesh frame of the input mesh into a plurality of submeshes, wherein the plurality of submeshes comprise one or more primitives.

[0025] Example 23: The method of any of examples 20 to 22 further comprising: storing, for each primitive, a correspondence between an original primitive index and the index of the each primitive in one or more submeshes the each primitive belongs to; storing following, for the each submesh: a list of neighbor submeshes of the each submesh, wherein the neighbor submeshes share a boundary; a list of at least one primitive that belongs to a submesh boundary; list of submeshes the at least one primitive belongs to; and a primitive index of the at least one primitive in the corresponding submeshes.

[0026] Example 24: The method of any of examples 22 or 23, wherein the segmentation is performed based on one or more criteria.

[0027] Example 25: The method of the example 23, wherein the one or more criteria comprises a target number of submeshes, a geometry-based criterion, or a semantic -based criterion.

[0028] Example 26: The method of any of the examples 22 to 25, wherein: when the one more primitive boundaries comprise one or more of the vertex boundaries or the edge boundaries, the segmentation comprises: a non-overlapping segmentation that groups triangles such that a triangle of the input mesh or the base mesh is comprised in exactly one submesh, wherein a set of vertices of the triangle belong to the one or more submeshes, and wherein the set of vertices comprises a set of interior vertices and / or a set of boundary vertices, and wherein the set of boundary vertices that belong to more than one submesh, and wherein the set of interior vertices belong to one submesh; and when the one more primitive boundaries comprise one or more of the face boundaries, the segmentation comprises: an overlapping segmentation that groups one or more triangles such that a triangle comprises a boundary face that belongs to two or more submeshes and a submesh boundary vertex that belongs to the two one or more submeshes.

[0029] Example 27: The method of any of the examples 21 to 26, wherein the one or more vertex boundaries belong to a polyline or piecewise linear curve that connects corresponding vertices with exactly one edge per pair of vertices, and wherein submesh face boundaries are defined for overlapping segmentations and comprises a strip of triangles.

[0030] Example 28: The method of any of the examples 21 to 26 further comprising: storing, for the each submesh, a list of faces that belonging to the submesh boundary, original indices of the face, and submesh indices of the face in the each submesh.

[0031] Example 29: The method of any of examples the previous examples the determining the zero or more traversal paths further comprises: determining a starting oriented primitive, a second primitive, and an end primitive; and iteratively identify a non-yet traversed neighboring boundary primitive of a last traversed boundary primitive, until reaching the end primitive.

[0032] Example 30: The method of any of the previous examples, wherein signaling traversal of the vertex boundary for the each submesh for zippering is done in an atlas or within an supplemental enhancement information (SEI) message.

[0033] Example 31: The method of example 30, wherein signaling within the SEI message comprises signaling following: a number of joint border [k] [p] for indicating a number of submesh joint borders for the zippering instances k and submesh indexes p; and joint borders indices [k] [p] [b] for indicating a bth index corresponding to a submesh that shares a border with a submesh p for a zippering instance k.

[0034] Example 32: The method of any of the examples 20 to 29, wherein signaling traversal of the vertex boundary for the each submesh comprises a zippering border implicit traversal.

[0035] Example 33: The method of example 32, wherein the zippering border implicit traversal comprises signaling a border implicit traversal type for indicating a traversal type that takes values a specified as following: the border implicit traversal type = 0 indicates that the traversal type comprises edge traversal; the border implicit traversal type = 1 indicates that the traversal type comprises triangle strip traversal; and the border implicit traversal type = 2 is reserved.

[0036] Example 34: The method of example 33 further comprising: providing following submesh vertices in the edge traversal: a start edge vertex[0] for indicating a first vertex index of a starting edge of a border traversal of submesh (p) with a submesh index (submeshindex) for a zippering instance k; a start edge vertex[l] for indicating a second vertex index of the starting edgeof the border traversal of the submesh (p); and an end vertex for indicating an end vertex index of the border traversal of the submesh p.

[0037] Example 35: The method of example 33, wherein the apparatus is caused to perform: providing following submesh faces in the triangle strip traversal: a start triangle for indicating a first triangle index of a starting triangle strip based border traversal of a submesh (p) with a submesh index (submeshindex) for the zippering instance k; a second triangle for indicating a second triangle index of the starting triangle strip based border traversal of the submesh (p); and an end triangle for indicating an end triangle index of the triangle strip based border traversal of the submesh (p).

[0038] Example 36: A method comprising: for a boundary between a first submesh and a second submesh, decoding a start primitive and an end primitive; and reproducing for the first submesh and the second submesh a traversal path based on the start primitive and the end primitive.

[0039] Example 37: The method of example 36, wherein a primitive comprises a vertex, an edge, or a face. In an example, the primitive comprises a vertex, an edge, a face, or a combination thereof.

[0040] Example 38: The method of any of examples 36 or 37 further comprising: determining corresponding pairs of boundary primitives from the first submesh and the second submesh; and fusing, pasting, zippering, or inpainting the first and the second submeshes along corresponding boundaries of the first and second submeshes.

[0041] Example 39: An apparatus comprising: for each submesh of a plurality of submeshes: means for determining zero or more traversal paths of one or more primitive boundaries the each submesh shares with one or more neighboring submeshes; and means for determining one or more corresponding traversal paths of the one or more primitive boundaries in one or more corresponding neighboring submeshes; and

[0042] means for signaling pairs of traversal paths of the one or more primitive boundaries between the one or more neighboring submeshes.

[0043] Example 40: The apparatus of example 39, wherein the apparatus further comprisesmeans for performing the methods as described in any of the examples 21 to 35.

[0044] Example 41: An apparatus comprising: for a boundary between a first submesh and a second submesh, means for decoding a start primitive and an end primitive; and means for reproducing for the first submesh and the second submesh a traversal path based on the start primitive and the end primitive.

[0045] Example 42: The apparatus of example 41, wherein the apparatus further comprises means for performing the methods as described in any of the examples 37 to 38.

[0046] Example 43: A computer readable medium comprising program instructions which, when executed by an apparatus, cause the apparatus to perform the methods as described in any of the examples 20 to 35.

[0047] Example 44: The computer readable medium of example 43, wherein the computer readable medium comprises a non-transitory computer readable medium.

[0048] Example 45: A computer readable medium comprising program instructions which, when executed by an apparatus, cause the apparatus to perform the methods as described in any of the examples 36 to 38.

[0049] Example 46: The computer readable medium of example 45, wherein the computer readable medium comprises a non-transitory computer readable medium.BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The foregoing embodiments and other features are explained in the following description, taken in connection with the accompanying drawings, wherein:

[0051] FIG. 1 shows schematically an electronic device employing embodiments of the examples described herein.

[0052] FIG. 2 shows schematically a user equipment suitable for employing embodiments of the examples described herein.

[0053] FIG. 3 illustrates a V-DMC encoder.

[0054] FIG. 4 illustrates a decoder scheme showing the substream decoding and reconstruction process to output a reconstructed dynamic mesh sequence.

[0055] FIG. 5 illustrates a concept of level of detail (LOD) in V-DMC.

[0056] FIG. 6 illustrates the way lifting operations predict a LOD from a lower resolution.

[0057] FIG. 7 illustrates segmentation of a mesh into three non-overlapping submeshes that can be independently decoded.

[0058] FIG. 8 illustrates reconstruction of lossy encoded non-overlapping submeshes leads to cracks at their boundaries.

[0059] FIG. 9 illustrates submesh boundary signaling.

[0060] FIG. 10 illustrates plurality of submeshes, pairs of corresponding vertices from the plurality of submeshes, and pairs of corresponding boundary edge traversals from the plurality of submeshes.

[0061] FIG. 11 is an example apparatus configured to implement the examples described herein.

[0062] FIG. 12 shows a representation of an example of non-volatile memory media used to store instructions that implement the examples described herein.

[0063] FIG. 13 is an example method, based on the examples described herein.

[0064] FIG. 14 is another example method, based on the examples described herein.DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0065] Described herein is a method and apparatus for encoding, signaling and / or decoding of inter submesh boundaries for video-based dynamic mesh coding.

[0066] The following describes in detail a suitable apparatus, method and possible mechanisms for encoding, signaling and / or decoding of inter submesh boundaries for video-based dynamic mesh coding according to embodiments. In this regard reference is first made to FIG. 1 and FIG. 2, where FIG. 1 shows an example block diagram of an apparatus 50. The apparatus may be an Internet of Things (loT) apparatus configured to perform various functions, such as for example, gathering information by one or more sensors, receiving or transmitting information, analyzing information gathered or received by the apparatus, or the like. The apparatus may comprise a video coding system, which may incorporate a codec. FIG. 2 shows a layout of an apparatus according to an example embodiment. The elements of FIG. 1 and FIG. 2 are explained next.

[0067] The apparatus 50 may for example be a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag, or other lower power device. However, it may be appreciated that embodiments of the examples described herein may be implemented within any electronic device or apparatus which may process data by neural networks.

[0068] The apparatus 50 may comprise a housing 30 for incorporating and protecting the device. The apparatus 50 further may comprise a display 32 in the form of a liquid crystal display. In other embodiments of the examples described herein the display may be any suitable display technology suitable to display an image or video. The apparatus 50 may further comprise a keypad 34. In other embodiments of the examples described herein any suitable data or user interface mechanism may be employed. For example the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.

[0069] The apparatus may comprise a microphone 36 or any suitable audio input which may be a digital or analog signal input. The apparatus 50 may further comprise an audio output device which in embodiments of the examples described herein may be any one of: an earpiece 38, speaker, or an analog audio or digital audio output connection. The apparatus 50 may also comprise a battery (or in other embodiments of the examples described herein the device may be poweredby any suitable mobile energy device such as solar cell, fuel cell or clockwork generator). The apparatus may further comprise a camera 42 capable of recording or capturing images and / or video. The apparatus 50 may further comprise an infrared port for short range line of sight communication to other devices. In other embodiments the apparatus 50 may further comprise any suitable short range communication solution such as for example a Bluetooth wireless connection or a USB / firewire wired connection.

[0070] The apparatus 50 may comprise a controller 56, processor or processor circuitry for controlling the apparatus 50. The controller 56 may be connected to memory 58 which in embodiments of the examples described herein may store both data in the form of image and audio data and / or may also store instructions for implementation on the controller 56. The controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and / or decoding of audio and / or video data or assisting in coding and / or decoding carried out by the controller.

[0071] The apparatus 50 may further comprise a card reader 48 and a smart card 46, for example a UICC and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.

[0072] The apparatus 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system or a wireless local area network. The apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and / or for receiving radio frequency signals from other apparatus(es).

[0073] The apparatus 50 may comprise a camera capable of recording or detecting individual frames which are then passed to the codec circuitry 54 or the controller for processing. The apparatus may receive the video image data for processing from another device prior to transmission and / or storage. The apparatus 50 may also receive either wirelessly or by a wired connection the image for coding / decoding. The structural elements of apparatus 50 described above represent examples of means for performing a corresponding function.

[0074] Throughout the detailed description, document ISO / IEC 23090-29 means the outputdocument MDS23075_WG07_N00680 of MPEG 143.

[0075] Volumetric video

[0076] There are many ways to capture and represent a volumetric frame. The format used to capture and represent it depends on the processing to be performed on the volumetric frame, and the target application using volumetric frame. Some example representations are listed below.

[0077] A volumetric frame may be represented as a point cloud. A point cloud is a set of unstructured points in 3D space, where each point is characterized by its position in a 3D coordinate system (e.g., Euclidean), and some corresponding attributes (e.g., color information provided as red, green, blue, alpha (RGBA) value, or normal vectors).

[0078] A volumetric frame may be represented as images, with or without depth, captured from multiple viewpoints in 3D space. In other words, it may be represented by one or more view frames (where a view is a projection of a volumetric scene on to a plane (the camera plane) using a real or virtual camera with known / computed extrinsics and intrinsics). Each view may be represented by a number of components (e.g., geometry, color, transparency, and occupancy picture), which may be part of the geometry picture or represented separately.

[0079] A volumetric frame may be represented as a mesh. A mesh is a collection of points, called vertices, and connectivity information between vertices, called edges. Vertices along with edges form faces. The combination of vertices, edges and faces may uniquely approximate shapes of objects.

[0080] Depending on the capture, a volumetric frame may provide viewers the ability to navigate a scene with six degrees of freedom, e.g., both translational and rotational movement of their viewing pose (which includes yaw, pitch, and role). The data to be coded for a volumetric frame may also be significant, as a volumetric frame may include many objects, and the positioning and movement of these objects in the scene may result in many dis-occluded regions. Furthermore, the interaction of light and materials in objects and surfaces in a volumetric frame may generate complex light fields that may produce texture variations for even a slight change of pose.

[0081] A sequence of volumetric frames is a volumetric video. Due to large amount ofinformation, storage and transmission of a volumetric video requires compression. A way to compress a volumetric frame may be to project the 3D geometry and related attributes into a collection of 2D images along with additional associated metadata. The projected 2D images may then be coded using 2D video and image coding technologies, for example, ISO / IEC 14496-10 (H.264 / AVC) and ISO / IEC 23008-2 (H.265 / HEVC). The metadata may be coded with technologies specified in specification such as ISO / IEC 23090-5. The coded images and the associated metadata may be stored or transmitted to a client that may decode and render the 3D volumetric frame.

[0082] Visual Volumetric Video-base Coding (V3C) - ISO / IEC 23090-5

[0083] ISO / IEC 23090-5 specifies the syntax, semantics, and process for coding volumetric video. The specified syntax is designed to be generic so that it may be reused for a variety of applications. Point clouds, immersive video with depth, and mesh representations may all use ISO / IEC 23090-5 standard with extensions that deal with the specific nature of the final representation. The purpose of the specification is to define how to decode and interpret the associated data (for example atlas data in ISO / IEC 23090-5) which tells a Tenderer how to interpret 2D frames to reconstruct a volumetric frame.

[0084] Two example applications of V3C (ISO / IEC 23090-5) have been defined, V-PCC (ISO / IEC 23090-5) and MIV (ISO / IEC 23090-12). MIV and V-PCC use number of V3C syntax elements with a slightly modified semantics. An example on how the generic syntax element may be differently interpreted by the application is pdu_projection_id.

[0085] In case of V-PCC the syntax element, pdu_projection_id specifies the index of the projection plane for the patch. There may be 6 or 18 projection planes in V-PCC, and they are implicit, e.g., pre-determined.

[0086] In case of MIV pdu_projection_id corresponds to a view ID, e.g., identifies which view the patch originated from. View IDs and their related information is explicitly provided in MIV view parameters list and may be tailored for each content.

[0087] MPEG 3DG (ISO SC29 WG7) group has started work on a third application of V3C - the mesh compression. It is also envisaged that mesh coding may re-use V3C syntax as much aspossible and may also slightly modify the semantics.

[0088] To differentiate between applications of V3C bitstream, that allow a client to properly interpret the decoded data, V3C uses the ptl_profile_toolset_idc parameter.

[0089] V3C - V3C bitstream

[0090] V3C bitstream is a sequence of bits that forms the representation of coded volumetric frames and the associated data making one or more coded V3C sequences (CVS). Where CVS is a sequence of bits identified and separated by appropriate delimiters, and is required to start with a VPS, includes a V3C unit, and includes one or more V3C units with atlas sub-bitstream or video sub-bitstream. Video sub-bitstreams and atlas sub-bitstreams may be referred to as V3C subbitstreams. Which V3C sub-bitstream a V3C unit includes and how to interpret it is identified by a V3C unit header in conjunction with VPS information.

[0091] V3C bitstream may be stored according to Annex C of ISO / IEC 23090-5 which specifies syntax and semantics of a sample stream format to be used by applications that deliver some or all of the V3C unit stream as an ordered stream of bytes or bits within which the locations of V3C unit boundaries need to be identifiable from patterns in the data.

[0092] Video-based Point Cloud Compression (V-PCC) - ISO / IEC 23090-5

[0093] The generic mechanism of V3C may be used by applications targeting volumetric content. One of such application is video-based point cloud compression (ISO / IEC 20390-5). V- PCC enables volumetric video coding for application in which a scene is represented by point cloud. V-PCC uses the patch data unit concept from V3C and for each patch assign one of 6 (18) pre-defined orthogonal camera views for reprojection.

[0094] MPEG Immersive Video (MIV) - ISO / IEC 23090-12

[0095] Another application of V3C is MPEG immersive video (ISO / IEC 23090-12). MIV enables volumetric video coding for applications in which a scene is recorded with multiple red, green, blue, and optionally depth (RGB(D)) cameras with overlapping fields of view (FoVs). One example setup is a linear array of cameras pointing towards a scene. This multi-scopic view of thescene allows a 3D reconstruction and therefore 6DoF / 3DoF+ consumption.

[0096] MIV uses the patch data unit concept from V3C and extends it by allow using application specific camera views for reprojection. In contrast to V-PCC, which uses pre-defined 6 or 18 orthogonal camera views for reprojection. Additionally, MIV introduces additional occupancy packing modes and other improvements to V3C base syntax. One such example is support for multiple atlases, for example when there is too much information to pack everything in a single video frame. It also adds support for common atlas data, which includes information that is shared between all atlases. This is particularly useful for storing camera details of the input camera models, which are frequently shared between different atlases.

[0097] Video-based dynamic mesh coding (V-DMC) - ISO / IEC 23090-29

[0098] V-DMC (ISO / IEC 23090-29) is another application form of V3C that aims on integration of mesh compression into the V3C family of standards. The standard is under development and at WD stage (MDS22775_WG07_N00611).

[0099] The retained technology after the CfP result analysis is based on multiresolution mesh analysis and coding. This approach includes:- generating a base-mesh that is a simplified (low resolution) mesh approximation of the original mesh, called base-mesh (this is done for all frames of the dynamic mesh sequence);- performing several mesh subdivision iterative steps (e.g., each triangle is converted into four triangles by connecting the triangle edge midpoints on the generated base mesh, generating other approximation meshes;- defining displacement vectors, also named error vectors, for each vertex of each mesh approximation;- for each subdivision level by adding the displacement vectors to the subdivided mesh vertices generates the best approximation of the original mesh at that resolution, given the base-mesh and prior subdivision levels;- the displacement vectors may undergo a lazy wavelet transform prior to compression; and- the attribute map of the original mesh is transferred to the deformed mesh at the highest resolution (e.g., subdivision level) such that texture coordinates are obtainedfor the deformed mesh and a new attribute map is generated.

[0100] FIG. 3 illustrates a V-DMC encoder 300. As input the V-DMC encoder 300 takes a dynamic mesh sequence 302 comprising textured mesh frames. A first module called preprocessing 304 generates an atlas 3data 06, a base-mesh sequence 308, a displacement sequence 310 and a texture (attribute) sequence 312. A module called atlas encoder 314 encodes the atlas data 306 for the whole input sequence and generates an atlas substream 316. A base mesh encoder 318 generates a coded base mesh substream 320 by using a static mesh codec with or without an animation codec. The displacement sequence 310 is encoded by a displacement encoder module 322 that generates a displacement substream 324 by using a video codec or using an arithmetic encoder as described in V-DMC Working Draft Annex J. The attribute encoder module 326 encodes the attribute sequence 312 to generate an attribute substream 328 using a video codec. These four substreams (or sub-bitstreams as synonym) are muxed together in a single V3C 330 bitstream by the multiplexer module 332.

[0101] For example, the V-DMC encoder generates compressed bitstreams, which later on are packed in V3C units and create V3C bitstream by concatenating one or more of the following V3C units, as illustrated in FIG. 3:- a sub-bitstream with the encoded base-mesh using a mesh codec;- a sub-bitstream with the displacement vectors;- packed in an image and encoded using a video codec, or- arithmetic encoded as defined in Annex J of WD ISO / IEC 23090, MDS22775_WG07_N00611 ;- a sub-bitstream with the attribute map encoded using a video codec; or- a sub-bitstream (atlas) that includes all metadata required to decode and reconstruct the mesh sequence based on the aforementioned sub-bitstreams. The signaling of the metadata is based on the V3C syntax and includes necessary extensions that are specific to meshes.

[0102] FIG. 4 illustrates a decoder scheme showing the substream decoding and reconstruction process 400 to output a reconstructed dynamic mesh sequence. The input V3C bitstream 402 is first demultiplexed by a demultiplexer module 404 to extract the atlas subbitstream 406, the base mesh sub-bitstream 408, the displacement sub-bitstream 410 and the attribute-sub-bitstream 412. The Atlas sub-bitstream 406 is decoded by an atlas decoder module414 to generate the atlas sequence 416. The base mesh sub-bitstream 408 is decoded by a base mesh decoder 418 to generate a base mesh sequence 420. The displacement sub-bitstream 410 is decoded by a displacement decoder 422 to generate a displacement sequence 424. The attribute- sub-bitstream 412 is decoded into a decoded attribute sequence 426 by an attribute decoder 428. The atlas sequence 416, the base mesh sequence 420 are further processed by the base-mesh processing module 430; and the displacement sequence 424 together with the atlas sequence 416 are further processed by the displacement processing module 432. The outputs of the base-mesh processing module 430 and the displacement processing module 432 are used to generate a reconstructed mesh sequence by the mesh module 434. The reconstruction module 436 takes the decoded attribute sequence 426 and output of the mesh module 434 to generate the reconstructed textured mesh sequence 438.

[0103] For example, FIG. 4 illustrates the V-DMC decoder scheme, where the V-DMC substreams are independently decoded generating:- Decoded atlas data,- Base mesh reconstructed data and its processing,- Decoded displacement data and its processing and- Decoded attribute data.

[0104] The reconstruction process that produces a reconstructed dynamic mesh sequence is also illustrated in FIG. 4.

[0105] Base-mesh bitstream (ISO / IEC 23090-29)

[0106] An elementary unit for the output of a base-mesh encoder (Annex H of ISO / IEC 23090-29) is a NAL unit.

[0107] A NAL unit may be defined as a syntax structure including an indication of the type of data to follow and bytes including that data in the form of a raw byte sequence pay load (RBSP) interspersed as necessary with emulation prevention bytes. An The RBSP may be defined as a syntax structure including an integer number of bytes that is encapsulated in a NAL unit. An RBSP is either empty or has the form of a string of data bits including syntax elements followed by an RBSP stop bit and followed by zero or more subsequent bits equal to 0.

[0108] NAL units may be categorized into base-mesh coding layer (BMCL) NAL units and non-BMCL NAL units. BMCL NAL units may be coded sub-mesh NAL units. A non-BMCL NAL unit may be for example one of the following types: a base-mesh sequence parameter set, a basemesh frame parameter set, a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end of sequence NAL unit, an end of bitstream NAL unit, or a filler data NAL unit. Parameter sets may be needed for the reconstruction of decoded bas-mesh, whereas many of the other non-BMCL NAL units are not necessary for the reconstruction of decoded sample values.

[0109] V-DMC specifications may include a set of constraints for associating data units (e.g. NAL units) into coded base-mesh access units.

[0110] Atlas data (ISO / IEC 23090-29)

[0111] The V-DMC standard provides semantic and signaling information that is required to process the decoded V-DMC substreams. The Atlas sequence parameter set is detailed in the following table, Table 1. It includes information on the subdivision parameters, displacement coordinates, transform parameters, number and parameters of the video attributes, and the like.

[0112] Atlas sequence parameter set V-DMC extension RBSP syntax is provided in following table, Table 1:Table 1 - Atlas sequence parameter set V-DMC extension RBSP syntax

[0113] Atlas tile information is provided in the following table, Table 2.Table 2 - Atlas frame attribute tile information

[0114] Subdivision and Lifting transform (ISO / IEC 23090-29)

[0115] The V-DMC framework utilizes the well-known concept of Level-of-Detail (LoD). FIG. 5 illustrates a concept of level of detail (LOD) in V-DMC. As illustrated in FIG. 5, different LoD levels (e.g., LoDO to LoD3) correspond to different sampling rates and qualities of meshes. It is desirable to process different LoD levels on a GPU for example for rendering, to optimize rendering speed. For example, when the mesh is rendered at a far distance in the viewport, it only is represented by a small number of pixels, and a coarse mesh representation (lowest LoD level, e.g., LoDO) is sufficient as finer sampling rates of the mesh may not lead to a better quality. When the mesh is rendered at a close distance in the viewport, then it is desirable to view it in full resolution (highest LoD level, e.g., LoD3) to maximize rendering quality. In between these cases, the LoD level may be optimized based on the area of the viewport that would be occupied by the mesh, and it is therefore desirable to represent the mesh in several LoDs for interactive rendering applications. In V-DMC, typically 3 LoD levels are defined, but some rendering applications may use for example up to 18 LoD levels. A given LoD level is generated from the previous LoD reconstruction by using a subdivision iteration and applying reconstructed displacement data as explained in previous paragraphs.

[0116] As per concept of Level of Detail (LOD) in V-DMC, s higher resolution (LOD2 or LOD3) is required for rendering from a close distance, while lower resolution (LODO, LODI) is sufficient for far distances. More LODs enable better tradeoffs between rendering performance and perceived quality of details.

[0117] The V-DMC framework further uses a lifting transform to decorrelate the signal defined on the mesh connectivity at each LOD in a hierarchical manner; the signal being the displacements and the normal, but can also include other attributes, for example. FIG. 6 illustrates a way in which a lifting operation predicts a LOD from a lower resolution.

[0118] The V-DMC framework further uses the concept of segmentation into submeshes that can each use different encoding parameters and can be independently decoded to allow for spatial random access or parallel decoding for example. FIG. 7 illustrates segmentation of a mesh into three non-overlapping submeshes that can be independently decoded.

[0119] When a V-DMC bitstream includes submeshes, reconstruction of submeshes is performed independently and may lead to cracks due to vertex position lossy coding. While inpainting methods may be used to reconnect submeshes after reconstruction, their complexity may be problematic for decoder implementations. Signaling matching pairs or triplets of boundary vertices introduces a significant metadata overhead that may also impact decoding throughput and overall decoding performance. The reconstruction needs to fuse the coordinates of the pairs of matched submesh boundary vertices to close the holes between the reconstructed submeshes. FIG. 8 illustrates reconstruction of lossy encoded non-overlapping submeshes leads to cracks at their boundaries.

[0120] V-DMC SEI messages have been proposed so far based on two different zippering approaches:- Distance-based (zp_method_type=l): the decoder receives indications on distance thresholds and has to perform a nearest neighbor search; this approach is computationally complex for the decoder but requires a very small amount, for example, one float for each submesh or for all submeshes, of metadata; and- Border point match zippering (zp_method_type=2): the decoder receives matching indices of pairs of vertices from submesh boundaries; this leads to a very large amount of metadata but involves no complex processing at the decoder side. Typically, hundreds of vertex index pairs using integers per submesh neigbor pair. For example when there are 3 submeshes S1,S2,S3 with a boundary between SI and S2 and another one between S2 and S3, then typically, 600 * 2 * 2 integers are needed, where 600 is a typical (but may be larger) number of neighbor vertices in submeshes, 2 as pairs of matching vertices are needed, and 2 because there are two boundaries (S1-S2 and S2-S3 respectively).

[0121] Signaling is as follows:

[0122] The zippering may refer to an operation of pulling the pairs of vertices on the traversal path to the same 3D location to avoid cracks appearing between submeshes (the cracks may appear because of the lossy coding).

[0123] In an example, after the pairs of boundary vertices are identified between submeshes, an example method includes averaging out the position of each vertex pair and set the same average value as the position of both the vertices. This minimizes the error caused by the lossy coding.

[0124] Various embodiments disclose signaling, decoding and encoding embodiments, at least, to solve the issue of a low complexity inpainting of submesh cracks by boundary signaling, while reducing the metadata overhead significantly compared to explicit boundary signaling.

[0125] This is performed, for example, by coding sets of matching submesh boundary primitives with a submesh primitive boundary traversal approach. The proposed embodiments rely on an efficient boundary search on the (sub)mesh connectivity that enables most of the boundary vertices to be signalled implicitly, e.g., without signaling except for the start and end of the traversal. In an example, a primitive may include one or more of a vertex, an edge, or a face; and a primitive boundary may include one or more of a vertex boundary, an edge boundary, or a face boundary. In an example, a primitive may include a vertex, an edge, a face, or a combination thereof; and a primitive boundary may include one or more of a vertex boundary, an edge boundary, a face boundary, or a combination thereof.

[0126] FIG. 9 illustrates submesh boundary signaling. As shown in FIG. 9, corresponding vertices have different indices in each submesh. Signaling each pair of corresponding vertices between two submeshes is not scalable and requires too much metadata, signaling traversals as described in the embodiments of this invention, provides a scalable solution to the problem where most vertices are obtained implicitly without a need for explicit signaling. FIG. 9 illustrates an example of a primitive being a vertex.

[0127] For a common boundary of N vertices 902, a verbose signaling 904 requires signaling explicitly N pairs of vertex indices from each submesh. For the proposed embodiments 906, two corresponding traversal paths have to be signalled, each of them requiring the explicit signaling of only 3 vertices to represent a starting oriented edge, as a starting point, and an end vertex. For example, a starting vertex, a vertex directly connected to the starting vertex that gives the orientation of the traversal path, and an ending vertex. In this example, there are 5*2 integers for the verbose method as submeshes SI and S2 share 5 vertices with pairs of indices in each submesh. The proposed traversal approach needs (2+l)*2 integers (2+1) for the traversal start and stop respectively and *2 because pairs of corresponding traversals in SI and S2 are needed. In this example, therefore there are 6 integers instead of 10.

[0128] In more realistic example, for verbose method, 600*2*2 integers are required for three submeshes, the proposed embodiments would require (2+1) * 2 * 2 integers (2 + 1 for the traversal start and stop respectively, 2 because of pairs of corresponding traversals, and 2 because there are two boundaries between S1-S2 and S2-S3). Therefore 2400 integers are required for the verbose method vs 12 integers for the proposed embodiments. For an example boundary of 100vertices, which is a common number for the current V-DMC framework, this means a reduction by 97% of the metadata signaling for the submesh edges.

[0129] Reducing the metadata overhead is important for all V3C standards to allow for high throughput and real-time decoding.

[0130] Therefore, with the proposed embodiments, the decoder complexity is very close to the one of border point match zippering method while the metadata bitrate remains close to the distance zippering method.

[0131] Further, another advantage of proposed approach is that it can handle submeshes with different subdivision iteration counts (e.g., LoDs), which is not the case for the current solutions proposed in V-DMC.

[0132] FIG. 10 illustrates plurality of submeshes, pairs of corresponding vertices from the plurality of submeshes, and pairs of corresponding boundary edge traversals from the plurality of submeshes. In 1002, the boundary between the plurality of submeshes SI and S2 is illustrated as a series of edges. Each vertex of this boundary comprises an index in S 1 (S l_v*) and a corresponding index in S2 (s2_v*). The example shows 5 neighbor vertices and 4 edges that connect these in the boundary polyline. 1004 lists corresponding vertices from submesh SI and S2 here with the same example as 1002. Given that submeshes typically have hundreds or more neighbor vertices, the solution of 1004 requires to encode long series of pairs of vertex indices, which leads to bitrate increases. In 1006 corresponding vertex traversals for submeshes SI and S2 are encoded for the examples of 1002 and 1004. The larger the number of vertices in the submesh boundary, the larger coding gain obtained by using the proposed vertex traversals of 1006.

[0133] Encoding embodiments

[0134] In an encoding embodiment, the input mesh frame is segmented into several submeshes according to a number of possible criteria, such as a target number of submeshes, a geometry-based criterion (partition along one axis for example), a semantic -based criterion (e.g., body parts), and the like. In the following, the segmentation may be applied to the input mesh or to the base mesh obtained after vertex decimation of the input mesh.

[0135] Following are two example segmentations with different primitives:

[0136] A non-overlapping segmentation that groups triangles such that a triangle of the original mesh is included in exactly one submesh, while vertices belong to one or more submeshes (those that belong to more than one submesh are called boundary vertices, the others being interior vertices). In an example, each submesh includes one or more triangles. For non-overlapping segmentation examples of primitives include vertices and / or edges.

[0137] An overlapping segmentation that also groups triangles but in such a way that a face may belong to two or more submeshes (boundary faces), while a submesh boundary vertex may be an interior vertex of another submesh. For overlapping segmentation examples of primitives include faces.

[0138] Sub mesh vertex boundaries in both types of segmentations belong to a polyline or piecewise linear curve that connects these vertices with exactly one edge per pair of vertices. Submesh face boundaries are typically defined for overlapping segmentations and includes a strip of triangles (such that each pair of neighboring boundary triangles shares exactly one edge).

[0139] As a result of the segmentation operation, the following information is stored for further use such as for each vertex, the correspondence between the original vertex index and its index in the submesh(es) it belongs to. In an example, an original index is the index of the vertex in the (decimated or not) base mesh before segmentation in submeshes. Once submeshes are created these vertex indices are modified (in practice all submeshes have vertex indices that start at zero or one and have a set of consecutive indices). For example, assume vertex 10 in the original (decimated or not) mesh is a boundary vertex between submeshes S 1 and S2, it may be that this vertex gets an index 0 in SI and 2 in S2 for example. Being able to know the correspondence between these vertex indices is key to understand how these vertices will be matched at the decoder.

[0140] Furthermore, for each submesh, the encoder stores:- the list of its neighbor submeshes, e.g., submeshes that share a boundary;- the list of vertices that belong to a submesh boundary, the submeshes they belong to and their vertex index in the corresponding submeshes; and / or- optionally the list of faces that belong to a submesh boundary (overlapping segmentation for example), and their original indices as well as their submeshindices in each submesh they belong to.

[0141] Then for each submesh, the encoder identifies a traversal of the vertex boundary it shares with a neighboring submesh and the corresponding traversal in the corresponding neighboring submesh as illustrated in FIG. 9. This means that when a submesh has two neighboring submeshes, then two boundary traversals are estimated for each neighboring submesh.

[0142] There are multiple ways to determine such a traversal, in one embodiment, a traversal of a neighboring vertex polyline is determined by a starting oriented edge (starting vertex, second vertex) and an end vertex. The choice of the starting oriented edge is an encoder choice; however depending on the topology of the boundary polyline, the starting oriented edge may be an open polyline (e.g., the extremities of the boundary polyline are not connected to each other) or a closed loop polyline (e.g., all its vertices have exactly two boundary vertex neighbors). In case of an open polyline, the starting oriented edge is required to start at one of the polyline extremities. In case of a closed polyline, any choice of starting oriented edge is valid. For example, the vertex with the smallest vertex index can be selected as a “virtual extremity” and its second vertex be its neighboring boundary vertex with smallest vertex index. The end vertex in the case of an open polyline is required to be the other extremity of the polyline, while in the case of an open polyline, the end vertex is the boundary vertex of the starting vertex that was not already selected as second vertex. In an example, the oriented edge gives the direction of the traversal path. This may be done by specifying an initial vector including a starting vertex and a next vertex.

[0143] Once the starting oriented edge and the end vertex are identified, the traversal of the boundary is uniquely defined as it suffices to iteratively identify the non-yet traversed neighboring boundary vertex of the last traversed boundary vertex, until reaching the end vertex.

[0144] In another example, the index of second vertex indicating the starting edge may be replaced by indicating the winding order of the traversal path around the edge of the submesh. The winding order may be enumerated to indicate either clockwise rotation or counter-clockwise rotation.

[0145] In another example, the index of the ending vertex may be replaced by the number of vertices that form the edge.

[0146] At the encoder, this information is available as we have knowledge of all vertices and their indices in the various submeshes.

[0147] For a pair of neighboring submeshes, a traversal must be determined for each “side” of the boundary, and the choice of the starting oriented edge and end vertex should preferably be consistent (meaning that the corresponding index of the starting, second and end vertices are respectively the same in the original mesh for both traversals). Another embodiment could rely on non-corresponding start and end vertices if one pair of corresponding vertices is encoded as well. This would however require additional processing to synchronize both traversals based on this pair.

[0148] Decoder embodiments

[0149] With the two traversals, it is possible for a decoder to uniquely identify each pair of corresponding neighbor vertices in both submeshes as shown in the following:Set Traversal_SiSj = { Si(va), Si(vb) }Select triangle SitO in Si that contains the edge SiSj_previous_edge, add SitO to the SiSj_Traversed set SiTs and set SiSj_previous_triangle = SitO.While (SiSj_previous_vertex != SiSj_end_vertex) do {Select triangle Sit in Si that belongs to triangle neighbors of SiSj_previous_triangle, (that does not belong to SiTs) and that has an edge Siel consisting of boundary vertices (of Si and Sj) SivO and Sivl with SivO equal to SiSj_previous_vertex, add Sit to SiTs, add Sivl to SiTb and add Siel to SiTe, set SiSj_previous_edge = Siel, set SiSj_previous_vertex = Sivl, set SiSj_previous_triangle = Sit append SiSj_previous_vertex to TraversalSiSj.}Perform the corresponding operations for Sj :Decode from syntax Sj Si_start_edge SieO = {Sjva, Sjvb}Decode from syntax Sj Si_end_vertex = Sjve / / Sive may be equal to Siva in case of close boundarySet SjSi_previous_vertex = SjvbSet SjSi_previous_edge = SjeOSet SjSi_Traversed triangles SjTs = { } / / emptySet SjSi_Traversed edges SjTe= {SjeO}

[0150] It is to be noted, that indices are those of the base mesh, this means that the number of boundary vertices in the above is not influenced by the subdivision iteration count for example.

[0151] In another embodiment, in case the subdivision iteration count of the two submeshes sharing a boundary are not the same, the zippering occurs at the base mesh level and a postprocess is used to correct T-junctions between the two submeshes, for example, using a dictionary of triangle transitions to modify the connectivity of the triangle with the smallest subdivision iteration count.

[0152] In another embodiment, the subdivision iteration count difference between the two submeshes is known and the matching procedure can occur after subdivision. The matching procedure takes into account the fact that the shared boundary in one submesh and in the other submesh do not have the same number of vertices but these numbers are related by a simple relationship. Indeed as a simple example, when the subdivision iteration count difference between the two submeshes is equal to one, the matching pairs association after reconstruction of the traversals includes matching each vertex in traversal order of the submesh with lower subdivision iteration count with even numbers in traversal order of the submesh boundary that has a larger subdivision iteration count. When the subdivision iteration count difference is larger and equal to d, then each vertex n in traversal order in the first submesh with lower subdivision count is matched with the d*n vertex in traversal order of the other submesh boundary. Similarly, the triangles of the submesh with lower subdivision count need to be processed so that unmatched vertices of higher subdivision layers are connected to these triangles without creating a T-junction.

[0153] In other embodiments, other primitive traversal types may be defined, for example, corresponding triangle strips for each submesh. For example, in case of an overlapping segmentation. An example triangle strip definition includes all vertices of the strip as specified in OpenGL, which is the implicit traversal knowing the vertices lie on a boundary. Therefore, the triangle strip requires to be defined differently in this embodiment, by specifying the starting triangle and an end vertex. It is possible to define it implicitly by performing a traversal search from the triangle last edge in vertex order (when triangles is defined as {v0, vl, v2], then the last edge is [v2,v0]), but other choices are possible). The traversal search looks for triangles comprising the edge [v2,v0] that are also boundary triangles, when none corresponds to that property and that the end vertex is not included in the last added triangle, then the search starts from the second edge[vl,v2] of the last triangle. When again, there is no candidate, the search starts from the previously added triangle non-tested edges until the end vertex is reached. This approach allows to handle special cases with non-manifold edges for example. Other primitive traversals may be used similarly such as line strips and line loops for edges, or triangle faces.

[0154] Reconstruction at decoder side

[0155] The decoder is able to reconstruct a list of pairs of corresponding boundary primitive (where primitive is a vertex) from each pair of connected submeshes and to fuse their vertex coordinates after reconstruction, in such a way that each pair of corresponding vertices uses a single vertex position. The choice of single vertex position may, for example, be:- the average position of each vertex position in the pair of corresponding vertices;- one of the vertex positions in the pair of corresponding vertices when one submesh is priority, where this priority is signaled along the bitstream or is left as a choice for the decoder, for example because one submesh is closer to the selected viewpoint in the rendering application; or- other combinations of the vertex positions to allow for a smooth boundary for example in a second pass filtering that can be defined along the bitstream as an SEI or in the atlas metadata or left as an end-device choice.

[0156] Signaling embodiments

[0157] Signaling Submesh boundaries for zippering may be done in the atlas (such as the asps or afps) or within an SEI message.

[0158] In case of an SEI message, the signaling may be as follows:

[0159] zp_zippering_number_ofjoint_borders[ k ][ p ] indicates the number of submesh joint borders for the zippering instance k and submesh index p.

[0160] zp_zippering_Joint_borders_indices[ k ][ p ] [ b ] indicates the bth index corresponding to the submesh that shares a border with submesh p for the zippering instance k.

[0161] The zippering border implicit traversal is signaled for example as follows:

[0162] zp_border_implicit_traversal_type indicates the traversal type that can take values as specified in the following table, Table 3:Table 3

[0163] In the Edge traversal zp_border_implicit_traversal_type, following three example submesh vertex indices are provided:- zpbit_start_edge_vertex[O] indicates the first vertex index of the starting edge of the border traversal of submesh p with submesh index submeshindex for the zippering instance k;- zpbit_start_edge_vertex[l] indicates the second vertex index of the starting edge of the of the border traversal of submesh p with submesh index submeshindex for the zippering instance k; and- zpbit_end_vertex indicates the end vertex index of the border traversal of submesh p with submesh index submeshindex for the zippering instance k.

[0164] In the Triangle Strip zp_border_implicit_traversal_type, following three example submesh face indices are provided:- Zpbit_triangle_strip_start_triangle indicates the first triangle index of the starting triangle strip based border traversal of submesh p with submesh index submeshindex for the zippering instance k;- zpbit_triangle_strip_second_triangle indicates the second triangle index of the starting triangle strip based border traversal of submesh p with submesh index submeshindex for the zippering instance k; and- zpbit_triangle_strip_end_triangle indicates the end triangle index of the triangle strip based border traversal of submesh p with submesh index submeshindex for the zippering instance k.

[0165] FIG. 11 is an example apparatus 1100, which may be implemented in hardware, configured to implement the examples described herein. The apparatus 1100 comprises at least one processor 1102 (e.g., an FPGA and / or CPU), at least one memory 1104 including computer program code 1105, the computer program code 1105 having instructions to carry out the methods described herein, wherein the at least one memory 1104 and the computer program code 1105 are configured to, with the at least one processor 1102, cause the apparatus 1100 to implement circuitry, a process, component, module, or function (implemented with control module 1106) to implement the examples described herein, including encoding, signaling and / or decoding of inter submesh boundaries for video-based dynamic mesh coding. Optionally included encoder 1130 of the control module 1106 performs encoding, and optionally included decoder 1140 implements decoding. The at least one memory 1104 may be a non-transitory memory, a transitory memory, a volatile memory (e.g., RAM), or a non-volatile memory (e.g., ROM).

[0166] The apparatus 1100 includes a display and / or I / O interface 1108, which includes user interface (UI) circuitry and elements, that may be used to display features or a status of the methods described herein (e.g., as one of the methods is being performed or at a subsequent time), or to receive input from a user such as with using a keypad, camera, touchscreen, touch area, microphone, biometric recognition, one or more sensors, and the like. The apparatus 1100 includes one or more communication, e.g., network (N / W) interfaces (I / F(s)) 1110. The communication I / F(s) 1110 may be wired and / or wireless and communicate over the Internet / other network(s) via any communication technique including via one or more links 1124. The communication I / F(s) 1110 may comprise one or more transmitters or one or more receivers.

[0167] The transceiver 1116 comprises one or more transmitters 1118 and one or more receivers 1120. The transceiver 1116 and / or communication I / F(s) 1110 may comprise standard well-known components such as an amplifier, filter, frequency-converter, (de)modulator, and encoder / decoder circuitries and one or more antennas, such as antennas 1114 used for communication over wireless link 1122.

[0168] The control module 1106 of the apparatus 1100 comprises one of or both parts 1106- 1 and / or 1106-2, which may be implemented in a number of ways. The control module 1106 may be implemented in hardware as control module 1106-1, such as being implemented as part of the at least one processor 1102. The control module 1106-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the control module 1106 may be implemented as control module 1106-2, which is implemented as computer program code (having corresponding instructions) 1105 and is executed by the at least one processor 1102. For instance, the at least one memory 1104 store instructions that, when executed by the at least one processor 1102, cause the apparatus 1100 to perform one or more of the operations as described herein. Furthermore, the at least one processor 1102, the at least one memory 1104, and example algorithms (e.g., as flowcharts and / or signaling diagrams), encoded as instructions, programs, or code, are means for causing performance of the operations described herein.

[0169] The apparatus 1100 to implement the functionality of control 1106 may correspond to any of the apparatuses depicted herein. Alternatively, apparatus 1100 and its elements may not correspond to any of the other apparatuses depicted herein, as apparatus 1100 may be part of a self- organizing / optimizing network (SON) node or other node, such as a node in a cloud.

[0170] The apparatus 1100 may also be distributed throughout the network including within and between apparatus 1100 and any network element.

[0171] Interface 1112 enables data communication and signaling between the various items of apparatus 1100, as shown in FIG. 11. For example, the interface 1112 may be one or more buses such as address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. Computer program code (e.g., instructions) 1105, including control 1106may comprise object-oriented software configured to pass data or messages between objects within computer program code 1105. The apparatus 1100 need not comprise each of the features mentioned, or may comprise other features as well. The various components of apparatus 1100 may at least partially reside in a common housing 1128, or a subset of the various components of apparatus 1100 may at least partially be located in different housings, which different housings may include housing 1128.

[0172] FIG. 12 shows a schematic representation of non-volatile memory media 1200a (e.g. computer / compact disc (CD) or digital versatile disc (DVD)) and 1200b (e.g. universal serial bus (USB) memory stick) and 1200c (e.g. cloud storage for downloading instructions and / or parameters 1202 or receiving emailed instructions and / or parameters 1202) storing instructions and / or parameters 1202 which when executed by a processor allows the processor to perform one or more of the operations of the methods described herein.

[0173] FIG. 13 is an example method 1300, based on the example embodiments described herein. At 1302, the method 1300 includes, for each submesh of a plurality of submeshes, determining zero or more traversal paths of one or more primitive boundaries the each submesh shares with one or more neighboring submeshes; and determining one or more corresponding traversal paths of the one or more primitive boundaries in one or more corresponding neighboring submeshes. At 1304, the method 1300 includes signaling pairs of traversal paths of the one or more primitive boundaries between the one or more neighboring submeshes.

[0174] In an example, a primitive comprise one or more of a vertex, an edge, or a face; and the one or more primitive boundaries comprise one or more of vertex boundaries, edge boundaries, or face boundaries.

[0175] In an example, a primitive comprise a vertex, an edge, a face, or a combination thereof; and the one or more primitive boundaries comprise vertex boundaries, edge boundaries, face boundaries, or combination thereof.

[0176] The method 1300 may be performed with an apparatus described herein, for example, the any apparatus of FIG. 1 to FIG. 4, any apparatus of FIG. 11, or any other apparatus described herein.

[0177] FIG. 1400 is another example method 1400, based on the example embodimentsdescribed herein. At 1402, the method 1400 includes, for a boundary between a first submesh and a second submesh, decoding a start primitive and an end primitive. At 1404, the method 1400 includes reproducing for the first submesh and the second submesh a traversal path based on the start primitive and the end primitive.

[0178] In an example, a primitive comprises a vertex, an edge, or a face.

[0179] In an example, the primitive comprises a vertex, an edge, a face, or a combination thereof.

[0180] The method 1400 may be performed with an apparatus described herein, for example, the any apparatus of FIG. 1 to FIG. 4, any apparatus of FIG. 11, or any other apparatus described herein.

[0181] References to a ‘computer’, ‘processor’, etc. should be understood to encompass not only computers having different architectures such as single / multi-processor architectures and sequential / parallel architectures but also specialized circuits such as field -programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device such as instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device, etc.

[0182] As used herein, the term ‘circuitry’, ‘circuit’ and variants may refer to any of the following: (a) hardware circuit implementations, such as implementations in analog and / or digital circuitry, and (b) combinations of circuits and software (and / or firmware), such as (as applicable): (i) a combination of processor(s) or (ii) portions of processor(s) / software including digital signal processor(s), software, and one or more memories that work together to cause an apparatus to perform various functions, and (c) circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even when the software or firmware is not physically present. As a further example, as used herein, the term ‘circuitry’ would also cover an implementation of merely a processor (or multiple processors) or a portion of a processor and its (or their) accompanying software and / or firmware. The term ‘circuitry’ would also cover, for example and when applicable to the particular element, a baseband integrated circuitor applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or another network device. Circuitry or circuit may also be used to mean a function or a process used to execute a method.

[0183] It should be understood that the foregoing description is only illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.

[0184] The following acronyms and abbreviations that may be found in the specification and / or the drawing figures are defined as follows (the abbreviations may be appended with each other or with other characters using e.g. a hyphen or dash (-), and may be case insensitive):ASPS atlas sequence parameter setAFPS atlas frame parameter setATL atlas tile layerAVC advanced video codingBMSPS base-mesh sequence parameter setBMFPS base-mesh frame parameter setBMTL base-mesh tile layerHEVC high efficiency video codingHLS high level syntaxMIV MPEG immersive videoV3C visual volumetric video-based codingV-DMC video-based dynamic mesh codingV-PCC video-Based Point Cloud Compression

Claims

CLAIMSWhat is claimed is:

1. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: for each submesh of a plurality of submeshes: determining zero or more traversal paths of one or more primitive boundaries the each submesh shares with one or more neighboring submeshes; and determining one or more corresponding traversal paths of the one or more primitive boundaries in one or more corresponding neighboring submeshes; and signaling pairs of traversal paths of the one or more primitive boundaries between the one or more neighboring submeshes.

2. The apparatus of claim 1, wherein a primitive comprise one or more of a vertex, an edge, or a face, and wherein the one or more primitive boundaries comprise one or more of vertex boundaries, edge boundaries, or face boundaries.

3. The apparatus of any of claims 1 or 2, wherein the apparatus is further caused to perform: segmenting input mesh frame or a base mesh frame of the input mesh into a plurality of submeshes, wherein the plurality of submeshes comprise one or more primitives.

4. The apparatus of any of claims 1 to 3, wherein the apparatus is further caused to perform: storing, for each primitive, a correspondence between an original primitive index and the index of the each primitive in one or more submeshes the each primitive belongs to; storing following, for the each submesh: a list of neighbor submeshes of the each submesh, wherein the neighbor submeshes share a boundary; a list of at least one primitive that belongs to a submesh boundary; list of submeshes the at least one primitive belongs to; and a primitive index of the at least one primitive in the corresponding submeshes.

5. The apparatus of any of claims 3 or 4, wherein the segmentation is performed based on one or more criteria.

6. The apparatus of the claim 4, wherein the one or more criteria comprises a target number of submeshes, a geometry-based criterion, or a semantic -based criterion.

7. The apparatus of any of the claims 3 to 6, wherein: when the one more primitive boundaries comprise one or more of the vertex boundaries or the edge boundaries, the segmentation comprises: a non-overlapping segmentation that groups triangles such that a triangle of the input mesh or the base mesh is comprised in exactly one submesh, wherein a set of vertices of the triangle belong to the one or more submeshes, and wherein the set of vertices comprises a set of interior vertices and / or a set of boundary vertices, and wherein the set of boundary vertices that belong to more than one submesh, and wherein the set of interior vertices belong to one submesh; and when the one more primitive boundaries comprise one or more of the face boundaries, the segmentation comprises: an overlapping segmentation that groups one or more triangles such that a triangle comprises a boundary face that belongs to two or more submeshes and a submesh boundary vertex that belongs to the two one or more submeshes.

8. The apparatus of any of the claims 2 to 7, wherein the one or more vertex boundaries belong to a polyline or piecewise linear curve that connects corresponding vertices with exactly one edge per pair of vertices, and wherein submesh face boundaries are defined for overlapping segmentations and comprises a strip of triangles.

9. The apparatus of any of the claims 2 to 7, wherein the apparatus is further caused to perform: storing, for the each submesh, a list of faces that belonging to the submesh boundary, original indices of the face, and submesh indices of the face in the each submesh.

10. The apparatus of any of the preceding claims, wherein for determining the zero or more traversal paths, the apparatus is further caused to perform: determining a starting oriented primitive, a second primitive, and an end primitive; and iteratively identify a non-yet traversed neighboring boundary primitive of a lasttraversed boundary primitive, until reaching the end primitive.

11. The apparatus of any of the previous claims, wherein signaling traversal of the vertex boundary for the each submesh for zippering is done in an atlas or within an supplemental enhancement information (SEI) message.

12. The apparatus of claim 11, wherein signaling within the SEI message comprises signaling following: a number of joint border [k][p] for indicating a number of submesh joint borders for the zippering instances k and submesh indexes p; and joint borders indices [k] [p][b] for indicating a bth index corresponding to a submesh that shares a border with a submesh p for a zippering instance k.

13. The apparatus of any of the claims 1 to 10, wherein signaling traversal of the vertex boundary for the each submesh comprises a zippering border implicit traversal.

14. The apparatus of claim 13, wherein the zippering border implicit traversal comprises signaling a border implicit traversal type for indicating a traversal type that takes values a specified as following: the border implicit traversal type = 0 indicates that the traversal type comprises edge traversal; the border implicit traversal type = 1 indicates that the traversal type comprises triangle strip traversal; and the border implicit traversal type = 2 is reserved.

15. The apparatus of claim 14, wherein the apparatus is further caused to perform: providing following submesh vertices in the edge traversal: a start edge vertex[0] for indicating a first vertex index of a starting edge of a border traversal of submesh (p) with a submesh index (submeshindex) for a zippering instance k; a start edge vertex[l] for indicating a second vertex index of the starting edge of the border traversal of the submesh (p); and an end vertex for indicating an end vertex index of the border traversal of the submeshP-16. The apparatus of claim 14, wherein the apparatus is caused to perform: providing following submesh faces in the triangle strip traversal: a start triangle for indicating a first triangle index of a starting triangle strip based border traversal of a submesh (p) with a submesh index (submeshindex) for the zippering instance k; a second triangle for indicating a second triangle index of the starting triangle strip based border traversal of the submesh (p); and an end triangle for indicating an end triangle index of the triangle strip based border traversal of the submesh (p).

17. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: for a boundary between a first submesh and a second submesh decoding a start primitive and an end primitive; and reproducing for the first submesh and the second submesh a traversal path based on the start primitive and the end primitive.

18. The apparatus of claim 17, wherein a primitive comprises a vertex, an edge, or a face.

19. The apparatus of any of claims 17 or 18, wherein the apparatus is further caused to perform: determining corresponding pairs of boundary primitives from the first submesh and the second submesh; and fusing, pasting, zippering, or inpainting the first and the second submeshes along corresponding boundaries of the first and second submeshes.

20. A method comprising: for each submesh of a plurality of submeshes: determining zero or more traversal paths of one or more primitive boundariesthe each submesh shares with one or more neighboring submeshes; and determining one or more corresponding traversal paths of the one or more primitive boundaries in one or more corresponding neighboring submeshes; and signaling pairs of traversal paths of the one or more primitive boundaries between the one or more neighboring submeshes.

21. The method of claim 20, wherein a primitive comprise one or more of a vertex, an edge, or a face, and wherein the one or more primitive boundaries comprise one or more of vertex boundaries, edge boundaries, or face boundaries.

22. The method of any of claims 20 or 21 further comprising: segmenting input mesh frame or a base mesh frame of the input mesh into a plurality of submeshes, wherein the plurality of submeshes comprise one or more primitives.

23. The method of any of claims 20 to 22 further comprising: storing, for each primitive, a correspondence between an original primitive index and the index of the each primitive in one or more submeshes the each primitive belongs to; storing following, for the each submesh: a list of neighbor submeshes of the each submesh, wherein the neighbor submeshes share a boundary; a list of at least one primitive that belongs to a submesh boundary; list of submeshes the at least one primitive belongs to; and a primitive index of the at least one primitive in the corresponding submeshes.

24. The method of any of claims 22 or 23, wherein the segmentation is performed based on one or more criteria.

25. The method of the claim 23, wherein the one or more criteria comprises a target number of submeshes, a geometry-based criterion, or a semantic -based criterion.

26. The method of any of the claims 22 to 25, wherein: when the one more primitive boundaries comprise one or more of the vertex boundaries or the edge boundaries, the segmentation comprises: a non-overlapping segmentation thatgroups triangles such that a triangle of the input mesh or the base mesh is comprised in exactly one submesh, wherein a set of vertices of the triangle belong to the one or more submeshes, and wherein the set of vertices comprises a set of interior vertices and / or a set of boundary vertices, and wherein the set of boundary vertices that belong to more than one submesh, and wherein the set of interior vertices belong to one submesh; and when the one more primitive boundaries comprise one or more of the face boundaries, the segmentation comprises: an overlapping segmentation that groups one or more triangles such that a triangle comprises a boundary face that belongs to two or more submeshes and a submesh boundary vertex that belongs to the two one or more submeshes.

27. The method of any of the claims 21 to 26, wherein the one or more vertex boundaries belong to a polyline or piecewise linear curve that connects corresponding vertices with exactly one edge per pair of vertices, and wherein submesh face boundaries are defined for overlapping segmentations and comprises a strip of triangles.

28. The method of any of the claims 21 to 26 further comprising: storing, for the each submesh, a list of faces that belonging to the submesh boundary, original indices of the face, and submesh indices of the face in the each submesh.

29. The method of any of preceding claims 20 to 28 the determining the zero or more traversal paths further comprises: determining a starting oriented primitive, a second primitive, and an end primitive; and iteratively identify a non-yet traversed neighboring boundary primitive of a last traversed boundary primitive, until reaching the end primitive.

30. The method of any of the claims 20 to 29, wherein signaling traversal of the vertex boundary for the each submesh for zippering is done in an atlas or within an supplemental enhancement information (SEI) message.

31. The method of claim 30, wherein signaling within the SEI message comprises signaling following: a number of joint border [k][p] for indicating a number of submesh joint borders for the zippering instances k and submesh indexes p; andjoint borders indices [k] [p][b] for indicating a bth index corresponding to a submesh that shares a border with a submesh p for a zippering instance k.

32. The method of any of the claims 20 to 29, wherein signaling traversal of the vertex boundary for the each submesh comprises a zippering border implicit traversal.

33. The method of claim 32, wherein the zippering border implicit traversal comprises signaling a border implicit traversal type for indicating a traversal type that takes values a specified as following: the border implicit traversal type = 0 indicates that the traversal type comprises edge traversal; the border implicit traversal type = 1 indicates that the traversal type comprises triangle strip traversal; and the border implicit traversal type = 2 is reserved.

34. The method of claim 33 further comprising: providing following submesh vertices in the edge traversal: a start edge vertex[0] for indicating a first vertex index of a starting edge of a border traversal of submesh (p) with a submesh index (submeshindex) for a zippering instance k; a start edge vertex[l] for indicating a second vertex index of the starting edge of the border traversal of the submesh (p); and an end vertex for indicating an end vertex index of the border traversal of the submesh P-35. The method of claim 33, wherein the apparatus is caused to perform: providing following submesh faces in the triangle strip traversal: a start triangle for indicating a first triangle index of a starting triangle strip based border traversal of a submesh (p) with a submesh index (submeshindex) for the zippering instance k; a second triangle for indicating a second triangle index of the starting triangle strip based border traversal of the submesh (p); and an end triangle for indicating an end triangle index of the triangle strip based border traversal of the submesh (p).

36. A method comprising: for a boundary between a first submesh and a second submesh, decoding a start primitive and an end primitive; and reproducing for the first submesh and the second submesh a traversal path based on the start primitive and the end primitive.

37. The method of claim 36, wherein a primitive comprises a vertex, an edge, or a face.

38. The method of any of claims 36 or 37further comprising: determining corresponding pairs of boundary primitives from the first submesh and the second submesh; and fusing, pasting, zippering, or inpainting the first and the second submeshes along corresponding boundaries of the first and second submeshes.

39. An apparatus comprising: for each submesh of a plurality of submeshes: means for determining zero or more traversal paths of one or more primitive boundaries the each submesh shares with one or more neighboring submeshes; and means for determining one or more corresponding traversal paths of the one or more primitive boundaries in one or more corresponding neighboring submeshes; and means for signaling pairs of traversal paths of the one or more primitive boundaries between the one or more neighboring submeshes.

40. The apparatus of claim 39, wherein the apparatus further comprises means for performing the methods as claimed in any of the claims 21 to 35.

41. An apparatus comprising: for a boundary between a first submesh and a second submesh, means for decoding a start primitive and an end primitive; and means for reproducing for the first submesh and the second submesh a traversal path based on the start primitive and the end primitive.

42. The apparatus of claim 41, wherein the apparatus further comprises means for performing the methods as claimed in any of the claims 37 to 38.

43. A computer readable medium comprising program instructions which, when executed by an apparatus, cause the apparatus to perform the methods as claimed in any of the claims 20 to 35.

44. The computer readable medium of claim 43, wherein the computer readable medium comprises a non-transitory computer readable medium.

45. A computer readable medium comprising program instructions which, when executed by an apparatus, cause the apparatus to perform the methods as claimed in any of the claims 36 to 38.

46. The computer readable medium of claim 45, wherein the computer readable medium comprises a non-transitory computer readable medium.

Images