Mesh data encoding device, mesh data encoding method, mesh data decoding device and mesh data decoding method

Abstract

A decoding method according to embodiments may comprise the steps of: decoding a base mesh within a bitstream; decoding displacements within the bitstream; and decoding attributes within the bitstream. An encoding method according to embodiments may comprise the steps of: encoding a base mesh of mesh data; encoding displacements of the mesh data; and encoding attributes of the mesh data.

Description

Mesh data encoding device, mesh data encoding method, mesh data decoding device and mesh data decoding method

[0001] The embodiments provide a method for providing Point Cloud content to provide various services to users, such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services.

[0002] A point cloud is a collection of points in 3D space. There is a problem in that it is difficult to generate point cloud data because there are many points in 3D space.

[0003] Mesh data refers to a form of data in which connectivity information between the vertices of a mesh is added to point cloud data.

[0004] There is a problem in that a large amount of throughput is required to transmit and receive dynamic mesh data.

[0005] The technical problem according to the embodiments is to provide a point mesh data transmission device, a transmission method, a mesh data reception device, and a reception method for efficiently transmitting and receiving mesh data in order to solve the aforementioned problems, etc.

[0006] The technical problem according to the embodiments is to provide a point mesh transmission device, a transmission method, a mesh data receiving device, and a receiving method for solving latency and encoding / decoding complexity.

[0007] However, the scope of rights of the embodiments is not limited to the technical problems described above, and may be extended to other technical problems that can be inferred by a person skilled in the art based on the entire content of this document.

[0008] To achieve the above-described purpose and other advantages, the decoding method according to the embodiments may include the step of decoding a base mesh in a bitstream; the step of decoding a displacement in a bitstream; and the step of decoding an attribute in a bitstream. The encoding method according to the embodiments may include the step of encoding a base mesh of mesh data; the step of encoding a displacement of mesh data; and the step of encoding an attribute of mesh data.

[0009] A mesh data transmission method, a transmission device, a mesh data reception method, and a reception device according to the embodiments can provide a high-quality mesh service.

[0010] A mesh data transmission method, a transmission device, a mesh data reception method, and a reception device according to the embodiments can achieve various video codec methods.

[0011] The mesh transmission method, transmission device, mesh data reception method, and reception device according to the embodiments can provide general-purpose mesh content such as autonomous driving services.

[0012] Drawings are included to further understand the embodiments, and the drawings illustrate the embodiments along with descriptions related to the embodiments. For a better understanding of the various embodiments described below, one must refer to the description of the embodiments below in relation to the following drawings, which include parts corresponding to similar reference numerals throughout the drawings.

[0013] FIG. 1 shows a V-DMC-based encoder and decoder according to embodiments.

[0014] FIG. 2 shows a system for providing dynamic mesh content according to embodiments.

[0015] FIG. 3 illustrates a V-MESH compression method according to embodiments.

[0016] FIG. 4 shows the pre-processing of V-MESH compression according to the embodiments.

[0017] FIG. 5 illustrates a mid-edge subdivision method according to embodiments.

[0018] FIG. 6 illustrates a displacement generation process according to embodiments.

[0019] FIG. 7 illustrates the V-DMC encoding process according to the embodiments.

[0020] FIG. 8 illustrates a lifting conversion process for displacement according to embodiments.

[0021] FIG. 9 illustrates the process of packing conversion coefficients according to embodiments into a 2D image.

[0022] FIG. 10 illustrates the attribute transfer process of the V-MESH compression method according to the embodiments.

[0023] FIG. 11 illustrates a V-DMC decoding process according to embodiments.

[0024] FIG. 12 illustrates a V-DMC encoding process according to embodiments.

[0025] FIG. 13 illustrates a V-DMC decoding process according to embodiments.

[0026] FIG. 14 shows a base mesh decoder according to embodiments.

[0027] FIG. 15 shows a V-DMC bitstream according to embodiments.

[0028] FIG. 16 shows the base mesh frame parameter set RBSP syntax according to the embodiments.

[0029] FIG. 17 illustrates a encoding method according to embodiments.

[0030] FIG. 18 illustrates a decoding method according to embodiments.

[0031] Preferred embodiments of the embodiments are described in detail, and examples thereof are shown in the accompanying drawings. The following detailed description, with reference to the accompanying drawings, is intended to describe preferred embodiments of the embodiments rather than merely embodiments that may be implemented according to the embodiments. The following detailed description includes details to provide a thorough understanding of the embodiments. However, it is obvious to those skilled in the art that the embodiments may be practiced without these details.

[0032] Most terms used in the embodiments are selected from those commonly used in the field, but some terms are chosen at the applicant's discretion, and their meanings are described in detail in the following description as necessary. Accordingly, the embodiments should be understood based on the intended meaning of the terms, rather than their mere names or meanings.

[0033] FIG. 1 shows a V-DMC-based encoder and decoder according to embodiments.

[0034] The basic structure of the currently ongoing V-DMC (v-mesh) is shown in Fig. 1. The encoder and decoder according to Fig. 1 perform the encoding and decoding processes of media representing a dynamic mesh using V3C technology. The preprocessor converts the input dynamic mesh representation into several V3C components (base mesh, displacement set, 2D representation of attributes, and atlas). The original mesh is simplified into a base mesh. The base mesh can be encoded using any mesh codec. Displacement vectors can be represented by a profile or encoded into V3C geometric video components using any video codec via SEI messages. For example, depending on the profile, displacement vectors (displacement data) can be encoded using arithmetic coding. Attribute data may include additional attributes. For example, texture or material information may be included as additional attributes and can be encoded using any video codec. Atlas data contains information on how to perform inverse reconstruction and is provided to the V3C decoding and / or rendering system. For example, atlas data may include methods for performing subdivision of the base mesh, methods for applying displacement vectors to the vertices of the subdivided mesh, and methods for applying attributes to the reconstructed mesh.

[0035] The encoder may be composed of a memory and at least one processor connected to the memory. The at least one processor may be configured to perform operations such as a preprocessor, an atlas encoder, a basemesh encoder, a displacement vector encoder, a video encoder, and a multiplexer.

[0036] The atlas encoding unit encodes the atlas of the mesh data to generate an atlas bitstream. The basemesh encoding unit encodes the basemesh of the mesh data to generate a basemesh bitstream. The displacement vector encoding unit encodes the displacement vector of the mesh data to generate a displacement vector bitstream. The video encoding unit encodes the attributes of the mesh data to generate an attribute bitstream. The encoder generates parameter information (which may be referred to as signaling information, metadata, etc.) related to each encoding. The encoder can generate a bitstream containing parameter information, the atlas, the basemesh, the displacement vector, and / or attributes.

[0037] The decoder may be composed of a memory and at least one processor connected to the memory. The at least one processor may be configured to perform operations such as a demultiplexer, an atlas decoder, a basemesh decoder, a displacement vector decoder, and a video decoder.

[0038] The atlas decoder decodes the atlas within the bitstream. The basemesh decoder decodes the basemesh within the bitstream. The displacement vector decoder decodes the displacement vector within the bitstream. The video decoder decodes the attributes within the bitstream. The decoder can perform each decoding operation based on parameter information within the bitstream. The decoder can reconstruct dynamic mesh data based on the atlas, displacement vector, attributes, and basemesh.

[0039] Below, the operation of the V-DMC encoder and decoder of FIG. 1 is explained in more detail.

[0040] FIG. 2 shows a system for providing dynamic mesh content according to embodiments.

[0041] The system of FIG. 2 includes a point cloud data transmission device (100) and a point cloud data reception device (110) according to embodiments. The point cloud data transmission device may include a dynamic mesh video acquisition unit (101), a dynamic mesh video encoder (102), a file / segment encapsulator (103), and a transmitter (104). The point cloud data reception device (110) may include a receiver (111), a file / segment decapsulator (112), a dynamic mesh video decoder (113), and a renderer (114). Each component of FIG. 1 may correspond to hardware, software, a processor, and / or a combination thereof. Hereinafter, the point cloud data transmission device according to embodiments may be interpreted as a term referring to the transmission device (100) or the dynamic mesh video encoder (hereinafter, encoder) (102). The point cloud data receiving device according to the embodiments may be interpreted as a term referring to the receiving device (110) or the dynamic mesh video decoder (hereinafter, decoder) (113).

[0042] The system of Fig. 2 can perform video-based dynamic mesh compression and decompression.

[0043] With advancements in 3D capture, modeling, and rendering, users can access various forms of 3D content, such as AR, XR, the metaverse, and holograms, across multiple platforms and devices. 3D content represents objects more sophisticatedly and realistically to enable users to enjoy immersive experiences, and for this purpose, the creation and use of 3D models require a large amount of data. Among the various types of 3D content, 3D meshes are widely used for efficient data utilization and realistic object representation. The embodiments include a series of processing steps in a system that uses such mesh content.

[0044] First, the method for compressing dynamic mesh data originates from the V-PCC (Video-based point cloud compression) standard technology. Point cloud data consists of data containing color information along with vertex coordinates (X, Y, Z). Mesh data refers to data where connectivity information between vertices is added to this vertex data. Content can be created in a mesh data format from the outset when generating content. Point cloud data can be converted into mesh data and used by adding connectivity information.

[0045] Currently, the MPEG standards organization defines the data types for dynamic mesh data as the following two types: Category 1: Mesh data containing texture maps as color information. Category 2: Mesh data containing vertex colors as color information.

[0046] Mesh coding standards for Category 1 data are currently being developed, and standardization work for Category 2 data is also scheduled to proceed in the future. As shown in Fig. 1, the entire process for providing mesh content services may include an acquisition process, an encoding process, a transmission process, a decoding process, a rendering process, and / or a feedback process.

[0047] To provide mesh content services, 3D data acquired through multiple cameras or special cameras can be processed into a mesh data type through a series of processes and then generated as a video. The generated mesh video is transmitted after undergoing a series of processes, and at the receiving end, the received data can be processed back into a mesh video and rendered. Through this, the mesh video is provided to the user, and the user can use the mesh content according to their intention through interaction.

[0048] A mesh compression system may include a transmission device and a reception device. The transmission device can encode mesh video to output a bitstream and transmit it to the reception device via a digital storage medium or network in the form of a file or streaming (streaming segment). The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0049] The transmission device may schematically include a mesh video acquisition unit, a mesh video encoder, and a transmission unit. The receiving device may schematically include a receiver, a mesh video decoder, and a renderer. The encoder may be referred to as a mesh video / image / picture / frame encoding device, and the decoder may be referred to as a mesh video / image / picture / frame decoding device. The transmitter may be included in the mesh video encoder. The receiver may be included in the mesh video decoder. The renderer may include a display unit, and the renderer and / or the display unit may be composed of separate devices or external components. The transmission device and the receiving device may further include separate internal or external modules / units / components for a feedback process.

[0050] Mesh data represents the surface of an object using multiple polygons. Each polygon is defined by a vertex in 3D space and connectivity information indicating how those vertices are connected. It may also include vertex attributes such as color and normals. Mapping information, which enables the surface of the mesh to be mapped onto a 2D planar area, can also be included as an attribute of the mesh. Mapping can generally be described by a set of parametric coordinates, referred to as UV coordinates or texture coordinates, associated with the mesh vertices. The mesh contains a 2D attribute map, which can be used to store high-resolution attribute information such as textures, normals, and displacements.

[0051] The mesh video acquisition unit may include processing 3D object data acquired through a camera, etc., into a mesh data type having the attributes described above through a series of processes, and generating a video composed of such mesh data. In a mesh video, the attributes of the mesh, namely vertices, polygons, connectivity information between vertices, color, normals, etc., may change over time. A mesh video having attributes and connectivity information that change over time in this way can be described as a dynamic mesh video.

[0052] A mesh video encoder can encode input mesh video into one or more video streams. A single video may contain multiple frames, and a single frame may correspond to a still image or picture. In this document, the term "mesh video" may include mesh images, frames, or pictures, and the terms mesh video and mesh images, frames, or pictures may be used interchangeably. A mesh video encoder can perform a Video-based Dynamic Mesh (V-Mesh) Compression procedure. To improve compression and coding efficiency, a mesh video encoder can perform a series of procedures such as prediction, transformation, quantization, and entropy coding. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0053] The file / segment encapsulation module can encapsulate encoded mesh video data and / or mesh video-related metadata into a file or the like. Here, the mesh video-related metadata may be received from a metadata processing module or the like. The metadata processing module may be included in the mesh video encoder or may be configured as a separate component / module. The encapsulation module can encapsulate the data into a file format such as ISOBMFF or process it into other forms such as DASH segments. Depending on the embodiment, the encapsulation module may include mesh video-related metadata in the file format. Mesh video metadata may be included, for example, in various levels of boxes in the ISOBMFF file format or as data within a separate track in the file. Depending on the embodiment, the encapsulation module may encapsulate the mesh video-related metadata itself into a file.

[0054] The transmission processing unit can apply transmission processing to mesh video data encapsulated according to the file format. The transmission processing unit may be included in the transmission unit or may be configured as a separate component / module. The transmission processing unit can process mesh video data according to any transmission protocol. Processing for transmission may include processing for delivery via a broadcast network and processing for delivery via broadband. According to an embodiment, the transmission processing unit may receive mesh video-related metadata from the metadata processing unit in addition to mesh video data, and apply transmission processing to it.

[0055] The transmission unit can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmission unit may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit can extract the bitstream and transmit it to a decoding device.

[0056] The receiver can receive mesh video data transmitted by a mesh video transmission device. Depending on the transmission channel, the receiver may receive mesh video data via a broadcast network or via broadband. Alternatively, it may receive mesh video data via a digital storage medium.

[0057] The receiving processing unit can perform processing on the received mesh video data according to the transmission protocol. The receiving processing unit may be included in the receiving unit or may be configured as a separate component or module. Corresponding to the processing for transmission performed on the transmitting side, the receiving processing unit may perform the reverse process of the aforementioned transmission processing unit. The receiving processing unit may transmit the acquired mesh video data to the decapsulation processing unit and the acquired mesh video-related metadata to the metadata parser. The mesh video-related metadata acquired by the receiving processing unit may be in the form of a signaling table.

[0058] The file / segment decapsulation module can decapsulate mesh video data in file format received from the receiving module. The decapsulation module can decapsulate files based on ISOBMFF, etc., to obtain a mesh video bitstream or mesh video-related metadata (metadata bitstream). The obtained mesh video bitstream can be transmitted to a mesh video decoder, and the obtained mesh video-related metadata (metadata bitstream) can be transmitted to a metadata processing module. The mesh video bitstream may also contain metadata (metadata bitstream). The metadata processing module may be included in the mesh video decoder or configured as a separate component / module. The mesh video-related metadata obtained by the decapsulation module may be in the form of boxes or tracks within the file format. If necessary, the decapsulation module may receive metadata required for decapsulation from the metadata processing module. Mesh video-related metadata may be passed to a Mesh video decoder and used in the Mesh video decoding process, or passed to a renderer and used in the Mesh video rendering process.

[0059] The Mesh Video Decoder can decode video by receiving a bitstream as input and performing operations corresponding to those of the Mesh Video Encoder. The decoded Mesh Video can be displayed through a display unit. The user can view all or part of the rendered result through a VR / AR display or a standard display.

[0060] The feedback process may include the process of transmitting various feedback information obtainable during the rendering / display process to the transmitting side or to the decoder of the receiving side. Interactivity in mesh video consumption may be provided through the feedback process. According to an embodiment, head orientation information, viewport information indicating the area the user is currently viewing, etc., may be transmitted during the feedback process. According to an embodiment, the user may interact with elements implemented in a VR / AR / MR / autonomous driving environment, and in this case, information related to such interaction may be transmitted to the transmitting side or the service provider side during the feedback process. According to an embodiment, the feedback process may not be performed.

[0061] Head orientation information can refer to information regarding the user's head position, angle, movement, etc. Based on this information, information about the area the user is currently viewing within the mesh video—that is, viewport information—can be calculated.

[0062] Viewport information may be information about the area currently being viewed by the user within a mesh video. Through this, gaze analysis can be performed to determine how the user consumes the mesh video and to what extent they gaze at specific areas of the mesh video. Gaze analysis may be performed at the receiving end and transmitted to the transmitting end via a feedback channel. Devices such as VR / AR / MR displays can extract the viewport area based on the user's head position / orientation, the vertical or horizontal FOV supported by the device, etc.

[0063] According to an embodiment, the aforementioned feedback information may not only be transmitted to the transmitting side but may also be consumed at the receiving side. That is, decoding and rendering processes at the receiving side may be performed using the aforementioned feedback information. For example, using head orientation information and / or viewport information, only the mesh video for the area currently viewed by the user may be preferentially decoded and rendered.

[0064] This document relates to dynamic mesh video compression as described above. The methods / embodiments disclosed in this document may be applied to the MPEG (Moving Picture Experts Group) Video-based Dynamic Mesh Compression Method (V-Mesh) standard or next-generation video / image coding standards. Dynamic mesh video compression is a method for processing mesh connection information and attributes that change over time, and it can perform lossy and lossless compression for various applications such as real-time communication, storage, free-viewpoint video, and AR / VR.

[0065] The dynamic mesh video compression method described below is based on MPEG's V-Mesh method.

[0066] In this document, "picture" or "frame" generally refers to a unit representing a single image of a specific time period.

[0067] A pixel or pel may refer to the smallest unit that constitutes a picture (or image). Additionally, the term 'sample' may be used as a counterpart to pixel. A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the lumina component, only the pixel / pixel value of the chroma component, or only the pixel / pixel value of the depth component.

[0068] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to that area. Depending on the case, the term unit may be used interchangeably with terms such as block or area. In general, an MxN block may include samples (or sample arrays) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0069] The encoding process of Fig. 2 is as follows.

[0070] Video-based dynamic mesh compression (V-Mesh) compression methods can provide a method for compressing dynamic mesh video data based on 2D video codecs such as HEVC and VVC. In the V-Mesh compression process, compression is performed by receiving the following data as input.

[0071] Input mesh: It includes the 3D coordinates (geometry) of the vertices constituting the mesh, normal information for each vertex, mapping information for mapping the mesh surface to a 2D plane, and connection information between the vertices constituting the surface. The surface of the mesh can be represented by triangles or polygons of greater size, and connection information between the vertices constituting each surface is stored according to a defined shape. The input mesh can be saved in the OBJ file format.

[0072] Attribute map: (Hereafter, Texture map is used with the same meaning): It contains information on the attributes (color, normals, displacement, etc.) of a mesh and stores data in the form of mapping the mesh surface onto a 2D image. Mapping which part of the mesh (surface or vertex) corresponds to each data point in this attribute map is based on the mapping information contained in the input mesh. Since the attribute map holds data for each frame of the mesh video, it can also be referred to as an attribute map video (or simply "attribute"). In the V-Mesh compression method, the attribute map primarily contains the mesh's color information and is stored in image file formats (PNG, BMP, etc.).

[0073] Material Library File: Contains material attribute information used in the mesh, specifically information that links the input mesh with its corresponding attribute map. This is saved in the Wavefront Material Template Library (MTL) file format.

[0074] In the V-Mesh compression method, the following data and information can be generated through the compression process.

[0075] Base Mesh: By simplifying (decimating) the input mesh through a preprocessing stage, objects from the input mesh are represented using the minimum number of vertices determined according to user criteria.

[0076] Displacement: Displacement information used to represent the input mesh as similarly as possible using the base mesh, expressed in the form of 3D coordinates.

[0077] Atlas information: This is metadata required to reconstruct a mesh using base mesh, displacement, and attribute map information. It can be generated and utilized as sub-units (sub-mesh, patch, etc.) that constitute the mesh.

[0078] Referring to FIGS. 3 to 7, a method for encoding mesh position information (vertices) is described, and referring to FIGS. 7-10 and others, a method for encoding attribute information (attribute map) by restoring mesh position information is described.

[0079] FIG. 3 illustrates a V-MESH compression method according to embodiments.

[0080] FIG. 3 illustrates the encoding process of FIG. 2, and the encoding process may include pre-processing and encoding processes. The encoder of FIG. 2 may include a pre-processor (200) and an encoder (201) as in FIG. 3. The transmitting device of FIG. 2 may be broadly referred to as an encoder, and the dynamic mesh video encoder of FIG. 2 may be referred to as an encoder. The V-Mesh compression method may include pre-processing (200) and encoding (201) processes as in FIG. 3. The pre-processor of FIG. 3 may be located in front of the encoder of FIG. 3. The pre-processor and encoder of FIG. 3 may be referred to as a single encoder.

[0081] The pre-processor can receive a static dynamic mesh and / or attribute map. The pre-processor can generate a base mesh and / or displacement through preprocessing. The pre-processor can receive feedback information from an encoder and generate a base mesh and / or displacement based on the feedback information.

[0082] The encoder can receive a base mesh, displacement, static and dynamic meshes, and / or attribute maps. The encoder can encode mesh-related data to generate a compressed bitstream.

[0083] FIG. 4 shows the pre-processing of V-MESH compression according to the embodiments.

[0084] Figure 4 shows the configuration and operation of the pre-processor of Figure 3.

[0085] FIG. 3 illustrates a process of performing preprocessing on an input mesh. The preprocessing process (200) may include four main steps: 1) generation of a Group of Frame (GoF), 2) mesh decimation, 3) UV parameterization, and 4) fitting subdivision surface (300). The pre-processor (200) may receive the input mesh, generate displacement and / or a base mesh, and transmit it to the encoder (201). The pre-processor (200) may transmit GoF information associated with GoF generation to the encoder (201).

[0086] Below, each step of Fig. 4 is explained.

[0087] GoF Generation: This is the process of creating a reference structure for mesh data. If the number of vertices, texture coordinates, vertex connectivity, and texture coordinate connectivity of the mesh in the previous frame and the current mesh are all identical, the previous frame can be set as the reference frame. In other words, if only the vertex coordinate values ​​differ between the current input mesh and the reference input mesh, inter-frame encoding can be performed. Otherwise, the frame undergoes intra-frame encoding.

[0088] Mesh Decimation: This is the process of simplifying the input mesh to generate a simplified mesh, or base mesh. After selecting vertices to remove from the original mesh based on user-defined criteria, the selected vertices and the triangles connected to them can be removed.

[0089] In the process of performing mesh decimation, information regarding the input mesh (voxelized), target triangle ratio (TTR), and minimum triangle component (CCCount) is passed as input, and a decimated mesh can be obtained as output. In this process, connected triangle components smaller than the set minimum triangle component (CCCount) can be removed.

[0090] UV Parameterization: This is the process of mapping 3D surfaces to a texture domain for a decimated mesh. Parameterization can be performed using a UV Atlas tool. Through this process, mapping information is generated regarding where each vertex of the decimated mesh can be mapped to on a 2D image. This mapping information is expressed and stored as texture coordinates, and the final base mesh is generated through this process.

[0091] OrthoAtlas technology is a technique that generates texture coordinates using orthographic projection. In orthoAtlas technology, the processes of patch generation and patch packing are performed sequentially. First, Connected Components (CCs) are generated by splitting adjacent triangles, and then the optimal CCs are merged using a cost function to generate a patch. The cost function can measure the cost based on the degree of distortion that occurs when orthogonally projecting the patch in each direction. Finally, texture coordinates can be calculated by packing the patch that minimizes the cost function into the texture domain. In the case of orthoAtlas technology, texture coordinates can be derived in the base mesh decoder without compressing texture coordinate and texture connection information during the base mesh encoding process.

[0092] Fitting subdivision surface: This is the process of performing subdivision on a decimated mesh. User-defined methods, such as the mid-edge method, can be applied as the subdivision method. A fitting process is performed to make the input mesh and the subdivision mesh similar to each other.

[0093] This is a process of performing fitting so that the mesh obtained by subdividing the base mesh becomes similar to the surface of the input mesh. As for the subdivision method, a user-defined method such as the mid-edge method (Fig. 5), loop method, or LS3 method may be applied.

[0094] FIG. 5 illustrates a mid-edge subdivision method according to embodiments.

[0095] Figure 5 illustrates the mid-edge method of the fitting subdivision surface described in Figure 4. Referring to Figure 5, an original mesh containing 4 vertices is subdivided to create a sub-mesh. A sub-mesh can be created by creating a new vertex at the midpoint of the edge between vertices.

[0096] When a fitted subdivided mesh (hereinafter referred to as the fitted subdivided mesh) is generated, displacement is calculated using this result and a pre-compressed and decoded base mesh (hereinafter referred to as the reconstructed base mesh). That is, the reconstructed base mesh is subdivided in the same way as the fitting subdivision surface. The positional difference between this result and the fitted subdivided mesh for each vertex becomes the displacement for each vertex. Since displacement represents a positional difference in three-dimensional space, it is also expressed as a value in the (x, y, z) space of the Cartesian coordinate system. Depending on user input parameters, (x, y, z) coordinate values ​​can be converted into (normal, tangential, bi-tangential) coordinate values ​​of the local coordinate system.

[0097] FIG. 6 illustrates a displacement generation process according to embodiments.

[0098] FIG. 6 illustrates in detail the displacement calculation method of a fitting subdivision surface (300) as described in FIG. 5.

[0099] An encoder and / or pre-processor according to the embodiments may include 1) a subdivision unit, 2) a local coordinate system calculation unit, and 3) a displacement calculation unit. The subdivision unit may receive a restored base mesh and generate a subdivided restored base mesh. The local coordinate system calculation unit may receive a fitted subdivided mesh and a subdivided restored base mesh and convert a coordinate system relating to the mesh to a local coordinate system. The local coordinate system calculation operation may be optional. The displacement calculation unit calculates the positional difference between the fitted subdivision mesh and the subdivided restored base mesh. For example, it may generate a positional difference value between the vertices of the two input meshes. The vertex positional difference value becomes the displacement.

[0100] The method and apparatus for transmitting point cloud data according to the embodiments can encode the point cloud as follows. The point cloud data according to the embodiments (which may be referred to simply as point cloud) may refer to data including vertex coordinates and color information. Point cloud is a term that includes mesh data, and in this document, point cloud and mesh data may be used interchangeably.

[0101] The V-Mesh compression (restoration) method according to the embodiments may include intra-frame encoding (Fig. 6) and inter-frame encoding (Fig. 7).

[0102] Intra-frame encoding or inter-frame encoding is performed based on the results of the aforementioned GoF generation. In the case of intra-frame encoding, the data to be compressed may include a base mesh, displacement, and attribute map. In the case of inter-frame encoding, the data to be compressed may include displacement, attribute map, and the motion field between the reference base mesh and the current base mesh.

[0103] FIG. 7 illustrates the V-DMC encoding process according to the embodiments.

[0104] The encoding process of Fig. 7 illustrates the encoding of Figs. 1 and 2 in detail.

[0105] A pre-processor receives an input mesh and can perform the aforementioned preprocessing. Through preprocessing, a base mesh and / or a fitted subdivided mesh can be generated. A quantizer can quantize the base mesh and / or the fitted subdivided mesh. A static mesh encoder can encode a static mesh. The static mesh encoder can generate a bitstream containing the encoded base mesh. A motion encoder can encode motion vectors for the base mesh based on inter-frame motion estimation and motion compensation for inter-prediction. An atlas encoder can encode an atlas for the vertices of the base mesh. The encoded base mesh can be reconstructed and inversely quantized through an inverse quantizer. A displacement computer receives the reconstructed mesh and, based on the fitted subdivided mesh, can generate displacement, which is the position difference. A lifting transform can receive the displacement and generate lifting coefficients. The quantizer can quantize the lifting coefficients. Depending on the encoding method, the image packing unit can pack the image based on the quantized lifting coefficients. The video encoder can encode the packed image. Depending on the encoding method, it can apply interprediction to the quantized lifting coefficients and encode the predicted lifting coefficients according to an arithmetic encoding method. The mesh restoration unit restores the deformed mesh through the restored displacement and the restored base mesh. The displacement data is restored, and the deformed mesh is restored based on the restored displacement data and the restored base mesh and provided to the attribute transfer. The attribute transfer receives the input mesh and / or input attribute map and generates an attribute map based on the restored deformed mesh. Push-pull padding can pad data into the attribute map based on a push-pull method. The color space converter can convert the space of the color component that is an attribute. The video encoder can encode the attributes.A multiplexer can generate a bitstream by multiplexing a compressed base mesh, compressed displacement, and compressed attributes.

[0106] Base Mesh Encoding: Base mesh compression methods can be divided into INTRA, INTER, and SKIP types depending on the base mesh type, and encoding can be performed in different ways for each. If the base mesh is of the INTRA type, it can be encoded using a static mesh encoding method. If the base mesh is of the INTER type, the motion field between the reference base mesh and the current base mesh can be encoded. If the current base mesh is of the SKIP type, the reference base mesh can be derived into the current base mesh.

[0107] After being encoded in the encoder, the base mesh can be subdivided into a subdivided mesh through a subdivision process. Subdivision algorithms such as mid-point subdivision and loop subdivision can be used.

[0108] Static Basemesh Encoding (Intra Basemesh Encoding): When performing intra encoding on the current basemesh, the base mesh generated during the preprocessing stage can be encoded using static mesh compression technology after undergoing a quantization process. Static mesh compression utilizes MPEG EdgeBreaker (MEB) technology, and the base mesh's vertex position information, mapping information (texture coordinates), vertex connectivity information, and normals are subject to compression.

[0109] The technology for compressing connection information can be encoded based on the edgebreaker algorithm. The edgebreaker algorithm is a technique that sequentially traverses triangles according to rules, maps symbols based on the characteristics of each triangle, and then encodes the corresponding symbols.

[0110] Techniques for compressing vertex location information can calculate predicted values ​​based on prediction techniques such as multiple parallelogram prediction, and then encode the residual value, which is the difference between the current vertex and the predicted value.

[0111] A technique for compressing mapping information (texture coordinates) can calculate a predicted value based on a prediction technique such as stretching, and then encode the residual value, which is the difference between the current mapping information (texture coordinates) and the predicted value.

[0112] Normal compression techniques can obtain predicted values ​​based on prediction techniques such as delta coding, multiple parallelogram prediction, and cross product-based prediction, and then encode the residual value, which is the difference between the current normal and the predicted value.

[0113] Motion Field Encoding (Inter Basemesh Encoding): Inter basemesh encoding can be performed when a one-to-one correspondence exists between the reference mesh and the current input mesh, differing only in vertex position information. When performing inter encoding, instead of compressing the base mesh, the difference between the vertices of the reference base mesh and the current base mesh—that is, the motion field—is calculated and this information is encoded. The reference base mesh is the result of quantizing already decoded base mesh data and is determined by the reference frame index.

[0114] The motion field can be encoded as is, or the predicted motion field can be calculated by averaging the motion fields of the restored vertices among those connected to the current vertex, and the residual motion field, which is the difference between the predicted motion field value and the current vertex's motion field value, can be encoded. This value can be encoded using entropy coding.

[0115] Displacement Encoding: After encoding the base mesh, reconstruction and inverse quantization are performed to reconstruct it. Once the base mesh is generated and subdivision is performed on it, the displacement between the result and the fitted subdivided mesh can be calculated. For effective encoding, data transform processes such as wavelet transform can be applied to the displacement information, and Figure 7 shows the process of transforming displacement information using the lifting transform in V-Mesh. The transform coefficients generated through the transformation process are quantized, and the quantized transform coefficients can be compressed through a video codec or through arithmetic coding, depending on the compression method.

[0116] When compressed through a video codec, the data is packed into a 2D image as shown in Figure 8. Transform coefficients are organized into one block for every N^2 (N*N) units, and each block can be packed in z-scan order. The number of horizontal blocks is fixed at N, while the number of vertical blocks can be determined by the number of vertices in the subdivided base mesh. Within a single block, transform coefficients can be packed by aligning them using Morton code. The packed images generate a displacement video for every GoF unit, and this displacement video can be encoded using an existing video compression codec.

[0117] When compressed via arithmetic coding, cross-frame prediction can be performed on the quantized displacement vector transformation coefficients. When cross-frame prediction is performed on the current quantized displacement vector transformation coefficients, the residual value, which is the difference between the current displacement vector transformation coefficient and the reference displacement vector transformation coefficient, can be encoded, and information about the reference target can be encoded. Depending on the displacement vector type, the quantized displacement vector transformation coefficients can be arithmetic encoded if it is of the INTRA type, and the residual value if it is of the INTER type. Arithmetic coding can be performed based on Context Adaptive Binary Arithmetic Coding (CABAC). The CABAC process can first binarize the displacement vector data and map it to a bin string. The bin string can be an output binarized into 0s and 1s, where each 0 or 1 can be a bin. Each bin can be arithmetic encoded using context information selected from the context model, and a process of updating probabilities can be performed.

[0118]

[0119] FIG. 8 illustrates a lifting conversion process for displacement according to embodiments.

[0120] FIG. 9 illustrates the process of packing conversion coefficients according to embodiments into a 2D image.

[0121] FIGS. 8-9 respectively illustrate the process of converting the displacement of the encoding process of FIG. 7 and the process of packing the conversion coefficients.

[0122] The encoding method according to the embodiments includes displacement encoding.

[0123] After encoding the base mesh through base mesh encoding and / or motion field encoding, a reconstructed base mesh is generated through reconstruction and inverse quantization. Subdivision is then performed on this reconstructed base mesh, and the displacement between the result and the fitted subdivided mesh generated through the fitting subdivision surface can be calculated. For effective encoding, a data transform process such as a wavelet transform can be applied to the displacement information.

[0124] FIG. 8 illustrates the process of transforming displacement information using a lifting transform in V-Mesh. The transformation coefficients generated through the transformation process are quantized and then packed into a 2D image as shown in FIG. 9. The transformation coefficients are organized into one block for every 256 (=16×16) units, and each block can be packed in z-scan order. The number of horizontal blocks is fixed at 16, while the number of vertical blocks can be determined by the number of vertices of the subdivided base mesh. Within a single block, the transformation coefficients can be packed by aligning them using a Morton code. The packed images generate a displacement video for every GoF unit, and this displacement video can be encoded using a conventional video compression codec.

[0125] Referring to FIG. 8, the base mesh (original) may include vertices and edges for LoD0. A first subdivision mesh generated by dividing the base mesh includes vertices generated by further dividing the edges of the base mesh. The first subdivision mesh includes vertices for LoD0 and vertices for LoD1. LoD1 includes subdivided vertices and vertices of the base mesh (LoD0). A second subdivision mesh may be generated by dividing the first subdivision mesh. The second subdivision mesh includes LoD2. LoD2 includes base mesh vertices (LoD0), LoD1 containing vertices additionally generated from LoD0, and vertices additionally divided from LoD1. LoD is a Level of Detail indicating the degree of detail; as the index of the level increases, the distance between vertices becomes closer, and the level of detail increases. LoD N includes the vertices included in the previous LoDN-1 as they are. When vertices are further subdivided through subdivision, the mesh can be encoded based on a prediction and / or update method by considering the previous vertices v1 and v2 and the subdivided vertex v. Instead of encoding the information for the current LoD N as is, the size of the bitstream can be reduced by generating residuals between the previous LoD N-1 and encoding the mesh using these residuals. The prediction process refers to the operation of predicting the current vertex v based on the previous vertices v1 and v2. Since adjacent subdivided meshes possess similar data, efficient encoding can be achieved by utilizing this property. The current vertex position information is predicted using the residuals from the previous vertex position information, and the previous vertex position information is updated using these residuals.

[0126] Referring to FIG. 9, the vertices have coefficients generated through a lifting transformation. The coefficients of the vertices related to the lifting transformation can be encoded by packing them into an image.

[0127] FIG. 10 illustrates the attribute transfer process of the V-MESH compression method according to the embodiments.

[0128] Figure 10 shows the detailed operation of the attribute transfer of the encoding of Figure 7.

[0129] The encoding according to the embodiments includes attribute map encoding.

[0130] Information regarding the input mesh is compressed through base mesh encoding, motion field encoding, and displacement encoding. The input mesh compressed during the encoding process is restored through base mesh decoding (intra frame), motion field encoding (inter frame), and displacement video decoding. The resulting restored deformed mesh (hereinafter referred to as Recon. deformed mesh) is used to compress the input attribute map, as shown in FIGS. 6 and 7. The Recon. deformed mesh possesses vertex position information, texture coordinates, and corresponding connection information, but lacks color information corresponding to the texture coordinates. Accordingly, as shown in FIG. 10, a new attribute map having color information corresponding to the texture coordinates of the reconstructed deformed mesh is generated through the attribute transfer process in the V-Mesh compression method.

[0131] Attribute transfer first checks for every point P(u, v) in the 2D texture domain whether the point belongs to a texture triangle of the reconstructed deformed mesh, and if it does, calculates the barycentric coordinates (α, β, γ) of P(u, v) according to that triangle T. Then, using the 3D vertex position of triangle T and (α, β, γ), calculates the 3D coordinates M(x, y, z) of P(u, v). In the input mesh domain, finds the vertex coordinates M'(x', y', z') corresponding to the position most similar to the calculated M(x, y, z) and triangle T' containing this point. Then, the center of mass coordinates (α', β', γ') of M'(x', y', z') in this triangle T' are calculated. Using the texture coordinates corresponding to the three vertices of Triangle T' and (α', β', γ'), texture coordinates (u', v') are calculated, and the color information corresponding to these coordinates is found in the Input Attribute Map. The color information found in this way is then assigned to the pixel location (u, v) in the new Attribute Map. If P(u, v) does not belong to any triangle, the pixel at that location in the new Attribute Map can be filled with a color value using a padding algorithm such as the Push-Pull algorithm.

[0132] The new attribute map generated through attribute transfer is bundled in GoF units to form an attribute map video, which is then compressed using a video codec.

[0133] Atlas Encoding: Atlas information may be transmitted during the aforementioned process. The Atlas consists of information required during the mesh decoding and / or rendering process, and may include information required during the process of performing subdivision, displacement decoding, base mesh decoding, etc., as well as tile information, patch information, etc. Atlas data may be encoded using Exp-Golomb coding, etc.

[0134] Referring to FIG. 10, the reference relationships between the input mesh, input attribute map, restored mesh, and generated attribute map can be seen.

[0135] The decoding process of Fig. 1 can perform the reverse process of the corresponding process of the encoding process of Fig. 1. The specific decoding process is as follows.

[0136] FIG. 11 illustrates a VV-DMC decoding process according to embodiments.

[0137] Figure 11 shows the configuration and operation of the decoder of the receiving device of Figure 1.

[0138] The input bitstream can be separated into a Basemesh sub-stream, a Displacement sub-stream, an Attribute map sub-stream, and an Atlas sub-stream.

[0139] The Atlas sub-stream can be decoded through Exp-Golomb coding, and as a result, information necessary for performing decoding, tile information, patch information, etc. can be obtained.

[0140] Depending on the basemesh type, if the basemesh sub-stream is of the INTRA type, it can be decoded through a static mesh decoder based on MEB (MPEG EdgeBreaker) technology, and as a result, the connectivity information, vertex geometry information, and vertex mapping information (texture coordinates) of the base mesh can be restored.

[0141] When the texture parameterization method in the encoder is orthoAtlas, the decoder can derive mapping information (texture coordinates) and attribute information (texture) connection information using vertex coordinates. The process of deriving mapping information (texture coordinates) and connection information can generate mapping information (texture coordinates) and attribute information (texture) connection information by calculating the homography transform of each face and then projecting the vertex based on it.

[0142] If the Basemesh type is INTER type, motion information can be decoded through entropy decoding and inverse prediction processes. The restored motion information is combined with the reference Basemesh that has already been restored and stored in the buffer to generate a Reconstructed quantized basemesh for the current frame. An inverse quantization process can be performed on the restored Basemesh.

[0143] If the displacement sub-stream is compressed through a video codec according to the compression method used in encoding, it is decoded into a displacement video through the video compression codec's decoder, and then the image unpacking process is performed.

[0144] When compressed through arithmetic coding, the displacement vector bitstream can decode binarized syntax elements through arithmetic decoding, and a Contextual Probability Model (CPM) can be adaptively determined according to each bin of the syntax elements, and arithmetic decoding can be performed by predicting the probability of occurrence of the bin through the CPM. The binarized syntax elements can be decoded through inverse binarization. Quantized displacement vector transformation coefficients can be derived from the decoded syntax elements. Depending on the displacement information type, if it is INTER (where inter prediction is performed), an inverse inter prediction process is performed using reference information for the quantized displacement coefficients.

[0145] The quantized displacement coefficient is restored as displacement information for each vertex through inverse quantization, inverse transform, and coordinate system transformation processes.

[0146] The restored Base mesh and the restored Displacement information are combined to generate the final Decoded mesh. The Attribute map sub-stream is decoded through the decoder of the video compression codec used in Encoding, and then restored to the final Attribute map through processes such as color format conversion.

[0147] The restored Decoded mesh and Decoded attribute map can be utilized at the receiving end as final mesh data available to the user.

[0148] The atlas decoder decodes atlas data within the bitstream.

[0149] When mesh data within the bitstream is encoded based on inter-prediction, the motion decoder derives the motion field of the basemesh of the current frame through motion estimation and compensation, based on the basemesh within the reference frame. When mesh data within the bitstream is encoded based on intra-prediction, the sectic decoder decodes the basemesh. Depending on the encoding method, it decodes displacement data by applying either arithmetic coding or video decoding. The video decoder decodes attribute data within the bitstream.

[0150] The decoding method of FIG. 11 can follow the inverse process of the encoding method according to the embodiments.

[0151] FIG. 12 illustrates a V-DMC encoding process according to embodiments.

[0152] FIG. 12 illustrates the configuration and operation of an encoder of a transmitting device such as FIG. 1 or FIG. 2. Each component of FIG. 12 corresponds to hardware, software, a processor, and / or a combination thereof.

[0153] Figure 12 shows the encoding process of V-Mesh technology.

[0154] The mesh preprocessing unit receives the original mesh as input and generates a decimated mesh. Decimation can be performed based on the number of target vertices or polygons constituting the mesh. For the decimated mesh, parameterization can be performed to generate mapping information (texture coordinates) and attribute information (texture) connection information per vertex. Additionally, floating-point mesh information can be quantized into fixed-point form. This result can be encoded as a base mesh through the static mesh encoding unit. The mesh preprocessing unit can generate additional vertices by performing mesh subdivision on the base mesh. Depending on the subdivision method, vertex connection information including the added vertices, texture coordinates, and texture coordinate connection information can be generated. The subdivided mesh can be fitted by adjusting vertex positions to resemble the original mesh, thereby generating a fitted subdivided mesh.

[0155] The base mesh generated through the mesh preprocessing unit can perform intra-encoding or inter-encoding depending on the base mesh type. If the base mesh frame undergoes intra-encoding, it can be compressed through the static mesh encoding unit. In this case, encoding can be performed on the base mesh's connectivity information, vertex geometry information, vertex texture information, normal information, etc. If the base mesh frame undergoes inter-encoding, the motion vector encoding unit is executed; using the base mesh and the reference-restored base mesh as inputs, the motion vector between the two meshes is calculated and its value encoded. The motion vector encoding unit performs connectivity-based prediction using previously encoded / decoded motion vectors as predictors and can encode the residual motion vector obtained by subtracting the predicted motion vector from the current motion vector. The base mesh bitstream generated through the base mesh encoding unit is transmitted to the multiplexer.

[0156] The encoded base mesh bitstream can generate a restored base mesh through the base mesh restoration unit.

[0157] The displacement vector calculation unit can perform mesh subdivision on the reconstructed base mesh. The displacement vector can be calculated as the difference in vertex positions between the subdivided reconstructed base mesh and the fitted subdivision mesh generated by the preprocessing unit. As a result, displacement vectors can be calculated for each vertex of the subdivision mesh. The displacement vector calculation unit can convert the displacement vector calculated in the 3D Cartesian coordinate system into a local coordinate system based on the normal vector of each vertex.

[0158] The displacement vector processing unit can transform the displacement vector for effective encoding. Depending on the embodiment, the transformation may be performed using a lifting transformation, a wavelet transformation, etc. Additionally, quantization can be performed on the transformed displacement vector values, i.e., the transformation coefficients. Different quantization parameters can be applied to each axis of the transformation coefficients, and the quantization parameters can be derived by the agreement between the encoder and decoder. The quantized displacement vector transformation coefficients calculated by the displacement vector processing unit can be encoded through the displacement vector video encoding unit or the displacement vector arithmetic encoding unit, depending on the compression method.

[0159] The displacement vector video encoding unit can pack displacement vector information that has undergone transformation and quantization into 2D images. A displacement vector video can be generated by bundling the packed 2D images for each frame, and the displacement vector video can be generated for each GoF (Group of Frame) unit of the input mesh. The generated displacement vector video can be encoded using a video compression codec. The generated displacement vector video bitstream is transmitted to the multiplexer.

[0160] In the displacement vector arithmetic encoding unit, for the quantized displacement vector transformation coefficients, if the displacement vector type is INTER type, inter-frame prediction can be performed. The inter-frame prediction process may be a process of calculating the residual value, which is the difference between the current transformation coefficient and the reference transformation coefficient. The displacement vector transformation coefficient or the residual value can be encoded through the arithmetic encoding process.

[0161] The displacement vector restored through the displacement vector restoration unit and the base mesh restored and subdivided through the base mesh restoration unit are restored through the mesh restoration unit, and the restored mesh contains restored vertices, connection information between vertices, texture coordinates, and connection information between texture coordinates.

[0162] The attribute information (texture map) of the original mesh can be regenerated into the attribute information (texture map) for the restored mesh through the attribute information (texture map) video generation unit. Vertex-specific color information contained in the original mesh's texture map can be assigned to the texture coordinates of the restored mesh. The texture maps regenerated for each frame can be grouped by GoF unit to generate a texture map video.

[0163] The generated texture map video can be encoded using a video compression codec through the texture map video encoding unit. The texture map video bitstream generated through encoding is transmitted to the multiplexer.

[0164] The atlas encoding unit can encode the atlas, which is additional information required for the mesh decoding and rendering processes. The generated atlas bitstream is transmitted to the multiplexer.

[0165] The generated base mesh bitstream, displacement vector bitstream, texture map bitstream, and atlas bitstream can be multiplexed into a single bitstream and transmitted to the receiver via the transmitter. Alternatively, the generated base mesh bitstream, displacement vector bitstream, texture map bitstream, and atlas bitstream can be generated into a file with one or more track data or encapsulated into segments and transmitted to the receiver (decoder) via the transmitter.

[0166] The data input unit can receive the original mesh and / or original texture map ('attribute'). The mesh preprocessing unit can simplify the original mesh to generate a base mesh and fit it to generate a refined mesh. If the mesh encoding method is inter-prediction, the motion vector encoding unit can generate motion vectors (motion fields) by referencing the restored base mesh within a previously processed reference frame and encode them based on a motion estimation and compensation method. If the mesh encoding method is intra-prediction, the static mesh encoding unit can encode the base mesh within the frame. The displacement vector calculation unit can calculate displacement vectors for vertices from the fitted refined mesh based on the restored base mesh. The displacement vector processing unit can process the displacement vectors into a form suitable for encoding. Depending on the encoding method for the displacement vectors, the displacement vectors can be encoded based on a video method or an arithmetic encoding method. The displacement vectors are restored and can be provided to the mesh restoration unit along with the restored base mesh. Based on the restored mesh, the attribute (texture map) video generation unit can generate a video for encoding the texture map using the original mesh and the texture map for the original mesh. The attributes are encoded based on the video method. The atlas is encoded by the atlas encoding unit.

[0167] FIG. 13 illustrates a V-DMC decoding process according to embodiments.

[0168] FIG. 13 corresponds to a decoder such as FIG. 1 to FIG. 3. Each component of FIG. 13 corresponds to hardware, software, a processor, and / or a combination thereof.

[0169] The received Mesh bitstream is demultiplexed into a compressed base mesh bitstream, displacement vector bitstream, attribute information (texture map) bitstream, and atlas bitstream after file / segment decapsulation.

[0170] If the current mesh has inter-frame encoding applied based on the frame header information, decoding can be performed on the base mesh bitstream in the motion vector decoder. The final motion vector can be restored by using the previously decoded motion vector as a predictor and adding it to the residual motion vector decoded from the bitstream. The current base mesh can be restored by adding the decoded motion vector to the reference base mesh.

[0171] If the current mesh has in-frame encoding applied based on the frame header information, the base mesh bitstream can restore the base mesh's connectivity information, vertex geometry, texture coordinates, normal information, etc., through the static mesh decoder.

[0172] In the base mesh restoration unit, a restored base mesh can be generated by performing inverse quantization on the decoded base mesh.

[0173] Depending on the encoding codec type, if the displacement vector bitstream is encoded through a video codec, it can be decoded using the video codec and then subjected to a reverse packing process. If encoded through arithmetic coding, arithmetic decoding can be performed through the displacement vector arithmetic decoding unit, and if inter-frame prediction is performed, the current displacement vector transformation coefficient can be generated by adding the residual value to the reference displacement vector transformation coefficient through inter-frame prediction.

[0174] The displacement vector restoration unit restores the displacement vector by applying the decoded displacement vector transformation coefficients to inverse quantization and inverse transformation processes. If the restored displacement vector is a value in the local coordinate system, an inverse transformation to the Cartesian coordinate system can be performed.

[0175] The mesh restoration unit can generate additional vertices by performing subdivision on the restored base mesh. Through subdivision, vertex connection information including the added vertices, texture coordinates, and connection information of the texture coordinates can be generated. The subdivided restored base mesh can be combined with the restored displacement vectors to generate the final restored mesh.

[0176] The texture map bitstream can be decoded as a video bitstream using a video codec in the texture map video decoder. The restored texture map contains color information for each vertex contained in the restored mesh, and the color value of the corresponding vertex can be retrieved from the texture map using the texture coordinates of each vertex.

[0177] The atlas bitstream can be decoded through the atlas decoding unit.

[0178] The restored mesh and texture map are displayed to the user through a rendering process using a mesh data renderer, etc.

[0179] The decoder receives the encoded bitstream and decodes the base mesh, displacement vector, attributes, and atlas within the bitstream based on the parameter information (which may be referred to as signaling information, metadata, etc.) contained within the bitstream. The decoding process may follow the inverse of the encoding process. Based on the decoded atlas, a mesh is reconstructed from the reconstructed base mesh and the reconstructed displacement mesh. Based on the reconstructed mesh and the reconstructed attributes, the mesh can be rendered.

[0180] A point cloud data encoding device and method according to the embodiments can encode mesh data and transmit a bitstream containing the encoded mesh data. A point cloud data decoding device and method according to the embodiments can receive a bitstream containing mesh data and decode the mesh data. A point cloud data encoding / decoding method / device according to the embodiments may be referred to simply as a method / device according to the embodiments. A point cloud data encoding / decoding method / device according to the embodiments may also be referred to as a mesh data encoding / decoding method / device according to the embodiments. Additionally, it may be used in this document simply as an encoding / decoding method / device.

[0181] Basemesh sub-bitstream

[0182] In the following, bitstream may refer to a basemesh sub-bitstream.

[0183] Overall basemesh codec characteristics, decoding operations, and post-decoding processes

[0184] FIG. 14 shows a base mesh decoder according to embodiments.

[0185] Below, a high-level description of the characteristics of the basemesh bitstream and the operations required for decoding is provided. The basemesh decoder includes two functional blocks represented by external codecs: a static mesh codec and a motion codec. The codecs of these functional blocks are determined first using a combination of a syntax element representing a basemesh profile or codec ID and an SEI message mapping the codec ID. FIG. 14 illustrates the elements of the basemesh decoder.

[0186] To provide additional functionality and flexibility for use in various video codec specifications, the base mesh can be divided into independent units called sub-meshes, which can be encoded and encapsulated into NAL units.

[0187] Bitstream format

[0188] In the following, the relationship between the NAL (network abstraction layer) unit stream and the NAL sample stream is specified and referred to as the NAL bitstream, respectively.

[0189] A bitstream can be one of two formats: the NAL unit stream format or the sample stream format. The NAL unit stream format is conceptually the more basic format and consists of a sequence of syntax elements called NAL units. This sequence is arranged in the order of decoding. In a NAL unit stream, constraints are imposed on the decoding order (and content) of the NAL units.

[0190] The NAL unit stream format is generally not used alone in any application because it requires additional information—namely, sub-bitstream size information—to decode the relevant sub-bitstream. One way to achieve this is to use the NAL sample stream format.

[0191] The NAL sample stream format can be constructed by arranging NAL units in the NAL unit stream format in decoding order and attaching a heading to each NAL unit specifying the exact size of the NAL unit in bytes. The NAL unit stream format can be extracted from the sample stream format by traversing the sample stream format, reading size information, and appropriately extracting each NAL unit.

[0192] The NAL sample stream format includes one or more coded basemesh sequences (CBMS) with a set of parameter sets identified by the four-character code 'bmsf'.

[0193] General basemesh sequence parameter set RBSP syntax

[0194] bmesh_sequence_parameter_set_rbsp( ) {Descriptorbmsps_sequence_parameter_set_idu(4)bmsps_max_sub_layers_minus1u(3)bmsps_temporal_id_nesting_flagu(1)bmesh_profile_tier_level( bmsps_max_sub_layers_minus1 ) bmsps_intra_mesh_codec_idu(8)bmsps_inter_mesh_codec_idu(8)bmsps_geometry_3d_bit_depth_minus1u(5)bmsps_geometry_msb_align_flagu(1)bmsps_mesh_attribute_countu(7)for( i = 0; i < bmsps_mesh_attribute_count; i++ ) { bmsps_mesh_attribute_index[ i ]u(7)bmsps_mesh_attribute_type_id[ i ]u(4)bmsps_mesh_attribute_dimension_minus1[ i ]u(6)bmsps_attribute_bit_depth_minus1[ i ]u(5)bmsps_attribute_msb_align_flag[ i ]u(1)} bmsps_log2_max_mesh_frame_order_cnt_lsb_minus4ue(v)bmsps_sub_layer_ordering_info_present_flagu(1)for( i = bmsps_sub_layer_ordering_info_present_flag?0:bmsps_max_sub_layers_minus1; i < bmsps_max_sub_layers_minus1 + 1;i++ ) { bmsps_max_dec_mesh_frame_buffering_minus1[ i ]ue(v)bmsps_max_num_reorder_frames[ i ]ue(v)bmsps_max_latency_increase_plus1[ i ]ue(v)} bmsps_long_term_ref_mesh_frames_flagu(1)bmsps_num_ref_mesh_frame_lists_in_bmspsue(v)for( i = 0; i < bmsps_num_ref_mesh_frame_lists_in_bmsps; i++ ) bmesh_ref_list_struct( i ) bmsps_inter_mesh_max_num_neighbours_minus1u(3)bmsps_codec_specific_parameters_present_flagu(1)if( bmsps_codec_specific_parameters_present_flag ) { bmsps_mesh_codec_prefix_length_minus1u(8)for( i = 0 ; i < bmsps_mesh_codec_prefix_length_minus1 + 1; i++ ) { bmsps_mesh_codec_prefix_data_byte[ i ]u(8)}} bmsps_motion_codec_specific_parameters_present_flagu(1)if( bmsps_motion_codec_specific_parameters_present_flag ) { bmsps_motion_codec_prefix_length_minus1u(8)for( i = 0 ; i < bmsps_motion_codec_prefix_length_minus1 + 1;i++ ) { bmsps_motion_codec_prefix_data_byte[ i ]u(8)}} bmsps_vui_parameters_present_flagu(1)if( bmsps_vui_parameters_present_flag ) bm_vui_parameters( ) bmsps_extension_present_flag(1)if( bmsps_extension_present_flag ) { bmsps_extension_countu(8)} if( bmsps_extension_count ){ bmsps_extensions_length_minus1ue(v)for( i = 0; i < bmsps_extension_count; i++ ) { bmsps_extension_type[ i ]u(8)bmsps_extension_length[ i ]u(16)bmsps_extension( bmsps_extension_type[ i ], bmsps_extension_length[ i ] )}} rbsp_trailing_bits( )} ;

[0195] General basemesh sequence parameter set RBSP semantics

[0196] The sequence parameter set ID (bmsps_sequence_parameter_set_id) provides an identifier for the basemesh sequence parameter set so that other syntax elements can reference it.

[0197] max_sub_layers_minus1 + 1 specifies the maximum number of temporal sublayers that may exist in each CBMS referencing a BMSPS. The value of bmsps_max_sub_layers_minus1 must be from 0 to 6 (inclusive).

[0198] The temporal ID nesting flag (bmsps_temporal_id_nesting_flag) specifies whether to further restrict inter-predictions for CBMSs referencing BMSPS when bmsps_max_sub_layers_minus1 is greater than 0. If bmsps_max_sub_layers_minus1 is 0, bmsps_temporal_id_nesting_flag must be equal to 1.

[0199] The syntax element temporal ID nesting flag (bmsps_temporal_id_nesting_flag) is used to indicate that temporal up-switching in CBMS is always possible, that is, from decoding up to a maximum basemesh temporal ID (BmTemporalId) tBmIdN to decoding up to a maximum BmTemporalId tBmIdM greater than tBmIdN.

[0200] The intra-mesh codec ID (bmsps_intra_mesh_codec_id) represents the mapping index of the codec identifier of the static mesh decoder used to decode the static basemesh subbitstream. The bmsps_intra_mesh_codec_id must be in the range from 0 to 255.

[0201] The intermesh codec ID (bmsps_inter_mesh_codec_id) represents the codec identifier mapping index of the decoder used for motion data decoding. bmsps_inter_mesh_codec_id must have a value from 0 to 255.

[0202] The basemesh attribute count (bmsps_mesh_attribute_count) represents the number of attributes associated with the basemesh. bmsps_mesh_attribute_count must be in the range of 0 to 127. The value represented by bmsps_mesh_attribute_count must be less than or equal to the number of attributes of the submesh contained in the bitstream.

[0203] For bitstream compatibility, all submeshes must have the same number of attributes.

[0204] Basemesh attribute index (bmsps_mesh_attribute_index[ i ]) represents the index of the basemesh attribute in the static mesh subbitstream associated with the i-th basemesh attribute. bmsps_mesh_attribute_index[ i ] must be in the range from 0 to 127.

[0205] For bitstream compatibility, all submeshes must have the same basemesh attribute index in the static mesh subbitstream associated with the i-th basemesh attribute.

[0206] The basemesh attribute type ID (bmsps_mesh_attribute_type_id[ i ]) represents the attribute type of the basemesh with index i.

[0207] Attribute bit depth minus 1 (bmsps_attribute_bit_depth_minus1[ i ]) plus 1 indicates the bit depth of the attribute with index i for the base mesh. bmsps_attribute_bit_depth_minus1[i] must be in the range of 0 to 31.

[0208] Basemesh profile, tier, and level syntax

[0209] bmesh_profile_tier_level( maxNumBmeshSubLayersMinus1 ) {Descriptorbmptl_tier_flagu(1)bmptl_profile_idcu(7)bmptl_reserved_zero_32bitsu(32)bmptl_level_idcu(8)for( i = 0; i < maxNumBmeshSubLayersMinus1+1 ; i++ ) {bmptl_sub_layer_profile_present_flag[ i ]u(1)bmptl_sub_layer_level_present_flag[ i ]u(1)if( bmptl_sub_layer_profile_present_flag[ i ] ) {bmptl_sub_layer_tier_flag[ i ]u(1)bmptl_sub_layer_profile_idc[ i ]u(7)}if( bmptl_sub_layer_level_present_flag[ i ] ) {bmptl_sub_layer_level_idc[ i ]u(8)}}bmptl_num_sub_profilesu(6)bmptl_extended_sub_profile_flagu(1)for( i = 0; i < bmptl_num_sub_profiles; i++ ) { bmptl_sub_profile_idc[ i ]u(v)} bmptl_toolset_constraints_present_flagu(1)if( bmptl_toolset_constraints_present_flag ) { bmesh_profile_toolset_constraints_information( )}}

[0210] 베이스메쉬 프로파일, 티어, 및 레벨 신택스(Basemesh profile, tier, and level semantics)

[0211] 티어 플래그(bmptl_tier_flag)는 레벨 지시자(bmptl_level_idc) 해석을 위한 티어 컨텍스트를 나타낸다.

[0212] The profile indicator (bmptl_profile_idc) represents the default mesh profile that CBMS adheres to.

[0213] If bmptl_reserved_zero_32bit exists, it must be equal to 0 in bitstreams compliant with the version of this document. Other values ​​for bmptl_reserved_zero_32bit are reserved for future use by ISO / IEC.

[0214] The level indicator (bmptl_level_idc) indicates the level compliant with the CBMS. The bitstream must not contain a bmptl_level_idc value other than the specified value. Other bmptl_level_idc values ​​are reserved by ISO / IEC for future use.

[0215] If the sublayer profile present flag (bmptl_sub_layer_profile_present_flag[i]) is 1, it indicates that profile information is present in the bmesh_profile_tier_level() syntax structure for the sublayer representation with BmTemporalId i. If bmptl_sub_layer_profile_present_flag[i] is 0, it indicates that profile information is not present in the bmesh_profile_tier_level() syntax structure for the sublayer representation with BmTemporalId i. When not present, bmtpl_sub_layer_profile_present_flag[i] must be equal to 0.

[0216] If the sublayer level present flag (bmptl_sub_layer_level_present_flag[i]) is 1, it specifies that level information is present in the bmesh_profile_tier_level() syntax structure for the sublayer representation with BmTemporalId i. If bmptl_sub_layer_level_present_flag[i]] is 0, it specifies that there is no level information in the bmesh_profile_tier_level() syntax structure for the sublayer representation with BmTemporalId i. If there is no information, bmptl_sub_layer_level_present_flag[i] is equal to 0.

[0217] The sublayer tier flag (bmptl_sub_layer_tier_flag[i]) specifies the layer context for interpreting the sublayer level indicator (bmptl_sub_layer_level_idc[i]). If absent, the value of bmptl_sub_layer_tier_flag[i] is inferred to be 0.

[0218] The sublayer profile specifier (bmptl_sub_layer_profile_idc[ i ]) specifies the basemesh profile for the sublayer representation with BmTemporalId i, and CBMSs adhere to this. Bitstreams must not contain bmptl_sub_layer_profile_idc[ i ] values ​​other than the specified values. Other values ​​for bmptl_sub_layer_profile_idc[ i ] are reserved by ISO / IEC for future use.

[0219] The sublayer level indicator (bmptl_sub_layer_level_idc[ i ]) indicates the level for the sublayer representation with BmTemporalId i, and CBMSs comply with this. Bitstreams must not contain bmptl_sub_layer_level_idc[ i ] values ​​other than the specified values. Other values ​​for bmptl_sub_layer_level_idc[ i ] are reserved for future use by ISO / IEC.

[0220] bmptl_sub_layer_tier_flag[ i ] may not exist, but bmptl_sub_layer_level_idc[ i ] may exist. In this case, the default value of bmptl_sub_layer_tier_flag[ i ] is required to interpret bmptl_sub_layer_level_idc[ i ].

[0221] If the toolset constraints present flag (bmptl_toolset_constraints_present_flag) is 1, it specifies that the additional syntax structure bmesh_profile_toolset_constraints_information() exists in the bitstream. If bmptl_toolset_constraints_present_flag is 0, it specifies that the syntax structure bmesh_profile_toolset_constraints_information() does not exist.

[0222] A encoding method / device according to embodiments (e.g., encoder in FIG. 1, transmitting device (100) in FIG. 2, acquisition unit (101), encoder (102), encapsulator (103), transmitter (104), pre-processing in FIG. 3, FIG. 4, FIG. 6, FIG. 7, encoder, multiplexer in FIG. 12, transmitting unit, bitstream and syntax generation in FIG. 15, FIG. 16, encoding method in FIG. 17, etc.) can encode mesh data, generate a bitstream, and encapsulate a file containing the bitstream.

[0223] A decoding method / device according to embodiments (a decoder in FIG. 1, a receiving device (110) in FIG. 2, a receiving unit (111), a decapsulator (112), a decoder (113), a renderer (114), a decoder in FIG. 11, a receiving unit in FIG. 13, a demultiplexer, a renderer, a base mesh decoder in FIG. 14, bitstream and syntax acquisition in FIG. 15 and 16, a decoding method in FIG. 18, etc.) receives a file containing a bitstream and can decode mesh data based on parameter information included in the bitstream.

[0224] The method and apparatus according to the embodiments may include and perform methods related to profiles, tiers, and levels for V-DMC basemesh coding.

[0225] The embodiments relate to V-DMC (Video-based Dynamic Mesh Compression), a method for compressing three-dimensional dynamic mesh data using an existing 2D video codec. The embodiments may include and perform profile and level configurations and signaling schemes for interoperability between a basemesh sub-bitstream, which is one of the components of a V-DMC bitstream, and a basemesh decoder for decoding it.

[0226] The embodiments relate to V-DMC (Video-based Dynamic Mesh Compression), a method for compressing three-dimensional dynamic mesh data using an existing 2D video codec, and define a profile and a level for interoperability between a basemesh sub-bitstream, which is one of the components of a V-DMC bitstream, and a basemesh decoder for decoding it, and define a syntax element for signaling it.

[0227] As of July 2024, V-DMC technology is in the CD stage, and active standardization is currently underway. In the current V-DMC standard, definitions for profiles, tiers, and levels for basemesh sub-bitstreams are required to ensure interoperability between basemesh sub-bitstreams and basemesh decoders.

[0228] The profile, tier, and level according to the embodiments define constraints on the encoded bitstream and limitations on the capability required to decode the bitstream. To ensure interoperability between the basemesh sub-bitstream and the basemesh decoder, the profile, tier, and level of the basemesh sub-bitstream must be defined.

[0229] The current V3C profile design includes two main components for video coding and V3C-specific tools. The former is signaled through codec groups, and the latter can be signaled through toolsets.

[0230] Since the BaseMesh sub-bitstream is a single component of the V3C bitstream, the concept of a toolset may be unnecessary, and only the CodecGroup component may be required for profiling the BaseMesh sub-bitstream.

[0231] Therefore, two-component profile design may no longer be efficient for basemesh sub-bitstreams, and one should consider using a simpler method to represent the profile of the basemesh sub-bitstream.

[0232] In addition, conformance regarding the reconstruction process of basemesh sub-bitstreams can be addressed in Appendix A of the V-DMC standard document (ISO / IEC 23090-29), and H.12 can address the conformance of the decoded basemesh sub-bitstream.

[0233] The embodiments relate to V-DMC (Video-based Dynamic Mesh Compression), a method for compressing three-dimensional dynamic mesh data using an existing 2D video codec. It defines profiles and levels for interoperability between a basemesh sub-bitstream, which is one of the components of a V-DMC bitstream, and a basemesh decoder for decoding it, and defines syntax elements for signaling them. Through these syntax elements, the basemesh decoder can determine in advance what level of decoding specification / function is required to receive and accurately decode the corresponding sub-bitstream, thereby providing means for interoperability (conformance or inter-operability) between a server providing the V-DMC bitstream and a receiver receiving, decoding, and processing the V-DMC bitstream. Furthermore, through the device / method according to the embodiments, the receiver can perform operations such as memory, buffer, and other resource allocation necessary to accurately decode the basemesh sub-bitstream.

[0234] The configuration, operation, and other features of the embodiments can be understood from the embodiments described with reference to the attached drawings.

[0235] V-DMC referred to in the embodiments may also be referred to as V-Mesh below and may be an expression used with the same meaning.

[0236] FIG. 15 shows a V-DMC bitstream according to embodiments.

[0237] An encoding method / device according to the embodiments can encode mesh data and generate parameter information regarding the mesh data (which may be referred to as signaling information, syntax elements, etc.) to generate a bitstream such as FIG. 15. A decoding method / device according to the embodiments can receive a bitstream such as FIG. 15 and decode the mesh data based on syntax elements within the bitstream.

[0238] The term V-DMC can also be referred to as V-Mesh and is used interchangeably. Dynamic mesh content can be encoded into a bitstream structure as shown in Fig. 15. Bitstreams can be generated based on the sample stream data unit used when encoding V3C content according to the V3C codec specification (ISO / IEC 23090-5).

[0239] The definitions of abbreviations referring to data contained within the bitstream are as follows: VPS: V3C / V-DMC Parameter Set, AD: Atlas Data, BMD: Base Mesh Data, DD: Displacement Data (Displacement data can be encoded using arithmetic coding), GVD: Geometry Video Data (Displacement data can be encoded based on a video codec), AVD: Attribute Video Data (Attribute data can be encoded based on video coding), PVD: Packing Video Data (Packing data can be encoded based on video coding), CAD: Common Atlas Data, OVD: Accupancy Video Data, ADD: Arithmetic Coded Displacement Data.

[0240] Referring to FIG. 15, the bitstream according to the embodiments may include a sample stream DMC header and a plurality of sample stream DMC units, and each of the plurality of sample stream DMC units may include VPS, AD, BMD, DD / GVD, AVD, and PVD, etc.

[0241] And the BMD of FIG. 15 may include information related to the profile, tier, and level of the basemesh sub-bitstream according to the embodiments.

[0242] Basemesh profiles, tiers, and levels

[0243] In the following, various factors are defined for a received basemesh sub-bitstream that allow a basemesh decoder to determine whether the bitstream is decodingable.

[0244] Overview of profiles, tiers, and levels

[0245] According to the embodiments, the profile, tier, and level represent constraints on the basemesh sub-bitstream and thus specify limitations on the capabilities required to decode the basemesh sub-bitstream. The profile, tier, and level can be used to represent interoperability points between individual basemesh decoder implementations.

[0246] According to the embodiments, each level of a basemesh tier represents a set of restrictions on the values ​​that a syntax element can have. The interpretation of the tiers and levels can be based on the values ​​of the bmptl profile toolset (bmptl_profile_toolset_idc). For a given basemesh profile, the levels of the tiers can generally correspond to specific basemesh decoder processing loads and memory capabilities.

[0247] According to the embodiments, the term “decoding” may mean only decoding a basemesh sub-bitstream and may not include reconstruction into 3D space. Additionally, each step included in the “decoding” process may be specified for each profile.

[0248] According to the embodiments, a phrase such as "bitstream" can be interpreted as a basemesh sub-bitstream of the operation point, and a phrase such as "coded basemesh access unit n" can be interpreted as "coded basemesh access unit n in the bitstream of the operation point." Here, "operation point" is an operation point associated with a profile, tier, or level.

[0249] According to the embodiments, BmHtidOP and BmTemporalID may represent the highest temporal layer existing in the basemesh sub-bitstream and the temporal ID of the basemesh sub-bitstream, and may each be equal to 0.

[0250] According to the embodiments, for each action point identified by BmHtidOP, profile, tier, and level information can be represented as a combination of syntax elements including bmptl_profile_codec_group_idc and bmptl_profile_toolset_idc, which are found in or derived from the bmesh_profile_tier_level() syntax structure of the basemesh SPS and, where possible, can be determined through external means.

[0251] Profile, tier, and level structure

[0252] The basemesh profile according to the embodiments can define conformance points that cover the decoded basemesh sub-bitstream and do not cover the reconstruction process.

[0253] Basemesh profile decoding performance can be defined by the syntax element bmptl_profile_codec_group_idc, which exists in the basemesh SPS or is obtained through external means if the basemesh SPS is unavailable.

[0254] According to the embodiments, a basemesh decoder that conforms to a basemesh profile at conformance point A (identified by the bmptl_profile_codec_group_idc syntax element) at a specific level (identified by the specific value of the bmptl_level_idc syntax element) of a specific tier (identified by the specific value of the bmptl_tier_flag syntax element) can decode all basemesh sub-bitstreams that satisfy all of the following conditions:

[0255] - Basemesh sub-bitstreams are instructed to adhere to supported basemesh profiles as specified in the Codec Group.

[0256] - BabeMesh sub-bitstreams are instructed to adhere to a basemesh level that is lower than or equal to the specified level.

[0257] - BabeMesh sub-bitstreams are instructed to adhere to a tier lower than or equal to the specified tier.

[0258] Profiles

[0259] General

[0260] Basically, a basic signaling method can be defined to determine whether the basemesh sub-bitstream included in the V-DMC bitstream is decodingable (i.e., whether the bitstream and the receiver are mutually compatible).

[0261] According to the embodiments, the bmptl profile codec group indicator (bmptl_profile_codec_group_idc) is a syntax element transmitted within a basemesh sequence parameter set, and if this value indicates 0, it means that it is a basemesh sub-bitstream based on the V-DMC standard (more specifically, ISO / IEC 23090-29 edition 1), and if a V-DMC decoder is designed based on this V-DMC standard, decoding of the sub-bitstream should be possible.

[0262] In this case, the corresponding basemesh sub-bitstream can be said to adhere to the basemesh main profile.

[0263] Alternatively, according to the embodiments, a component codec mapping SEI message capable of signaling that a basemesh sub-bitstream has been coded based on the V-DMC standard may be utilized, in which case a value of 'bmsh' may be used as a 4cc code value within the SEI message.

[0264] Alternatively, according to the embodiments, a Basemesh Main Still profile may be defined using a value of 'sbmc' with a bmptl_profile_codec_group_idc of 1 or a 4cc code value of a component codec mapping SEI message. In this case, basemesh coding supports only still images, and inter-basemesh frame coding is not used in this stream. Not using inter-basemesh frame coding means not using a motion codec.

[0265] Table 1 provides the list of defined profiles for the basemesh and the allowed values ​​for the syntax element bmptl profile codec group idc, respectively.

[0266] Basemesh Profilebmptl_profile_codec_group_idc4CC codeBasemesh Main0'bmsh'Reserved1..126-MP4RA127provided by external means or may be determined by component codec mapping SEI message (F.2.7) in ISO / IEC 23090-5

[0267] Basemesh Main profile

[0268] According to the embodiments, if the basemesh sub-bitstream complies with the basemesh main profile, it must comply with the constraints specified below. That is, the basemesh sub-bitstream must comply with the constraint that it must not have specifications that deviate from those specified below. According to the embodiments, it cannot have specifications such as bmsps_max_sub_layers_minus1 having a value of 1 or greater.

[0269] Basemesh sub-bitstreams that comply with the basemesh main profile must comply with the following constraints:

[0270] - Active BMSPS shall have bmsps_max_sub_layers_minus1 equal to 0 only.

[0271] - The tier and level constraints specified for the Basemesh Main profile shall be fulfilled.

[0272] Additional constraints apply here.

[0273] Whether a basemesh sub-bitstream corresponds to the basemesh main profile is indicated when a component codec mapping SEI message is present, such as bmptl_profile_codec_group_idc being 0 or ccm_codec_4cc being 'bmsh'.

[0274] Tiers and levels

[0275] According to the embodiments, the tier and level values ​​may define steps for the specifications of a basemesh frame or basemesh sequence that can be processed by a basemesh sub-bitstream and a basemesh decoder that conforms to the basemesh main profile. Generally, a receiver capable of decoding a specific level value can decode a bitstream having specifications lower than that level, including that level value.

[0276] According to the embodiments, the following and Table 4 may indicate minimum specifications for decoding bitstreams according to each level value. The level of the receiver / decoder is determined according to its respective performance (e.g., memory, buffer, processing speed, etc.), and the bitstream that can be decoded can be determined using these levels. Indicators for determining such performance may be based on the maximum value of vertices included in a single basemesh frame, the maximum size of the input buffer for storing compressed basemesh sub-bitstreams, the bitrate of the basemesh sub-bitstream that can be processed, and the maximum value of the number of maximum submeshes and vertices that can be processed per unit time (e.g., 1 second).

[0277] In other words, the receiver / decoder must guarantee basic performance to satisfy the size of the input buffer required at each level and the number of mesh vertices that can be processed per unit time in order to decode a specific bitstream, and the numbers expressed in Table 4 are examples and can be replaced with other values.

[0278] General tier and level limits

[0279] According to the embodiments, to compare tier performance, a tier with ptl_tier_flag 0 can be considered a lower tier than a tier with ptl_tier_flag 1.

[0280] According to the embodiments, to compare level performance, a specific level of a specific tier may be considered a lower level when the ptl_level_idc value of that level is smaller than the value of another level of the same tier.

[0281] To express constraints, the following is specified:

[0282] - A coded basemesh access unit n is called the nth basemesh access unit in decoding order, and the first basemesh access unit can be called access unit 0 (i.e., the 0th basemesh access unit).

[0283] Basemesh n can be said to be the coded basemesh of basemesh access unit n or the corresponding decoded basemesh.

[0284] According to the embodiments, a basemesh sub-bitstream of a specific tier and level that complies with all profiles may comply with the following constraints for each basemesh sub-bitstream conformity test specified below:

[0285] 1. The base mesh frame size (BmFrameSize) is less than or equal to the maximum base mesh frame size (MaxBmFrameSize). Here, MaxBmFrameSize is specified in Table 4 and BmFrameSize can be equal to the syntax element bmfps_num_vertices of BMFPS.

[0286] 2. Geometry 3D bit depth minus 1 (bmsps_geometry_3d_bit_depth_minus1) + 1 must be less than or equal to Max Geometry Bit Depth (MaxGeoBitDepth), where MaxGeoBitDepth is specified in Table 4.

[0287] 3. The number of base mesh attributes (bmsps_mesh_attribute_count) must be less than or equal to the maximum number of base mesh attributes (MaxBmAttrCount), and MaxMbAttrCount is specified in Table 4.

[0288] 4. Attribute bit depth minus 1 (bmsps_attribute_bit_depth_minus1) + 1 must be less than or equal to the maximum attribute bit depth (MaxAttrBitDepth), where MaxAttrBitDepth is specified in Table 4.

[0289] 5. For the BMCL HRD parameter, for at least one value of i in the range from 0 to cbmb_cnt_minus1[HTid]), the CBMB size (CbmbSize[i]) is less than or equal to CbmbBmclFactor * MaxCBMB. Here, CbmbSize[i] is specified based on the selected parameters as specified below, CbmbBmclFactor is specified in Table 4, and MaxCBMB is specified in Table 4 in bits of CbmbBmclFactor.

[0290] 6. For the BNAL HRD parameter, the CBMB size (CbmbSize[i]) is less than or equal to CbmbNalFactor * MaxCBMB for at least one value of i in the range from 0 to cbmb_cnt_minus1[HTid]. Here, CbmbSize[i] is specified below based on the selected parameters as specified below, CbmbNalFactor is specified in Table 4, and MaxCBMB is specified in Table 4 in CbmbNalFactor bit units.

[0291] 7. For each coded picture, the value of BinCountsInNalUnits must be less than or equal to (32÷3) * NumBytesInVclNalUnits + ( RawMinCuBits * PicSizeInMinCbsY ) ÷ 32.

[0292] Profile specific level limits for the basemesh dynamic profiles

[0293] Levels related to the base mesh can be configured as follows.

[0294] According to the embodiments, the tier and level observed by the basemesh sub-bitstream can be indicated by the syntax elements bmptl_tier_flag and bmptl_level_idc.

[0295] According to the embodiments, bmptl_level_idc can be set to a value 30 times the level number specified in Table 4.

[0296] LevelMax # vertices per basemesh frameMaxBmFrameSizeMax CBMB sizeMaxCBMBSizein 1 000 000 bitsMax basemesh bit rateMaxBmBRin 1 000 000 bits / sMax # submeshes per basemesh frameMaxNumSubMeshesMax submeshes rateMaxSubMeshesRate(submeshes / s)Max basemesh vertices rateMaxBmVertsRate(1 000 vertices / s)1.016 000101033601 9201.532 000202067203 8402.064 0004040121 4407 6802.5128 0008080242 88015 3603.0256 000160160485 76030 7203.5512 0003203209611 52061 4404.01 024 00064064019223 040122 8804.52 048 0001 2801 28038446 080245 760

[0297] Table 4 may specify general basemesh and BMSPS-related limits for each level of each tier.

[0298] According to the embodiments, the following variables may be specified:

[0299] - The variable BmFrameVertCount can be set to be equal to bmfps_num_vertices.

[0300] - The variable NumBmeshSubMeshes can represent the number of sub-meshes in the base mesh frame.

[0301] According to the embodiments, the variable t0 may represent, in milliseconds, a time instance in which the first bit of the coded atlas access unit 0 begins to enter the CAB.

[0302] If t1 is greater than or equal to t0, the variable BmVertsPerSec is defined and is set to be equal to the total number of BmFrameVertCount of basemesh frames output for all values ​​t1 <= t < t1 + 1000.

[0303] If t1 is greater than or equal to t0, the variable SubMeshesPerSec is defined and is set to equal the total number of NumBmeshSubMeshes of the basemesh frame output for all values ​​where t1 <= t < t1 + 1000.

[0304] Basemesh sub-bitstreams that comply with the basemesh dynamic profile at specific tiers and levels comply with the following constraints for each basemesh sub-bitstream conformance test.

[0305] - NumBmeshSubMeshs is less than or equal to MaxNumSubMeshs

[0306] - SubMeshesPerSec is less than or equal to MaxSubMeshesRate

[0307] - BmVertsPerSec is less than or equal to MaxBmVertsRate

[0308] MaxBmFrame, MaxNumSubMeshes, MaxSubMeshesRate and MaxBmVertsRate are specified in Table 4.

[0309] The following variables are specified:

[0310] - The variable fR can be set to 1÷1000.

[0311] - The variable MaxSubmeshSize represents the maximum total number of bytes expected to be associated with all decoded elements of a submesh.

[0312] - The variable aR represents BmFrameVertCount÷MaxBmVertsRate.

[0313] - A coded basemesh access unit n is called the nth coded basemesh access unit in decoding order, and the first basemesh access unit can be called access unit 0 (i.e., the 0th basemesh access unit).

[0314] - Basemesh n is said to be a coded basemesh or a decoded basemesh frame of the corresponding coded basemesh access unit n.

[0315] - The variable MaxNumSubLayersMinus1 can be set to 0.

[0316] The following restrictions apply:

[0317] 1. For BMCL HRD parameters, for at least one i value within the range from 0 to hrd_cbmb_cnt_minus1[Htid], CbmbSize[i] is less than or equal to MaxCBMBSize. Here, CbmbSize[i] is specified based on the selected parameter, and MaxCBMBSize can be specified in bits in Table 4.

[0318] 2. For BMNAL HRD parameters, for at least one i value within the range from 0 to hrd_cbmb_cnt_minus1[Htid], CbmbSize[i] is less than or equal to NalBmclRatio * MaxCBMBSize. Here, CbmbSize[i] is specified based on the selected parameter, MaxCBMBSize is specified in bits from Table 4, and NalBmclRatio can be set to 1.1.

[0319] 3. The nominal removal time of a coded basemesh access unit n (where n is greater than 0) in the CBMB can satisfy the constraint that AuNominalRemovalTime[n] - AuCbmbRemovalTime[n - 1] is greater than or equal to Max(aR, fR).

[0320] 4. The difference between consecutive output times of basemesh in DBMB can satisfy the constraint that DbmbOutputInterval[n] is greater than or equal to Max(aR, fR) if basemesh frame n is not the last basemesh of the basemesh sub-bitstream being output.

[0321] 5. For BMCL HRD parameters, for at least one value of i in the range from 0 to hrd_cbmb_cnt_minus1[Htid], BitRate[i] is less than or equal to MaxBmBR. Here, BitRate[i] is specified based on the selected parameter and MaxBmBR can be specified in Table 4.

[0322] 6. For the BMNAL HRD parameter, for at least one value of i in the range from 0 to hrd_cbmb_cnt_minus1[Htid], BitRate[i] is less than or equal to NalBmclRatio * MaxBmBR. Here, BitRate[i] is specified based on the selected parameter, MaxBmBR is specified in Table 4, and NalBmClRatio can be set to 1.1.

[0323] 7. For coded basemesh access unit 0, define the variables maxSubmeshBytes and nB as follows:

[0324] maxSubmeshBytes = MaxSubMeshesRate * MaxSubmeshSize

[0325] nB = Max( MaxNumSubMeshes * MaxSubmeshSize, fR * MaxSubmeshBytes ) +

[0326] maxSubmeshBytes * ( AuCbmbRemovalTime

[0000] - AuNominalRemovalTime

[0000] )

[0327] The values ​​of MaxNumSubMeshes and MaxSubMeshesRate are specified in Table 4.

[0328] The sum of the NumBytesInNalUnit variables for the coded basemesh access unit 0 is less than or equal to nB.

[0329] 8. If the coded basemesh access unit n is greater than 0, the variables maxSubmeshBytes and nB are defined as follows:

[0330] maxSubmeshBytes = MaxSubMeshesRate * MaxSubmeshSize

[0331] nB = maxSubmeshBytes * ( AuCbmbRemovalTime[ n ] - AuCbmbRemovalTime[ n - 1 ] )

[0332] Here, MaxSubMeshesRate is specified in Table 4.

[0333] The sum of the NumBytesInNalUnit variable for basemesh access unit n is less than or equal to nB.

[0334] The Basemesh frame parameter set (BMFPS) represents the characteristics of each basemesh frame and can define a new syntax element called bmfps_num_vertices, which specifically represents the number of mesh vertices included in each basemesh frame.

[0335] Basemesh Frame Parameter Set RBSP Syntax

[0336] General Basemesh Frame Parameter Set RBSP Syntax

[0337] FIG. 16 shows the base mesh frame parameter set RBSP syntax according to the embodiments.

[0338] Basemesh Frame Parameter Set RBSP Semantics

[0339] General Basemesh Frame Parameter Set RBSP Semantics

[0340] The BMFPS sequence parameter set ID (bmfps_sequence_parameter_set_id) represents the value of bmsps_sequence_parameter_set_id for the active basemesh sequence parameter set.

[0341] The BMFPS parameter set ID (bmfps_parameter_set_id) identifies the basemesh frame parameter set so that it can be referenced by other syntax elements.

[0342] BMFPS vertex count (bmfps_num_vertices) can represent the number of vertices in a basemesh frame.

[0343] If the BMFPS output flag present flag (bmfps_output_flag_present_flag) is equal to 1, it indicates that the bmsh_output_flag syntax element exists in the associated submesh header. If bfps_output_flag_present_flag is equal to 0, it indicates that the bmsh_output_flag syntax element does not exist in the associated submesh header.

[0344] Receiver Operation Flow

[0345] The receiver determines the supported V-DMC basemesh profile / level based on its respective specifications (computational capability, memory, storage space, etc.). In this case, the indicators serving as the basis for the judgment may be based on defined content.

[0346] The intex elements included in the BMSPS and BMFPS contained in the basemesh sub-bitstream of the received V-DMC bitstream are parsed. Through this, the profile and level of the basemesh sub-bitstream can be determined.

[0347] It is possible to determine whether a basemesh sub-bitstream can be decoded by utilizing the profile / level / tier information of the received bitstream.

[0348] If decoding is possible, resource allocation capable of decoding the corresponding bitstream can be performed by executing a defined series of operations.

[0349] According to the embodiments, the basemesh sequence parameter set may include syntax elements bmsps_max_sub_layers_minus1, bmsps_mesh_attribute_count, bmsps_attribute_bit_depth_minus1, etc.

[0350] And the basemesh sequence parameter set may include bmesh_profile_tier_level(). bmesh_profile_tier_level may include text elements bmptl_profile_codec_group_idc, bmptl_profile_idc, bmptl_tier_flag, bmptl_level_idc, etc.

[0351] FIG. 17 illustrates different encoding methods in the embodiments.

[0352] The encoding method of Fig. 17 can follow the inverse process of the decoding method of Fig. 18.

[0353] The method according to the embodiments may include the step of encoding a base mesh of mesh data (S1700); the step of encoding a displacement of mesh data (S1710); and / or the step of encoding attributes of mesh data (S1720); etc.

[0354] For the steps of encoding the base mesh of the mesh data within the bitstream (S1700), encoding the displacement of the mesh data (S1710), and encoding the attributes of the mesh data (S1720), one may refer to the description of the encoding method / device according to the embodiments (Fig. 1 encoder, Fig. 2 transmitting device (100), acquisition unit (101), encoder (102), encapsulator (103), transmitter (104), pre-processing in Fig. 3, Fig. 4, Fig. 6, Fig. 7 encoder, multiplexer, transmitting unit in Fig. 12 encoder, multiplexer, transmitting unit, bitstream and syntax generation in Fig. 15, Fig. 16 bitstream and syntax generation in Fig. 17 encoding method, etc.).

[0355] The bitstream may include a basemesh sub-bitstream. And the step of encoding the basemesh (S1700) may include the step of encoding the basemesh within the basemesh sub-bitstream based on profile information, tier information, and level information.

[0356] An encoder according to the embodiments may generate information regarding the profile, tier, and level of a basemesh sub-bitstream, include it in the basemesh sub-bitstream within the bitstream, and transmit it to a decoder. By generating information regarding the profile, tier, and level of a basemesh sub-bitstream and transmitting it to a decoder, the encoder according to the embodiments may enable the decoder to determine whether the basemesh sub-bitstream can be decoded.

[0357] Information regarding the profile, tier, and level according to the embodiments may be syntax elements for each.

[0358] Profile information according to the embodiments may include a profile codec group indicator (bmptl_profile_codec_group_idc) or a profile indicator (bmptl_profile_idc). The profile information may indicate that the basemesh sub-bitstream complies with at least one of the basemesh main profile, the basemesh main still profile, or the basemesh MP4RA profile.

[0359] The tier information according to the embodiments may be a tier flag (bmptl_tier_flag). The tier information may indicate a specific tier that the basemesh sub-bitstream adheres to. The tier for the first value of the tier flag may indicate that it is lower than the tier for the second value of the tier flag.

[0360] Level information according to the embodiments may be a level indicator (bmptl_level_idc). The level information may indicate a specific level observed by the basemesh sub-bitstream. The level indicator may indicate a level value regarding a tier flag.

[0361] Based on profile information according to the embodiments, the basemesh sub-bitstream may follow constraints on the profile. The constraints on the basemesh profile may include at least one of the constraints that the basemesh sub-bitstream includes a maximum number of sublayers minus 1 (bmsps_max_sub_layers_minus1) with a value of 0, or the constraints that the tier and level constraints on the basemesh profile are satisfied.

[0362] Constraints for a basemesh main profile according to the embodiments may include at least one of the following: a constraint in which the activated basemesh sequence parameter set (BMSPS) includes a maximum number of sublayers minus 1 (bmsps_max_sub_layers_minus1) with a value of 0; or a constraint in which constraints for tiers and levels for the basemesh main profile are satisfied.

[0363] The activated basemesh sequence parameter set (BMSPS) according to the embodiments may be generated by the encoding method, etc., according to the embodiments and included in the bitstream. For the activated basemesh sequence parameter set (BMSPS), refer to the description of the bmesh_sequence_parameter_set_rbsp syntax described above.

[0364] The temporal sublayer according to the embodiments may mean a temporally scalable layer of a temporally scalable basemesh sub-bitstream and may be composed of a BMCL (Basemesh Coding Layer) NAL (Network Abstraction Layer) unit having a BmTemporalId variable of a specific value and a non-BMCL NAL unit associated therewith.

[0365] The temporal sub-layer according to the embodiments refers to an access unit within the coded base mesh. Here, the maximum sub-layer may refer to the maximum value of the temporal sub-layer related to the access unit for the base mesh.

[0366] If the maximum number of sublayers according to the embodiments is constrained to 0, the decoder according to the embodiments has a constraint of decoding the base mesh based on one access unit rather than multiple sublayers.

[0367] The basemesh sub-bitstream according to the embodiments can comply with constraints on the tier and level based on the tier information and the level information.

[0368] The tier information and level information according to the embodiments may represent the minimum specifications of a decoder for decoding a basemesh sub-bitstream.

[0369] Constraints on tiers and levels according to the embodiments include: a constraint that the number of basemesh-related attributes (bmsps_mesh_attribute_count) is less than or equal to the maximum number of basemesh attributes (MaxBmAttrCount); a constraint that the bit depth of basemesh-related attributes minus 1 (bmsps_attribute_bit_depth_minus1) plus 1 is less than or equal to the maximum attribute bit depth (MaxAttrBitDepth); for BMCL (Basemesh Coding Layer) HRD (Hypothetical Reference Decoder) parameters, a constraint that the coded basemesh buffer (CBMB) size (CbmbSize) is less than or equal to the value related to the maximum CBMB size; or for BMNAL (Basemesh Network Abstraction Layer) HRD parameters, a constraint that the CBMB size (CbmbSize) is less than or equal to the product of the maximum CBMB size and the NAL-related factor; It may include at least one of the following.

[0370] At this time, the number of attributes associated with the base mesh (bmsps_mesh_attribute_count) and the bit depth minus 1 of attributes associated with the base mesh (bmsps_attribute_bit_depth_minus1) correspond to syntax elements included in the BMSPS and may be signaling information.

[0371] At least one of the number of maximum basemesh attributes, maximum attribute bit depth, or maximum CBMB size according to the embodiments may be a value derived based on a level indicator.

[0372] The maximum number of base mesh attributes (MaxBmAttrCount), maximum attribute bit depth (MaxAttrBitDepth), and maximum CBMB size according to the embodiments may be specified in Table 4.

[0373] Table 4 according to the embodiments may represent variables derived based on level information or tier information. Table 4 may include values ​​for the maximum number of vertices per basemesh frame, maximum CBMB size, maximum number of basemesh frames that can be processed per unit time (MaxBmBR), maximum number of submesh frames that can be processed per unit time (MaxSubMeshesRate), and maximum number of basemesh vertices that can be processed per unit time (MaxBmVertsRate), based on the level information or tier information. Furthermore, Table 4 may include various variables derived based on level information or tier information, but is not limited thereto. For example, Table 4 may further include the maximum number of basemesh attributes (MaxBmAttrCount) and the maximum attribute bit depth (MaxAttrBitDepth) derived based on the level information or tier information.

[0374] The encoder according to the embodiments can generate a tier flag and / or a level indicator and encode a basemesh sub-bitstream that complies with constraints based on variables derived based on the level indicator or the tier flag as shown in Table 4.

[0375] A submesh according to the embodiments refers to an independently decodingable region of a base mesh. A base mesh frame according to the embodiments includes a base mesh, and the maximum number of vertices of the base mesh included in the base mesh frame can be derived according to the level constraints according to the embodiments. The maximum number of submesh(s) of the base mesh of the base mesh frame according to the embodiments can be derived according to the level constraints according to the embodiments.

[0376] The maximum CBMB size according to the embodiments can be set according to the level information, such as 10 when the level is 1, 20 when the level is 2, and 40 when the level is 3, as indicated as Max CBMB size in Table 4.

[0377] According to the embodiments, two types of HRD parameter sets (BMNAL HRD parameters and BMCL HRD parameters) may be used. The HRD parameter sets may be passed through the bm_hrd_parameters() syntax element.

[0378] The CBMB size (CbmbSize) according to the embodiments may be set by syntax elements according to the embodiments. For example, the CBMB size may be set by the SubBmHrdFlag, CBMB size value minus 1 (cbmb_size_value_minus1), CBMB size scale (cbmb_size_scale) syntax elements, etc.

[0379] The encoder according to the embodiments can encode a basemesh coding layer based on a Hypothetical Reference Decoder (HRD). The HRD according to the embodiments refers to a virtual reference decoder and can support seamless decoding operations by inputting, storing, and outputting data to be input / decoded / output into a buffer based on a buffer model and according to a bit rate. The HRD parameters according to the embodiments may include parameters regarding buffer size, timing, bit rate, etc., to support buffer operations for decoding. The buffer for HRD regarding the basemesh according to the embodiments may include a Coded Basemesh Buffer (CBMB) that stores the coded basemesh and / or a Decoded Basemesh Buffer (DBMB) that stores the decoded basemesh.

[0380] A basemesh sub-bitstream according to the embodiments complies with constraints on a dynamic profile based on tier information and level information, and the constraints on the dynamic profile may include at least one of: a constraint that, for a Basemesh Coding Layer (BMCL) Hypothetical Reference Decoder (HRD) parameter, the bit rate (BitRate) is less than or equal to the maximum basemesh bit rate (MaxBmBR); or, for a Basemesh Network Abstraction Layer (BMAL) Hypothetical Reference Decoder (HRD) parameter, the bit rate (BitRate) is less than or equal to the product of the NAL / BMCL ratio (NalBmclRatio) and the maximum basemesh bit rate (MaxBmBR).

[0381] The bit rate according to the embodiments may be set by syntax elements according to the embodiments. For example, the bit rate may be set by the bit rate value minus 1 (bm_bit_rate_value_minus1[ i ]), bit rate scale (bm_bit_rate_scale) syntax elements, etc.

[0382] The maximum basemesh bit rate according to the embodiments may refer to the maximum number of basemesh bits that can be processed per hour and may be set based on level information and / or tier information. Referring to Table 4, the maximum basemesh bit rate (MaxBmBR) may have a value in units of 1,000,000 bits / s. For example, it may be set to a value of 10 when the level is 1, 20 when the level is 2, etc.

[0383] The NAL / BMCL ratio (NalBmclRatio) according to the embodiments can be set to a specific value, for example, NalBmclRatio can be set to 1.1.

[0384] Constraints for the dynamic profile according to the embodiments may further include a constraint that the nominal removal time of basemesh access unit n (where n is greater than 0) coded in the CBMB is AuNominalRemovalTime[n] - AuCbmbRemovalTime[n - 1], which is greater than or equal to Max(aR, fR). Here, the access unit nominal removal time (AuNominalRemovalTime[n]) represents the nominal removal time of access unit n, and the access unit CBMB removal time (n-1) represents the CBMB removal time of access unit n-1. The variable aR may be set to the basemesh frame vertex count (BmFrameVertCount) / basemesh vertex rate (MaxBmVertsRate). The basemesh vertex rate may be a value derived from Table 4. And fR may be set to a specific value, for example, 1÷1000.

[0385] Constraints for dynamic profiles according to embodiments may further include a constraint that DbmbOutputInterval[n] is greater than or equal to Max(aR, fR) when the difference between consecutive output times of basemesh in DBMB is not the last basemesh of the basemesh sub-bitstream to which basemesh frame n is output. In this case, the decoded basemesh output interval (DbmbOutputInterval) may be the difference between the output time of the first basemesh frame and the output time of the basemesh frame where BasemeshFrameOutputFlag is 1.

[0386] Constraints on the dynamic profile according to the embodiments may include, for the BMCL HRD parameter, a constraint that for at least one value of i within the range of 0 to hrd_cbmb_cnt_minus1[Htid], BitRate[i] is less than or equal to MaxBmBR. Here, BitRate[i] is specified based on the selected parameter, and MaxBmBR can be specified in Table 4. The value of the number of coded basemesh buffers (bm_hrd_cbmb_cnt_minus1[i]) plus 1 specifies the number of alternate CBMB specifications in the CBMS bitstream when HighestTid is equal to i. The value of the number of coded basemesh buffers (cbmb_cnt_minus1[i]) must be in the range of 0 to 31. If this value is missing, the value of cbmb_cnt_minus1[i] is assumed to be 0.

[0387] Constraints on the dynamic profile according to the embodiments may include, for the BMNAL HRD parameter, a constraint that for at least one i value within the range from 0 to hrd_cbmb_cnt_minus1[Htid], BitRate[i] is less than or equal to NalBmclRatio * MaxBmBR. Here, BitRate[i] is specified based on the selected parameter, MaxBmBR is specified in Table 4, and NalBmClRatio can be set to 1.1.

[0388] FIG. 18 illustrates a decoding method according to embodiments.

[0389] The decoding method of Fig. 18 can follow the inverse process of the encoding method of Fig. 17.

[0390] The decoding method according to the embodiments may include the step of decoding a base mesh in a bitstream (S1800); the step of decoding a displacement in a bitstream (S1810); and / or the step of decoding an attribute in a bitstream (S1820); etc.

[0391] For the steps of decoding the base mesh (S1800), decoding displacement within the bitstream (S1810), and decoding attributes (S1820), one may refer to the description of the decoding method / device according to the embodiments (Fig. 1 decoder, Fig. 2 receiving device (110), receiving unit (111), decapsulator (112), decoder (113), renderer (114), Fig. 11 decoder, Fig. 13 receiving unit, demultiplexer, renderer, Fig. 14 base mesh decoder, Fig. 15, Fig. 16 bitstream and syntax acquisition, Fig. 18 decoding method, etc.).

[0392] A decoder according to the embodiments can decode mesh data within a bitstream based on information regarding a profile, tier, and level according to the embodiments. For example, mesh data within a bitstream can be decoded by comparing the tier capability and / or level capability of the decoder.

[0393] In the decoder according to the embodiments, the decoding level may be determined based on the performance of the decoder (e.g., memory, buffer, operation speed, etc.). Additionally, the decoder may determine whether a received bitstream is a decoding bitstream based on profile information, tier information, and level information included in the bitstream. For example, if the decoding level of the decoder is higher than the level of the received bitstream, the decoder may determine that the received bitstream is a decoding bitstream. Furthermore, if the decoding level of the decoder is lower than the level of the received bitstream, the decoder may determine that the received bitstream is an undecoding bitstream.

[0394] The decoder according to the embodiments can increase decoding efficiency by determining in advance whether a bitstream is decodingable based on information regarding profile, tier, and level.

[0395] The encoding method and decoding method according to the embodiments can derive a level for decoding a base mesh by considering the capacity and / or performance of the decoder. If the performance of the decoder and / or the network environment is poor, the number of vertex attributes of the base mesh, the depth value of the vertex of the base mesh, and the size of the buffer for HRD-based decoding of the base mesh can be constrained to be small based on a low level. Conversely, if the performance of the decoder and / or the network environment is good, the number of vertex attributes of the base mesh, the depth value of the vertex of the base mesh, and the size of the buffer for HRD-based decoding of the base mesh can be constrained to be large based on a high level.

[0396] A device for decoding encoded mesh data according to the embodiments can decode mesh data based on the values ​​of variables (e.g., column elements of the aforementioned Table 4, etc.) that are constrained to a specific level of a specific tier in a specific profile, taking into account capabilities related to its profile, tier, level, etc.

[0397] Information regarding the profile, tier, and level according to the embodiments may be syntax elements for each.

[0398] The step of decoding a base mesh according to the embodiments (S1800) may include decoding a base mesh in a bitstream and a base mesh in a sub-bitstream based on profile information, tier information, and level information.

[0399] Profile information according to the embodiments may include a profile codec group indicator (bmptl_profile_codec_group_idc) or a profile indicator (bmptl_profile_idc). The profile information may indicate that the basemesh sub-bitstream complies with at least one of the basemesh main profile, the basemesh main still profile, or the basemesh MP4RA profile.

[0400] The tier information according to the embodiments may be a tier flag (bmptl_tier_flag). The tier information may indicate a specific tier that the basemesh sub-bitstream adheres to. The tier for the first value of the tier flag may indicate that it is lower than the tier for the second value of the tier flag.

[0401] Level information according to the embodiments may be a level indicator (bmptl_level_idc). The level information may indicate a specific level observed by the basemesh sub-bitstream. The level indicator may indicate a level value regarding a tier flag.

[0402] Based on profile information according to the embodiments, the basemesh sub-bitstream may follow constraints on the profile. The constraints on the basemesh profile may include at least one of the constraints that the basemesh sub-bitstream includes a maximum number of sublayers minus 1 (bmsps_max_sub_layers_minus1) with a value of 0, or the constraints that the tier and level constraints on the basemesh profile are satisfied.

[0403] Constraints for a basemesh main profile according to the embodiments may include at least one of the following: a constraint in which the activated basemesh sequence parameter set (BMSPS) includes a maximum number of sublayers minus 1 (bmsps_max_sub_layers_minus1) with a value of 0; or a constraint in which constraints for tiers and levels for the basemesh main profile are satisfied.

[0404] The activated basemesh sequence parameter set (BMSPS) according to the embodiments may be generated by the encoding method, etc., according to the embodiments and included in the bitstream. For the activated basemesh sequence parameter set (BMSPS), refer to the description of the bmesh_sequence_parameter_set_rbsp syntax described above.

[0405] The temporal sublayer according to the embodiments may mean a temporally scalable layer of a temporally scalable basemesh sub-bitstream and may be composed of a BMCL (Basemesh Coding Layer) NAL (Network Abstraction Layer) unit having a BmTemporalId variable of a specific value and a non-BMCL NAL unit associated therewith.

[0406] The temporal sub-layer according to the embodiments refers to an access unit within the coded base mesh. Here, the maximum sub-layer may refer to the maximum value of the temporal sub-layer related to the access unit for the base mesh.

[0407] If the maximum number of sublayers according to the embodiments is constrained to 0, the decoder according to the embodiments has a constraint of decoding the base mesh based on one access unit rather than multiple sublayers.

[0408] According to embodiments, the basemesh sub-bitstream can comply with constraints on the tier and level based on tier information and level information.

[0409] The tier information and level information according to the embodiments may represent the minimum specifications of a decoder for decoding a basemesh sub-bitstream.

[0410] The decoder according to the embodiments must guarantee performance to satisfy the maximum number of basemesh attributes, maximum attribute bit depth, maximum CBMB size, and maximum number of regular counts that can be processed per unit time, etc., required in tier information and level information.

[0411] Constraints on tiers and levels according to the embodiments include: a constraint that the number of basemesh-related attributes (bmsps_mesh_attribute_count) is less than or equal to the maximum number of basemesh attributes (MaxBmAttrCount); a constraint that the bit depth of basemesh-related attributes minus 1 (bmsps_attribute_bit_depth_minus1) plus 1 is less than or equal to the maximum attribute bit depth (MaxAttrBitDepth); for BMCL (Basemesh Coding Layer) HRD (Hypothetical Reference Decoder) parameters, a constraint that the coded basemesh buffer (CBMB) size (CbmbSize) is less than or equal to the value related to the maximum CBMB size; or for BMNAL (Basemesh Network Abstraction Layer) HRD parameters, a constraint that the CBMB size (CbmbSize) is less than or equal to the product of the maximum CBMB size and the NAL-related factor; It may include at least one of the following.

[0412] The number of attributes associated with the base mesh (bmsps_mesh_attribute_count) and the bit depth minus 1 of the attributes associated with the base mesh (bmsps_attribute_bit_depth_minus1) according to the embodiments correspond to syntax elements included in the BMSPS and may be signaling information.

[0413] At least one of the number of maximum basemesh attributes, maximum attribute bit depth, or maximum CBMB size according to the embodiments may be a value derived based on level information.

[0414] The maximum number of base mesh attributes (MaxBmAttrCount), maximum attribute bit depth (MaxAttrBitDepth), and maximum CBMB size according to the embodiments may be specified in Table 4.

[0415] Table 4 according to the embodiments may represent variables derived based on level information or tier information. Table 4 may include values ​​for the maximum number of vertices per basemesh frame, maximum CBMB size, maximum number of basemesh frames that can be processed per unit time (MaxBmBR), maximum number of submesh frames that can be processed per unit time (MaxSubMeshesRate), and maximum number of basemesh vertices that can be processed per unit time (MaxBmVertsRate), based on the level information or tier information. Furthermore, Table 4 may include various variables based on the level information or tier information, but is not limited thereto. For example, Table 4 may further include the maximum number of basemesh attributes (MaxBmAttrCount) and the maximum attribute bit depth (MaxAttrBitDepth).

[0416] A submesh according to the embodiments refers to an independently decodingable region of a base mesh. A base mesh frame according to the embodiments includes a base mesh, and the maximum number of vertices of the base mesh included in the base mesh frame can be derived according to the level constraints according to the embodiments. The maximum number of submesh(s) of the base mesh of the base mesh frame according to the embodiments can be derived according to the level constraints according to the embodiments.

[0417] In addition, the maximum CBMB size can be set according to the level information, such as 10 when the level is 1, 20 when the level is 2, and 40 when the level is 3.

[0418] According to the embodiments, two types of HRD parameter sets (BMNAL HRD parameters and BMCL HRD parameters) may be used. The HRD parameter sets may be passed through the bm_hrd_parameters() syntax element.

[0419] The CBMB size (CbmbSize) according to the embodiments may be set by syntax elements according to the embodiments. For example, the CBMB size may be set by the SubBmHrdFlag, CBMB size value minus 1 (cbmb_size_value_minus1), CBMB size scale (cbmb_size_scale) syntax elements, etc.

[0420] The decoder according to the embodiments can decode a basemesh coding layer based on a Hypothetical Reference Decoder (HRD). The HRD according to the embodiments refers to a virtual reference decoder and can support seamless decoding operations by inputting, storing, and outputting data to be input / decoded / output into a buffer based on a buffer model and according to a bit rate. The HRD parameters according to the embodiments may include parameters regarding buffer size, timing, bit rate, etc., to support buffer operations for decoding. The buffer for HRD regarding the basemesh according to the embodiments may include a Coded Basemesh Buffer (CBMB) that stores the coded basemesh and / or a Decoded Basemesh Buffer (DBMB) that stores the decoded basemesh.

[0421] The decoder according to the embodiments can store base mesh data in a buffer based on buffer capacity derived according to constraints regarding the HRD buffer size, and can decode mesh data by referencing the base mesh data stored in the buffer.

[0422] A basemesh sub-bitstream according to the embodiments complies with constraints on a dynamic profile based on tier information and level information, and the constraints on the dynamic profile may include at least one of: a constraint that, for a Basemesh Coding Layer (BMCL) Hypothetical Reference Decoder (HRD) parameter, the bit rate (BitRate) is less than or equal to the maximum basemesh bit rate (MaxBmBR); or, for a Basemesh Network Abstraction Layer (BMAL) Hypothetical Reference Decoder (HRD) parameter, the bit rate (BitRate) is less than or equal to the product of the NAL / BMCL ratio (NalBmclRatio) and the maximum basemesh bit rate (MaxBmBR).

[0423] BitRate can be set by syntax elements according to the embodiments. For example, BitRate can be set by bit rate value minus 1 (bm_bit_rate_value_minus1[ i ]), bit rate scale (bm_bit_rate_scale) syntax elements, etc.

[0424] In this case, the maximum basemesh bit rate may refer to the maximum number of basemesh bits that can be processed per hour and may be set based on level information and / or tier information. Referring to Table 4, the maximum basemesh bit rate (MaxBmBR) may have a value in units of 1,000,000 bits / s. For example, the maximum basemesh bit rate may be set to a value of 10 when the level is 1, 20 when the level is 2, etc.

[0425] The NAL / BMCL ratio (NalBmclRatio) according to the embodiments can be set to a specific value, for example, NalBmclRatio can be set to 1.1.

[0426] Constraints for the dynamic profile according to the embodiments may further include a constraint that the nominal removal time of basemesh access unit n (where n is greater than 0) coded in the CBMB is AuNominalRemovalTime[n] - AuCbmbRemovalTime[n - 1] is greater than or equal to Max(aR, fR). Here, the access unit nominal removal time (AuNominalRemovalTime[n]) represents the nominal removal time of access unit n, and the access unit CBMB removal time (n-1) represents the CBMB removal time of access unit n-1. The variable aR can be set to the basemesh frame vertex count (BmFrameVertCount) / basemesh vertex rate (MaxBmVertsRate). The basemesh vertex rate may be a value set according to Table 4. And fR can be set to a specific value, for example, 1÷1000.

[0427] Constraints for dynamic profiles according to embodiments may further include a constraint that DbmbOutputInterval[n] is greater than or equal to Max(aR, fR) when the difference between consecutive output times of basemesh in DBMB is not the last basemesh of the basemesh sub-bitstream to which basemesh frame n is output. In this case, the decoded basemesh output interval (DbmbOutputInterval) may be the difference between the output time of the first basemesh frame and the output time of the basemesh frame where BasemeshFrameOutputFlag is 1.

[0428] Constraints on the dynamic profile according to the embodiments may include, for the BMCL HRD parameter, a constraint that for at least one value of i within the range of 0 to hrd_cbmb_cnt_minus1[Htid], BitRate[i] is less than or equal to MaxBmBR. Here, BitRate[i] is specified based on the selected parameter, and MaxBmBR can be specified in Table 4. The value of the number of coded basemesh buffers (bm_hrd_cbmb_cnt_minus1[i]) plus 1 specifies the number of alternate CBMB specifications in the CBMS bitstream when HighestTid is equal to i. The value of the number of coded basemesh buffers (cbmb_cnt_minus1[i]) must be in the range of 0 to 31. If this value is missing, the value of cbmb_cnt_minus1[i] is assumed to be 0.

[0429] Constraints on the dynamic profile according to the embodiments may include, for the BMNAL HRD parameter, a constraint that for at least one i value within the range from 0 to hrd_cbmb_cnt_minus1[Htid], BitRate[i] is less than or equal to NalBmclRatio * MaxBmBR. Here, BitRate[i] is specified based on the selected parameter, MaxBmBR is specified in Table 4, and NalBmClRatio can be set to 1.1.

[0430] The embodiments including FIGS. 17 to 18 can provide the following technical effects.

[0431] The embodiments define profiles and levels for interoperability between a basemesh sub-bitstream, which is one of the components of a V-DMC bitstream, and a basemesh decoder for decoding it, and define syntax elements for signaling it.

[0432] These syntax elements enable the basemesh decoder to determine in advance what level of decoding specification / function is required to receive and accurately decode the corresponding sub-bitstream, thereby providing a means for interoperability between a server providing the V-DMC bitstream and a receiver receiving, decoding, and processing the V-DMC bitstream.

[0433] Through the method provided in this embodiment, the receiver can perform operations such as memory, buffer, and other resource allocation necessary to accurately decode the basemesh sub-bitstream.

[0434] The embodiments have been described in terms of methods and / or devices, and the description of the methods and the description of the devices may be applied complementarily.

[0435] Although the drawings have been described separately for the convenience of explanation, it is also possible to design a new embodiment by combining the embodiments described in each drawing. Furthermore, designing a computer-readable recording medium containing a program for executing the previously described embodiments, as required by a person skilled in the art, falls within the scope of the embodiments. The apparatus and method according to the embodiments are not limited to the configuration and method of the embodiments described above; rather, the embodiments may be configured by selectively combining all or part of each embodiment to allow for various modifications. Although preferred embodiments have been illustrated and described, the embodiments are not limited to the specific embodiments described above. It is not only possible for a person skilled in the art to make various modifications without departing from the essence of the embodiments claimed in the claims, but such modifications should not be understood individually from the technical concept or perspective of the embodiments.

[0436] Various components of the device of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. Various components of the embodiments may be implemented as a single chip, for example, a single hardware circuit. Depending on the embodiments, the components according to the embodiments may each be implemented as separate chips. Depending on the embodiments, at least one of the components of the device according to the embodiments may be composed of one or more processors capable of executing one or more programs, and one or more programs may include instructions for performing or executing any one or more of the operations / methods according to the embodiments. Executable instructions for performing the methods / operations of the device according to the embodiments may be stored in non-transient CRMs or other computer program products configured to be executed by one or more processors, or may be stored in transient CRMs or other computer program products configured to be executed by one or more processors. Additionally, memory according to the embodiments may be used as a concept that includes not only volatile memory (e.g., RAM, etc.) but also non-volatile memory, flash memory, PROM, etc. In addition, it may also include implementation in the form of carrier waves, such as transmission over the Internet. Furthermore, processor-readable recording media are distributed across networked computer systems, allowing processor-readable code to be stored and executed in a distributed manner.

[0437] In this document, “ / ” and “,” are interpreted as “and / or.” For example, “A / B” is interpreted as “A and / or B,” and “A, B” is interpreted as “A and / or B.” Additionally, “A / B / C” means “at least one of A, B and / or C.” Also, “A, B, C” means “at least one of A, B and / or C.” Additionally, in this document, “or” is interpreted as “and / or.” For example, “A or B” may mean 1) “A” alone, 2) “B” alone, or 3) “A and B.” In other words, “or” in this document may mean “additionally or alternatively.”

[0438] Terms such as "first," "second," etc., may be used to describe various components of the embodiments. However, the interpretation of the various components according to the embodiments should not be limited by these terms. These terms are merely used to distinguish one component from another. For example, the first user input signal may be referred to as the second user input signal. Similarly, the second user input signal may be referred to as the first user input signal. The use of these terms should be interpreted as not departing from the scope of the various embodiments. Although the first user input signal and the second user input signal are both user input signals, they do not mean the same user input signals unless clearly indicated in the context.

[0439] The terms used to describe the embodiments are intended for the purpose of describing specific embodiments and are not intended to limit the embodiments. As used in the description of the embodiments and in the claims, the singular is intended to include the plural unless explicitly indicated in the context. Expressions of and / or are used to mean including all possible combinations between the terms. Expressions of include describe the presence of features, numbers, steps, elements, and / or components and do not imply the exclusion of additional features, numbers, steps, elements, and / or components. Conditional expressions such as "if" or "when" used to describe the embodiments are not limited to being optional. It is intended to be interpreted as "when a specific condition is satisfied," "when a related action is performed in response to a specific condition," or "when a related definition is interpreted."

[0440] Additionally, operations according to the embodiments described herein may be performed by a transmitting and receiving device including memory and / or a processor, depending on the embodiments. The memory may store programs for processing / controlling operations according to the embodiments, and the processor may control various operations described in this document. The processor may be referred to as a controller, etc. Operations in the embodiments may be performed by firmware, software, and / or a combination thereof, and the firmware, software, and / or a combination thereof may be stored in the processor or in memory.

[0441] Meanwhile, the operation according to the embodiments described above may be performed by a transmitting device and / or a receiving device according to the embodiments. The transmitting and receiving device may include a transmitting and receiving unit for transmitting and receiving media data, a memory for storing instructions (program code, algorithm, flowchart and / or data) for a process according to the embodiments, and a processor for controlling the operations of the transmitting and receiving devices.

[0442] The processor may be referred to as a controller, etc., and may correspond, for example, to hardware, software, and / or a combination thereof. The operation according to the embodiments described above may be performed by the processor. Additionally, the processor may be implemented as an encoder / decoder, etc., for the operation of the embodiments described above.

[0443]

[0444] As described above, the relevant details have been explained in the best mode for carrying out the embodiments.

[0445]

[0446] As described above, the embodiments may be applied wholly or partially to point cloud data transmission and reception devices and systems.

[0447] Those skilled in the art may make various changes or modifications to the embodiments within the scope of the embodiments.

[0448] The embodiments may include modifications / variations, and such modifications / variations do not exceed the scope of the claims and their equivalents.

Claims

Step of decoding the basemesh within the bitstream; A step of decoding displacement within the bitstream; and A step of decoding attributes within the bitstream; comprising Decryption method. In paragraph 1, The step of decoding the basemesh includes the step of decoding the basemesh within the basemesh sub-bitstream within the bitstream based on profile information, tier information, and level information. Decryption method. In paragraph 2, The above profile information indicates that the basemesh sub-bitstream conforms to at least one of the basemesh main profile or the basemesh main steel profile. Decryption method. In paragraph 3, Based on the above profile information, the basemesh sub-bitstream follows the constraints of the profile, and The constraints on the above profile are: The above basemesh sub-bitstream includes a constraint containing the maximum number of sublayers minus 1 (bmsps_max_sub_layers_minus1) with a value of 0; or Constraints that satisfy the tier and level constraints for the above base mesh profile; including at least one of, Decryption method. In paragraph 2, The above tier information is the tier flag (tier_flag), and The tier with respect to the first value of the above tier flag indicates that it is lower than the tier with respect to the second value of the above tier flag, and The above level information is a level indicator (level_idc), and The above level indicator indicates the level value regarding the above tier flag, Decryption method. In paragraph 5, The above basemesh sub-bitstream complies with constraints on tiers and levels based on the tier flag and the level indicator, and The constraints on the above tiers and levels are: Constraint that the number of attributes associated with the above base mesh (bmsps_mesh_attribute_count) is less than or equal to the maximum number of base mesh attributes (MaxBmAttrCount); Constraint that the bit depth minus 1 (bmsps_attribute_bit_depth_minus1) plus 1 of the attribute associated with the above base mesh is less than or equal to the maximum attribute bit depth (MaxAttrBitDepth); For BMCL (Basemesh Coding Layer) HRD (Hypothetical Reference Decoder) parameters, the constraint that the Coded Basemesh Buffer (CBMB) size (CbmbSize) is less than or equal to a value related to the maximum CBMB size; or For BMNAL (Basemesh Network Abstraction Layer) HRD parameters, the constraint that the CBMB size (CbmbSize) is less than or equal to the product of the maximum CBMB size and the NAL (Network Abstraction Layer) related factor; including at least one of, Decryption method. In paragraph 6, At least one of the number of the maximum basemesh attributes, the maximum attribute bit depth, or the maximum CBMB size is derived based on the level indicator, Decryption method. In Paragraph 7, The maximum number of vertices per base mesh frame for the above base mesh, and the maximum number of submeshes per base mesh frame for the above base mesh are derived based on the level indicator, Decryption method. In paragraph 5, The above basemesh sub-bitstream complies with constraints on the dynamic profile based on the profile information, the tier flag, and the level indicator, and The constraints on the above dynamic profile are: For BMCL (Basemesh Coding Layer) HRD (Hypothetical Reference Decoder) parameters, the BitRate is constrained to be less than or equal to the Max Basemesh BitRate (MaxBmBR); or For the BMNAL (Basemesh Network Abstraction Layer) HRD (Hypothetical Reference Decoder) parameter, the bitrate (BitRate) includes at least one of the constraints that the bitrate (BitRate) is less than or equal to the product of the NAL / BMCL ratio (NalBmclRatio) and the maximum basemesh bitrate (MaxBmBR), Decryption method. In Paragraph 9, The above maximum basemesh bit rate is derived based on the level indicator, Decryption method. Memory; and At least one processor connected to the memory; comprising, wherein the at least one processor: Decoding the basemesh within the bitstream; Decoding displacement within the above bitstream; and Configured to decode attributes within the above bitstream, Decoding device. Step of encoding the base mesh of the mesh data; A step of encoding the displacement of the above mesh data; and A step of encoding attributes of the above mesh data; comprising Encoding method. Memory; and At least one processor connected to the memory; comprising, wherein the at least one processor: Encode the base mesh of the mesh data; Encoding the displacement of the above mesh data; and Configured to encode the attributes of the above mesh data, Encoding device. A computer-readable storage medium for storing a bitstream generated by the method according to paragraph 12. Step of acquiring a bitstream for mesh data, The bitstream is generated based on the steps of: encoding the base mesh of the mesh data; encoding the displacement of the mesh data; and encoding the attributes of the mesh data; and A method comprising the step of transmitting data including the bitstream above.

Images