Mesh data transmission device, mesh data transmission method, mesh data reception device, and mesh data reception method

Abstract

A decoding method according to embodiments may comprise the steps of: decoding a base mesh in a bitstream; decoding displacement data in the bitstream; and decoding attribute data in the bitstream, wherein the bitstream includes data units, each data unit includes a data unit header and a data unit payload, the data unit header includes type information indicating a type of data carried in the data unit payload, and atlas data, the base mesh, the displacement data, and the attribute data are parsed from the bitstream on the basis of the type information.

Description

Mesh data transmission device, mesh data transmission method, mesh data reception device, and mesh data reception method

[0001] The embodiments provide a method for providing 3D content to provide users with various services such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services.

[0002] Among 3D content, point cloud data and mesh data are collections of points in 3D space. However, the sheer number of points in 3D space makes it difficult to generate point cloud or mesh data.

[0003] That is, there is a problem that a lot of processing is required to transmit and receive 3D data with a large amount of points, such as point cloud data or mesh data.

[0004] The technical problem according to the embodiments is to provide a device and method for efficiently transmitting and receiving mesh data in order to solve the problems described above.

[0005] The technical problem according to the embodiments is to provide a device and method for resolving latency and encoding / decoding complexity of mesh data.

[0006] The technical problem according to the embodiments is to provide a device and method for accurately identifying and transmitting / receiving arithmetic encoded displacement data.

[0007] However, the scope of the embodiments is not limited to the aforementioned technical tasks, and the scope of the embodiments may be expanded to other technical tasks that can be inferred by a person skilled in the art based on the entire contents of this document.

[0008] To achieve the above-described purpose and other advantages, a decoding method according to embodiments may include a step of decoding a base mesh in a bitstream, a step of decoding displacement data in the bitstream, and a step of decoding attribute data in the bitstream.

[0009] According to embodiments, a bitstream includes data units, each data unit is composed of a data unit header and a data unit payload, the data unit header includes type information indicating a type of data carried by the data unit payload, and atlas data, the basemesh, the displacement data, and the attribute data can be parsed from the bitstream based on the type information.

[0010] According to embodiments, the displacement data is encoded based on a video codec or encoded based on an arithmetic codec, and the displacement data encoded based on the video codec and the displacement data encoded based on the arithmetic codec can be identified by the type information.

[0011] According to embodiments, the step of decoding displacement data within the bitstream may perform arithmetic codec-based decoding on the displacement data based on parameter set information, if the type information indicates that the data carried as the corresponding data unit payload is displacement data encoded based on the arithmetic codec.

[0012] According to embodiments, the parameter set information may include indication information indicating whether arithmetic encoded displacement data is linked to the corresponding atlas.

[0013] According to embodiments, if the instruction information indicates that arithmetic-encoded displacement data is linked to the atlas, the parameter set information may further include displacement-related information for a decoding process of the displacement data.

[0014] According to embodiments, the displacement-related information may include information indicating a method for transforming displacement vector samples decoded from the arithmetic-encoded displacement data for the corresponding atlas.

[0015] According to embodiments, the step of decoding displacement data within the bitstream may perform video codec-based decoding on the displacement data if the type information indicates that the data carried as the corresponding data unit payload is displacement data encoded based on the video codec.

[0016] According to embodiments, a decoding device includes a memory and at least one processor connected to the memory, wherein the at least one processor can be configured to decode a basemesh within a bitstream, decode displacement data within the bitstream, and decode attribute data within the bitstream.

[0017] According to embodiments, the encoding method may include a step of encoding a base mesh of mesh data, a step of encoding displacement data of the mesh data, and a step of encoding attribute data of the base mesh data.

[0018] According to embodiments, the displacement data may be encoded based on a video codec or based on an arithmetic codec.

[0019] According to embodiments, the encoded basemesh, the encoded displacement data, and the encoded attribute data are multiplexed into a bitstream and transmitted, the bitstream includes data units, each data unit is composed of a data unit header and a data unit payload, the data unit header includes type information indicating a type of data carried by the data unit payload, and the data unit payload includes one of the encoded basemesh, the encoded displacement data, and the encoded attribute data, and the displacement data encoded based on the video codec and the displacement data encoded based on the arithmetic codec can be identified by the type information.

[0020] According to embodiments, the encoding device includes a memory and at least one processor connected to the memory, wherein the at least one processor may be configured to encode a base mesh of the mesh data, encode displacement data of the mesh data, and encode attribute data of the mesh data.

[0021] According to embodiments, a computer-readable storage medium can store a bitstream generated by the encoding method.

[0022] According to embodiments, the transmission method may include obtaining a bitstream for mesh data and transmitting data including the bitstream.

[0023] According to embodiments, the bitstream may be generated based on a step of encoding a base mesh of the mesh data, a step of encoding displacement data of the mesh data, and a step of encoding attribute data of the mesh data.

[0024] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can provide a quality 3D service.

[0025] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can achieve various video codec methods.

[0026] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can provide general-purpose 3D content such as autonomous driving services.

[0027] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments signal displacement data encoded / decoded by video coding and / or arithmetic coding by distinguishing them into V3C unit types, so that a receiver can effectively recognize and decode them.

[0028] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments signal displacement-related information related to arithmetic-coded displacement data in the V-DMC extension of the V3C parameter set, thereby enabling a receiver to effectively decode the arithmetic-coded displacement data.

[0029] The drawings are included to further understand the embodiments, and the drawings illustrate the embodiments together with the description related to the embodiments. For a better understanding of the various embodiments described below, reference should be made to the following description of the embodiments in conjunction with the following drawings, in which like reference numerals correspond to corresponding parts throughout the drawings.

[0030] FIG. 1 illustrates a system for providing dynamic mesh content according to embodiments.

[0031] Figure 2 illustrates a V-MESH compression method according to embodiments.

[0032] Figure 3 illustrates pre-processing of V-MESH compression according to embodiments.

[0033] Figure 4 illustrates a mid-edge subdivision method according to embodiments.

[0034] Figure 5 illustrates a displacement generation process according to embodiments.

[0035] Figure 6 illustrates an intra-frame encoding process of V-MESH data according to embodiments.

[0036] Figure 7 illustrates an inter-frame encoding process of V-MESH data according to embodiments.

[0037] Figure 8 illustrates a lifting conversion process for displacement according to embodiments.

[0038] Figure 9 illustrates a process of packing transformation coefficients into a 2D image according to embodiments.

[0039] Figure 10 illustrates an attribute transfer process of a V-MESH compression method according to embodiments.

[0040] Figure 11 illustrates an intra-frame decoding process of V-MESH data according to embodiments.

[0041] Figure 12 shows an inter-frame decoding processor of V-MESH data.

[0042] Fig. 13 is a drawing showing an example of a transmitting device according to embodiments.

[0043] Fig. 14 is a drawing showing an example of a receiving device according to embodiments.

[0044] FIG. 15 is a diagram showing an example of a dynamic mesh bitstream structure encoded and transmitted in a transmitting device of the present disclosure.

[0045] FIG. 16 is a diagram showing an example of the syntax structure of a V3C unit header according to embodiments.

[0046] Figure 17 shows examples of data types of V3C units assigned to V3C unit types according to embodiments.

[0047] FIG. 18 is a diagram showing an example of the syntax structure of a V3C unit payload according to embodiments.

[0048] FIG. 19 is a diagram showing an example of a dynamic mesh bitstream structure according to embodiments.

[0049] FIG. 20 is a diagram showing an example of the syntax structure of vps_vdmc_extension() according to embodiments.

[0050] FIG. 21 is a diagram showing an example of the syntax structure of displacement_information() according to embodiments.

[0051] FIG. 22 is a diagram showing examples of allowed syntax element values ​​for V-DMC toolset profile components according to embodiments.

[0052] FIG. 23 is a drawing showing another example of a receiving device according to embodiments.

[0053] Figure 24 shows an example of a detailed block diagram of a mesh decoding module according to embodiments.

[0054] Figure 25 is a flowchart showing an example of a transmission method according to embodiments.

[0055] Fig. 26 is a flowchart showing an example of a receiving method according to embodiments.

[0056] Preferred embodiments of the embodiments are described in detail, examples of which are illustrated in the accompanying drawings. The following detailed description, with reference to the accompanying drawings, is intended to illustrate preferred embodiments of the embodiments, rather than merely show embodiments that can be implemented according to the embodiments. The following detailed description includes details to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may be practiced without these details.

[0057] While most of the terms used in the examples are commonly used in the field, some terms were arbitrarily selected by the applicant, and their meanings are described in detail in the following descriptions as needed. Therefore, the examples should be understood based on the intended meaning of the terms, not simply their names or meanings.

[0058] With the recent development of 3D data modeling and rendering technology, research on creating and processing 3D data is being conducted in various fields such as Virtual Reality (VR), Augmented Reality (AR), autonomous driving, Computer-Aided Design (CAD) / Computer-Aided Manufacturing (CAM), and Geographic Information Systems (GIS). 3D data can be represented as point clouds, meshes, etc., depending on the representation format. Among these, a mesh is composed of geometric information expressing the coordinate values ​​of each vertex (or point), connection information indicating the connection relationship between vertices, a texture map expressing the color information of the mesh surface as 2D image data, and texture coordinates indicating mapping information between the surface of the mesh and the texture map. In the present disclosure, a mesh is defined as a dynamic mesh if one or more of the elements that make up the mesh change over time, and a static mesh if they do not change. In other words, dynamic mesh data may refer to mesh data that has an object or movement.

[0059] Because dynamic mesh data has a large amount of data for elements that constitute the mesh compared to two-dimensional image data, technologies have been developed to efficiently compress this large amount of mesh data to store and transmit it.

[0060] FIG. 1 illustrates a system for providing dynamic mesh content according to embodiments.

[0061] The system of FIG. 1 includes a transmitting device (100) and a receiving device (110) according to embodiments. The transmitting device (100) may include a mesh video acquisition unit (101), a mesh video encoder (102), a file / segment encapsulator (103), and a transmitter (104). The receiving device (110) may include a receiving unit (111), a file / segment decapsulator (112), a mesh video decoder (113), and a renderer (114). Each component of FIG. 1 may correspond to hardware, software, a processor, and / or a combination thereof. Hereinafter, the mesh data transmitting device according to embodiments may be interpreted as a term referring to a 3D data transmitting device or transmitting device (100), or a mesh video encoder (hereinafter, referred to as an encoder) (102). The mesh data receiving device according to the embodiments may be interpreted as a term referring to a 3D data receiving device or receiving device (110), or a mesh video decoder (hereinafter, decoder) (113).

[0062] The system of FIG. 1 can perform video-based dynamic mesh compression and decompression.

[0063] Advances in 3D capture, modeling, and rendering have enabled users to consume diverse forms of 3D content, such as AR, XR, metaverse, and holograms, across multiple platforms and devices. 3D content increasingly represents objects with greater precision and realism, enabling users to enjoy immersive experiences. To achieve this, the creation and use of 3D models requires a significant amount of data. Among various types of 3D content, 3D meshes are widely used for efficient data utilization and realistic object representation. Embodiments include a series of processing steps in a system that utilizes such mesh content.

[0064] First, the method of compressing dynamic mesh data starts from the V-PCC (Video-based point cloud compression) standard technology for point cloud data. Point cloud data is data that has color information at the coordinates (X, Y, Z) of a vertex (or point). In the present disclosure, the coordinates (i.e., position information) of a vertex are referred to as geometry information, the color information of a vertex is referred to as attribute information, and the geometry information and attribute information are referred to as vertex information or point cloud data. The vertex information to which connectivity information between vertices is added is referred to as mesh data. When creating content, it can be created in the form of mesh data from the beginning. Alternatively, it can be used by converting it into mesh data by adding connectivity information to point cloud data.

[0065] Currently, the MPEG standards body defines two types of dynamic mesh data: Category 1: Mesh data with texture maps as color information. Category 2: Mesh data with vertex colors as color information.

[0066] Mesh coding standards for Category 1 data are currently under development, and work on Category 2 data standards is also planned for the future. The overall process for providing mesh content services may include acquisition, encoding, transmission, decoding, rendering, and / or feedback, as shown in Figure 1.

[0067] To provide mesh content services, 3D data acquired through multiple cameras or specialized cameras can be processed into mesh data types through a series of processes and then converted into video. The generated mesh video is then transmitted through a series of processes, and the receiving end can then reprocess the received data into mesh video and render it. This allows mesh video to be presented to users, who can then interact with the mesh content according to their intended intent.

[0068] A mesh compression system may include a transmitting device (100) and a receiving device (110) as shown in FIG. 1. The transmitting device (100) may encode mesh video to output a bitstream, and transmit the bitstream to the receiving device (110) in the form of a file or streaming (streaming segment) via a digital storage medium or a network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0069] In the above transmitting device (100), the encoder may be called a mesh video / video / picture / frame encoding device, and in the receiving device (110), the decoder may be called a mesh video / video / picture / frame decoding device. The transmitter may be included in a mesh video encoder. The receiver may be included in a mesh video decoder. The renderer (114) may include a display unit, and the renderer and / or the display unit may be configured as separate devices or external components. The transmitting device (100) and the receiving device (110) may further include separate internal or external modules / units / components for a feedback process.

[0070] Mesh data represents the surface of an object as a number of polygons. Each polygon is defined by vertices in 3D space and connection information that describes how the vertices are connected. It can also contain vertex attributes such as vertex color and normal. Mapping information that allows the surface of the mesh to be mapped to a 2D planar area can also be included in the attributes of the mesh. The mapping can be described as a set of parameter coordinates, generally called UV coordinates or texture coordinates, associated with the mesh vertices. Meshes contain 2D attribute maps, which can be used to store high-resolution attribute information such as textures, normals, and displacement. Here, displacement can be used interchangeably with displacement, displacement information, or displacement vectors (i.e., displacement vectors).

[0071] The mesh video acquisition unit (101) may include processing 3D object data acquired through a camera, etc. into a mesh data type having the attributes described above through a series of processes and generating a video composed of such mesh data. The mesh video may have attributes of the mesh, such as vertices, polygons, connection information between vertices, colors, normals, etc., that may change over time. A mesh video having attributes and connection information that change over time in this way may be expressed as a dynamic mesh video.

[0072] A mesh video encoder (102) can encode an input mesh video into one or more video streams. One video can include multiple frames, and one frame can correspond to a still image / picture. In this document, a mesh video can include a mesh image / frame / picture, and a mesh video can be used interchangeably with a mesh image / frame / picture. The mesh video encoder (102) can perform a Video-based Dynamic Mesh (V-Mesh) Compression procedure. The mesh video encoder (102) can perform a series of procedures such as prediction, transformation, quantization, and entropy coding for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0073] The file / segment encapsulator (103) can encapsulate encoded mesh video data and / or mesh video-related metadata in the form of a file, etc. Here, the mesh video-related metadata may be received from a metadata processing unit, etc. The metadata processing unit may be included in the mesh video encoder (102) or may be configured as a separate component / module. The file / segment encapsulator (103) can encapsulate the corresponding data in a file format such as ISOBMFF, or process it in the form of other DASH segments, etc. The file / segment encapsulator (103) may include mesh video-related metadata in the file format according to an embodiment. The mesh video metadata may be included in boxes at various levels in the ISOBMFF file format, for example, or may be included as data in a separate track within the file. Depending on the embodiment, the file / segment encapsulator (103) may encapsulate the mesh video related metadata itself into a file.

[0074] The transmission processing unit can process encapsulated mesh video data for transmission according to the file format. The transmission processing unit can be included in the transmission unit (104) or can be configured as a separate component / module. The transmission processing unit can process mesh video data according to any transmission protocol. The processing for transmission can include processing for transmission through a broadcast network or processing for transmission through broadband. According to an embodiment, the transmission processing unit can receive not only mesh video data but also mesh video-related metadata from the metadata processing unit and process it for transmission.

[0075] The transmission unit (104) can transmit encoded video / image information or data output in the form of a bitstream to the reception unit (111) of the reception device (110) via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (104) can include an element for generating a media file through a predetermined file format and can include an element for transmission via a broadcasting / communication network. The reception unit (111) can extract the bitstream and transmit it to a decoding device.

[0076] The receiving unit (111) can receive mesh video data transmitted by a mesh data transmission device. Depending on the channel through which it is transmitted, the receiving unit (111) can receive mesh video data through a broadcast network, through a broadband, or through a digital storage medium.

[0077] The receiving processing unit can perform processing according to the transmission protocol on the received mesh video data. The receiving processing unit can be included in the receiving unit (111) or can be configured as a separate component / module. In order to correspond to the processing performed for transmission on the transmitting side, the receiving processing unit can perform the reverse process of the aforementioned transmission processing unit. The receiving processing unit can transfer the acquired mesh video data to the file / segment decapsulator (112) and transfer the acquired mesh video-related metadata to the metadata parser. The mesh video-related metadata acquired by the receiving processing unit can be in the form of a signaling table.

[0078] The file / segment decapsulator (112) can decapsulate mesh video data in the form of a file received from a receiving processing unit. The file / segment decapsulator (112) can decapsulate files according to ISOBMFF, etc., to obtain a mesh video bitstream or mesh video-related metadata (metadata bitstream). The obtained mesh video bitstream can be transmitted to the mesh video decoder (113), and the obtained mesh video-related metadata (metadata bitstream) can be transmitted to the metadata processing unit. The mesh video bitstream may include metadata (metadata bitstream). The metadata processing unit may be included in the mesh video decoder (113) or may be configured as a separate component / module. The mesh video-related metadata obtained by the file / segment decapsulator (112) may be in the form of a box or track within a file format. The file / segment decapsulator (112) may receive metadata required for decapsulation from the metadata processing unit, if necessary. The mesh video related metadata may be passed to the mesh video decoder (113) and used in the mesh video decoding procedure, or may be passed to the renderer (114) and used in the mesh video rendering procedure.

[0079] The mesh video decoder (113) can receive a bitstream and perform a reverse operation corresponding to the operation of the mesh video encoder (102) to decode the video / image. The decoded mesh video / image can be displayed through the display unit of the renderer (114). The user can view all or part of the rendered result through a VR / AR display or a general display.

[0080] The feedback process may include a process of transmitting various feedback information that may be acquired during the rendering / display process to the transmitter or to the decoder on the receiver. Interactivity may be provided in mesh video consumption through the feedback process. Depending on the embodiment, head orientation information, viewport information indicating the area that the user is currently viewing, etc. may be transmitted during the feedback process. Depending on the embodiment, the user may interact with things implemented in the VR / AR / MR / autonomous driving environment, in which case information related to the interaction may be transmitted to the transmitter or the service provider during the feedback process. Depending on the embodiment, the feedback process may not be performed.

[0081] Head orientation information can refer to information about the user's head position, angle, and movement. Based on this information, information about the area the user is currently viewing within the mesh video, i.e. viewport information, can be calculated.

[0082] Viewport information can be information about the area the user is currently viewing in the mesh video. This can be used to perform gaze analysis to determine how the user consumes the mesh video, which area of ​​the mesh video they are gazing at, and for how long. Gaze analysis can be performed on the receiving side and transmitted to the transmitting side through a feedback channel. Devices such as VR / AR / MR displays can extract the viewport area based on the user's head position / orientation, the vertical or horizontal FOV supported by the device, etc.

[0083] Depending on the embodiment, the aforementioned feedback information may not only be transmitted to the transmitter but may also be consumed by the receiver. That is, the aforementioned feedback information may be utilized to perform decoding, rendering, and other processes on the receiver. For example, head orientation information and / or viewport information may be utilized to preferentially decode and render only the mesh video for the area currently being viewed by the user.

[0084] This document relates to embodiments of dynamic mesh video compression as described above. The method / embodiment disclosed in this document can be applied to the Video-based Dynamic Mesh Compression (V-Mesh) standard of the Moving Picture Experts Group (MPEG) or the next-generation video / image coding standard. Dynamic mesh video compression is a method for processing mesh connection information and attributes that change over time, and it can perform lossy and lossless compression for various applications such as real-time communication, storage, free-viewpoint video, and AR / VR.

[0085] The dynamic mesh video compression method described below is based on MPEG's V-Mesh method.

[0086] In this document, picture / frame can generally mean a unit representing one video of a specific time period.

[0087] A pixel or pel can refer to the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, only the pixel / pixel value of the chroma component, or only the pixel / pixel value of the depth component.

[0088] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0089] As described above, the encoding process of Fig. 1 is as follows.

[0090] That is, the video-based dynamic mesh compression (V-Mesh) compression method can provide a method of compressing dynamic mesh video data based on 2D video codecs such as HEVC (High Efficiency Video Coding) and VVC (Versatile Video Coding). The V-Mesh compression process receives the following data as input and performs compression.

[0091] Input mesh: Contains the 3D coordinates of the vertices that make up the mesh, normal information for each vertex, mapping information that maps the mesh surface to a 2D plane, and connection information between the vertices that make up the surface. The mesh surface can be expressed as triangles or more polygons, and connection information between the vertices that make up each surface is stored according to a set shape. The input mesh can be saved in the OBJ file format.

[0092] Attribute map: (Hereinafter, texture map is also used in the same meaning): Contains information about the attributes of the mesh (color, normal, displacement, etc.), and stores data in the form of a mapping of the surface of the mesh onto a 2D image. Mapping which part (surface or vertex) of the mesh each data of this attribute map corresponds to is based on the mapping information contained in the input mesh. Since the attribute map has data for each frame of the mesh video, it can also be expressed as an attribute map video. The attribute map in the V-Mesh compression method mainly contains the color information of the mesh, and is saved in an image file format (PNG, BMP, etc.).

[0093] Material Library File: Contains information about the material attributes used in a mesh, and in particular, information that links the input mesh to its corresponding attribute map. It is saved in the Wavefront Material Template Library (MTL) file format.

[0094] In the V-Mesh compression method, the following data and information can be generated through the compression process.

[0095] Base mesh: The input mesh is simplified (decimated) through a pre-processing process, thereby expressing the objects of the input mesh using the minimum number of vertices determined by the user's standards.

[0096] Displacement: Displacement information used to express the input mesh as similarly as possible to the base mesh, and is expressed in the form of 3D coordinates.

[0097] Atlas information: This is the metadata required to reconstruct a mesh using base mesh, displacement, and attribute map information. It can be created and utilized as sub-mesh units (such as patches) that make up the mesh.

[0098] Referring to FIGS. 2 to 7, a method for encoding mesh position information (or vertex position information) is described, and referring to FIGS. 6 to 10, etc., a method for encoding attribute information (attribute map) by restoring mesh position information is described.

[0099] Figure 2 illustrates a V-MESH compression method according to embodiments.

[0100] Fig. 2 illustrates the encoding process of Fig. 1, and the encoding process may include a pre-processing process and an encoding process. The mesh video encoder (102) of Fig. 1 may include a pre-processor (200) and an encoder (201) as in Fig. 2. In addition, the transmitting device of Fig. 1 may be broadly referred to as an encoder, and the mesh video encoder (102) of Fig. 1 may be referred to as an encoder. The V-Mesh compression method may include a pre-processing process (Pre-processing, 200) and an encoding process (Encoding, 201) as in Fig. 2. The pre-processor (200) of Fig. 2 may be located in front of the encoder (201) of Fig. 2. The pre-processor (200) and the encoder (201) of Fig. 2 may be referred to as a single encoder.

[0101] The pre-processor (200) can receive a static of a dynamic mesh (M(i)) and / or an attribute map (A(i)). The pre-processor (200) can generate a base mesh (m(i)) and / or a displacement (or displacement) (d(i)) through pre-processing. The pre-processor (200) can receive feedback information from the encoder (201) and generate the base mesh and / or the displacement based on the feedback information.

[0102] The encoder (201) can receive a base mesh (m(i)), a displacement (d(i)), a static of a dynamic mesh (M(i)), and / or an attribute map (A(i)). In the present disclosure, at least one of the base mesh (m(i)), the displacement (d(i)), the static of a dynamic mesh (M(i)), and / or the attribute map (A(i)) can be referred to as mesh-related data. The encoder (201) can encode the mesh-related data to generate a compressed bitstream.

[0103] Figure 3 illustrates a pre-processing process of V-MESH compression according to embodiments.

[0104] Fig. 3 illustrates the configuration and operation of the preprocessor of Fig. 2. In Fig. 3, the input mesh may include a static of a dynamic mesh (M(i)) and / or an attribute map (A(i)). In addition, the input mesh may include three-dimensional coordinates of vertices constituting the mesh, normal information of each vertex, mapping information for mapping the mesh surface to a 2D plane, connection information between vertices constituting the surface, etc.

[0105] Fig. 3 shows a process of performing pre-processing on an input mesh. The pre-processing process (200) may largely include four steps: 1) GoF (Group of Frame) generation, 2) Mesh Decimation, 3) UV parameterization, and 4) Fitting subdivision surface (300). According to embodiments, GoF generation may be referred to as a GoF generation process or a GoF generation unit, mesh simplification may be referred to as a mesh simplification process or a mesh simplification unit, UV parameterization may be referred to as a UV parameterization process or a UV parameterization unit, and the fitting subdivision surface may be referred to as a fitting subdivision surface process or a fitting subdivision surface unit. The pre-processor (200) can generate displacement and / or base meshes from the received input mesh and transmit them to the encoder (201). The pre-processor (200) can transmit GoF information associated with GoF generation to the encoder (201).

[0106] Below, each step of Fig. 3 is described.

[0107] GoF Generation: This is the process of generating a reference structure for mesh data. If the number of vertices, the number of texture coordinates, the vertex connection information, and the texture coordinate connection information of the mesh of the previous frame and the current mesh are all the same, the previous frame can be set as the reference frame. That is, if only the vertex coordinate values ​​are different between the current input mesh and the reference input mesh, the encoder (201) can perform inter frame encoding. Otherwise, intra frame encoding is performed for the corresponding frame.

[0108] Mesh Decimation: This process simplifies the input mesh to create a simplified mesh, or base mesh. Vertices to be removed from the original mesh are selected based on user-defined criteria, and the selected vertices and the triangles connected to them can be removed.

[0109] In the process of performing mesh simplification (Mesh decimation), the input mesh (voxelized), target triangle ratio (TTR), and minimum triangle component (CCCount) information are passed as input, and the simplified mesh (decimated mesh) can be obtained as output. In this process, connected triangle components smaller than the set minimum triangle component (CCCount) can be removed.

[0110] UV parameterization: This is the process of mapping a 3D surface of a decimated mesh into a texture domain. Parameterization can be performed using the UVAtlas tool. This process generates mapping information, which indicates where each vertex of the decimated mesh can be mapped to on a 2D image. This mapping information is expressed and stored as texture coordinates, and through this process, the final base mesh is created.

[0111] Fitting subdivision surface (300): This is a process of performing subdivision on a decimated mesh (i.e., a simplified mesh having texture coordinates). The displacement and base mesh generated through this process are output to the encoder (201). A user-defined method, such as a mid-edge method, may be applied as the subdivision method. A fitting process is performed so that the input mesh and the mesh on which the subdivision has been performed become similar to each other. In the present disclosure, the mesh on which the fitting process has been performed is referred to as a fitted subdivision mesh (or fitted subdivision mesh).

[0112] Figure 4 illustrates a mid-edge subdivision method according to embodiments.

[0113] Figure 4 illustrates the mid-edge method of the fitting subdivision surface described in Figure 3. Referring to Figure 4, an original mesh containing four vertices is subdivided to generate a sub-mesh. A sub-mesh can be generated by creating a new vertex in the middle of the edge between the vertices. Then, a fitting process is performed so that the input mesh and the sub-mesh become similar to each other, thereby generating a fitted sub-division mesh.

[0114] When a fitted subdivided mesh (hereinafter referred to as a fitted subdivided mesh) is generated, displacement is calculated using this result and a pre-compressed and decoded base mesh (hereinafter referred to as a reconstructed base mesh). That is, the reconstructed base mesh is subdivided in the same way as the fitting subdivision surface. The difference in position of each vertex between this result and the fitted subdivided mesh is the displacement for each vertex. Since displacement represents the position difference in three-dimensional space, it is also expressed as a value in the (x, y, z) space of a Cartesian coordinate system. Depending on the user input parameters, the (x, y, z) coordinate values ​​can be converted to (normal, tangential, bi-tangential) coordinate values ​​of the local coordinate system.

[0115] Fig. 5 illustrates a displacement generation process according to embodiments. The displacement generation process of Fig. 5 may be performed in a pre-processor (200) or in an encoder (201).

[0116] FIG. 5 illustrates in detail the displacement calculation method of the fitting subdivision surface (300) as described in FIG. 4.

[0117] An encoder and / or pre-processor according to embodiments may include 1) a subdivision unit, 2) a local coordinate system calculation unit, and 3) a displacement calculation unit. The subdivision unit may perform subdivision on a reconstructed base mesh to generate a subdivided reconstructed base mesh. Here, the restoration of the base mesh may be performed in the pre-processor (200) or in the encoder (201). The local coordinate system calculation unit may receive a fitted subdivision mesh and a subdivided reconstructed base mesh, and may convert a coordinate system of the mesh into a local coordinate system based on the fitted subdivision mesh and the subdivided reconstructed base mesh. The local coordinate system calculation operation may be optional. The displacement calculation unit may calculate a positional difference between the fitted subdivision mesh and the subdivided reconstructed base mesh. For example, a positional difference value between vertices of two input meshes may be generated. The vertex positional difference value becomes a displacement.

[0118] The mesh data transmission method and device according to the embodiments can encode mesh data as follows. Mesh data is a term including point cloud data. Point cloud data (which may be referred to as point cloud for short) according to the embodiments can refer to data including vertex coordinates (or geometry information) and color information (or attribute information). In addition, geometry images, attribute images, occupancy maps, and additional information (or patch information) generated through patch generation and packing based on vertex coordinates and color information are also referred to as point cloud data. Therefore, point cloud data including connection information can be referred to as mesh data. In this document, point cloud and mesh data can be used interchangeably.

[0119] The V-Mesh compression (reconstruction) method according to the embodiments may include intra frame encoding (Fig. 6) and inter frame encoding (Fig. 7).

[0120] Based on the results of the GoF generation described above, intra-frame encoding or inter-frame encoding is performed. In the case of intra-encoding, the data to be compressed may be a base mesh, displacement, attribute map, etc. In the case of inter-encoding, the data to be compressed may be a displacement, attribute map, and a motion field between a reference base mesh and the current base mesh.

[0121] Fig. 6 illustrates an intra-frame encoding process of a V-MESH compression method according to embodiments. Each component for the intra-frame encoding process of Fig. 6 corresponds to hardware, software, a processor, and / or a combination thereof.

[0122] The encoding process of FIG. 6 details the encoding of the mesh video encoder (102) of FIG. 1. That is, it shows the configuration of the mesh video encoder (102) when the encoding of FIG. 1 is an intra-frame method. The encoder of FIG. 6 may include a pre-processor (200) and / or an encoder (201). The pre-processor (200) and encoder (201) of FIG. 6 may correspond to the pre-processor (200) and encoder (201) of FIG. 3.

[0123] The preprocessor (200) can receive an input mesh and perform the preprocessing described above. The preprocessing can generate a base mesh and / or a fitted subdivision mesh.

[0124] The quantizer (411) of the encoder (201) can quantize the base mesh and / or the fitted subdivided mesh. The static mesh encoder (412) can encode the static mesh (i.e., the quantized base mesh) and generate a bitstream (i.e., a compressed base mesh bitstream) including the encoded base mesh. The static mesh decoder (413) can decode the encoded static mesh (i.e., the encoded base mesh). The inverse quantizer (414) can inversely quantize the quantized static mesh (i.e., the base mesh) to output a reconstructed (or restored) base mesh. The displacement calculation unit (415) can generate displacements (or displacements) based on the reconstructed static mesh (i.e., the base mesh) and the fitted subdivided mesh. According to embodiments, the displacement calculation unit (415) calculates displacement, which is the position difference between each vertex of the subdivided base mesh and the fitted subdivided mesh after subdividing (or refining) the restored base mesh. In other words, the displacement is a displacement vector, which is the position difference between the vertices of the two meshes so that the fitted subdivided (or refining) mesh becomes similar to the original mesh. The forward linear lifting unit (416) can perform lifting transformation on the input displacement to generate lifting coefficients (or transform coefficients). The quantizer (417) can quantize the lifting coefficients. The image packing unit (418) can pack an image based on the quantized lifting coefficients. The video encoder (419) can encode the packed image. That is, the quantized lifting coefficients are packed into one frame as a 2D image by the image packing unit (418), compressed through the video encoder (419), and output as a displacement bitstream (i.e., compressed displacement bitstream).

[0125] A video decoder (420) decodes a compressed displacement bitstream. An image unpacking unit (421) can perform unpacking on the decoded displacement frame to output quantized lifting coefficients. A dequantizer (422) can dequantize the quantized lifting coefficients. An inverse linear lifting unit (423) applies inverse lifting to the inverse quantized lifting coefficients to generate restored displacement. A mesh restoration unit (424) reconstructs and deforms a mesh using the restored displacement output from the inverse linear lifting unit (423) and the restored base mesh (or subdivided restored base mesh) output from the inverse quantization unit (414). The present disclosure refers to the reconstructed and deformed mesh as a restored deformed mesh.

[0126] The attribute transfer (425) receives an input mesh and / or an input attribute map, and regenerates an attribute map based on the restored deformed mesh. The attribute map refers to a texture map corresponding to attribute information among mesh data components, and in the present disclosure, the attribute map and the texture map may be used interchangeably. The push-pull padding (426) may pad data in the attribute map based on the push-pull method. The color space conversion unit (427) may convert the space of the color component of the attribute map. For example, the attribute map may be converted from an RGB color space to a YUV color space. The video encoder (428) may encode the attribute map and output it as a compressed attribute bitstream.

[0127] A multiplexer (430) can generate a compressed bitstream by multiplexing a compressed base mesh bitstream, a compressed displacement bitstream, and a compressed attribute bitstream.

[0128] In Fig. 6, the displacement calculation unit (415) may be included in the pre-processor (200). In addition, at least one of the quantizer (411), the static mesh encoder (412), the static mesh decoder (413), and the inverse quantizer (414) may be included in the pre-processor (200).

[0129] As described in FIG. 6, the intra-frame encoding method includes base mesh encoding (also called static mesh encoding). That is, when performing intra-frame encoding on the current input mesh frame, the base mesh generated in the pre-processing process of the pre-processor (200) can be encoded using a static mesh compression technology in a static mesh encoder (412) after undergoing a quantization process in a quantizer (411). In the V-Mesh compression method, for example, Draco technology is applied to base mesh encoding, and vertex position information, mapping information (texture coordinates), vertex connection information, etc. of the base mesh become compression targets.

[0130] The encoder of Fig. 6 generates a bitstream by compressing the base mesh, displacement, and attributes within the frame, and the encoder of Fig. 7 generates a bitstream by compressing the motion, displacement, and attributes between the current frame and the reference frame.

[0131] Fig. 7 illustrates an inter-frame encoding process of a V-MESH compression method according to embodiments. Each component for the inter-frame encoding process of Fig. 7 corresponds to hardware, software, a processor, and / or a combination thereof.

[0132] The encoding process of Fig. 7 details the encoding of Fig. 1. That is, it shows the configuration of an encoder when the encoding of Fig. 1 is an inter-frame method. The encoder of Fig. 7 may include a pre-processor (200) and / or an encoder (201). The pre-processor (200) and encoder (201) of Fig. 7 may correspond to the pre-processor (200) and encoder (201) of Fig. 3.

[0133] For a description of the components corresponding to the encoding operation of FIG. 6 among the encoding operations of FIG. 7, refer to the description of FIG. 6. That is, the operation of the quantizer (511), displacement calculation unit (515), wavelet transformer (516), quantizer (517), image packing unit (518), video encoder (519), video decoder (520), image unpacking unit (521), inverse quantizer (522), inverse wavelet transformer (523), mesh restoration unit (524), attribute transfer (525), push-pull padding (526), ​​color space conversion unit (527), video encoder (528), and multiplexer (530) of FIG. 7 is similar to that of the quantizer (411), static mesh encoder (412), static mesh decoder (413), inverse quantizer (414), displacement calculation unit (415), forward linear lifting unit (416), quantizer (417), image Since the operations described in the packing unit (418), video encoder (419), video decoder (420), image unpacking unit (421), inverse quantizer (422), inverse linear lifting unit (423), mesh restoration unit (424), attribute transfer (425), push-pull padding (426), color space conversion unit (427), video encoder (428), and multiplexer (430) are the same or similar, a detailed description thereof is omitted in FIG. 7 to avoid redundant description.

[0134] In Fig. 7, for inter-frame based encoding, the motion encoder (512) can obtain a motion vector between the two base meshes based on the restored quantized reference base mesh and the quantized current base mesh, and then encode the motion vector to output a compressed motion bitstream. The motion encoder (512) can be referred to as a motion vector encoder. The base mesh restoration unit (513) can restore the base mesh based on the restored quantized reference base mesh and the encoded motion vector. The restored base mesh is dequantized in the dequantizer (514) and then output to the displacement calculation unit (515).

[0135] In Fig. 7, the displacement calculation unit (515) may be included in the pre-processor (200). In addition, at least one of the quantizer (511), the motion encoder (512), the base mesh restoration unit (513), and the inverse quantizer (514) may be included in the pre-processor (200).

[0136] As described in Fig. 7, the inter-frame encoding method may include motion field encoding (also called motion vector encoding). Inter-frame encoding may be performed when a one-to-one correspondence of vertices is established between a reference mesh and a current input mesh, and only the position information of the vertices is different. When performing inter-frame encoding, instead of compressing the base mesh, the difference between the vertices of the reference base mesh and the current base mesh, i.e., the motion field (also called motion vector), may be calculated and encoded to encode this information. The reference base mesh is the result of quantizing the already decoded base mesh data and is determined according to the reference frame index determined in the GoF generation. The motion field may also be encoded as a value. Alternatively, the predicted motion field can be calculated by averaging the motion fields of the restored vertices among the vertices connected to the current vertex, and the residual motion field, which is the difference between the predicted motion field value and the motion field value of the current vertex, can be encoded. This residual motion field value can be encoded using entropy coding.The process of encoding displacement and attribute maps, excluding the motion field encoding process of inter frame encoding, is the same as the structure of the intra frame encoding method except for the base mesh encoding.

[0137] Figure 8 illustrates a lifting conversion process for displacement according to embodiments.

[0138] Figure 9 illustrates a process of packing transformation coefficients (or lifting coefficients) according to embodiments into a 2D image.

[0139] Figures 8 and 9 illustrate the process of transforming displacement and packing transform coefficients of the encoding process of Figures 6 and 7, respectively.

[0140] The encoding method according to the embodiments includes displacement encoding.

[0141] After base mesh encoding and / or motion field encoding, a reconstructed base mesh is generated through restoration and dequantization, and the displacement between the result of performing subdivision on the reconstructed base mesh and the fitted subdivided mesh generated through the fitting subdivision surface can be calculated (see 415 in FIG. 6 or 515 in FIG. 7). For effective encoding, a data transform process such as wavelet transform can be applied to the displacement information (see 416 in FIG. 6 or 516 in FIG. 7).

[0142] FIG. 8 shows a process of transforming displacement information using a lifting transform in the forward linear lifting unit (416) of FIG. 6 or the wavelet transformer (516) of FIG. 7. For example, a linear wavelet-based lifting transform may be performed. The transform coefficients generated through the transform process are quantized in a quantizer (417 or 517) and then packed into a 2D image through an image packing unit (418 or 518) as in FIG. 9. The transform coefficients are configured as one block for every 256 (= 16×16) units, and each block can be packed in a z-scan order. The horizontal number of blocks is fixed to 16, but the vertical number of blocks can be determined according to the number of vertices of the subdivided base mesh. Transform coefficients can be packed by aligning them with Morton codes within a single block. The packed images generate displacement videos for each GoF unit, and these displacement videos can be encoded using a conventional video compression codec in a video encoder (419 or 519).

[0143] Referring to FIG. 8, a base mesh (original) may include vertices and edges for LoD (Level of Detail) 0. A first subdivision mesh generated by dividing (or subdividing) the base mesh includes vertices generated by further dividing (or subdividing) edges of the base mesh. The first subdivision mesh includes vertices for LoD 0 and vertices for LoD 1. LoD 1 includes the subdivided vertices and the vertices of the base mesh (LoD 0). The first subdivision mesh may be further divided (or subdivided) to generate a second subdivision mesh. The second subdivision mesh includes LoD 2. LoD 2 includes base mesh vertices (LoD 0), LoD 1 including vertices further divided (or subdivided) from LoD 0, and vertices further divided (or subdivided) from LoD 1. LoD is a level of detail that indicates the degree of detail of mesh data content. As the level index increases, the distance between vertices becomes closer and the level of detail increases. In other words, the smaller the LoD value, the lower the detail of the mesh data content, and the larger the LoD value, the higher the detail of the mesh data content. LoD N contains the vertices included in the previous LoDN-1 as is. When a mesh (or vertex) is further divided through subdivision, the mesh can be encoded based on a prediction and / or update method by considering the previous vertices v1, v2, and the subdivided vertex v. Instead of directly encoding information about the current LoD N, a residual value between the previous LoD N-1 can be generated and the mesh can be encoded using the residual value to reduce the bitstream size. The prediction process refers to the operation of predicting the current vertex v using the previous vertices v1 and v2. Since adjacent subdivision meshes have similar data, this property can be utilized for efficient encoding.Current vertex position information is predicted as a residual of previous vertex position information, and the previous vertex position information is updated through the residual. In the present disclosure, vertex, apex, and point may be used with the same meaning. In addition, LoDs may be defined during the subdivision process of the base mesh. According to embodiments, the subdivision process of the base mesh may be performed in the pre-processor (200) or in a separate component / module.

[0144] Referring to FIG. 9, a vertex has a transform coefficient (also called a lifting coefficient) generated through a lifting transformation. The transform coefficient of a vertex related to a lifting transformation can be packed into an image by an image packing unit (418 or 518) and then encoded by a video encoder (419 or 519).

[0145] Figure 10 illustrates an attribute transfer process of a V-MESH compression method according to embodiments.

[0146] According to the embodiments, FIG. 10 shows the detailed operation of the attribute transfer (425 or 525) of the encoding of FIG. 6, FIG. 7, etc.

[0147] Encoding according to embodiments includes attribute map encoding. According to embodiments, attribute map encoding may be performed in the video encoder (428) of FIG. 6 or the video encoder (528) of FIG. 7.

[0148] According to embodiments, in the present disclosure, the encoder compresses information about the input mesh through base mesh encoding (i.e., intra encoding), motion field encoding (i.e., inter encoding), and displacement encoding. In the encoding process, the compressed input mesh is restored through base mesh decoding (intra frame), motion field decoding (inter frame), and displacement video decoding processes, and the restored result, the reconstructed deformed mesh (hereinafter referred to as Recon. deformed mesh), is used to compress the input attribute map as shown in FIGS. 6 and 7. The reconstructed deformed mesh (Recon. deformed mesh) has position information of vertices, texture coordinates, and corresponding connection information, but does not have color information corresponding to the texture coordinates. Therefore, as shown in Fig. 10, in the V-Mesh compression method, a new attribute map having color information corresponding to the texture coordinates of the reconstructed deformed mesh is regenerated through the attribute transfer process of attribute transfer (425 or 525).

[0149] According to embodiments, attribute transfer (425 or 525) first checks whether each point P(u, v) of a 2D texture domain belongs to a texture triangle of a reconstructed deformed mesh, and if it exists in a texture triangle T, the barycentric coordinate of P(u, v) according to the triangle T ( , , ) is calculated. And the 3D vertex positions of triangle T and ( , , ) is used to compute the 3D coordinates M(x, y, z) of P(u, v). Find the vertex coordinates M'(x', y', z') and the triangle T' containing this vertex that corresponds to the most similar position to the calculated M(x, y, z) in the input mesh domain. Then, the center of mass coordinates of M'(x', y', z') in this triangle T' ( ', ', ') is calculated. The texture coordinates corresponding to the three vertices of Triangle T' and ( ', ', ') is used to calculate the texture coordinates (u', v'), and the color information corresponding to these coordinates is found in the input attribute map. The color information found in this way is immediately assigned to the pixel location (u, v) of the new attribute map. If P(u, v) does not belong to any triangle, the pixel at that location in the new attribute map can be filled with a color value using a padding algorithm, such as the push-pull algorithm of push-pull padding (426 or 526).

[0150] The new attribute map generated through attribute transfer (425 or 525) is grouped into GoF units to form an attribute map video, which is compressed using the video codec of the video encoder (428 or 528).

[0151] Referring to Figure 10, the reference relationship between the input mesh, the input attribute map, the reconstructed deformed mesh, and the regenerated attribute map can be seen.

[0152] The decoding process of Fig. 1 can perform the reverse process of the corresponding process of the encoding process of Fig. 1. The specific decoding process is as follows.

[0153] FIG. 11 illustrates an intra-frame decoding (or intra-decoding) process of V-Mesh technology according to embodiments.

[0154] Fig. 11 illustrates the configuration and operation of the mesh video decoder (113) of the receiving device of Fig. 1. In addition, Fig. 11 can restore mesh data by performing the reverse process of the intra-frame encoding process of Fig. 6. Each component for the intra-frame decoding process of Fig. 11 corresponds to hardware, software, and / or a combination thereof.

[0155] First, the bitstream (i.e., compressed bitstream) received and input to the demultiplexer (611) of the intra frame decoding unit (610) can be separated into a mesh substream, a displacement substream, an attribute map substream, and a substream containing patch information of the mesh, such as V-PCC / V3C. The term V-PCC (Video-based Point Cloud Compression) used in this document can be used with the same meaning as V3C (Visual Volumetric Video-based Coding), and the two terms can be used interchangeably. Therefore, the term V-PCC in this document can be interpreted as the term V3C.

[0156] According to embodiments, the mesh sub-stream may be input to a static mesh decoder (612) and decoded, the displacement sub-stream may be input to a video decoder (613) and decoded, and the attribute map sub-stream may be input to a video decoder (617) and decoded.

[0157] According to embodiments, the mesh sub-stream is decoded through a decoder (612) of a static mesh codec used in encoding, such as Google Draco, and as a result, a reconstructed quantized base mesh, for example, connection information, vertex geometry information, vertex texture coordinates, etc. of the base mesh can be reconstructed.

[0158] According to embodiments, the displacement sub-stream is decoded into displacement video through a decoder (613) of a video compression codec used in encoding, and is restored as displacement information for each vertex (i.e., Recon. displacements) through an image unpacking process of an image unpacking unit (614), an inverse quantization process of an inverse quantizer (615), and an inverse transform process of an inverse linear lifting unit (616).

[0159] According to embodiments, the base mesh restored by the static mesh decoder (612) is inverse quantized by the inverse quantizer (620) and then output to the mesh restoration unit (630). The mesh restoration unit (630) reconstructs and restores the deformed mesh (i.e., decoded mesh) through the restored displacement output from the inverse linear lifting unit (616) and the restored base mesh output from the inverse quantizer (620). That is, the inverse quantized restored base mesh is combined with the restored displacement information to generate the final decoded mesh. In the present disclosure, the final decoded mesh is referred to as a reconstructed deformed mesh.

[0160] According to embodiments, an attribute map sub-stream is decoded through a decoder (617) corresponding to a video compression codec used in encoding, and then restored to a final attribute map (i.e., decoded attribute map) through a color conversion unit (640) through processes such as color format conversion and color space conversion.

[0161] According to embodiments, the restored decoded mesh and decoded attribute map can be utilized by the receiver as final mesh data that can be utilized by the user.

[0162] Referring to FIG. 11, the received compressed bitstream includes patch information, a mesh substream, a displacement substream, and an attribute map substream. A substream is interpreted as a term referring to a part of a bitstream included in a bitstream. The bitstream includes patch information (data), mesh information (data), displacement information (data), and attribute map information (data).

[0163] As described above, the decoder of FIG. 11 performs the following intra-frame decoding operations. The static mesh decoder (612) decodes the mesh sub-stream to generate a reconstructed quantized base mesh, and the inverse quantizer (620) applies the quantization parameters of the quantizer inversely to generate the reconstructed base mesh. The video decoder (613) decodes the displacement sub-stream, the image unpacking unit (614) unpacks the images of the decoded displacement video, and the inverse quantizer (615) inversely quantizes the quantized images. The inverse linear lifting unit (616) applies a lifting transform in the reverse process of the encoder to generate the reconstructed displacement. The mesh restoration unit (630) generates a reconstructed deformed mesh based on the reconstructed base mesh and the reconstructed displacement. The video decoder (617) decodes the attribute map sub-stream, and the color conversion unit (640) converts the color format and / or space of the decoded attribute map to generate a decoded attribute map.

[0164] Figure 12 illustrates the inter-frame decoding (or inter-decoding) process of V-Mesh technology.

[0165] Fig. 12 illustrates the configuration and operation of the mesh video decoder (113) of the receiving device of Fig. 1. In addition, Fig. 12 can restore mesh data by performing the reverse process of the inter-frame encoding process of Fig. 7. Each component for the inter-frame decoding process of Fig. 12 corresponds to hardware, software, and / or a combination thereof.

[0166] First, the bitstream received and input to the demultiplexer (711) of the intra frame decoding unit (710) can be separated into a motion sub-stream (also called a motion sub-stream or motion vector sub-stream), a displacement sub-stream, an attribute map sub-stream, and a sub-stream including patch information of a mesh such as V3C / V-PCC.

[0167] According to embodiments, the motion sub-stream may be input to a motion decoder (712) and decoded, the displacement sub-stream may be input to a video decoder (713) and decoded, and the attribute map sub-stream may be input to a video decoder (717) and decoded.

[0168] According to embodiments, a motion sub-stream is decoded through entropy decoding and inverse prediction processes in a motion decoder (712) and restored into motion information (or motion vector information). A base mesh restoration unit (718) combines the restored motion information with a reference base mesh that has already been restored and stored to generate a reconstructed quantized base mesh for the current frame. An inverse quantizer (720) applies inverse quantization to the restored quantized base mesh to generate a reconstructed base mesh. A video decoder (713) decodes a displacement sub-stream, an image unpacking unit (714) unpacks an image of the decoded displacement video, and an inverse quantizer (715) inversely quantizes a quantized image. The inverse linear lifting unit (716) applies a lifting transformation in the reverse process of the encoder to generate a restored displacement. The mesh restoration unit (730) generates a reconstructed deformed mesh, i.e., a final decoded mesh, based on the restored base mesh and the restored displacement.

[0169] According to embodiments, the video decoder (717) decodes the attribute map sub-stream in the same manner as intra decoding, and the color conversion unit (740) converts the color format and / or space of the decoded attribute map to generate a decoded attribute map. The decoded mesh and the decoded attribute map can be utilized by the receiver as final mesh data that can be utilized by the user.

[0170] Referring to Fig. 12, the bitstream includes motion information (also called motion vectors), displacement, and an attribute map. Since Fig. 12 performs inter-frame decoding, it further includes a process of decoding inter-frame motion information. The motion information is decoded, and a restored quantized base mesh for the motion information is generated based on the reference base mesh, thereby generating a restored base mesh. For a description of the operation of Fig. 12, which is identical to that of Fig. 11, refer to the description of Fig. 11.

[0171] Fig. 13 illustrates a mesh data transmission device according to embodiments.

[0172] FIG. 13 corresponds to the transmitting device (100) or mesh video encoder (102) of FIG. 1, the encoder (preprocessor and encoder) of FIG. 2, FIG. 6, or FIG. 7, and / or a transmitting encoding device corresponding thereto. Each component of FIG. 13 corresponds to hardware, software, a processor, and / or a combination thereof.

[0173] The operation process of a transmitter for compressing and transmitting dynamic mesh data using V-Mesh compression technology may be as shown in Fig. 13. The transmitter of Fig. 13 may perform an intra-frame encoding (or intra-encoding or intra-screen encoding) process and / or an inter-frame encoding (or inter-encoding or inter-screen encoding) process.

[0174] The pre-processor (811) receives the original mesh as input and generates a simplified mesh (decimated mesh) (or base mesh) and a fitted decimated mesh (or subdivision). Simplification can be performed based on the target number of vertices or target number of polygons that constitute the mesh. Parameterization, which generates texture coordinates and texture connection information per vertex, can be performed on the simplified mesh. For example, parameterization is a process of mapping a 3D surface to a texture domain for the decimated mesh. If parameterization is performed using the UVAtlas tool, mapping information is generated that can identify where each vertex of the decimated mesh can be mapped on a 2D image. The mapping information is expressed and stored as texture coordinates, and the final base mesh is generated through this process. In addition, the work of quantizing the mesh information in floating-point form into fixed-point form can be performed. This result can be output as a base mesh to a motion vector encoder (813) or a static mesh encoder (814) through a switching unit (812). The pre-processor (811) can perform mesh subdivision on the base mesh to generate additional vertices. Depending on the subdivision method, vertex connection information, texture coordinates, and texture coordinate connection information including the added vertices can be generated. The pre-processor (811) can generate a fitted subdivided mesh by adjusting the vertex positions so that the subdivided mesh becomes similar to the original mesh.

[0175] According to embodiments, the base mesh is output to a motion vector encoder (813) via a switching unit (812) when performing inter-encoding for the corresponding mesh frame, and is output to a static mesh encoder (814) via a switching unit (812) when performing intra-encoding for the corresponding mesh frame. The motion vector encoder (813) may be referred to as a motion encoder.

[0176] For example, when performing intra-encoding (or intra-frame encoding) on ​​the corresponding mesh frame, the base mesh can be compressed through a static mesh encoder (814). In this case, encoding can be performed on connection information, vertex geometry information, vertex texture information, normal information, etc. of the base mesh. The base mesh bitstream generated through encoding is transmitted to a multiplexer (823).

[0177] As another example, when performing inter-encoding (or inter-frame encoding) for the corresponding mesh frame, the motion vector encoder (813) can receive a base mesh and a reference reconstructed base mesh (or a reconstructed quantized reference base mesh) as input, calculate a motion vector between the two meshes, and encode the value. In addition, the motion vector encoder (813) can perform prediction based on connection information using a previously encoded / decoded motion vector as a predictor, and encode a residual motion vector obtained by subtracting the predicted motion vector from the current motion vector. The motion vector bitstream generated through encoding is transmitted to the multiplexer (823).

[0178] The base mesh restoration unit (815) can receive the base mesh encoded by the static mesh encoder (814) or the motion vector encoded by the motion vector encoder (813) and generate a reconstructed base mesh. For example, the base mesh restoration unit (815) can perform static mesh decoding on the base mesh encoded by the static mesh encoder (814) to restore the base mesh. At this time, quantization can be applied before the static mesh decoding, and inverse quantization can be applied after the static mesh decoding. As another example, the base mesh restoration unit (815) can restore the base mesh based on the reconstructed quantized reference base mesh and the motion vector encoded by the motion vector encoder (813). The reconstructed base mesh is output to the displacement calculation unit (816) and the mesh restoration unit (820).

[0179] The displacement calculation unit (816) can perform mesh refinement on the restored base mesh. The displacement calculation unit (816) can calculate a displacement vector, which is a difference value between the vertex positions of the restored base mesh and the fitted subdivision (or refined) mesh generated by the pre-processor (811). At this time, the displacement vector can be calculated as many times as the number of vertices of the refined mesh. The displacement calculation unit (816) can convert the displacement vector calculated in the 3D Cartesian coordinate system into a local coordinate system based on the normal vector of each vertex.

[0180] The displacement vector video generation unit (817) may include a linear lifting unit, a quantizer, and an image packing unit. That is, in the displacement vector video generation unit (817), the linear lifting unit may transform the displacement vector for effective encoding. The transformation may be performed by a lifting transformation, a wavelet transformation, etc., according to embodiments. In addition, quantization may be performed in a quantizer on the transformed displacement vector value, i.e., the transform coefficient. At this time, a different quantization parameter may be applied to each axis of the transform coefficient, and the quantization parameter may be derived according to an encoder / decoder agreement. The transformed and quantized displacement vector information may be packed into a 2D image in the image packing unit. The displacement vector video generation unit (817) may generate a displacement vector video by bundling packed 2D images for each frame, and the displacement vector video may be generated for each GoF (Group of Frame) unit of the input mesh.

[0181] The displacement vector video encoder (818) can encode the generated displacement vector video using a video compression codec. The generated displacement vector video bitstream is transmitted to a multiplexer (823).

[0182] The displacement vector restoration unit (819) may include a video decoder, an image unpacking unit, an inverse quantizer, and an inverse linear lifting unit. That is, the displacement vector restoration unit (819) performs decoding on an encoded displacement vector in the video decoder, performs image unpacking in the image unpacking unit, performs inverse quantization in the inverse quantizer, and then performs inverse transformation in the inverse linear lifting unit to restore the displacement vector. The restored displacement vector is output to the mesh restoration unit (820). The mesh restoration unit (820) restores a deformed mesh based on the base mesh restored by the base mesh restoration unit (815) and the displacement vector restored by the displacement vector restoration unit (819). The restored mesh (or referred to as a restored deformed mesh) has restored vertices, connection information between vertices, texture coordinates, and connection information between texture coordinates.

[0183] The texture map video generation unit (821) can regenerate a texture map based on the texture map (or attribute map) of the original mesh and the restored deformed mesh output from the mesh restoration unit (820). According to embodiments, the texture map video generation unit (821) can assign color information per vertex of the texture map of the original mesh to the texture coordinates of the restored deformed mesh. According to embodiments, the texture map video generation unit (821) can generate a texture map video by grouping the regenerated texture maps by GoF unit for each frame.

[0184] The generated texture map video can be encoded using a video compression codec of the texture map video encoder (822). The texture map video bitstream generated through encoding is transmitted to a multiplexer (823).

[0185] A multiplexer (823) multiplexes a motion vector bitstream (e.g., in case of inter encoding), a base mesh bitstream (e.g., in case of intra encoding), a displacement vector bitstream, and a texture map bitstream into a single bitstream. The single bitstream can be transmitted to a receiver via a transmitter (824). Alternatively, the motion vector bitstream, the base mesh bitstream, the displacement vector bitstream, and the texture map bitstream can be generated as a file with one or more track data or encapsulated into segments and transmitted to a receiver via the transmitter (824).

[0186] Referring to FIG. 13, a transmitting device (encoder) can encode a mesh in an intra-frame or inter-frame manner. A transmitting device according to intra-encoding can generate a base mesh, a displacement vector (or referred to as displacement), and a texture map (or referred to as attribute map). A transmitting device according to inter-encoding can generate a motion vector (or referred to as motion), a displacement vector (or referred to as displacement), and a texture map (or referred to as attribute map). The texture map obtained from the data input unit is generated and encoded based on the restored mesh. Displacement is generated and encoded through the difference in vertex positions between the base mesh and the divided (or subdivided or subdivided) mesh. More specifically, the displacement is the difference in position between the fitted sub-divided mesh and the sub-divided restored base mesh, i.e., the difference in vertex positions between the two meshes. In addition, the base mesh is generated by simplifying and encoding the original mesh through pre-processing. Motion is generated as motion vectors for the mesh of the current frame based on the reference base mesh of the previous frame.

[0187] Fig. 14 illustrates a mesh data receiving device according to embodiments.

[0188] Fig. 14 corresponds to the receiving device (110) or mesh video decoder (113) of Fig. 1, the decoder of Fig. 11 or Fig. 12, and / or the receiving decoding device corresponding thereto. Each component of Fig. 14 corresponds to hardware, software, a processor, and / or a combination thereof. The receiving (decoding) operation of Fig. 14 may follow the reverse process of the corresponding process of the transmitting (encoding) operation of Fig. 13.

[0189] The bitstream of the mesh data received by the receiver (910) is demultiplexed into a compressed motion vector bitstream (e.g., inter decoding) or a base mesh bitstream (e.g., intra decoding), a displacement vector bitstream, and a texture map bitstream after file / segment decapsulation in the demultiplexer (911). For example, if the current mesh has inter-screen encoding (i.e., inter encoding) applied, the motion vector bitstream is received, demultiplexed, and then output to the motion vector decoder (913) via the switching unit (912). As another example, if the current mesh has intra-screen encoding (i.e., intra encoding) applied, the base mesh bitstream is received, demultiplexed, and then output to the static mesh decoder (914) via the switching unit (912). Here, the motion vector decoder (913) may be referred to as a motion decoder.

[0190] According to embodiments, if the current mesh has inter-screen encoding applied according to frame header information, the motion vector decoder (913) can perform decoding on the motion vector bitstream. According to embodiments, the motion vector decoder (913) can reconstruct the final motion vector by adding the previously decoded motion vector as a predictor to the residual motion vector decoded from the bitstream.

[0191] According to embodiments, if the current mesh has been subjected to in-screen encoding according to frame header information, the static mesh decoder (914) can decode the base mesh bitstream to restore connection information, vertex geometry information, texture coordinates, normal information, etc. of the base mesh.

[0192] According to embodiments, the base mesh restoration unit (915) can restore the current base mesh based on the decoded motion vector or the decoded base mesh. For example, if the current mesh has inter-screen encoding applied, the base mesh restoration unit (915) can generate a restored base mesh by adding the decoded motion vector to the reference base mesh and then performing inverse quantization. As another example, if the current mesh has intra-screen encoding applied, the base mesh restoration unit (915) can generate a restored base mesh by performing inverse quantization on the base mesh decoded through the static mesh decoder (914).

[0193] According to embodiments, the displacement vector video decoder (917) can decode the displacement vector bitstream as a video bitstream using a video codec.

[0194] According to embodiments, the displacement vector restoration unit (918) extracts displacement vector transform coefficients from the decoded displacement vector video, and restores the displacement vector by applying inverse quantization and inverse transformation processes to the extracted displacement vector transform coefficients. To this end, the displacement vector restoration unit (918) may include an image unpacking unit, an inverse quantizer, and an inverse linear lifting unit. If the restored displacement vector is a value in a local coordinate system, a process of inversely transforming it into a Cartesian coordinate system may be performed.

[0195] The mesh restoration unit (916) can generate additional vertices by performing subdivision on the restored base mesh. Through subdivision, vertex connection information including the added vertices, texture coordinates, and texture coordinate connection information can be generated. At this time, the mesh restoration unit (916) can generate a final restored mesh (or a restored deformed mesh) by combining the subdivided restored base mesh with the restored displacement vector.

[0196] According to embodiments, the texture map video decoder (919) can decode the texture map bitstream as a video bitstream using a video codec to restore the texture map. The restored texture map has color information for each vertex contained in the restored mesh, and the color value of each vertex can be obtained from the texture map using the texture coordinates of each vertex.

[0197] According to embodiments, the mesh restored by the mesh restoration unit (916) and the texture map restored by the texture map video decoder (919) are shown to the user through a rendering process in the mesh data renderer (920).

[0198] Referring to FIG. 14, a receiving device (decoder) can decode a mesh in an intra-frame or inter-frame manner. A receiving device according to intra-decoding can receive a base mesh, a displacement vector (or referred to as displacement), a texture map (or referred to as attribute map), and render mesh data based on the restored mesh and the restored texture map. A receiving device according to inter-decoding can receive a motion vector (or referred to as motion), a displacement vector (or referred to as displacement), a texture map (or referred to as attribute map), and render mesh data based on the restored mesh and the restored texture map.

[0199] A mesh data transmission device and method according to embodiments may pre-process mesh data, encode the pre-processed mesh data, and transmit a bitstream including the encoded mesh data. A point mesh data reception device and method according to embodiments may receive a bitstream including mesh data and decode the mesh data. The mesh data transmission and reception method / device according to embodiments may be abbreviated as method / device according to embodiments. The mesh data transmission and reception method / device according to embodiments may also be referred to as 3D data transmission and reception method / device or point cloud data transmission and reception method / device.

[0200] As described above, the V-Mesh method can encode displacement data generated during the encoding process through a video codec-based encoder, a zero run-length encoder, an arithmetic encoder, etc. That is, in the displacement vector encoder, the displacement data can be packed into a 2D image and then encoded using a 2D video codec (i.e., a video compression codec), or can be zero run-length encoded, or can be arithmetic encoded to generate a displacement vector video bitstream.

[0201] In the present disclosure, displacement data may be referred to as displacement vector, displacement vector transformation coefficient, displacement information, or geometry data. Additionally, a texture map may be referred to as texture map data, texture data, attribute data, attribute information, or attribute map.

[0202] Currently, the unit type for the displacement data component among the V-DMC (Video-based dynamic mesh coding) components is classified as V3C_GVD. However, V3C_GVD is a type for cases where displacement data is encoded based on a predefined video codec. However, in the case of displacement data, as described above, it can be encoded based on an arithmetic codec in addition to a video codec. However, currently, it is not possible to distinguish this by unit type when displacement data is encoded based on an arithmetic codec. In other words, it is impossible.

[0203] In this disclosure, the V3C unit type (vuh_unit_type) influences the instantiation of a decoder required to correctly decode NAL units carried as the payload of V3C units. Currently, the association between an arithmetic coded (AC) displacement sub-bitstream and a correct decoder is not clearly indicated. That is, a unique association exists between a V3C unit type and a sub-bitstream that requires unique decoding capabilities. Assigning the same V3C unit type to an arithmetic coded (AC) displacement sub-bitstream and a video coded geometry / displacement sub-bitstream breaks this unique association.

[0204] That is, since the AC displacement sub-bitstream requires completely different decoding capabilities from the video coded geometry / displacement sub-bitstream, the sub-bitstream containing the AC displacement data must be explicitly represented separately from the video sub-bitstream. Therefore, it is more efficient to have a separate V3C unit for the AC displacement sub-bitstream, which allows the presence of the AC displacement NAL unit to be clearly indicated in the payload of the V3C unit.

[0205] For this reason, the present disclosure proposes signaling for the case where displacement data, which is one of the components constituting dynamic mesh data, is encoded based on an arithmetic codec.

[0206] The present disclosure proposes a method for signaling by adding a new type to the V3C data format to identify arithmetic-coded displacement data.

[0207] According to embodiments, a dynamic mesh content receiver can decode a bitstream constituting the received dynamic mesh content by dividing it into components. The present disclosure adds a new type (e.g., V3C_ADD) for V-DMC data (e.g., arithmetic-coded displacement data) to the V3C data format defined in the V3C codec and signals it, thereby enabling the receiver to use an appropriate decoding method according to each component constituting the V-DMC.

[0208] That is, the present disclosure will be described in detail below with respect to a method for adding a new data type for arithmetic-coded displacement data, a method for defining displacement sub-bitstream data for arithmetic-coded displacement data, a method for signaling displacement information (also referred to as displacement-related information) by defining it in a V-DMC extension of a V3C parameter set for arithmetic-coded displacement data, a method for decoding arithmetic-coded displacement data, a method for nominal format conversion for reconstruction of arithmetic-coded displacement data, and a method for signaling support for a toolset profile for arithmetic-coded displacement data.

[0209] The V-DMC referred to in this disclosure may also be referred to as V-Mesh, and the terms are used with the same meaning. Dynamic mesh data refers to a type of mesh data, which is a form of point cloud data, and refers to mesh data in which objects and / or people corresponding to objects change over time, i.e., have movement.

[0210] FIG. 15 is a diagram showing an example of a dynamic mesh bitstream structure encoded and transmitted by a transmitting device of the present disclosure. That is, dynamic mesh content can be encoded into a bitstream structure such as FIG. 15 and transmitted to a receiving device.

[0211] In the present disclosure, the dynamic mesh bitstream of FIG. 15 is, as an embodiment, generated and output by one of the encoding devices / encoding methods of FIG. 1, FIG. 2, FIG. 3, FIG. 6, FIG. 7, FIG. 13, or FIG. 25. The dynamic mesh bitstream of the present disclosure can be transmitted / received in either a V3C unit stream format or a V3C sample stream format. The V-DMC bitstream of the present disclosure can follow the V3C bitstream structure defined in the V3C codec specification (ISO / IEC 23090-5) to be described later. At this time, the existing V3C bitstream structure may be followed, but some V3C units may not be used, and some structures for V-DMC only, such as V-DMC extensions, may be followed. The present disclosure describes, as an embodiment, the use of a sample stream data unit used when encoding V3C content of the V3C codec specification (ISO / IEC 23090-5) as shown in FIG. 15.

[0212] A bitstream (referred to as a V-DMC bitstream or dynamic mesh bitstream) transmitted from a transmitting device to a receiving device of the present disclosure may be composed of a sample stream DMC header and a plurality of sample stream DMC units. In the present disclosure, the sample stream DMC header may be referred to as a sample stream header, and the sample stream DMC unit may be referred to as a sample stream data unit.

[0213] At this time, if the sample stream DMC unit follows the V3C codec specification (ISO / IEC 23090-5), each sample stream DMC unit can be composed of V3C sample stream size information and a V3C unit. The V3C unit is again composed of a V3C unit header (V3C_unit_header) and a V3C unit payload (V3C_unit_payload).

[0214] The above V3C sample stream size information specifies the size of the subsequent V3C unit in bytes. At this time, the sample stream DMC header can indicate the accuracy of the V3C sample stream size information in all sample stream DMC units in bytes.

[0215] The above V3C unit header includes type information (vuh_unit_type) indicating the type of data carried by the corresponding V3C unit payload. The V3C unit payload may carry one of a V3C / V-DMC parameter set (VPC), Atlas Data (AD), Base Mesh Data (BMD), Geometry Video Data (GVD), Attribute Video Data (AVD), Packed Video Data (PVD), and Displacement Data (ADD) according to the type information (vuh_unit_type). In the present disclosure, attribute video data is used interchangeably with attribute data and has the same meaning, and geometry video data is used interchangeably with geometry data and has the same meaning.

[0216] Here, VPS may include parameter set information such as decoder configuration information related to mesh encoding / decoding and a sequence header. Atlas data (AD) may include additional information such as 2D mapping or texture mapping for a 3D object. Base mesh data (BMD) is compressed base mesh data for mesh encoding / decoding. In addition, geometry video data (GVD) is displacement data compressed (i.e., encoded) using a video codec, and attribute video data (AVD) is attribute or texture data (or texture map information) compressed (i.e., encoded) using a video codec. In addition, packing video data (PVD) is packing texture map and displacement information compressed (i.e., encoded) based on a video codec, and ADD (or DD) is displacement data compressed (i.e., encoded) based on an arithmetic codec. FIG. 16 is a diagram showing an example of the syntax structure of a V3C unit header (V3C_unit_header()) according to embodiments.

[0217] The V3C unit header of the present disclosure may include vuh_unit_type. The vuh_unit_type is V3C unit type information indicating the type of data carried by the corresponding V3C unit payload.

[0218] Figure 17 shows examples of data types of V3C units assigned to vuh_unit_type according to embodiments.

[0219] Referring to FIG. 17, if the value of vuh_unit_type is 0, it indicates that the data included in the V3C unit payload of the corresponding V3C unit is a V3C parameter set (V3C_VPS), if it is 1, it indicates that it is atlas data (V3C_AD), if it is 2, it indicates that it is occupancy video data (V3C_OVD), if it is 3, it indicates that it is geometry video data (V3C_GVD), if it is 4, it indicates that it is attribute video data (V3C_AVD), if it is 5, it indicates that it is packed video data (V3C_PVD), if it is 6, it indicates that it is common atlas data (V3C_CAD), if it is 7, it indicates that it is base mesh data (V3C_BMD), and if it is 8, it indicates that it is arithmetic-coded displacement data (referred to as V3C_ADD or V3C_DD). Since the meaning, order, deletion, addition, etc. of the values ​​assigned to the vuh_unit_type field in the present disclosure can be easily changed by those skilled in the art, the present disclosure will not be limited to the above embodiment.

[0220] If the bitstream transmitted / received in the present disclosure is a dynamic mesh-based bitstream, the occupancy data whose vhu_unit_type is V3C_OVD is not included in the bitstream as shown in FIG. 15.

[0221] That is, the present disclosure assigns 8 as the value of vuh_unit_type, and if the value of vuh_unit_type is 8, it indicates that it is arithmetic-coded displacement data.

[0222] According to embodiments, depending on the value of the vuh_unit_type field, the V3C unit header may further include additional information as follows.

[0223] For example, if the above vuh_unit_type indicates one of V3C_AVD, V3C_GVD, V3C_OVD, V3C_AD, V3C_CAD, V3C_PVD, V3C_BMD, V3C_ADD, the V3C unit header may further include vuh_v3c_parameter_set_id.

[0224] The above vuh_v3c_parameter_set_id specifies the value of vps_v3c_parameter_set_id for the active V3C VPS.

[0225] If the above vuh_unit_type indicates one of V3C_AVD, V3C_GVD, V3C_OVD, V3C_AD, V3C_PVD, V3C_BMD, V3C_ADD, the V3C unit header may further include vuh_atlas_id.

[0226] The above vuh_atlas_id specifies the identifier (ID) of the atlas corresponding to the current V3C unit.

[0227] The above V3C unit header may further include vuh_attribute_index, vuh_attribute_partition_index, vuh_map_index, and vuh_auxiliary_video_flag if the above vuh_unit_type indicates V3C_AVD, and may further include vuh_map_index, vuh_auxiliary_video_flag, and vuh_reserved_zero_12bits if the above vuh_unit_type indicates V3C_GVD. Additionally, the V3C unit header may further include vuh_reserved_zero_17bits if the vuh_unit_type indicates one of V3C_OVD, V3C_AD, V3C_PVD, V3C_BMD, and V3C_ADD, may further include vuh_reserved_zero_23bits if it indicates V3C_CAD, and otherwise may further include vuh_reserved_zero_27bits.

[0228] The above vuh_attribute_index indicates the index of attribute video data carried as an attribute video data unit.

[0229] The above vuh_attribute_partition_index indicates the index of the attribute dimension group carried as an attribute video data unit.

[0230] The above vuh_map_index, if this field exists, indicates the map index of the current attribute stream.

[0231] If the value of the vuh_auxiliary_video_flag field is 1, it may indicate that the related attribute video data unit contains only raw and / or EOM (Enhanced Occupancy Mode) coded points. As another example, if the value of the vuh_auxiliary_video_flag field is 0, it may indicate that the related attribute video data unit may contain raw and / or EOM coded points. In some embodiments, raw and / or EOM coded points may also be referred to as PCM (Pulse Code Modulation) coded points.

[0232] The above vuh_map_index, if this field exists, indicates the map index of the current geometry stream.

[0233] If the value of the above vuh_auxiliary_video_flag is 1, it may indicate that the related geometry video data unit contains only raw and / or EOM coded points. As another example, if the value of the vuh_auxiliary_video_flag field is 0, it may indicate that the related geometry video data unit may contain raw and / or EOM coded points. In some embodiments, the raw and / or EOM coded points may also be referred to as PCM (Pulse Code Modulation) coded points.

[0234] The above vuh_reserved_zero_12bits, vuh_reserved_zero_17bits, and vuh_reserved_zero_23bits are reserved fields for future use.

[0235] That is, the present disclosure can add a V3C type, i.e., a V3C_ADD or V3C_DD type, as shown in FIG. 17 to identify arithmetic coded displacement data, and can signal by adding a type corresponding to a V3C unit header and / or a V3C unit payload.

[0236] Fig. 18 is a diagram showing an example of the syntax structure of a V3C unit payload (V3C_unit_payload) according to embodiments. In Fig. 18, numBytesInV3CPayload indicates the size of the corresponding V3C unit, which can be specified by the V3C sample stream size information.

[0237] The V3C unit payload of FIG. 18 may include one of the V3C parameter set (v3c_parameter_set(numBytesInV3CPayload)), the atlas sub-bitstream (atlas_sub_bitstream(numBytesInV3CPayload)), the video sub-bitstream (video_sub_bitstream(numBytesInV3CPayload)), the basemesh sub-bitstream (basemesh_sub_bitstream(numBytesInV3CPayload)), and the displacement sub-bitstream (displacement_sub_bitstream(numBytesInV3CPayload)) depending on the vuh_unit_type field value of the corresponding V3C unit header.

[0238] For example, if the vuh_unit_type field indicates a V3C parameter set (V3C_VPS), the V3C unit payload includes a V3C parameter set (v3c_parameter_set(numBytesInV3CPayload)) that contains overall encoding information of the bitstream, and if it indicates atlas data (V3C_AD) or common atlas data (V3C_CAD), it includes an atlas sub bitstream (atlas_sub_bitstream(numBytesInV3CPayload)) that carries the atlas data or common atlas data. And if the vuh_unit_type field indicates accumulation video data (V3C_OVD), the V3C unit payload includes an accumulation video sub-bitstream (video_sub_bitstream(numBytesInV3CPayload)) carrying accumulation video data, if it indicates geometry video data (V3C_GVD), it includes a geometry video sub-bitstream (video_sub_bitstream(numBytesInV3CPayload)) carrying geometry video data, if it indicates attribute video data (V3C_AVD), it includes an attribute video sub-bitstream (video_sub_bitstream(numBytesInV3CPayload)) carrying attribute video data, and if it indicates packed video data (V3C_PVD), it includes a packed video sub-bitstream (video_sub_bitstream(numBytesInV3CPayload)) carrying packed video data. That is, in the present disclosure, packet video data (i.e., texture and displacement data packed into one frame) is transmitted to a receiving device through a V3C unit corresponding to V3C_PVD (vuh_unit_type == V3C_PVD).In addition, if the vuh_unit_type field indicates basemesh data (V3C_BMD), the V3C unit payload includes a basemesh sub-bitstream (basemesh_sub_bitstream(numBytesInV3CPayload)) that carries the basemesh data, and if it indicates displacement data (V3C_ADD), the V3C unit payload includes a displacement sub-bitstream (displacement_sub_bitstream(numBytesInV3CPayload)) that carries the arithmetic-coded displacement data. In the present disclosure, attribute video sub-bitstream is used interchangeably with attribute sub-bitstream, attribute video bitstream, and attribute bitstream with the same meaning. Geometry video sub-bitstream is used interchangeably with geometry sub-bitstream, geometry video bitstream, and geometry bitstream with the same meaning.

[0239] The displacement_sub_bitstream(numBytesInV3CPayload) contains a portion of an arithmetic-encoded displacement unit stream of size numBytes as an ordered stream of bytes or bits, within which the positions of unit boundaries can be identified through patterns in the data. The format of this displacement unit stream can be identified by a 4CC code defined in ptl_profile_codec_group_idc or a component codec mapping SEI (Supplemental Enhancement Information) message.

[0240] And, as shown in FIGS. 16 to 18, V3C_ADD added to the V3C unit type of the V3C unit header can be defined as follows. That is, the basemesh and displacement data unit payloads, that is, the basemesh unit payload that carries the basemesh and the displacement data unit payload that carries the arithmetic-coded displacement data, correspond to the basemesh data unit (e.g., the NAL unit defined in ISO / IEC 23090-29) and the arithmetic-coded (AC) displacement data unit (e.g., the NAL unit defined in ISO / IEC 23090-29), respectively. And, the basemesh and displacement data unit payloads can be decoded by an appropriate decoder specified by profiles. In addition, the basemesh and displacement data unit payloads, if present, can be decoded by an appropriate decoder specified by the component codec mapping SEI message.

[0241] FIG. 19 is a diagram showing an example of a DMC (or V3C) bitstream structure according to embodiments, and is an example of a V3C bitstream in a V3C unit stream format.

[0242] In Fig. 19, the V3C bitstream can be composed of V3C sample stream precision information and multiple sample stream V3C units. In addition, each sample stream V3C unit is composed of V3C sample stream size information and a V3C unit. The V3C unit is again composed of a V3C unit header (V3C_unit_header) and a V3C unit payload (V3C_unit_payload). The V3C sample stream precision information can represent the precision of the V3C sample stream size information in all sample stream V3C units in bytes.

[0243] The above V3C sample stream size information specifies the size of the subsequent V3C unit in bytes.

[0244] The above V3C unit header includes type information (vuh_unit_type) indicating the type of data carried by the corresponding V3C unit payload. The V3C unit payload carries one of V3C parameter set data, atlas data, occupancy video data, geometry video data (i.e., video coded displacement data), attribute video data, common atlas data, packed video data, base mesh data, or arithmetic coded displacement data, depending on the type information (vuh_unit_type). For example, if the type information (vuh_unit_type) of the above V3C unit header indicates a V3C parameter set (V3C_VPS), the V3C unit payload includes a V3C parameter set (v3c_parameter_set()) that includes overall encoding information of the bitstream, and if it indicates atlas data (V3C_AD), it includes an atlas sub bitstream (atlas_sub_bitstream()) that carries the atlas data. And, if the type information (vuh_unit_type) of the V3C unit header indicates accumulator video data (V3C_OVD), the V3C unit payload includes a video sub-bitstream (video_sub_bitstream()) carrying the accumulator video data, if it indicates geometry video data (V3C_GVD, i.e., displacement data encoded with a video codec), the V3C unit payload includes a video sub-bitstream (video_sub_bitstream()) carrying the geometry video data, and if it indicates attribute video data (V3C_AVD), the V3C unit payload includes a video sub-bitstream (video_sub_bitstream()) carrying the attribute video data.In addition, if the type information (vuh_unit_type) of the V3C unit header indicates packed video data (V3C_PVD), the V3C unit payload may include a video sub-bitstream (video_sub_bitstream()) carrying packed video data, if it indicates base mesh data (V3C_BMD), the V3C unit payload may include a base mesh sub-bitstream (basemesh_sub_bitstream()) carrying base mesh data, and if it indicates displacement data (V3C_ADD), the V3C unit payload may include a displacement sub-bitstream (displacement_sub_bitstream()) carrying arithmetic-encoded displacement data (Displacements Data).

[0245] Here, the V3C parameter set (V3C_VPS) may include parameter set information such as decoder configuration information related to mesh encoding / decoding and a sequence header. The atlas data (V3C_AD) may include additional information such as 2D mapping or texture mapping for a 3D object. In addition, the geometry video data (V3C_GVD) is displacement data compressed based on a video codec, and the attribute video data (V3C_AVD) is attribute or texture data compressed by a video codec. In addition, the base mesh data (V3C_BMD) is compressed base mesh information for mesh encoding / decoding, and the packed video data is data in which a texture map and displacement information are packed into one frame and compressed. In addition, the displacement data (V3C_ADD) is displacement data compressed based on an arithmetic codec.

[0246] As in Fig. 19, if the type information (vuh_unit_type) of the V3C unit header indicates atlas data (V3C_AD), the corresponding V3C unit payload includes an atlas sub bitstream (atlas_sub_bitstream()) carrying the atlas data.

[0247] At this time, the atlas sub bitstream (atlas_sub_bitstream()) carries the atlas data in NAL sample stream format.

[0248] In the present disclosure, the NAL sample stream carrying atlas data is composed of NAL unit size precision information and multiple sample stream NAL units, similar to the V3C sample stream. Each sample stream NAL unit is composed of NAL unit size information and a NAL unit. The NAL unit is further composed of a NAL unit header (NAL_unit_header) and a NAL unit payload (NAL_unit_payload).

[0249] The above NAL unit size precision information can indicate the precision of NAL unit size information in all sample stream NAL units in bytes.

[0250] The above NAL unit size information specifies the size of the subsequent NAL unit in bytes.

[0251] The above NAL unit header includes type information indicating the type of data carried by the corresponding NAL unit payload. The NAL unit payload carries one of ASPS (atlas sequence parameter set), AAPS (atlas adaptation parameter set), AFPS (atlas frame parameter set), prefix essential SEI message, atlas tile layer information, and suffix essential SEI message according to the type information of the NAL unit header. In the present disclosure, the prefix essential SEI message and the suffix essential SEI message are transmitted via sel_rbsp(). Here, rbsp (Raw Byte Sequence Payload) is specified as a sequential sequence of bytes.

[0252] In the present disclosure, if the type information of the NAL unit header indicates NAL_PREFIX_ESEI or NAL_SUFFIX_ESEI, the corresponding NAL unit payload can carry component codec mapping information (e.g., component codec mapping information) through sel_rbsp().

[0253] The following describes a method for signaling displacement-related information (also called displacement information) by defining it in the V-DMC extension of the V3C parameter set for arithmetic coded displacement data of the present disclosure.

[0254] According to embodiments, the present disclosure may include vps_vdmc_extension() as an extension of v3c_parameter_set.

[0255] Fig. 20 is a diagram showing an example of a syntax structure of vps_vdmc_extension() according to embodiments. In Fig. 20, vps_vdmc_extension() may be repeated as many times as the number of atlases (i.e., vps_atlas_count_minus1 plus 1) and may include the following information. In the present disclosure, vps_atlas_count_minus1 is included in v3c_parameter_set, and 1 is added to vps_atlas_count_minus1 to indicate the total number of atlases supported in the current bitstream. v3c_parameter_set may further include vps_atlas_id[ k ]. vps_atlas_id[ k ] specifies the ID of the atlas with index k.

[0256] That is, vps_vdmc_extension() can include vps_ext_bmesh_data_substream_codec_id[j], vps_ext_bmesh_geometry_3d_bit_depth_minus1[j], vps_ext_bmesh_geometry_3d_msb_align_flag[j], and vps_ext_bmesh_data_attribute_count[j]. there is.

[0257] The above vps_ext_bmesh_data_substream_codec_id[j] indicates the identifier of the codec used to decode basemesh data for the atlas with atlas ID j. This codec can be identified through profiles, component codec mapping SEI messages, or external means.

[0258] The value obtained by adding 1 to the above vps_ext_bmesh_geometry_3d_bit_depth_minus1[j] represents the bit depth of the vertex coordinates in the base mesh for the atlas whose atlas ID is j.

[0259] The above vps_ext_bmesh_geometry_3d_msb_align_flag[j] indicates how the decoded vertex coordinate samples in the basemesh are converted to vertex coordinate samples at the nominal bit depth for the atlas with atlas ID j.

[0260] The above vps_ext_bmesh_data_attribute_count[j] indicates the number of attributes in the base mesh for the atlas whose atlas ID is j.

[0261] The above vps_vdmc_extension() may be repeated as many times as the above vps_ext_bmesh_data_attribute_count[j] and may further include vps_ext_bmesh_attribute_index[ j ][ i ], vps_ext_bmesh_attribute_bit_depth_minus1[ j ][ i ], vps_ext_bmesh_attribute_msb_align_flag[ j ][ i ], and vps_ext_bmesh_attribute_type[ j ][ i ].

[0262] The above vps_ext_bmesh_attribute_index[ j ][ i ] represents the index of the basemesh attribute carried in the basemesh bitstream for the i-th attribute of the atlas whose atlas ID is j.

[0263] The value obtained by adding 1 to the above vps_ext_bmesh_attribute_bit_depth_minus1[ j ][ i ] represents the bit depth of the ith attribute in the base mesh for the atlas whose atlas ID is j.

[0264] The above vps_ext_bmesh_attribute_msb_align_flag[ j ][ i ] indicates how the decoded attribute samples in the basemesh are converted to attribute samples at the nominal attribute bit depth for the atlas with atlas ID j and attribute with index i.

[0265] The above vps_ext_bmesh_attribute_type[ j ][ i ] indicates the attribute type of the ith attribute signaled through the basemesh for the atlas with atlas ID j.

[0266] According to embodiments, vps_vdmc_extension() may further include displacement-related information. In the present disclosure, whether displacement-related information is included may be determined based on the value of vps_ac_displacement_present_flag[ j ].

[0267] If the value of the above vps_ac_displacement_present_flag[ j ] is 0, it indicates that the atlas whose atlas ID is j is not associated with arithmetic encoded displacement data. That is, if the value of the above vps_ac_displacement_present_flag[ j ] is 0, it indicates that the atlas whose atlas ID is j does not have arithmetic encoded displacement data associated with it. If the value of the above vps_ac_displacement_present_flag[ j ] is 1, it indicates that the atlas whose atlas ID is j should be associated with arithmetic encoded displacement data. That is, if the value of the above vps_ac_displacement_present_flag[ j ] is 1, it indicates that the atlas whose atlas ID is j has arithmetic encoded displacement data associated with it.

[0268] According to embodiments, the vps_vdmc_extension() may further include displacement-related information (displacement_information(j)) if the value of vps_ac_displacement_present_flag[ j ] is 1. A detailed description of displacement_information( j ) will be provided later with reference to FIG. 21.

[0269] According to embodiments, vps_vdmc_extension() may further include vps_ext_attribute_frame_width[ j ][ i ] and vps_ext_attribute_frame_height[ j ][ i ], repeating as many times as the value of ai_attribute_count[ j ].

[0270] The above ai_attribute_count[ j ] represents the number of attributes related to the atlas whose atlas ID is j.

[0271] The above vps_ext_attribute_frame_width[ j ][ i ] represents the width of the atlas attribute frame in integer luma sample units for the atlas whose atlas ID is j.

[0272] The above vps_ext_attribute_frame_height[ j ][ i ] represents the height of the atlas attribute frame in integer luma sample units for the atlas whose atlas ID is j.

[0273] The present disclosure has the following bitstream standard compliance requirements, and is divided into Option A and Option B. Option A is a bitstream standard compliance requirement, in which, for an atlas whose vps_geometry_video_present_flag[ j ] value is 1 or whose pin_geometry_present_flag[ j ] value is 1, the vps_ac_displacement_present_flag[ j ] value must be 0 for an atlas with the same atlas ID j. Option B is a bitstream standard compliance requirement, in which the sum of the vps_geometry_video_present_flag[ j ], pin_geometry_present_flag[ j ], and vps_ac_displacement_present_flag[ j ] values ​​cannot exceed 1, and this is applied to an atlas with an atlas ID j.

[0274] The above vps_geometry_video_present_flag[ j ] is information included in v3c_parameter_set. If the value of the above vps_geometry_video_present_flag[ j ] is 0, it indicates that the atlas with atlas ID j does not have geometry video data associated with it, and if it is 1, it indicates that the atlas with atlas ID j has geometry video data associated with it. If this vps_geometry_video_present_flag[ j ] does not exist, its value is inferred to be 1.

[0275] The above pin_geometry_present_flag[ j ] is information belonging to packing information included in the extension of v3c_parameter_set. If the value of pin_geometry_present_flag[ j ] is 0, it indicates that the packed video frames of the atlas whose atlas ID is j do not include areas having geometry data, and if it is 1, it indicates that the packed video frames of the atlas whose atlas ID is j include areas having geometry data. If pin_geometry_present_flag[ j ] does not exist, its value is inferred to be 0.

[0276] At this time, as a bitstream standard compliance requirement, if the value of vps_geometry_video_present_flag[ j ] is 1 for the atlas with atlas ID j, the value of pin_geometry_present_flag[ j ] must be 0 for the same atlas ID j. Conversely, if the value of pin_geometry_present_flag[ j ] is 1 for the atlas with atlas ID j, the value of vps_geometry_video_present_flag[ j ] must be 0 for the same atlas ID j.

[0277] FIG. 21 is a diagram showing an example of a syntax structure of displacement-related information (displacement_information(atlasID)) according to embodiments.

[0278] That is, the V3C parameter set (v3c_parameter_set()) may include vps_vdmc_extension(), and vps_vdmc_extension() may include displacement_information(atlasID) when vps_ac_displacement_present_flag[j] is true (i.e., indicating that the corresponding atlas should be associated with arithmetic-encoded displacement data) while repeating as many times as the value of vps_atlas_count_minus1 plus 1. For example, when vps_ac_displacement_present_flag[j] with atlas ID j is true, displacement information (displacement_information(atlasID)) corresponding to atlas ID j is included. The present disclosure may refer to displacement_information(atlasID) as displacement-related information.

[0279] displacement_information(atlasID) in Fig. 21 may include di_disp_dimension[atlasID], di_disp_count_per_frame[atlasID], and di_disp_msb_align_flag[atlasID].

[0280] The value of di_disp_dimension [j] plus 1 indicates the nominal dimension of displacement to which all arithmetic coded displacement data for the atlas with atlas ID j shall be converted to.

[0281] The above di_disp_count_per_frame [j] value indicates the nominal number of displacement values ​​to which all arithmetic encoded displacement data for the atlas with atlas ID j shall be converted.

[0282] The above di_disp_msb_align_flag [j] value indicates how the decoded displacement vector samples in the arithmetic encoded displacement data for the atlas with atlas ID j shall be converted to.

[0283] The above displacement_information(atlasID) may further include di_codec_id[ j ] and di_3d_bit_depth_minus1[ j ].

[0284] The above di_codec_id[ j ] represents the mapping index of the codec identifier of the arithmetic codec used to compress displacement information for the atlas whose atlas ID is j.

[0285] The value obtained by adding 1 to the above di_3d_bit_depth_minus1[ j ] represents the nominal bit depth of the 3D displacement coefficients to which all arithmetic encoded displacement data will be converted for the atlas whose atlas ID is j.

[0286] The following describes an additional decoding process for arithmetic-encoded displacement data. This decoding process is, in one embodiment, performed in the mesh data decoding module of FIG. 23, the AC displacement decoder (15014) of FIG. 24, and / or the step of decoding displacement in the bitstream (S32012) of FIG. 26. Furthermore, this decoding process may also be performed in the dynamic mesh video decoder (113) of FIG. 1 or the displacement vector video decoder (917) of FIG. 14.

[0287] The following additional decoding process is invoked for each associated V3C sub-bitstream component.

[0288] That is, the decoding process of a V3C sub-bitstream component corresponding to a basemesh component can be determined by checking whether vuh_unit_type matches V3C_BMD, or by external means if there is no V3C unit header.

[0289] Additionally, the decoding process of a V3C sub-bitstream component corresponding to an arithmetic-encoded displacement component can be determined by checking whether vuh_unit_type matches V3C_ADD, or by external means if there is no V3C unit header.

[0290] The following describes a nominal format conversion method for reconstruction of arithmetic-encoded displacement data. That is, the present disclosure adds the following arithmetic-encoded displacement nominal format conversion content.

[0291] More specifically, this process converts the decoded displacement frames, DecDispFrames, into a nominal format. At this time, the variables dispNumComp, dispHeight, dispWidth, vertNumComp, and vertMSBAlignFlag can be set as follows. Here, ConvAtlasID can be the atlas identifier associated with the nominal format conversion.

[0292] That is, the variable dispNumComp is a variable that sets the dimension of displacement data, and in one embodiment, the value of di_disp_dimension[ConvAtlasID] is assigned (dispNumComp = di_disp_dimension[ConvAtlasID]).

[0293] The variable dispHeight is a variable that sets the height of displacement data (e.g., the height of the displacement frame containing the displacement data), and is fixed to 1 in one embodiment (disHeight=1).

[0294] The variable dispWidth is a variable that sets the width of the displacement data (e.g., the width of the displacement frame containing the displacement data), and in one embodiment, the value of di_disp_count_per_frame[ConvAtlasID] is assigned (dispWidth = di_disp_count_per_frame[ConvAtlasID]).

[0295] The variable vertNumComp is a variable that sets the number of components, and is fixed to 3 in one embodiment (vertNumComp = 3).

[0296] The variable vertMSBAlignFlag is a variable that sets whether the most significant bit is aligned. In one embodiment, the value of di_disp_msb_align_flag[ConvAtlasID] is assigned (vertMSBAlignFlag = di_disp_msb_align_flag[ConvAtlasID]).

[0297] The output of this process is a 2D array of structures called DispFramesNF representing the displacement frames decoded in nominal format, where the dimension corresponds to the displacement composition time index.

[0298] At this time, to convert the decoded displacement into a nominal format, several processing steps can be performed depending on the original format of the decoded displacement. In particular, decDispFramesNBD[frameIdx] is initialized to DecDispFrames[frameIdx].

[0299] Additionally, to convert the bit depth of the decoded displacement frames to the required nominal bit depth, the present disclosure is called for each frame index frameIdx, taking as input the variables DecDispBitDepth, totalVertCount, vertNumComp, VerCoordsBitDepthNF, vertMSBAlignFlag of the decoded displacement frame DecDispFrames[frameIdx] and the array DecDispFrame of the decoded displacement frame DecDispFrames[frameIdx]. The output of this process is the displacement frame in the nominal bit depth, i.e., the DecDispFrame of decDispFramesNBD[frameIdx]. The variable DecDispBitDepth is updated to VerCoordsBitDepthNF.

[0300] Additionally, to identify missing frames in the sequence and reorder the input, this process takes as input the variables DecDispFrameCount, dispNumComp, dispHeight, dispWidth and the arrays decDispFramesNBD, DecDispOutOrdIdx, DecDispCompTime. The output of this process is the number of frames to be output numOutOrdDispFrames, the decoded displacement frames sorted by their output order index decDispFramesONF, a 1D array framePresent indicating whether a frame is present or not, and a 1D array decDispCompTimeONF indicating the composition time of each frame sorted by its output order index.

[0301] And, to align the composition time of the decoded displacement frame with the composition time of the decoded atlas frame in the output order, the variables NumDecAtlasFrames, numOutOrdDispFrames, dispNumComp, dispHeight, dispWidth and the arrays DecAtlasOutOrdIdx, DecAtlasCompTime, decDispFramesONF, decDispCompTimeONF, framePresent are used as input. The output of this process is the number of frames to be output rangeCompTimeIdx, the output composition time of each frame dispCompTime, and the frame sequence DispFramesNF in the nominal format aligned with the decoded atlas composition times.

[0302] The following describes how to signal support for a toolset profile for arithmetic-encoded displacement data. This describes the V-DMC Main and V-DMC Main Still toolset profile components added in this disclosure.

[0303] According to embodiments, the V-DMC toolset profile components represent a V-DMC Main (ptl_profile_toolset_idc = 128) or a V-DMC Main Still (ptl_profile_toolset_idc = 128 and ptc_one_v3c_frame_only_flag = 1) toolset profile component, which adheres to the syntax element restrictions specified in FIG. 22.

[0304] FIG. 22 is a diagram illustrating examples of allowed syntax element values ​​for V-DMC toolset profile components according to embodiments. If a syntax element is not mentioned in FIG. 22, the component is not restricted by the toolset profile component.

[0305] In the present disclosure, vps_ac_displacement_present_flag can be used as a syntax element representing a V-DMC toolset profile component, as shown in FIG. 22. If a specific profile does not allow arithmetic coding (AC) displacement data, the profile setting can be enabled by setting the syntax element vps_ac_displacement_present_flag to 0.

[0306] FIG. 23 is a diagram showing another example of a receiving device according to embodiments. The receiving device of FIG. 23 may be referred to as a dynamic mesh content receiving device. The receiving device of FIG. 23 may correspond to the receiving device of FIG. 1, the receiving device of FIG. 11, the receiving device of FIG. 12, or the receiving device of FIG. 14. Therefore, parts not described in FIG. 23 will refer to the description of the receiving device of FIG. 1, the receiving device of FIG. 11, the receiving device of FIG. 12, or the receiving device of FIG. 14. The elements of the receiving device illustrated in FIG. 23 may be implemented by hardware, software, a processor connected to a memory, and / or a combination thereof. That is, the elements of the receiving device of FIG. 23 may be implemented by hardware, software, firmware, or a combination thereof, including one or more processors or integrated circuits configured to communicate with one or more memories, although not illustrated in the drawing. One or more processors may perform at least one of the operations and / or functions of the elements of the receiving device of FIG. 23 described above. Furthermore, one or more processors may operate or execute a set of software programs and / or instructions for performing the operations and / or functions of the elements of the receiving device of FIG. 23.

[0307] According to embodiments, a bitstream of mesh data (also called a V3C bitstream) encapsulated in a file at a transmitting device and delivered to a receiving device through a delivery module is decapsulated in a file decapsulation module. If the bitstream of mesh data is not encapsulated in a file format at the transmitting device, the decapsulation process at the receiving device is omitted. In the present disclosure, a base mesh sub-bitstream may be referred to as a base mesh bitstream, an atlas sub-bitstream may be referred to as an atlas bitstream, an attribute sub-bitstream may be referred to as an attribute bitstream, a geometry sub-bitstream may be referred to as a geometry bitstream, a PVD video sub-bitstream may be referred to as a PVD video bitstream, and a displacement sub-bitstream may be referred to as a displacement bitstream.

[0308] That is, the stored and / or received dynamic mesh content passes through the delivery module and the file decapsulation module and can be in a form similar to the bitstream structure of FIG. 15 (i.e., dynamic mesh bitstream or V3C bitstream).

[0309] These bitstreams are provided to the mesh data decoding module of Fig. 23.

[0310] Figure 24 shows an example of a detailed block diagram of a mesh decoding module according to embodiments.

[0311] According to the embodiment, the demultiplexer (15011) of the mesh decoding module can separate an atlas sub-bitstream, a base mesh sub-bitstream, an arithmetic-encoded displacement sub-bitstream, an attribute sub-bitstream, etc. from an input bitstream using the type information (vuh_unit_type) included in the V3C unit header and provide them to each decoder. In the present disclosure, the demultiplexer (15011) can include a bitstream parser that parses a dynamic mesh bitstream. That is, the bitstream parser can parse the V3C unit header and the V3C unit payload constituting the bitstream, and collect each sub-bitstream by unit type defined in FIG. 16. For example, if the V3C unit type included in the V3C unit header is V3C_ADD (or V3C_DD), it can be known that displacement data has been received encoded based on an arithmetic codec, and a corresponding decoding method can be used.

[0312] In the present disclosure, an atlas sub-bitstream is decoded in an atlas decoder (15012), a base mesh sub-bitstream is decoded in a base mesh decoder (15013), an arithmetic-encoded displacement sub-bitstream is decoded in an AC displacement decoder (15014), and an attribute sub-bitstream is decoded in a video decoder (15015). If displacement data is received encoded based on a video codec, it is decoded in the video decoder (15015). Additionally, if packed video data is received encoded, it can be decoded in the video decoder (15015). In the present disclosure, atlas data, basemesh, arithmetic-coded displacement data, attribute data coded with a video codec, geometry data coded with a video codec, or packed video data coded with a video codec may each be carried as a V3C unit payload or may be carried as a NAL unit (NALU).

[0313] More specifically, the atlas decoder (15012) can decode the atlas sub-bitstream to restore the atlas data, and provide the restored atlas data to necessary blocks, for example, a base mesh processing unit (15013), a displacement processing unit (15014), and a mesh restoration unit (15018).

[0314] The above base mesh decoder (15013) can restore the final motion vector by adding the residual motion vector decoded from the base mesh sub-bitstream to the previously decoded motion vector as a predictor if the current mesh has inter-screen encoding applied. The above base mesh decoder (15013) can restore the connection information, vertex geometry information, texture coordinates, normal information, etc. of the base mesh by statically decoding the base mesh sub-bitstream if the current mesh has intra-screen encoding applied.

[0315] The base mesh processing unit (15016) restores the current base mesh based on the decoded motion vector or the decoded base mesh, and provides the restored current base mesh to the mesh restoration unit (15018). For example, if the current mesh has inter-screen encoding applied, the base mesh processing unit (15016) can generate a restored base mesh by adding the decoded motion vector to the reference base mesh and then performing inverse quantization. As another example, if the current mesh has intra-screen encoding applied, the base mesh processing unit (15016) can generate a restored base mesh by performing inverse quantization on the decoded base mesh.

[0316] The above AC displacement decoder (15014) restores displacement data by arithmetic decoding the displacement sub-bitstream, and provides the restored displacement data to the displacement processing unit (15017).

[0317] The present disclosure can collect displacement-related information by parsing displacement_information() of FIG. 21, which is signaled by the vps_ac_displacement_present_flag value (true or 1) in the V-DMC extension (i.e., vps_vdmc_extension()) of the V3C parameter set of FIG. 20, when arithmetic coded displacement data is received. This process may be performed in a bitstream parser, an AC displacement decoder (15014), or a separate metadata parsing unit (not shown).

[0318] Displacement-related information parsed from displacement_information() of Fig. 21 may include at least one of di_disp_dimension[atlasID], di_disp_count_per_frame[atlasID], di_disp_msb_align_flag[atlasID], di_codec_id[atlasID], or di_3d_bit_depth_minus1[atlasID]. For a detailed description of each field, refer to the description of Fig. 21. In addition, the additional decoding process performed in the AC displacement decoder (15014) using this displacement-related information (i.e., di_disp_dimension[atlasID], di_disp_count_per_frame[atlasID], di_disp_msb_align_flag[atlasID]), di_codec_id[atlasID], or di_3d_bit_depth_minus1[atlasID]) has been described in detail above, so a detailed description thereof will be omitted here.

[0319] The video decoder (15015) decodes the attribute sub-bitstream based on a video codec to restore attribute data and provides the restored attribute data to the reconstruction unit (15019). In addition, the video decoder (15015) can decode the geometry sub-bitstream based on a video codec to restore geometry data and provide the restored geometry data to the reconstruction unit (15019). In addition, the video decoder (15015) can separate displacement data and attribute data (or texture map) encoded by the video codec from the packed video sub-bitstream and decode them respectively to restore the geometry data and the attribute data. At this time, the restored geometry data can be provided to the displacement processing unit (15017), and the restored attribute data can be provided to the reconstruction unit (15019).

[0320] The displacement processing unit (15017) can perform a reverse transformation of the coordinate system of the restored displacement data. For example, if the restored displacement data is a value of the local coordinate system, a process of reverse transformation to the Cartesian coordinate system can be performed.

[0321] The above mesh restoration unit (15018) can restore the final restored mesh by combining the restored current base mesh and the restored displacement data based on the restored atlas data. At this time, the mesh restoration unit (15018) can perform subdivision on the restored current base mesh to generate additional vertices. Through subdivision, vertex connection information including the added vertices, texture coordinates, and texture coordinate connection information, etc. can be generated. In addition, the mesh restoration unit (15018) can combine the subdivided restored base mesh with the restored displacement data to generate the final restored mesh (or referred to as a restored deformed mesh).

[0322] The above reconstruction unit (15019) can combine the restored mesh and the restored attribute video data to provide the reconstructed dynamic mesh data to the mesh data processing / rendering module of FIG. 23.

[0323] The above reconstructed dynamic mesh data can be rendered through a process such as post-processing in the mesh data processing / rendering module.

[0324] Fig. 25 is a flowchart illustrating an example of an encoding method according to embodiments. The encoding method according to embodiments may include a step of encoding a base mesh of mesh data (S31011), a step of encoding a displacement of mesh data (S31012), and a step of encoding an attribute of mesh data (S31013).

[0325] In the step (S31011) of encoding the base mesh of the above mesh data, if intra encoding or intra frame encoding is performed on the corresponding mesh frame, the base mesh can be encoded through a static mesh encoder. In this case, encoding can be performed on connection information, vertex geometry information, vertex texture information, normal information, etc. of the base mesh. In the step (S31011) of encoding the base mesh of the above mesh data, if inter encoding or inter frame encoding is performed on the corresponding mesh frame, a motion vector encoder can calculate a motion vector between the base mesh and the reference restored base mesh (or the restored quantized reference base mesh) and encode the value. In addition, connection information-based prediction can be performed using a previously encoded / decoded motion vector as a predictor, and a residual motion vector obtained by subtracting the predicted motion vector from the current motion vector can be encoded. At this time, the V3C unit type (vuh_unit_type) included in the V3C unit header of the V3C unit carrying the encoded base mesh is assigned V3C_BMD (i.e., 7).

[0326] The step (S31012) of encoding the displacement of the above mesh data may perform video codec-based encoding on the displacement data or arithmetic codec-based encoding. At this time, the V3C unit type (vuh_unit_type) included in the V3C unit header of the V3C unit that carries the displacement data encoded based on the video codec is assigned V3C_GVD (i.e., 3), and the V3C unit type (vuh_unit_type) included in the V3C unit header of the V3C unit that carries the displacement data encoded based on the arithmetic codec is assigned V3C_ADD or V3C_DD (i.e., 8). In addition, the step of encoding the displacement of the mesh data (S31012) may convert the coordinate system of the displacement data from the 3D Cartesian coordinate system to the Local coordinate system before encoding the displacement data. If the displacement data is encoded based on the arithmetic codec, displacement-related information such as FIGS. 20 and 21 may be signaled in the V3C parameter set and transmitted to the receiving device. A detailed description of the displacement-related information is omitted here since it has been described in FIGS. 20 and 21.

[0327] The step (S31013) of encoding the attributes of the above mesh data may perform encoding based on a video codec for the attribute data (or texture map). At this time, the V3C unit type (vuh_unit_type) included in the V3C unit header of the V3C unit carrying the attribute data encoded based on the video codec is assigned V3C_AVD (i.e., 4).

[0328] The above encoding method may further include a step of transmitting a bitstream including an encoded base mesh, encoded displacement data, and encoded attribute data. The bitstream transmitted in the above step may be transmitted to a receiving device in a structure as shown in FIG. 15.

[0329] The encoding method of the present disclosure can be performed by an encoding device (encoder). The encoding device includes a memory and at least one processor connected to the memory, and the at least one processor can be configured to encode a base mesh of mesh data, encode a displacement of the mesh data, and encode an attribute of the mesh data.

[0330] Embodiments further include a computer-readable storage medium storing a bitstream generated by the method according to FIG. 25.

[0331] Embodiments further include a method comprising the steps of obtaining a bitstream for mesh data, the bitstream being generated based on the steps of encoding a basemesh of the mesh data, encoding a displacement of the mesh data, and encoding an attribute of the mesh data, and transmitting data including the bitstream.

[0332] Fig. 26 is a flowchart showing an example of a decoding method according to embodiments. The decoding method according to embodiments may include a step of decoding a base mesh in a bitstream (S32011), a step of decoding a displacement in the bitstream (S32012), and a step of decoding an attribute in the bitstream (S32013). The decoding step of Fig. 26 may further include a step of receiving a bitstream including a base mesh, displacement data, and attribute data, or a file in which a bitstream is encapsulated. When the file is received, the receiving step performs a decapsulation process to extract a bitstream, and when the bitstream is received, the decapsulation process is omitted. The bitstream is referred to as a bitstream of mesh data or a V3C bitstream, and has a structure as shown in Fig. 15. The above receiving step can separate an atlas sub-bitstream, a basemesh sub-bitstream, a displacement sub-bitstream, and an attribute sub-bitstream (or attribute video sub-bitstream) from the bitstream according to the type information of the V3C unit header.

[0333] The step (S32011) of decoding the base mesh in the bitstream may, if the current mesh has been subjected to inter-screen encoding, use a previously decoded motion vector as a predictor to restore a final motion vector by adding it to a residual motion vector decoded from the base mesh sub-bitstream. The step (S32011) of decoding the base mesh in the bitstream may, if the current mesh has been subjected to intra-screen encoding, statically decode the base mesh sub-bitstream to restore connection information, vertex geometry information, texture coordinates, normal information, etc. of the base mesh.

[0334] The step (S32012) of decoding displacement in the bitstream performs decoding on the displacement sub-bitstream based on the video codec if the displacement data is encoded based on the video codec, and performs decoding based on the arithmetic codec if the displacement data is encoded based on the arithmetic codec. For example, if the V3C unit type included in the V3C unit header is V3C_ADD (or V3C_DD), it can be known that the displacement data was received encoded based on the arithmetic codec, and if the V3C unit type is V3C_GVD, it can be known that the displacement data was received encoded based on the video codec. In addition, a decoding method suitable for the encoding method can be used.

[0335] The present disclosure can collect displacement-related information by parsing displacement_information() of FIG. 21, which is signaled by the vps_ac_displacement_present_flag value (true or 1) in the V-DMC extension (i.e., vps_vdmc_extension()) of the V3C parameter set of FIG. 20, when arithmetic coded displacement data is received.

[0336] Displacement-related information parsed from displacement_information() of Fig. 21 may include at least one of di_disp_dimension[atlasID], di_disp_count_per_frame[atlasID], di_disp_msb_align_flag[atlasID], di_codec_id[atlasID], or di_3d_bit_depth_minus1[atlasID]. For a detailed description of each field, refer to the description of Fig. 21. In addition, the additional decoding process performed in the step (S32012) of decoding the displacement in the bitstream using the displacement-related information (i.e., di_disp_dimension[atlasID], di_disp_count_per_frame[atlasID], di_disp_msb_align_flag[atlasID], di_codec_id[atlasID], or di_3d_bit_depth_minus1[atlasID]) has been described in detail above, so a detailed description thereof will be omitted here.

[0337] The step of decoding an attribute in the above bitstream (S32013) restores the attribute data by decoding the attribute sub-bitstream based on a video codec.

[0338] The base mesh, displacement data, and attribute data decoded in steps S32011-S32013 can be rendered after undergoing post-processing such as mesh restoration and reconstruction.

[0339] The decoding method of the present disclosure can be performed by a decoding device (decoder). The decoding device includes a memory and at least one processor connected to the memory, and the at least one processor can be configured to decode a basemesh within a bitstream, decode a displacement within the bitstream, and decode an attribute within the bitstream.

[0340] As explained so far, the method proposed in this disclosure extends the existing V3C unit for data constituting dynamic mesh content as shown in FIGS. 15 to 19, so that each component can be signaled separately. Among these, displacement data can be encoded / decoded using video coding and / or arithmetic coding depending on the encoding method, and the distinction between the two codecs is signaled by distinguishing it by the V3C unit type, so that the receiver can effectively recognize it.

[0341] And, by signaling displacement-related information related to arithmetic-coded displacement data in the V-DMC extension of the V3C parameter set as in FIGS. 20 and 21, the receiver can effectively decode the arithmetic-coded displacement data.

[0342] Additionally, by adding vps_ac_displacement_present_flag to the V-DMC profile as shown in Fig. 22, the receiver can determine whether arithmetic coded displacement data is included through profile information.

[0343] Each of the parts, modules, or units described above may be software, processors, or hardware parts that execute sequential execution processes stored in memory (or storage units). Each of the steps described in the embodiments described above may be performed by processors, software, or hardware parts. Each of the modules / blocks / units described in the embodiments described above may operate as a processor, software, or hardware. In addition, the methods presented in the embodiments may be implemented as code. This code may be written on a processor-readable storage medium and thus may be read by a processor provided by an apparatus.

[0344] Furthermore, throughout the specification, when a part is said to "include" a component, this does not exclude other components, unless otherwise specifically stated, but rather implies the inclusion of other components. Furthermore, terms such as "part" described in the specification mean a unit that processes at least one function or operation, which may be implemented using hardware, software, or a combination of hardware and software.

[0345] For convenience of explanation, this specification has been described separately in each drawing. However, it is also possible to design new embodiments by combining the embodiments described in each drawing. Furthermore, designing a computer-readable recording medium containing a program for executing the previously described embodiments, as required by those skilled in the art, is also within the scope of the embodiments.

[0346] The devices and methods according to the embodiments are not limited to the configurations and methods of the embodiments described above, but the embodiments may be configured by selectively combining all or part of each embodiment so that various modifications can be made.

[0347] Although preferred embodiments of the embodiments have been illustrated and described, the embodiments are not limited to the specific embodiments described above, and various modifications may be made by those skilled in the art to which the present disclosure pertains without departing from the spirit or scope of the embodiments claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the embodiments.

[0348] The various components of the devices of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. The various components of the embodiments may be implemented by a single chip, for example, a single hardware circuit. The components according to the embodiments may be implemented by separate chips. At least one of the components of the devices of the embodiments may be configured with one or more processors capable of executing one or more programs, and the one or more programs may perform, or include instructions for performing, one or more of the operations / methods according to the embodiments. The executable instructions for performing the methods / operations of the devices of the embodiments may be stored in non-transitory CRMs or other computer program products configured to be executed by one or more processors, or may be stored in temporary CRMs or other computer program products configured to be executed by one or more processors. In addition, the memory according to the embodiments may be used as a concept including not only volatile memory (e.g., RAM, etc.), but also non-volatile memory, flash memory, PROM, etc. Additionally, it may include implementations in the form of carrier waves, such as transmissions via the Internet. Furthermore, processor-readable recording media may be distributed across network-connected computer systems, allowing processor-readable code to be stored and executed in a distributed manner.

[0349] In this document, " / " and "," are interpreted as "and / or". For example, "A / B" is interpreted as "A and / or B", and "A, B" is interpreted as "A and / or B". Additionally, "A / B / C" means "at least one of A, B, and / or C". Also, "A, B, C" means "at least one of A, B, and / or C". Additionally, "or" in this document is interpreted as "and / or". For example, "A or B" can mean 1) "A" only, 2) "B" only, or 3) "A and B". In other words, "or" in this document can mean "additionally or alternatively".

[0350] Various elements of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. Various elements of the embodiments may be implemented on a single chip, such as a hardware circuit. In some embodiments, the embodiments may optionally be implemented on separate chips. In some embodiments, at least one of the elements of the embodiments may be implemented within one or more processors that include instructions for performing operations according to the embodiments.

[0351] Additionally, the operations according to the embodiments described in this document may be performed by a transceiver device including one or more memories and / or one or more processors according to the embodiments. One or more memories may store programs for processing / controlling the operations according to the embodiments, and one or more processors may control various operations described in this document. One or more processors may be referred to as a controller, etc. The operations according to the embodiments may be performed by firmware, software, and / or a combination thereof, and the firmware, software, and / or a combination thereof may be stored in a processor or a memory.

[0352] Terms such as "first" and "second" may be used to describe various components of the embodiments. However, the various components according to the embodiments should not be interpreted in a limited manner by these terms. These terms are merely used to distinguish one component from another. For example, a first user input signal may be referred to as a "second user input signal." Similarly, a second user input signal may be referred to as a "first user input signal." The use of these terms should be interpreted as not departing from the scope of the various embodiments. Although "first user input signal" and "second user input signal" are both user input signals, they do not necessarily mean the same user input signals unless the context clearly indicates otherwise.

[0353] The terminology used to describe the embodiments is for the purpose of describing particular embodiments and is not intended to be limiting of the embodiments. As used in the description of the embodiments and in the claims, the singular is intended to include the plural unless the context clearly dictates otherwise. The expressions “and / or” are used to mean all possible combinations of the terms. The expression “comprises” or “includes” describes the presence of features, numbers, steps, elements, and / or components, but does not mean that additional features, numbers, steps, elements, and / or components are not included. Conditional expressions such as “if” or “when” used to describe the embodiments are not intended to be limited to only optional cases. When a specific condition is satisfied, a related action is performed in response to a specific condition, or a related definition is intended to be interpreted.

[0354] As described above, the relevant contents have been described in the best form for carrying out the embodiments.

[0355] As described above, the embodiments may be applied, in whole or in part, to mesh data transmission and reception devices and systems. Those skilled in the art will appreciate that various modifications and variations may be made to the embodiments within the scope of the embodiments. The embodiments may include modifications and variations, and such modifications and variations do not depart from the scope of the claims and their equivalents.

Claims

1. Step of decoding the base mesh in the bitstream; A step of decoding displacement data within the bitstream; and A step of decoding attribute data within the bitstream; comprising: How to decode.

2. In paragraph 1, The above bitstream contains data units, Each data unit consists of a data unit header and a data unit payload. The above data unit header includes type information indicating the type of data carried as the data unit payload, Based on the above type information, the atlas data, the base mesh, the displacement data, and the attribute data are parsed from the bitstream. How to decode.

3. In paragraph 2, The above displacement data is encoded based on a video codec or encoded based on an arithmetic codec, The displacement data encoded based on the above video codec and the displacement data encoded based on the above arithmetic codec are identified by the above type information. How to decode.

4. In the third paragraph, the step of decoding displacement data in the bitstream If the above type information indicates that the data carried as the corresponding data unit payload is displacement data encoded based on the arithmetic codec, then arithmetic codec-based decoding is performed on the displacement data based on the parameter set information. How to decode.

5. In paragraph 4, The above parameter set information includes instruction information indicating whether arithmetic encoded displacement data is linked to the corresponding atlas. How to decode.

6. In paragraph 5, If the above instruction information indicates that arithmetic-encoded displacement data is connected to the corresponding atlas, the parameter set information further includes displacement-related information for the decoding process of the displacement data. How to decode.

7. In paragraph 6, The displacement-related information includes information indicating a method of transforming displacement vector samples decoded from the arithmetic-encoded displacement data for the corresponding atlas. How to decode.

8. In the third paragraph, the step of decoding displacement data in the bitstream If the above type information indicates that the data carried as the data unit payload is displacement data encoded based on the video codec, then video codec-based decoding is performed on the displacement data. How to decode.

9. Memory; and At least one processor connected to the memory; At least one processor of the above: Decode the basemesh within the bitstream; Decoding displacement data within the bitstream; and configured to decode attribute data within the above bitstream; Decoding device.

10. Step of encoding the base mesh of mesh data; A step of encoding displacement data of the above mesh data; and A step of encoding attribute data of the above mesh data; comprising; Encoding method.

11. In paragraph 10, The above displacement data is encoded based on a video codec or encoded based on an arithmetic codec. Encoding method.

12. In paragraph 11, The encoded base mesh, the encoded displacement data, and the encoded attribute data are multiplexed into a bitstream and transmitted, The above bitstream contains data units, Each data unit consists of a data unit header and a data unit payload. The above data unit header includes type information indicating the type of data carried as the data unit payload, The above data unit payload includes one of the encoded base mesh, the encoded displacement data, and the encoded attribute data, The displacement data encoded based on the above video codec and the displacement data encoded based on the above arithmetic codec are identified by the above type information. Encoding method.

13. Memory; and At least one processor connected to the memory; At least one processor of the above: Encode the base mesh of mesh data; Encode displacement data of the above mesh data; and Encoding attribute data of the above mesh data; configured to do so; Encoding device.

14. A computer-readable storage medium storing a bitstream generated by the method according to Article 10.

15. Step of obtaining bitstream for mesh data, The bitstream is generated based on the steps of encoding a base mesh of the mesh data; encoding displacement data of the mesh data; and encoding attribute data of the mesh data; and A method comprising the step of transmitting data including the bitstream.

Images