Mesh data transmission device, mesh data transmission method, mesh data reception device, and mesh data reception method

Abstract

A decoding method according to embodiments may comprise: a base mesh processing step of decoding a base mesh or a motion vector of the base mesh included in a bitstream; a displacement information processing step of decoding displacement information included in the bitstream; and a texture map processing step of decoding texture map information included in the bitstream.

Description

Mesh data transmission device, mesh data transmission method, mesh data reception device, and mesh data reception method

[0001] The embodiments provide a method for providing 3D content to provide users with various services such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services.

[0002] Among 3D content, point cloud data and mesh data are collections of points in 3D space. However, the sheer number of points in 3D space makes it difficult to generate point cloud or mesh data.

[0003] That is, there is a problem that a lot of processing is required to transmit and receive 3D data with a large amount of points, such as point cloud data or mesh data.

[0004] The technical problem according to the embodiments is to provide a device and method for efficiently transmitting and receiving mesh data in order to solve the problems described above.

[0005] The technical problem according to the embodiments is to provide a device and method for resolving latency and encoding / decoding complexity of mesh data.

[0006] However, the scope of the embodiments is not limited to the aforementioned technical tasks, and the scope of the embodiments may be expanded to other technical tasks that can be inferred by a person skilled in the art based on the entire contents of this document.

[0007] To achieve the above-described purpose and other advantages, a decoding method according to embodiments may include a step of receiving a bitstream including mesh data and a step of decoding the mesh data.

[0008] According to embodiments, the step of decoding the mesh data may include a base mesh processing step of performing decoding on a base mesh or a motion vector of the base mesh included in the bitstream, a displacement information processing step of performing decoding on displacement information included in the bitstream, and a texture map processing step of performing decoding on texture map information included in the bitstream.

[0009] According to embodiments, the base mesh processing step may include a step of performing decoding using entropy context information for a current base mesh in intra mode and a step of storing the entropy context information used for the decoding, and the entropy context information used for decoding the current base mesh may be initialized by inheriting the entropy context information stored after decoding of a previous base mesh.

[0010] According to embodiments, the base mesh processing step may include, if in inter mode, a step of performing decoding using entropy context information for a motion vector of a current base mesh and a step of storing the entropy context information used for the decoding, and the entropy context information used for decoding the motion vector of the current base mesh may be initialized by inheriting the entropy context information stored after decoding the motion vector of a previous base mesh.

[0011] According to embodiments, the displacement information processing step may include a step of performing decoding using entropy context information on current displacement information and a step of storing the entropy context information used for the decoding, and the entropy context information used for decoding the current displacement information may be initialized by inheriting the entropy context information stored after decoding of previous displacement information, and the decoding may be arithmetic-based decoding.

[0012] A decoding device according to embodiments may include a memory and at least one processor connected to the memory, wherein the at least one processor may be configured to receive a bitstream including mesh data and decode the mesh data.

[0013] According to embodiments, the at least one processor may include a base mesh processing unit that performs decoding on a base mesh or a motion vector of the base mesh included in the bitstream, a displacement information processing unit that performs decoding on displacement information included in the bitstream, and a texture map processing unit that performs decoding on texture map information included in the bitstream.

[0014] According to embodiments, the base mesh processing unit, in the intra mode, performs decoding using entropy context information for the current base mesh, stores the entropy context information used for the decoding, and the entropy context information used for decoding the current base mesh can be initialized by inheriting the entropy context information stored after decoding of the previous base mesh.

[0015] According to embodiments, the base mesh processing unit, in the inter mode, performs decoding using entropy context information for a motion vector of a current base mesh, stores the entropy context information used for the decoding, and the entropy context information used for decoding the motion vector of the current base mesh can be initialized by inheriting the entropy context information stored after decoding the motion vector of a previous base mesh.

[0016] According to embodiments, the displacement information processing unit performs decoding on current displacement information using entropy context information, stores the entropy context information used for the decoding, and the entropy context information used for decoding the current displacement information is initialized by inheriting the entropy context information stored after decoding of the previous displacement information, and the decoding may be arithmetic-based decoding.

[0017] According to embodiments, the encoding method may include a step of encoding mesh data and a step of transmitting a bitstream including the encoded mesh data.

[0018] According to embodiments, the encoding step may include a base mesh processing step of performing encoding on a base mesh generated by simplifying an original mesh or a motion vector of the base mesh, a displacement information processing step of performing encoding on displacement information generated based on the base mesh, and a texture map processing step of performing encoding on a texture map.

[0019] According to embodiments, the base mesh processing step includes a step of performing encoding using entropy context information for the current base mesh if it is an intra mode, and a step of storing the entropy context information used for the encoding, wherein the entropy context information used for encoding the current base mesh can be initialized by inheriting the entropy context information stored after encoding of a previous base mesh.

[0020] According to embodiments, the base mesh processing step includes a step of performing encoding using entropy context information for a motion vector of a current base mesh in an inter mode, and a step of storing the entropy context information used for the encoding, wherein the entropy context information used for encoding the motion vector of the current base mesh can be initialized by inheriting the entropy context information stored after encoding the motion vector of a previous base mesh.

[0021] According to embodiments, the displacement information processing step includes a step of performing encoding using entropy context information on current displacement information, and a step of storing the entropy context information used for the encoding, wherein the entropy context information used for encoding the current displacement information is initialized by inheriting the entropy context information stored after encoding of previous displacement information, and the encoding may be arithmetic-based encoding.

[0022] According to embodiments, a computer program stored on a computer-readable recording medium can be combined with a computer, which is hardware, to perform the above method.

[0023] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can provide a quality 3D service.

[0024] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can achieve various video codec methods.

[0025] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can provide general-purpose 3D content such as autonomous driving services.

[0026] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can store entropy context information when a motion vector of a static mesh or a dynamic mesh is coded, and can reduce the bit amount of motion vectors of a static mesh and a dynamic mesh and improve encoding performance by using the stored context information when coding the next static mesh or motion vector.

[0027] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can reduce the bit amount of the displacement vector and improve encoding performance by also storing and using context information of the displacement vector in the same manner when the coding type of the displacement vector is arithmetic coding.

[0028] The drawings are included to further understand the embodiments, and the drawings illustrate the embodiments together with the description related to the embodiments. For a better understanding of the various embodiments described below, reference should be made to the following description of the embodiments in conjunction with the following drawings, in which like reference numerals correspond to corresponding parts throughout the drawings.

[0029] FIG. 1 illustrates a system for providing dynamic mesh content according to embodiments.

[0030] Figure 2 illustrates a V-MESH compression method according to embodiments.

[0031] Figure 3 illustrates pre-processing of V-MESH compression according to embodiments.

[0032] Figure 4 illustrates a mid-edge subdivision method according to embodiments.

[0033] Figure 5 illustrates a displacement generation process according to embodiments.

[0034] Figure 6 illustrates an intra-frame encoding process of V-MESH data according to embodiments.

[0035] Figure 7 illustrates an inter-frame encoding process of V-MESH data according to embodiments.

[0036] Figure 8 illustrates a lifting conversion process for displacement according to embodiments.

[0037] Figure 9 illustrates a process of packing transformation coefficients into a 2D image according to embodiments.

[0038] Figure 10 illustrates an attribute transfer process of a V-MESH compression method according to embodiments.

[0039] Figure 11 illustrates an intra-frame decoding process of V-MESH data according to embodiments.

[0040] Figure 12 shows an inter-frame decoding processor of V-MESH data.

[0041] Fig. 13 is a drawing showing an example of a transmitting device according to embodiments.

[0042] Fig. 14 is a drawing showing an example of a receiving device according to embodiments.

[0043] Fig. 15 is a drawing showing another example of a transmitting device according to embodiments.

[0044] FIG. 16 is a detailed block diagram showing an example of a motion vector encoder according to embodiments.

[0045] FIG. 17(a) is a detailed block diagram showing another example of a motion vector encoder according to embodiments, and FIG. 17(b) is a diagram showing another example of a motion vector encoding process according to embodiments.

[0046] FIG. 18(a) and FIG. 18(b) are diagrams showing another example of a motion vector encoding process according to embodiments.

[0047] FIG. 19(a) and FIG. 19(b) are diagrams showing another example of a motion vector encoding process according to embodiments.

[0048] FIG. 20 is a detailed block diagram showing an example of a static mesh encoder according to embodiments.

[0049] FIG. 21 is a detailed block diagram showing another example of a static mesh encoder according to embodiments.

[0050] FIG. 22(a) is a detailed block diagram showing another example of a static mesh encoder according to embodiments, and FIG. 22(b) is a diagram showing another example of a static mesh encoding process according to embodiments.

[0051] FIG. 23 is a detailed block diagram showing an example of a displacement vector encoder according to embodiments.

[0052] Fig. 24 is a detailed block diagram showing another example of a displacement vector encoder according to embodiments.

[0053] FIG. 25 is a detailed block diagram showing another example of a displacement vector encoder according to embodiments.

[0054] FIG. 26 is a detailed block diagram showing another example of a displacement vector encoder according to embodiments.

[0055] Fig. 27 is a diagram showing an example of a displacement vector arithmetic encoding unit according to embodiments.

[0056] Fig. 28 is a diagram showing an example of an encoding process of a sub-block displacement vector transformation coefficient encoding unit according to embodiments.

[0057] Fig. 29 is a detailed block diagram showing another example of a displacement vector encoder based on arithmetic coding according to embodiments.

[0058] FIG. 30 is a diagram showing an example of context information storage and inheritance according to displacement vector type according to embodiments.

[0059] Fig. 31 is a block diagram showing a configuration of a receiving device according to embodiments.

[0060] Fig. 32(a) is a detailed block diagram showing an example of a displacement vector coordinate system inverse transformation unit (15020) according to embodiments.

[0061] FIG. 32(b) is a detailed block diagram showing another example of a displacement vector coordinate system inverse transformation unit (15020) according to embodiments.

[0062] FIG. 33 is a detailed block diagram showing an example of a motion vector decoder according to embodiments.

[0063] FIG. 34 is a detailed block diagram showing another example of a motion vector decoder according to embodiments.

[0064] FIG. 35 is a detailed block diagram showing an example of a static mesh decoder according to embodiments.

[0065] FIG. 36 is a detailed block diagram showing another example of a static mesh decoder according to embodiments.

[0066] FIG. 37 is a detailed block diagram showing another example of a static mesh decoder according to embodiments.

[0067] Fig. 38 is a diagram showing an example of a syntax structure when the type of a submesh that inherits context information according to embodiments is S_SUMMESH.

[0068] FIG. 39 is a detailed block diagram showing an example of a displacement vector decoder according to embodiments.

[0069] FIG. 40 is a detailed block diagram showing another example of a displacement vector decoder according to embodiments.

[0070] FIG. 41 is a detailed block diagram showing another example of a displacement vector decoder according to embodiments.

[0071] FIG. 42 is a detailed block diagram showing another example of a displacement vector decoder according to embodiments.

[0072] Fig. 43 is a diagram showing an example of a displacement vector conversion coefficient decoding unit according to embodiments.

[0073] Fig. 44 is a diagram showing an example of a decoding process of a sub-block displacement vector conversion coefficient decoding unit according to embodiments.

[0074] Fig. 45 is a detailed block diagram showing another example of a displacement vector decoder based on arithmetic coding according to embodiments.

[0075] Figure 46 is a diagram showing an example of context information storage and inheritance according to displacement vector type according to embodiments.

[0076] FIG. 47 is a diagram showing an example of the syntax structure of a sequence parameter set (bmesh_sequence_parameter_set_rbsp()) according to embodiments.

[0077] FIG. 48 is a diagram showing an example of the syntax structure of a submesh header (bmesh_submesh_header()) according to embodiments.

[0078] FIG. 49 is a diagram showing an example of the syntax structure of a submesh unit (bmesh_submesh_unit()) according to embodiments.

[0079] FIG. 50 is a diagram showing another example of the syntax structure of a submesh header (bmesh_submesh_header()) according to embodiments.

[0080] FIG. 51 is a diagram showing an example of the syntax structure of a frame parameter set (bmesh_frame_parameter_set_rbsp( )) according to embodiments.

[0081] FIG. 52 is a diagram showing another example of the syntax structure of a sequence parameter set (displ_sequence_parameter_set_rbsp()) according to embodiments of the present invention.

[0082] FIG. 53 is a diagram showing an example of the syntax structure of a displacement vector header (displ_header()) according to embodiments.

[0083] FIG. 54 is a diagram showing another example of the syntax structure of a sequence parameter set (displ_sequence_parameter_set_rbsp()) according to embodiments of the present invention.

[0084] FIG. 55 is a diagram showing another example of the syntax structure of a displacement vector header (displ_header()) according to embodiments.

[0085] FIG. 56 is a diagram showing another example of the syntax structure of a displacement vector header (displ_header()) according to embodiments.

[0086] Figure 57 is a flowchart showing an example of a transmission method according to embodiments.

[0087] Figure 58 is a flowchart showing an example of a receiving method according to embodiments.

[0088] Preferred embodiments of the embodiments are described in detail, examples of which are illustrated in the accompanying drawings. The following detailed description, with reference to the accompanying drawings, is intended to illustrate preferred embodiments of the embodiments, rather than merely show embodiments that can be implemented according to the embodiments. The following detailed description includes details to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may be practiced without these details.

[0089] While most of the terms used in the examples are commonly used in the field, some terms were arbitrarily selected by the applicant, and their meanings are described in detail in the following descriptions as needed. Therefore, the examples should be understood based on the intended meaning of the terms, not simply their names or meanings.

[0090] With the recent development of 3D data modeling and rendering technology, research on creating and processing 3D data is being conducted in various fields such as Virtual Reality (VR), Augmented Reality (AR), autonomous driving, Computer-Aided Design (CAD) / Computer-Aided Manufacturing (CAM), and Geographic Information Systems (GIS). 3D data can be represented as point clouds, meshes, etc., depending on the representation format. Among these, a mesh is composed of geometric information expressing the coordinate values ​​of each vertex (or point), connection information indicating the connection relationship between vertices, a texture map expressing the color information of the mesh surface as 2D image data, and texture coordinates indicating mapping information between the surface of the mesh and the texture map. In the present disclosure, a mesh is defined as a dynamic mesh if one or more of the elements that make up the mesh change over time, and a static mesh if they do not change. In other words, dynamic mesh data may refer to mesh data that has an object or movement.

[0091] Because dynamic mesh data has a large amount of data for elements that constitute the mesh compared to two-dimensional image data, technologies have been developed to efficiently compress this large amount of mesh data to store and transmit it.

[0092] FIG. 1 illustrates a system for providing dynamic mesh content according to embodiments.

[0093] The system of FIG. 1 includes a transmitting device (100) and a receiving device (110) according to embodiments. The transmitting device (100) may include a mesh video acquisition unit (101), a mesh video encoder (102), a file / segment encapsulator (103), and a transmitter (104). The receiving device (110) may include a receiving unit (111), a file / segment decapsulator (112), a mesh video decoder (113), and a renderer (114). Each component of FIG. 1 may correspond to hardware, software, a processor, and / or a combination thereof. Hereinafter, the mesh data transmitting device according to embodiments may be interpreted as a term referring to a 3D data transmitting device or transmitting device (100), or a mesh video encoder (hereinafter, referred to as an encoder) (102). The mesh data receiving device according to the embodiments may be interpreted as a term referring to a 3D data receiving device or receiving device (110), or a mesh video decoder (hereinafter, decoder) (113).

[0094] The system of FIG. 1 can perform video-based dynamic mesh compression and decompression.

[0095] Advances in 3D capture, modeling, and rendering have enabled users to consume diverse forms of 3D content, such as AR, XR, metaverse, and holograms, across multiple platforms and devices. 3D content increasingly represents objects with greater precision and realism, enabling users to enjoy immersive experiences. To achieve this, the creation and use of 3D models requires a significant amount of data. Among various types of 3D content, 3D meshes are widely used for efficient data utilization and realistic object representation. Embodiments include a series of processing steps in a system that utilizes such mesh content.

[0096] First, the method of compressing dynamic mesh data starts from the V-PCC (Video-based point cloud compression) standard technology for point cloud data. Point cloud data is data that has color information at the coordinates (X, Y, Z) of a vertex (or point). In the present disclosure, the coordinates (i.e., position information) of a vertex are referred to as geometry information, the color information of a vertex is referred to as attribute information, and the geometry information and attribute information are referred to as vertex information or point cloud data. The vertex information to which connectivity information between vertices is added is referred to as mesh data. When creating content, it can be created in the form of mesh data from the beginning. Alternatively, it can be used by converting it into mesh data by adding connectivity information to point cloud data.

[0097] Currently, the MPEG standards body defines two types of dynamic mesh data: Category 1: Mesh data with texture maps as color information. Category 2: Mesh data with vertex colors as color information.

[0098] Mesh coding standards for Category 1 data are currently under development, and work on Category 2 data standards is also planned for the future. The overall process for providing mesh content services may include acquisition, encoding, transmission, decoding, rendering, and / or feedback, as shown in Figure 1.

[0099] To provide mesh content services, 3D data acquired through multiple cameras or specialized cameras can be processed into mesh data types through a series of processes and then converted into video. The generated mesh video is then transmitted through a series of processes, and the receiving end can then reprocess the received data into mesh video and render it. This allows mesh video to be presented to users, who can then interact with the mesh content according to their intended intent.

[0100] A mesh compression system may include a transmitting device (100) and a receiving device (110) as shown in FIG. 1. The transmitting device (100) may encode mesh video to output a bitstream, and transmit the bitstream to the receiving device (110) in the form of a file or streaming (streaming segment) via a digital storage medium or a network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0101] In the above transmitting device (100), the encoder may be called a mesh video / video / picture / frame encoding device, and in the receiving device (110), the decoder may be called a mesh video / video / picture / frame decoding device. The transmitter may be included in a mesh video encoder. The receiver may be included in a mesh video decoder. The renderer (114) may include a display unit, and the renderer and / or the display unit may be configured as separate devices or external components. The transmitting device (100) and the receiving device (110) may further include separate internal or external modules / units / components for a feedback process.

[0102] Mesh data represents the surface of an object as a number of polygons. Each polygon is defined by vertices in 3D space and connection information that describes how the vertices are connected. It can also contain vertex attributes such as vertex color and normal. Mapping information that allows the surface of the mesh to be mapped to a 2D planar area can also be included in the attributes of the mesh. The mapping can be described as a set of parameter coordinates, generally called UV coordinates or texture coordinates, associated with the mesh vertices. Meshes contain 2D attribute maps, which can be used to store high-resolution attribute information such as textures, normals, and displacement. Here, displacement can be used interchangeably with displacement, displacement information, or displacement vectors (i.e., displacement vectors).

[0103] The mesh video acquisition unit (101) may include processing 3D object data acquired through a camera, etc. into a mesh data type having the attributes described above through a series of processes and generating a video composed of such mesh data. The mesh video may have attributes of the mesh, such as vertices, polygons, connection information between vertices, colors, normals, etc., that may change over time. A mesh video having attributes and connection information that change over time in this way may be expressed as a dynamic mesh video.

[0104] A mesh video encoder (102) can encode an input mesh video into one or more video streams. One video can include multiple frames, and one frame can correspond to a still image / picture. In this document, a mesh video can include a mesh image / frame / picture, and a mesh video can be used interchangeably with a mesh image / frame / picture. The mesh video encoder (102) can perform a Video-based Dynamic Mesh (V-Mesh) Compression procedure. The mesh video encoder (102) can perform a series of procedures such as prediction, transformation, quantization, and entropy coding for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0105] The file / segment encapsulator (103) can encapsulate encoded mesh video data and / or mesh video-related metadata in the form of a file, etc. Here, the mesh video-related metadata may be received from a metadata processing unit, etc. The metadata processing unit may be included in the mesh video encoder (102) or may be configured as a separate component / module. The file / segment encapsulator (103) can encapsulate the corresponding data in a file format such as ISOBMFF, or process it in the form of other DASH segments, etc. The file / segment encapsulator (103) may include mesh video-related metadata in the file format according to an embodiment. The mesh video metadata may be included in boxes at various levels in the ISOBMFF file format, for example, or may be included as data in a separate track within the file. Depending on the embodiment, the file / segment encapsulator (103) may encapsulate the mesh video related metadata itself into a file.

[0106] The transmission processing unit can process encapsulated mesh video data for transmission according to the file format. The transmission processing unit can be included in the transmission unit (104) or can be configured as a separate component / module. The transmission processing unit can process mesh video data according to any transmission protocol. The processing for transmission can include processing for transmission through a broadcast network or processing for transmission through broadband. According to an embodiment, the transmission processing unit can receive not only mesh video data but also mesh video-related metadata from the metadata processing unit and process it for transmission.

[0107] The transmission unit (104) can transmit encoded video / image information or data output in the form of a bitstream to the reception unit (111) of the reception device (110) via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (104) can include an element for generating a media file through a predetermined file format and can include an element for transmission via a broadcasting / communication network. The reception unit (111) can extract the bitstream and transmit it to a decoding device.

[0108] The receiving unit (111) can receive mesh video data transmitted by a mesh data transmission device. Depending on the channel through which it is transmitted, the receiving unit (111) can receive mesh video data through a broadcast network, through a broadband, or through a digital storage medium.

[0109] The receiving processing unit can perform processing according to the transmission protocol on the received mesh video data. The receiving processing unit can be included in the receiving unit (111) or can be configured as a separate component / module. In order to correspond to the processing performed for transmission on the transmitting side, the receiving processing unit can perform the reverse process of the aforementioned transmission processing unit. The receiving processing unit can transfer the acquired mesh video data to the file / segment decapsulator (112) and transfer the acquired mesh video-related metadata to the metadata parser. The mesh video-related metadata acquired by the receiving processing unit can be in the form of a signaling table.

[0110] The file / segment decapsulator (112) can decapsulate mesh video data in the form of a file received from a receiving processing unit. The file / segment decapsulator (112) can decapsulate files according to ISOBMFF, etc., to obtain a mesh video bitstream or mesh video-related metadata (metadata bitstream). The obtained mesh video bitstream can be transmitted to the mesh video decoder (113), and the obtained mesh video-related metadata (metadata bitstream) can be transmitted to the metadata processing unit. The mesh video bitstream may include metadata (metadata bitstream). The metadata processing unit may be included in the mesh video decoder (113) or may be configured as a separate component / module. The mesh video-related metadata obtained by the file / segment decapsulator (112) may be in the form of a box or track within a file format. The file / segment decapsulator (112) may receive metadata required for decapsulation from the metadata processing unit, if necessary. The mesh video related metadata may be passed to the mesh video decoder (113) and used in the mesh video decoding procedure, or may be passed to the renderer (114) and used in the mesh video rendering procedure.

[0111] The mesh video decoder (113) can receive a bitstream and perform a reverse operation corresponding to the operation of the mesh video encoder (102) to decode the video / image. The decoded mesh video / image can be displayed through the display unit of the renderer (114). The user can view all or part of the rendered result through a VR / AR display or a general display.

[0112] The feedback process may include a process of transmitting various feedback information that may be acquired during the rendering / display process to the transmitter or to the decoder on the receiver. Interactivity may be provided in mesh video consumption through the feedback process. Depending on the embodiment, head orientation information, viewport information indicating the area that the user is currently viewing, etc. may be transmitted during the feedback process. Depending on the embodiment, the user may interact with things implemented in the VR / AR / MR / autonomous driving environment, in which case information related to the interaction may be transmitted to the transmitter or the service provider during the feedback process. Depending on the embodiment, the feedback process may not be performed.

[0113] Head orientation information can refer to information about the user's head position, angle, and movement. Based on this information, information about the area the user is currently viewing within the mesh video, i.e. viewport information, can be calculated.

[0114] Viewport information can be information about the area the user is currently viewing in the mesh video. This can be used to perform gaze analysis to determine how the user consumes the mesh video, which area of ​​the mesh video they are gazing at, and for how long. Gaze analysis can be performed on the receiving side and transmitted to the transmitting side through a feedback channel. Devices such as VR / AR / MR displays can extract the viewport area based on the user's head position / orientation, the vertical or horizontal FOV supported by the device, etc.

[0115] Depending on the embodiment, the aforementioned feedback information may not only be transmitted to the transmitter but may also be consumed by the receiver. That is, the aforementioned feedback information may be utilized to perform decoding, rendering, and other processes on the receiver. For example, head orientation information and / or viewport information may be utilized to preferentially decode and render only the mesh video for the area currently being viewed by the user.

[0116] This document relates to embodiments of dynamic mesh video compression as described above. The method / embodiment disclosed in this document can be applied to the Video-based Dynamic Mesh Compression (V-Mesh) standard of the Moving Picture Experts Group (MPEG) or the next-generation video / image coding standard. Dynamic mesh video compression is a method for processing mesh connection information and attributes that change over time, and it can perform lossy and lossless compression for various applications such as real-time communication, storage, free-viewpoint video, and AR / VR.

[0117] The dynamic mesh video compression method described below is based on MPEG's V-Mesh method.

[0118] In this document, picture / frame can generally mean a unit representing one video of a specific time period.

[0119] A pixel or pel can refer to the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, only the pixel / pixel value of the chroma component, or only the pixel / pixel value of the depth component.

[0120] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0121] As described above, the encoding process of Fig. 1 is as follows.

[0122] That is, the video-based dynamic mesh compression (V-Mesh) compression method can provide a method of compressing dynamic mesh video data based on 2D video codecs such as HEVC (High Efficiency Video Coding) and VVC (Versatile Video Coding). The V-Mesh compression process receives the following data as input and performs compression.

[0123] Input mesh: Contains the 3D coordinates of the vertices that make up the mesh, normal information for each vertex, mapping information that maps the mesh surface to a 2D plane, and connection information between the vertices that make up the surface. The mesh surface can be expressed as triangles or more polygons, and connection information between the vertices that make up each surface is stored according to a set shape. The input mesh can be saved in the OBJ file format.

[0124] Attribute map: (Hereinafter, texture map is also used in the same meaning): Contains information about the attributes of the mesh (color, normal, displacement, etc.), and stores data in the form of a mapping of the surface of the mesh onto a 2D image. Mapping which part (surface or vertex) of the mesh each data of this attribute map corresponds to is based on the mapping information contained in the input mesh. Since the attribute map has data for each frame of the mesh video, it can also be expressed as an attribute map video. The attribute map in the V-Mesh compression method mainly contains the color information of the mesh, and is saved in an image file format (PNG, BMP, etc.).

[0125] Material Library File: Contains information about the material attributes used in a mesh, and in particular, information that links the input mesh to its corresponding attribute map. It is saved in the Wavefront Material Template Library (MTL) file format.

[0126] In the V-Mesh compression method, the following data and information can be generated through the compression process.

[0127] Base mesh: The input mesh is simplified (decimated) through a pre-processing process, thereby expressing the objects of the input mesh using the minimum number of vertices determined by the user's standards.

[0128] Displacement: Displacement information used to express the input mesh as similarly as possible to the base mesh, and is expressed in the form of 3D coordinates.

[0129] Atlas information: This is the metadata required to reconstruct a mesh using base mesh, displacement, and attribute map information. It can be created and utilized as sub-mesh units (such as patches) that make up the mesh.

[0130] Referring to FIGS. 2 to 7, a method for encoding mesh position information (or vertex position information) is described, and referring to FIGS. 6 to 10, etc., a method for encoding attribute information (attribute map) by restoring mesh position information is described.

[0131] Figure 2 illustrates a V-MESH compression method according to embodiments.

[0132] Fig. 2 illustrates the encoding process of Fig. 1, and the encoding process may include a pre-processing process and an encoding process. The mesh video encoder (102) of Fig. 1 may include a pre-processor (200) and an encoder (201) as in Fig. 2. In addition, the transmitting device of Fig. 1 may be broadly referred to as an encoder, and the mesh video encoder (102) of Fig. 1 may be referred to as an encoder. The V-Mesh compression method may include a pre-processing process (Pre-processing, 200) and an encoding process (Encoding, 201) as in Fig. 2. The pre-processor (200) of Fig. 2 may be located in front of the encoder (201) of Fig. 2. The pre-processor (200) and the encoder (201) of Fig. 2 may be referred to as a single encoder.

[0133] The pre-processor (200) can receive a static of a dynamic mesh (M(i)) and / or an attribute map (A(i)). The pre-processor (200) can generate a base mesh (m(i)) and / or a displacement (or displacement) (d(i)) through pre-processing. The pre-processor (200) can receive feedback information from the encoder (201) and generate the base mesh and / or the displacement based on the feedback information.

[0134] The encoder (201) can receive a base mesh (m(i)), a displacement (d(i)), a static of a dynamic mesh (M(i)), and / or an attribute map (A(i)). In the present disclosure, at least one of the base mesh (m(i)), the displacement (d(i)), the static of a dynamic mesh (M(i)), and / or the attribute map (A(i)) can be referred to as mesh-related data. The encoder (201) can encode the mesh-related data to generate a compressed bitstream.

[0135] Figure 3 illustrates a pre-processing process of V-MESH compression according to embodiments.

[0136] Fig. 3 illustrates the configuration and operation of the preprocessor of Fig. 2. In Fig. 3, the input mesh may include a static of a dynamic mesh (M(i)) and / or an attribute map (A(i)). In addition, the input mesh may include three-dimensional coordinates of vertices constituting the mesh, normal information of each vertex, mapping information for mapping the mesh surface to a 2D plane, connection information between vertices constituting the surface, etc.

[0137] Fig. 3 shows a process of performing pre-processing on an input mesh. The pre-processing process (200) may largely include four steps: 1) GoF (Group of Frame) generation, 2) Mesh Decimation, 3) UV parameterization, and 4) Fitting subdivision surface (300). According to embodiments, GoF generation may be referred to as a GoF generation process or a GoF generation unit, mesh simplification may be referred to as a mesh simplification process or a mesh simplification unit, UV parameterization may be referred to as a UV parameterization process or a UV parameterization unit, and the fitting subdivision surface may be referred to as a fitting subdivision surface process or a fitting subdivision surface unit. The pre-processor (200) can generate displacement and / or base meshes from the received input mesh and transmit them to the encoder (201). The pre-processor (200) can transmit GoF information associated with GoF generation to the encoder (201).

[0138] Below, each step of Fig. 3 is described.

[0139] GoF Generation: This is the process of generating a reference structure for mesh data. If the number of vertices, the number of texture coordinates, the vertex connection information, and the texture coordinate connection information of the mesh of the previous frame and the current mesh are all the same, the previous frame can be set as the reference frame. That is, if only the vertex coordinate values ​​are different between the current input mesh and the reference input mesh, the encoder (201) can perform inter frame encoding. Otherwise, intra frame encoding is performed for the corresponding frame.

[0140] Mesh Decimation: This process simplifies the input mesh to create a simplified mesh, or base mesh. Vertices to be removed from the original mesh are selected based on user-defined criteria, and the selected vertices and the triangles connected to them can be removed.

[0141] In the process of performing mesh simplification (Mesh decimation), the input mesh (voxelized), target triangle ratio (TTR), and minimum triangle component (CCCount) information are passed as input, and the simplified mesh (decimated mesh) can be obtained as output. In this process, connected triangle components smaller than the set minimum triangle component (CCCount) can be removed.

[0142] UV parameterization: This is the process of mapping a 3D surface of a decimated mesh into a texture domain. Parameterization can be performed using the UVAtlas tool. This process generates mapping information, which indicates where each vertex of the decimated mesh can be mapped to on a 2D image. This mapping information is expressed and stored as texture coordinates, and through this process, the final base mesh is created.

[0143] Fitting subdivision surface (300): This is a process of performing subdivision on a decimated mesh (i.e., a simplified mesh having texture coordinates). The displacement and base mesh generated through this process are output to the encoder (201). A user-defined method, such as a mid-edge method, may be applied as the subdivision method. A fitting process is performed so that the input mesh and the mesh on which the subdivision has been performed become similar to each other. In the present disclosure, the mesh on which the fitting process has been performed is referred to as a fitted subdivision mesh (or fitted subdivision mesh).

[0144] Figure 4 illustrates a mid-edge subdivision method according to embodiments.

[0145] Figure 4 illustrates the mid-edge method of the fitting subdivision surface described in Figure 3. Referring to Figure 4, an original mesh containing four vertices is subdivided to generate a sub-mesh. A sub-mesh can be generated by creating a new vertex in the middle of the edge between the vertices. Then, a fitting process is performed so that the input mesh and the sub-mesh become similar to each other, thereby generating a fitted sub-division mesh.

[0146] When a fitted subdivided mesh (hereinafter referred to as a fitted subdivided mesh) is generated, displacement is calculated using this result and a pre-compressed and decoded base mesh (hereinafter referred to as a reconstructed base mesh). That is, the reconstructed base mesh is subdivided in the same way as the fitting subdivision surface. The difference in position of each vertex between this result and the fitted subdivided mesh is the displacement for each vertex. Since displacement represents the position difference in three-dimensional space, it is also expressed as a value in the (x, y, z) space of a Cartesian coordinate system. Depending on the user input parameters, the (x, y, z) coordinate values ​​can be converted to (normal, tangential, bi-tangential) coordinate values ​​of the local coordinate system.

[0147] Fig. 5 illustrates a displacement generation process according to embodiments. The displacement generation process of Fig. 5 may be performed in a pre-processor (200) or in an encoder (201).

[0148] FIG. 5 illustrates in detail the displacement calculation method of the fitting subdivision surface (300) as described in FIG. 4.

[0149] An encoder and / or pre-processor according to embodiments may include 1) a subdivision unit, 2) a local coordinate system calculation unit, and 3) a displacement calculation unit. The subdivision unit may perform subdivision on a reconstructed base mesh to generate a subdivided reconstructed base mesh. Here, the restoration of the base mesh may be performed in the pre-processor (200) or in the encoder (201). The local coordinate system calculation unit may receive a fitted subdivision mesh and a subdivided reconstructed base mesh, and may convert a coordinate system of the mesh into a local coordinate system based on the fitted subdivision mesh and the subdivided reconstructed base mesh. The local coordinate system calculation operation may be optional. The displacement calculation unit may calculate a positional difference between the fitted subdivision mesh and the subdivided reconstructed base mesh. For example, a positional difference value between vertices of two input meshes may be generated. The vertex positional difference value becomes a displacement.

[0150] The mesh data transmission method and device according to the embodiments can encode mesh data as follows. Mesh data is a term including point cloud data. Point cloud data (which may be referred to as point cloud for short) according to the embodiments can refer to data including vertex coordinates (or geometry information) and color information (or attribute information). In addition, geometry images, attribute images, occupancy maps, and additional information (or patch information) generated through patch generation and packing based on vertex coordinates and color information are also referred to as point cloud data. Therefore, point cloud data including connection information can be referred to as mesh data. In this document, point cloud and mesh data can be used interchangeably.

[0151] The V-Mesh compression (reconstruction) method according to the embodiments may include intra frame encoding (Fig. 6) and inter frame encoding (Fig. 7).

[0152] Based on the results of the GoF generation described above, intra-frame encoding or inter-frame encoding is performed. In the case of intra-encoding, the data to be compressed may be a base mesh, displacement, attribute map, etc. In the case of inter-encoding, the data to be compressed may be a displacement, attribute map, and a motion field between a reference base mesh and the current base mesh.

[0153] Fig. 6 illustrates an intra-frame encoding process of a V-MESH compression method according to embodiments. Each component for the intra-frame encoding process of Fig. 6 corresponds to hardware, software, a processor, and / or a combination thereof.

[0154] The encoding process of FIG. 6 details the encoding of the mesh video encoder (102) of FIG. 1. That is, it shows the configuration of the mesh video encoder (102) when the encoding of FIG. 1 is an intra-frame method. The encoder of FIG. 6 may include a pre-processor (200) and / or an encoder (201). The pre-processor (200) and encoder (201) of FIG. 6 may correspond to the pre-processor (200) and encoder (201) of FIG. 3.

[0155] The preprocessor (200) can receive an input mesh and perform the preprocessing described above. The preprocessing can generate a base mesh and / or a fitted subdivision mesh.

[0156] The quantizer (411) of the encoder (201) can quantize the base mesh and / or the fitted subdivided mesh. The static mesh encoder (412) can encode the static mesh (i.e., the quantized base mesh) and generate a bitstream (i.e., a compressed base mesh bitstream) including the encoded base mesh. The static mesh decoder (413) can decode the encoded static mesh (i.e., the encoded base mesh). The inverse quantizer (414) can inversely quantize the quantized static mesh (i.e., the base mesh) to output a reconstructed (or restored) base mesh. The displacement calculation unit (415) can generate displacements (or displacements) based on the reconstructed static mesh (i.e., the base mesh) and the fitted subdivided mesh. According to embodiments, the displacement calculation unit (415) calculates displacement, which is the position difference between each vertex of the subdivided base mesh and the fitted subdivided mesh after subdividing (or refining) the restored base mesh. In other words, the displacement is a displacement vector, which is the position difference between the vertices of the two meshes so that the fitted subdivided (or refining) mesh becomes similar to the original mesh. The forward linear lifting unit (416) can perform lifting transformation on the input displacement to generate lifting coefficients (or transform coefficients). The quantizer (417) can quantize the lifting coefficients. The image packing unit (418) can pack an image based on the quantized lifting coefficients. The video encoder (419) can encode the packed image. That is, the quantized lifting coefficients are packed into one frame as a 2D image by the image packing unit (418), compressed through the video encoder (419), and output as a displacement bitstream (i.e., compressed displacement bitstream).

[0157] A video decoder (420) decodes a compressed displacement bitstream. An image unpacking unit (421) can perform unpacking on the decoded displacement frame to output quantized lifting coefficients. A dequantizer (422) can dequantize the quantized lifting coefficients. An inverse linear lifting unit (423) applies inverse lifting to the inverse quantized lifting coefficients to generate restored displacement. A mesh restoration unit (424) reconstructs and deforms a mesh using the restored displacement output from the inverse linear lifting unit (423) and the restored base mesh (or subdivided restored base mesh) output from the inverse quantization unit (414). The present disclosure refers to the reconstructed and deformed mesh as a restored deformed mesh.

[0158] The attribute transfer (425) receives an input mesh and / or an input attribute map, and regenerates an attribute map based on the restored deformed mesh. The attribute map refers to a texture map corresponding to attribute information among mesh data components, and in the present disclosure, the attribute map and the texture map may be used interchangeably. The push-pull padding (426) may pad data in the attribute map based on the push-pull method. The color space conversion unit (427) may convert the space of the color component of the attribute map. For example, the attribute map may be converted from an RGB color space to a YUV color space. The video encoder (428) may encode the attribute map and output it as a compressed attribute bitstream.

[0159] A multiplexer (430) can generate a compressed bitstream by multiplexing a compressed base mesh bitstream, a compressed displacement bitstream, and a compressed attribute bitstream.

[0160] In Fig. 6, the displacement calculation unit (415) may be included in the pre-processor (200). In addition, at least one of the quantizer (411), the static mesh encoder (412), the static mesh decoder (413), and the inverse quantizer (414) may be included in the pre-processor (200).

[0161] As described in FIG. 6, the intra-frame encoding method includes base mesh encoding (also called static mesh encoding). That is, when performing intra-frame encoding on the current input mesh frame, the base mesh generated in the pre-processing process of the pre-processor (200) can be encoded using a static mesh compression technology in a static mesh encoder (412) after undergoing a quantization process in a quantizer (411). In the V-Mesh compression method, for example, Draco technology is applied to base mesh encoding, and vertex position information, mapping information (texture coordinates), vertex connection information, etc. of the base mesh become compression targets.

[0162] The encoder of Fig. 6 generates a bitstream by compressing the base mesh, displacement, and attributes within the frame, and the encoder of Fig. 7 generates a bitstream by compressing the motion, displacement, and attributes between the current frame and the reference frame.

[0163] Fig. 7 illustrates an inter-frame encoding process of a V-MESH compression method according to embodiments. Each component for the inter-frame encoding process of Fig. 7 corresponds to hardware, software, a processor, and / or a combination thereof.

[0164] The encoding process of Fig. 7 details the encoding of Fig. 1. That is, it shows the configuration of an encoder when the encoding of Fig. 1 is an inter-frame method. The encoder of Fig. 7 may include a pre-processor (200) and / or an encoder (201). The pre-processor (200) and encoder (201) of Fig. 7 may correspond to the pre-processor (200) and encoder (201) of Fig. 3.

[0165] For a description of the components corresponding to the encoding operation of FIG. 6 among the encoding operations of FIG. 7, refer to the description of FIG. 6. That is, the operation of the quantizer (511), displacement calculation unit (515), wavelet transformer (516), quantizer (517), image packing unit (518), video encoder (519), video decoder (520), image unpacking unit (521), inverse quantizer (522), inverse wavelet transformer (523), mesh restoration unit (524), attribute transfer (525), push-pull padding (526), ​​color space conversion unit (527), video encoder (528), and multiplexer (530) of FIG. 7 is similar to that of the quantizer (411), static mesh encoder (412), static mesh decoder (413), inverse quantizer (414), displacement calculation unit (415), forward linear lifting unit (416), quantizer (417), image Since the operations described in the packing unit (418), video encoder (419), video decoder (420), image unpacking unit (421), inverse quantizer (422), inverse linear lifting unit (423), mesh restoration unit (424), attribute transfer (425), push-pull padding (426), color space conversion unit (427), video encoder (428), and multiplexer (430) are the same or similar, a detailed description thereof is omitted in FIG. 7 to avoid redundant description.

[0166] In Fig. 7, for inter-frame based encoding, the motion encoder (512) can obtain a motion vector between the two base meshes based on the restored quantized reference base mesh and the quantized current base mesh, and then encode the motion vector to output a compressed motion bitstream. The motion encoder (512) can be referred to as a motion vector encoder. The base mesh restoration unit (513) can restore the base mesh based on the restored quantized reference base mesh and the encoded motion vector. The restored base mesh is dequantized in the dequantizer (514) and then output to the displacement calculation unit (515).

[0167] In Fig. 7, the displacement calculation unit (515) may be included in the pre-processor (200). In addition, at least one of the quantizer (511), the motion encoder (512), the base mesh restoration unit (513), and the inverse quantizer (514) may be included in the pre-processor (200).

[0168] As described in Fig. 7, the inter-frame encoding method may include motion field encoding (also called motion vector encoding). Inter-frame encoding may be performed when a one-to-one correspondence of vertices is established between a reference mesh and a current input mesh, and only the position information of the vertices is different. When performing inter-frame encoding, instead of compressing the base mesh, the difference between the vertices of the reference base mesh and the current base mesh, i.e., the motion field (also called motion vector), may be calculated and encoded to encode this information. The reference base mesh is the result of quantizing the already decoded base mesh data and is determined according to the reference frame index determined in the GoF generation. The motion field may also be encoded as a value. Alternatively, the predicted motion field can be calculated by averaging the motion fields of the restored vertices among the vertices connected to the current vertex, and the residual motion field, which is the difference between the predicted motion field value and the motion field value of the current vertex, can be encoded. This residual motion field value can be encoded using entropy coding.The process of encoding displacement and attribute maps, excluding the motion field encoding process of inter frame encoding, is the same as the structure of the intra frame encoding method except for the base mesh encoding.

[0169] Figure 8 illustrates a lifting conversion process for displacement according to embodiments.

[0170] Figure 9 illustrates a process of packing transformation coefficients (or lifting coefficients) according to embodiments into a 2D image.

[0171] Figures 8 and 9 illustrate the process of transforming displacement and packing transform coefficients of the encoding process of Figures 6 and 7, respectively.

[0172] The encoding method according to the embodiments includes displacement encoding.

[0173] After base mesh encoding and / or motion field encoding, a reconstructed base mesh is generated through restoration and dequantization, and the displacement between the result of performing subdivision on the reconstructed base mesh and the fitted subdivided mesh generated through the fitting subdivision surface can be calculated (see 415 in FIG. 6 or 515 in FIG. 7). For effective encoding, a data transform process such as wavelet transform can be applied to the displacement information (see 416 in FIG. 6 or 516 in FIG. 7).

[0174] FIG. 8 shows a process of transforming displacement information using a lifting transform in the forward linear lifting unit (416) of FIG. 6 or the wavelet transformer (516) of FIG. 7. For example, a linear wavelet-based lifting transform may be performed. The transform coefficients generated through the transform process are quantized in a quantizer (417 or 517) and then packed into a 2D image through an image packing unit (418 or 518) as in FIG. 9. The transform coefficients are configured as one block for every 256 (= 16×16) units, and each block can be packed in a z-scan order. The horizontal number of blocks is fixed to 16, but the vertical number of blocks can be determined according to the number of vertices of the subdivided base mesh. Transform coefficients can be packed by aligning them with Morton codes within a single block. The packed images generate displacement videos for each GoF unit, and these displacement videos can be encoded using a conventional video compression codec in a video encoder (419 or 519).

[0175] Referring to FIG. 8, a base mesh (original) may include vertices and edges for LoD (Level of Detail) 0. A first subdivision mesh generated by dividing (or subdividing) the base mesh includes vertices generated by further dividing (or subdividing) edges of the base mesh. The first subdivision mesh includes vertices for LoD 0 and vertices for LoD 1. LoD 1 includes the subdivided vertices and the vertices of the base mesh (LoD 0). The first subdivision mesh may be further divided (or subdivided) to generate a second subdivision mesh. The second subdivision mesh includes LoD 2. LoD 2 includes base mesh vertices (LoD 0), LoD 1 including vertices further divided (or subdivided) from LoD 0, and vertices further divided (or subdivided) from LoD 1. LoD is a level of detail that indicates the degree of detail of mesh data content. As the level index increases, the distance between vertices becomes closer and the level of detail increases. In other words, the smaller the LoD value, the lower the detail of the mesh data content, and the larger the LoD value, the higher the detail of the mesh data content. LoD N contains the vertices included in the previous LoDN-1 as is. When a mesh (or vertex) is further divided through subdivision, the mesh can be encoded based on a prediction and / or update method by considering the previous vertices v1, v2, and the subdivided vertex v. Instead of directly encoding information about the current LoD N, a residual value between the previous LoD N-1 can be generated and the mesh can be encoded using the residual value to reduce the bitstream size. The prediction process refers to the operation of predicting the current vertex v using the previous vertices v1 and v2. Since adjacent subdivision meshes have similar data, this property can be utilized for efficient encoding.Current vertex position information is predicted as a residual of previous vertex position information, and the previous vertex position information is updated through the residual. In the present disclosure, vertex, apex, and point may be used with the same meaning. In addition, LoDs may be defined during the subdivision process of the base mesh. According to embodiments, the subdivision process of the base mesh may be performed in the pre-processor (200) or in a separate component / module.

[0176] Referring to FIG. 9, a vertex has a transform coefficient (also called a lifting coefficient) generated through a lifting transformation. The transform coefficient of a vertex related to a lifting transformation can be packed into an image by an image packing unit (418 or 518) and then encoded by a video encoder (419 or 519).

[0177] Figure 10 illustrates an attribute transfer process of a V-MESH compression method according to embodiments.

[0178] According to the embodiments, FIG. 10 shows the detailed operation of the attribute transfer (425 or 525) of the encoding of FIG. 6, FIG. 7, etc.

[0179] Encoding according to embodiments includes attribute map encoding. According to embodiments, attribute map encoding may be performed in the video encoder (428) of FIG. 6 or the video encoder (528) of FIG. 7.

[0180] According to embodiments, in the present disclosure, the encoder compresses information about the input mesh through base mesh encoding (i.e., intra encoding), motion field encoding (i.e., inter encoding), and displacement encoding. In the encoding process, the compressed input mesh is restored through base mesh decoding (intra frame), motion field decoding (inter frame), and displacement video decoding processes, and the restored result, the reconstructed deformed mesh (hereinafter referred to as Recon. deformed mesh), is used to compress the input attribute map as shown in FIGS. 6 and 7. The reconstructed deformed mesh (Recon. deformed mesh) has position information of vertices, texture coordinates, and corresponding connection information, but does not have color information corresponding to the texture coordinates. Therefore, as shown in Fig. 10, in the V-Mesh compression method, a new attribute map having color information corresponding to the texture coordinates of the reconstructed deformed mesh is regenerated through the attribute transfer process of attribute transfer (425 or 525).

[0181] According to embodiments, attribute transfer (425 or 525) first checks whether each point P(u, v) of a 2D texture domain belongs to a texture triangle of a reconstructed deformed mesh, and if it exists in a texture triangle T, the barycentric coordinate of P(u, v) according to the triangle T ( , , ) is calculated. And the 3D vertex positions of triangle T and ( , , ) is used to compute the 3D coordinates M(x, y, z) of P(u, v). Find the vertex coordinates M'(x', y', z') and the triangle T' containing this vertex that corresponds to the most similar position to the calculated M(x, y, z) in the input mesh domain. Then, the center of mass coordinates of M'(x', y', z') in this triangle T' ( ', ', ') is calculated. The texture coordinates corresponding to the three vertices of Triangle T' and ( ', ', ') is used to calculate the texture coordinates (u', v'), and the color information corresponding to these coordinates is found in the input attribute map. The color information found in this way is immediately assigned to the pixel location (u, v) of the new attribute map. If P(u, v) does not belong to any triangle, the pixel at that location in the new attribute map can be filled with a color value using a padding algorithm, such as the push-pull algorithm of push-pull padding (426 or 526).

[0182] The new attribute map generated through attribute transfer (425 or 525) is grouped into GoF units to form an attribute map video, which is compressed using the video codec of the video encoder (428 or 528).

[0183] Referring to Figure 10, the reference relationship between the input mesh, the input attribute map, the reconstructed deformed mesh, and the regenerated attribute map can be seen.

[0184] The decoding process of Fig. 1 can perform the reverse process of the corresponding process of the encoding process of Fig. 1. The specific decoding process is as follows.

[0185] FIG. 11 illustrates an intra-frame decoding (or intra-decoding) process of V-Mesh technology according to embodiments.

[0186] Fig. 11 illustrates the configuration and operation of the mesh video decoder (113) of the receiving device of Fig. 1. In addition, Fig. 11 can restore mesh data by performing the reverse process of the intra-frame encoding process of Fig. 6. Each component for the intra-frame decoding process of Fig. 11 corresponds to hardware, software, and / or a combination thereof.

[0187] First, the bitstream (i.e., compressed bitstream) received and input to the demultiplexer (611) of the intra frame decoding unit (610) can be separated into a mesh substream, a displacement substream, an attribute map substream, and a substream containing patch information of the mesh, such as V-PCC / V3C. The term V-PCC (Video-based Point Cloud Compression) used in this document can be used with the same meaning as V3C (Visual Volumetric Video-based Coding), and the two terms can be used interchangeably. Therefore, the term V-PCC in this document can be interpreted as the term V3C.

[0188] According to embodiments, the mesh sub-stream may be input to a static mesh decoder (612) and decoded, the displacement sub-stream may be input to a video decoder (613) and decoded, and the attribute map sub-stream may be input to a video decoder (617) and decoded.

[0189] According to embodiments, the mesh sub-stream is decoded through a decoder (612) of a static mesh codec used in encoding, such as Google Draco, and as a result, a reconstructed quantized base mesh, for example, connection information, vertex geometry information, vertex texture coordinates, etc. of the base mesh can be reconstructed.

[0190] According to embodiments, the displacement sub-stream is decoded into displacement video through a decoder (613) of a video compression codec used in encoding, and is restored as displacement information for each vertex (i.e., Recon. displacements) through an image unpacking process of an image unpacking unit (614), an inverse quantization process of an inverse quantizer (615), and an inverse transform process of an inverse linear lifting unit (616).

[0191] According to embodiments, the base mesh restored by the static mesh decoder (612) is inverse quantized by the inverse quantizer (620) and then output to the mesh restoration unit (630). The mesh restoration unit (630) reconstructs and restores the deformed mesh (i.e., decoded mesh) through the restored displacement output from the inverse linear lifting unit (616) and the restored base mesh output from the inverse quantizer (620). That is, the inverse quantized restored base mesh is combined with the restored displacement information to generate the final decoded mesh. In the present disclosure, the final decoded mesh is referred to as a reconstructed deformed mesh.

[0192] According to embodiments, an attribute map sub-stream is decoded through a decoder (617) corresponding to a video compression codec used in encoding, and then restored to a final attribute map (i.e., decoded attribute map) through a color conversion unit (640) through processes such as color format conversion and color space conversion.

[0193] According to embodiments, the restored decoded mesh and decoded attribute map can be utilized by the receiver as final mesh data that can be utilized by the user.

[0194] Referring to FIG. 11, the received compressed bitstream includes patch information, a mesh substream, a displacement substream, and an attribute map substream. A substream is interpreted as a term referring to a part of a bitstream included in a bitstream. The bitstream includes patch information (data), mesh information (data), displacement information (data), and attribute map information (data).

[0195] As described above, the decoder of FIG. 11 performs the following intra-frame decoding operations. The static mesh decoder (612) decodes the mesh sub-stream to generate a reconstructed quantized base mesh, and the inverse quantizer (620) applies the quantization parameters of the quantizer inversely to generate the reconstructed base mesh. The video decoder (613) decodes the displacement sub-stream, the image unpacking unit (614) unpacks the images of the decoded displacement video, and the inverse quantizer (615) inversely quantizes the quantized images. The inverse linear lifting unit (616) applies a lifting transform in the reverse process of the encoder to generate the reconstructed displacement. The mesh restoration unit (630) generates a reconstructed deformed mesh based on the reconstructed base mesh and the reconstructed displacement. The video decoder (617) decodes the attribute map sub-stream, and the color conversion unit (640) converts the color format and / or space of the decoded attribute map to generate a decoded attribute map.

[0196] Figure 12 illustrates the inter-frame decoding (or inter-decoding) process of V-Mesh technology.

[0197] Fig. 12 illustrates the configuration and operation of the mesh video decoder (113) of the receiving device of Fig. 1. In addition, Fig. 12 can restore mesh data by performing the reverse process of the inter-frame encoding process of Fig. 7. Each component for the inter-frame decoding process of Fig. 12 corresponds to hardware, software, and / or a combination thereof.

[0198] First, the bitstream received and input to the demultiplexer (711) of the intra frame decoding unit (710) can be separated into a motion sub-stream (also called a motion sub-stream or motion vector sub-stream), a displacement sub-stream, an attribute map sub-stream, and a sub-stream including patch information of a mesh such as V3C / V-PCC.

[0199] According to embodiments, a motion sub-stream may be input to a motion decoder (712) and decoded, a displacement sub-stream may be input to a video decoder (713) and decoded, and an attribute map sub-stream may be input to a video decoder (717) and decoded.

[0200] According to embodiments, a motion sub-stream is decoded through entropy decoding and inverse prediction processes in a motion decoder (712) and restored into motion information (or motion vector information). A base mesh restoration unit (718) combines the restored motion information with a reference base mesh that has already been restored and stored to generate a reconstructed quantized base mesh for the current frame. An inverse quantizer (720) applies inverse quantization to the restored quantized base mesh to generate a reconstructed base mesh. A video decoder (713) decodes a displacement sub-stream, an image unpacking unit (714) unpacks an image of the decoded displacement video, and an inverse quantizer (715) inversely quantizes a quantized image. The reverse linear lifting unit (716) applies a lifting transformation in the reverse process of the encoder to generate a restored displacement. The mesh restoration unit (730) generates a reconstructed deformed mesh, i.e., a final decoded mesh, based on the restored base mesh and the restored displacement.

[0201] According to embodiments, the video decoder (717) decodes the attribute map sub-stream in the same manner as intra decoding, and the color conversion unit (740) converts the color format and / or space of the decoded attribute map to generate a decoded attribute map. The decoded mesh and the decoded attribute map can be utilized by the receiver as final mesh data that can be utilized by the user.

[0202] Referring to Fig. 12, the bitstream includes motion information (also called motion vectors), displacement, and an attribute map. Since Fig. 12 performs inter-frame decoding, it further includes a process of decoding inter-frame motion information. The motion information is decoded, and a restored quantized base mesh for the motion information is generated based on the reference base mesh, thereby generating a restored base mesh. For a description of the operation of Fig. 12, which is identical to that of Fig. 11, refer to the description of Fig. 11.

[0203] Fig. 13 illustrates a mesh data transmission device according to embodiments.

[0204] FIG. 13 corresponds to the transmitting device (100) or mesh video encoder (102) of FIG. 1, the encoder (preprocessor and encoder) of FIG. 2, FIG. 6, or FIG. 7, and / or a transmitting encoding device corresponding thereto. Each component of FIG. 13 corresponds to hardware, software, a processor, and / or a combination thereof.

[0205] The operation process of a transmitter for compressing and transmitting dynamic mesh data using V-Mesh compression technology may be as shown in Fig. 13. The transmitter of Fig. 13 may perform an intra-frame encoding (or intra-encoding or intra-screen encoding) process and / or an inter-frame encoding (or inter-encoding or inter-screen encoding) process.

[0206] The pre-processor (811) receives the original mesh as input and generates a simplified mesh (decimated mesh) (or base mesh) and a fitted decimated mesh (or subdivision). Simplification can be performed based on the target number of vertices or target number of polygons that constitute the mesh. Parameterization, which generates texture coordinates and texture connection information per vertex, can be performed on the simplified mesh. For example, parameterization is a process of mapping a 3D surface to a texture domain for the decimated mesh. If parameterization is performed using the UVAtlas tool, mapping information is generated that can identify where each vertex of the decimated mesh can be mapped on a 2D image. The mapping information is expressed and stored as texture coordinates, and the final base mesh is generated through this process. In addition, the work of quantizing the mesh information in floating-point form into fixed-point form can be performed. This result can be output as a base mesh to a motion vector encoder (813) or a static mesh encoder (814) through a switching unit (812). The pre-processor (811) can perform mesh subdivision on the base mesh to generate additional vertices. Depending on the subdivision method, vertex connection information, texture coordinates, and texture coordinate connection information including the added vertices can be generated. The pre-processor (811) can generate a fitted subdivided mesh by adjusting the vertex positions so that the subdivided mesh becomes similar to the original mesh.

[0207] According to embodiments, the base mesh is output to a motion vector encoder (813) via a switching unit (812) when performing inter-encoding for the corresponding mesh frame, and is output to a static mesh encoder (814) via a switching unit (812) when performing intra-encoding for the corresponding mesh frame. The motion vector encoder (813) may be referred to as a motion encoder.

[0208] For example, when performing intra-encoding (or intra-frame encoding) on ​​the corresponding mesh frame, the base mesh can be compressed through a static mesh encoder (814). In this case, encoding can be performed on connection information, vertex geometry information, vertex texture information, normal information, etc. of the base mesh. The base mesh bitstream generated through encoding is transmitted to a multiplexer (823).

[0209] As another example, when performing inter-encoding (or inter-frame encoding) for the corresponding mesh frame, the motion vector encoder (813) can receive a base mesh and a reference reconstructed base mesh (or a reconstructed quantized reference base mesh) as input, calculate a motion vector between the two meshes, and encode the value. In addition, the motion vector encoder (813) can perform prediction based on connection information using a previously encoded / decoded motion vector as a predictor, and encode a residual motion vector obtained by subtracting the predicted motion vector from the current motion vector. The motion vector bitstream generated through encoding is transmitted to the multiplexer (823).

[0210] The base mesh restoration unit (815) can receive the base mesh encoded by the static mesh encoder (814) or the motion vector encoded by the motion vector encoder (813) and generate a reconstructed base mesh. For example, the base mesh restoration unit (815) can perform static mesh decoding on the base mesh encoded by the static mesh encoder (814) to restore the base mesh. At this time, quantization can be applied before the static mesh decoding, and inverse quantization can be applied after the static mesh decoding. As another example, the base mesh restoration unit (815) can restore the base mesh based on the reconstructed quantized reference base mesh and the motion vector encoded by the motion vector encoder (813). The reconstructed base mesh is output to the displacement calculation unit (816) and the mesh restoration unit (820).

[0211] The displacement calculation unit (816) can perform mesh refinement on the restored base mesh. The displacement calculation unit (816) can calculate a displacement vector, which is a difference value between the vertex positions of the restored base mesh and the fitted subdivision (or refined) mesh generated by the pre-processor (811). At this time, the displacement vector can be calculated as many times as the number of vertices of the refined mesh. The displacement calculation unit (816) can convert the displacement vector calculated in the 3D Cartesian coordinate system into a local coordinate system based on the normal vector of each vertex.

[0212] The displacement vector video generation unit (817) may include a linear lifting unit, a quantizer, and an image packing unit. That is, in the displacement vector video generation unit (817), the linear lifting unit may transform the displacement vector for effective encoding. The transformation may be performed by a lifting transformation, a wavelet transformation, etc., according to embodiments. In addition, quantization may be performed in a quantizer on the transformed displacement vector value, i.e., the transform coefficient. At this time, a different quantization parameter may be applied to each axis of the transform coefficient, and the quantization parameter may be derived according to an encoder / decoder agreement. The transformed and quantized displacement vector information may be packed into a 2D image in the image packing unit. The displacement vector video generation unit (817) may generate a displacement vector video by bundling packed 2D images for each frame, and the displacement vector video may be generated for each GoF (Group of Frame) unit of the input mesh.

[0213] The displacement vector video encoder (818) can encode the generated displacement vector video using a video compression codec. The generated displacement vector video bitstream is transmitted to a multiplexer (823).

[0214] The displacement vector restoration unit (819) may include a video decoder, an image unpacking unit, an inverse quantizer, and an inverse linear lifting unit. That is, the displacement vector restoration unit (819) performs decoding on an encoded displacement vector in the video decoder, performs image unpacking in the image unpacking unit, performs inverse quantization in the inverse quantizer, and then performs inverse transformation in the inverse linear lifting unit to restore the displacement vector. The restored displacement vector is output to the mesh restoration unit (820). The mesh restoration unit (820) restores a deformed mesh based on the base mesh restored by the base mesh restoration unit (815) and the displacement vector restored by the displacement vector restoration unit (819). The restored mesh (or referred to as a restored deformed mesh) has restored vertices, connection information between vertices, texture coordinates, and connection information between texture coordinates.

[0215] The texture map video generation unit (821) can regenerate a texture map based on the texture map (or attribute map) of the original mesh and the restored deformed mesh output from the mesh restoration unit (820). According to embodiments, the texture map video generation unit (821) can assign color information per vertex of the texture map of the original mesh to the texture coordinates of the restored deformed mesh. According to embodiments, the texture map video generation unit (821) can generate a texture map video by grouping the regenerated texture maps by GoF unit for each frame.

[0216] The generated texture map video can be encoded using a video compression codec of a texture map video encoder (822). The texture map video bitstream generated through encoding is transmitted to a multiplexer (823).

[0217] A multiplexer (823) multiplexes a motion vector bitstream (e.g., in case of inter encoding), a base mesh bitstream (e.g., in case of intra encoding), a displacement vector bitstream, and a texture map bitstream into a single bitstream. The single bitstream can be transmitted to a receiver via a transmitter (824). Alternatively, the motion vector bitstream, the base mesh bitstream, the displacement vector bitstream, and the texture map bitstream can be generated as a file with one or more track data or encapsulated into segments and transmitted to a receiver via the transmitter (824).

[0218] Referring to FIG. 13, a transmitting device (encoder) can encode a mesh in an intra-frame or inter-frame manner. A transmitting device according to intra-encoding can generate a base mesh, a displacement vector (or referred to as displacement), and a texture map (or referred to as attribute map). A transmitting device according to inter-encoding can generate a motion vector (or referred to as motion), a displacement vector (or referred to as displacement), and a texture map (or referred to as attribute map). The texture map obtained from the data input unit is generated and encoded based on the restored mesh. Displacement is generated and encoded through the difference in vertex positions between the base mesh and the divided (or subdivided or subdivided) mesh. More specifically, the displacement is the difference in position between the fitted sub-divided mesh and the sub-divided restored base mesh, i.e., the difference in vertex positions between the two meshes. In addition, the base mesh is generated by simplifying and encoding the original mesh through pre-processing. Motion is generated as motion vectors for the mesh of the current frame based on the reference base mesh of the previous frame.

[0219] Fig. 14 illustrates a mesh data receiving device according to embodiments.

[0220] Fig. 14 corresponds to the receiving device (110) or mesh video decoder (113) of Fig. 1, the decoder of Fig. 11 or Fig. 12, and / or the receiving decoding device corresponding thereto. Each component of Fig. 14 corresponds to hardware, software, a processor, and / or a combination thereof. The receiving (decoding) operation of Fig. 14 may follow the reverse process of the corresponding process of the transmitting (encoding) operation of Fig. 13.

[0221] The bitstream of the mesh data received by the receiver (910) is demultiplexed into a compressed motion vector bitstream (e.g., inter decoding) or a base mesh bitstream (e.g., intra decoding), a displacement vector bitstream, and a texture map bitstream after file / segment decapsulation in the demultiplexer (911). For example, if the current mesh has inter-screen encoding (i.e., inter encoding) applied, the motion vector bitstream is received, demultiplexed, and then output to the motion vector decoder (913) via the switching unit (912). As another example, if the current mesh has intra-screen encoding (i.e., intra encoding) applied, the base mesh bitstream is received, demultiplexed, and then output to the static mesh decoder (914) via the switching unit (912). Here, the motion vector decoder (913) may be referred to as a motion decoder.

[0222] According to embodiments, if the current mesh has inter-screen encoding applied according to frame header information, the motion vector decoder (913) can perform decoding on the motion vector bitstream. According to embodiments, the motion vector decoder (913) can reconstruct the final motion vector by adding the previously decoded motion vector as a predictor to the residual motion vector decoded from the bitstream.

[0223] According to embodiments, if the current mesh has been subjected to in-screen encoding according to frame header information, the static mesh decoder (914) can decode the base mesh bitstream to restore connection information, vertex geometry information, texture coordinates, normal information, etc. of the base mesh.

[0224] According to embodiments, the base mesh restoration unit (915) can restore the current base mesh based on the decoded motion vector or the decoded base mesh. For example, if the current mesh has inter-screen encoding applied, the base mesh restoration unit (915) can generate a restored base mesh by adding the decoded motion vector to the reference base mesh and then performing inverse quantization. As another example, if the current mesh has intra-screen encoding applied, the base mesh restoration unit (915) can generate a restored base mesh by performing inverse quantization on the base mesh decoded through the static mesh decoder (914).

[0225] According to embodiments, the displacement vector video decoder (917) can decode the displacement vector bitstream as a video bitstream using a video codec.

[0226] According to embodiments, the displacement vector restoration unit (918) extracts displacement vector transform coefficients from the decoded displacement vector video, and restores the displacement vector by applying inverse quantization and inverse transformation processes to the extracted displacement vector transform coefficients. To this end, the displacement vector restoration unit (918) may include an image unpacking unit, an inverse quantizer, and an inverse linear lifting unit. If the restored displacement vector is a value in a local coordinate system, a process of inversely transforming it into a Cartesian coordinate system may be performed.

[0227] The mesh restoration unit (916) can generate additional vertices by performing subdivision on the restored base mesh. Through subdivision, vertex connection information including the added vertices, texture coordinates, and texture coordinate connection information can be generated. At this time, the mesh restoration unit (916) can generate a final restored mesh (or a restored deformed mesh) by combining the subdivided restored base mesh with the restored displacement vector.

[0228] According to embodiments, the texture map video decoder (919) can decode the texture map bitstream as a video bitstream using a video codec to restore the texture map. The restored texture map has color information for each vertex contained in the restored mesh, and the color value of each vertex can be obtained from the texture map using the texture coordinates of each vertex.

[0229] According to embodiments, the mesh restored by the mesh restoration unit (916) and the texture map restored by the texture map video decoder (919) are shown to the user through a rendering process in the mesh data renderer (920).

[0230] Referring to FIG. 14, a receiving device (decoder) can decode a mesh in an intra-frame or inter-frame manner. A receiving device according to intra-decoding can receive a base mesh, a displacement vector (or referred to as displacement), a texture map (or referred to as attribute map), and render mesh data based on the restored mesh and the restored texture map. A receiving device according to inter-decoding can receive a motion vector (or referred to as motion), a displacement vector (or referred to as displacement), a texture map (or referred to as attribute map), and render mesh data based on the restored mesh and the restored texture map.

[0231] A mesh data transmission device and method according to embodiments may pre-process mesh data, encode the pre-processed mesh data, and transmit a bitstream including the encoded mesh data. A point mesh data reception device and method according to embodiments may receive a bitstream including mesh data and decode the mesh data. The mesh data transmission and reception method / device according to embodiments may be abbreviated as the method / device according to embodiments. The mesh data transmission and reception method / device according to embodiments may also be referred to as a 3D data transmission and reception method / device or a point cloud data transmission and reception method / device.

[0232] As described above, in the V-Mesh method, when performing intra-encoding (or intra frame encoding) on ​​the corresponding mesh frame, the base mesh can be encoded through a static mesh encoder. In this case, encoding can be performed on connection information, vertex geometry information, vertex texture information, normal information, etc. of the base mesh. In addition, when performing inter-encoding (or inter frame encoding) on ​​the corresponding mesh frame, the motion vector between the base mesh and the reference restored base mesh (or the restored quantized reference base mesh) can be calculated and its value can be encoded through a motion vector encoder.

[0233] In particular, in the current V-DMC standard, the coding type is determined as intra mode or inter mode according to the one-to-one relationship for the base mesh, regardless of the coding structure of the displacement vector and texture map. That is, in intra mode, the static mesh is coded, and in inter mode, the motion vector is coded. At this time, since the context information (called context or context information) is initialized for each frame and then used, unnecessary initialization is performed if the same information exists among the context information between frames.

[0234] To improve this, the present disclosure can improve the encoding performance of motion vectors and displacement vectors of static meshes and dynamic meshes by storing the context information and retrieving and using the context information stored in the previous frame when coding the next static mesh, motion vector, and displacement vector. That is, the present disclosure proposes a method of storing entropy context information when coding the motion vectors of static meshes and dynamic meshes and using the stored context information when coding the next static mesh or motion vector. In this way, the present disclosure can reduce the bit amount and improve the encoding performance of motion vectors of static meshes and dynamic meshes by using the context information stored in advance without initializing the context information for each frame. In addition, when the coding type of the displacement vector is arithmetic coding, the present disclosure can improve the encoding performance of the displacement vector by also storing and using the context information of the displacement vector in the same way.

[0235] In this way, the present disclosure relates to V-DMC, a method of compressing 3D dynamic mesh data using an existing 2D video codec, and describes a method of storing and encoding / decoding contextual information during the coding process of motion vectors and displacement vectors of a static mesh and a dynamic mesh, and syntax and semantics information related thereto.

[0236] In the present disclosure, geometric information (or referred to as geometry or geometric data) is one of the elements that constitute a mesh, and includes vertices (or points), edges, polygons, etc. Here, a vertex defines a position in 3D space, an edge represents connection information between vertices, and a polygon forms the surface of the mesh with a combination of edges and vertices. That is, each vertex that constitutes the mesh represents a position in 3D space, and is expressed, for example, by x, y, z coordinates (i.e., canonical coordinate system). And, a polygon may be a triangle or a square. That is, geometry forms the skeleton of a 3D model, thereby defining the shape of the model and visually expressing it when rendered.

[0237] In the present disclosure, vertex, apex, and point may be used interchangeably. That is, a vertex has coordinates in 3D space, and a triangular or quadrilateral polygon can be created through connections between multiple vertices. In addition, V-DMC referred to in the present disclosure may also be referred to as V-mesh, and the two terms are expressions used interchangeably.

[0238] In the present disclosure, displacement information can be acquired based on a refined mesh (or referred to as a sub-mesh). That is, a fitting process is performed to make the input mesh and the sub-mesh similar to each other, and the difference in the positions of each vertex of the fitted subdivision mesh and the refined restored base mesh is generated by performing refinement on the restored base mesh. The present disclosure refers to this vertex position difference value as a displacement vector. In the present disclosure, the displacement vector may be used interchangeably with the same meaning as displacement, displacement information, or displacement data. In addition, the displacement video may be used interchangeably with the same meaning as displacement vector video or displacement vector transformation coefficient video, and the displacement vector may be used interchangeably with the same meaning as displacement vector transformation coefficient or displacement vector coefficient.

[0239] The following describes the process of storing entropy context information when coding motion vectors of static meshes or dynamic meshes and using the stored context information when coding the next static mesh or motion vector, with reference to the drawings. In addition, when the coding type of the displacement vector is arithmetic coding, the process of storing and using the context information of the displacement vector in the same manner will be described with reference to the drawings.

[0240] Fig. 15 illustrates a transmitting device according to embodiments. The transmitting device of Fig. 15 may be referred to as a mesh data transmitting device or an encoder or an encoder of a transmitting device or a V-Mesh encoder or a dynamic mesh encoder.

[0241] FIG. 15 corresponds to the transmitting device (100) or the mesh video encoder (102) of FIG. 1, the encoder (pre-processor and encoder) of FIG. 2, FIG. 6, or FIG. 7, the transmitting device of FIG. 13, and / or the transmitting encoding device corresponding thereto. Therefore, parts not described in FIG. 15 will refer to the description of the transmitting device (100) or the mesh video encoder (102) of FIG. 1, the encoder (pre-processor and encoder) of FIG. 2, FIG. 6, or FIG. 7, and the transmitting device of FIG. 13. The elements of the transmitting device illustrated in FIG. 15 may be implemented by hardware, software, a processor connected to a memory, and / or a combination thereof. That is, the elements of the transmitting device of FIG. 15 may be implemented by hardware, software, firmware, or a combination thereof, including one or more processors or integrated circuits configured to communicate with one or more memories, although not illustrated in the drawing. One or more processors may perform at least one of the operations and / or functions of the elements of the transmitting device of FIG. 15 described above. Furthermore, one or more processors may operate or execute a set of software programs and / or instructions for performing the operations and / or functions of the elements of the transmitting device of FIG. 15. The execution order of each block in FIG. 15 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0242] In the present disclosure, the operation process of a transmitter for compressing and transmitting dynamic mesh data using V-Mesh compression technology may be as shown in FIG. 15. The transmitter of FIG. 15 may support both an intra-frame encoding (or intra-encoding or intra-screen encoding) process and / or an inter-frame encoding (or inter-encoding or inter-screen encoding) process.

[0243] In Fig. 15, the mesh simplification unit (11011) simplifies the input original mesh through a mesh simplification algorithm to generate a base mesh (or simplified base mesh or simplified mesh). At this time, mesh simplification can be performed based on the number of target vertices or target polygons constituting the mesh. For example, a method such as decimation can be used as a mesh simplification algorithm that simplifies the original mesh. That is, the decimation method can be a process of selecting vertices to be removed from the original mesh using a certain reference point, and then removing the selected vertices and the triangles connected to the selected vertices.

[0244] That is, the mesh simplification unit (11011) can simplify the input mesh by the target number of vertices or the target number of faces. At this time, the simplification process can be performed through various methods such as triangle collapse and edge collapse.

[0245] According to embodiments, the base mesh simplified in the mesh simplification unit (11011) is provided to the mesh parameterization unit (11012) and the mesh refinement unit (11018).

[0246] The mesh parameterization unit (11012) performs a process of mapping a 3D surface to a texture domain for a simplified mesh (decimated mesh). That is, the mesh parameterization unit (11012) generates texture coordinates and texture connection information of the input mesh. In one embodiment, the mesh parameterization unit (11012) may perform parameterization using a UV Atlas tool. Through this process, mapping information is generated regarding which location on a 2D image each vertex of the simplified mesh (decimated mesh) can be mapped to. The mapping information is expressed and stored as texture coordinates, and through this process, the final base mesh is generated. That is, the mesh parameterization unit (11012) performs parameterization to generate texture coordinates (UV coordinates) and texture connection information per vertex of the input mesh (i.e., simplified mesh or simplified base mesh).

[0247] The final base mesh (or base mesh having a texture map) generated by the parameterization unit (11012) is input to the mesh quantization unit (11013) and quantized. According to embodiments, the mesh quantization unit (11013) may perform the task of quantizing floating-point type mesh information (e.g., geometry information (x, y, z) or / and texture coordinates (u, v), normal information (nx, ny, nz), etc.) into fixed-point type. That is, the mesh quantization unit (11013) may quantize vertex coordinates and texture coordinates of the base mesh. According to embodiments, quantization for specific components may be omitted.

[0248] The above mesh subdivision unit (11018) subdivides the base mesh simplified by the mesh simplification unit (11011). That is, the mesh subdivision unit (11018) can perform mesh subdivision on the base mesh to generate additional vertices. Depending on the subdivision method, vertex connection information, texture coordinates, and texture coordinate connection information including the added vertices can be generated. At this time, depending on the subdivision method, geometry information connection information, texture coordinate connection information, and texture coordinates can be implicitly derived and generated. According to embodiments, the mesh subdivision unit (11018) can perform subdivision through a method such as mid-edge, Loop, or Catmul&Clark.

[0249] In the present disclosure, the level and LoD can be defined in the process of subdividing the base mesh as described in FIG. 8. In this case, subdividing the base mesh is performed in a mesh subdividing unit (11018), as an example.

[0250] More specifically, in the mesh refinement unit (11018), mesh refinement can be performed n times by user parameters or a promise of the encoder (i.e., transmitting device) / decoder (i.e., receiving device). According to embodiments, the vertices of the base mesh are vertices of level 0 (R0), the vertices newly generated by performing refinement once are vertices of level 1 (R1), … the vertices newly generated by performing refinement n times are vertices of level n (R n ) can be defined as vertices of level 0 (R0). For convenience of explanation, the present disclosure refers to vertices of level 0 (R0) as level 0 vertices, vertices of level 1 (R1) as level 1 vertices, …, level n (R n ) can be called level n vertices. Also, level 0 is called the 0th level (R0), level 1 is called the 1st level (R1), … level n is called the nth level (R n ) can be called.

[0251] According to embodiments, LoD0, LoD1, … LoDn can be defined as in the following mathematical expression 1.

[0252] [Mathematical Formula 1]

[0253] LoD0 = R0

[0254] LoD1 = R0+ R1= LoD0 + R1

[0255] …

[0256] LoDn = R0+ R1+, … + R n = LoDn-1 + R n

[0257] That is, the base mesh includes vertices for LoD0. And, the first subdivision mesh generated by subdividing the base mesh once includes vertices for LoD1. That is, LoD1 includes vertices of the base mesh (i.e., R0 vertices) and vertices of a newly generated level 1 by subdividing the base mesh (i.e., R1 vertices). In addition, the second subdivision mesh generated by subdividing the first subdivision mesh again includes vertices for LoD2. That is, LoD2 includes vertices of the base mesh (i.e., R0 vertices), vertices of a newly generated level 1 by subdividing the base mesh (i.e., R1 vertices), and vertices of a newly generated level 2 by subdividing the first subdivision mesh (i.e., R2 vertices).

[0258] In this way, LoD represents the level of detail of mesh data content, and as the LoD (or level) index increases, the distance between vertices becomes closer and the level of detail increases. In other words, the smaller the LoD value, the lower the detail of the mesh data content, and the larger the LoD value, the higher the detail of the mesh data content.

[0259] According to embodiments, the mesh fitting unit (11019) can perform fitting by adjusting vertex positions so that the mesh subdivided by the mesh subdivision unit (11018) becomes similar to the original mesh, thereby generating a fitted subdivided mesh.

[0260] The present disclosure may be referred to as a pre-processor including a mesh simplification unit (11011), a mesh parameterization unit (11012), a mesh refinement unit (11018), and a mesh fitting unit (11019). At this time, the mesh simplification unit (11011), the mesh parameterization unit (11012), the mesh refinement unit (11018), and the mesh fitting unit (11019) may be omitted, and when the corresponding process is omitted, the original mesh is output to the mesh quantization unit (11013). According to embodiments, the pre-processor may further include a displacement vector calculation unit (11020). At this time, coordinate information of the original mesh can be applied as input to the displacement vector calculation unit (11020), and according to embodiments, the displacement vector encoding process (i.e., displacement vector calculation unit (11020), displacement vector coordinate system conversion unit (11021), displacement vector encoder (11022)) can be omitted.

[0261] According to embodiments, the quantized base mesh in the mesh quantization unit (11013) can be output to a motion vector encoder (11015) or a static mesh encoder (11016) through a switching unit (11014).

[0262] According to embodiments, the base mesh is output to a motion vector encoder (11015) through a switching unit (11014) when inter-encoding (or the type of the current base mesh is inter type) is performed for the corresponding mesh frame, and is output to a static mesh encoder (11016) through a switching unit (11014) when intra-encoding (or the type of the current base mesh is intra type) is performed for the corresponding mesh frame. The motion vector encoder (11015) may be referred to as a motion encoder.

[0263] For example, when performing intra encoding or intra frame encoding for the corresponding mesh frame, the base mesh can be compressed through a static mesh encoder (11016). In this case, encoding can be performed on connection information, vertex geometry information, vertex texture information, normal information, etc. of the base mesh. That is, vertex coordinates, vertex connection information, texture coordinates, texture connection information, etc. of the mesh can be encoded in the static mesh encoder (11016). The base mesh bitstream generated through encoding is transmitted to a multiplexer (not shown).

[0264] As another example, when performing inter-encoding (or inter-frame encoding) on ​​the corresponding mesh frame, the motion vector encoder (11015) can receive a base mesh and a reference reconstructed base mesh (or a reconstructed quantized reference base mesh) as input, calculate a motion vector between the two meshes, and encode the value. In addition, the motion vector encoder (11015) can perform prediction based on connection information using a previously encoded / decoded motion vector as a predictor, and entropy-encode a differential motion vector (or residual motion vector) obtained by subtracting the predicted motion vector from the current motion vector. In the present disclosure, motion vector encoding can be performed on a vertex basis, a sub-mesh basis, or a sub-group basis. Then, the motion vector bitstream generated through the encoding is transmitted to a multiplexer (not shown) as a base mesh bitstream. That is, in the case of intra-frame encoding, the static mesh bitstream is input to the multiplexer as a base mesh bitstream, and in the case of inter-frame encoding, the motion vector bitstream is input to the multiplexer as a base mesh bitstream.

[0265] In FIG. 15, the base mesh decoder (11017) can receive a base mesh encoded by a static mesh encoder (11016) or a motion vector encoded by a motion vector encoder (11015) or a reference base mesh to generate a reconstructed base mesh.

[0266] According to embodiments, the base mesh decoder (11017) performs restoration of the base mesh according to the encoding type of the current mesh or the type of the current base mesh (inter-screen encoding or intra-screen encoding or skip). For example, the base mesh decoder (11017) can restore the base mesh by performing static mesh decoding on the base mesh encoded by the static mesh encoder (11016). At this time, quantization can be applied before the static mesh decoding, and inverse quantization can be applied after the static mesh decoding. That is, when intra-screen encoding is performed, inverse quantization can be performed on the quantized base mesh through the mesh quantization unit (11013) to restore the current base mesh. As another example, the base mesh decoder (11017) can restore the base mesh based on the restored quantized reference base mesh and the motion vector encoded by the motion vector encoder (11015). That is, when inter-screen encoding is performed, the current base mesh can be generated by decoding the motion vector by the motion vector decoding method and then applying (i.e., adding) the decoded motion vector to the reference restoration base mesh. At this time, if the motion vector is not quantized, the motion vector restoration process is omitted and the current base mesh can be restored using the motion vector calculated by the motion vector encoder (11015). As another example, if the type of the current base mesh is skip, the reference base mesh becomes the restored base mesh. The restored base mesh is output to the displacement vector calculation unit (11020) and the mesh dequantization unit (11024).

[0267] According to embodiments, the displacement vector calculation unit (11020) can perform mesh refinement on the restored base mesh. In addition, the displacement vector calculation unit (11020) can calculate a displacement vector, which is a difference value of vertex positions between the restored base mesh that has been refined and the fitted subdivision (or refined) mesh generated by the mesh fitting unit (11019). At this time, the displacement vector can be calculated as many times as the number of vertices of the refined mesh. That is, the displacement vector of the number of vertices of the refined mesh can be calculated through the displacement vector calculation unit (11020).

[0268] According to embodiments, the displacement vector coordinate system conversion unit (11021) can output the vertex displacement vector calculated in a three-dimensional Cartesian coordinate system (i.e., (x, y, z) space) (or canonical coordinate system) as is, or can convert it into a local coordinate system (i.e., normal, tangential, bi-tangential coordinate system) based on the normal vector of each vertex. At this time, the normal vector can be calculated for each subdivided vertex based on the geometry information and connection information of the surrounding vertices.

[0269] According to embodiments, among the normal, tangential, and bi-tangential components, only the normal component can be encoded. This means that when a coordinate system transformation is applied according to the agreement between the encoder and the decoder, only the normal component is always encoded, or the encoder can decide to signal a 1-bit flag (onlyNormFlag) and transmit it to the receiver.

[0270] At this time, the normal vector can be calculated for each subdivided vertex based on the geometric information and / or connection information of the surrounding vertices.

[0271] In addition, whether or not to perform displacement vector coordinate system transformation is determined by an agreement between the encoder / decoder, or whether or not to perform displacement vector coordinate system transformation is determined by transmitting a flag (applyLocalCoord) for whether or not to perform displacement vector coordinate system transformation in units such as sequence, GOF (Group of frame), frame, and sub-mesh.

[0272] That is, in the displacement vector coordinate system conversion unit (11021), whether or not to convert the displacement vector coordinate system can be determined by an encoder (i.e., transmitting device) / decoder (i.e., receiving device) agreement, or the displacement vector coordinate system conversion status flag (applyLocalCoord), which is information that can identify whether or not to convert the displacement vector coordinate system, can be signaled in signaling information (e.g., atlas sequence parameter set, ASPS) and transmitted to the receiving device. For example, if the value of the displacement vector coordinate system conversion status flag (applyLocalCoord) syntax (or field) is 0, the canonical coordinate system is used as is, and if it is 1, it can indicate that conversion to the local coordinate system has been performed.

[0273] According to embodiments, the displacement vector encoder (11022) can perform encoding on the displacement vector or displacement vector transformation coefficient output from the displacement vector calculation unit (11020) or the displacement vector coordinate system transformation unit (11021). In the present disclosure, the displacement vector encoder (11022) can perform encoding through a video codec-based encoder, a zero run length encoder, an arithmetic encoder, or the like. For example, when the encoding method is video codec-based encoding, the displacement vector encoder (11022) can encode the displacement vector or the displacement vector transformation coefficient by packing it into a frame. That is, in the displacement vector encoder (11022), the displacement vector transform coefficients can be packed into a 2D image and then encoded using a 2D video codec (i.e., a video compression codec), or zero run-length encoded, or arithmetic encoded to generate a displacement vector video bitstream.

[0274] In the present disclosure, the displacement vector encoding method may be various. According to embodiments, one of {video codec-based encoding method and zero run-length encoding method}, {zero run-length encoding method and arithmetic encoding method}, {video codec-based encoding method and arithmetic encoding method}, video codec-based encoding method, zero run-length encoding method, and arithmetic encoding method may be determined through information (e.g., displacement_method flag) or an index for identifying the displacement vector encoding method.

[0275] According to embodiments, a displacement vector encoding method may be determined according to a profile defined in an encoder / decoder, and an index (profile_toolset_idc) indicating profile information may be signaled and transmitted to a decoder of a receiving device, and the decoder of the receiving device may determine a displacement vector decoding method according to profile_toolset_idc.

[0276] According to embodiments, a displacement vector video bitstream encoded and generated by a displacement vector encoder (11022) is transmitted to a multiplexer (not shown). According to embodiments, a method for selecting encoding of the displacement vector encoder (11022) may use a displacement vector encoder promised in an encoder (i.e., a transmitting side) / decoder (i.e., a receiving side), or may analyze the characteristics of a displacement vector in an encoder on the transmitting side and transmit the type of a selected displacement vector encoder to a decoder on the receiving side.

[0277] According to embodiments, the displacement vector restoration unit (11023) can restore the displacement vector by performing the reverse process of displacement vector encoding on the displacement vector or displacement vector transform coefficient encoded by the displacement vector encoder (11022). That is, the displacement vector restoration unit (11023) can perform displacement vector depacking depending on the method of encoding the displacement vector, for example, when encoding is based on a video codec. In other words, the bitstream encoded by the 2D video encoder (i.e., the video codec-based encoder) after being packed into a 2D image / video is decoded by the 2D video decoder and depacked. In addition, the displacement vector restoration unit (11023) can additionally perform inverse quantization, inverse transformation, etc. depending on whether quantization and transformation processes are performed during the displacement vector encoding process. For example, inverse quantization is performed on the quantized transform coefficients on which inverse packing has been performed, and inverse transformation is performed to calculate the restored displacement vector.

[0278] According to embodiments, the mesh dequantization unit (11024) can dequantize vertex coordinates or texture coordinates of the restored base mesh by the reverse process of quantization. That is, the mesh dequantization unit (11024) receives restoration geometry information (x, y, z) and / or texture coordinates (u, v) and / or normal information (nx, ny, nz) of the restored base mesh and performs dequantization. If the quantization process is omitted in the mesh quantization unit (11013), the dequantization process is also omitted in the mesh dequantization unit (11024).

[0279] According to embodiments, the mesh restoration unit (11025) can restore a mesh based on a restored displacement vector output from the displacement vector restoration unit (11023) and a restored base mesh (or a dequantized restored base mesh) output from the mesh dequantization unit (11024). More specifically, the mesh restoration unit (11025) can perform subdivision on the restored base mesh output from the mesh dequantization unit (11024) and add the restored displacement vector from the displacement vector restoration unit (11023) to generate a reconstructed deformed mesh. That is, the mesh restoration unit (11025) performs subdivision on the restored base mesh, which is restored by performing dequantization in the mesh dequantization unit (11024), to generate subdivided vertex position information, texture coordinates, and connection information. In addition, the mesh restoration unit generates restored vertex position information by adding a restored displacement vector to the subdivided vertex position information. The mesh restored by the mesh restoration unit (11025) (or referred to as restored mesh or restored deformed mesh) has restored vertices, connection information between vertices, texture coordinates, and connection information between texture coordinates. The restored mesh (or referred to as restored mesh or restored deformed mesh) generated by the mesh restoration unit (11025) is provided to the texture map generation unit (11026).

[0280] According to embodiments, the texture map generation unit (11026) can regenerate the texture map of the current mesh based on the texture map (or attribute map) of the original mesh and the mesh restored by the mesh restoration unit (11025). That is, the texture map generation unit (11026) can generate the texture map of the restored mesh through the relationship between the texture coordinates and connection information of the restored mesh and the original mesh and the texture map of the original mesh.

[0281] According to embodiments, the texture map generation unit (11026) can assign color information per vertex of the texture map of the original mesh to the texture coordinates of the restored base mesh (or the restored deformed mesh). According to embodiments, the texture map generation unit (11026) can generate a texture map (or texture map video) by grouping the regenerated texture maps by GoF unit for each frame.

[0282] The texture map generated by the texture map generation unit (11026) may be encoded by the texture map encoder (11027). That is, the texture map encoder (11027) stacks the texture map generated by the texture map generation unit (11026) in the frame order of the mesh to form a texture map video, and encodes it by the 2D video encoder. For example, the texture map encoder (11027) may encode the texture map using a 2D video codec-based encoder, a zero run length encoder, an entropy coding-based arithmetic encoder, etc. In addition, the texture map encoder (11027) may further perform color space conversion of the texture map. For example, if the color space of the texture map is RGB444, encoding may be performed after conversion to a color space such as YUV420, YUV444, etc. Then, the texture map substream (or texture map video bitstream) generated through texture map encoding is transmitted to a multiplexer (not shown).

[0283] According to embodiments, the type of texture map encoder (11027) may include a video encoder (e.g., VVC, HEVC, etc.), an entropy coding-based encoder, etc. In addition, a method for selecting a texture map encoder (11027) may use a texture map encoder promised in an encoder (i.e., a transmitting side) / decoder (i.e., a receiving side), or may transmit the type of texture map encoder selected by the encoder on the transmitting side to the decoder on the receiving side.

[0284] According to embodiments, a multiplexer (not shown) may multiplex an input base mesh bitstream, a displacement vector bitstream, and a texture map bitstream into a single bitstream and then transmit the multiple bitstreams to a receiving device. Alternatively, the base mesh bitstream, the displacement vector bitstream, and the texture map bitstream may be encapsulated into a file / segment and transmitted to the receiving device.

[0285] According to embodiments, the bitstream multiplexed in the multiplexer may be transmitted over a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as a USB, SD, CD, DVD, Blu-ray, HDD, or SSD.

[0286] Below is a detailed description of the motion vector encoder.

[0287] Fig. 16 is a detailed block diagram showing an example of a motion vector encoder according to embodiments. The motion vector encoder of Fig. 16 corresponds to the motion vector encoder of Fig. 7, Fig. 13, or Fig. 15. Each component of Fig. 16 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in Fig. 16 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0288] In the present disclosure, a motion vector encoder can perform motion vector encoding when a reference base mesh and a current base mesh have a 1-to-1 mapping relationship. According to embodiments, prediction is performed based on already encoded MV(s) around each vertex, a residual MV, which is the difference between the predicted MV and the original motion vector (MV), is generated, and then the residual MV is entropy-coded and transmitted.

[0289] According to an embodiment, the motion vectors of FIG. 16 may be divided into groups by a method defined according to an agreement between the encoder / decoder, and a prediction mode may be determined for each group, and an encoding process may be performed.

[0290] That is, in the group mode decision unit, motion vectors can be divided into one or more groups by a method defined according to an agreement between the encoder / decoder.

[0291] In this disclosure, the definition of a group can vary (1<=group_size<=N, N=number of vertices of the base mesh), and the size of a group can be determined by an encoder / decoder agreement or through explicit signaling.

[0292] According to embodiments, the group mode decision unit may change the vertex order based on the vertex position information of the base mesh for group division into an order such as a 3D Morton code or a 3D Hilbert curve, and then perform group division based on the changed vertices.

[0293] According to embodiments, the present disclosure may signal and transmit to a receiving device a 1-bit flag (group_skip_flag) that determines whether to drive all motion vectors within a group to 0 on a group basis.

[0294] According to embodiments, when group_skip_flag is 0, the skip flag (group_comp_skip_flag) for each component (x, y, z axis) of each motion vector may be additionally parsed to determine whether to induce the motion of each component to 0.

[0295] At this time, if group_skip_flag=0, group_comp_skip_flag[0]=1, and group_comp_skip_flag[1]=1, the skip flag group_comp_skip_flag[2] of the third component can be implicitly derived as 0.

[0296] According to embodiments, if the current group is not skipped, the motion vector or motion value of the current group can be determined by signaling one representative (residual) motion vector for the motion of each component or a representative (residual) motion for each component. In this case, a flag indicating whether to use the representative (residual) motion vector or the representative (residual) motion for each component can be additionally transmitted.

[0297] According to embodiments, a flag for determining whether to perform prediction on a representative motion vector or a representative motion may be signaled and transmitted, wherein prediction of the current group may be performed based on the motion vector values ​​of the previous motion group in encoder / decoder order.

[0298] According to embodiments, when prediction is performed on a vertex-by-vertex basis and residual motion vectors are encoded, prediction modes such as a zero motion vector and / or an average motion vector with respect to surrounding vertices and / or an MV value of the i-th closest surrounding vertex may be used as a prediction method.

[0299] Also, when calculating the average value of the MV of the k vertices for which encoding has already been performed, and / or Two prediction values ​​can be used depending on whether rounding is used when predicting the mean value, and the two modes can be configured as separate prediction modes.

[0300] At this time, the definition of surrounding vertices can mean 1) the maximum k vertices among the vertices connected to the current vertex by edges, as determined by the encoder / decoder agreement, 2) the maximum k vertices among the vertices that are close in distance among the vertices that have already been encoded, etc.

[0301] According to embodiments, the MV prediction unit can perform MV prediction on a group or vertex basis through the prediction unit and prediction method determined in the group mode determination unit.

[0302] According to embodiments, the residual MV generation unit can generate a residual MV by differentiating the MV of the current group or vertex from the predicted MV signal generated by the MV prediction unit.

[0303] According to embodiments, the residual MV entropy encoding unit can entropy code the residual MV generated by the residual MV generation unit. At this time, entropy encoding for the motion of each component can be performed through the isZero, ... isK (1<=K) flags and mv_sign, mv_rem, etc. Here, the isZero flag may mean a flag indicating whether the absolute value of the current motion is 0, and isK may mean a flag indicating whether the absolute value of the current motion is equal to K.

[0304] And, for mv_sign, if the isZero flag is 0, that is, if the absolute value of the current motion is greater than 0, it can be entropy encoded.

[0305] For mv_rem, if the isZero, … isK flags are all 0, (abs(v)-K) can be encoded using an entropy encoding method such as exponential-Golomb encoding. abs(v) represents the current motion absolute value.

[0306] In some embodiments, the gtZero, … gtK flags may be used to indicate whether isZero, … isK is greater than K.

[0307] At this time, for each flag and mv_rem, encoding can be performed in entropy encoding based on context information or bypass mode using fixed probability.

[0308] According to embodiments, the motion vector encoder can encode the motion vector of the current base mesh by inheriting the stored entropy context information after encoding the motion vector of the previous base mesh.

[0309] FIG. 17(a) is a detailed block diagram showing another example of a motion vector encoder according to embodiments, and FIG. 17(b) is a diagram showing another example of a motion vector encoding process according to embodiments.

[0310] That is, FIG. 17(a) and FIG. 17(b) show a process in which, when the motion vector of the current base mesh is encoded in the motion vector encoding unit, the entropy context information used to encode the motion vector is stored through the motion vector entropy context information storage unit after encoding is completed, and initialization is performed through the context information when encoding the motion vector of the next frame. For example, assuming that the motion vector of base mesh 1 of FIG. 17(b) is encoded in the motion vector encoding unit, the entropy context information used to encode the motion vector of base mesh 1 is stored in the motion context information memory (i.e., the motion vector entropy context information storage unit) after encoding is completed. Thereafter, when the motion vector of the next frame, i.e., the motion vector of base mesh 2, is encoded in the motion vector encoding unit, the entropy context information of base mesh 2 is initialized with the entropy context information of base mesh 1 loaded from the motion context information memory. That is, the entropy context information of the previous frame stored in the motion vector entropy context information storage unit is provided to the motion vector encoding unit through the entropy context information buffer when encoding the motion vector of the next frame and is set as the entropy context information of the next frame. This means that the current motion vector is encoded using the entropy context information stored in advance when encoding the current motion vector. In other words, the motion vector encoding of the current base mesh means that it is performed by inheriting the previous entropy context information (or motion vector entropy context information) stored in the motion context information memory (i.e., the motion vector entropy context information storage unit). The motion vector entropy context information storage unit and the entropy context information buffer of Fig. 17(a) may correspond to the motion context information memory of Fig. 17(b).

[0311] The following describes the storage and inheritance of motion vector entropy context information. According to embodiments, whether to allow context information storage and inheritance at a higher level is determined through a 1-bit flag (motion_entropy_continuation_enabled), and if the flag is 1, whether to additionally inherit context information for the current base mesh or sub-mesh unit can be determined through a 1-bit flag (motion_entropy_continuation_flag).

[0312] According to embodiments, if the difference in FoC value between the previous base mesh storing context information and the current base mesh exceeds a certain threshold based on the FoC (frame of Count), the motion_entropy_continuation_flag can be implicitly induced to 0.

[0313] In the present disclosure, motion_entropy_continuation_flag is a flag that determines whether to perform entropy context information inheritance when encoding / decoding motion vectors. In the present disclosure, motion vector encoding / decoding may be performed on a vertex basis, a sub-mesh basis, a group basis, or a frame parameter set (fps) basis.

[0314] According to embodiments, when motion_entropy_continuation_enabled is 1, the motion vector of the current base mesh is encoded through the motion vector entropy context information storage unit, and then the entropy context information can be stored. That is, when motion_entropy_continuation_enabled is 1, the motion vector of the current base mesh is encoded using the entropy context information stored in the motion vector entropy context information storage unit, and then the entropy context information of the current base mesh is stored in the motion vector entropy context information storage unit.

[0315] At this time, the context information stored in the motion vector entropy context information storage unit may include one or more of a motion vector of a group unit signaled by the group mode decision unit, whether to skip a group for the motion, a group prediction mode, and a flag or index for encoding / decoding a residual MV generated by group unit or vertex unit.

[0316] According to embodiments, when the current base mesh is divided into two or more sub-meshes and encoding is performed, the context information of each sub-mesh can be stored in the motion vector entropy context information storage unit together with sub-mesh identification information (i.e., sub-mesh id).

[0317] Fig. 18(a) and Fig. 18(b) are diagrams showing another example of a motion vector encoding process according to embodiments. That is, Fig. 18(a) is a diagram showing an example of a context information reference base mesh corresponding to context information stored together with a submesh id in a motion vector entropy context information storage unit, in units of submeshes, and Fig. 18(b) is a diagram showing an example of a current base mesh inheriting the context information of a submesh stored in a motion vector entropy context information storage unit, in units of submeshes. For example, the context information of a first submesh stored together with a submesh id in a motion vector entropy context information storage unit can be inherited by the first submesh and the second submesh of the current submesh.

[0318] As in Fig. 18(a) and Fig. 18(b), when context information is stored in the motion vector entropy context information storage unit along with the submesh id for each submesh, when encoding the motion of the current base mesh, the motion vector entropy context information storage unit can determine the context information id to be inherited from the encoder (i.e., the motion vector encoding unit) and explicitly transmit the context information id to initialize the current submesh.

[0319] FIG. 19(a) and FIG. 19(b) are diagrams showing another example of a motion vector encoding process according to embodiments. That is, FIG. 19(a) is a diagram showing an example of a context information reference base mesh corresponding to context information stored in a motion vector entropy context information storage unit, in units of submeshes, and FIG. 19(b) is a diagram showing an example of a current base mesh that inherits the context information of a submesh stored in a motion vector entropy context information storage unit, in units of submeshes. For example, the context information of the fourth submesh stored in the motion vector entropy context information storage unit can be inherited by the first to fourth submeshes of the current submesh.

[0320] In some embodiments, when the number of sub-meshes is two or more, context information of a specific sub-mesh may be stored according to the encoder / decoder agreement.

[0321] Additionally, context information of the sub-mesh with the lowest id or the highest id can be stored.

[0322] The following is a detailed description of a static mesh encoder. Fig. 20 is a detailed block diagram showing an example of a static mesh encoder according to embodiments, and Fig. 21 is a detailed block diagram showing another example of a static mesh encoder according to embodiments. The static mesh encoders of Figs. 20 and 21 correspond to the static mesh encoders of Fig. 6, Fig. 13, or Fig. 15. Each component of Figs. 20 and 21 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in Figs. 20 and 21 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0323] According to embodiments, a static mesh can be separated into CCs (connected components) and encoding of CC units can be performed in a static mesh encoder.

[0324] According to embodiments, the mesh preprocessing unit may perform various preprocessing operations prior to encoding the input static mesh data. According to embodiments, the mesh preprocessing unit may perform corner table generation for vertex coordinates, corner table generation for UV coordinates, processing for non-manifold mesh processing, dummy vertex generation for hole filling, and / or quantization for attributes.

[0325] The connection information encoding unit performs encoding on the mesh connection information based on the edgebreaker algorithm. Here, the edgebreaker algorithm is a process of mapping the connection information of the current triangle to one of five symbols, C, R, E, L, or S, depending on the connection relationship between the triangle to be encoded and the surrounding triangles. The mapped symbol can be subjected to entropy encoding.

[0326] The vertex coordinate encoding unit performs a prediction on the current vertex coordinate based on the connection information of the static mesh and the previously encoded vertex coordinates, and performs entropy encoding on the differential vertex coordinate values ​​between the predicted vertex coordinates and the original vertex coordinates. According to embodiments, the prediction may be performed by differential prediction, parallelogram prediction, and / or multi-parallelogram prediction.

[0327] The UV coordinate encoding unit can perform prediction on the current UV coordinate based on the connection information of the static mesh and the already encoded UV coordinate, and can perform entropy encoding on the differential UV coordinate value between the predicted UV coordinate and the original UV coordinate.

[0328] The normal vector encoding unit may perform a prediction on the current normal vector based on the connection information of the static mesh and the already encoded normal vector, and may perform entropy encoding on the differential normal vector value between the predicted normal vector and the original normal vector. According to embodiments, in the case of the normal vector, entropy encoding may be performed after conversion into a 2D vector or octahedron conversion is performed based on the sum of the vector components of each axis being 1.

[0329] According to embodiments, in the case of symbols and / or differential signals of connection information, vertex coordinates, and UV coordinates as in FIG. 21, symbol mapping and residual signal generation may be performed for each component through a connection information symbol calculation unit, a residual vertex coordinate calculation unit, a residual UV coordinate calculation unit, and a residual normal vector calculation unit, and the symbol and residual signal may be stored, and after symbol mapping and residual signal generation of all components are completed, entropy encoding may be performed through a mesh entropy encoding unit.

[0330] In some embodiments, the static mesh encoder may encode the current base mesh by inheriting the entropy context information stored after encoding the previous base mesh.

[0331] FIG. 22(a) is a detailed block diagram showing another example of a static mesh encoder according to embodiments, and FIG. 22(b) is a diagram showing another example of a static mesh encoding process according to embodiments.

[0332] That is, FIG. 22(a) and FIG. 22(b) show a process in which, when the current base mesh is encoded in the static mesh encoding unit, the entropy context information used to encode the base mesh is stored through the static mesh entropy context information storage unit after encoding is complete, and initialization is performed through the context information when encoding the static mesh of the next frame. For example, assuming that the static mesh 0 of FIG. 22(b) is encoded in the static mesh encoding unit, the entropy context information used to encode the base mesh 0 is stored in the context information memory (i.e., the static mesh entropy context information storage unit) after encoding is complete. Thereafter, when the static mesh encoding unit encodes the base mesh 1 of the next frame, the entropy context information of the base mesh 1 is initialized with the entropy context information of the base mesh 0 loaded from the context information memory. That is, the entropy context information of the previous frame stored in the static mesh entropy context information storage unit is provided to the static mesh encoding unit through the static mesh entropy context information buffer when encoding the base mesh of the next frame and is set as the entropy context information of the next frame. This means that the current base mesh is encoded using the entropy context information stored in advance when encoding the current base mesh. In other words, the static mesh encoding of the current base mesh means that it is performed by inheriting the previous entropy context information stored in the context information memory (i.e., the static mesh entropy context information storage unit). The static mesh entropy context information storage unit and the static mesh entropy context information buffer of Fig. 22(a) may correspond to the context information memory of Fig. 22(b).

[0333] The following is a description of static mesh entropy context information storage and inheritance. According to embodiments, the present disclosure determines whether context information storage and inheritance are allowed at a higher level through a 1-bit flag (smesh_entropy_continuation_enabled), and when the flag is 1, additionally determines whether context information is inherited by the current base mesh or sub-mesh unit through a 1-bit flag (smesh_entropy_continuation_flag).

[0334] According to embodiments, if the difference in FoC value between the previous base mesh and the current base mesh that stored context information is greater than a certain threshold based on the FoC (frame of Count), smesh_entropy_continuation_flag can be implicitly induced to 0.

[0335] According to embodiments, only some of the context information used in static mesh encoding may be stored according to an encoder / decoder agreement, and the remaining context information may be initialized using the probability information agreed upon between the encoder / decoder at each static mesh encoding.

[0336] According to embodiments, the inheritance of entropy context information may be determined for some information, such as connection information symbols, (residual) vertex coordinates, (residual) UV coordinates, (residual) normal vectors, etc., for which entropy encoding is performed, through encoder / decoder agreements or explicit flag or index signaling. For example, when the context information to be inherited is determined by explicitly signaling a flag or index, the encoder may signal whether or not to inherit entropy in units of mesh frames or submesh, and at this time, the inheritance may be determined by configuring an inheritance flag for connection information and an inheritance flag for other attributes, or through a separate flag for each component.

[0337] According to embodiments, only some of the context information used in static mesh encoding may be stored according to an agreement between the encoder / decoder, and the remaining context information may be initialized with the probability information agreed upon between the encoder / decoder at each static mesh encoding.

[0338] The following is a detailed description of a displacement vector encoder. In the present disclosure, the displacement vector encoder can encode displacement vectors based on a 2D video codec or arithmetic coding. At this time, profile information or flags can be signaled at a higher level for the determined encoding method.

[0339] The displacement vector encoder described below corresponds to the displacement vector video encoder or displacement vector encoder of FIG. 6, FIG. 7, FIG. 13 or FIG. 15.

[0340] Fig. 23 is a detailed block diagram illustrating an example of a displacement vector encoder according to embodiments. Each component of Fig. 23 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in Fig. 23 may be changed, some blocks may be omitted, and some blocks may be newly added. In particular, Fig. 23 is an example of displacement vector encoding performed via a 2D video encoder.

[0341] More specifically, the displacement vector transformation unit can perform lifting transformation, wavelet transformation, etc. on the displacement vector of the (x, y, z) or (n, t, bt) coordinate system to transform the displacement vector of the (x, y, z) or (n, t, bt) coordinate system into a displacement vector transformation coefficient. In the (n, t, bt) coordinate system, n represents normal, t represents tangential, and bt represents bi-tangential. That is, the displacement vector of the (x, y, z) or (n, t, bt) coordinate system can be transformed through the displacement vector transformation unit (12011). According to embodiments, when the coordinate system transformation is performed to the (n, t, bt) coordinate system, a 1D scalar displacement vector of the normal (n) component may be applied as an input of the displacement vector transformation unit to perform transformation, quantization, and encoding on the displacement value of the normal component.

[0342] If lifting transformation is performed in the displacement vector transformation section, the vertex R of the kth subdivision level k R as a predictor when performing prediction t (t <k 또는 t<=k)의 세분화 정점 변위 벡터를 통해 k번째 세분화 레벨의 변위 벡터 예측을 수행할 수 있다. 실시예들에 따라 변위 벡터의 예측 수행 시 현재 정점보다 세분화 레벨이 낮은 정점 중 연결정보를 기반으로 가까운 n개의 점을 평균 또는 거리 기반 가중평균예측을 수행할 수 있다. 실시예들에 따라 메쉬 세분화 단계에서 현재 정점을 생성하기 위해 사용한 n개의 정점의 변위 벡터를 기반으로 예측을 수행할 수 있다. 그리고, 리프팅 변환이 수행되는 경우, 예측으로 생성되는 잔차 신호를 통해 예측에 사용된 정점의 변위 벡터를 업데이트 하는 과정이 수행될 수 있다.

[0343] According to embodiments, the displacement vector transform coefficient quantization unit can perform quantization on the displacement vector value transformed by the displacement vector transform unit, i.e., the displacement vector transform coefficient. In the present disclosure, the displacement vector transform coefficient is used interchangeably with the displacement vector coefficient with the same meaning. According to embodiments, the displacement vector transform coefficient can be quantized through different quantization parameters for each axis, and the quantization parameter or scaling parameter can be derived by the encoder / decoder agreement to determine the quantization rate for each LoD level.

[0344] According to embodiments, the displacement vector transform coefficient packing unit can pack N quantized displacement vector transform coefficients into a 2D image of size WxH. That is, if the displacement vector encoder encodes the displacement vector transform coefficients using a video codec-based encoding method, the process of packing the displacement vector transform coefficients into a frame as a 2D image is performed in the displacement vector transform coefficient packing unit. In other words, the displacement vector transform coefficient packing unit can pack the displacement vector transform coefficients into a 2D image, and the displacement vector image / video encoding unit can encode the packed 2D images using a video compression codec.

[0345] As described above, displacement vectors can be encoded based on arithmetic coding.

[0346] Fig. 24 is a detailed block diagram showing another example of a displacement vector encoder according to embodiments. Fig. 25 is a detailed block diagram showing another example of a displacement vector encoder according to embodiments. Fig. 26 is a detailed block diagram showing another example of a displacement vector encoder according to embodiments. That is, Figs. 24 to 26 are examples of detailed block diagrams of a displacement vector encoder that encodes a displacement vector based on arithmetic coding. Each component of Figs. 24 to 26 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in Figs. 24 to 26 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0347] The displacement vector encoder of Fig. 24 includes a displacement vector conversion unit, a displacement vector conversion coefficient quantization unit, and a displacement vector arithmetic encoding unit, and can perform displacement vector conversion, displacement vector conversion coefficient quantization, and displacement vector arithmetic encoding on an input displacement vector to output a displacement vector bitstream.

[0348] The displacement vector encoder of Fig. 25 includes a displacement vector conversion unit, a displacement vector conversion coefficient quantization unit, a displacement vector conversion level inter-screen prediction unit, and a displacement vector arithmetic encoding unit, and can output a displacement vector bitstream by performing displacement vector conversion, displacement vector conversion coefficient quantization, displacement vector conversion level inter-screen prediction, and displacement vector arithmetic encoding on an input displacement vector.

[0349] The displacement vector encoder of Fig. 26 includes a displacement vector conversion unit, a displacement vector conversion coefficient inter-screen prediction unit, a differential displacement vector conversion coefficient quantization unit, and a displacement vector arithmetic encoding unit, and can output a displacement vector bitstream by performing displacement vector conversion, displacement vector conversion coefficient inter-screen prediction, differential displacement vector conversion coefficient quantization, and displacement vector arithmetic encoding on an input displacement vector.

[0350] The following is a detailed description of the displacement vector arithmetic encoding unit of FIGS. 24 to 26.

[0351] Fig. 27 is a diagram showing an example of a displacement vector arithmetic encoding unit according to embodiments. Fig. 27 is repeatedly performed in sub-block units.

[0352] That is, the displacement vector is divided into subblocks, and arithmetic encoding is performed on a subblock basis. According to embodiments, the displacement vector is divided into subblock sizes determined through an agreement or explicit signaling between the encoder and decoder, and encoding / decoding can be performed on a subblock basis.

[0353] At this time, a 1-bit flag (nz_block_flag) can be signaled depending on whether a non-zero displacement vector exists within the subblock (i.e., if it is not the last subblock).

[0354] If a non-zero displacement vector exists in the current subblock, nz_block_flag is signaled as 1, and encoding can then be performed in the subblock displacement vector transform coefficient encoding unit.

[0355] Fig. 28 is a diagram showing an example of an encoding process of a sub-block displacement vector transformation coefficient encoding unit according to embodiments. Fig. 28 performs encoding on a vertex basis.

[0356] In Fig. 28, the isK flag may be a flag indicating whether the absolute value of the displacement vector transformation coefficient currently being encoded is equal to K, and depending on the embodiment, the gtZero, gtOne, … gtK flags may be used instead of the isZero, isOne, … isK flags. In this case, the gtK flag may be a flag indicating whether the absolute value of the transformation coefficient currently being encoded is greater than K.

[0357] That is, if Fig. 28 is not the last vertex, it signals the isZero flag, and if the value of the isZero flag is 0, it signals the sign flag and the isOne flag. If the isOne flag is 0, it signals the isK flag, and if the value of the isK flag is 0, it sets abs(disp)-(K+1) to disp_rem.

[0358] Meanwhile, as described above, when a displacement vector is encoded using an arithmetic coding method, its entropy context information can be stored and the stored context information can be used when encoding the next displacement vector, thereby reducing the bit amount of the displacement vector and improving encoding performance.

[0359] That is, in the present disclosure, when a displacement vector is encoded through arithmetic coding in a displacement vector encoder, entropy context information used in arithmetic coding of the displacement vector is stored, and when encoding the displacement vector of the next frame, encoding can be performed by inheriting the previously stored context information.

[0360] Fig. 29 is a detailed block diagram showing another example of a displacement vector encoder based on arithmetic coding according to embodiments. That is, when the current displacement vector is encoded in the displacement vector encoding unit, the entropy context information used to encode the displacement vector is stored through the displacement vector entropy context information storage unit after the encoding is completed, and initialization can be performed using the context information when encoding the displacement vector of the next frame. In other words, the entropy context information of the previous frame stored in the displacement vector entropy context information storage unit is provided to the displacement vector encoding unit through the entropy context information buffer when encoding the displacement vector of the next frame, and is set as the entropy context information of the next frame. This means that the displacement vector is encoded using the entropy context information stored in advance when encoding the current displacement vector. In other words, the encoding of the current displacement vector is performed by inheriting the previous entropy context information stored in the displacement vector entropy context information storage unit.

[0361] The following is a description of displacement vector entropy context information storage and inheritance. According to embodiments, the present disclosure determines whether context information storage and inheritance are allowed at a higher level through a 1-bit flag (dsps_entropy_continuation_enabled), and when the flag is 1, additionally signals a 1-bit flag (dh_entropy_continuation_flag) that determines whether context information is inherited for the current displacement vector.

[0362] According to embodiments, if the difference in FoC value between the previous displacement vector storing context information and the current displacement vector exceeds a certain threshold based on the FoC (frame of Count), dh_entropy_continuation_flag can be implicitly induced to 0.

[0363] According to an embodiment, only some of the context information used for displacement vector encoding may be stored according to an agreement between the encoder / decoder, and the remaining context information may be initialized with the probability information agreed upon between the encoder / decoder at the time of each displacement vector encoding.

[0364] In some embodiments, when context information is not inherited, the initial probability of each context information may be determined according to an agreement between the encoder and decoder. In this case, the context information may be initialized with a probability of 0.5, and when managing probability information with 16-bit precision, the probability value may be initialized to 0x8000.

[0365] According to embodiments, when storing context information in a displacement vector entropy context information storage unit, the upper m-bits of the probability value can be stored according to an agreement between the encoder / decoder.

[0366] In the case where the probability is managed with n-bit precision and the probability value is stored with m-bit precision according to the embodiments, the stored probability value (p s ) can be used to initialize the current context information probability (p) using the following mathematical expression 2.

[0367] [Equation 2]

[0368] p=p s <<(nm)

[0369] Fig. 30 is a diagram showing an example of context information storage and inheritance according to displacement vector types according to embodiments. That is, Fig. 30 shows an embodiment in which context information storage and inheritance are performed according to displacement vector types.

[0370] According to embodiments, the displacement vector entropy context information storage unit can be divided into an I-type displacement vector context information memory and a P-type displacement vector context information memory. That is, the displacement vector entropy context information storage unit separately stores entropy context information according to the type of displacement vector, and the displacement vector encoding unit can perform encoding by inheriting context information suitable for the current displacement vector type. For example, the entropy context information used to encode a displacement vector of type I in the displacement vector encoding unit is stored in the I-type displacement vector context information memory after encoding is completed. Thereafter, when encoding an I-type displacement vector of the next frame in the displacement vector encoding unit, the context information of the displacement vector is initialized with the entropy context information of the previous displacement vector loaded (inherited) from the I-type displacement vector memory, and encoding is performed. As another example, the entropy context information used to encode a displacement vector of type P in the displacement vector encoding unit is stored in the P-type displacement vector context information memory after encoding is completed. Afterwards, when encoding the P-type displacement vector of the next frame in the displacement vector encoding unit, the context information of the displacement vector is initialized with the entropy context information of the previous displacement vector loaded (inherited) from the P-type displacement vector memory, and encoding is performed.

[0371] Fig. 31 illustrates a receiving device according to embodiments. In the present disclosure, the receiving device of Fig. 31 may be referred to as a mesh data receiving device or decoder or a decoder of a receiving device or a V-Mesh decoder or a dynamic mesh decoder.

[0372] FIG. 31 corresponds to the receiving device (110) or mesh video decoder (113) of FIG. 1, the decoder of FIG. 11 or FIG. 12, the receiving device of FIG. 14, and / or the receiving decoding device corresponding thereto. Therefore, parts not described in FIG. 31 will refer to the description of the receiving device (110) or mesh video decoder (113) of FIG. 1, the decoder of FIG. 11 or FIG. 12, and the receiving device of FIG. 14. The elements of the receiving device illustrated in FIG. 31 may be implemented by hardware, software, a processor connected to a memory, and / or a combination thereof. That is, the elements of the receiving device of FIG. 31 may be implemented by hardware, software, firmware, or a combination thereof, including one or more processors or integrated circuits configured to communicate with one or more memories, although not illustrated in the drawing. One or more processors may perform at least one of the operations and / or functions of the elements of the receiving device of FIG. 31 described above. Furthermore, one or more processors may operate or execute a set of software programs and / or instructions for performing the operations and / or functions of the elements of the receiving device of FIG. 31. The execution order of each block in FIG. 31 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0373] FIG. 31 may largely include a base mesh decoding unit, a displacement information decoding unit, and a texture map decoding unit. According to embodiments, the base mesh decoding unit may include a switching unit (15011), a motion vector decoder (15012), a static mesh decoder (15013), a base mesh restoration unit (15014), a mesh refinement unit (15015), and a mesh restoration unit (15016). According to embodiments, the displacement information decoding unit may include a displacement vector decoder (15017) and a displacement vector coordinate system inverse transformation unit (15020).

[0374] According to embodiments, a bitstream of mesh data received by a receiver (not shown) may be demultiplexed into a base mesh bitstream, a displacement vector bitstream, and a texture map bitstream after file / segment decapsulation in a demultiplexer (not shown). If the current mesh has inter-screen encoding (i.e., inter-encoding) applied, the base mesh bitstream may be a motion vector bitstream.

[0375] According to embodiments, the base mesh bitstream is provided to a motion vector decoder (15012) via a switching unit (15011) or to a static mesh decoder (15013).

[0376] For example, if the current mesh has inter-screen encoding (i.e., inter encoding) applied, the base mesh bitstream, i.e., the motion vector bitstream, is received, demultiplexed, and then output to the motion vector decoder (15012) through the switching unit (15011). As another example, if the current mesh has intra-screen encoding (i.e., intra encoding) applied, the base mesh bitstream is received, demultiplexed, and then output to the static mesh decoder (15013) through the switching unit (15011). Here, the motion vector decoder (15012) may be referred to as a motion decoder.

[0377] According to embodiments, the motion vector decoder (15012) can perform decoding on a motion vector bitstream on a vertex-by-vertex basis or a subgroup basis.

[0378] According to embodiments, the motion vector decoder (15012) can reconstruct a final motion vector by adding a differential motion vector (i.e., a residual motion vector) decoded from a bitstream using a previously decoded motion vector as a predictor. That is, the motion vector decoder (15012) can decode a differential motion vector (or a residual motion vector) in units of vertices or subgroups (or subblocks) through a motion vector bitstream, and perform prediction based on connection information using a previously decoded motion vector as a predictor to decode the motion vector by adding it to the residual motion vector.

[0379] According to embodiments, the static mesh decoder (15013) can decode the base mesh bitstream to restore connection information, vertex geometry information, texture coordinates (i.e., attribute geometry information), normal information, etc. of the base mesh.

[0380] According to embodiments, the base mesh restoration unit (15014) may restore the current base mesh based on the decoded motion vector or the decoded base mesh. For example, if the current mesh has inter-screen encoding applied, the base mesh restoration unit (15014) may add the decoded (or restored) motion vector to the reference base mesh and then perform inverse quantization to generate a restored base mesh (i.e., the current base mesh). As another example, if the current mesh has intra-screen encoding applied, the base mesh restoration unit (15014) may perform inverse quantization on the decoded (or restored) base mesh through the static mesh decoder (15012) to generate a restored base mesh (i.e., the current base mesh). According to embodiments, the inverse quantization unit may be omitted.

[0381] According to embodiments, the mesh subdivision unit (15015) can generate additional vertices by performing subdivision on the base mesh. The present disclosure can implicitly derive and generate geometry information connection information, texture coordinate connection information, and texture coordinates according to the subdivision method.

[0382] According to embodiments, the mesh refinement unit (15015) can perform refinement through methods such as mid-edge, Loop, and Catmul&Clark.

[0383] According to embodiments, mesh refinement in the mesh refinement unit (15015) may be performed n times by user parameters or a promise of the encoder / decoder. According to embodiments, vertices of the base mesh are vertices of level 0 (R0), vertices newly generated by performing refinement once are vertices of level 1 (R1), … vertices newly generated by performing refinement n times are vertices of level n (R n ) can be defined as the vertices of the 0th level (R0), level 1 as the 1st level (R1), … level n as the nth level (R n ) can be called.

[0384] According to embodiments, LoD0, LoD1, … LoDn can be defined as in the following mathematical expression 3.

[0385] [Equation 3]

[0386] LoD0 = R0

[0387] LoD1 = R0+ R1= LoD0 + R1

[0388] …

[0389] LoDn = R0+ R1+, … + R n = LoDn-1 + R n

[0390] That is, the base mesh includes vertices for LoD0. And, the first subdivision mesh generated by subdividing the base mesh once includes vertices for LoD1. That is, LoD1 includes vertices of the base mesh (i.e., R0 vertices) and vertices of a newly generated level 1 by subdividing the base mesh (i.e., R1 vertices). In addition, the second subdivision mesh generated by subdividing the first subdivision mesh again includes vertices for LoD2. That is, LoD2 includes vertices of the base mesh (i.e., R0 vertices), vertices of a newly generated level 1 by subdividing the base mesh (i.e., R1 vertices), and vertices of a newly generated level 2 by subdividing the first subdivision mesh (i.e., R2 vertices).

[0391] According to embodiments, the displacement vector decoder (15017) may perform video codec-based decoding on the demultiplexed displacement vector bitstream as a video bitstream, or perform zero run-length decoding, or perform arithmetic decoding. In the present disclosure, the displacement vector decoder may be used interchangeably with the displacement vector transform decoder.

[0392] According to embodiments, the displacement vector decoder (15017) can restore the displacement vector by decoding the displacement vector in a reverse process of the displacement vector encoding method of the transmitting side.

[0393] According to embodiments, the displacement vector coordinate system inversion unit (15020) can perform a process of inverting the displacement vector decoded by the displacement vector decoder (15017) into a Cartesian (or canonical) coordinate system (x, y, z) if the displacement vector is a value of a local coordinate system (n, t, bt). The output of the displacement vector coordinate system inversion unit (15020) is provided to the mesh restoration unit (15016).

[0394] That is, in the encoder of the transmitting device, the vertex displacement vector calculated in the (x,y,z) space can be converted to a (normal, tangential, bi-tangential) coordinate system (or local coordinate system) based on the normal vector of each vertex. At this time, the normal vector can be calculated for each subdivided vertex based on the geometric information and connection information of the surrounding vertices.

[0395] Fig. 32(a) is a detailed block diagram showing an example of a displacement vector coordinate system inverse transformation unit (15020) according to embodiments. Fig. 32(b) is a detailed block diagram showing another example of a displacement vector coordinate system inverse transformation unit (15020) according to embodiments.

[0396] More specifically, the displacement vector coordinate system inverse transformation unit (15020) parses the coordinate system transformation flag (applyLocalCoord or asps_vmc_ext_displacement_coordinate_system) included in the signaling information in units of sequences, GoFs (group of frames), frames, or sub-mesh, and if its value is 1, it can perform inverse transformation of the inverse quantized (or inversely transformed) restored displacement vector from the local coordinate system (n, t, b) to the canonical coordinate system (x, y, z).

[0397] In addition, a normal vector per vertex is calculated based on the restored vertex position information of the restored base mesh, and a normal value of a newly created vertex can be assigned by interpolating the normal vector of the restored base mesh for the vertex additionally created through the subdivision process (see Fig. 32(a)).

[0398] At this time, in the case of interpolation, interpolation can be performed by averaging or distance-based weighting the normal information of the base mesh used for subdivision. Alternatively, the normal information of the base mesh can be used as is for subdivided vertices on the same plane.

[0399] According to the embodiments, after performing subdivision on the restored base mesh, normal vectors can be calculated for vertices generated through the mesh subdivision unit (15015) and vertices of the base mesh (see Fig. 32(b)).

[0400] And, through the calculated normal vector per vertex, the tangential and bi-tangential vectors orthogonal to the normal vector can be calculated and the displacement vector coordinate system inverse transformation can be performed. The following mathematical expression 4 disp n (0), disp n (1), disp n (2) refers to the results of normal, tangential, and bitangential components obtained by performing inverse transformation and inverse quantization.

[0401] [Equation 4]

[0402]

[0403] In some embodiments, coordinate system inversion can always be performed without flag transmission.

[0404] According to embodiments, the mesh restoration unit (15016) restores the mesh based on the mesh refined in the mesh refinement unit (15015) and the restored displacement vector output from the displacement vector coordinate system inverse transformation unit (15020).

[0405] According to embodiments, the received and demultiplexed texture map bitstream is input to a texture map decoder (15021). According to embodiments, the texture map decoder (15021) can decode the texture map through a 2D scalable decoder. That is, the texture map decoder (15021) can restore the texture map by applying 2D scalable decoding to the texture map. In this way, the texture map decoder (15021) receives the texture map bitstream as input and performs a process of decoding the texture map. At this time, the type of texture map decoder may include a video decoder, a zero run length decoder, an arithmetic decoder, etc. According to embodiments, the texture map decoder (15021) can perform color space conversion of the texture map.

[0406] When the base mesh bitstream, displacement vector bitstream, and texture map bitstream encoded by the dynamic mesh encoder of the transmitting device are transmitted in this way, the decoder of the receiving device restores the mesh by decoding each bitstream. First, the base mesh is decoded by the motion vector or static mesh decoder depending on whether it is an inter or intra frame, and the geometry information is restored along with the decoded displacement vector information through subdivision.

[0407] The following is a detailed description of the motion vector decoder.

[0408] Fig. 33 is a detailed block diagram showing an example of a motion vector decoder according to embodiments. The motion vector decoder of Fig. 33 corresponds to the motion vector decoder of Fig. 12, Fig. 14, or Fig. 31. Each component of Fig. 33 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in Fig. 33 may be changed, some blocks may be omitted, and some blocks may be newly added. In Fig. 33, motion vector decoding is performed in groups.

[0409] According to embodiments, the group mode decision unit can determine whether to skip the current motion group as a group and / or whether to skip each motion component (x, y, z axis) and / or whether to use a group-based prediction mode and / or whether to predict a group representative value, based on a flag or index that is explicitly or implicitly determined.

[0410] According to embodiments, the group mode decision unit may parse and determine a 1-bit flag (group_skip_flag) that determines whether to drive all motion vectors within a group to 0 on a group basis.

[0411] According to embodiments, the group mode decision unit may additionally parse the component-specific skip flag (group_comp_skip_flag) of each motion vector when group_skip_flag is 0 to determine whether to induce the motion of each component to 0. At this time, when group_skip_flag=0, group_comp_skip_flag[0]=1, and group_comp_skip_flag[1]=1, the skip flag group_comp_skip_flag[2] of the third component may be implicitly inducible to 0.

[0412] According to embodiments, the group mode decision unit may parse a flag for determining whether to restore the motion vector or motion value of the current group by signaling one representative (residual) motion vector for the motion vector or motion of each component or one representative (residual) motion for each component, if the motion vector or motion of the current group is not skipped. In this case, if the flag is 1, one representative motion vector or motion value may be parsed and restored to restore the MV of all vertices for the current group.

[0413] According to embodiments, the group mode decision unit may additionally parse a 1-bit flag to determine whether to use the representative motion vector or the representative motion as the restoration MV, or to perform restoration by adding the representative motion vector or the motion value after performing prediction. At this time, the definition of the group may vary (1<=group_size<=N, N=number of vertices of the base mesh), and the size of the group may be determined according to an agreement between the encoder / decoder or may be explicitly determined through parsing.

[0414] According to embodiments, for group division, the vertex order can be changed to a 3D Moulton code or a 3D Hilbert curve based on the vertex position information of the base mesh, and then group division can be performed based on the changed vertices.

[0415] According to embodiments, the motion vector decoder can decode the motion vector of the current base mesh by inheriting the stored entropy context information after decoding the motion vector of the previous base mesh.

[0416] FIG. 34 is a detailed block diagram showing another example of a motion vector decoder according to embodiments.

[0417] That is, Fig. 34 shows a process of storing the entropy context information used to decode the motion vector through the motion vector entropy context information storage unit after decoding is completed when the motion vector of the current base mesh is decoded based on the reference restoration base mesh in the motion vector decoding unit, and performing initialization through the context information when decoding the motion vector of the next frame. That is, the entropy context information of the previous frame stored in the motion vector entropy context information storage unit is provided to the motion vector decoding unit through the entropy context information buffer when decoding the motion vector of the next frame, and is set as the entropy context information of the next frame. This means that the current motion vector is decoded using the entropy context information stored in advance when decoding the current motion vector. In other words, it means that the motion vector decoding of the current base mesh is performed by inheriting the previous entropy context information (or motion vector entropy context information) stored in the motion vector entropy context information storage unit.

[0418] The following describes the storage and inheritance of motion vector entropy context information. According to embodiments, whether to allow context information storage and inheritance at a higher level is determined through a 1-bit flag (motion_entropy_continuation_enabled), and if the flag is 1, whether to additionally inherit context information for the current base mesh or sub-mesh unit can be determined through a 1-bit flag (motion_entropy_continuation_flag).

[0419] According to embodiments, the motion vector decoding unit may implicitly induce motion_entropy_continuation_flag to 0 when the difference in FoC values ​​between the previous base mesh storing context information and the current base mesh exceeds a certain threshold based on the FoC (frame of Count).

[0420] According to embodiments, the motion vector decoding unit can perform motion vector decoding by inheriting context information of a mesh or submesh referenced by the current motion vector. That is, the context information of the mesh or submesh referenced by the current motion vector, i.e., the inherited context information, is provided to the motion vector decoding unit through an entropy context information buffer from a motion vector entropy context information storage unit. At this time, the motion vector decoding unit can inherit the context information only when the type of the reference mesh is a type (P_SUBMESH) that performs decoding for a motion vector through the motion vector decoding unit.

[0421] According to embodiments, among the context information used for motion vector decoding, only some of the context information may be stored according to an agreement between the encoder and decoder, and the remaining context information may be initialized with the probability information agreed upon between the encoder and decoder at each motion vector decoding.

[0422] In some embodiments, when context information is not inherited, the initial probability of each context information may be determined according to an agreement between the encoder and decoder. In this case, the context information may be initialized with a probability of 0.5, and when managing probability information with 16-bit precision, the probability value may be initialized to 0x8000.

[0423] According to embodiments, when storing context information in a motion vector entropy context information storage unit, the upper m-bits of the probability values ​​can be stored according to an agreement between the encoder / decoder.

[0424] In cases where the probability is managed with n-bit precision and the probability value is stored with m-bit precision according to embodiments, the current context information probability (p) can be initialized using the stored probability value (p_s) using the following mathematical expression 5.

[0425] [Equation 5]

[0426] p=p s <<(nm)

[0427] According to embodiments, when motion_entropy_continuation_enabled is 1, the motion vector of the current base mesh may be decoded through the motion vector entropy context information storage unit, and then the entropy context information may be stored. That is, when motion_entropy_continuation_enabled is 1, the motion vector of the current base mesh is decoded using the entropy context information stored in the motion vector entropy context information storage unit, and then the entropy context information of the current base mesh is stored in the motion vector entropy context information storage unit. At this time, the context information stored may include one or more of a motion vector of a group unit signaled by the group mode decision unit, whether to skip a group for a motion, whether to skip each motion component, whether to transmit a group representative value, whether to predict a group unit, a group prediction mode, and a flag or index for decoding a residual MV generated by a group unit or a vertex unit.

[0428] According to embodiments, when the current base mesh is divided into two or more sub-meshes and decoding is performed, the context information of each sub-mesh can be stored together with the sub-mesh ID in the motion vector entropy context information storage unit.

[0429] According to embodiments, when context information is stored together with a submesh id for each submesh, as shown in FIGS. 18(a) and 18(b), when decoding a motion vector of the current base mesh, the motion vector entropy context information storage unit determines the context information id to be inherited from the decoder (e.g., the motion vector decoding unit). That is, the context information to be inherited can be explicitly determined by parsing the context information id for initializing the current submesh.

[0430] According to embodiments, when the number of sub-meshes is two or more, context information of a specific sub-mesh may be stored in the motion vector entropy context information storage unit according to an agreement between the encoder and decoder. In this case, as shown in FIG. 19(a) and FIG. 19(b), context information of the sub-mesh with the lowest ID or the highest ID may be stored.

[0431] The following is a detailed description of a static mesh decoder. Fig. 35 is a detailed block diagram showing an example of a static mesh decoder according to embodiments, and Fig. 36 is a detailed block diagram showing another example of a static mesh decoder according to embodiments. The static mesh decoders of Figs. 35 and 36 correspond to the static mesh decoders of Figs. 11, 14, or 31. Each component of Figs. 35 and 36 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in Figs. 35 and 36 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0432] According to embodiments, a static mesh can be separated into CCs (connected components) and decoding of CC units can be performed in a static mesh decoder.

[0433] According to embodiments, the connection information decoding unit performs decoding on the connection information of the mesh based on the edgebreaker algorithm. As an example, the connection information decoding unit can restore the connection information based on the decoded symbol (C or R or E or L or S) through connection information symbol entropy decoding.

[0434] According to embodiments, the vertex coordinate decoding unit may perform a prediction on the current vertex coordinate based on the connection information of the static mesh and the already decoded vertex coordinates, and may restore the vertex coordinate by adding the entropy-decoded differential vertex coordinates. According to embodiments, the prediction may be performed by differential prediction and / or parallelogram prediction and / or multiple parallelogram prediction.

[0435] According to embodiments, the UV coordinate decoding unit can perform a prediction on the current UV coordinate based on the connection information of the static mesh and the already decoded UV coordinate, and restore the UV coordinate by adding the entropy-decoded differential UV coordinate.

[0436] According to embodiments, the normal vector decoding unit may perform prediction on the current normal vector based on the connection information of the static mesh and the already decoded normal vector, and may perform restoration by adding the entropy-decoded residual normal vector. According to embodiments, when the normal vector is encoded after undergoing a transformation such as an octrahedron transformation in the transmitting device, the normal vector decoding unit may perform an inverse transformation on the residual normal vector or the restored normal vector.

[0437] According to embodiments, the mesh post-processing unit may perform dequantization and / or dummy vertex removal and / or generation of duplicate vertices for attribute information including mesh position information. In this case, dummy vertex removal may be a process of removing newly generated vertices after decoding at the receiving device to remove holes during encoding at the transmitting device. In addition, duplicate vertex generation may be a process of restoring duplicate vertices that were merged into one to remove non-manifolds of the input mesh during lossless encoding / decoding, and then separating the duplicate vertices again.

[0438] According to embodiments, in the case of symbols or / and differential signals of connection information, vertex coordinates, UV coordinates as in FIG. 36, after decoding is performed on all symbols and residual signals through a mesh entropy decoding unit, the symbols and residual signals are stored, and in the subsequent decoding process, decoding can be performed on each piece of information based on the symbols and residual signals.

[0439] In some embodiments, the static mesh decoder may inherit the entropy context information stored after decoding the previous base mesh to decode the current base mesh.

[0440] FIG. 37 is a detailed block diagram showing another example of a static mesh decoder according to embodiments.

[0441] That is, FIG. 37 shows a process in which, when the current base mesh is decoded in the static mesh decoding unit, the entropy context information used to decode the base mesh is stored through the static mesh entropy context information storage unit after decoding is complete, and initialization is performed through the context information when decoding the static mesh of the next frame. That is, the entropy context information of the previous frame stored in the static mesh entropy context information storage unit is provided to the static mesh decoding unit through the static mesh entropy context information buffer when decoding the base mesh of the next frame, and is set as the entropy context information of the next frame. This means that the current base mesh is decoded using the entropy context information stored in advance when decoding the current base mesh. In other words, the static mesh decoding of the current base mesh is performed by inheriting the previous entropy context information stored in the static mesh entropy context information storage unit.

[0442] The following is a description of static mesh entropy context information storage and inheritance. According to embodiments, the present disclosure determines whether context information storage and inheritance are allowed at a higher level by parsing a 1-bit flag (bmsps_smesh_entropy_continuation_enabled), and if the flag is 1, additionally determines whether context information is inherited by the current base mesh or sub-mesh unit by parsing a 1-bit flag (bmsh_smesh_entropy_continuation_flag).

[0443] According to embodiments, if the difference in FoC value between the previous base mesh that stored context information and the current base mesh exceeds a certain threshold based on the FoC (frame of Count), bmsh_smesh_entropy_continuation_flag can be implicitly set to 0.

[0444] According to embodiments, only some of the context information used for static mesh decoding may be stored according to an agreement between the encoder and decoder, and the remaining context information may be initialized with the probability information agreed upon between the encoder and decoder at each static mesh decoding.

[0445] In some embodiments, when context information is not inherited, the initial probability of each context information may be determined according to an agreement between the encoder and decoder. In this case, the context information may be initialized with a probability of 0.5, and when managing probability information with 16-bit precision, the probability value may be initialized to 0x8000.

[0446] According to embodiments, when storing context information in the entropy context information storage unit of a static mesh, the upper m-bits of the probability values ​​can be stored according to an agreement between the encoder / decoder.

[0447] In cases where the probability is managed with n-bit precision and the probability value is stored with m-bit precision according to the embodiments, the current context information probability (p) can be initialized using the stored probability value (p_s) using the following mathematical expression 6.

[0448] [Equation 6]

[0449] p=p s <<(nm)

[0450] According to embodiments, for some information such as connection information symbols, (residual) vertex coordinates, (residual) UV coordinates, (residual) normal vectors, etc. for which entropy decoding is performed, inheritance of entropy context information may be performed through an agreement between the encoder / decoder or explicit flag or index parsing. If the context information to be inherited is determined by explicitly parsing a flag or index, the decoder (e.g., static mesh decoder) may parse whether entropy is inherited for each mesh frame or submesh. At this time, the inheritance may be determined by configuring an inheritance flag for connection information and an inheritance flag for other attributes, or through a separate flag for each component.

[0451] According to embodiments, only some of the context information used for static mesh decoding may be stored according to an agreement between the encoder and decoder, and the remaining context information may be initialized with the probability information agreed upon between the encoder and decoder at each static mesh decoding.

[0452] According to embodiments, the static mesh decoder may omit parsing of a separate context inheritance flag and implicitly set the context inheritance flag to 1 if the type parsed through the mesh header or sub-mesh header is a type that performs context inheritance defined in the encoder / decoder.

[0453] Fig. 38 is a diagram showing an example of a syntax structure when the type of a submesh that inherits context information according to embodiments is S_SUMMESH.

[0454] That is, Fig. 38 is a case where the type of the submesh that inherits context information is parsed as S_SUBMESH. If the type of the current submesh is S_SUBMESH, the stored context information of the previous submesh is inherited to initialize the current context information, and the static mesh can be decoded through the same process as the static mesh coding of I_SUBMESH.

[0455] The following is a detailed description of the displacement vector decoder. In the present disclosure, the displacement vector decoder can be determined based on a profile or flag parsed at a higher level during 2D video codec-based decoding or arithmetic coding-based decoding.

[0456] The displacement vector decoder described below corresponds to the displacement vector video decoder or displacement vector decoder of FIG. 11, FIG. 12, FIG. 14 or FIG. 31.

[0457] Fig. 39 is a detailed block diagram showing an example of a displacement vector decoder according to embodiments. Each component of Fig. 39 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in Fig. 39 may be changed, some blocks may be omitted, and some blocks may be newly added. In particular, Fig. 39 is an example when displacement vector decoding is performed through a 2D video decoder. That is, when displacement vector decoding is performed through a 2D video decoder, decoding may be performed through Fig. 39.

[0458] According to embodiments, the video decoding unit applies a 2D video decoder such as H.264, HEVC, or VVC to the input displacement vector bitstream to perform decoding on a displacement vector transform coefficient image / video.

[0459] According to embodiments, the displacement vector transform coefficient depacking unit performs displacement vector transform coefficient depacking for each frame on a restored displacement vector transform coefficient video restored through a video decoding unit. At this time, displacement vector transform coefficient assignment corresponding to each vertex of the restored mesh may be performed. In addition, the displacement vector transform coefficient depacking unit may perform depacking from a restored displacement vector transform coefficient image corresponding to the current mesh frame according to a scanning order defined by an encoder / decoder agreement or a scanning order parsed into a higher-level unit (sequence, frame, etc.).

[0460] According to embodiments, the displacement vector transform coefficient inverse quantization unit performs inverse quantization on displacement vector transform coefficients allocated per vertex through the displacement vector transform coefficient inverse packing unit. At this time, the transform coefficients may be quantized through different quantization parameters for each axis, and the quantization rate may be determined for each LoD level by deriving quantization parameters or scaling parameters through an agreement between the encoder / decoder.

[0461] According to embodiments, the displacement vector inverse transform unit performs an inverse transform of the transform performed in the encoder of the transmitting device on the inverse quantized displacement vector coefficients to output displacement vectors. According to embodiments, a lifting inverse transform, a wavelet inverse transform, etc. may be performed. If a lifting inverse transform is performed, the vertex R of the kth subdivision level k R as a predictor when performing prediction k (t <k 또는 t<=k)의 세분화 정점 변위 벡터를 통해 k번째 세분화 레벨의 변위벡터 예측을 수행할 수 있다. 실시예들에 따라 변위벡터의 예측 수행 시 현재 정점보다 세분화 레벨이 낮은 정점 중 연결정보를 기반으로 가까운 n개의 점을 평균 또는 거리기반 가중평균예측을 수행할 수 있다. 실시예들에 따라 메쉬 세분화 단계에서 현재 정점을 생성하기 위해 사용한 n개의 정점의 변위 벡터를 기반으로 예측을 수행할 수 있다. 그리고, 리프팅 역변환이 수행되는 경우 파싱받은 잔차 신호를 통해 인코더에서 예측에 사용된 정점의 변위 벡터를 업데이트 하는 과정이 수행될 수 있다.

[0462] As described above, displacement vectors can be decoded based on arithmetic coding.

[0463] Fig. 40 is a detailed block diagram showing another example of a displacement vector decoder according to embodiments. Fig. 41 is a detailed block diagram showing another example of a displacement vector decoder according to embodiments. Fig. 42 is a detailed block diagram showing another example of a displacement vector decoder according to embodiments. That is, Figs. 40 to 42 are examples of detailed block diagrams of a displacement vector decoder that decodes a displacement vector based on arithmetic coding. Each component of Figs. 40 to 42 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in Figs. 40 to 42 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0464] The displacement vector decoder of Fig. 40 includes a displacement vector transform coefficient decoding unit, a displacement vector transform coefficient inverse quantization unit, and a displacement vector inverse transform unit, and can restore a displacement vector by performing displacement vector transform coefficient arithmetic decoding, displacement vector transform coefficient inverse quantization, and displacement vector inverse transform on an input displacement vector bitstream.

[0465] The displacement vector decoder of Fig. 41 includes a displacement vector transform coefficient decoding unit, a displacement vector transform level inter-screen prediction and restoration unit, a displacement vector transform coefficient inverse quantization unit, and a displacement vector inverse transform unit, and can restore a displacement vector by performing displacement vector transform coefficient arithmetic decoding, displacement vector transform level inter-screen prediction and restoration, displacement vector transform coefficient inverse quantization, and displacement vector inverse transform on an input displacement vector bitstream.

[0466] The displacement vector decoder of FIG. 42 includes a displacement vector transform coefficient decoding unit, a displacement vector transform coefficient inverse quantization unit, a displacement vector transform coefficient inter-frame prediction and restoration unit, and a displacement vector inverse transform unit, and can restore a displacement vector by performing displacement vector transform coefficient arithmetic decoding, displacement vector transform coefficient inverse quantization, displacement vector transform coefficient inter-frame prediction / restoration, and displacement vector inverse transform on an input displacement vector bitstream.

[0467] That is, when a displacement vector is decoded through arithmetic coding, displacement vector decoding can be performed through the displacement vector conversion coefficient decoding unit of FIGS. 40 to 42.

[0468] The following is a detailed description of the displacement vector conversion coefficient decoding unit of FIGS. 40 to 42.

[0469] Fig. 43 is a diagram showing an example of a displacement vector conversion coefficient decoding unit according to embodiments. Fig. 43 is repeatedly performed in sub-block units.

[0470] That is, the displacement vector is divided into subblock sizes determined through an agreement or explicit signaling between the encoder / decoder, and decoding can be performed on a subblock basis. At this time, a 1-bit flag (nz_block_flag) indicating whether a non-zero displacement vector exists within the subblock is parsed, and when the flag is 0, all displacement vector transformation coefficient values ​​within the current subblock can be restored to 0.

[0471] At this time, if nz_block_flag is 1, the vertex unit displacement vector transformation coefficient within the subblock can be restored through the subblock displacement vector transformation coefficient decoding unit.

[0472] According to embodiments, when the displacement vector type parsed at the upper level is a type that performs inter-screen prediction (P_DISPL), displacement vector transformation level inter-screen prediction and restoration can be performed in the displacement vector transformation level inter-screen prediction and restoration unit as shown in FIG. 41. At this time, the displacement vector transformation level inter-screen prediction and restoration unit can perform a process of adding the restoration transformation coefficient level value of the reference frame stored in the buffer to the current restoration transformation coefficient level value.

[0473] According to embodiments, displacement vector transformation coefficient inter-screen prediction and restoration may be performed in the displacement vector transformation coefficient inter-screen prediction and restoration unit as in FIG. 42. In this case, the displacement vector transformation coefficient inter-screen prediction and restoration unit may perform a process of adding the restoration transformation coefficient value of the reference frame stored in the buffer to the current restoration transformation coefficient value.

[0474] Fig. 44 is a diagram showing an example of a decoding process of a sub-block displacement vector transformation coefficient decoding unit according to embodiments. Fig. 44 performs decoding on a vertex basis.

[0475] That is, when nz_block_flag is 1 in FIG. 43, the sub-block displacement vector transform coefficient decoding unit can be performed, and the sub-block displacement vector transform coefficient decoding unit can operate as in FIG. 44.

[0476] In Fig. 44, the isK flag may be a flag indicating whether the displacement vector transformation coefficient currently being decoded is equal to K, and depending on the embodiment, the gtZero, gtOne, … gtK flags may be used instead of the isZero, isOne, … isK flags. In this case, the gtK flag may be a flag indicating whether the transformation coefficient currently being decoded is greater than K.

[0477] That is, if Fig. 44 is not the last vertex, the isZero flag is parsed, and if the value of the isZero flag is 0, the sign flag and the isOne flag are parsed. If the isOne flag is 0, the isK flag is parsed, and if the value of the isK flag is 0, disp_rem is parsed.

[0478] And, if the sign is 0, the value of the displacement vector (disp) is set to disp_rem + K + 1, and if it is not 0, the value of the displacement vector (disp) is set to -(disp_rem + K + 1) (i.e., the negative of disp_rem + K + 1).

[0479] Meanwhile, when the displacement vector transform coefficient is decoded using an arithmetic coding method as described above, the entropy context information thereof can be stored and the decoding performance of the displacement vector can be improved by using the stored context information when decoding the next displacement vector transform coefficient.

[0480] That is, in the present disclosure, when a displacement vector transform coefficient is decoded based on arithmetic coding in a displacement vector decoder, entropy context information used for decoding the displacement vector transform coefficient is stored, and when decoding the displacement vector transform coefficient of the next frame, decoding can be performed by inheriting the previously stored context information.

[0481] Fig. 45 is a detailed block diagram showing another example of a displacement vector decoder based on arithmetic coding according to embodiments. As shown in Fig. 45, entropy context information is stored through the displacement vector entropy context information storage unit, and initialization can be performed using the context information when decoding the displacement vector of the next frame.

[0482] That is, when the current displacement vector (or displacement vector transformation coefficient) is decoded in the displacement vector decoding unit, the entropy context information used to arithmetic decode the displacement vector (or displacement vector transformation coefficient) is stored through the displacement vector entropy context information storage unit after the arithmetic decoding is completed, and initialization can be performed through the context information when arithmetic decoding the displacement vector of the next frame. In other words, the entropy context information of the previous frame stored in the displacement vector entropy context information storage unit is provided to the displacement vector decoding unit through the entropy context information buffer when decoding the displacement vector of the next frame, and is set as the entropy context information of the next frame. This means that the displacement vector is decoded using the entropy context information stored in advance when decoding the current displacement vector. In other words, the arithmetic decoding of the current displacement vector is performed by inheriting the previous entropy context information stored in the displacement vector entropy context information storage unit (i.e., the entropy context information used for arithmetic decoding).

[0483] The following is a description of displacement vector entropy context information storage and inheritance. According to embodiments, the displacement vector decoding unit may determine whether context information storage and inheritance are permitted at a higher level by parsing a 1-bit flag (dsps_entropy_continuation_enabled), and if the flag is 1, may additionally determine whether context information is inherited for the current displacement vector by parsing a 1-bit flag (dh_entropy_continuation_flag).

[0484] According to embodiments, the displacement vector decoding unit may implicitly induce dh_entropy_continuation_flag to 0 when the difference in FoC value between the previous displacement vector storing context information and the current displacement vector exceeds a certain threshold based on the FoC (frame of Count).

[0485] According to embodiments, the displacement vector decoding unit may store only some of the context information used for displacement vector decoding according to the encoder / decoder agreement, and may perform an initialization process for the remaining context information with the probability information agreed upon between the encoder / decoder at each displacement vector decoding.

[0486] According to embodiments, the displacement vector decoding unit may determine the initial probability of each contextual information according to the encoder / decoder agreement when contextual information is not inherited. In this case, the contextual information may be initialized with a probability of 0.5, and when managing probability information with 16-bit precision, the probability value may be initialized to 0x8000.

[0487] According to embodiments, when storing context information in a displacement vector entropy context information storage unit, the upper m-bits of the probability value can be stored according to an agreement between the encoder / decoder.

[0488] In cases where the probability is managed with n-bit precision and the probability value is stored with m-bit precision according to embodiments, the current context information probability (p) can be initialized using the stored probability value (p_s) using the following mathematical expression 7.

[0489] [Equation 7]

[0490] p=p s <<(nm)

[0491] Figure 46 is a diagram showing an example of context information storage and inheritance according to displacement vector types according to embodiments. That is, Figure 46 shows an embodiment in which context information storage and inheritance are performed according to displacement vector types.

[0492] According to embodiments, the displacement vector entropy context information storage unit stores entropy context information separately according to the type of displacement vector, and the displacement vector decoding unit can perform decoding by inheriting context information suitable for the current displacement vector type. At this time, context information storage and inheritance can be performed according to the displacement vector type, as shown in FIG. 46.

[0493] According to embodiments, the displacement vector entropy context information storage unit can be divided into an I-type displacement vector context information memory and a P-type displacement vector context information memory. That is, the displacement vector entropy context information storage unit separately stores entropy context information according to the type of displacement vector, and the displacement vector decoding unit can perform decoding by inheriting context information suitable for the current displacement vector type. For example, the entropy context information used to decode the I-type displacement vector in the displacement vector decoding unit is stored in the I-type displacement vector context information memory after decoding is completed. Thereafter, when the displacement vector decoding unit decodes the I-type displacement vector of the next frame, the context information of the displacement vector is initialized with the entropy context information of the previous displacement vector loaded (inherited) from the I-type displacement vector memory, and decoding is performed. As another example, the entropy context information used to decode the P-type displacement vector in the displacement vector decoding unit is stored in the P-type displacement vector context information memory after decoding is completed. Thereafter, when the displacement vector decoding unit decodes the P-type displacement vector of the next frame, the context information of the displacement vector is initialized with the entropy context information of the previous displacement vector loaded (inherited) from the P-type displacement vector memory, and decoding is performed.

[0494] According to embodiments, signaling information in a transmitting device (or an encoder of the transmitting device) may be generated by a metadata processing unit (not shown, may be referred to as a metadata generator, etc.) and provided to corresponding blocks in the transmitting device and / or a receiving device (or a decoder of the receiving device), and a metadata parser (not shown) of the receiving device may parse the received signaling information and provide it to the corresponding blocks. According to embodiments, each block of the receiving device may perform each operation based on the signaling information.

[0495] According to embodiments, the decoder of the present disclosure may parse a 1-bit flag for allowing contextual information storage and inheritance for meshes of the current higher level (e.g., sequence parameter set (SPS) etc.) of the base mesh to determine.

[0496] According to embodiments, the flag for determining whether context information is allowed may be parsed for static mesh encoding of a static mesh encoder and motion vector encoding of a motion vector encoder, or may be parsed as a single flag. According to embodiments, the flag for determining whether context information is allowed may be parsed for static mesh decoding of a static mesh decoder and motion vector decoding of a motion vector decoder, or may be parsed as a single flag.

[0497] FIG. 47 is a diagram showing an example of the syntax structure of a sequence parameter set (bmesh_sequence_parameter_set_rbsp()) according to embodiments.

[0498] In Fig. 47, bmsps_motion_entropy_continuation_enabled is a flag that determines whether to allow entropy context information storage and inheritance when encoding / decoding motion vectors in sps units. If the value of bmsps_motion_entropy_continuation_enabled is 0, it indicates that context information storage and inheritance of the mesh referencing the corresponding sps are not allowed, and if it is 1, it indicates that context information storage and inheritance of the mesh referencing the corresponding sps are allowed.

[0499] bmsps_smesh_entropy_continuation_enable is a flag that determines whether to allow entropy context information storage and inheritance when encoding / decoding static meshes in sps units. If the value of bmsps_smesh_entropy_continuation_enable is 0, it indicates that context information storage and inheritance of meshes referencing the corresponding sps are not allowed, and if it is 1, it indicates that context information storage and inheritance of meshes referencing the corresponding sps are allowed.

[0500] According to embodiments, when decoding a base mesh in a static mesh decoder or when decoding a motion vector in a motion vector decoder depending on intra mode or inter mode, as in FIG. 47, 1-bit flags (bmsps_smesh_entropy_continuation_enabled, bmsps_motion_entropy_continuation_enabled) for the static mesh and motion vectors can be parsed from the sps of the base mesh to determine whether to allow storage and inheritance of context information.

[0501] FIG. 48 is a diagram showing an example of the syntax structure of a submesh header (bmesh_submesh_header()) according to embodiments.

[0502] In Fig. 48, bmsh_smesh_entropy_continuation_flag is a flag that determines whether to perform entropy context information inheritance when encoding / decoding static meshes in units of submesh. If the value of bmsh_smesh_entropy_continuation_flag is 0, it indicates that the context information of the corresponding submesh is initialized with a probability defined according to the agreement between the encoder / decoder, and if it is 1, it can indicate that the entropy context information of the current submesh is initialized with the stored context information.

[0503] bmsh_motion_entropy_continuation_flag is a flag that determines whether to inherit entropy context information when encoding / decoding motion vectors on a submesh basis. If the value of bmsh_motion_entropy_continuation_flag is 0, it indicates that the motion vector context information of the corresponding submesh is initialized with a probability defined according to the encoder / decoder agreement. If it is 1, it can indicate that the entropy context information of the motion vector of the current submesh is initialized with the stored context information.

[0504] According to embodiments, when decoding a base mesh in a static mesh decoder or decoding a motion vector in a motion vector decoder according to intra mode or inter mode, a flag (bmsh_smesh_entropy_continuation_flag, bmsh_motion_entropy_continuation_flag) for determining whether to inherit context information at the level of the base mesh as in FIG. 48 may be parsed for static mesh decoding and motion vector decoding, respectively, according to a flag for allowing storage and inheritance of context information parsed at a higher level. Or, in the case of an encoder, it may be parsed for static mesh encoding and motion vector encoding, respectively.

[0505] That is, whether or not the context information of the static mesh and motion vector is inherited can be parsed in the sps of Fig. 47, and whether or not the context information of the static mesh and motion vector is inherited can be parsed in the base mesh (i.e., submesh header) of Fig. 48.

[0506] FIG. 49 is a diagram showing an example of the syntax structure of a submesh unit (bmesh_submesh_unit()) according to embodiments.

[0507] A submesh unit is identified by submeshID and can contain various information depending on the bmsh_type. For example, if bmsh_type is I_SUBMESH, it can contain bm_intra_submesh_unit(submeshID, unitSize), if bmsh_type is P_SUBMESH, it can contain bm_inter_submesh_unit(submeshID, unitSize), if bmsh_type is SKIP_SUBMESH, it can contain bm_skip_submesh_unit(), and if bmsh_type is S_SUBMESH, it can contain load_submesh_context(submeshID) and bm_intra_submesh_unit(submeshID, unitSize).

[0508] FIG. 50 is a diagram showing another example of the syntax structure of a submesh header (bmesh_submesh_header()) according to embodiments.

[0509] If bmsh_type is S_SUBMESH, bmsh_smesh_entropy_continuation_flag can be signaled with a value of 1. And, if bmsh_type is P_SUBMESH and bmsps_motion_entropy_continuation_enabled is true, bmsh_motion_entropy_continuation_flag can be included.

[0510] bmsh_motion_entropy_continuation_flag is a flag that determines whether to inherit entropy context information when encoding / decoding motion vectors on a submesh basis. If the value of bmsh_motion_entropy_continuation_flag is 0, it indicates that the motion vector context information of the corresponding submesh is initialized with a probability defined according to the encoder / decoder agreement. If it is 1, it can indicate that the entropy context information of the motion vector of the current submesh is initialized with the stored context information.

[0511] According to embodiments, when inheriting context information of a previous frame during static mesh encoding / decoding as in FIG. 49, the mesh can be defined as a separate type (S_SUBMESH).

[0512] In addition, in a situation like the embodiment of Fig. 48 where the flag for whether to allow context information storage and inheritance is parsed separately for the static mesh and the motion vector, if the current mesh is encoded / decoded as S_SUBMESH, a separate type that inherits context information, as in Fig. 50, the flag (bmsh_smesh_entropy_continuation_flag) for determining whether the static mesh inherits context information can be explicitly derived as 1.

[0513] That is, FIGS. 49 and 50 are examples of defining a separate type S_Submesh when parsing whether to allow inheritance of context information of static mesh and motion vector from sps and when parsing whether to allow inheritance at the base mesh level, respectively.

[0514] FIG. 51 is a diagram showing an example of the syntax structure of a frame parameter set (bmesh_frame_parameter_set_rbsp( )) according to embodiments.

[0515] In Fig. 51, bfps_static_entropy_continuation_flag is a flag that determines whether to perform entropy context information inheritance when encoding / decoding static meshes in units of fps. If the value of bfps_static_entropy_continuation_flag is 0, it indicates that the mesh referencing the corresponding fps performs initialization of context information with a probability defined according to the encoder / decoder agreement when encoding / decoding static meshes. If it is 1, it can indicate that the entropy context information of the current mesh is initialized with the stored context information.

[0516] bfps_motion_entropy_continuation_flag is a flag that determines whether to perform entropy context information inheritance when encoding / decoding motion vectors in units of fps. If the value of bfps_motion_entropy_continuation_flag is 0, it indicates that the mesh referencing the corresponding fps performs initialization of context information with a probability defined according to the encoder / decoder agreement when encoding / decoding motion vectors. If it is 1, it can indicate that the stored context information initializes the entropy context information of the current motion vector.

[0517] According to embodiments, when decoding a base mesh in a static mesh decoder or when decoding a motion vector in a motion vector decoder depending on intra mode or inter mode, as in FIG. 51, whether to allow context storage and inheritance for the static mesh and motion vectors at the fps of the base mesh can be parsed through a 1-bit flag to determine whether to allow context storage and inheritance. Then, at the base mesh level, whether to allow inheritance for the static mesh and motion vectors can be parsed through a single flag, as in FIG. 48 or FIG. 49, or FIG. 50 above.

[0518] Additionally, even if the top-level flag is 1, context inheritance for static meshes and motion vectors may not be performed depending on certain conditions promised by the encoder / decoder.

[0519] In some embodiments, for static mesh encoding / decoding, context information may be inherited only when the FoC (frame of Count) with the immediately encoded / decoded I_SUBMESH (or S_SUBMESH in FIG. 49) is less than a certain threshold.

[0520] According to embodiments, in the case of motion vector encoding / decoding, if the type of the mesh being referenced is I_SUBMESH, context information cannot be inherited, and according to embodiments, context information can be inherited and used only if the type of the reference mesh is P_SUBMESH.

[0521] According to embodiments, in the case of motion vector encoding / decoding, context information can be inherited only when the FoC (frame of Count) with the immediately encoded / decoded reference mesh is less than a specific threshold.

[0522] According to embodiments, the displacement vector decoder may parse and determine a 1-bit flag for whether to allow context information storage and inheritance for the displacement vector of the current upper level at the upper level (sps, etc.) of the displacement vector, as shown in FIG. 52 or FIG. 54.

[0523] According to embodiments, the flag for determining whether to allow storage and inheritance of context information may be parsed for each of the I type and P type displacement vectors, or one flag may be parsed.

[0524] Figures 52 and 54 are examples of parsing a 1-bit flag from the sps of a displacement vector to determine whether to allow storage and inheritance of context information.

[0525] According to embodiments, the displacement vector decoder may determine whether to allow storage and inheritance of context information by parsing the flags (dsps_static_entropy_continuation_enabled, dsps_inter_entropy_continuation_enabled) for the displacement vectors of type I and type P, respectively, as shown in FIG. 52.

[0526] In addition, in a situation such as in FIG. 52, where a flag for determining whether to allow storage and inheritance of context information of a displacement vector is parsed according to the type, the type of the displacement vector that inherits the context information can be separately specified (S_DISPL) without performing prediction using the previous restoration transformation coefficient or transformation level at the displacement vector level as in FIG. 53, and in the case of the corresponding displacement vector type, dh_static_entropy_continuation_flag can be implicitly induced to 1.

[0527] FIG. 52 is a diagram showing another example of the syntax structure of a sequence parameter set (displ_sequence_parameter_set_rbsp()) according to embodiments of the present invention.

[0528] In Fig. 52, dsps_static_entropy_continuation_enable is a flag that determines whether to allow entropy context information storage and inheritance when encoding / decoding displacement vectors in sps units for displacement vectors of the I_DISPL (or S_DISPL) type. If the value of dsps_static_entropy_continuation_enabled is 0, it indicates that context information storage and inheritance of displacement vectors referencing the corresponding sps are not allowed, and if it is 1, it indicates that context information storage and inheritance of displacement vectors referencing the corresponding sps are allowed.

[0529] dsps_inter_entropy_continuation_enable is a flag that determines whether to allow entropy context information storage and inheritance when encoding / decoding displacement vectors in sps units for displacement vectors of the P_DISPL type. If the value of dsps_inter_entropy_continuation_enabled is 0, it indicates that context information storage and inheritance of displacement vectors referencing the corresponding sps are not allowed, and if it is 1, it indicates that context information storage and inheritance of displacement vectors referencing the corresponding sps are allowed.

[0530] FIG. 53 is a diagram showing an example of the syntax structure of a displacement vector header (displ_header()) according to embodiments.

[0531] In Fig. 53, dh_static_entropy_continuation_flag is a flag that determines whether to perform entropy context information inheritance when encoding / decoding displacement vectors of the I_DISPL (or S_DISPL) type. If the value of dh_static_entropy_continuation_flag is 0, it indicates that the context information of the corresponding displacement vector is initialized with a probability defined according to the encoder / decoder agreement, and if it is 1, it can indicate that the entropy context information of the current displacement vector is initialized with the stored context information.

[0532] dh_inter_entropy_continuation_flag is a flag that determines whether to inherit entropy context information when encoding / decoding displacement vectors of the P_DISPL type. If the value of dh_inter_entropy_continuation_flag is 0, it indicates that the context information of the corresponding displacement vector is initialized with a probability defined according to the encoder / decoder agreement, and if it is 1, it can indicate that the entropy context information of the current displacement vector is initialized with the stored context information.

[0533] Figures 52 and 53 are examples of parsing, respectively, whether to allow inheritance of context information of displacement vectors in sps and whether to allow inheritance at the displacement vector level according to P type and I type when decoding displacement vectors in a displacement vector decoder.

[0534] That is, in an embodiment in which whether to allow inheritance of context information of displacement vector is parsed in sps and whether to allow inheritance is parsed according to P type and I type at the displacement vector level, Fig. 53 is an embodiment in which a separate type S_DISPL is specified.

[0535] FIG. 54 is a diagram showing another example of the syntax structure of a sequence parameter set (displ_sequence_parameter_set_rbsp()) according to embodiments of the present invention.

[0536] In Fig. 54, dsps_entropy_continuation_enable is a flag that determines whether to allow entropy context information storage and inheritance when encoding / decoding displacement vectors in sps units. If the value of dsps_entropy_continuation_enable is 0, it indicates that context information storage and inheritance of displacement vectors referencing the corresponding sps are not allowed, and if it is 1, it indicates that context information storage and inheritance of displacement vectors referencing the corresponding sps are allowed.

[0537] According to embodiments, when decoding a displacement vector in a displacement vector decoder, as in FIG. 54, a flag for determining whether to allow storage and inheritance of context information may be parsed as one flag (dsps_entropy_continuation_enabled) for P-type and I-type displacement vectors to determine whether to allow storage and inheritance of context information. In this case, the displacement vector decoder may parse one flag (dh_entropy_continuation_flag) for determining whether to allow inheritance of context information based on the flag for allowing storage and inheritance of context information parsed from a higher level at the displacement vector level, as in FIG. 55.

[0538] In addition, in an embodiment such as FIG. 54, which parses a flag for determining whether to allow storage and inheritance of context information of a displacement vector as one, decoding can be performed by inheriting the context information of the displacement vector only when the type of the current displacement vector is P type (P_DISPL), as in FIG. 56.

[0539] Fig. 55 is a diagram showing another example of the syntax structure of a displacement vector header (displ_header()) according to embodiments. Fig. 56 is a diagram showing another example of the syntax structure of a displacement vector header (displ_header()) according to embodiments.

[0540] In FIG. 55 and FIG. 56, dh_entropy_continuation_flag is a flag that determines whether to perform entropy context information inheritance when encoding / decoding displacement vectors. If the value of dh_entropy_continuation_flag is 0, it indicates that the context information of the corresponding displacement vector is initialized with a probability defined according to the encoder / decoder agreement, and if it is 1, it can indicate that the entropy context information of the current displacement vector is initialized with the stored context information.

[0541] Fig. 57 is a flowchart showing an example of a transmission method according to embodiments. The transmission method according to embodiments may include a step of encoding mesh data (S31011) and a step of transmitting a bitstream including the encoded mesh data (S31012). In one embodiment, the bitstream transmitted in step (S31012) includes a base mesh bitstream, a displacement vector bitstream, and a texture map bitstream.

[0542] According to embodiments, the step of encoding mesh data (S31011) may include a process of encoding a base mesh, a process of encoding displacement vectors or displacement vector transformation coefficients, and a process of encoding a texture map.

[0543] That is, in the process of encoding the base mesh, the original mesh to be transmitted is first simplified and mesh parameterized to generate a base mesh. The generated base mesh is quantized, and in the case of an inter-frame, a motion vector is calculated from the previously referenced restored base mesh and the motion vector is encoded. In the case of an intra-frame, the base mesh (or static mesh) is encoded through static mesh encoding and transmitted as a base mesh bitstream.

[0544] The present disclosure relates to a method for encoding a base mesh (or static mesh) by storing entropy context information used to encode the base mesh in an entropy context information storage unit after the encoding is completed, and initialization can be performed using the context information when encoding the base mesh of the next frame. That is, the context information of the base mesh of the next frame can be initialized using the entropy context information stored in the entropy context information storage unit. In other words, the base mesh of the next frame can perform encoding by inheriting the entropy context information stored in the entropy context information storage unit.

[0545] In the present disclosure, when a motion vector of a current base mesh is motion vector encoded, entropy context information used to encode the motion vector of the base mesh is stored in an entropy context information storage unit after encoding is completed, and initialization can be performed using the context information when encoding the motion vector of the base mesh of the next frame. That is, the context information of the motion vector of the base mesh of the next frame can be initialized using the entropy context information stored in the entropy context information storage unit. In other words, the motion vector of the base mesh of the next frame can be encoded by inheriting the entropy context information stored in the entropy context information storage unit.

[0546] At this time, the encoding of the base mesh, the encoding of the motion vector of the base mesh, whether inheritance is allowed, the description of whether inheritance is allowed, and the signaling are omitted here, with reference to the descriptions of FIGS. 16 to 22 and FIGS. 47 to 51.

[0547] And in the process of encoding the displacement vector (or displacement information), the displacement vector is calculated between the mesh data that has been simplified through mesh simplification, refined and fitted, and the mesh data restored from the previously encoded base mesh. In order to efficiently encode the calculated displacement vector, the displacement vector coordinate system is transformed into a local coordinate system, and the displacement vector in the local coordinate system is transformed and quantized into a displacement vector transformation coefficient, and then encoded into a displacement vector bitstream and transmitted.

[0548] The present disclosure stores entropy context information used to encode a displacement vector in an entropy context information storage unit after encoding is completed, and initialization can be performed using the context information when encoding the displacement vector of the next frame. In other words, the context information of the displacement vector of the next frame can be initialized using the entropy context information stored in the entropy context information storage unit. In other words, the displacement vector of the next frame can be encoded by inheriting the entropy context information stored in the entropy context information storage unit.

[0549] At this time, the encoding of the displacement vector, whether inheritance is allowed, description of whether inheritance is allowed, and signaling are omitted here, with reference to the descriptions of FIGS. 23 to 30 and FIGS. 52 to 56.

[0550] Finally, in the process of encoding the texture map, a new texture map having color information corresponding to the texture coordinates of the restored mesh is generated through a texture map generation unit (11026), and the generated texture map is encoded through a texture map encoder (i.e., a 2D video encoder) (11027) and transmitted as a texture bitstream.

[0551] In the step (S31012) of transmitting a bitstream including the above mesh data, the base mesh bitstream, displacement vector bitstream, and texture bitstream generated as described above in the step (S31011) of encoding the above mesh data are generated into one bitstream through a multiplexing unit, and transmitted to a receiving device through a transmitting unit.

[0552] Fig. 58 is a flowchart showing an example of a receiving method according to embodiments. The receiving method according to embodiments may include a step (S32011) of receiving a bitstream including mesh data and a step (S32012) of decoding the mesh data included in the bitstream. In one embodiment, the step (S32011) of receiving the bitstream including mesh data receives a bitstream including a base mesh bitstream, a displacement vector bitstream, and a texture map bitstream. In addition, the step (S32011) of receiving the bitstream including mesh data also receives signaling information including an atlas sequence parameter set (ASPS). At this time, the signaling information may be received while being included in the bitstream and is also referred to as metadata.

[0553] According to embodiments, the step of decoding mesh data (S32012) may include a process of decoding a base mesh bitstream, a process of decoding a displacement vector bitstream, and a process of decoding a texture map bitstream.

[0554] That is, in the step (S32011) of receiving a bitstream including mesh data, the received bitstream is demultiplexed into a base mesh bitstream, a displacement vector bitstream, and a texture map bitstream through a demultiplexing unit, and then a process of decoding each of them is performed.

[0555] First, in the process of decoding the base mesh bitstream, the base mesh bitstream is decoded through a motion vector encoder (15012) for inter-frames and a static mesh decoder (15013) for intra-frames. The decoded base mesh is then passed through a base mesh restoration unit (15014) to undergo mesh refinement.

[0556] The present disclosure relates to a method for decoding a base mesh (or static mesh) by storing entropy context information used for decoding the base mesh in an entropy context information storage unit after decoding is completed, and initialization can be performed using the context information when decoding the base mesh of the next frame. That is, the context information of the base mesh of the next frame can be initialized using the entropy context information stored in the entropy context information storage unit. In other words, the base mesh of the next frame can perform decoding by inheriting the entropy context information stored in the entropy context information storage unit.

[0557] In the present disclosure, when a motion vector of a current base mesh is motion vector decoded, entropy context information used to decode the motion vector of the base mesh is stored in an entropy context information storage unit after decoding is completed, and initialization can be performed using the context information when decoding the motion vector of the base mesh of the next frame. That is, the context information of the motion vector of the base mesh of the next frame can be initialized using the entropy context information stored in the entropy context information storage unit. In other words, the motion vector of the base mesh of the next frame can be decoded by inheriting the entropy context information stored in the entropy context information storage unit.

[0558] At this time, the decoding of the base mesh, the decoding of the motion vector of the base mesh, whether inheritance is allowed, the description of whether inheritance is allowed, and the signaling are omitted here, with reference to the descriptions of FIGS. 32 to 38 and FIGS. 47 to 51.

[0559] And, in the process of decoding the displacement vector bitstream, the displacement vector bitstream is decoded in the reverse order of encoding through a displacement vector decoder (15017), inverse quantization and inverse transformation are performed, and then inversely transformed to a coordinate system in a displacement vector coordinate system inverse transformation unit (15020), and then restored as mesh geometry information together with base mesh data.

[0560] The present disclosure stores entropy context information used to decode a displacement vector in an entropy context information storage unit after decoding is completed, and initialization can be performed using the context information when decoding the displacement vector of the next frame. That is, the context information of the displacement vector of the next frame can be initialized using the entropy context information stored in the entropy context information storage unit. In other words, the displacement vector of the next frame can be decoded by inheriting the entropy context information stored in the entropy context information storage unit.

[0561] At this time, the decoding of the displacement vector, whether inheritance is allowed, description of whether inheritance is allowed, and signaling are omitted here, with reference to the descriptions of FIGS. 39 to 46 and FIGS. 52 to 56.

[0562] And, in the process of decoding the texture map bitstream, the received texture map bitstream is decoded through a texture map decoder (15021). The decoded texture map is used to generate a final restored mesh together with the restored geometry information in the mesh wall unit (15016).

[0563] As described above, in the present disclosure, after coding a static mesh or motion vector, entropy context information is stored, and this context information can be used for coding the next base mesh or motion vector. The same method can be applied to displacement vectors when the coding type is arithmetic coding. In addition, after decoding a static mesh or motion vector, entropy context information can be stored, and this context information can be used for decoding the next base mesh or motion vector. The same method can be applied to displacement vectors when the decoding type is arithmetic decoding.

[0564] This allows for bit reduction and improved encoding / decoding performance while maintaining existing compression performance by loading and using pre-stored context information without performing unnecessary initialization work.

[0565] Each of the parts, modules, or units described above may be software, processors, or hardware parts that execute sequential execution processes stored in memory (or storage units). Each of the steps described in the embodiments described above may be performed by processors, software, or hardware parts. Each of the modules / blocks / units described in the embodiments described above may operate as a processor, software, or hardware. In addition, the methods presented in the embodiments may be implemented as code. This code may be written on a processor-readable storage medium and thus may be read by a processor provided by an apparatus.

[0566] Furthermore, throughout the specification, when a part is said to "include" a component, this does not exclude other components, unless otherwise specifically stated, but rather implies the inclusion of other components. Furthermore, terms such as "part" described in the specification mean a unit that processes at least one function or operation, which may be implemented using hardware, software, or a combination of hardware and software.

[0567] For convenience of explanation, this specification has been described separately in each drawing. However, it is also possible to design new embodiments by combining the embodiments described in each drawing. Furthermore, designing a computer-readable recording medium containing a program for executing the previously described embodiments, as required by those skilled in the art, is also within the scope of the embodiments.

[0568] The devices and methods according to the embodiments are not limited to the configurations and methods of the embodiments described above, but the embodiments may be configured by selectively combining all or part of each embodiment so that various modifications can be made.

[0569] Although preferred embodiments of the embodiments have been illustrated and described, the embodiments are not limited to the specific embodiments described above, and various modifications may be made by those skilled in the art to which the present disclosure pertains without departing from the spirit or scope of the embodiments claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the embodiments.

[0570] The various components of the devices of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. The various components of the embodiments may be implemented by a single chip, for example, a single hardware circuit. The components according to the embodiments may be implemented by separate chips. At least one of the components of the devices of the embodiments may be configured with one or more processors capable of executing one or more programs, and the one or more programs may perform, or include instructions for performing, one or more of the operations / methods according to the embodiments. The executable instructions for performing the methods / operations of the devices of the embodiments may be stored in non-transitory CRMs or other computer program products configured to be executed by one or more processors, or may be stored in temporary CRMs or other computer program products configured to be executed by one or more processors. In addition, the memory according to the embodiments may be used as a concept including not only volatile memory (e.g., RAM, etc.), but also non-volatile memory, flash memory, PROM, etc. Additionally, it may include implementations in the form of carrier waves, such as transmissions via the Internet. Furthermore, processor-readable recording media may be distributed across network-connected computer systems, allowing processor-readable code to be stored and executed in a distributed manner.

[0571] In this document, " / " and "," are interpreted as "and / or". For example, "A / B" is interpreted as "A and / or B", and "A, B" is interpreted as "A and / or B". Additionally, "A / B / C" means "at least one of A, B, and / or C". Also, "A, B, C" means "at least one of A, B, and / or C". Additionally, "or" in this document is interpreted as "and / or". For example, "A or B" can mean 1) "A" only, 2) "B" only, or 3) "A and B". In other words, "or" in this document can mean "additionally or alternatively".

[0572] Various elements of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. Various elements of the embodiments may be implemented on a single chip, such as a hardware circuit. In some embodiments, the embodiments may optionally be implemented on separate chips. In some embodiments, at least one of the elements of the embodiments may be implemented within one or more processors that include instructions for performing operations according to the embodiments.

[0573] Additionally, the operations according to the embodiments described in this document may be performed by a transceiver device including one or more memories and / or one or more processors according to the embodiments. One or more memories may store programs for processing / controlling the operations according to the embodiments, and one or more processors may control various operations described in this document. One or more processors may be referred to as a controller, etc. The operations according to the embodiments may be performed by firmware, software, and / or a combination thereof, and the firmware, software, and / or a combination thereof may be stored in a processor or a memory.

[0574] Terms such as "first" and "second" may be used to describe various components of the embodiments. However, the various components according to the embodiments should not be interpreted in a limited manner by these terms. These terms are merely used to distinguish one component from another. For example, a first user input signal may be referred to as a "second user input signal." Similarly, a second user input signal may be referred to as a "first user input signal." The use of these terms should be interpreted as not departing from the scope of the various embodiments. Although "first user input signal" and "second user input signal" are both user input signals, they do not necessarily mean the same user input signals unless the context clearly indicates otherwise.

[0575] The terminology used to describe the embodiments is for the purpose of describing particular embodiments and is not intended to be limiting of the embodiments. As used in the description of the embodiments and in the claims, the singular is intended to include the plural unless the context clearly dictates otherwise. The expressions “and / or” are used to mean all possible combinations of the terms. The expression “comprises” or “includes” describes the presence of features, numbers, steps, elements, and / or components, but does not mean that additional features, numbers, steps, elements, and / or components are not included. Conditional expressions such as “if” or “when” used to describe the embodiments are not intended to be limited to only optional cases. When a specific condition is satisfied, a related action is performed in response to a specific condition, or a related definition is intended to be interpreted.

[0576] As described above, the relevant contents have been described in the best form for carrying out the embodiments.

[0577] As described above, the embodiments may be applied, in whole or in part, to mesh data transmission and reception devices and systems. Those skilled in the art will appreciate that various modifications and variations may be made to the embodiments within the scope of the embodiments. The embodiments may include modifications and variations, and such modifications and variations do not depart from the scope of the claims and their equivalents.

Claims

1. A step of receiving a bitstream containing mesh data; and A step of decoding the above mesh data; comprising: How to decode.

2. In the first paragraph, the step of decoding the mesh data A base mesh processing step for performing decoding on a base mesh or a motion vector of a base mesh included in the above bitstream; A displacement information processing step for performing decoding on displacement information included in the bitstream; and A decoding method comprising a texture map processing step of performing decoding on texture map information included in the above bitstream.

3. In the second paragraph, the base mesh processing step If it is an intra mode, a step of performing decoding using entropy context information for the current base mesh, and A step of storing entropy context information used for the above decoding is included, A decoding method in which the entropy context information used for decoding the current base mesh is initialized by inheriting the entropy context information stored after decoding the previous base mesh.

4. In the second paragraph, the base mesh processing step If it is an inter mode, a step of performing decoding using entropy context information for the motion vector of the current base mesh, and A step of storing entropy context information used for the above decoding is included, A decoding method in which the entropy context information used for decoding the motion vector of the current base mesh is initialized by inheriting the entropy context information stored after decoding the motion vector of the previous base mesh.

5. In the second paragraph, the displacement information processing step A step of performing decoding using entropy context information on the current displacement information, and A step of storing entropy context information used for the above decoding is included, The entropy context information used for decoding the current displacement information above is initialized by inheriting the entropy context information stored after decoding the previous displacement information. The above decoding is a decoding method that is an arithmetic-based decoding.

6. Memory; and At least one processor connected to the memory; At least one processor of the above: Receive a bitstream containing mesh data; and configured to decode the above mesh data; Decoding device.

7. In the 6th paragraph, the at least one processor, A base mesh processing unit that performs decoding on a base mesh or a motion vector of a base mesh included in the above bitstream; A displacement information processing unit that performs decoding on displacement information included in the bitstream; and A decoding device including a texture map processing unit that performs decoding on texture map information included in the above bitstream.

8. In the 7th paragraph, the base mesh processing unit, In intra mode, decoding is performed using entropy context information for the current base mesh, Stores the entropy context information used for the above decoding, A decoding device in which the entropy context information used for decoding the current base mesh is initialized by inheriting the entropy context information stored after decoding the previous base mesh.

9. In the 7th paragraph, the base mesh processing unit, In inter mode, decoding is performed using entropy context information for the motion vector of the current base mesh, Stores the entropy context information used for the above decoding, A decoding device in which the entropy context information used for decoding the motion vector of the current base mesh is initialized by inheriting the entropy context information stored after decoding the motion vector of the previous base mesh.

10. In paragraph 7, the displacement information processing unit, Decoding is performed using entropy context information for the current displacement information, Stores the entropy context information used for the above decoding, The entropy context information used for decoding the current displacement information above is initialized by inheriting the entropy context information stored after decoding the previous displacement information. The above decoding is a decoding device that is an arithmetic-based decoding.

11. Step of encoding mesh data; and A step of transmitting a bitstream including the encoded mesh data; comprising: Encoding method.

12. In the 11th paragraph, the encoding step A base mesh processing step that performs encoding on a base mesh generated by simplifying the original mesh or on the motion vectors of the base mesh; A displacement information processing step for performing encoding on displacement information generated based on the above base mesh; and An encoding method comprising a texture map processing step of performing encoding on a texture map.

13. In the 12th paragraph, the base mesh processing step If it is an intra mode, a step of performing encoding using entropy context information for the current base mesh, and A step of storing entropy context information used for the above encoding is included, An encoding method in which the entropy context information used for encoding the current base mesh is initialized by inheriting the entropy context information saved after encoding the previous base mesh.

14. In the 12th paragraph, the base mesh processing step If it is an inter mode, a step of performing encoding using entropy context information for the motion vector of the current base mesh, and A step of storing entropy context information used for the above encoding is included, An encoding method in which the entropy context information used for encoding the motion vector of the current base mesh is initialized by inheriting the entropy context information stored after encoding the motion vector of the previous base mesh.

15. In the 12th paragraph, the displacement information processing step A step of performing encoding using entropy context information on the current displacement information, and A step of storing entropy context information used for the above encoding is included, The entropy context information used for encoding the current displacement information above is initialized by inheriting the entropy context information stored after encoding the previous displacement information. The above encoding is an encoding method that is an arithmetic-based encoding.

Images