Mesh data transmission device, mesh data transmission method, mesh data reception device, and mesh data reception method

Abstract

A mesh data decoding method according to embodiments may comprise the steps of: decoding a base mesh in a bitstream; decoding displacement data in the bitstream; and decoding attribute data in the bitstream, wherein the step of decoding the displacement data in the bitstream includes a step of inversely transforming the displacement data and additionally inversely transforming the inversely transformed displacement data.

Description

Mesh data transmission device, mesh data transmission method, mesh data reception device and mesh data reception method

[0001] The embodiments provide a method for providing 3D content to provide various services to users, such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services.

[0002] Among 3D content, point cloud data or mesh data is a set of points in 3D space. However, there is a problem in that it is difficult to generate point cloud data or mesh data because there is a large amount of points in 3D space.

[0003] In other words, there is a problem in that a large amount of throughput is required to transmit and receive 3D data with a large amount of points, such as point cloud data or mesh data.

[0004] The technical problem according to the embodiments is to provide an apparatus and method for efficiently transmitting and receiving mesh data in order to solve the aforementioned problems, etc.

[0005] The technical problem according to the embodiments is to provide an apparatus and method for solving the latency and encoding / decoding complexity of mesh data.

[0006] The technical problem according to the embodiments is to provide an apparatus and method for improving the compression efficiency of displacement data.

[0007] However, the scope of rights of the embodiments is not limited to the technical problems described above, and may be extended to other technical problems that can be inferred by a person skilled in the art based on the entire content of this document.

[0008] To achieve the above-described purpose and other advantages, the decoding method according to the embodiments may include the steps of decoding a base mesh in a bitstream, decoding displacement data in the bitstream, and decoding attribute data in the bitstream.

[0009] According to embodiments, the step of decoding the displacement data may include decoding the displacement data included in the bitstream based on a video codec, inversely packing the decoded displacement data, inversely quantizing the inversely packed displacement data, and inversely converting the inversely quantized displacement data.

[0010] According to embodiments, the step of inversely transforming the displacement data may include the step of performing inverse transformation on the displacement data of the vertices of the decoded base mesh.

[0011] According to the embodiments, the step of inversely transforming the displacement data can be performed by applying connectivity information of the vertices of the base mesh to a component prediction method to inversely transform the displacement data.

[0012] According to embodiments, the step of inversely transforming the displacement data may include, when the decoded base mesh is subdivided into a plurality of subdivision levels, grouping the plurality of subdivision levels into one or more groups and performing inverse transformation on the displacement data of each vertex on a group basis.

[0013] According to the embodiments, the inverse transformation of the group unit can be performed based on at least one of a component prediction-based method, a lifting-based method, and an anomalous cosine transform-based method.

[0014] According to the embodiments, the step of inversely transforming the displacement data may further include the step of performing an additional lifting inverse transformation on the displacement data for which the inverse transformation was performed in the group unit.

[0015] According to embodiments, the bitstream may include parameters related to the conversion of the displacement data.

[0016] According to embodiments, a decoding device includes a memory and at least one processor connected to the memory, and the at least one processor may be configured to decode a base mesh in a bitstream, decode displacement data in the bitstream, and decode attribute data in the bitstream.

[0017] According to embodiments, the at least one processor may further include a displacement decoding unit that decodes displacement data included in the bitstream based on a video codec, a reverse packing unit that reverse packs the decoded displacement data, a reverse quantization unit that reverse quantizes the reverse packed displacement data, and a reverse conversion unit that reverse converts the reverse quantized displacement data.

[0018] According to embodiments, the encoding method may include the steps of encoding a base mesh of mesh data, encoding displacement data of the mesh data, and encoding attribute data of the mesh data.

[0019] According to embodiments, the step of encoding the displacement data may include the step of converting the displacement data, the step of quantizing the converted displacement data, the step of packing the quantized displacement data, and the step of encoding the packed displacement data based on a video codec.

[0020] According to embodiments, the encoding device includes a memory and at least one processor connected to the memory, and the at least one processor may be configured to encode a base mesh of mesh data, encode displacement data of mesh data, and encode attribute data of mesh data.

[0021] According to embodiments, a computer-readable storage medium can store a bitstream generated by the encoding method.

[0022] According to embodiments, the transmission method may include the step of acquiring a bitstream for mesh data, the bitstream being generated based on the step of encoding a base mesh of the mesh data, the step of encoding displacement data of the mesh data, and the step of encoding attribute data of the mesh data, and the step of transmitting data including the bitstream.

[0023] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can provide a high-quality 3D service.

[0024] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can achieve various video codec methods.

[0025] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can provide general-purpose 3D content such as autonomous driving services.

[0026] The mesh data transmission method and mesh data transmission device according to the embodiments perform a first lifting transformation of the displacement vector and additionally perform a second transformation for each subdivision level, and the mesh data reception method and mesh data reception device according to the embodiments additionally perform a second inverse transformation of the displacement vector for each subdivision level, so that the encoding performance of the encoder of the transmission device can be improved by reducing the redundancy of the displacement vector and the amount of transmitted bits can be reduced, and the decoding performance of the decoder of the reception device can be improved by performing decoding with the reduced amount of bits.

[0027] Drawings are included to further understand the embodiments, and the drawings illustrate the embodiments along with descriptions related to the embodiments. For a better understanding of the various embodiments described below, one must refer to the description of the embodiments below in relation to the following drawings, which include parts corresponding to similar reference numerals throughout the drawings.

[0028] FIG. 1(a) and FIG. 1(b) are drawings showing examples of encoders and decoders according to embodiments.

[0029] FIG. 2 shows a system for providing dynamic mesh content according to embodiments.

[0030] FIG. 3 shows a V-MESH compression method according to embodiments.

[0031] FIG. 4 shows the pre-processing of V-MESH compression according to the embodiments.

[0032] FIG. 5 illustrates a mid-edge subdivision method according to embodiments.

[0033] Figure 6 shows a displacement generation process according to embodiments.

[0034] FIG. 7 illustrates the encoding process of mesh data according to embodiments.

[0035] FIG. 8 illustrates the lifting conversion process for displacement according to the embodiments.

[0036] FIG. 9 illustrates the process of packing conversion coefficients according to embodiments into a 2D image.

[0037] FIG. 10 illustrates the attribute transfer process of the V-MESH compression method according to the embodiments.

[0038] FIG. 11 illustrates the decoding process of mesh data according to embodiments.

[0039] FIG. 12 is a drawing showing an example of a transmitting device according to embodiments.

[0040] FIG. 13 is a drawing showing an example of a receiving device according to embodiments.

[0041] FIG. 14 is a drawing showing another example of a transmitting device according to embodiments.

[0042] FIG. 15 is a detailed block diagram showing an example of a displacement vector encoder according to embodiments.

[0043] FIG. 16 is a detailed block diagram showing another example of a displacement vector encoder according to embodiments.

[0044] FIG. 17 is a block diagram showing an example of a displacement vector conversion unit according to embodiments.

[0045] FIG. 18 is a diagram showing an example of a displacement vector first-order transformation process of a displacement vector first-order transformation unit according to embodiments.

[0046] FIG. 19 is a diagram showing an example of a displacement vector quadratic transformation process of a displacement vector quadratic transformation unit according to embodiments.

[0047] FIG. 20 is a diagram showing an example of a method for determining a secondary transformation group of a secondary transformation group determination unit according to embodiments.

[0048] FIG. 21 is a diagram showing an example of signaling of secondary conversion method parameters according to embodiments.

[0049] FIG. 22 is a diagram showing an example of a second-order transformation method based on discrete cosine transform according to embodiments.

[0050] FIG. 23 is a diagram showing an example of a second transformation method based on a lifting wavelet transform according to embodiments.

[0051] FIG. 24 (a) is a drawing showing an example of a parallelogram prediction method according to embodiments, and FIG. 24 (b) is a drawing showing an example of a multiple parallelogram prediction method according to embodiments.

[0052] FIG. 25 (a) is a diagram showing an example of a trapezoid prediction method according to embodiments, and FIG. 25 (b) is a diagram showing an example of a multiple trapezoid prediction method according to embodiments.

[0053] FIG. 26 is a drawing showing another example of a receiving device according to embodiments.

[0054] FIG. 27 is a detailed block diagram showing an example of a displacement vector coordinate system inverse transformation unit according to embodiments.

[0055] FIG. 28 is a detailed block diagram showing another example of a displacement vector coordinate system inverse transformation unit according to embodiments.

[0056] FIG. 29 is a detailed block diagram showing an example of a displacement vector decoder according to embodiments.

[0057] FIG. 30 is a detailed block diagram showing another example of a displacement vector decoder according to embodiments.

[0058] FIG. 31 is a block diagram showing an example of a displacement vector inverse transformation unit according to embodiments.

[0059] FIG. 32 is a drawing showing an example of a displacement vector second-order inverse transformation unit according to embodiments.

[0060] FIG. 33 is a diagram showing an example of a second-order inverse transform method based on discrete cosine transform according to embodiments.

[0061] FIG. 34 is a diagram showing an example of a second-order inverse transform method based on lifting wavelet transform according to embodiments.

[0062] FIG. 35 is a diagram showing an example of a first-order inverse displacement vector transformation of a first-order inverse displacement vector transformation unit according to embodiments.

[0063] FIG. 36 is a diagram showing the syntax structure of a lifting conversion parameter set according to the first embodiment.

[0064] FIG. 37 is a diagram showing the syntax structure of a lifting conversion parameter set according to a second embodiment.

[0065] FIG. 38 is a diagram showing the syntax structure of a lifting conversion parameter set according to a third embodiment.

[0066] FIG. 39 is a flowchart showing an example of an encoding method according to embodiments.

[0067] FIG. 40 is a flowchart showing an example of a decoding method according to embodiments.

[0068] Preferred embodiments of the embodiments are described in detail, and examples thereof are shown in the accompanying drawings. The following detailed description, with reference to the accompanying drawings, is intended to describe preferred embodiments of the embodiments rather than merely embodiments that may be implemented according to the embodiments. The following detailed description includes details to provide a thorough understanding of the embodiments. However, it is obvious to those skilled in the art that the embodiments may be practiced without these details.

[0069] Most terms used in the embodiments are selected from those commonly used in the field, but some terms are chosen at the applicant's discretion, and their meanings are described in detail in the following description as necessary. Accordingly, the embodiments should be understood based on the intended meaning of the terms, rather than their mere names or meanings.

[0070] With the recent advancement of 3D data modeling and rendering technologies, research on generating and processing 3D data is being conducted in various fields such as Virtual Reality (VR), Augmented Reality (AR), autonomous driving, CAD (Computer-Aided Design) / CAM (Computer-Aided Manufacturing), and GIS (Geographic Information System). Depending on the representation format, 3D data can be represented as a point cloud, a mesh, etc. Among these, a mesh consists of geometry information representing coordinate values ​​for each vertex (or point), connectivity information representing the relationship between vertices, a texture map representing color information of the mesh surface as 2D image data, and texture coordinates representing mapping information between the mesh surface and the texture map. In this disclosure, a case where one or more of the elements constituting the mesh change over time is defined as a dynamic mesh, and a case where they do not change is defined as a static mesh. That is, dynamic mesh data may refer to mesh data that has an object or movement.

[0071] Because dynamic mesh data has a large amount of data for the elements that make up the mesh compared to 2D image data, technologies to efficiently compress it have been developed to store and transmit massive amounts of mesh data.

[0072] FIGS. 1(a) and FIGS. 1(b) show a V-DMC-based encoder and decoder according to embodiments. In particular, FIGS. 1(a) shows an encoder, and FIGS. 1(b) shows a decoder.

[0073] The basic structure of the currently ongoing V-DMC (v-mesh) is as shown in FIG. 1 (a) and FIG. 1 (b). The encoder according to FIG. 1 (a) and the decoder according to FIG. 1 (b) perform the encoding and decoding processes of media representing the dynamic mesh using Visual Volumetric Video-based Coding (V3C) technology. The pre-processor converts the input dynamic mesh representation into several V3C components (base mesh, displacement set, 2D representation of attributes, and atlas). The original mesh is simplified into a base mesh. The base mesh can be encoded using any mesh codec. The displacement vector can be encoded into a V3C geometry video component using any video codec based on a profile or SEI (supplemental enhancement information) message. For example, depending on the profile, displacement vectors (or displacement data) may be encoded through a video codec-based encoder, a zero-run length encoder, an arithmetic encoder, etc. Attribute data may include additional attributes. For example, texture or material information may be included as additional attributes and may be encoded based on any video codec. Atlas data contains information on how to perform inverse reconstruction and is provided to the V3C (or v-mesh) decoding and / or rendering system of the receiving device. For example, atlas data may include methods for subdividing the base mesh, methods for applying displacement vectors to the vertices of the subdivided mesh, methods for applying attributes to the reconstructed mesh, etc.

[0074] An encoder according to the embodiments may be composed of a memory and at least one processor connected to the memory. The at least one processor may be configured to perform operations such as a pre-processor, an atlas encoding unit, a basemesh encoding unit, a displacement vector encoding unit, a video encoding unit, and a multiplexer.

[0075] The atlas encoding unit generates an atlas bitstream by encoding the atlas of the mesh data. The basemesh encoding unit generates a basemesh bitstream by encoding the basemesh of the mesh data. The displacement vector encoding unit generates a displacement vector bitstream by encoding the displacement vector of the mesh data. The video encoding unit generates an attribute bitstream by encoding the attributes of the mesh data. The encoder according to the embodiments may generate parameter information related to each encoding (which may be referred to as signaling information, metadata, etc.). The encoder according to the embodiments may generate a compressed bitstream including parameter information, an atlas, a basemesh, a displacement vector, and / or attributes, etc.

[0076] The decoder according to the embodiments may be composed of a memory and at least one processor connected to the memory. The at least one processor may be configured to perform operations such as a demultiplexer, an atlas decoding unit, a basemesh decoding unit, a displacement vector decoding unit, and a video decoding unit.

[0077] The atlas decoding unit decodes the atlas within the bitstream. The basemesh decoding unit decodes the basemesh within the bitstream. The displacement vector decoding unit decodes the displacement vector within the bitstream. The video decoding unit decodes the attributes within the bitstream. The decoder according to the embodiments may perform each decoding operation based on parameter information within the bitstream. In the decoder according to the embodiments, the basemesh processing unit restores the current basemesh from the decoded basemesh based on the atlas and / or parameter information. In the decoder according to the embodiments, the displacement processing unit restores the displacement vector by performing coordinate system transformation, etc., of the decoded displacement vector based on the atlas and / or parameter information. In the decoder according to the embodiments, the mesh restoration unit restores the final mesh by combining the restored basemesh and the restored displacement vector based on the atlas and / or parameter information. The restored mesh processing unit of the decoder according to the embodiments can generate and render a reconstructed dynamic mesh image by combining the decoded attribute (or texture map) with the restored final mesh. That is, the reconstructed dynamic mesh image can be displayed to a user.

[0078] Below, the operation of the V-DMC encoder and decoder of FIG. 1 is explained in more detail.

[0079] FIG. 2 shows a system for providing dynamic mesh content according to embodiments.

[0080] The system of FIG. 2 includes a transmitting device (100) and a receiving device (110) according to embodiments. The transmitting device (100) may include a mesh video acquisition unit (101), a mesh video encoder (102), a file / segment encapsulator (103), and a transmitter (104). The receiving device (110) may include a receiving unit (111), a file / segment decapsulator (112), a mesh video decoder (113), and a renderer (114). Each component of FIG. 2 may correspond to hardware, software, a processor, and / or a combination thereof. Hereinafter, the mesh data transmitting device according to embodiments may be interpreted as a term referring to a 3D data transmitting device or a transmitting device (100), or a mesh video encoder (hereinafter, encoder) (102). The mesh data receiving device according to the embodiments may be interpreted as a term referring to a 3D data receiving device or a receiving device (110), or a mesh video decoder (hereinafter, decoder) (113).

[0081] The system of Fig. 2 can perform video-based dynamic mesh compression and decompression.

[0082] With advancements in 3D capture, modeling, and rendering, users can access various forms of 3D content, such as AR, XR, the metaverse, and holograms, across multiple platforms and devices. 3D content represents objects more sophisticatedly and realistically to enable users to enjoy immersive experiences, and for this purpose, the creation and use of 3D models require a large amount of data. Among the various types of 3D content, 3D meshes are widely used for efficient data utilization and realistic object representation. The embodiments include a series of processing steps in a system that uses such mesh content.

[0083] First, the method for compressing dynamic mesh data originates from the V-PCC (Video-based point cloud compression) standard technology for point cloud data. Point cloud data consists of data containing color information in addition to the coordinates (X, Y, Z) of vertices (or points). In this disclosure, the coordinates of a vertex (i.e., location information) are referred to as geometry information, and the color information of a vertex is referred to as attribute information; the geometry information and attribute information combined are referred to as vertex information or point cloud data. Mesh data refers to vertex information to which connectivity information between vertices has been added. Content can be created in the form of mesh data from the beginning when generating content. Alternatively, it can be converted into mesh data by adding connectivity information to point cloud data for use.

[0084] Currently, the MPEG standards organization defines the data types of dynamic mesh data as the following two types.

[0085] Category 1: Mesh data containing a texture map with color information.

[0086] Category 2: Mesh data with vertex colors as color information.

[0087] Currently, mesh coding standards for Category 1 data are in progress, and standardization work for Category 2 data is also planned for the future. The entire process for providing mesh content services may include an acquisition process, an encoding process, a transmission process, a decoding process, a rendering process, and / or a feedback process, as shown in Fig. 2.

[0088] To provide mesh content services, 3D data acquired through multiple cameras or special cameras can be processed into a mesh data type through a series of processes and then generated as a video. The generated mesh video is transmitted after undergoing a series of processes, and at the receiving end, the received data can be processed back into a mesh video and rendered. Through this, the mesh video is provided to the user, and the user can use the mesh content according to their intention through interaction.

[0089] A mesh compression system may include a transmitting device (100) and a receiving device (110) as shown in FIG. 2. The transmitting device (100) may encode a mesh video and output a bitstream, which may be transmitted to the receiving device (110) via a digital storage medium or network in the form of a file or streaming (streaming segment). The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.

[0090] In the transmitting device (100), the encoder may be referred to as a mesh video / image / picture / frame encoding device, and in the receiving device (110), the decoder may be referred to as a mesh video / image / picture / frame decoding device. The transmitter may be included in the mesh video encoder. The receiver may be included in the mesh video decoder. The renderer (114) may include a display unit, and the renderer and / or the display unit may be composed of separate devices or external components. The transmitting device (100) and the receiving device (110) may further include separate internal or external modules / units / components for the feedback process.

[0091] Mesh data represents the surface of an object using multiple polygons. Each polygon is defined by a vertex in 3D space and connectivity information indicating how those vertices are connected. It may also include vertex attributes such as vertex color and normals. Mapping information, which enables the surface of the mesh to be mapped to a 2D planar area, may also be included in the mesh attributes. Mapping can generally be described by a set of parametric coordinates, referred to as UV coordinates or texture coordinates, associated with the mesh vertices. The mesh contains a 2D attribute map, which can be used to store high-resolution attribute information such as textures, normals, and displacements. Here, "displacement" may be used interchangeably with "displacement information" or "displacement vector."

[0092] The mesh video acquisition unit (101) may include processing three-dimensional object data acquired through a camera, etc., into a mesh data type having the attributes described above through a series of processes, and generating a video composed of such mesh data. In the mesh video, the attributes of the mesh, namely vertices, polygons, connectivity information between vertices, color, normals, etc., may change over time. A mesh video having attributes and connectivity information that change over time in this way can be described as a dynamic mesh video.

[0093] A mesh video encoder (102) can encode an input mesh video into one or more video streams. A single video may contain multiple frames, and a single frame may correspond to a still image / picture. In this document, the term mesh video may include mesh images / frames / pictures, and mesh video may be used interchangeably with mesh images / frames / pictures. The mesh video encoder (102) can perform a video-based dynamic mesh (V-Mesh) compression procedure. The mesh video encoder (102) can perform a series of procedures such as prediction, transformation, quantization, and entropy coding for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0094] The file / segment encapsulation module (103) can encapsulate encoded mesh video data and / or mesh video-related metadata in the form of a file or the like. Here, the mesh video-related metadata may be received from a metadata processing unit or the like. The metadata processing unit may be included in the mesh video encoder (102) or may be configured as a separate component / module. The file / segment encapsulation module (103) can encapsulate the data into a file format such as ISOBMFF or process it into other forms such as DASH segments. Depending on the embodiment, the file / segment encapsulation module (103) may include mesh video-related metadata in the file format. The mesh video metadata may be included, for example, in various levels of boxes in the ISOBMFF file format or as data within a separate track in the file. According to an embodiment, the file / segment encapsulator (103) can encapsulate the mesh video-related metadata itself into a file.

[0095] The transmission processing unit may apply processing for transmission to the mesh video data encapsulated according to the file format. The transmission processing unit may be included in the transmission unit (104) or may be configured as a separate component / module. The transmission processing unit may process the mesh video data according to any transmission protocol. Processing for transmission may include processing for transmission via a broadcast network and processing for transmission via broadband. According to an embodiment, the transmission processing unit may receive not only the mesh video data but also mesh video-related metadata from the metadata processing unit and apply processing for transmission to it.

[0096] The transmission unit (104) can transmit encoded video / image information or data output in the form of a bitstream to the receiving unit (111) of the receiving device (110) via a digital storage medium or network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (104) may include elements for creating a media file through a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit (111) can extract the bitstream and transmit it to a decoding device.

[0097] The receiver (111) can receive mesh video data transmitted by the mesh data transmission device. Depending on the transmission channel, the receiver (111) may receive mesh video data through a broadcast network or through broadband. Alternatively, it may receive mesh video data through a digital storage medium.

[0098] The receiving processing unit can perform processing according to the transmission protocol on the received mesh video data. The receiving processing unit may be included in the receiving unit (111) or may be configured as a separate component / module. In order to correspond to the processing for transmission performed at the transmission side, the receiving processing unit may perform the reverse process of the aforementioned transmission processing unit. The receiving processing unit may transmit the acquired mesh video data to a file / segment decapsulator (112) and transmit the acquired mesh video related metadata to a metadata parser. The mesh video related metadata acquired by the receiving processing unit may be in the form of a signaling table.

[0099] The file / segment decapsulator (112) can decapsulate mesh video data in the form of a file received from the receiving processing unit. The file / segment decapsulator (112) can decapsulate files according to ISOBMFF, etc., to obtain a mesh video bitstream or mesh video-related metadata (metadata bitstream). The obtained mesh video bitstream can be transmitted to a mesh video decoder (113), and the obtained mesh video-related metadata (metadata bitstream) can be transmitted to a metadata processing unit. The mesh video bitstream may include metadata (metadata bitstream). The metadata processing unit may be included in the mesh video decoder (113) or may be configured as a separate component / module. The mesh video-related metadata obtained by the file / segment decapsulator (112) may be in the form of boxes or tracks within the file format. The file / segment decapsulator (112) may receive metadata required for decapsulation from the metadata processing unit if necessary. The mesh video-related metadata may be passed to the mesh video decoder (113) to be used in the mesh video decoding process, or passed to the renderer (114) to be used in the mesh video rendering process.

[0100] The mesh video decoder (113) can receive a bitstream and perform an inverse operation corresponding to the operation of the mesh video encoder (102) to decode the video / image. The decoded mesh video / image can be displayed through the display unit of the renderer (114). The user can view all or part of the rendered result through a VR / AR display or a general display.

[0101] The feedback process may include the process of transmitting various feedback information, which can be obtained during the rendering / display process, to the transmitting side or to the decoder of the receiving side. Interactivity in mesh video consumption may be provided through the feedback process. According to an embodiment, head orientation information, viewport information indicating the area the user is currently viewing, etc., may be transmitted during the feedback process. According to an embodiment, the user may interact with elements implemented in a VR / AR / MR / autonomous driving environment, and in this case, information related to such interaction may be transmitted to the transmitting side or the service provider side during the feedback process. According to an embodiment, the feedback process may not be performed.

[0102] Head orientation information can refer to information regarding the user's head position, angle, movement, etc. Based on this information, information about the area the user is currently viewing within the mesh video—that is, viewport information—can be calculated.

[0103] Viewport information may be information about the area currently being viewed by the user in the mesh video. Through this, gaze analysis can be performed to determine how the user consumes the mesh video and which areas of the mesh video they gaze at for how long. Gaze analysis may be performed at the receiving end and transmitted to the transmitting end via a feedback channel. Devices such as VR / AR / MR displays can extract the viewport area based on the user's head position / orientation, the vertical or horizontal FOV supported by the device, etc.

[0104] According to the embodiment, the aforementioned feedback information may not only be transmitted to the transmitting side but may also be consumed at the receiving side. That is, decoding and rendering processes at the receiving side may be performed using the aforementioned feedback information. For example, using head orientation information and / or viewport information, only the mesh video for the area currently viewed by the user may be preferentially decoded and rendered.

[0105] This document relates to embodiments of dynamic mesh video compression as described above. The methods / embodiments disclosed in this document may be applied to the MPEG (Moving Picture Experts Group) Video-based Dynamic Mesh Compression Method (V-Mesh) standard or next-generation video / image coding standards. Dynamic mesh video compression is a method for processing mesh connection information and attributes that change over time, and it can perform lossy and lossless compression for various applications such as real-time communication, storage, free-viewpoint video, and AR / VR.

[0106] The dynamic mesh video compression method described below is based on MPEG's V-Mesh method.

[0107] In this document, "picture" or "frame" generally refers to a unit representing a single image of a specific time period.

[0108] A pixel or pel may refer to the smallest unit that constitutes a picture (or image). Additionally, the term 'sample' may be used as a counterpart to pixel. A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the lumina component, only the pixel / pixel value of the chroma component, or only the pixel / pixel value of the depth component.

[0109] A unit may represent a basic unit of image processing. A unit may include at least one of a specific area of ​​a picture and information related to that area. Depending on the case, the term unit may be used interchangeably with terms such as block or area. In general, an MxN block may include samples (or sample arrays) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0110] As mentioned above, the encoding process of Fig. 2 is as follows.

[0111] In other words, the Video-based Dynamic Mesh Compression (V-Mesh) compression method can provide a method for compressing dynamic mesh video data based on 2D video codecs such as HEVC (High Efficiency Video Coding) and VVC (Versatile Video Coding). In the V-Mesh compression process, compression is performed by receiving the following data as input.

[0112] Input mesh: Contains the 3D coordinates of the vertices constituting the mesh, normal information for each vertex, mapping information for mapping the mesh surface to a 2D plane, and connection information between the vertices constituting the surface. The surface of the mesh can be represented by triangles or polygons of greater size, and connection information between the vertices constituting each surface is stored according to a defined shape. The input mesh can be saved in the OBJ file format.

[0113] Attribute map: (Hereafter, Texture map is used with the same meaning): It contains information on the attributes (color, normals, displacement, etc.) of a mesh and stores data in the form of mapping the mesh surface onto a 2D image. Mapping which part of the mesh (surface or vertex) corresponds to each data point in this attribute map is based on the mapping information contained in the input mesh. Since the attribute map holds data for each frame of the mesh video, it can also be referred to as an attribute map video. In the V-Mesh compression method, the attribute map primarily contains the mesh's color information and is stored in image file formats (PNG, BMP, etc.).

[0114] Material Library File: Contains material attribute information used in the mesh, specifically information that links the input mesh with its corresponding attribute map. This is saved in the Wavefront Material Template Library (MTL) file format.

[0115] In the V-Mesh compression method, the following data and information can be generated through the compression process.

[0116] Base Mesh: By simplifying (decimating) the input mesh through a pre-processing process, objects in the input mesh are represented using the minimum number of vertices determined according to user criteria.

[0117] Displacement: Displacement information used to represent the input mesh as similarly as possible using the base mesh, and is expressed in the form of 3D coordinates.

[0118] Atlas information: This is metadata required to reconstruct a mesh using base mesh, displacement, and attribute map information. It can be generated and utilized as sub-units (sub-mesh, patch, etc.) that constitute the mesh.

[0119] Referring to FIGS. 3 to 7, a method for encoding mesh position information (or vertex position information) is described, and referring to FIGS. 7 to 10, a method for restoring mesh position information and encoding attribute information (attribute map) is described.

[0120] FIG. 3 shows a V-MESH compression method according to embodiments.

[0121] FIG. 3 illustrates the encoding process of FIG. 2, and the encoding process may include a pre-processing process and an encoding process. The mesh video encoder (102) of FIG. 2 may include a pre-processor (200) and an encoder (201) as in FIG. 3. Additionally, the transmitting device of FIG. 2 may be broadly referred to as an encoder, and the mesh video encoder (102) of FIG. 2 may be referred to as an encoder. The V-Mesh compression method may include a pre-processing process (Pre-processing, 200) and an encoding process (Encoding, 201) as in FIG. 3. The pre-processor (200) of FIG. 3 may be located in front of the encoder (201) of FIG. 3. The pre-processor (200) and the encoder (201) of FIG. 3 may be referred to as a single encoder.

[0122] The pre-processor (200) can receive a static of dynamic mesh (M(i)) and / or an attribute map (A(i)). The pre-processor (200) can generate a base mesh (m(i)) and / or a displacement (d(i)) through pre-processing. The pre-processor (200) can receive feedback information from the encoder (201) and generate the base mesh and / or the displacement based on the feedback information.

[0123] The encoder (201) may receive a base mesh (m(i)), a displacement (d(i)), a static (M(i)) of a dynamic mesh, and / or an attribute map (A(i)). In the present disclosure, at least one of the base mesh (m(i)), the displacement (d(i)), the static (M(i)) of a dynamic mesh, and / or an attribute map (A(i)) may be referred to as mesh-related data. The encoder (201) may encode the mesh-related data to generate a compressed bitstream.

[0124] Figure 4 shows the pre-processing process of V-MESH compression according to the embodiments.

[0125] FIG. 4 illustrates the configuration and operation of the pre-processor of FIG. 3. In FIG. 4, the input mesh may include a static of dynamic mesh (M(i)) and / or an attribute map (A(i)). Additionally, the input mesh may include 3D coordinates of vertices constituting the mesh, normal information for each vertex, mapping information for mapping the mesh surface to a 2D plane, and connection information between vertices constituting the surface.

[0126] FIG. 4 illustrates a process of performing pre-processing on an input mesh. The pre-processing process (200) may include four main steps: 1) GoF (Group of Frame) generation, 2) Mesh Decimation, 3) UV parameterization, and 4) Fitting subdivision surface (300). According to the embodiments, GoF generation may be referred to as the GoF generation process or GoF generation section, Mesh Decimation as the Mesh Decimation process or Mesh Decimation section, UV parameterization as the UV parameterization process or UV parameterization section, and Fitting subdivision surface as the Fitting subdivision surface process or Fitting subdivision surface section. The pre-processor (200) can generate displacement and / or base mesh from the received input mesh and transmit it to the encoder (201). The pre-processor (200) can transmit GoF information associated with GoF generation to the encoder (201).

[0127] Below, each step of Fig. 4 is explained.

[0128] GoF Generation: This is the process of generating a reference structure for mesh data. If the number of vertices, the number of texture coordinates, vertex connection information, and texture coordinate connection information of the mesh of the previous frame and the current mesh are all identical, the previous frame can be set as the reference frame. That is, if only the vertex coordinate values ​​differ between the current input mesh and the reference input mesh, the encoder (201) can perform inter-frame encoding. Otherwise, intra-frame encoding is performed for the corresponding frame.

[0129] Mesh Decimation: This is the process of simplifying the input mesh to generate a simplified mesh, or base mesh. After selecting vertices to remove from the original mesh based on user-defined criteria, the selected vertices and the triangles connected to them can be removed.

[0130] In the process of performing mesh decimation, information regarding the input mesh (voxelized), target triangle ratio (TTR), and minimum triangle component (CCCount) is passed as input, and a decimated mesh can be obtained as output. In this process, connected triangle components smaller than the set minimum triangle component (CCCount) can be removed.

[0131] UV Parameterization: This is the process of mapping 3D surfaces to a texture domain for a decimated mesh. Parameterization can be performed using a UV Atlas tool. Through this process, mapping information is generated regarding where each vertex of the decimated mesh can be mapped to on a 2D image. This mapping information is expressed and stored as texture coordinates, and the final base mesh is generated through this process.

[0132] OrthoAtlas technology is a technique that generates texture coordinates using orthographic projection. In orthoAtlas technology, the processes of patch generation and patch packing are performed sequentially. First, Connected Components (CCs) are generated by splitting adjacent triangles, and then the optimal CCs are merged using a cost function to generate the patch. The cost function can measure the cost based on the degree of distortion that occurs when orthographically projecting the patch in each direction. Finally, texture coordinates can be calculated by packing the patch that minimizes the cost function into the texture domain. With orthoAtlas technology, texture coordinates can be derived in the base mesh decoder without compressing texture coordinate and texture connection information during the base mesh encoding process.

[0133] Fitting subdivision surface (300): This is a process of performing subdivision on a decimated mesh (i.e., a decimated mesh having texture coordinates). The displacement generated through this process and the base mesh are output to the encoder (201). A user-defined method, such as the mid-edge method, may be applied as the subdivision method. A fitting process is performed so that the input mesh and the mesh that has undergone subdivision become similar to each other. In this disclosure, the mesh on which the fitting process has been performed is referred to as the fitted subdivided mesh (or fitted subdivided mesh). This process is a process of performing fitting so that the mesh that has undergone subdivision on the base mesh becomes similar to the surface of the input mesh. As for the subdivision method, a user-defined method such as the Mid-edge method (see Fig. 5), Loop method, and LS3 method may be applied.

[0134] FIG. 5 illustrates a mid-edge subdivision method according to embodiments.

[0135] Figure 5 illustrates the mid-edge method of the fitting subdivision surface described in Figure 4. Referring to Figure 5, an original mesh containing four vertices is subdivided to create a sub-mesh. A sub-mesh can be created by generating a new vertex at the midpoint of the edges between vertices. Then, a fitting process is performed so that the input mesh and the sub-mesh become similar to each other, thereby creating a fitted subdivided mesh.

[0136] When a fitted subdivided mesh (hereinafter referred to as the fitted subdivided mesh) is generated, displacement is calculated using this result and a pre-compressed and decoded base mesh (hereinafter referred to as the reconstructed base mesh). That is, the reconstructed base mesh is subdivided in the same way as the fitting subdivision surface. The positional difference between this result and the fitted subdivided mesh for each vertex becomes the displacement for each vertex. Since displacement represents a positional difference in three-dimensional space, it is also expressed as a value in the (x, y, z) space of the Cartesian coordinate system. Depending on user input parameters, (x, y, z) coordinate values ​​can be converted into (normal, tangential, bi-tangential) coordinate values ​​of the local coordinate system.

[0137] FIG. 6 illustrates a displacement generation process according to embodiments. The displacement generation process of FIG. 6 may be performed in a pre-processor (200) or in an encoder (201).

[0138] Figure 6 illustrates in detail the method of calculating the displacement of the fitting subdivision surface (300) as described in Figure 5.

[0139] The encoder and / or pre-processor according to the embodiments may include 1) a subdivision unit, 2) a local coordinate system calculation unit, and 3) a displacement calculation unit. The subdivision unit may perform subdivision on the restored base mesh to generate a subdivided restored base mesh. Here, the restoration of the base mesh may be performed in the pre-processor (200) or in the encoder (201). The local coordinate system calculation unit receives the fitted subdivided mesh and the subdivided restored base mesh, and may convert the coordinate system regarding the mesh to a local coordinate system based on them. The local coordinate system calculation operation may be optional. The displacement calculation unit calculates the position difference between the fitted subdivision mesh and the subdivided restored base mesh. For example, it may generate a position difference value between the vertices of the two input meshes. The vertex position difference value becomes the displacement.

[0140] The mesh data transmission method and apparatus according to the embodiments can encode mesh data as follows. Mesh data is a term that includes point cloud data. Point cloud data according to the embodiments (which may be referred to as point cloud for short) may refer to data including vertex coordinates (or referred to as geometry information) and color information (or referred to as attribute information). Additionally, geometry images, attribute images, accusation maps, and additional information (or referred to as patch information) generated through patch generation and packing based on vertex coordinates and color information are also referred to as point cloud data. Therefore, point cloud data including connection information may be referred to as mesh data. In this document, point cloud and mesh data may be used interchangeably.

[0141] The V-Mesh compression (restoration) method according to the embodiments may include intra-frame encoding and inter-frame encoding.

[0142] Intra-frame encoding or inter-frame encoding is performed based on the results of the aforementioned GoF generation. In the case of intra-frame encoding, the data to be compressed may include the base mesh, displacement, and attribute map. In the case of inter-frame encoding, the data to be compressed may include displacement, attribute map, and the motion field between the reference base mesh and the current base mesh.

[0143] Figure 7 illustrates a V-DMC encoding process according to embodiments.

[0144] The elements of the transmitting device illustrated in FIG. 7 may be implemented in hardware, software, processors connected to memory, and / or combinations thereof. That is, the elements of the transmitting device of FIG. 7 may be implemented in hardware, software, firmware, or a combination thereof, including one or more processors or integrated circuits configured to communicate with one or more memories, although not illustrated in the drawing. One or more processors may perform at least one of the operations and / or functions of the elements of the transmitting device of FIG. 7 described above. Additionally, one or more processors may operate or execute a set of software programs and / or instructions for performing the operations and / or functions of the elements of the transmitting device of FIG. 7. The execution order of each block in FIG. 7 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0145] In the present disclosure, the operation process of a transmitting end for compressing and transmitting dynamic mesh data using V-Mesh compression technology may be as shown in FIG. 7. The transmitting device of FIG. 7 may support both an intra-frame encoding (or intra-encoding or intra-frame encoding) process and / or an inter-frame encoding (or inter-encoding or inter-frame encoding) process.

[0146] The encoding process of FIG. 7 illustrates in detail the encoding of the mesh video encoder (102) of FIG. 2. The encoder of FIG. 7 may include a pre-processor (200) and / or an encoder (201). The pre-processor (200) and encoder (201) of FIG. 7 may correspond to the pre-processor (200) and encoder (201) of FIG. 4.

[0147] The pre-processor (200) receives an input mesh and can perform the aforementioned pre-processing. Through pre-processing, a base mesh and / or a fitted subdivided mesh can be generated.

[0148] The quantizer (411) of the encoder (201) can quantize the base mesh and / or the fitted subdivided mesh.

[0149] According to embodiments, the base mesh quantized in the mesh quantization unit (411) may be output to a static mesh encoder (413) or a motion vector encoder (414) through a switching unit (412). According to embodiments, the base mesh is output to a motion vector encoder (414) through the switching unit (412) when inter-encoding is performed on the corresponding mesh frame, and is output to a static mesh encoder (413) through the switching unit (412) when intra-encoding is performed on the corresponding mesh frame. The motion vector encoder (414) may be referred to as a motion encoder.

[0150] For example, when intra-frame encoding is performed on the corresponding mesh frame, the base mesh can be compressed through a static mesh encoder (413). In this case, encoding can be performed on the connection information, vertex geometry information, vertex texture information, normal information, etc. of the base mesh. That is, the vertex coordinates, vertex connection information, texture coordinates, texture connection information, etc. of the mesh can be encoded in the static mesh encoder (413). The base mesh bitstream generated through encoding is transmitted to a multiplexer (not shown).

[0151] As another example, when performing inter-frame encoding for the corresponding mesh frame, the motion vector encoder (414) may take the current base mesh and the reference reconstructed base mesh (or reconstructed quantized reference base mesh) as inputs, calculate the motion vector between the two meshes, and encode the value. Additionally, the motion vector encoder (414) may perform a prediction based on connectivity information using a previously encoded / decoded motion vector as a predictor, and entropy-encode the difference motion vector (or residual motion vector) obtained by subtracting the predicted motion vector from the current motion vector. Depending on the embodiments, motion vector encoding may be performed at the vertex level or at the subgroup level. The motion vector bitstream generated through motion vector encoding is transmitted to a multiplexer (not shown) as a base mesh bitstream. That is, in the case of in-frame encoding, the static mesh bitstream is input to the multiplexer as a base mesh bitstream, and in the case of inter-frame encoding, the motion vector bitstream is input to the multiplexer as a base mesh bitstream.

[0152] In FIG. 7, the base mesh restoration unit (415) can generate a reconstructed base mesh by receiving a base mesh encoded in the static mesh encoder (413) or a motion vector encoded in the motion vector encoder (414). The base mesh restoration unit (415) performs the restoration of the base mesh according to the encoding type of the current mesh (inter-frame encoding or intra-frame encoding). For example, the base mesh restoration unit (415) can restore the base mesh by performing static mesh decoding on the base mesh encoded in the static mesh encoder (413). At this time, quantization can be applied before static mesh decoding, and inverse quantization can be applied in the inverse quantization unit (416) after static mesh decoding. That is, when intra-frame encoding is performed, the current base mesh can be restored by performing inverse quantization in the inverse quantization unit (416) on the base mesh quantized through the mesh quantization unit (411). As another example, the base mesh restoration unit (415) can restore the base mesh based on the restored quantized reference base mesh and the motion vector encoded by the motion vector encoder (414). That is, when cross-frame encoding is performed, the motion vector can be decoded using a motion vector decoding method, and then the decoded motion vector can be applied (i.e., added) to the reference restored base mesh to generate the current base mesh. In this case, if the motion vector is not quantized, the motion vector restoration process is omitted, and the current base mesh can be restored using the motion vector calculated by the motion vector encoder (414). The restored base mesh is output to the displacement vector calculation unit (417) and the mesh restoration unit (425).

[0153] According to the embodiments, the displacement vector calculation unit (417) can perform mesh subdivision on the restored base mesh. Additionally, the displacement vector calculation unit (417) can calculate a displacement vector, which is the difference in vertex positions between the subdivided restored base mesh and the fitted subdivided (or subdivided) mesh generated by the pre-processor (200). That is, the displacement vector is the difference in positions between the vertices of the two meshes so that the fitted subdivided (or subdivided) mesh becomes similar to the original mesh. At this time, the displacement vector can be calculated for the number of vertices of the subdivided mesh. That is, the displacement vector for the number of vertices of the subdivided (subdivided) mesh can be calculated through the displacement vector calculation unit (417).

[0154] The lifting transformation unit (418) can perform a lifting transformation on the input displacement vector to generate a lifting coefficient (or displacement vector transformation coefficient). The quantizer (419) can quantize the lifting coefficient, i.e., the displacement vector transformation coefficient.

[0155] In the present disclosure, the displacement vector or quantized displacement vector transformation coefficient can be encoded through a 2D video codec-based encoding method, and / or a zero-run length encoding method and / or an arithmetic encoding method, etc.

[0156] If an arithmetic encoding method is used, the displacement vector or quantized displacement vector transformation coefficient is encoded based on an arithmetic codec in the arithmetic encoding unit (421) after inter prediction in the inter prediction unit (420), and if a 2D video codec-based encoding method is used, the displacement vector or quantized displacement vector transformation coefficient is encoded based on a 2D video codec in the video encoding unit (423) after image packing in the image packing unit (422) and can be output as a displacement bitstream (i.e., compressed displacement bitstream). For example, the image packing unit (422) can pack an image based on quantized lifting coefficients (i.e., displacement vector transformation coefficients). The video encoding unit (423) can encode the packed image. That is, the quantized lifting coefficients are packed into a frame as a 2D image by the image packing unit (422), compressed through the video encoding unit (423), and output as a displacement bitstream (i.e., compressed displacement bitstream).

[0157] The displacement vector restoration unit (424) may include a video decoder, an image unpacking unit, an inverse quantizer, and an inverse linear lifting unit. That is, the displacement vector restoration unit (424) performs decoding on the encoded displacement vector in the video decoder, performs image unpacking in the image unpacking unit, performs inverse quantization in the inverse quantizer, and then performs inverse transformation in the inverse linear lifting unit to restore the displacement vector. The restored displacement vector is output to the mesh restoration unit (425). The mesh restoration unit (425) restores the deformed mesh based on the base mesh restored in the base mesh restoration unit (415) and the displacement vector restored in the displacement vector restoration unit (424). That is, the mesh restoration unit (425) restores the reconstructed and deformed mesh through the restored displacement output from the displacement vector restoration unit (424) and the restored base mesh (or subdivided restored base mesh) output from the inverse quantization unit (416). The present disclosure refers to the reconstructed and deformed mesh as the restored deformed mesh. The restored mesh (or restored deformed mesh) has restored vertices, connection information between vertices, texture coordinates, and connection information between texture coordinates.

[0158] The attribute transfer (426) receives an input mesh and / or an input attribute map and regenerates an attribute map based on the restored deformed mesh. An attribute map refers to a texture map corresponding to attribute information among the mesh data components, and in this disclosure, attribute map and texture map may be used interchangeably. Push-pull padding (427) can pad data into the attribute map based on a push-pull method. A color space conversion unit (428) can convert the space of the color components of the attribute map. For example, the attribute map can be converted from the RGB color space to the YUV color space. A video encoding unit (429, or referred to as a video encoder) can encode the attribute map and output it as a compressed attribute bitstream.

[0159] According to the embodiments, the atlas encoder (430) can generate a compressed atlas bitstream by encoding atlas information (or atlas data). The atlas bitstream generated through atlas information encoding is transmitted to a multiplexer (431). In the present disclosure, the atlas may be information required in a mesh reconstruction process and may refer to information such as tiles and patches. Additionally, the atlas information may refer to data required in processes such as 2D mapping of 3D objects, texture mapping information, mesh decoding, and mesh reconstruction, and may include additional information such as subdivision methods, transformation methods, quantization methods, and the location and size of patches within the atlas frame. In the present disclosure, the atlas information may be encoded through Exp-Golomb coding, etc., of the atlas encoder (430).

[0160] According to embodiments, a multiplexer (430) can multiplex an input compressed base mesh bitstream, a compressed displacement (or displacement vector) bitstream, a compressed attribute (or texture map) bitstream, and a compressed atlas bitstream to produce a single compressed bitstream. The multiplexed bitstream can be encapsulated into one or more tracks of a file.

[0161] According to the embodiments, the multiplexed bitstream or file from the multiplexer (431) may be transmitted over a network or stored on a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, etc., and the digital storage medium may include various storage media such as a USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.

[0162] In summary, for base meshes, encoding can be performed in different ways depending on the base mesh type (INTRA type, INTER type, SKIP type). If the base mesh is of the INTRA type, it can be encoded using the static mesh encoding method. If the base mesh is of the INTER type, the motion field between the reference base mesh and the current base mesh can be encoded. If the current base mesh is of the SKIP type, the reference base mesh can be guided to the current base mesh.

[0163] After being encoded in an encoder, the decoded base mesh can be processed to obtain a subdivided mesh. Subdivision algorithms such as mid-point subdivision and loop subdivision can be used.

[0164] Static Base Mesh Encoding (Intra Base Mesh Encoding): When performing intra encoding on the current basemesh, the base mesh generated during the pre-processing stage can be encoded using static mesh compression technology after undergoing a quantization process. Static mesh compression utilizes MPEG EdgeBreaker (MEB) technology, and the base mesh's vertex position information, mapping information (texture coordinates), vertex connectivity information, and normals are subject to compression.

[0165] Connection information can be encoded and compressed based on the edgebreaker algorithm. The edgebreaker algorithm is a technique that sequentially traverses triangles according to rules, maps symbols based on the characteristics of each triangle, and then encodes those symbols.

[0166] Techniques for compressing vertex location information can calculate predicted values ​​based on prediction techniques such as multiple parallelogram prediction, and then encode the residual value, which is the difference between the current vertex and the predicted value.

[0167] A technique for compressing mapping information (texture coordinates) can calculate a predicted value based on a prediction technique such as stretching, and then encode the residual value, which is the difference between the current mapping information (texture coordinates) and the predicted value.

[0168] Normal compression techniques can obtain predicted values ​​based on prediction techniques such as delta coding, multiple parallelogram prediction, and cross product-based prediction, and then encode the residual value, which is the difference between the current normal and the predicted value.

[0169] Motion Field Encoding (Inter Basemesh Encoding): Inter basemesh encoding can be performed when a one-to-one correspondence exists between the reference mesh and the current input mesh, differing only in their vertex position information. When performing inter encoding, instead of compressing the base mesh, the difference between the vertices of the reference base mesh and the current base mesh—that is, the motion field—is calculated and this information is encoded. The reference base mesh is the result of quantizing already decoded base mesh data and is determined by the reference frame index.

[0170] The motion field can be encoded as is, or the predicted motion field can be calculated by averaging the motion fields of the restored vertices among the vertices connected to the current vertex, and the residual motion field, which is the difference between this predicted motion field value and the current vertex's motion field value, can be encoded. This value can be encoded using entropy coding.

[0171] Displacement Encoding: After Base mesh encoding, reconstruction and inverse quantization are performed to Recon. A base mesh is generated, and the displacement between the result of performing subdivision on it and the Fitted subdivided mesh can be calculated. For effective encoding, a data transform process such as a wavelet transform can be applied to the displacement information, and Figure 8 shows the process of transforming displacement information using a lifting transform in V-Mesh. The displacement vector transformation coefficients generated through the transformation process are quantized, and the quantized transformation coefficients may be compressed through a video codec or through arithmetic encoding depending on the compression method.

[0172] When compressed through a video codec, the 2D image is packed as shown in Fig. 9. Transform coefficients are organized into one block for every N^2 (N*N) units, and each block can be packed in z-scan order. The number of horizontal blocks is fixed at N, while the number of vertical blocks can be determined by the number of vertices of the subdivided base mesh. Within a single block, the transform coefficients can be packed by aligning them using Morton code. The packed images generate a displacement video for every GoF unit, and this displacement video can be encoded using an existing video compression codec.

[0173] When compressed via arithmetic encoding, cross-frame prediction can be performed on the quantized displacement vector transformation coefficients. When cross-frame prediction is performed on the current quantized displacement vector transformation coefficients, the residual value, which is the difference between the current displacement vector transformation coefficient and the reference displacement vector transformation coefficient, can be encoded, and information about the reference target can be encoded. Depending on the displacement vector type, the quantized displacement vector transformation coefficients can be arithmetic encoded if it is of the INTRA type, and the residual value if it is of the INTER type. Arithmetic encoding can be performed based on Context Adaptive Binary Arithmetic Coding (CABAC). The CABAC process can first binarize the displacement vector data and map it to a bin string. The bin string can be an output binarized into 0s and 1s, where each 0 or 1 can be a bin. Each bin can be arithmetic encoded using context information selected from the context model, and a process of updating probabilities can be performed.

[0174] FIG. 8 illustrates the lifting conversion process for displacement according to the embodiments.

[0175] FIG. 9 illustrates the process of packing a conversion factor (or lifting factor) according to embodiments into a 2D image.

[0176] Figures 8 and 9 respectively show the process of converting the displacement of the encoding process of Figure 7 and the process of packing the conversion coefficients.

[0177] The encoding method according to the embodiments includes displacement encoding.

[0178] After base mesh encoding and / or motion field encoding, a reconstructed base mesh is generated through reconstruction and inverse quantization, and the displacement between the result of performing subdivision on this reconstructed base mesh and the fitted subdivided mesh generated through the fitting subdivision surface can be calculated (417 in FIG. 7). For effective encoding, a data transform process such as a wavelet transform can be applied to the displacement information (418 in FIG. 7).

[0179] FIG. 8 shows the process of transforming displacement information using a lifting transform in the lifting transform unit (418) of FIG. 7. For example, a linear wavelet-based lifting transform may be performed. The transformation coefficients generated through the transformation process are quantized in a quantizer (419) and then packed into a 2D image as shown in FIG. 9 through an image packing unit (422). The transformation coefficients are organized into one block for every 256 (=16×16) units, and each block can be packed in z-scan order. The number of horizontal blocks is fixed at 16, while the number of vertical blocks can be determined according to the number of vertices of the subdivided base mesh. The transformation coefficients can be packed by aligning them with a Morton code within a single block. Packed images generate a displacement video for each GoF unit, and this displacement video can be encoded using an existing video compression codec in a video encoding unit (423, or referred to as a video encoder).

[0180] Referring to FIG. 8, the base mesh (original) may include vertices and edges for Level of Detail (LoD) 0. A first subdivision mesh generated by dividing (or subdividing) the base mesh includes vertices generated by further dividing (or subdividing) the edges of the base mesh. The first subdivision mesh includes vertices for LoD0 and vertices for LoD1. LoD1 includes the subdivided vertices and the vertices of the base mesh (LoD0). A second subdivision mesh may be generated by dividing (or subdividing) the first subdivision mesh again. The second subdivision mesh includes LoD2. LoD2 includes the base mesh vertices (LoD0), LoD1 which includes vertices further divided (or subdivided) from LoD0, and vertices further divided (or subdivided) from LoD1. LoD is a Level of Detail that indicates the degree of detail in the mesh data content. As the level index increases, the distance between vertices decreases, and the level of detail increases. In other words, a smaller LoD value indicates lower detail in the mesh data content, while a larger LoD value indicates higher detail. LoD N includes the vertices contained in the previous LoD N-1. When a mesh (or vertex) is further subdivided through subdivision, the mesh can be encoded based on a prediction and / or update method by considering the previous vertices v1 and v2 and the subdivided vertex v. Instead of encoding the information for the current LoD N as is, residuals between the previous LoD N-1 can be generated and encoded using these residuals to reduce the size of the bitstream. The prediction process refers to the operation of predicting the current vertex v based on the previous vertices v1 and v2. Since adjacent subdivided meshes contain similar data, efficient encoding can be achieved by utilizing this property.The current vertex position information is predicted as a residual for the previous vertex position information, and the previous vertex position information is updated through the residual. In this disclosure, vertex, vertex, and point may be used interchangeably. Also, LoDs may be defined during the refinement process of the base mesh. According to embodiments, the refinement process of the base mesh may be performed in a pre-processor (200) or in a separate component / module.

[0181] Referring to FIG. 9, the vertex has a transformation coefficient (or lifting coefficient) generated through a lifting transformation. The transformation coefficient of the vertex related to the lifting transformation can be packed into an image by the image packing unit (422) and then encoded by the video encoding unit (423).

[0182] FIG. 10 illustrates the attribute transfer process of the V-MESH compression method according to the embodiments.

[0183] According to the embodiments, FIG. 10 shows the detailed operation of the attribute transfer (426) of FIG. 7.

[0184] The encoding according to the embodiments includes attribute map encoding. According to the embodiments, attribute map encoding can be performed in the video encoding unit (429) of FIG. 7.

[0185] According to embodiments, the encoder in the present disclosure compresses information about an input mesh through base mesh encoding (i.e., intra encoding), motion field encoding (i.e., inter encoding), and displacement encoding. The input mesh compressed during the encoding process is restored through base mesh decoding (intra frame), motion field decoding (inter frame), and displacement video decoding processes, and the restored result, the reconstructed deformed mesh (hereinafter referred to as Recon. deformed mesh), is used to compress an input attribute map as shown in FIG. 7. The reconstructed deformed mesh has vertex position information, texture coordinates, and corresponding connection information, but does not have color information corresponding to the texture coordinates. Accordingly, as shown in FIG. 10, in the V-Mesh compression method, a new attribute map having color information corresponding to the texture coordinates of the restored deformed mesh is regenerated through the attribute transfer process of the attribute transfer (426).

[0186] According to embodiments, attribute transfer (426) first checks for all points P(u, v) in a 2D texture domain whether the corresponding vertex belongs to a texture triangle of the reconstructed deformed mesh, and if it exists within a texture triangle T, the barycentric coordinate of P(u, v) according to that triangle T ( , , Calculate ). And the 3D vertex positions of triangle T and ( , , Calculate the 3D coordinates M(x, y, z) of P(u, v) using ). Find the vertex coordinates M'(x', y', z') corresponding to the location most similar to the calculated M(x, y, z) in the input mesh domain, and triangle T' containing this vertex. Then, find the coordinates of the centroid of M'(x', y', z') in this triangle T' ( ', ', Calculate '). Texture coordinates corresponding to the three vertices of Triangle T' and ( ', ', Texture coordinates (u', v') are calculated using '), and color information corresponding to these coordinates is found in the input attribute map. The color information found in this way is then assigned to the (u, v) pixel location in the new attribute map. If P(u, v) does not belong to any triangle, the pixel at that location in the new attribute map can be filled with a color value using a padding algorithm, such as the push-pull algorithm of push-pull padding (427).

[0187] The new attribute map generated through attribute transfer (426) is grouped into GoF units to form an attribute map video, which is then compressed using the video codec of the video encoding unit (429).

[0188] Referring to Fig. 10, the reference relationships between the input mesh, the input attribute map, the reconstructed deformed mesh, and the regenerated attribute map can be seen.

[0189] The decoding process of Fig. 2 can perform the reverse process of the corresponding process of the encoding process of Fig. 2. The specific decoding process is as follows.

[0190] FIG. 11 illustrates the decoding process of V-Mesh technology according to embodiments.

[0191] FIG. 11 illustrates the configuration and operation of a mesh video decoder (113) of the receiving device of FIG. 2. Additionally, FIG. 11 can restore mesh data by performing the reverse process of the encoding process of FIG. 7. In the present disclosure, the receiving device of FIG. 11 may be referred to as a mesh data receiving device, a decoder, a decoder of a receiving device, a V-Mesh decoder, or a dynamic mesh decoder.

[0192] The elements of the receiving device illustrated in FIG. 11 may be implemented in hardware, software, processors connected to memory, and / or combinations thereof. That is, the elements of the receiving device of FIG. 11 may be implemented in hardware, software, firmware, or a combination thereof, including one or more processors or integrated circuits configured to communicate with one or more memories, although not illustrated in the drawing. One or more processors may perform at least one of the operations and / or functions of the elements of the receiving device of FIG. 11 described above. Additionally, one or more processors may operate or execute a set of software programs and / or instructions for performing the operations and / or functions of the elements of the receiving device of FIG. 11. The execution order of each block in FIG. 11 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0193] FIG. 11 may largely include a demultiplexer (611), an atlas decoder (612), and a decoding unit (620).

[0194] According to embodiments, a bitstream of mesh data (i.e., a compressed bitstream) received by a receiver (not shown) may be demultiplexed in a demultiplexer (611) into a base mesh bitstream (or base mesh sub-stream), a displacement vector bitstream (or displacement sub-stream), a texture map bitstream (or attribute map sub-stream or texture map sub-stream), and / or an atlas bitstream after file / segment decapsulation. If the bitstream of mesh data is not encapsulated in the form of a file at the transmitting device, the decapsulation process at the receiving device is omitted. If the current mesh is inter-encoded, the base mesh bitstream may be a motion vector bitstream.

[0195] According to embodiments, an atlas bitstream is provided to an atlas decoder (612). The atlas decoder (612) can decode the atlas bitstream to restore atlas information. The restored atlas information can be used in a mesh decoding process. According to embodiments, the process in which the atlas information is used may be a subdivision process, a displacement vector restoration process, etc., and may include information such as tiles and patches.

[0196] According to the embodiments, the atlas bitstream can be decoded through the Exp-Golomb coding of the atlas decoder (612). At this time, the atlas may be information required in the mesh reconstruction process and may refer to information such as tiles and patches. Also, the atlas data may refer to data required in the mesh decoding, mesh reconstruction, etc. process and may include a subdivision method, a transformation method, a quantization method, the location and size of patches within the atlas frame, etc.

[0197] According to the embodiments, the base mesh bitstream is provided to the motion vector decoder (623) or to the static mesh decoder (622) through the switching unit (621).

[0198] For example, if the current mesh is inter-encoded, the base mesh bitstream, i.e., the motion vector bitstream, is received, demultiplexed, and then output to the motion vector decoder (623) through the switching unit (621). As another example, if the current mesh is intra-encoded, the base mesh bitstream is received, demultiplexed, and then output to the static mesh decoder (622) through the switching unit (621). Here, the motion vector decoder (623) may be referred to as a motion decoder.

[0199] According to the embodiments, the motion vector decoder (623) can perform decoding on the motion vector bitstream on a vertex-by-vertex or subgroup-by-subgroup basis.

[0200] According to embodiments, the motion vector decoder (623) can restore the final motion vector by adding the difference motion vector (i.e., residual motion vector) decoded from the bitstream using the previously decoded motion vector as a predictor. That is, the motion vector decoder (623) can decode the difference motion vector (or residual motion vector) in vertex or subgroup (or subblock) units through the motion vector bitstream, and decode the motion vector by adding the residual motion vector by performing a connection information-based prediction using the previously decoded motion vector as a predictor.

[0201] According to the embodiments, the static mesh decoder (622) can decode the base mesh bitstream to restore the connection information, vertex geometry information, texture coordinates (i.e., attribute geometry information), normal information, etc. of the base mesh. That is, the static mesh decoder (622) can restore the connection information, vertex geometry information, vertex texture coordinates, etc. of the restored quantized base mesh, for example, the base mesh.

[0202] According to embodiments, the base mesh restoration unit (631) can restore the current base mesh based on the decoded motion vector or the decoded base mesh. For example, if the current mesh is subjected to inter-frame encoding, the base mesh restoration unit (631) can generate the restored base mesh (i.e., the current base mesh) by adding the decoded (or restored) motion vector to the reference base mesh and then performing inverse quantization. As another example, if the current mesh is subjected to intra-frame encoding, the base mesh restoration unit (631) can generate the restored base mesh (i.e., the current base mesh) by performing inverse quantization on the base mesh decoded (or restored) through the static mesh decoder (622). According to embodiments, inverse quantization may be omitted.

[0203] According to the embodiments, the displacement sub-bitstream is provided to the arithmetic decoding unit (625) or to the video decoding unit (627) through the switching unit (624) depending on the decoding method.

[0204] For example, if the decoding method is an arithmetic codec method, the arithmetic decoding unit (625) decodes the displacement substream based on the arithmetic codec, and the inverse prediction unit (626) performs the inverse prediction process on the decoded displacement information and outputs it to the inverse quantization unit (629). As another example, if the decoding method is a 2D video codec method, the video decoding unit (627) decodes the displacement substream based on the 2D video codec, and the image unpacking unit (628) unpacks the image of the decoded displacement video and outputs it to the inverse quantization unit (629).

[0205] The displacement information provided by the above-mentioned inverse prediction unit (626) or image unpacking unit (628) is inversely quantized in the inverse quantization unit (629) and inversely transformed in the inverse linear lifting unit (630) to be restored as displacement information for each vertex (i.e., Recon. displacements).

[0206] According to the embodiments, the mesh restoration unit (632) restores a reconstructed and deformed mesh through the restored displacement output from the inverse linear lifting unit (630) and the restored base mesh output from the base mesh restoration unit (631) (i.e., decoded mesh). That is, the inversely quantized restored base mesh is combined with the restored displacement information to generate a final decoded mesh. In this disclosure, the final decoded mesh is referred to as a reconstructed deformed mesh.

[0207] According to the embodiments, the attribute map sub-stream is decoded through a video decoding unit (633) corresponding to the video compression codec used in encoding, and then restored to a final attribute map (i.e., decoded attribute map) through processes such as color format conversion and color space conversion in a color conversion unit (634).

[0208] According to the embodiments, the restored decoded mesh and decoded attribute map can be utilized at the receiving end as final mesh data that can be utilized by the user.

[0209] To summarize Fig. 11, the base mesh sub-stream can be decoded through a static mesh decoder (622) based on MEB (MPEG EdgeBreaker) technology, for example, if it is of the INTRA type, depending on the base mesh type, and as a result, connection information, vertex geometry information, vertex mapping information (texture coordinates), etc. of the base mesh can be restored.

[0210] In the case where the texture parameterization method in the encoder according to the embodiments is orthoAtlas, the decoder can derive mapping information (texture coordinates) and attribute information (texture) connection information using vertex coordinates. The process of deriving mapping information (texture coordinates) and connection information can generate mapping information (texture coordinates) and attribute information (texture) connection information by calculating the homography transform of each face and then projecting the vertex based on this.

[0211] According to the embodiments, when the base mesh type is an INTER type, motion information can be decoded through entropy decoding and inverse prediction processes. The decoded motion information is combined with a reference base mesh that has already been restored and stored in a buffer to generate a Reconstructed quantized base mesh for the current frame. An inverse quantization process can be performed on the restored base mesh.

[0212] In other words, when mesh data within the bitstream is encoded based on inter-prediction, the motion vector decoder derives the motion field of the basemesh of the current frame through motion estimation and compensation, based on the basemesh within the reference frame. When mesh data within the bitstream is encoded based on intra-prediction, the static mesh decoder decodes the basemesh.

[0213] Depending on the compression method used by the encoder, if the displacement sub-stream is compressed through a video codec, for example, it is decoded into displacement video through the decoder of the video compression codec, and then an image unpacking process is performed.

[0214] As another example, when compressed through arithmetic coding, the displacement sub-stream can be decoded into binarized syntax elements through arithmetic decoding, and a Contextual Probability Model (CPM) can be adaptively determined according to each bin of the syntax elements, and arithmetic decoding can be performed by predicting the probability of bin occurrence through the CPM. The binarized syntax elements can be decoded through inverse binarization. Quantized displacement vector transformation coefficients can be derived from the decoded syntax elements. As another example, when the displacement information type is INTER (when inter prediction is performed), an inverse inter prediction process is performed using reference information for the quantized displacement vector transformation coefficients.

[0215] The quantized displacement vector displacement coefficients are restored as displacement information for each vertex through inverse quantization, inverse transformation, and coordinate system transformation processes.

[0216] The restored base mesh and restored displacement information are combined to generate the final decoded mesh. The attribute map sub-stream is decoded through the decoder of the video compression codec used in the encoder, and then restored to the final attribute map through processes such as color format conversion.

[0217] The restored decoded mesh and decoded attribute map can be utilized at the receiving end as final mesh data available to the user.

[0218] FIG. 12 shows a mesh data transmission device according to embodiments.

[0219] FIG. 12 corresponds to the transmitting device (100) or mesh video encoder (102) of FIG. 2, the encoder (preprocessor and encoder) of FIG. 3 or FIG. 7, and / or the corresponding transmitting encoding device. Each component of FIG. 12 corresponds to hardware, software, a processor, and / or a combination thereof.

[0220] The operation process of the transmitting end for compressing and transmitting dynamic mesh data using V-Mesh compression technology may be as shown in FIG. 12. The transmitting device of FIG. 12 may perform an intra-frame encoding (or intra-encoding or intra-frame encoding) process and / or an inter-frame encoding (or inter-encoding or inter-frame encoding) process.

[0221] The pre-processor (811) receives the original mesh as input and generates a subdivided mesh that is fitted with the decimated mesh (or base mesh). Decimation can be performed based on the number of target vertices or the number of target polygons constituting the mesh. For the decimated mesh, parameterization (or parameterization) can be performed to generate texture coordinates and texture connection information per vertex. For example, parameterization is the process of mapping a 3D surface to a texture domain for the decimated mesh. If parameterization is performed using a UV Atlas tool, mapping information is generated that can identify where each vertex of the decimated mesh can be mapped to on a 2D image. The mapping information is stored in the form of texture coordinates, and through this process, the final base mesh is generated. Additionally, the task of quantizing floating-point mesh information into fixed-point form can be performed. This result can be output as a base mesh to a motion vector encoder (813) or a static mesh encoder (814) through a switching unit (812). The pre-processor (811) can generate additional vertices by performing mesh subdivision on the base mesh. Depending on the subdivision method, vertex connection information including the added vertices, texture coordinates, and connection information of texture coordinates can be generated. The pre-processor (811) can generate a fitted subdivided mesh by adjusting vertex positions so that the subdivided mesh becomes similar to the original mesh.

[0222] According to the embodiments, when inter-encoding is performed on the corresponding mesh frame, the base mesh is output to the motion vector encoder (813) through the switching unit (812), and when intra-encoding is performed on the corresponding mesh frame, it is output to the static mesh encoder (814) through the switching unit (812). The motion vector encoder (813) may be referred to as a motion encoder.

[0223] For example, when intra-frame encoding is performed on the corresponding mesh frame, the base mesh can be compressed through a static mesh encoder (814). In this case, encoding can be performed on the connection information, vertex geometry information, vertex texture information, normal information, etc. of the base mesh. The base mesh bitstream generated through encoding is transmitted to a multiplexer (823).

[0224] As another example, when inter-frame encoding is performed on the corresponding mesh frame, the motion vector encoder (813) can take a base mesh and a reference restored base mesh (or restored quantized reference base mesh) as inputs, calculate a motion vector between the two meshes, and encode the value. Additionally, the motion vector encoder (813) can perform a prediction based on connection information using a previously encoded / decoded motion vector as a predictor, and encode a residual motion vector obtained by subtracting the predicted motion vector from the current motion vector. The motion vector bitstream generated through encoding is transmitted to a multiplexer (823).

[0225] The base mesh restoration unit (815) can generate a reconstructed base mesh by receiving a base mesh encoded in the static mesh encoder (814) or a motion vector encoded in the motion vector encoder (813) as input. For example, the base mesh restoration unit (815) can restore the base mesh by performing static mesh decoding on the base mesh encoded in the static mesh encoder (814). At this time, quantization can be applied before static mesh decoding, and inverse quantization can be applied after static mesh decoding. As another example, the base mesh restoration unit (815) can restore the base mesh based on the restored quantized reference base mesh and the motion vector encoded in the motion vector encoder (813). The restored base mesh is output to the displacement calculation unit (816) and the mesh restoration unit (820).

[0226] The displacement calculation unit (816) can perform mesh subdivision on the restored base mesh. The displacement calculation unit (816) can calculate a displacement vector, which is the difference in vertex positions between the subdivided restored base mesh and the fitted subdivision (or subdivided) mesh generated by the pre-processor (811). At this time, the displacement vector can be calculated for as many vertices as the number of vertices of the subdivided mesh. The displacement calculation unit (816) can convert the displacement vector calculated in a 3D Cartesian coordinate system into a local coordinate system based on the normal vector of each vertex.

[0227] The displacement vector video generation unit (817) may include a linear lifting unit, a quantizer, and an image packing unit. That is, in the displacement vector video generation unit (817), the linear lifting unit can transform the displacement vector for effective encoding. Depending on the embodiments, the transformation may be performed as a lifting transformation, a wavelet transformation, etc. Additionally, quantization can be performed in the quantizer on the transformed displacement vector value, i.e., the transformation coefficient. At this time, different quantization parameters can be applied to each axis of the transformation coefficient, and the quantization parameters can be derived by the agreement of the encoder / decoder. The displacement vector information that has undergone transformation and quantization can be packed into a 2D image in the image packing unit. The displacement vector video generation unit (817) can generate a displacement vector video by bundling the packed 2D images for each frame, and the displacement vector video can be generated for each GoF (Group of Frame) unit of the input mesh.

[0228] The displacement vector video encoder (818) can encode the generated displacement vector video using a video compression codec. The generated displacement vector video bitstream is transmitted to a multiplexer (823).

[0229] The displacement vector restoration unit (819) may include a video decoder, an image unpacking unit, an inverse quantizer, and an inverse linear lifting unit. That is, the displacement vector restoration unit (819) performs decoding on the encoded displacement vector in the video decoder, performs image unpacking in the image unpacking unit, performs inverse quantization in the inverse quantizer, and then performs inverse transformation in the inverse linear lifting unit to restore the displacement vector. The restored displacement vector is output to the mesh restoration unit (820). The mesh restoration unit (820) restores a deformed mesh based on the base mesh restored in the base mesh restoration unit (815) and the displacement vector restored in the displacement vector restoration unit (819). The restored mesh (or referred to as the restored deformed mesh) has restored vertices, connection information between vertices, texture coordinates, and connection information between texture coordinates.

[0230] The texture map video generation unit (821) can regenerate a texture map based on the texture map (or attribute map) of the original mesh and the restored deformed mesh output from the mesh restoration unit (820). According to embodiments, the texture map video generation unit (821) can assign vertex-specific color information of the original mesh's texture map to the texture coordinates of the restored deformed mesh. According to embodiments, the texture map video generation unit (821) can generate a texture map video by grouping the texture maps regenerated for each frame into GoF units.

[0231] The generated texture map video can be encoded using the video compression codec of the texture map video encoder (822). The texture map video bitstream generated through encoding is transmitted to the multiplexer (823).

[0232] The multiplexer (823) multiplexes the motion vector bitstream (e.g., for inter-encoding), base mesh bitstream (e.g., for intra-encoding), displacement vector bitstream, and texture map bitstream into a single bitstream. The single bitstream can be transmitted to the receiver via the transmitter (824). Alternatively, the motion vector bitstream, base mesh bitstream, displacement vector bitstream, and texture map bitstream can be encapsulated into a file with one or more track data or segments and transmitted to the receiver via the transmitter (824).

[0233] Referring to FIG. 12, the transmitting device (encoder) can encode the mesh using an intra-frame or inter-frame method. The transmitting device according to intra-encoding can generate a base mesh, a displacement vector (or displacement), and a texture map (or attribute map). The transmitting device according to inter-encoding can generate a motion vector (or motion), a displacement vector (or displacement), and a texture map (or attribute map). The texture map obtained from the data input unit is generated and encoded based on the restored mesh. The displacement is generated and encoded through the difference in vertex positions between the base mesh and the divided (or subdivided) mesh. More specifically, the displacement is the difference in position between the fitted subdivided mesh and the subdivided restored base mesh, that is, the difference in vertex positions between the two meshes. The base mesh is generated by simplifying and encoding the original mesh through pre-processing. Motion is generated as motion vectors for the mesh of the current frame based on the reference base mesh of the previous frame.

[0234] FIG. 13 shows a mesh data receiving device according to embodiments.

[0235] FIG. 13 corresponds to the receiving device (110) or mesh video decoder (113) of FIG. 2, the decoder of FIG. 11, and / or a corresponding receiving decoding device. Each component of FIG. 13 corresponds to hardware, software, a processor, and / or a combination thereof. The receiving (decoding) operation of FIG. 13 may follow the reverse process of the corresponding process of the transmitting (encoding) operation of FIG. 12.

[0236] The bitstream of mesh data received by the receiver (910) is demultiplexed in the demultiplexer (911) into a compressed motion vector bitstream (e.g., inter-decoding) or base mesh bitstream (e.g., intra-decoding), displacement vector bitstream, and texture map bitstream after file / segment decapsulation. For example, if the current mesh is inter-frame encoding (i.e., inter-encoding), the motion vector bitstream is received, demultiplexed, and then output to the motion vector decoder (913) via the switching unit (912). As another example, if the current mesh is intra-frame encoding (i.e., intra-encoding), the base mesh bitstream is received, demultiplexed, and then output to the static mesh decoder (914) via the switching unit (912). Here, the motion vector decoder (913) may be referred to as the motion decoder.

[0237] According to embodiments, if the current mesh has inter-frame encoding applied according to the frame header information, the motion vector decoder (913) can perform decoding on the motion vector bitstream. According to embodiments, the motion vector decoder (913) can use the previously decoded motion vector as a predictor and add it to the residual motion vector decoded from the bitstream to restore the final motion vector.

[0238] According to the embodiments, if the current mesh has in-screen encoding applied according to the frame header information, the static mesh decoder (914) can decode the base mesh bitstream to restore the connection information, vertex geometry information, texture coordinates, normal information, etc. of the base mesh.

[0239] According to the embodiments, the base mesh restoration unit (915) can restore the current base mesh based on the decoded motion vector or the decoded base mesh. For example, if the current mesh has inter-frame encoding applied, the base mesh restoration unit (915) can generate the restored base mesh by adding the decoded motion vector to the reference base mesh and then performing inverse quantization. As another example, if the current mesh has intra-frame encoding applied, the base mesh restoration unit (915) can generate the restored base mesh by performing inverse quantization on the base mesh decoded through the static mesh decoder (914).

[0240] According to the embodiments, the displacement vector video decoder (917) can decode the displacement vector bitstream as a video bitstream using a video codec or decode it using an arithmetic codec. That is, depending on the encoding codec type, if the displacement vector bitstream is encoded through a video codec, for example, after decoding using a video codec, a reverse packing process can be performed. As another example, if it is encoded through arithmetic coding, arithmetic decoding can be performed on the displacement vector bitstream through a displacement vector arithmetic decoding unit, and if inter-frame prediction is performed, the current displacement vector transformation coefficient can be generated by adding the residual value to the reference displacement vector transformation coefficient through inter-frame prediction.

[0241] According to the embodiments, the displacement vector restoration unit (918) extracts displacement vector transformation coefficients from the decoded displacement vector video and restores the displacement vector by applying inverse quantization and inverse transformation processes to the extracted displacement vector transformation coefficients. To this end, the displacement vector restoration unit (918) may include an image unpacking unit, an inverse quantizer, and an inverse linear lifting unit. If the restored displacement vector is a value in the local coordinate system, an inverse transformation process to the Cartesian coordinate system may be performed.

[0242] The mesh restoration unit (916) can generate additional vertices by performing subdivision on the restored base mesh. Through subdivision, vertex connection information including the added vertices, texture coordinates, and connection information of the texture coordinates can be generated. At this time, the mesh restoration unit (916) can generate a final restored mesh (or restored deformed mesh) by combining the subdivided restored base mesh with the restored displacement vector.

[0243] According to the embodiments, the texture map video decoder (919) can restore the texture map by decoding the texture map bitstream as a video bitstream using a video codec. The restored texture map has color information for each vertex contained in the restored mesh, and the color value of the corresponding vertex can be obtained from the texture map using the texture coordinates of each vertex.

[0244] According to the embodiments, the mesh restored in the mesh restoration unit (916) and the texture map restored in the texture map video decoder (919) are shown to the user through a rendering process in the mesh data renderer (920).

[0245] Referring to FIG. 13, a receiving device (decoder) can decode a mesh in an intra-frame or inter-frame manner. A receiving device according to intra-decoding receives a base mesh, a displacement vector (or displacement), and a texture map (or attribute map), and can render mesh data based on the restored mesh and the restored texture map. A receiving device according to inter-decoding receives a motion vector (or motion), a displacement vector (or displacement), and a texture map (or attribute map), and can render mesh data based on the restored mesh and the restored texture map.

[0246] A mesh data transmission device and method according to the embodiments may pre-process mesh data, encode the pre-processed mesh data, and transmit a bitstream containing the encoded mesh data. A point mesh data reception device and method according to the embodiments may receive a bitstream containing mesh data and decode the mesh data. A mesh data transmission and reception method / device according to the embodiments may be referred to simply as a method / device according to the embodiments. A mesh data transmission and reception method / device according to the embodiments may also be referred to as a 3D data transmission and reception method / device or a point cloud data transmission and reception method / device.

[0247] As described above, in this disclosure, displacement information may be obtained based on a subdivided mesh (or referred to as a sub-mesh). That is, the difference in vertex positions between the subdivided reconstructed base mesh, generated by performing a fitting process to make the input mesh and the sub-mesh similar to each other and performing subdivision on the reconstructed base mesh, is calculated. In this disclosure, this value of the difference in vertex positions is referred to as a displacement vector. In this disclosure, the term displacement vector may be used interchangeably with the same meaning as displacement, displacement information, or displacement data. Furthermore, the term displacement video may be used interchangeably with the same meaning as displacement vector video or displacement vector transformation coefficient video, and the term displacement vector may be used interchangeably with the same meaning as displacement vector transformation coefficient or displacement vector coefficient.

[0248] In this disclosure, each vertex constituting the mesh represents a position in three-dimensional space and is expressed, for example, by x, y, and z coordinates (i.e., a canonical coordinate system). Also, the polygon may be a triangle or a quadrilateral. In this disclosure, vertex, vertex, and point may be used interchangeably. That is, a vertex has coordinates in 3D space, and a polygon of triangle or quadrilateral can be generated through connections between multiple vertices. Furthermore, the V-DMC referred to in this disclosure may also be referred to as V-mesh below, and the two terms are expressions used interchangeably.

[0249] The transmitting device of the present disclosure (or encoder or V-DMC encoder) may perform video codec-based encoding, zero-run length encoding, or arithmetic encoding on a displacement vector or displacement vector transformation coefficients. For example, in a displacement vector encoder, displacement vector transformation coefficients may be packed into a 2D image and then encoded using a 2D video codec (i.e., a video compression codec), or zero-run-length encoded, or arithmetic encoded to generate a displacement vector bitstream. In the present disclosure, the terms displacement vector bitstream and displacement vector video bitstream may be used interchangeably.

[0250] Conversely, the receiving device (or decoder or V-DMC decoder) can perform video codec-based decoding, zero-run length decoding, or arithmetic decoding on the displacement vector bitstream.

[0251] According to the embodiments, when performing video codec-based encoding on a displacement vector in a V-DMC encoder, the process may involve video encoding into a displacement vector bitstream after performing lifting transform, quantization, and image packing. In this case, the V-DMC decoder restores the displacement vector by performing the inverse process of the V-DMC encoder, namely video decoding, image inverse packing, inverse quantization, and inverse lifting transform of the displacement vector bitstream.

[0252] In this way, the V-DMC encoder / decoder performs a lifting transformation on the displacement vector of the entire level of details and signals / parses the resulting signal. In this disclosure, the lifting transformation (encoder) or the lifting inverse transformation (decoder) is referred to as the first-order transformation / inverse transformation or the first-order lifting transformation / inverse transformation.

[0253] In this embodiment, to further eliminate redundancy in the signal resulting from the first lifting transformation, a second transformation is performed on the signal resulting from the first lifting transformation for each subdivision level, and the resulting signal is signaled / parsed. Additionally, the present disclosure can determine the operation for the second transformation on a group basis, such as by forming groups of specific subdivision levels and selecting the same second transformation method for each subdivision level belonging to the group, or by omitting the second transformation of the subdivision levels belonging to the group. In this way, by additionally performing a second transformation for each subdivision level after performing the first lifting transformation of the displacement vector, the present disclosure can reduce the amount of transmission bits and improve encoding performance by reducing the redundancy of the displacement vector.

[0254] In other words, the present disclosure relates to a method for signaling / parsing the resulting signal by performing a secondary transformation method for each subdivision level on the signal resulting from a lifting transformation performed during the process of compressing 3D dynamic mesh data using an existing 2D video codec, in order to additionally remove redundancy in the signal resulting from said lifting transformation, and to related syntax and semantic information. This is because, although the current V-DMC standard performs a lifting transformation of displacement vectors as a step to increase the compression ratio, redundancy in the resulting signal still exists. The present disclosure reduces the redundancy of displacement vectors by performing a secondary transformation method for each subdivision level and signaling / parsing the resulting signal.

[0255] The following is a detailed description of a method, apparatus, syntax, and semantics for determining whether to perform a secondary transformation method for each subdivision level, or for groups of subdivision levels according to their characteristics and determining whether to perform the secondary transformation or to perform a specified secondary transformation depending on whether to perform the secondary transformation, after performing a lifting transformation of a displacement vector between dynamic mesh encoding / decoding according to the embodiments, and a method, apparatus, syntax, and semantics for omitting the secondary transformation or performing a specified secondary transformation depending on the determined decision.

[0256] FIG. 14 shows a transmitting device according to embodiments. The transmitting device of FIG. 14 may be referred to as a mesh data transmitting device or an encoder or an encoder of a transmitting device or a V-Mesh encoder or a dynamic mesh encoder.

[0257] FIG. 14 corresponds to the transmitting device (100) or mesh video encoder (102) of FIG. 1, the encoder (preprocessor and encoder) of FIG. 2 or FIG. 7, the transmitting device of FIG. 12 and / or the corresponding transmitting encoding device. Therefore, for parts not described in FIG. 14, reference is made to the description of the transmitting device (100) or mesh video encoder (102) of FIG. 1, the encoder (preprocessor and encoder) of FIG. 2 or FIG. 7, and the transmitting device of FIG. 12. The elements of the transmitting device illustrated in FIG. 14 may be implemented in hardware, software, a processor connected to memory, and / or a combination thereof. That is, the elements of the transmitting device of FIG. 14 may be implemented in hardware, software, firmware, or a combination thereof, including one or more processors or integrated circuits configured to communicate with one or more memories, although not illustrated in the drawings. One or more processors may perform at least one of the operations and / or functions of the elements of the transmitting device of FIG. 14 described above. Additionally, one or more processors may operate or execute a set of software programs and / or instructions for performing the operations and / or functions of the elements of the transmitting device of FIG. 14. The execution order of each block in FIG. 14 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0258] In the present disclosure, the operation process of a transmitting end for compressing and transmitting dynamic mesh data using V-Mesh compression technology may be as shown in FIG. 14. The transmitting device of FIG. 14 may support both an intra-frame encoding (or intra-encoding or intra-frame encoding) process and / or an inter-frame encoding (or inter-encoding or inter-frame encoding) process.

[0259] In FIG. 14, the mesh simplification unit (11011) simplifies the input original mesh through a mesh simplification algorithm to generate a base mesh (or simplified base mesh or simplified mesh). At this time, mesh simplification can be performed based on the number of target vertices or target polygons constituting the mesh. For example, methods such as decimation can be used as a mesh simplification algorithm to simplify the original mesh. That is, the decimation method may be a process of selecting vertices to be removed from the original mesh based on a reference point, and then removing the selected vertices and the triangles connected to the selected vertices.

[0260] That is, the mesh simplification unit (11011) can perform simplification of the input mesh to a target number of vertices or a target number of faces. In other words, the number of vertices or faces of the original mesh is determined based on the number of vertices or faces of the original mesh to have a number less than or equal to that number, and the mesh can be simplified to have the determined number of vertices or faces. At this time, the simplification process can be performed through various methods such as triangle collapse and edge collapse.

[0261] According to the embodiments, the simplified base mesh in the mesh simplification unit (11011) is provided to the mesh parameterization (or texture parameterization) unit (11012) and the mesh subdivision unit (11018).

[0262] The mesh parameterization unit (11012) performs the process of mapping a 3D surface to a texture domain for a decimated mesh. That is, the mesh parameterization unit (11012) generates texture coordinates and texture connection information of the input mesh. In one embodiment, the mesh parameterization unit (11012) may perform parameterization using a UV Atlas tool. Through this process, mapping information is generated regarding where each vertex of the decimated mesh can be mapped to on a 2D image. The mapping information is expressed as texture coordinates and stored, and through this process, a final base mesh is generated. That is, the mesh parameterization unit (11012) generates a base mesh by performing parameterization that generates texture coordinates (UV coordinates) and texture connection information per vertex of the input mesh (i.e., the decimated mesh or the decimated base mesh).

[0263] The final base mesh (or base mesh having a texture map) generated in the above parameterization unit (11012) is input to the mesh quantization unit (11013) and quantized.

[0264] According to embodiments, the mesh quantization unit (11013) can perform the task of quantizing mesh information in floating-point form (e.g., geometry information (x,y,z) or / and texture coordinates (u,v), normal information (nx, ny, nz), etc.) into a fixed-point form. That is, the mesh quantization unit (11013) can quantize the vertex coordinates and texture coordinates of the base mesh. According to embodiments, quantization for specific components may be omitted.

[0265] The mesh subdivision unit (11018) subdivides the base mesh simplified in the mesh simplification unit (11011). That is, the mesh subdivision unit (11018) can generate additional vertices by performing mesh subdivision on the base mesh. At this time, the generated vertices may include information such as geometry information, connectivity information, and texture coordinates calculated according to the subdivision method. In other words, geometry information, connectivity information, texture coordinate connectivity information, and texture coordinates can be implicitly derived and generated according to the subdivision method. According to the embodiments, the mesh subdivision unit (11018) can perform subdivision through methods such as mid-edge, Loop, and Catmul & Clark.

[0266] More specifically, in the mesh subdivision unit (11018), mesh subdivision may be performed n times by user parameters or by an agreement between the encoder (i.e., the transmitting device) and the decoder (i.e., the receiving device). According to embodiments, the number of mesh subdivisions in this disclosure may be determined by an agreement between the encoder and the decoder, or by receiving a mesh subdivision count parameter (subdivision_iteration_count) signaled / parsed in units such as a sequence, GOF (Group Of Frames), frame, or submesh. According to embodiments, the number of times mesh subdivision has been performed may be defined as the level of details.

[0267] According to the embodiments, the vertices of the base mesh are the vertices of level 0 (R0), the vertices newly generated by performing subdivision 1 are the vertices of level 1 (R1), … , the vertices newly generated by performing subdivision n are the vertices of level n (R n It can be defined as the vertices of ). For convenience of explanation, the present disclosure refers to the vertices of Level 0 (R0) as Level 0 vertices, the vertices of Level 1 (R1) as Level 1 vertices, … , Level n (Rn The vertices of ) can be referred to as vertices of level n. Also, level 0 is the 0th level (R0), level 1 is the 1st level (R1), ..., level n is the nth level (R n It may be referred to as ). In addition, the present disclosure covers Level 0 to Level n (R0 to R n ) can be referred to as refinement level 0 to refinement level n or subdivision level 0 to subdivision level n or LoD level 0 to LoD level n.

[0268] According to the embodiments, LoD0, LoD1, … , LoDn can be defined as follows. For example, vertex LoD included in any refinement level n. n It can be defined as follows.

[0269] LoD0 = R0

[0270] LoD1 = R0 + R1 = LoD0 + R1

[0271] …

[0272] LoDn = R0 + R1 + ..., ..., + R n = LoDn-1 + R n = R0∪ R1∪, … , ∪ R n

[0273] That is, the base mesh contains vertices for LoD0. And, the first subdivision mesh created by subdividing the base mesh once contains vertices for LoD1. That is, LoD1 includes the vertices of the base mesh (i.e., R0 vertices) and the vertices of Level 1 newly created by subdividing the base mesh (i.e., R1 vertices). Also, the second subdivision mesh created by subdividing the first subdivision mesh again contains vertices for LoD2. That is, LoD2 includes the vertices of the base mesh (i.e., R0 vertices), the vertices of Level 1 newly created by subdividing the base mesh (i.e., R1 vertices), and the vertices of Level 2 newly created by subdividing the first subdivision mesh (i.e., R2 vertices).

[0274] As such, LoD represents the degree of detail of the mesh data content; as the LoD (or level) index increases, the distance between vertices decreases, and the level of detail increases. In other words, a smaller LoD value indicates lower detail in the mesh data content, while a larger LoD value indicates higher detail in the mesh data content.

[0275] According to the embodiments, different mesh subdivision methods may be performed in each subdivision process, and the mesh subdivision method to be performed in each subdivision process may be determined by an agreement between encoders / decoders or by receiving a mesh subdivision method parameter (subdivision_method) signaled / parsed in units such as sequences, GOFs, frames, and submeshes.

[0276] According to the embodiments, the mesh fitting unit (11019) can generate a fitted subdivided mesh by performing fitting through vertex position adjustment and deformation, etc., so that the mesh subdivided in the mesh subdivision unit (11018) becomes similar to the original mesh.

[0277] According to the embodiments, the mesh simplification unit (11011), mesh parameterization unit (11012), mesh subdivision unit (11018), and mesh fitting unit (11019) may be omitted, and if the processes are omitted, the original mesh may be applied as an input to the mesh quantization unit (11013).

[0278] At this time, coordinate information of the original mesh may be applied as input to the displacement vector calculation unit (11020), and depending on the embodiments, the displacement vector encoding process (displacement vector calculation unit (11020), displacement vector coordinate system conversion unit (11021), and displacement vector encoder (11022)) may be omitted.

[0279] The present disclosure may be referred to as a pre-processor comprising a mesh simplification unit (11011), a mesh parameterization unit (11012), a mesh subdivision unit (11018), and a mesh fitting unit (11019). According to embodiments, the pre-processor may further include a displacement vector calculation unit (11020).

[0280] According to embodiments, a base mesh that is quantized or not quantized in the mesh quantization unit (11013) may be output to a motion vector encoder (11015) or a static mesh encoder (11016) through a switching unit (11014). According to embodiments, the base mesh is output to a motion vector encoder (11015) through the switching unit (11014) when inter-encoding is performed on the corresponding mesh frame, and is output to a static mesh encoder (11016) through the switching unit (11014) when intra-encoding is performed on the corresponding mesh frame. The motion vector encoder (11015) may be referred to as a motion encoder.

[0281] For example, when intra-frame encoding is performed on the corresponding mesh frame, the base mesh can be compressed through a static mesh encoder (11016). In this case, encoding can be performed on the connection information, vertex geometry information, vertex texture information, normal information, etc. of the quantized or non-quantized base mesh. That is, the vertex coordinates, vertex connection information, texture coordinates, texture connection information, etc. of the mesh can be encoded in the static mesh encoder (11016). The base mesh bitstream generated through encoding is transmitted to a multiplexer (not shown).

[0282] As another example, when performing inter-frame encoding for the corresponding mesh frame, the motion vector encoder (11015) may take the current base mesh and the reference restored base mesh (or the restored quantized reference base mesh) as inputs, calculate the motion vector between the two meshes, and perform encoding on the calculated motion vector. Additionally, the motion vector encoder (11015) may perform a prediction based on connectivity information using a previously encoded / decoded motion vector as a predictor, and perform encoding on the difference motion vector (or residual motion vector) obtained by subtracting the predicted motion vector from the current motion vector. Depending on the embodiments, motion vector encoding may be performed at the vertex level or subgroup level, or it may be omitted. The motion vector bitstream generated through motion vector encoding is transmitted to a multiplexer (not shown) as a base mesh bitstream. That is, in the case of in-frame encoding, the static mesh bitstream is input to the multiplexer as a base mesh bitstream, and in the case of inter-frame encoding, the motion vector bitstream is input to the multiplexer as a base mesh bitstream.

[0283] In FIG. 14, the base mesh restoration unit (11017, or base mesh decoder) can generate a reconstructed base mesh by receiving a base mesh encoded in a static mesh encoder (11016) or a motion vector encoded in a motion vector encoder (11015). According to embodiments, the base mesh restoration unit (11017) performs the restoration of the current base mesh according to the encoding type of the current base mesh (inter-frame encoding or intra-frame encoding).

[0284] For example, the base mesh restoration unit (11017) can restore the base mesh by performing static mesh decoding on the base mesh encoded by the static mesh encoder (11016). At this time, quantization can be applied before static mesh decoding, and inverse quantization can be applied after static mesh decoding. That is, when in-frame encoding is performed, the current base mesh can be restored by performing inverse quantization on the base mesh quantized through the mesh quantization unit (11013). More specifically, when in-frame encoding is performed, the connection information, vertex geometry information, vertex texture coordinates, normal information, etc. of the current base mesh encoded through the static mesh encoder (11016) can be decoded (i.e. restored).

[0285] As another example, the base mesh restoration unit (11017) can restore the base mesh based on the restored quantized reference base mesh and the motion vector encoded by the motion vector encoder (11015). That is, when cross-frame encoding is performed, the motion vector can be decoded using a motion vector decoding method, and then the decoded motion vector can be applied (i.e., added) to the reference restored base mesh to generate the current base mesh. More specifically, when cross-frame encoding is performed, the motion vector encoded through the motion vector encoder (11015) can be decoded, and the decoded motion vector can be added to the reference restored base mesh to decode (i.e., restore) the connection information, vertex geometry information, vertex texture coordinates, normal information, etc. of the current base mesh. At this time, if the motion vector is not quantized, the motion vector restoration process can be omitted, and the current base mesh can be restored using the motion vector calculated by the motion vector encoder (11015). The restored base mesh is output to the displacement vector calculation unit (11020), the displacement vector coordinate system transformation unit (11021), and / or the mesh inverse quantization unit through the mesh subdivision unit (11024). That is, the restored base mesh can be generated by performing inverse quantization on the connection information, vertex geometry information, vertex texture coordinates, normal information, etc. of the current base mesh decoded (or restored) in the base mesh restoration unit (11017). According to the embodiments, if quantization for a specific component is not performed, inverse quantization for that component may be omitted.

[0286] According to the embodiments, the mesh subdivision unit (11024) can perform mesh subdivision on the restored base mesh. The displacement vector calculation unit (11020) can calculate a displacement vector, which is a value of the difference in vertex-by-vertex position information between the restored base mesh subdivided by the mesh subdivision unit (11024) and the fitted subdivision (or subdivided) mesh generated by the mesh fitting unit (11019). At this time, the displacement vector can be calculated as many times as the number of vertices of the subdivided mesh. That is, the same number of displacement vectors as the number of vertices of the restored mesh subdivided through the mesh subdivision unit (11024) or the displacement vector calculation unit (11020) can be calculated.

[0287] According to embodiments, the displacement vector coordinate system transformation unit (11021) can transform the displacement vector of each vertex calculated in a 3D Cartesian coordinate system (i.e., (x, y, z) space) into a local coordinate system (i.e., normal, tangential, bi-tangential coordinate system) based on the normal vector of each vertex. At this time, the normal vector can be calculated for each subdivided vertex based on geometry information and / or connectivity information of surrounding vertices. In the present disclosure, whether to transform the displacement vector coordinate system may be determined by an agreement between the encoder and decoder, or by signaling a coordinate system transformation flag (applyLocalCoord) in units such as sequences, GOFs, frames, and submeshes. According to embodiments, when a coordinate system transformation is applied, only the normal component of the (normal, tangential, bi-tangential) coordinate system may be encoded. This can be determined by the agreement between the encoder and decoder, where if a coordinate system transformation is applied, only the normal component is encoded, or by the encoder signaling a 1-bit flag (onlyNormFlag) and the decoder parsing the 1-bit flag (onlyNormFlag).

[0288] According to embodiments, the displacement vector encoder (11022) can perform encoding on the displacement vector or displacement vector transformation coefficients output from the displacement vector calculation unit (11020) or the displacement vector coordinate system transformation unit (11021). In the present disclosure, the displacement vector encoder (11022) can perform encoding of the displacement vector through a 2D video codec-based encoder such as H.264, HEVC, VVC, etc., and / or a zero-run length encoder, and / or an arithmetic encoder, etc. In this case, the displacement vector encoder (11022) can selectively use only one of the 2D video codec-based encoding method, the zero-run length encoding method, and the arithmetic encoding method, or use one or more various combinations to encode the displacement vector or quantized displacement vector transformation coefficients. For example, if the encoding method is a video codec-based encoding, the displacement vector encoder (11022) can encode by packing the displacement vector transformation coefficients into a frame. That is, the displacement vector transformation coefficients in the displacement vector encoder (11022) can be packed into a 2D image and then encoded using a 2D video codec (i.e., a video compression codec), or zero run-length encoded, or arithmetic encoded to be generated into a displacement vector video bitstream.

[0289] According to the embodiments, a displacement vector encoding method may be determined based on a profile defined in the encoder / decoder, and an index (profileToolsetIdx) representing the profile information may be signaled and transmitted to the receiver, and the decoder of the receiver may determine a displacement vector decoding method based on the profileToolsetIdx. Additionally, the encoding / decoding method of the displacement vector may be determined by an agreement between the encoder / decoder or determined by the encoder / decoder by signaling the encoding method determined by the encoder as an index (dispEncType), and an index (profilesetIdx) for the profile information defined between the encoder / decoder may be signaled and transmitted according to the determined encoding method.

[0290] According to embodiments, the displacement vector video bitstream generated by encoding in the displacement vector encoder (11022) is transmitted to a multiplexer (not shown). According to embodiments, the encoding selection method of the displacement vector encoder (11022) may use a displacement vector encoder agreed upon at the encoder (i.e., the transmitting side) / decoder (i.e., the receiving side), or the type of displacement vector encoder selected by analyzing the characteristics of the displacement vector at the encoder on the transmitting side may be transmitted to the decoder on the receiving side.

[0291] According to the embodiments, the displacement vector decoder (or displacement vector restoration unit) (11023) can restore the displacement vector by performing the inverse process of displacement vector encoding on the displacement vector encoded in the displacement vector encoder (11022) or the displacement vector transformation coefficients. That is, the displacement vector decoder (11023) can perform displacement vector inverse packing depending on the method of encoding the displacement vector, for example, if it is encoded based on a video codec. In other words, it performs decoding with a 2D video decoder and inverse packing on a bitstream that is packed into a 2D image / video and encoded through a 2D video encoder (i.e., a video codec-based encoder). Additionally, the displacement vector decoder (11023) can additionally perform inverse quantization, inverse transformation, etc., depending on whether quantization and transformation processes are performed during the displacement vector encoding process. For example, inverse quantization is performed on the quantized transform coefficients that have undergone inverse packing, and an inverse transform is performed to calculate the restored displacement vector.

[0292] According to embodiments, a mesh inverse quantization unit (not shown) may inversely quantize vertex coordinates or texture coordinates of a restored base mesh as the reverse process of quantization and output them to a mesh subdivision unit (11024). More specifically, the mesh inverse quantization unit performs inverse quantization with inputs such as restored geometry information (x,y,z) and / or texture coordinates (u,v) and / or normal information (nx, ny, nz) of the restored base mesh. According to embodiments, inverse quantization for specific components may be omitted. Additionally, if the quantization process is omitted in the mesh quantization unit (11013), the inverse quantization process is also omitted in the mesh inverse quantization unit.

[0293] According to the embodiments, the displacement vector coordinate system inverse transformation unit (11025) can perform coordinate system transformation as the inverse process of the displacement vector coordinate system transformation unit (11021) when coordinate system transformation is applied in the displacement vector coordinate system transformation unit (11021).

[0294] According to embodiments, the mesh restoration unit (11028) can restore a mesh based on the restored displacement vector output from the displacement vector decoder (11023) or the displacement vector coordinate system inverse transformation unit (11025) and the restored base mesh (or inversely quantized restored base mesh) output from the mesh subdivision unit (11024) or the mesh inverse quantization unit. More specifically, the mesh restoration unit (11028) can generate a reconstructed deformed mesh by adding the displacement vector restored from the displacement vector decoder (11023) to the restored base mesh subdivided from the mesh subdivision unit (11024). According to embodiments, the mesh subdivision unit (11024) performs subdivision on the restored base mesh, which has been restored by inverse quantization in the mesh inverse quantization unit, to generate subdivided vertex position information, texture coordinates, and connection information. Then, the mesh restoration unit (11028) generates restored vertex position information by adding a restored displacement vector to the subdivided vertex position information. The mesh restored by the mesh restoration unit (11028) (or referred to as the restored mesh or restored deformed mesh) has restored vertices, connection information between vertices, texture coordinates, and connection information between texture coordinates. The restored mesh (or referred to as the restored mesh or restored deformed mesh) generated by the mesh restoration unit (11028) is provided to the texture map generation unit (11026).

[0295] According to the embodiments, the texture map generation unit (11026) can regenerate the texture map of the current mesh based on the texture map (or attribute map) of the original mesh and the mesh restored by the mesh restoration unit (11028). That is, the texture map generation unit (11026) can generate the texture map of the restored mesh through the relationship between the texture coordinates and connection information of the restored mesh and the original mesh and the texture map of the original mesh.

[0296] According to embodiments, the texture map generation unit (11026) can assign vertex-specific color information of the original mesh's texture map to the texture coordinates of the restored base mesh (or restored deformed mesh). According to embodiments, the texture map generation unit (11026) can generate a texture map (or texture map video) by grouping the texture maps regenerated for each frame into GoF units.

[0297] The texture map generated by the texture map generation unit (11026) can be encoded by the texture map encoder (11027). For example, the texture map encoder (11027) can encode the texture map using a 2D video codec-based encoder, a zero-run length encoder, an entropy coding-based arithmetic encoder, etc. That is, the texture map generated through the texture map generation unit (11026) is stacked in the order of the mesh frames to form a texture map video, and encoding is performed through the 2D video encoder. In addition, the texture map encoder (11027) can further perform color space conversion of the texture map. That is, if the color space of the texture map is RGB444, encoding can be performed after converting it to a color space such as YUV420 or YUV444. Then, the texture map substream (or texture map video bitstream) generated through texture map encoding is transmitted to a multiplexer (not shown).

[0298] According to the embodiments, the type of texture map encoder (11027) may include a video encoder (e.g., VVC, HEVC, etc.), an entropy coding-based encoder, etc. Furthermore, the method of selecting the texture map encoder (11027) may involve using a texture map encoder agreed upon at the encoder (i.e., the transmitting side) and the decoder (i.e., the receiving side), or transmitting the type of texture map encoder selected from the encoder at the transmitting side to the decoder at the receiving side.

[0299] In this disclosure, a base mesh bitstream output to a motion vector encoder (11015) or a static mesh encoder (11016) may be referred to as base mesh data or a base mesh sub-bitstream. Additionally, a displacement vector bitstream output from a displacement vector encoder (11022) may be referred to as displacement vector data, displacement data, displacement information, or a displacement vector sub-bitstream. In this case, the displacement data may be displacement data encoded based on a video codec or displacement data encoded based on an arithmetic codec. In this disclosure, displacement data encoded based on a video codec may be referred to as video codec-based displacement data, and displacement data encoded based on an arithmetic codec may be referred to as arithmetic codec-based displacement data. Additionally, the texture map bitstream output from the texture map encoder (11027) may be referred to as texture map data, attribute data, attribute information, attribute video data, texture map sub-bitstream, or attribute sub-bitstream.

[0300] According to embodiments, a multiplexer (not shown) may multiplex an input base mesh bitstream, a displacement vector bitstream, and a texture map bitstream into a single bitstream (i.e., a dynamic mesh bitstream) and then transmit it to a receiving device. The present disclosure may refer to the bitstream multiplexed and output from the multiplexer as a dynamic mesh bitstream or a V-DMC bitstream. In the present disclosure, the dynamic mesh bitstream or V-DMC bitstream may be configured in the multiplexer or in a separate block / module.

[0301] Additionally, the multiplexed bitstream may be encapsulated into one or more tracks of a file and transmitted to a receiving device. According to embodiments, the multiplexed bitstream or file from the multiplexer may be transmitted over a network or stored on a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, etc., and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.

[0302] As described above, the displacement vector encoder (11022) can encode the displacement vector through a 2D video codec such as H.264, HEVC, VVC, or an arithmetic coding method. According to the embodiments, when the displacement vector is encoded through a 2D video codec, the displacement vector encoding can be performed through the process of FIG. 15. Additionally, when the displacement vector is encoded through an arithmetic coding method, the displacement vector encoding can be performed through the process of FIG. 16.

[0303] FIG. 15 is a detailed block diagram showing an example of a displacement vector encoder according to embodiments. Each component of FIG. 15 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in FIG. 15 may be changed, some blocks may be omitted, and some blocks may be newly added. In particular, FIG. 23 is an example when displacement vector encoding is performed through a 2D video encoder.

[0304] More specifically, the displacement vector transformation unit can perform a lifting transformation, a wavelet transformation, etc., on the displacement vector of the (x,y,z) or (n,t,bt) coordinate system to transform the displacement vector of the (x,y,z) or (n,t,bt) coordinate system into a displacement vector transformation coefficient. In the (n,t,bt) coordinate system, n represents normal, t represents tangential, and bt represents bi-tangential. That is, the displacement vector of the (x,y,z) or (n,t,bt) coordinate system can be transformed through the displacement vector transformation unit. According to the embodiments, when a coordinate system transformation to the (n,t,bt) coordinate system is performed, a 1D scalar displacement vector of the normal (n) component is applied as an input to the displacement vector transformation unit, and transformation, quantization, and encoding can be performed on the displacement value of the normal component.

[0305] According to embodiments, the transformation of the displacement vector may be performed through a first transformation process and a second transformation process. According to embodiments, the first transformation process and / or the second transformation process may be omitted. A detailed description will be provided later.

[0306] If a lifting transformation is performed during the first transformation and / or second transformation, vertex R of the k-th refinement level k When performing prediction, use R as the predictor. t(t <k 또는 t<=k)의 세분화 정점 변위 벡터를 통해 k번째 세분화 레벨의 변위 벡터 예측을 수행할 수 있다. 실시예들에 따라 변위 벡터의 예측 수행 시 현재 정점보다 세분화 레벨이 낮은 정점 중 연결정보를 기반으로 가까운 n개의 점을 평균 또는 거리 기반 가중평균예측을 수행할 수 있다. 실시예들에 따라 메쉬 세분화 단계에서 현재 정점을 생성하기 위해 사용한 n개의 정점의 변위 벡터를 기반으로 예측을 수행할 수 있다. 그리고, 리프팅 변환이 수행되는 경우, 예측으로 생성되는 잔차 신호를 통해 예측에 사용된 정점의 변위 벡터를 업데이트 하는 과정이 수행될 수 있다.

[0307] According to the embodiments, the displacement vector transformation coefficient quantization unit may perform quantization on the displacement vector value, i.e., the displacement vector transformation coefficient, transformed through the displacement vector transformation unit. In this disclosure, the displacement vector transformation coefficient is used interchangeably with the displacement vector coefficient. According to the embodiments, the displacement vector transformation coefficient may be quantized using different quantization parameters for each axis and / or level of subdivision, and the quantization parameters or scaling parameters may be derived according to an agreement between the encoder and decoder or from the encoder and decoder. Alternatively, the quantization parameters or scaling parameters may be determined through signaling / parsing between the encoder and decoder.

[0308] According to the embodiments, the displacement vector transformation coefficient packing unit can perform packing of N quantized displacement vector transformation coefficients into a 2D image of size WxH. That is, if the displacement vector transformation coefficients are encoded using a video codec-based encoding method in a displacement vector encoder, the process of packing the displacement vector transformation coefficients into a 2D image in a frame is performed in the displacement vector transformation coefficient packing unit. In other words, the displacement vector transformation coefficient packing unit packs the displacement vector transformation coefficients into a 2D image, and the video encoding unit can encode the packed 2D images using a video compression codec and output them as a displacement vector bitstream (or referred to as a displacement bitstream). That is, the video encoding unit can encode the displacement vector transformation coefficients packed into an image through a 2D video codec and output a displacement vector bitstream.

[0309] As mentioned above, displacement vectors can be encoded based on arithmetic coding.

[0310] FIG. 16 is a detailed block diagram showing another example of a displacement vector encoder according to embodiments. Each component of FIG. 16 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in FIG. 16 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0311] The displacement vector encoder of FIG. 16 includes a displacement vector transformation unit, a displacement vector transformation coefficient quantization unit, a displacement vector prediction unit, and a displacement vector arithmetic encoding unit, and can output a displacement vector bitstream by performing displacement vector transformation, displacement vector transformation coefficient quantization, displacement vector prediction (e.g., displacement vector transformation level inter-frame prediction), and displacement vector arithmetic encoding on an input displacement vector. According to embodiments, the displacement vector prediction unit may generate residual transformation coefficients for the current transformation coefficients by using the transformation coefficients of a reference frame stored in a buffer as predictors. According to embodiments, the order of the displacement vector transformation coefficient quantization unit and the displacement vector prediction unit may be reversed, and the displacement vector prediction unit may be omitted.

[0312] Meanwhile, as described above, the transformation of the displacement vector in the displacement vector transformation unit of FIG. 15 can be performed through a first transformation process and a second transformation process. Depending on the embodiments, the first transformation process and / or the second transformation process may be omitted.

[0313] FIG. 17 is a block diagram showing an example of a displacement vector transformation unit according to embodiments. The displacement vector transformation unit of FIG. 17 may include a displacement vector first transformation unit and a displacement vector second transformation unit.

[0314] According to embodiments, the transformation of the displacement vector may be performed by applying the subdivided mesh and the displacement vector to a static mesh encoder (11016). In this case, connectivity information of the subdivided mesh is calculated, and the displacement vector can be transformed through a component prediction method based on the calculated connectivity information. As an example, connectivity information of the subdivided mesh may be calculated in a static mesh encoder such as DRACO, and the displacement vector can be transformed through a component prediction method.

[0315] According to embodiments, the displacement vector first transformation unit of FIG. 17 can perform a first transformation on the input displacement vector. According to embodiments, if the displacement vector first transformation unit is omitted, the displacement vector applied as input can be output as is. Also, wavelet transformation, lifting transformation, etc., can be applied as the first transformation. According to embodiments, if a lifting transformation is performed, the transformation of the displacement vector can be performed through the process of FIG. 18.

[0316] FIG. 18 is a diagram showing an example of the displacement vector first-order transformation process of the displacement vector first-order transformation unit according to the embodiments. In the displacement vector first-order transformation unit according to the embodiments, the lifting transformation can be performed n times as shown in FIG. 18, and the number of lifting transformations can be determined and signaled according to the mesh subdivision count parameter (subdivision_iteration_count).

[0317] In the present disclosure, the lifting transformation process can be performed at the level of mesh subdivision. Additionally, during the lifting transformation process, a lifting transformation prediction unit and a lifting transformation update unit may be performed.

[0318] According to the embodiments, the lifting transformation prediction unit is the vertex R of the k-th subdivision level. k For the displacement vector of, t(t <k 또는 t≤k)번째 세분화 레벨의 정점 R t A residual displacement vector can be generated by performing a prediction using the displacement vector of as a predictor.

[0319] According to the embodiments, when performing the prediction of a displacement vector, the lifting transformation prediction unit can perform the prediction by averaging or distance-based weighted averaging the displacement vectors of m nearby vertices based on connection information among vertices with a lower level of refinement than the current vertex.

[0320] According to the embodiments, when performing a displacement vector prediction, the prediction can be performed based on the displacement vectors of m vertices used to generate the current vertex according to the subdivision method in the mesh subdivision step.

[0321] According to the embodiments, when performing a displacement vector prediction, the prediction can be performed based on the normal vectors of the current vertex and the vertices adjacent to the current vertex.

[0322] According to the embodiments, the lifting transformation prediction unit can generate a residual signal (i.e., a residual displacement vector) through the difference between the displacement vector of the segmentation level and the predicted displacement vector.

[0323] In Fig. 18, the lifting transformation update unit performs the process of updating the displacement vector of the vertex used for prediction through the residual displacement vector (i.e., residual signal) generated by the lifting transformation prediction unit.

[0324] According to the embodiments, the weights used during the update process may be determined by an agreement between the encoder and decoder or determined and signaled by the encoder.

[0325] According to the embodiments, the weights used during the update process may be derived based on information such as the current level of refinement and the number of edges connected to a vertex.

[0326] According to the embodiments, the displacement vector second transformation unit may perform a second transformation on the displacement vector for which a first transformation has been performed in the displacement vector first transformation unit. At this time, the second transformation may be performed by applying various embodiments or may be omitted.

[0327] The following is an explanation of the quadratic transformation of displacement vectors.

[0328] According to the embodiments, whether to perform a quadratic transformation in the displacement vector quadratic transformation unit may be implicitly determined by the encoder / decoder agreement.

[0329] According to an embodiment, the present disclosure may determine whether to perform a secondary transformation and signal a signal indicating whether to perform a secondary transformation, for example, a secondary transformation enable flag (secondary_transform_enable_flag), to the signaling information and transmit it to the decoder of the receiving device.

[0330] For example, if the value of secondary_transform_enable_flag is 0, the secondary displacement vector transformation part can be omitted for the displacement vectors of all subdivision level vertices.

[0331] As another example, when the value of secondary_transform_enable_flag is 1, the displacement vector quadratic transformation can be performed only on the displacement vectors of the base mesh (segmentation level 0) vertices.

[0332] According to an embodiment, a secondary transformation of the displacement vector can be performed on the displacement vectors of all subdivision level vertices. At this time, the target for the secondary transformation in the secondary transformation unit may be implicitly determined according to an encoder / decoder agreement or determined by signaling a secondary transformation target flag (secondary_transform_target_flag), and the result of the transformation may be signaled in the signaling information and transmitted to the decoder of the receiving device.

[0333] According to the embodiments, a secondary transformation type can be determined, and the determined secondary transformation type can be signaled to the signaling information as a secondary transformation type parameter (secondary_transform_type) and transmitted to the decoder of the receiving device.

[0334] For example, if the value of secondary_transform_type is 0, the secondary transformation of the displacement vectors can be omitted for the displacement vectors of all subdivision level vertices.

[0335] As another example, when the value of secondary_transform_type is 1, the displacement vector quadratic transformation can be performed in the displacement vector quadratic transformation section only for the displacement vector of the base mesh (segmentation level 0) vertices.

[0336] As another example, when the value of secondary_transform_type is 2, a displacement vector quadratic transformation can be performed for the displacement vectors of all subdivision level vertices in the displacement vector quadratic transformation unit.

[0337] According to embodiments, parameters for a secondary transformation in a displacement vector secondary transformation unit may be signaled in units such as sequences, GOFs, frames, submeshes, mesh patches, etc. The parameters for the secondary transformation may include a secondary transformation execution parameter (e.g., secondary_transform_enable_flag), a secondary transformation target parameter (e.g., secondary_transform_target_flag), a secondary transformation type parameter (e.g., secondary_transform_type), a secondary transformation method parameter (e.g., secondary_transform_method), and an index parameter of a unit group determined at each level of subdivision (e.g., secondary_transform_group_index[k]). In the present disclosure, the parameters for the secondary transformation may be included in information related to the displacement vector transformation.

[0338] According to embodiments, the parameters for the second transformation may be transmitted in different units independently of the parameters for the first transformation. In this case, if the first transformation is omitted, the parameters for the first transformation may not be signaled, and only the parameters for the second transformation may be signaled. According to embodiments, the parameters for the second transformation may be transmitted in the same unit dependent on the parameters for the first transformation.

[0339] According to embodiments, the secondary transformation in the displacement vector secondary transformation unit may be performed on the displacement vectors of all subdivision level vertices. In this case, if the primary transformation of the displacement vector is performed as a lifting transformation, the secondary transformation in the displacement vector secondary transformation unit may be performed only on the displacement vectors of the base mesh (i.e., subdivision level 0) vertices, which may be implicitly determined by the encoder / decoder agreement or determined by signaling the secondary transformation target flag (secondary_transform_target_flag).

[0340] FIG. 19 is a diagram showing an example of a displacement vector quadratic transformation process of a displacement vector quadratic transformation unit according to embodiments. The displacement vector quadratic transformation unit according to embodiments may include a quadratic transformation group determining unit, a quadratic transformation method determining unit, and a quadratic transformation performing unit.

[0341] According to embodiments, if the displacement vector quadratic transformation is omitted, the first-order transformed displacement vector applied as input may be output as is. According to embodiments, subdivision levels may be grouped into groups, and the displacement vector quadratic transformation for the subdivision levels belonging to each group may be performed in the displacement vector quadratic transformation unit. In this case, according to embodiments, the same quadratic transformation method may be applied to the subdivision levels belonging to the group, or the quadratic transformation may be omitted for all subdivision levels belonging to the group. That is, depending on the characteristics of the data, the same quadratic transformation may be applied (or omitted) for each subdivision level of the displacement vector. For example, LoD0 and LoD1 may be grouped into Group 1 for lifting transformation as a quadratic transformation, while LoD2 and LoD3 may be grouped into Group 2 for the quadratic transformation to be omitted. Additionally, if the number of subdivision levels increases, the quadratic transformation method may not be signaled for all subdivision levels, but only a portion may be transmitted for each group.

[0342] According to the embodiments, when a displacement vector quadratic transformation is performed in the displacement vector quadratic transformation unit for the displacement vectors of all subdivision level vertices, the displacement vector quadratic transformation can be performed through the quadratic transformation method determination unit and the quadratic transformation execution unit in group units determined by the quadratic transformation group determination unit of FIG. 19.

[0343] According to the embodiments, when a displacement vector quadratic transformation is performed in the displacement vector quadratic transformation unit only for the displacement vector of the base mesh (segmentation level 0) vertex, a displacement vector quadratic transformation can be performed for the displacement vector of the base mesh vertex through the quadratic transformation method determination unit and the quadratic transformation execution unit. At this time, since the quadratic transformation is performed only on the base mesh, the lifting transformation cannot be performed, so the quadratic transformation method determination unit in FIG. 19 may be omitted, and the quadratic transformation method may be determined as a component prediction-based method according to the encoder / decoder agreement.

[0344] FIG. 20 is a diagram showing an example of a secondary transformation group determination method of a secondary transformation group determination unit according to embodiments.

[0345] According to embodiments, the secondary transformation group determination unit may determine each unit group to which a secondary transformation is performed on the primary transformed displacement vector. According to embodiments, each unit group may be composed of the displacement vector of a newly generated vertex at each level of subdivision. For example, when the number of mesh subdivisions performed in the mesh subdivision unit is 3, and the vertex of the base mesh is defined as R0, the vertex newly generated by performing subdivision 1 time is defined as R1, the vertex newly generated by performing subdivision 2 times is defined as R2, and the vertex newly generated by performing subdivision 3 times is defined as R3, the displacement vector of R0 may be composed of Group 0 (i.e., Group 0), the displacement vector of R1 may be composed of Group 1 (i.e., Group 1), the displacement vector of R2 may be composed of Group 2 (i.e., Group 2), and the displacement vector of R3 may be composed of Group 3 (i.e., Group 3), thereby forming a total of 4 unit groups. According to embodiments, each unit group may consist of displacement vectors of newly generated vertices at multiple subdivision levels.

[0346] As another example, if the number of mesh subdivisions performed in the mesh subdivision section is 3, as shown in FIG. 20, the displacement vector of R0 can be configured as group 0 (i.e., group 0), and the displacement vectors of R1, R2, and R3 can be configured as group 1 (i.e., group 1), thereby forming a total of 2 unit groups.

[0347] According to the embodiments, the unit group may be implicitly determined according to the encoder / decoder agreement.

[0348] According to the embodiments, the index parameter (secondary_transform_group_index[k]) of a unit group determined at each subdivision level can be signaled to the signaling information and transmitted to the decoder of the receiving device. Here, k represents each subdivision level.

[0349] For example, if the number of mesh subdivisions performed in the mesh subdivision section is 3, and the displacement vector of R0 is group 0, and the displacement vectors of R1, R2, and R3 are group 1, the index parameter of the unit group can be signaled with the following values.

[0350] That is, secondary_transform_group_index[0]=0

[0351] secondary_transform_group_index[1]=1

[0352] secondary_transform_group_index[2]=1

[0353] secondary_transform_group_index[3]=1

[0354] FIG. 21 is a diagram showing an example of secondary_transform_method parameter signaling according to embodiments.

[0355] For example, if the value of secondary_transform_method is 0, secondary transformation can be omitted; if 1, a component prediction-based method; if 2, a lifting wavelet transform-based method; and if 3, a discrete cosine transform-based method.

[0356] In FIG. 19, the secondary transformation method determination unit can determine whether to perform a secondary transformation for each group and the secondary transformation method. The secondary transformation method (e.g., secondary_transform_method) may include a component prediction-based method, a lifting wavelet-based method, a Discrete Cosine Transform-based method, etc.

[0357] According to the embodiments, whether to perform a secondary transformation for each unit group and the secondary transformation method may be determined according to the encoder / decoder agreement. According to the embodiments, whether to perform a secondary transformation for each unit group and the secondary transformation method may determine the secondary transformation method parameter (secondary_transform_method) in units such as sequence, GOF, frame, submesh, etc., and transmit it to the decoder of the receiving device by signaling the signaling information as shown in FIG. 21.

[0358] In FIG. 19, the second transformation execution unit generates a second transformed displacement vector by performing a second transformation method determined by the second transformation method determination unit on a group unit determined by the second transformation group determination unit. At this time, the second transformation method may include a component prediction-based method, a lifting wavelet transform-based method, a discrete cosine transform-based method, etc. Additionally, the second transformation for the current displacement vector group may be omitted.

[0359] According to the embodiments, if it is determined that a second transformation for a current displacement vector group is omitted, the second transformation for the group (i.e., the second transformation performed in the second transformation execution unit) may be omitted, and the first transformed displacement vector of the group may be output as is to the displacement vector transformation coefficient quantization unit.

[0360] According to embodiments, when a second transformation is performed by a discrete cosine transform-based method, the present disclosure may divide a current displacement vector group (e.g., group 0) into m subgroups as shown in FIG. 22, and apply a 1D discrete cosine transform to each subgroup to generate a second transformed displacement vector.

[0361] FIG. 22 is a diagram showing an example of a discrete cosine transform-based quadratic transformation method according to embodiments. In FIG. 22, the 1D discrete cosine transform may be performed using a formula such as DCT-II or through a value obtained by approximating and integerizing a formula such as DCT-II. According to embodiments, the quadratic transformation execution unit may sort the vertices of the current group in a manner such as a Z-order curve or a Hilbert curve, and perform the discrete cosine transform-based quadratic transformation method using a displacement vector of the sorted vertex order.

[0362] According to embodiments, when a secondary transformation is performed by a lifting wavelet transform-based method, the present disclosure can generate a secondary transformed displacement vector through a process such as that shown in FIG. 23.

[0363] FIG. 23 is a diagram showing an example of a lifting wavelet transform-based quadratic transformation method according to embodiments. According to embodiments, the quadratic transformation performing unit may align the vertices of the current group in a manner such as a Z-order curve or a Hilbert curve, and perform the lifting wavelet transform-based quadratic transformation method with a displacement vector of the aligned vertex order.

[0364] Referring to FIG. 23, the dividing unit divides the displacement vector of the current group into a plurality of subgroups, for example, two subgroups (a first subgroup and a second subgroup). According to embodiments, the dividing unit may divide the current group into displacement vectors having even and odd indices based on the sorted order of the vertices.

[0365] The prediction unit of FIG. 23 generates a residual displacement vector of the second subgroup by predicting the displacement vector of the second subgroup through the displacement vector of the first subgroup. That is, the residual displacement vector of the second subgroup corresponds to the difference between the predicted displacement vector of the second subgroup and the displacement vector of the divided second subgroup. According to an embodiment, the prediction unit may perform the prediction through the average or weighted sum of the displacement vectors of adjacent vertices based on the vertex alignment order.

[0366] The update unit of FIG. 23 compensates the displacement vector of the first subgroup used for prediction through the residual displacement vector of the second subgroup. According to embodiments, the update weights may be determined by an agreement between the encoder and the decoder, or determined and signaled by the encoder and transmitted to the decoder of the receiving device.

[0367] The merging unit of FIG. 23 merges the displacement vectors of the two divided subgroups to generate a quadratic transformed displacement vector of the current group. According to embodiments, the merging unit may merge the first subgroup compensation displacement vector and the second subgroup residual displacement vector to generate a quadratic transformed displacement vector of the current group (i.e., a quadratic transformed displacement vector group). According to embodiments, the quadratic transformed displacement vector may be merged so that it has the same vertex alignment order as the input displacement vector. As an example, each displacement vector of the two subgroups may be merged by crossing.

[0368] According to an embodiment, the secondary transformed displacement vector can be merged to have a different vertex alignment order than the input displacement vector. As an example, the displacement vector of the first subgroup can be merged so that it follows the displacement vector of the second subgroup.

[0369] According to embodiments, when a secondary transformation is performed using a component prediction-based method, the present disclosure can generate a secondary transformed displacement vector by performing a prediction based on connectivity information of the current group vertex.

[0370] According to embodiments, if the current group consists of vertices of a base mesh, the connectivity information may be encoded through a static mesh encoder (e.g., 413 in FIG. 7, 814 in FIG. 12, or 11016 in FIG. 14) or a motion vector encoder (e.g., 414 in FIG. 7, 813 in FIG. 12, or 11015 in FIG. 14). In this case, the connectivity information of the base mesh vertices restored through a base mesh restoration unit (e.g., 424 in FIG. 7, 819 in FIG. 12, or 11017 in FIG. 14) may be applied to and used by the static mesh encoder or the motion vector encoder.

[0371] According to the embodiments, the connectivity information may be calculated by applying a mesh connectivity coding method, such as DRACO or Edgebreaker, to the vertices of the current group.

[0372] According to the embodiments, a quadratic transformed displacement vector can be generated as shown in Equation 1 below by performing a delta encoding method based on the connectivity information of the current group vertex.

[0373] [Mathematical Formula 1]

[0374] a n = d n - d' n-1

[0375] In the above mathematical formula 1, d n is the displacement vector of the nth vertex of the current group, d' n-1 is the restored (inverse quantization after quantization) displacement vector of the (n-1)th vertex of the current group, a n represents the nth displacement vector transformed by the second degree.

[0376] According to embodiments, the present disclosure can generate a quadratic transformed displacement vector as shown in Equation 2 below by performing a Parallelogram Prediction method based on connectivity information of the current group vertices.

[0377] [Mathematical Formula 2]

[0378]

[0379] In the above mathematical formula 2, pred n represents the predicted displacement vector for the displacement vector of the nth vertex of the current group, and R'0, L'0, and O'0 represent the restored (quantized followed by in-quantized) displacement vectors of the vertices of adjacent parallelograms used for prediction, as shown in FIG. 24 (a). And, in the above Equation 2, d n is the displacement vector of the nth vertex of the current group, a n represents the nth displacement vector that has been transformed by the second degree. That is, FIG. 24 (a) is a diagram showing an example of a parallelogram prediction method according to embodiments.

[0380] According to embodiments, the present disclosure can generate a quadratic transformed displacement vector as shown in Equation 3 below by performing a multi-parallelogram prediction method based on connectivity information of the current group vertices.

[0381] [Mathematical Formula 3]

[0382]

[0383] In the above mathematical formula 3, pred n is the predicted displacement vector for the displacement vector of the nth vertex of the current group, k is the number of neighboring parallelograms used for prediction as in Fig. 24 (b), and R' i , L' i , O' i represents the restored (quantized followed by inverse quantization) displacement vector of each parallelogram vertex used for prediction, as shown in Fig. 24(b). And, in the above Equation 3, d n is the displacement vector of the nth vertex of the current group, a n represents the nth displacement vector that has been transformed by the second degree. That is, FIG. 24 (b) is a diagram showing an example of a multiple parallelogram prediction method according to embodiments.

[0384] According to the embodiments, the quadratic transformation unit can generate a quadratic transformed displacement vector as shown in Equation 4 below by performing a trapezoid prediction method based on connectivity information of the current group vertex.

[0385] [Mathematical Formula 4]

[0386]

[0387] In the above mathematical formula 4, pred n represents the predicted displacement vector for the displacement vector of the nth vertex of the current group, and R'0, L'0, O'0, and R'1 represent the restored (quantized followed by in-quantized) displacement vectors of neighboring vertices used for prediction as shown in FIG. 25 (a). Furthermore, in the above Equation 4, d n is the displacement vector of the nth vertex of the current group, a n represents the nth displacement vector that has been transformed by the second degree. That is, FIG. 25 (a) is a diagram showing an example of a trapezoidal prediction method according to embodiments.

[0388] According to the embodiments, the quadratic transformation unit can generate a quadratic transformed displacement vector as shown in Equation 5 below by performing a Multi-trapezoid Prediction method based on connectivity information of the current group vertex.

[0389] [Mathematical Formula 5]

[0390]

[0391] In the above mathematical formula 5, pred n is the predicted displacement vector for the displacement vector of the nth vertex of the current group, k is the number of neighboring face structures used for prediction as in Fig. 25 (b), and R' i , L' i , O' i , R' i+1represents the restored (quantized followed by inverse quantization) displacement vector of the vertices of each face structure used for prediction, as shown in Fig. 25(b). And, in the above Equation 5, d n is the displacement vector of the nth vertex of the current group, a n represents the nth displacement vector that has been transformed by the second degree. That is, FIG. 25 (b) is a diagram showing an example of a multi-trapezoidal prediction method according to embodiments.

[0392] FIG. 26 is a drawing showing another example of a receiving device according to embodiments. In the present disclosure, the receiving device of FIG. 26 may be referred to as a mesh data receiving device or a decoder or a decoder of a receiving device or a V-Mesh decoder or a dynamic mesh decoder.

[0393] FIG. 26 corresponds to the receiving device (110) or mesh video decoder (113) of FIG. 1, the decoder of FIG. 11, the receiving device of FIG. 13, and / or a corresponding receiving decoding device. Therefore, any parts not described in FIG. 26 are to be understood by referring to the descriptions of the receiving device (110) or mesh video decoder (113) of FIG. 1, the decoder of FIG. 11, and the receiving device of FIG. 13. The elements of the receiving device illustrated in FIG. 26 may be implemented in hardware, software, a processor connected to memory, and / or a combination thereof. That is, the elements of the receiving device of FIG. 26 may be implemented in hardware, software, firmware, or a combination thereof, including one or more processors or integrated circuits configured to communicate with one or more memories, although not illustrated in the drawings. One or more processors may perform at least one of the operations and / or functions of the elements of the receiving device of FIG. 26 described above. In addition, one or more processors may operate or execute a set of software programs and / or instructions for performing operations and / or functions of the elements of the receiving device of FIG. 26. The execution order of each block in FIG. 26 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0394] FIG. 26 may largely include a base mesh decoding unit, a displacement information decoding unit, and a texture map decoding unit. According to embodiments, the base mesh decoding unit may include a switching unit (15011), a motion vector decoder (15012), a static mesh decoder (15013), a base mesh restoration unit (15014), a mesh subdivision unit (15015), and a mesh restoration unit (15016). According to embodiments, the displacement information decoding unit may include a displacement vector decoder (15017) and a displacement vector coordinate system inverse transformation unit (15020).

[0395] According to embodiments, a bitstream of mesh data received by a receiver (not shown) may be demultiplexed into a base mesh bitstream, a displacement vector bitstream, and / or a texture map bitstream in a demultiplexer (not shown) after file / segment decapsulation. If the bitstream of mesh data is not encapsulated in the form of a file at the transmitting device, the decapsulation process at the receiving device is omitted. If the current mesh is inter-encoded, the base mesh bitstream may be a motion vector bitstream.

[0396] According to embodiments, the base mesh bitstream is provided to the motion vector decoder (15012) or to the static mesh decoder (15013) through the switching unit (15011).

[0397] For example, if the current mesh is inter-encoded, the base mesh bitstream, i.e., the motion vector bitstream, is received, demultiplexed, and then output to the motion vector decoder (15012) through the switching unit (15011). As another example, if the current mesh is intra-encoded, the base mesh bitstream is received, demultiplexed, and then output to the static mesh decoder (15013) through the switching unit (15011). Here, the motion vector decoder (15012) may be referred to as a motion decoder.

[0398] According to the embodiments, the motion vector decoder (15012) can perform decoding on the motion vector bitstream on a vertex-by-vertex or subgroup-by-subgroup basis. That is, the motion vector decoder (15012) can decode the current motion vector.

[0399] According to embodiments, the motion vector decoder (15012) can perform a connection information-based prediction using a previously decoded motion vector as a predictor and restore the final (i.e., current) motion vector by adding a difference motion vector (i.e., residual motion vector) decoded from the basemesh bitstream to the predicted motion vector. That is, the motion vector decoder (15012) can decode the difference motion vector (or residual motion vector) at the vertex or subgroup (or subblock) level through the motion vector bitstream and decode the motion vector by performing a connection information-based prediction using a previously decoded motion vector as a predictor and adding it to the residual motion vector. According to embodiments, decoding of the motion vector at the vertex level or subgroup level may be performed or omitted.

[0400] According to the embodiments, the static mesh decoder (15013) can decode the base mesh bitstream to restore the connection information, vertex geometry information, texture coordinates (i.e., attribute geometry information), normal information, etc. of the current base mesh.

[0401] According to the embodiments, the base mesh restoration unit (15014) can restore the current base mesh based on the decoded motion vector or the decoded base mesh. For example, if the current mesh has inter-frame encoding applied, the base mesh restoration unit (15014) can generate the restored base mesh (i.e., the current base mesh) by adding the motion vector decoded through the motion vector decoder (15012) to the reference restored base mesh, decoding the connection information, vertex geometry information, vertex texture coordinates, normal information, etc. of the current base mesh, and performing inverse quantization. As another example, if the current mesh has intra-frame encoding applied, the base mesh restoration unit (15014) can generate the restored base mesh (i.e., the current base mesh) by performing inverse quantization on the connection information, vertex geometry information, vertex texture coordinates, normal information, etc. of the current base mesh decoded (or restored) through the static mesh decoder (15012). According to the embodiments, if quantization for a specific component is not performed, inverse quantization for that component may be omitted.

[0402] According to the embodiments, the mesh subdivision unit (15015) can generate additional vertices by performing subdivision on the base mesh restored by the base mesh restoration unit (15014). The present disclosure can implicitly derive and generate geometry information connection information, texture coordinate connection information, and texture coordinates according to the subdivision method.

[0403] According to the embodiments, the mesh subdivision unit (15015) can perform subdivision through methods such as mid-edge, Loop, Catmul & Clark.

[0404] According to embodiments, mesh subdivision in the mesh subdivision unit (15015) may be performed n times by user parameters or by a promise of the encoder / decoder. According to embodiments, the vertices of the base mesh are the vertices of refinement level 0 (R0), the vertices newly generated by performing subdivision once are the vertices of refinement level 1 (R1), ..., the vertices newly generated by performing subdivision n times are the vertices of refinement level n (R n It can be defined by the vertices of ). Also, refinement level 0 is the 0th refinement level (R0), refinement level 1 is the 1st refinement level (R1), ..., refinement level n is the nth refinement level (R n It can be called ).

[0405] According to the embodiments, LoD0, LoD1, ..., LoDn can be defined as follows.

[0406] LoD0 = R0

[0407] LoD1 = R0 + R1 = LoD0 + R1

[0408] …

[0409] LoDn = R0 + R1 + ..., + R n = LoDn-1 + R n == R0∪ R1∪, ..., ∪ R n

[0410] That is, the base mesh contains vertices for LoD0. And, the first subdivision mesh generated by subdividing the base mesh once contains vertices for LoD1. That is, LoD1 includes the vertices of the base mesh (i.e., R0 vertices) and the vertices of refinement level 1 newly generated by subdividing the base mesh (i.e., R1 vertices). Additionally, the second subdivision mesh generated by subdividing the first subdivision mesh again contains vertices for LoD2. That is, LoD2 includes the vertices of the base mesh (i.e., R0 vertices), the vertices of refinement level 1 newly generated by subdividing the base mesh (i.e., R1 vertices), and the vertices of refinement level 2 newly generated by subdividing the first subdivision mesh (i.e., R2 vertices).

[0411] According to embodiments, the displacement vector decoder (15017) may perform decoding based on a 2D video codec such as H.264, HEVC, VVC, etc., as a video bitstream, or perform zero-run-length decoding or arithmetic decoding for the demultiplexed displacement vector bitstream. In this case, the displacement vector transformation factor decoding method in the present disclosure may vary, and according to embodiments, one of the following methods may be determined through the dispEncType flag or index: {video codec-based decoding method, zero-run-length decoding method}, {video codec-based decoding method, zero-run-length decoding method}, {video codec-based decoding method, arithmetic decoding method}, video codec-based decoding method, zero-run-length decoding method, or arithmetic decoding method. That is, the decoding method of the displacement vector can be determined by an agreement between the encoder and the decoder, or by parsing the decoding method determined by the encoder into an index (dispEncType). Depending on the determined decoding method, an index (profilesetIdx) for profile information defined between the encoder and the decoder can be signaled from the encoder and parsed. According to the embodiments, the decoding method of the displacement vector transformation coefficient can be determined according to the profile defined in the encoder and the decoder, and the decoder can determine the displacement vector decoding method according to the profileToolsetIdx by receiving an index (profileToolsetIdx) representing the profile information. Here, as one embodiment, the decoder is a displacement vector decoder (15017).

[0412] According to the embodiments, the displacement vector decoder (15017) can restore the displacement vector by decoding the displacement vector as the reverse process of the displacement vector encoding method of the transmitting side.

[0413] According to the embodiments, the displacement vector coordinate system inverse transformation unit (15020) can perform the process of inversely transforming the displacement vector decoded by the displacement vector decoder (15017) into a Cartesian (or canonical) coordinate system (x,y,z) if the value is in a local coordinate system (n,t,bt). The output of the displacement vector coordinate system inverse transformation unit (15020) is provided to the mesh restoration unit (15016). That is, the displacement vector coordinate system inverse transformation unit (15020) can inversely transform the decoded displacement vector from a (normal, tangential, bi-tangential) coordinate system to a (x,y,z) spatial coordinate system based on the normal vector of each vertex. In other words, the encoder of the transmitting device can perform a transformation of the vertex displacement vector calculated in the (x,y,z) space into a (normal, tangential, bi-tangential) coordinate system (or local coordinate system) based on the normal vector of each vertex. At this time, the normal vector can be calculated for each subdivided vertex based on the geometry information and connectivity information of the surrounding vertices.

[0414] FIG. 27 is a detailed block diagram showing an example of a displacement vector coordinate system inverse transformation unit (15020) according to embodiments.

[0415] FIG. 28 is a detailed block diagram showing another example of a displacement vector coordinate system inverse transformation unit (15020) according to embodiments.

[0416] More specifically, the displacement vector coordinate system inverse transformation unit (15020) can parse the coordinate system transformation flag (applyLocalCoord) included in the signaling information in the order of sequence or GoF (group of frame) or frame or sub-mesh, and if the value is 1, perform an inverse transformation of the inversely quantized (or inversely transformed) restored displacement vector from the local coordinate system (n,t,b) to the canonical coordinate system (x,y,z).

[0417] In addition, a normal vector per vertex is calculated based on the vertex position information of the reconstructed base mesh, and a normal value for a newly generated vertex can be assigned by interpolating the vertex normal vector of the reconstructed base mesh calculated for the additional vertices generated through the subdivision process (see FIG. 27). That is, the normal vector of each vertex can be composed of a normal vector calculated based on the vertex position information of the reconstructed base mesh and a normal vector calculated by interpolating the normal vector of the reconstructed base mesh. In this case, for interpolation, the normal information of the base mesh used for subdivision can be averaged or distance-based weighted summed to perform interpolation. Alternatively, the normal information of the base mesh can be used as is for the subdivided vertices of the same plane.

[0418] According to the embodiments, after performing subdivision on the restored base mesh, normal vectors can be calculated for the vertices generated through the mesh subdivision unit (15015) and the vertices of the base mesh (see FIG. 28). That is, the normal vector of each vertex can be composed of a normal vector calculated based on the vertex position information of the subdivided mesh.

[0419] According to the embodiments, whether to inversely transform the displacement vector coordinate system may be determined by an agreement between the encoder and decoder, or by receiving a coordinate system transformation flag (e.g., applyLocalCoord) from the transmitting side in units such as sequence, GOF, frame, submesh, etc.

[0420] According to the embodiments, the mesh restoration unit (15016) restores the mesh based on the restored mesh subdivided in the mesh subdivision unit (15015) and the restored displacement vector output from the displacement vector coordinate system inverse transformation unit (15020). That is, the mesh restoration unit (15016) can generate the restored mesh by adding the restored displacement vector to the vertex-specific position information of the subdivided restored mesh.

[0421] According to the embodiments, the received and demultiplexed texture map bitstream is input to a texture map decoder (15021). According to the embodiments, the texture map decoder (15021) can restore a texture map from the texture map bitstream. At this time, the type of texture map decoder (15012) may be a video decoder, a zero-run length decoder, an arithmetic decoder, etc. According to the embodiments, the texture map decoder (15021) can perform color space conversion of the texture map. That is, after texture map decoding, color space conversion can be performed to RGB444, YUV420, YUV444, etc.

[0422] In this way, when the base mesh bitstream, displacement vector bitstream, and texture map bitstream encoded by the dynamic mesh encoder of the transmitting device are transmitted, the decoder of the receiving device restores the mesh by decoding each bitstream. First, the base mesh is decoded by the motion vector or static mesh decoder depending on whether it is an inter-frame or intra-frame, and the geometry information is restored along with the displacement vector information decoded through subdivision.

[0423] As described above, the displacement vector decoder (15017) may decode the displacement vector through a 2D video codec and may decode the displacement vector through an arithmetic codec.

[0424] FIG. 29 is a detailed block diagram showing an example of a displacement vector decoder according to embodiments. The displacement vector decoder according to embodiments may include a displacement vector video decoding unit, a displacement vector transformation coefficient inverse packing unit, a displacement vector transformation coefficient inverse quantization unit, and a displacement vector inverse transformation unit. Each component of FIG. 29 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in FIG. 29 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0425] According to the embodiments, the displacement vector video decoding unit performs displacement vector decoding through a 2D video decoder such as H.264, HEVC, or VVC. That is, the displacement vector video decoding unit can decode a transformation factor image from a displacement vector bitstream through a 2D video codec. In the present disclosure, the video codec-based displacement vector bitstream input to the displacement vector video decoding unit may be used interchangeably with the same meaning as a displacement vector video bitstream, a displacement information video bitstream, or a displacement information bitstream.

[0426] According to the embodiments, the displacement vector transformation coefficient decoding method in the displacement vector video decoding unit may be determined as a specific decoding method by an agreement between the encoder and the decoder, or it may be determined by receiving the encoding method determined by the encoder as a flag (dispEncType) or index.

[0427] The above displacement vector transformation coefficient inverse packing unit can perform inverse packing from the transformation coefficient image restored by the displacement vector video decoding unit according to the scanning order defined by the encoder / decoder agreement, or the scanning order parsed in units such as sequences or frames. That is, the displacement vector transformation coefficients decoded based on a video codec by the displacement vector video decoding unit undergo an inverse packing process in the displacement vector transformation coefficient inverse packing unit.

[0428] The above displacement vector transformation coefficient quantization unit can perform inverse quantization for the displacement vector transformation coefficient.

[0429] The above displacement vector inverse transformation unit will be explained in detail later.

[0430] FIG. 30 is a detailed block diagram showing another example of a displacement vector decoder according to embodiments. The displacement vector decoder according to embodiments may include a displacement vector video arithmetic decoding unit, a displacement vector prediction and transformation unit, a displacement vector transformation coefficient inverse quantization unit, and a displacement vector inverse transformation unit. Each component of FIG. 30 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in FIG. 30 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0431] According to the embodiments, the displacement vector video arithmetic decoding unit performs displacement vector decoding through an arithmetic decoder.

[0432] The above displacement vector prediction and restoration unit can restore displacement vector transformation coefficients by adding the transformation coefficients of the reference frame stored in the buffer and the residual transformation coefficients restored by the above displacement vector video arithmetic decoding unit. The above displacement vector transformation coefficient quantization unit can perform inverse quantization on the displacement vector transformation coefficients.

[0433] According to the embodiments, the order of the displacement vector prediction and restoration unit and the displacement vector transformation coefficient inverse quantization unit may be changed, and the displacement vector prediction and restoration unit may be omitted.

[0434] In FIG. 29 or FIG. 30, the displacement vector inverse transform unit outputs displacement vectors by performing the inverse transform of the transform performed by the encoder of the transmitting device on the inverse quantized displacement vector transform coefficients. According to embodiments, a lifting inverse transform, a wavelet inverse transform, etc., may be performed. If a lifting inverse transform is performed in the displacement vector inverse transform unit, a process of updating the displacement vector of the vertex used for prediction in the encoder through the parsed residual signal may be performed. That is, the displacement vector inverse transform unit performs the inverse transform of the inverse quantized displacement vector coefficients. At this time, the inverse transform may be a lifting inverse transform, a wavelet inverse transform, etc., depending on the embodiments.

[0435] According to embodiments, the displacement vector inverse transform unit may perform a second inverse transform process and a first inverse transform process. According to embodiments, if the first transform and / or second transform process in the encoder is omitted, the first inverse transform process and / or the second inverse transform process may be omitted.

[0436] The above second inverse transformation process corresponds to the second transformation process of the encoder of the transmitting device, and the first inverse transformation process corresponds to the first transformation process of the encoder of the transmitting device. This is an example, and the inverse transformation process of the decoder of the receiving device corresponding to the second transformation process of the encoder of the transmitting device may be called the first inverse transformation process, and the inverse transformation process of the decoder of the receiving device corresponding to the first transformation process of the encoder of the transmitting device may be called the second inverse transformation process.

[0437] FIG. 31 is a block diagram showing an example of a displacement vector inverse transformation unit according to embodiments. The displacement vector inverse transformation unit according to embodiments may include a displacement vector second-order inverse transformation unit and a displacement vector first-order inverse transformation unit.

[0438] That is, the displacement vector inverse transformation unit of the present disclosure can perform the inverse transformation process of the transformation performed in the encoder on the displacement vector.

[0439] According to the embodiments, when the coordinate system of the displacement vector in the encoder is transformed into a (normal, tangential, bi-tangential) coordinate system and the transformation is performed on the 1D scalar displacement value of the normal component, the displacement vector inverse transformation unit can perform the inverse transformation on the displacement value of the normal component.

[0440] According to the embodiments, the inverse transformation of the displacement vector can be performed through a second-order inverse transformation process and a first-order inverse transformation process as shown in FIG. 31.

[0441] According to the embodiments, if the first conversion and / or second conversion process in the encoder is omitted, the first inverse conversion process and / or the second inverse conversion process may be omitted.

[0442] According to embodiments, a subdivided mesh and a displacement vector are applied to a static mesh decoder, and an inverse transformation of the displacement vector can be performed in the static mesh decoder. That is, connectivity information of the subdivided mesh is calculated in the static mesh decoder, and the displacement vector can be inversely transformed through a component prediction method based on the calculated connectivity information. As an example, connectivity information of the subdivided mesh is calculated in a static mesh decoder such as DRACO, and the displacement vector can be inversely transformed through a component prediction method.

[0443] According to the embodiments, the second inverse transformation in the displacement vector second inverse transformation unit may be performed by applying various embodiments or omitted.

[0444] The following is an explanation of the inverse displacement vector transformation.

[0445] According to embodiments, the displacement vector second inverse transform unit can perform a second inverse transform on the displacement vector decoded from the bitstream.

[0446] According to embodiments, whether to perform a secondary inverse transform may be implicitly determined by an agreement between the encoder and the decoder. According to embodiments, whether to perform a secondary inverse transform may be determined by parsing a secondary transform execution flag (secondary_transform_enable_flag) received and included in the signaling information.

[0447] For example, if the value of secondary_transform_enable_flag is 0, the inverse displacement vector transformation can be omitted for the displacement vectors of all subdivision level vertices.

[0448] As another example, when the value of secondary_transform_enable_flag is 1, the displacement vector inverse transformation can be performed in the displacement vector inverse transformation section only for the displacement vector of the base mesh (segmentation level 0) vertices.

[0449] According to embodiments, a secondary inverse transformation of the displacement vector for the displacement vector of all subdivision level vertices may be performed in a secondary inverse transformation unit. At this time, the target for performing the secondary inverse transformation of the displacement vector may be implicitly determined according to an agreement between the encoder and decoder, or determined by parsing a secondary transformation target flag (secondary_transform_target_flag) received and included in the signaling information.

[0450] According to the embodiments, the secondary transformation type parameter (secondary_transform_type) received and included in the signaling information can be parsed to determine the secondary transformation type.

[0451] For example, if the value of secondary_transform_type is 0, the inverse displacement vector transformation can be omitted for the displacement vectors of all subdivision level vertices.

[0452] As another example, when the value of secondary_transform_type is 1, the displacement vector inverse transformation can be performed in the displacement vector inverse transformation section only for the displacement vector of the base mesh (segmentation level 0) vertices.

[0453] As another example, when the value of secondary_transform_type is 2, the inverse displacement vector transformation can be performed in the inverse displacement vector transformation unit for the displacement vectors of all subdivision level vertices.

[0454] According to embodiments, parameters for secondary transformation may be received by being included in signaling information in units such as sequences, GOFs, frames, submeshes, mesh patches, etc., and may be determined by parsing from the received signaling information. The parameters for secondary transformation may include at least one of a secondary transformation execution parameter (e.g., secondary_transform_enable_flag), a secondary transformation target parameter (e.g., secondary_transform_target_flag), a secondary transformation type parameter (e.g., secondary_transform_type), a secondary transformation method parameter (e.g., secondary_transform_method), and an index parameter of a unit group determined at each level of subdivision (e.g., secondary_transform_group_index[k]). According to embodiments, the parameters for secondary transformation may be transmitted in different units independently of the parameters for primary transformation. In this case, if the first transformation is omitted in the encoder, the parameters for the first transformation are not parsed, and only the parameters for the second transformation are parsed to determine whether to perform the second transformation and how to perform the second transformation.

[0455] According to the embodiments, the parameters for the second transformation may be dependent on the parameters for the first transformation and may be transmitted in the same unit.

[0456] According to embodiments, the displacement vector secondary inverse transform may be performed on the displacement vectors of all subdivision level vertices. In this case, if the displacement vector primary transform is performed as a lifting inverse transform, the displacement vector secondary inverse transform may be performed only on the displacement vectors of the base mesh vertices, which may be implicitly determined by the encoder / decoder agreement or determined by parsing the secondary transform target flag (secondary_transform_target_flag) received and included in the signaling information.

[0457] According to the embodiments, if the displacement vector second inverse transformation is omitted, the displacement vector decoded from the bitstream can be output as is to the displacement vector coordinate system inverse transformation unit.

[0458] According to the embodiments, when the displacement vector inverse transformation for the displacement vector of all subdivision level vertices is performed in the displacement vector inverse transformation unit, as shown in FIG. 32, the displacement vector inverse transformation can be performed through the inverse transformation method determination unit and the inverse transformation execution unit in group units determined by the inverse transformation group determination unit.

[0459] FIG. 32 is a diagram showing an example of a displacement vector second inverse transform unit according to embodiments. According to embodiments, when a displacement vector second inverse transform is performed only on the displacement vector of a base mesh (segmentation level 0) vertex, the displacement vector second inverse transform can be performed on the displacement vector of the base mesh vertex through a second inverse transform method determination unit and a second inverse transform execution unit. In this case, since the second inverse transform is performed only on the base mesh, the second inverse transform method determination unit (i.e., second inverse transform method determination method) can be omitted because the lifting transform cannot be performed due to this, and the second inverse transform method can be determined as a component prediction-based method according to the encoder / decoder agreement.

[0460] In FIG. 32, the second inverse transform group determination unit can determine each unit group on which a second inverse transform is to be performed on the displacement vector decoded from the bitstream. According to embodiments, each unit group may consist of the displacement vector of a newly generated vertex at each level of subdivision.

[0461] For example, when the number of mesh subdivisions performed in the mesh subdivision unit is 3, and the vertices of the base mesh are defined as R0, the vertices newly created by performing subdivision 1 time are defined as R1, the vertices newly created by performing subdivision 2 times are defined as R2, and the vertices newly created by performing subdivision 3 times are defined as R3, the second inverse transformation group determination unit can form a total of 4 unit groups by configuring the displacement vector of R0 as group 0, the displacement vector of R1 as group 1, the displacement vector of R2 as group 2, and the displacement vector of R3 as group 3.

[0462] According to embodiments, each unit group may consist of displacement vectors of newly generated vertices at multiple subdivision levels.

[0463] For example, if the number of mesh subdivisions performed in the mesh subdivision unit is 3, the secondary inverse transform group determination unit may form a total of 2 unit groups by configuring the displacement vector of R0 as group 0 and the displacement vectors of R1, R2, and R3 as group 1, as shown in FIG. 20. According to embodiments, the unit groups may be determined according to an encoder / decoder agreement. According to embodiments, the unit groups may be determined by parsing the unit group index parameter (secondary_transform_group_index[k]) from the received signaling information for each subdivision level. Here, k represents each subdivision level.

[0464] For example, if the number of mesh subdivisions performed in the mesh subdivision unit is 3, and the index parameter of the unit group (secondary_transform_group_index[k]) is parsed from the received signaling information as follows, the secondary inverse transform group determination unit can determine the displacement vector of R0 as group 0 and the displacement vectors of R1, R2, and R3 as group 1.

[0465] That is, secondary_transform_group_index[0]=0

[0466] secondary_transform_group_index[1]=1

[0467] secondary_transform_group_index[2]=1

[0468] secondary_transform_group_index[3]=1

[0469] In FIG. 32, the second inverse transform method determination unit can determine whether to perform a second inverse transform for each group and determine the second inverse transform method. The second inverse transform method may include a base mesh component prediction-based method, a lifting wavelet-based method, a Discrete Cosine Transform-based method, etc.

[0470] According to the embodiments, subdivision levels can be grouped into groups, and a secondary inverse transformation of the displacement vector for each subdivision level belonging to the group can be performed by the secondary inverse transformation execution unit of the secondary inverse transformation unit of the displacement vector. At this time, the secondary inverse transformation execution unit may apply the same secondary inverse transformation method to the subdivision levels belonging to the group, or omit the secondary inverse transformation for all subdivision levels belonging to the group.

[0471] According to the embodiments, whether to perform a second inverse transformation for each unit group and the method of the second inverse transformation may be determined according to the encoder / decoder agreement.

[0472] For example, if the displacement vector of vertex R0 of the base mesh in the encoder is configured as group 0 and a quadratic transformation is performed only on group 0, then the inverse quadratic transformation process can be performed only on group 0 containing the displacement vector of vertex R0 of the base mesh.

[0473] According to the embodiments, whether to perform a secondary inverse transform for each unit group and the secondary inverse transform method can be determined by parsing the secondary transform method parameter (secondary_transform_method) received by signaling information in units such as sequences, GOFs, frames, and submeshes. Taking FIG. 21 as an example, if the value of secondary_transform_method is 0, the secondary inverse transform method determination unit can determine that the secondary inverse transform is omitted, i.e., not performed. Additionally, if the value of secondary_transform_method is 1, the secondary inverse transform method determination unit determines the component prediction-based method as the secondary inverse transform method; if the value of secondary_transform_method is 2, the lifting wavelet transform-based method as the secondary inverse transform method; and if the value of secondary_transform_method is 3, the discrete cosine transform-based method as the secondary inverse transform method.

[0474] In FIG. 32, the second inverse transformation performing unit may omit the second inverse transformation based on the result determined by the second inverse transformation method determining unit, or perform the second inverse transformation using the determined method.

[0475] The above second inverse transformation execution unit generates a second inverse transformed displacement vector by performing the second inverse transformation method determined by the above second inverse transformation method determination unit in the group unit determined by the above second inverse transformation group determination unit.

[0476] According to the embodiments, if it is determined that the second inverse transformation for the current displacement vector group is omitted, the second inverse transformation for the group may be omitted, and the displacement vector decoded from the bitstream of the group may be output as is to the displacement vector coordinate system inverse transformation unit.

[0477] According to the embodiments, when a sorting method for the vertices of the current group is performed in the encoder, a process for mapping the vertex order after the second inverse transformation process to the vertex order determined in the mesh subdivision process may be performed.

[0478] According to the embodiments, when the second inverse transform performing unit is determined to perform a second inverse transform, it may perform a second inverse transform based on a component prediction-based method, a lifting wavelet transform-based method, a discrete cosine transform-based method, or other transform methods not mentioned herein.

[0479] The following is a detailed description of performing the second-order inverse transform using a discrete cosine transform-based method.

[0480] According to embodiments, when a second-order inverse transform is performed using a discrete cosine transform-based method, as shown in FIG. 33, the current displacement vector group can be divided into m subgroups, and a second-order inverse transformed displacement vector can be generated by applying a 1D inverse discrete cosine transform to each subgroup. The 1D inverse discrete cosine transform can be performed using a formula such as DCT-III or using a value obtained by approximating and converting a formula such as DCT-III into an integer.

[0481] FIG. 33 is a diagram showing an example of a second-order inverse transform method based on discrete cosine transform according to embodiments. For example, if the current displacement vector group is group 0 (i.e., group 0), group 0 is divided into m subgroups, and the displacement vector of each subgroup is second-order inverse transformed based on discrete cosine transform.

[0482] The following is a detailed description of performing a second-order inverse transform using a lifting wavelet transform-based method.

[0483] FIG. 34 is a diagram showing an example of a second-order inverse transform method based on a lifting wavelet transform according to embodiments. According to embodiments, when a second-order inverse transform is performed using a lifting wavelet transform-based method, the second-order inverse transform performing unit can generate a second-order transformed displacement vector through a process such as that shown in FIG. 34.

[0484] The dividing section of FIG. 34 divides the displacement vector of the current group into a plurality of subgroups, for example, two subgroups (a first subgroup compensation displacement vector and a second subgroup residual displacement vector), according to the method of merging in the encoder (i.e., the merging section of FIG. 23).

[0485] The update unit of FIG. 34 restores the displacement vector of the first subgroup by inversely compensating the compensated displacement vector of the first subgroup, which is compensated through the residual displacement vector of the second subgroup. According to embodiments, the update weights may be determined by an agreement between the encoder and the decoder, or by receiving and parsing information signaled from the encoder.

[0486] The prediction unit of FIG. 34 restores the displacement vector of the second subgroup by adding the residual displacement vector of the second subgroup to the predictor predicted through the displacement vector of the first subgroup. According to embodiments, the prediction may be performed through the average or weighted sum of the displacement vectors of adjacent vertices based on the vertex alignment order.

[0487] The merging section of Fig. 34 merges the displacement vectors of two subgroups according to the method divided by the encoder (i.e., the dividing section of Fig. 23) to generate the second-order inverse transformed displacement vector of the current group.

[0488] The following is a detailed description of performing a second-order inverse transform using a component prediction-based method.

[0489] According to the embodiments, when a second inverse transformation is performed using a component prediction-based method, the second inverse transformation performing unit can generate a second inverse transformed displacement vector by performing a prediction based on connectivity information of the current group vertex.

[0490] According to the embodiments, when the current group is composed of vertices of the base mesh, the connectivity information may be the connectivity information of the base mesh vertices restored through the base mesh restoration unit. That is, when the current group is composed of vertices of the base mesh, the second inverse transformation unit performs component prediction based on the connectivity information of the base mesh vertices restored through the base mesh restoration unit to generate a second inverse transformed displacement vector.

[0491] According to the embodiments, the connectivity information may be calculated by applying a mesh connectivity coding method, such as DRACO or Edgebreaker, to the vertices of the current group.

[0492] According to the embodiments, a second-order inverse transformed displacement vector can be generated as shown in Equation 6 below by performing a delta encoding method based on the connectivity information of the current group vertex.

[0493] [Mathematical Formula 6]

[0494]

[0495] d' in the above mathematical formula 6 n and d' n-1 is the restored displacement vector of the nth and (n-1)th vertices of the current group, a' n represents the nth displacement vector of the current group decoded from the bitstream.

[0496] According to the embodiments, the second inverse transform performing unit may generate a second inverse transformed displacement vector as shown in Equation 7 below by performing a parallelogram prediction method (e.g., (a) of FIG. 24) based on connectivity information of the current group vertex.

[0497] [Mathematical Formula 7]

[0498]

[0499] In the above mathematical formula 7, pred n is the predicted displacement vector for the displacement vector of the nth vertex of the current group,

[0500] R'0, L'0, and O'0 represent the restored displacement vectors of the vertices of adjacent parallelograms used for prediction, as shown in FIG. 24 (a). In Equation 7 above, d' n is the restored displacement vector of the nth vertex of the current group, a' n represents the nth displacement vector of the current group decoded from the bitstream.

[0501] According to the embodiments, the second inverse transform performing unit can generate a second inverse transformed displacement vector as shown in Equation 8 by performing a multi-parallelogram prediction method (e.g., (b) of FIG. 24) based on connectivity information of the current group vertex.

[0502] [Mathematical Formula 8]

[0503]

[0504] In the above mathematical formula 8, pred n is the predicted displacement vector for the displacement vector of the nth vertex of the current group, k is the number of neighboring parallelograms used for prediction as in Fig. 24 (b), and R' i , L' i , O' irepresents the restored displacement vector of each vertex of the parallelogram used for prediction, as shown in Fig. 24(b). In the above Equation 8, d' n is the restored displacement vector of the nth vertex of the current group, a' n represents the nth displacement vector of the current group decoded from the bitstream.

[0505] According to the embodiments, the second inverse transform performing unit may generate a second inverse transformed displacement vector as shown in Equation 9 below by performing a trapezoid prediction method (e.g., (a) of FIG. 25) based on connectivity information of the current group vertex.

[0506] [Mathematical Formula 9]

[0507]

[0508] In the above mathematical formula 9, pred n represents the predicted displacement vector for the displacement vector of the nth vertex of the current group, and R'0, L'0, O'0, and R'1 represent the restored displacement vectors of neighboring vertices used for prediction as shown in FIG. 25 (a). In the above Equation 9, d' n is the restored displacement vector of the nth vertex of the current group, a' n represents the nth displacement vector of the current group decoded from the bitstream.

[0509] According to embodiments, the second inverse transform performing unit may generate a second inverse transformed displacement vector as shown in Equation 10 below by performing a multi-trapezoid prediction method (e.g., (b) of FIG. 25) based on connectivity information of the current group vertex.

[0510] [Mathematical Formula 10]

[0511]

[0512] In the above mathematical formula 10, pred nrepresents the predicted displacement vector for the displacement vector of the nth vertex of the current group, k represents the number of neighboring face structures used for prediction as shown in FIG. 25 (b), and R'0, L'0, O'0, and R'1 represent the restored displacement vectors of the vertices of each face structure used for prediction as shown in FIG. 25 (b). In the above Equation 10, d' n is the restored displacement vector of the nth vertex of the current group, a' n represents the nth displacement vector of the current group decoded from the bitstream.

[0513] According to embodiments, the displacement vector first inverse transform unit may perform a first inverse transform on the displacement vector that has been secondarily inverse transformed in the displacement vector second inverse transform unit. According to embodiments, if the displacement vector first inverse transform is omitted, the secondly inverse transformed displacement vector may be output as is to the displacement vector coordinate system inverse transform unit. According to embodiments, if the displacement vector second inverse transform is omitted, the firstly inverse transformed displacement vector may be output to the displacement vector coordinate system inverse transform unit.

[0514] The above displacement vector first-order inverse transform unit can perform a first-order inverse transform using wavelet transform, lifting transform, etc.

[0515] FIG. 35 is a diagram showing an example of the first-order inverse transformation of a displacement vector in a first-order inverse transformation unit according to embodiments. That is, when a lifting inverse transformation is performed, the inverse transformation of the displacement vector can be performed through the process of FIG. 35. The lifting inverse transformation can be performed n times as shown in FIG. 35, and the number of lifting inverse transformations can be determined according to the mesh subdivision count parameter (subdivision_iteration_count) parsed from the signaling information transmitted from the encoder. The lifting inverse transformation process can be performed in units of mesh subdivision levels.

[0516] According to embodiments, the lifting inverse transform update unit may perform a process of updating the displacement vector of the vertex used for prediction through the second-order inverse transformed displacement vector (i.e., the first-order transformed displacement vector from the encoder of the transmitting device). According to embodiments, the weights used during the update process may be determined by an agreement between the encoder and the decoder, or may be determined by parsing information received by signaling information from the encoder. According to embodiments, the weights used during the update process may be derived based on information such as the current level of refinement and the number of edges connected to the vertex.

[0517] According to embodiments, the lifting inverse transformation prediction unit is the vertex R of the k-th refinement level. k When performing displacement vector prediction, t(t <k 또는 t<=k)번째 세분화 레벨의 정점 R t The displacement vector of can be used as a predictor to predict the displacement vector of the k-th level of refinement. Then, the displacement vector can be restored through the sum of the predicted displacement vector and the second-order inverse transformed displacement vector (first-order transformed displacement vector).

[0518] According to embodiments, when performing displacement vector prediction, the prediction can be performed by averaging or distance-based weighted averaging the displacement vectors of m vertices that are close based on connectivity information among vertices with a refinement level lower than or equal to that of the current vertex. According to embodiments, when performing displacement vector prediction, the prediction can be performed based on the displacement vectors of m vertices used to generate the current vertex according to the refinement method in the mesh refinement step. According to embodiments, when performing displacement vector prediction, the prediction can be performed based on the normal vectors of the current vertex and vertices adjacent to the current vertex.

[0519] As described above, the transformation of the displacement vector in the encoder can be performed through a first transformation process and a second transformation process, and the inverse transformation of the displacement vector in the decoder can be performed through a second inverse transformation process and a first inverse transformation process. In addition, the first transformation process and the second transformation process of the displacement vector in the encoder may be omitted, either or both. Similarly, the second inverse transformation process and the first inverse transformation process of the displacement vector in the decoder may be omitted, either or both.

[0520] To this end, a transmitting device may signal information related to displacement vector transformation into signaling information and transmit it to a receiving device, and the receiving device may parse information related to displacement vector transformation from the received signaling information to perform or omit the inverse transformation of the displacement vector. At this time, information related to displacement vector transformation may be transmitted in units such as a sequence, a Group of Frames (GOF), a frame, or a submesh. The information related to displacement vector transformation in this disclosure includes parameters for a second transformation and may be referred to as parameters, syntax elements, or fields related to displacement vector transformation. As another example, in an encoder / decoder, displacement vector transformation / inverse transformation may be derived (or determined) by an agreement between the encoder / decoder.

[0521] In the present disclosure, signaling information may include an atlas sequence parameter set (ASPS), and said atlas sequence parameter set may include information related to displacement vector transformation. According to embodiments, the atlas sequence parameter set may include a lifting transformation parameter set (vdmc_lifting_transform_parameters()), and said lifting transformation parameter set (vdmc_lifting_transform_parameters()) may include information related to said displacement vector transformation.

[0522] According to embodiments, information related to displacement vector transformation may include parameters for a secondary transformation. The parameters for the secondary transformation may include one or more of vltp_secondary_transform_enable_flag, vltp_secondary_transform_target_flag, vltp_secondary_transform_group_index, vltp_secondary_transform_method, and vltp_secondary_transform_type.

[0523] According to the embodiments, information related to displacement vector transformation can be signaled to a set of lifting transformation parameters in various ways. The present disclosure describes the information related to displacement vector transformation signaled to a set of lifting transformation parameters as divided into first to third embodiments as follows.

[0524] FIG. 36 is a diagram showing an example of the syntax structure of a lifting transformation parameter set (vdmc_lifting_transform_parameters(ltpIndex, subdivisionCount)) according to a first embodiment. In FIG. 36, ltpIndex represents the index of the lifting transformation parameter set, and subdivisionCount represents the current subdivision iteration count. According to embodiments, a mesh subdivision count parameter (subdivision_iteration_count) is signaled to ASPS, and this mesh subdivision count parameter (subdivision_iteration_count) is stored in the variable subdivisionCount.

[0525] In FIG. 36, if the value of vltp_skip_update_flag[ltpIndex] is 1, it indicates that the update step of the lifting transform applied to the displacement is skipped, and if the value of vltp_skip_update_flag[ltpIndex] is 0, it indicates that the update step of the lifting transform applied to the displacement is not skipped.

[0526] vltp_secondary_transform_enable_flag[ltpIndex] is a parameter indicating whether to perform a second transformation. For example, in the encoder / decoder, if the value of vltp_secondary_transform_enable_flag is 0, the second transformation and inverse transformation are omitted for the displacement vectors of all subdivision level vertices. And, if the value of vltp_secondary_transform_enable_flag is 1, the encoder / decoder performs the second transformation and inverse transformation on the displacement vectors of the base mesh (subdivision level 0) vertices. In this case, the second transformation and inverse transformation of the base mesh (subdivision level 0) vertices are performed using a component prediction-based method.

[0527] vltp_log2_lifting_prediction_weight[ltpIndex] is a parameter representing the weight coefficient used for lifting transformation prediction.

[0528] FIG. 37 is a diagram showing an example of the syntax structure of a lifting transformation parameter set (vdmc_lifting_transform_parameters(ltpIndex, subdivisionCount)) according to a second embodiment. In FIG. 37, ltpIndex represents the index of the lifting transformation parameter set, and subdivisionCount represents the current number of subdivision iterations. According to the embodiments, a mesh subdivision count parameter (subdivision_iteration_count) is signaled to ASPS, and this mesh subdivision count parameter (subdivision_iteration_count) is stored in the variable subdivisionCount.

[0529] In FIG. 37, if vltp_skip_update_flag[ltpIndex] is equal to 1, it indicates that the update step of the lifting transform applied to the displacement is skipped, and if vltp_skip_update_flag[ltpIndex] is equal to 0, it indicates that the update step of the lifting transform applied to the displacement is not skipped.

[0530] vltp_secondary_transform_enable_flag[ltpIndex] is a parameter indicating whether to perform a second transformation. For example, in the encoder / decoder, if the value of vltp_secondary_transform_enable_flag is 0, the second transformation and inverse transformation are omitted for the displacement vectors of all subdivision level vertices. And, if the value of vltp_secondary_transform_enable_flag is 1, the encoder / decoder performs the second transformation and inverse transformation for the base mesh (subdivision level 0) vertices or the displacement vectors of all subdivision level vertices. In this case, the second transformation and inverse transformation of the base mesh (subdivision level 0) vertices can be performed using a component prediction-based method.

[0531] vltp_secondary_transform_target_flag[ltpIndex] is a parameter indicating the target for the quadratic transformation. For example, if the value of vltp_secondary_transform_target_flag is 0, the encoder / decoder performs quadratic and inverse transformations on the displacement vectors of the base mesh (segmentation level 0) vertices. In this case, the quadratic and inverse transformations are performed on the displacement vectors of the base mesh (segmentation level 0) vertices using a component prediction-based method. For another example, if the value of vltp_secondary_transform_target_flag is 1, the encoder / decoder performs quadratic and inverse transformations on the displacement vectors of all segmentation level vertices. According to the embodiments, when the value of vltp_secondary_transform_enable_flag[ltpIndex] is true (e.g., 1), the lifting transformation parameter set may include vltp_secondary_transform_target_flag[ltpindex].

[0532] vltp_secondary_transform_group_index[ltpIndex][i] is a parameter representing the group number to which the displacement vector of the i-th subdivision level belongs. That is, in vltp_secondary_transform_group_index[ltpIndex][i], i represents the subdivision level, and the above parameter represents the group number to which the displacement vector for each subdivision level belongs. For example, when subdivisionCount is 3 and the vertices of the base mesh are defined as R0, the vertices newly created by performing subdivision 1 are defined as R1, the vertices newly created by performing subdivision 2 are defined as R2, and the vertices newly created by performing subdivision 3 are defined as R3, when the unit group index parameter is parsed as follows, the displacement vector of R0 can be determined as group 0, and the displacement vectors of R1, R2, and R3 as group 1.

[0533] secondary_transform_group_index[0]=0

[0534] secondary_transform_group_index[1]=1

[0535] secondary_transform_group_index[2]=1

[0536] secondary_transform_group_index[3]=1

[0537] At this time, the number of unit groups for the second transformation can be determined by the group number parsed for the last level of refinement.

[0538] According to embodiments, if vltp_secondary_transform_enable_flag[ltpIndex] is true (e.g., 1) and vltp_secondary_transform_target_flag[ltpindex]) is true (e.g., 1), the lifting transformation parameter set includes vltp_secondary_transform_group_index[ltpIndex][i], and vltp_secondary_transform_group_index[ltpIndex][i] can be stored (or assigned) to the variable secondaryTransformGroupIndex[ltpIndex][i], where i represents the current refinement step (or level).

[0539] vltp_log2_lifting_prediction_weight[ltpIndex] is a parameter representing the weight coefficient used for lifting transformation prediction.

[0540] According to embodiments, when vltp_secondary_transform_enable_flag[ltpIndex] is true (e.g., 1) and vltp_secondary_transform_target_flag[ltpindex]) is true (e.g., 1), secondaryTransformGroupIndex[ltpIndex][subdivisionCount] may be assigned (or stored) to the variable secondaryTransformGroupCount[ltpIndex]. And, the lifting transformation parameter set may include vltp_secondary_transform_method[ltpIndex][i] while iterating as many times as the value of secondaryTransformGroupCount[ltpIndex].

[0541] The above vltp_secondary_transform_method[ltpIndex][i] is a parameter representing the second transformation method to be performed for each group. Here, i represents the group number. According to embodiments, the encoder / decoder may determine the second transformation method for each group as shown in FIG. 21 based on the above vltp_secondary_transform_method[ltpIndex][i]. For example, if the value of the above vltp_secondary_transform_method[ltpIndex][i] is 0, the second transformation of the i-th group may be omitted; if it is 1, the second transformation of the i-th group may be applied using a component prediction-based method; if it is 2, the second transformation of the i-th group may be applied using a lifting wavelet transform technique; and if it is 3, the second transformation of the i-th group may be applied using a discrete cosine transform-based method. According to the embodiments, the secondary transformation method determined through the vltp_secondary_transform_method[ltpIndex][i] may be configured to include some of the methods specified in FIG. 21 and / or additional methods other than those specified.

[0542] According to embodiments, the present disclosure may apply the same secondary transformation method to each group or omit secondary transformation for all subdivision levels belonging to a group. In this case, according to embodiments, if the number of subdivision levels increases, the secondary transformation method may be signaled by group rather than signaling the secondary transformation method for each subdivision level.

[0543] FIG. 38 is a diagram showing an example of the syntax structure of a lifting transformation parameter set (vdmc_lifting_transform_parameters(ltpIndex, subdivisionCount)) according to a third embodiment. In FIG. 38, ltpIndex represents the index of the lifting transformation parameter set, and subdivisionCount represents the current subdivision iteration count. According to embodiments, a mesh subdivision count parameter (subdivision_iteration_count) is signaled to ASPS, and this mesh subdivision count parameter (subdivision_iteration_count) is stored in the variable subdivisionCount.

[0544] In FIG. 38, if vltp_skip_update_flag[ltpIndex] is equal to 1, it indicates that the update step of the lifting transform applied to the displacement is skipped, and if vltp_skip_update_flag[ltpIndex] is equal to 0, it indicates that the update step of the lifting transform applied to the displacement is not skipped.

[0545] vltp_secondary_transform_type[ltpIndex] is a parameter indicating the type of quadratic transformation. For example, if the value of vltp_secondary_transform_type[ltpIndex] is 0, the encoder / decoder omits quadratic and inverse transformations for the displacement vectors of all subdivision level vertices. For another example, if the value of vltp_secondary_transform_type is 1, the encoder / decoder performs quadratic and inverse transformations on the displacement vectors of the base mesh (subdivision level 0) vertices. In this case, the quadratic and inverse transformations of the base mesh (subdivision level 0) vertices are performed using a component prediction-based method. For yet another example, if the value of vltp_secondary_transform_type is 2, the encoder / decoder performs quadratic and inverse transformations on the displacement vectors of all subdivision level vertices. The above vltp_secondary_transform_type[ltpIndex] can be assigned (or stored) to the variable secondaryTransformType[ltpIndex].

[0546] vltp_secondary_transform_group_index [ltpIndex][i] is a parameter representing the group number to which the displacement vector belongs for each subdivision level. Here, i represents the subdivision level. In this case, the number of unit groups for the secondary transformation can be determined by the group number parsed for the last subdivision level. According to embodiments, if secondaryTransformType[ltpIndex] is 2 for each subdivision level, the vltp_secondary_transform_group_index [ltpIndex][i] may be included in the lifting transformation parameter set, and vltp_secondary_transform_group_index [ltpIndex][i] may be assigned (or stored) to the variable secondaryTransformGroupIndex [ltpIndex][i].

[0547] vltp_log2_lifting_prediction_weight[ltpIndex] is a parameter representing the weight coefficient used for lifting transformation prediction.

[0548] vltp_secondary_transform_method[ltpIndex][i] is a parameter representing the second transformation method to be performed for each group. Here, i represents the group number. According to embodiments, the encoder / decoder may determine the second transformation method for each group as shown in FIG. 21 based on the vltp_secondary_transform_method[ltpIndex][i]. For example, if the value of vltp_secondary_transform_method[ltpIndex][i] is 0, the second transformation of the i-th group may be omitted; if it is 1, the second transformation of the i-th group may be applied using a component prediction-based method; if it is 2, the second transformation of the i-th group may be applied using a lifting wavelet transform technique; and if it is 3, the second transformation of the i-th group may be applied using a discrete cosine transform-based method. According to the embodiments, the secondary transformation method determined through the vltp_secondary_transform_method[ltpIndex][i] may be configured to include some of the methods specified in FIG. 21 and / or additional methods other than those specified.

[0549] According to embodiments, the present disclosure may apply the same secondary transformation method to each group or omit secondary transformation for all subdivision levels belonging to a group. In this case, according to embodiments, if the number of subdivision levels increases, the secondary transformation method may be signaled by group rather than signaling the secondary transformation method for each subdivision level.

[0550] FIG. 39 is a flowchart showing an example of an encoding method according to embodiments. The encoding method according to embodiments may include the step of encoding a base mesh of mesh data (S31011), the step of encoding a displacement of mesh data (S31012), and the step of encoding attributes of mesh data (S31013).

[0551] In the step of encoding the base mesh of the above mesh data (S31011), if intra-frame encoding is performed for the corresponding mesh frame, the base mesh can be encoded using a static mesh encoder. In this case, encoding can be performed on the connectivity information, vertex geometry information, vertex texture information, normal information, etc. of the base mesh. In the step of encoding the base mesh of the above mesh data (S31011), if inter-frame encoding is performed for the corresponding mesh frame, a motion vector encoder can calculate the motion vector between the base mesh and the reference restored base mesh (or the restored quantized reference base mesh) and encode the value. Additionally, a prediction based on connectivity information can be performed using a previously encoded / decoded motion vector as a predictor, and the residual motion vector obtained by subtracting the predicted motion vector from the current motion vector can be encoded.

[0552] The step of encoding the displacement of the mesh data (S31012) may perform video codec-based encoding or arithmetic codec-based encoding on the displacement data. The step of encoding the displacement of the mesh data (S31012) may convert the coordinate system of the displacement data from a 3D Cartesian coordinate system to a local coordinate system before encoding the displacement data. If the displacement data is encoded based on a video codec, it may be encoded through a 2D video codec after undergoing displacement vector transformation, displacement vector transformation coefficient quantization, and displacement vector transformation coefficient packing processes.

[0553] According to embodiments, the displacement vector transformation in the present disclosure may be performed through a first transformation process and a second transformation process as described in FIGS. 17 to 25. In this case, the first transformation process and / or the second transformation process may be omitted. That is, the displacement vector second transformation may be omitted for the displacement vectors of all subdivision level vertices, the displacement vector second transformation may be performed only for the displacement vectors of the base mesh (subdivision level 0) vertices, or the displacement vector second transformation may be performed for the displacement vectors of all subdivision level vertices. If the displacement vector second transformation is performed only for the displacement vectors of the base mesh (subdivision level 0) vertices, the subdivided mesh and the displacement vector may be applied to a static mesh encoder to perform the transformation of the displacement vector. At this time, connectivity information of the subdivided mesh is calculated, and based on the calculated connectivity information, the displacement vector of the base mesh (subdivision level 0) vertex can be transformed through a component prediction method. In addition, if the quadratic transformation of the displacement vector is omitted, the quadratic transformed displacement vector applied as input can be output as is. According to the embodiments, subdivision levels may be grouped into groups, and the quadratic transformation of the displacement vector for the subdivision levels belonging to each group may be performed. In this case, the same quadratic transformation method may be performed for the subdivision levels belonging to the group, or the quadratic transformation may be omitted for all subdivision levels belonging to the group.

[0554] A detailed description of displacement vector transformations including first and second transformations of the present disclosure is omitted here to avoid redundancy, with reference to the descriptions in FIGS. 14 to 25.

[0555] In the present disclosure, parameters for secondary transformation may be signaled in units such as sequences, GOFs, frames, submeshes, and mesh patches, and transmitted to a decoder of a receiving device. The parameters for secondary transformation may include secondary transformation execution parameters (e.g., secondary_transform_enable_flag), secondary transformation target parameters (e.g., secondary_transform_target_flag), secondary transformation type parameters (e.g., secondary_transform_type), secondary transformation method parameters (e.g., secondary_transform_method), and index parameters of unit groups determined at each level of subdivision (e.g., secondary_transform_group_index). A detailed description of the parameters for secondary transformation is to be made with reference to the descriptions in FIGS. 36 to 38, and is omitted here to avoid redundant descriptions.

[0556] The step of encoding attributes of the mesh data (S31013) can perform encoding based on a video codec for the attribute data (or texture map).

[0557] The above encoding method may further include the step of transmitting a bitstream comprising an encoded base mesh, encoded displacement data, encoded attribute data and / or atlas data.

[0558] The encoding method of the present disclosure may be performed by an encoding device (encoder). The encoding device includes a memory and at least one processor connected to the memory, and the at least one processor may be configured to encode a base mesh of mesh data, encode a displacement of mesh data, and encode attributes of mesh data.

[0559] The embodiments further include a computer-readable storage medium that stores a bitstream generated by the method according to FIG. 39.

[0560] The embodiments further include a method comprising the steps of acquiring a bitstream for mesh data, said bitstream being generated based on the steps of encoding a base mesh of the mesh data, encoding a displacement of the mesh data, and encoding attributes of the mesh data, and transmitting data including said bitstream.

[0561] FIG. 40 is a flowchart showing an example of a decoding method according to embodiments. The decoding method according to embodiments may include a step of decoding a base mesh within a bitstream (S32011), a step of decoding a displacement within a bitstream (S32012), and a step of decoding an attribute within a bitstream (S32013). The decoding step of FIG. 40 may further include a step of receiving a bitstream containing a base mesh, displacement data, and attribute data, or a file in which the bitstream is encapsulated. The receiving step performs a decapsulation process to extract the bitstream when the file is received, and omits the decapsulation process when the bitstream is received.

[0562] In the step of decoding the base mesh within the bitstream (S32011), if the current mesh is subject to inter-frame encoding, the motion vector decoder may use the previously decoded motion vector as a predictor and add it to the residual motion vector decoded from the base mesh bitstream to restore the final motion vector. In the step of decoding the base mesh within the bitstream (S32011), if the current mesh is subject to intra-frame encoding, the static mesh decoder may statically decode the base mesh bitstream to restore the connection information, vertex geometry information, texture coordinates, normal information, etc. of the base mesh.

[0563] The step of decoding displacement within the bitstream (S32012) performs decoding on the displacement vector bitstream based on a video codec if the displacement data is encoded based on a video codec, and performs decoding based on an arithmetic codec if it is encoded based on an arithmetic codec. At this time, the displacement vector decoded based on a video codec or an arithmetic codec may undergo an inverse quantization process and an inverse transform process.

[0564] According to embodiments, the displacement vector inverse transformation process may include a second inverse transformation process and a first inverse transformation process. According to embodiments, if the first transformation and / or second transformation process in the encoder is omitted, the first inverse transformation process and / or the second inverse transformation process may be omitted.

[0565] The above second inverse transformation process corresponds to the second transformation process of the encoder of the transmitting device, and the first inverse transformation process corresponds to the first transformation process of the encoder of the transmitting device.

[0566] According to the embodiments, the displacement vector inverse transformation process can perform a second inverse transformation process and a first inverse transformation process by parsing parameters for a second transformation received and included in the signaling information, or it can omit one or both of the second inverse transformation process and the first inverse transformation process.

[0567] According to the embodiments, if the first conversion and / or second conversion process in the encoder is omitted, the first inverse conversion process and / or the second inverse conversion process may be omitted.

[0568] According to the embodiments, the subdivided mesh and the displacement vector are applied to the static mesh decoder of the step (S32011) of decoding the base mesh, and the inverse transformation of the displacement vector can be performed in the static mesh decoder. That is, connectivity information of the subdivided mesh is calculated in the static mesh decoder, and the displacement vector can be inversely transformed through a component prediction method based on the calculated connectivity information.

[0569] A detailed description of the displacement vector inverse transformation, including the second-order inverse transformation and the first-order inverse transformation of the present disclosure, will be omitted here to avoid redundancy, by referring to the description in FIGS. 26 to 35.

[0570] In the present disclosure, parameters for secondary transformation may be received by signaling in units such as sequences, GOFs, frames, submeshes, and mesh patches. The parameters for secondary transformation may include secondary transformation execution parameters (e.g., secondary_transform_enable_flag), secondary transformation target parameters (e.g., secondary_transform_target_flag), secondary transformation type parameters (e.g., secondary_transform_type), secondary transformation method parameters (e.g., secondary_transform_method), and index parameters of unit groups determined at each level of subdivision (e.g., secondary_transform_group_index). A detailed description of the parameters for secondary transformation is to be made with reference to the descriptions in FIGS. 36 to 38, and is omitted here to avoid redundant descriptions.

[0571] The step of decoding attributes within the bitstream (S32013) decodes the attribute (or texture map) bitstream based on a video codec to restore attribute data.

[0572] The base mesh, displacement data, and attribute data decoded in the above steps S32011-S32013 can be rendered after undergoing post-processing steps such as mesh restoration and reconstruction.

[0573] The decoding method of the present disclosure may be performed by a decoding device (decoder). The decoding device includes a memory and at least one processor connected to the memory, and the at least one processor may be configured to decode a base mesh in a bitstream, decode a displacement in a bitstream, and decode an attribute in a bitstream.

[0574] As described above, the present disclosure can perform a second transformation for each subdivision level on the signal resulting from the lifting transformation of the displacement vector currently performed in the encoder / decoder of a conventional V-DMC, and transmit the resulting signal to a receiving device. Additionally, the decoder of the receiving device can parse the signaled signal and perform a second inverse transformation of the displacement vector for each subdivision level. Through this, the encoding performance of the encoder of the transmitting device can be improved by reducing the redundancy of the displacement vector and the amount of transmitted bits can be reduced, and the decoding performance of the decoder of the receiving device can be improved by performing decoding with the reduced amount of bits.

[0575] Each of the aforementioned parts, modules, or units may be software, processors, or hardware parts that execute successive processes stored in memory (or storage units). Each step described in the aforementioned embodiments may be performed by processors, software, or hardware parts. Each module / block / unit described in the aforementioned embodiments may operate as a processor, software, or hardware. Additionally, the methods presented in the embodiments may be executed as code. This code may be written to a storage medium that is readable by a processor and thus read by a processor provided by the apparatus.

[0576] Furthermore, throughout the specification, when a part is described as “comprising” a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Also, terms such as “…part” as used in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0577] Although the drawings have been described separately for convenience of explanation in this specification, it is also possible to design a new embodiment by combining the embodiments described in each drawing. Furthermore, according to the needs of a person skilled in the art, designing a computer-readable recording medium on which a program for executing the previously described embodiments is recorded falls within the scope of the embodiments.

[0578] The apparatus and method according to the embodiments are not limited to the configurations and methods of the embodiments described above; rather, the embodiments may be configured by selectively combining all or part of each embodiment so that various modifications can be made.

[0579] Although preferred embodiments have been illustrated and described, the embodiments are not limited to the specific embodiments described above, and various modifications can be made by those skilled in the art without departing from the essence of the embodiments claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the embodiments.

[0580] Various components of the device according to the embodiments may be implemented by hardware, software, firmware, or a combination thereof. Various components of the embodiments may be implemented as a single chip, for example, a single hardware circuit. Components according to the embodiments may each be implemented as separate chips. At least one of the components of the device according to the embodiments may be composed of one or more processors capable of executing one or more programs, and one or more programs may include instructions for performing or executing any one or more of the operations / methods according to the embodiments. Executable instructions for performing the methods / operations of the device according to the embodiments may be stored in non-transient CRMs or other computer program products configured to be executed by one or more processors, or may be stored in transient CRMs or other computer program products configured to be executed by one or more processors. Additionally, memory according to the embodiments may be used as a concept that includes not only volatile memory (e.g., RAM, etc.) but also non-volatile memory, flash memory, PROM, etc. In addition, it may also include implementation in the form of a carrier wave, such as transmission over the Internet. Furthermore, processor-readable recording media may be distributed across networked computer systems, allowing processor-readable code to be stored and executed in a distributed manner.

[0581] In this document, " / " and "," are interpreted as "and / or." For example, "A / B" is interpreted as "A and / or B," and "A, B" is interpreted as "A and / or B." Additionally, "A / B / C" means "at least one of A, B, and / or C." Also, "A, B, C" means "at least one of A, B, and / or C." Additionally, in this document, "or" is interpreted as "and / or." For example, "A or B" may mean 1) "A" only, 2) "B" only, or 3) "A and B." In other words, "or" in this document may mean "additionally or alternatively."

[0582] Various elements of the embodiments may be performed by hardware, software, firmware, or a combination thereof. Various elements of the embodiments may be performed on a single chip, such as a hardware circuit. Depending on the embodiments, the embodiments may optionally be performed on individual chips. Depending on the embodiments, at least one of the elements of the embodiments may be performed within one or more processors that include instructions for performing operations according to the embodiments.

[0583] Additionally, the operation according to the embodiments described herein may be performed by a transceiver device comprising one or more memories and / or one or more processors according to the embodiments. One or more memories may store programs for processing / controlling the operation according to the embodiments, and one or more processors may control the various operations described in this document. One or more processors may be referred to as controllers, etc. The operations of the embodiments may be performed by firmware, software, and / or a combination thereof, and the firmware, software, and / or a combination thereof may be stored in a processor or in memory.

[0584] Terms such as "first," "second," etc., may be used to describe various components of the embodiments. However, the interpretation of the various components according to the embodiments should not be limited by these terms. These terms are merely used to distinguish one component from another. For example, the first user input signal may be referred to as the second user input signal. Similarly, the second user input signal may be referred to as the first user input signal. The use of these terms should be interpreted as not departing from the scope of the various embodiments. Although the first user input signal and the second user input signal are both user input signals, they do not imply the same user input signals unless clearly indicated in the context.

[0585] The terms used to describe the embodiments are intended for the purpose of describing specific embodiments and are not intended to limit the embodiments. As used in the description of the embodiments and in the claims, the singular is intended to include the plural unless explicitly indicated in the context. Expressions of and / or are used to mean including all possible combinations between the terms. Expressions of “include” describe the presence of features, numbers, steps, elements, and / or components and do not imply the exclusion of additional features, numbers, steps, elements, and / or components. Conditional expressions such as “if” or “when” used to describe the embodiments are not limited to being optional. It is intended to be interpreted as when a specific condition is satisfied, to perform a related action in response to a specific condition, or to interpret the related definition.

[0586] As described above, the relevant details have been explained in the best mode for carrying out the embodiments.

[0587] As described above, the embodiments may be applied wholly or partially to mesh data transmission and reception devices and systems. Those skilled in the art may make various changes or modifications to the embodiments within the scope of the embodiments. The embodiments may include changes / modifications, and such changes / modifications do not deviate from the scope of the claims and their equivalents.

Claims

Step of decoding the basemesh within the bitstream; A step of decoding displacement data within the bitstream; and A step of decoding attribute data within the bitstream; comprising Decoding method. In claim 1, the step of decoding the displacement data is, A step of decoding displacement data included in the above bitstream based on a video codec, Step of backpacking the above decoded displacement data, The step of inversely quantizing the above-mentioned inversely packed displacement data, and A decoding method comprising the step of inversely transforming the above inversely quantized displacement data. In claim 2, the step of inversely transforming the displacement data is, A decoding method comprising the step of performing an inverse transformation on the displacement data of the vertices of the decoded base mesh. In claim 3, the step of inversely transforming the displacement data is, A decoding method that performs an inverse transformation of the displacement data by applying the connectivity information of the vertices of the base mesh to a component prediction method. In claim 2, the step of inversely transforming the displacement data is, A decoding method comprising the step of, when the above-decoded base mesh is subdivided into a plurality of subdivision levels, grouping the plurality of subdivision levels into one or more groups and performing an inverse transformation on the displacement data of each vertex on a group basis. In Article 5, The above group unit inverse transformation is a decoding method performed based on at least one of a component prediction-based method, a lifting-based method, and an odd cosine transform-based method. In claim 5, the step of inversely transforming the displacement data is, A decoding method further comprising the step of performing an additional lifting inverse transformation on displacement data for which an inverse transformation has been performed on the above group units. In Article 2, The above bitstream is a decoding method including parameters related to the conversion of the above displacement data. Memory; and At least one processor connected to the memory; comprising, The above at least one processor is: Decoding the basemesh within the bitstream; Decoding displacement data within the above bitstream; and Configured to decode attribute data within the above bitstream, Decoding device. In claim 9, the above at least one processor, A displacement decoding unit that decodes displacement data included in the above bitstream based on a video codec, A reverse packing unit that reverse packs the above-decoded displacement data, An inverse quantization unit that inversely quantizes the above-mentioned inversely packed displacement data, and A decoding device further comprising an inverse transformation unit that inversely transforms the above-mentioned inversely quantized displacement data. Step of encoding the base mesh of the mesh data; A step of encoding displacement data of the above mesh data; and A step of encoding attribute data of the above mesh data; comprising Encoding method. In claim 11, the step of encoding the displacement data is, Step of converting the above displacement data, Step of quantizing the above converted displacement data, The step of packing the above-mentioned quantized displacement data, and An encoding method comprising the step of encoding the above-mentioned packed displacement data based on a video codec. Memory; and At least one processor connected to the memory; comprising, wherein the at least one processor: Encode the base mesh of the mesh data; Encoding the displacement data of the above mesh data; and Configured to encode the attribute data of the above mesh data; Encoding device. A computer-readable storage medium for storing a bitstream generated by the method according to paragraph 12. Step of acquiring a bitstream for mesh data, The above bitstream is generated based on the steps of: encoding a base mesh of the mesh data; encoding displacement data of the mesh data; and encoding attribute data of the mesh data; and A method comprising the step of transmitting data including the bitstream above.

Images