Mesh data transmission device, mesh data transmission method, mesh data reception device, and mesh data reception method

Abstract

A mesh data decoding method according to embodiments may comprise: a step of receiving a base mesh bitstream, a displacement vector bitstream, a texture map bitstream, and signaling information; a base mesh processing step of restoring a base mesh from the base mesh bitstream; a displacement information processing step of restoring displacement information from the displacement vector bitstream; a restoration step of restoring a mesh on the basis of the base mesh and the displacement information; and a texture map processing step of restoring texture maps from the texture map bitstream.

Description

Mesh data transmission device, mesh data transmission method, mesh data reception device, and mesh data reception method

[0001] The embodiments provide a method for providing 3D content to provide users with various services such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and autonomous driving services.

[0002] Among 3D content, point cloud data and mesh data are collections of points in 3D space. However, the sheer number of points in 3D space makes it difficult to generate point cloud or mesh data.

[0003] That is, there is a problem that a lot of processing is required to transmit and receive 3D data with a large amount of points, such as point cloud data or mesh data.

[0004] The technical problem according to the embodiments is to provide a device and method for efficiently transmitting and receiving mesh data in order to solve the problems described above.

[0005] The technical problem according to the embodiments is to provide a device and method for resolving latency and encoding / decoding complexity of mesh data.

[0006] The technical problem according to the embodiments is to provide a device and method for efficiently performing encoding and decoding of a displacement vector.

[0007] However, the scope of the embodiments is not limited to the aforementioned technical tasks, and the scope of the embodiments may be expanded to other technical tasks that can be inferred by a person skilled in the art based on the entire contents of this document.

[0008] In order to achieve the above-described purpose and other advantages, a mesh data decoding method according to embodiments may include a step of receiving a base mesh bitstream, a displacement vector bitstream, a texture map bitstream, and signaling information, a base mesh processing step of restoring a base mesh from the base mesh bitstream, a displacement information processing step of restoring displacement information from the displacement vector bitstream, a restoration step of restoring a mesh based on the base mesh and the displacement information, and a texture map processing step of restoring texture maps from the texture map bitstream.

[0009] According to embodiments, the texture map processing step may include a step of decoding the texture map bitstream based on the signaling information to restore texture maps, a step of determining whether at least one texture map is omitted from the texture map bitstream based on the signaling information, and a step of generating the at least one omitted texture map based on the signaling information and at least one reference frame if it is determined that there is the at least one omitted texture map.

[0010] According to embodiments, the omission unit of the at least one texture map may be at least one of a frame unit, a patch unit, a sub-mesh unit, a tile unit, and a slice unit.

[0011] According to embodiments, the signaling information may include information for identifying whether the at least one texture map is omitted and reference texture map information related to the at least one reference frame.

[0012] According to embodiments, a mesh data decoding device may include a receiving unit that receives a base mesh bitstream, a displacement vector bitstream, a texture map bitstream, and signaling information, a base mesh processing unit that restores a base mesh from the base mesh bitstream, a displacement information processing unit that restores displacement information from the displacement vector bitstream, a restoration unit that restores a mesh based on the base mesh and the displacement information, and a texture map processing unit that restores a texture map from the texture map bitstream.

[0013] According to embodiments, the texture map processing unit may decode the texture map bitstream based on the signaling information to restore texture maps, determine whether at least one texture map is omitted from the texture map bitstream based on the signaling information, and, if it is determined that there is at least one omitted texture map, generate the at least one omitted texture map based on the signaling information and at least one reference frame.

[0014] According to embodiments, the omission unit of the at least one texture map may be at least one of a frame unit, a patch unit, a sub-mesh unit, a tile unit, and a slice unit.

[0015] According to embodiments, the signaling information may include information for identifying whether the at least one texture map is omitted and reference texture map information related to the at least one reference frame.

[0016] According to embodiments, a method for encoding mesh data may include encoding an original mesh, and transmitting a bitstream including the encoded mesh and signaling information.

[0017] According to embodiments, the encoding step may include a base mesh processing step of encoding a base mesh generated by simplifying an original mesh to generate a base mesh bitstream, a displacement information processing step of encoding displacement information generated based on the base mesh to generate a displacement vector bitstream, a mesh restoration step of restoring a mesh based on the encoded base mesh and the encoded displacement information, and a texture map processing step of determining whether to omit at least one texture map among texture maps generated based on the original mesh and the restored mesh, and encoding texture maps that are not omitted to generate a texture map bitstream.

[0018] According to embodiments, the texture map processing step may determine whether to omit the current texture map by comparing the similarity between the current texture map and the reference texture map before encoding the generated texture maps.

[0019] According to embodiments, the texture map processing step may omit coding and transmission of the current texture map if the difference between the signal-to-noise ratio (PSNR) value calculated through the current texture map and the PSNR value calculated through the reference texture map is less than a preset threshold value.

[0020] According to embodiments, the omission unit of the at least one texture map may be at least one of a frame unit, a patch unit, a sub-mesh unit, a tile unit, and a slice unit.

[0021] According to embodiments, the signaling information may include information for identifying whether the at least one texture map is omitted and reference texture map information related to at least one reference frame for generating the omitted texture map.

[0022] According to embodiments, the reference texture map information may include information for identifying a direction of the at least one reference frame.

[0023] According to embodiments, a computer program stored on a computer-readable recording medium can be combined with a computer, which is hardware, to perform the above method.

[0024] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can provide a quality 3D service.

[0025] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can achieve various video codec methods.

[0026] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can provide general-purpose 3D content such as autonomous driving services.

[0027] The mesh data transmission method, mesh data transmission device, mesh data reception method, and mesh data reception device according to the embodiments can obtain the effects of reducing bits of the transmitted texture map, improving transmission speed, and reducing the complexity of the encoder / decoder by omitting coding and transmission of the current texture map when the similarity between the reference texture map of the dynamic mesh and the current texture map is high.

[0028] The drawings are included to further understand the embodiments, and the drawings illustrate the embodiments together with the description related to the embodiments. For a better understanding of the various embodiments described below, reference should be made to the following description of the embodiments in conjunction with the following drawings, in which like reference numerals correspond to corresponding parts throughout the drawings.

[0029] FIG. 1 illustrates a system for providing dynamic mesh content according to embodiments.

[0030] Figure 2 illustrates a V-MESH compression method according to embodiments.

[0031] Figure 3 illustrates pre-processing of V-MESH compression according to embodiments.

[0032] Figure 4 illustrates a mid-edge subdivision method according to embodiments.

[0033] Figure 5 illustrates a displacement generation process according to embodiments.

[0034] Figure 6 illustrates an intra-frame encoding process of V-MESH data according to embodiments.

[0035] Figure 7 illustrates an inter-frame encoding process of V-MESH data according to embodiments.

[0036] Figure 8 illustrates a lifting conversion process for displacement according to embodiments.

[0037] Figure 9 illustrates a process of packing transformation coefficients into a 2D image according to embodiments.

[0038] Figure 10 illustrates an attribute transfer process of a V-MESH compression method according to embodiments.

[0039] Figure 11 illustrates an intra-frame decoding process of V-MESH data according to embodiments.

[0040] Figure 12 shows an inter-frame decoding processor of V-MESH data.

[0041] Fig. 13 is a drawing showing an example of a transmitting device according to embodiments.

[0042] Fig. 14 is a drawing showing an example of a receiving device according to embodiments.

[0043] Fig. 15 is a drawing showing another example of a transmitting device according to embodiments.

[0044] FIG. 16 is a flowchart showing an example of a method for determining whether to omit texture map coding according to embodiments.

[0045] FIG. 17 is a diagram showing an example of a method for determining whether to omit a texture map using a texture map maximum distance parameter according to embodiments.

[0046] FIG. 18 is a diagram showing an example of whether a texture map is omitted and a reference frame index signaled in sub-mesh units according to embodiments.

[0047] FIG. 19 is a drawing showing an example of a frame relationship of a referenced texture map when texture map coding is omitted in sub-mesh units according to embodiments.

[0048] FIG. 20 is a drawing showing another example of a receiving device according to embodiments.

[0049] FIG. 21 is a drawing showing an example of a detailed block diagram of a texture map decoder according to embodiments.

[0050] FIG. 22 is a diagram showing an example of setting the texture map omission flag to 0 according to embodiments.

[0051] Fig. 23 is a flowchart showing an example of deriving a texture map omission flag in a texture map omission flag determination unit according to embodiments.

[0052] FIG. 24 is a diagram showing an example of whether a texture map is omitted and a reference frame index signaled in sub-mesh units according to embodiments.

[0053] FIG. 25 is a drawing showing an example of a frame relationship of a referenced texture map when texture map coding is omitted in sub-mesh units according to embodiments.

[0054] FIG. 26 is a diagram showing an example of the syntax structure of an atlas frame parameter set (AFPS) according to embodiments.

[0055] Fig. 27 is a table showing examples of whether to omit a texture map and the reference direction of afps_texture_skip_refDirection_idx according to embodiments.

[0056] FIG. 28 is a diagram showing an example of the syntax structure of an atlas tile header according to embodiments.

[0057] FIG. 29 is a diagram showing an example of a syntax structure of an atlas tile data unit according to embodiments.

[0058] FIG. 30 is a table showing examples of determining a patch mode according to a delimiter when the coding type of the current atlas tile according to embodiments is I_TILE.

[0059] FIG. 31 is a table showing examples of determining a patch mode according to a delimiter when the coding type of the current atlas tile according to embodiments is P_TILE.

[0060] FIG. 32 is a diagram showing an example of the syntax structure of a submesh header according to embodiments.

[0061] FIG. 33 is a diagram showing another example of the syntax structure of a submesh header according to embodiments.

[0062] FIG. 34 is a diagram showing another example of the syntax structure of an atlas sequence parameter set (ASPS) according to embodiments.

[0063] FIG. 35 is a diagram showing another example of the syntax structure of an atlas frame parameter set (AFPS) according to embodiments.

[0064] Figure 36 is a table showing the definition of afve_texturemap_skip_flag and afve_ref_direction_idx according to embodiments.

[0065] FIG. 37 is a drawing showing an example of texture map omission according to embodiments.

[0066] FIG. 38(a) and FIG. 38(b) are drawings showing an example of determining whether to omit a texture map and a reference information index on a frame-by-frame basis according to embodiments.

[0067] FIG. 39 is a diagram showing an example of a syntax structure of an atlas tile layer according to embodiments.

[0068] FIG. 40 is a diagram showing an example of a syntax structure of an atlas tile header according to embodiments.

[0069] FIG. 41 is a diagram showing an example of a syntax structure of patch information data according to embodiments.

[0070] FIG. 42 is a diagram showing an example of a syntax structure of a patch data unit according to embodiments.

[0071] FIG. 43 is a diagram showing an example of a syntax structure of an inter-patch data unit according to embodiments.

[0072] FIG. 44 is a diagram showing an example of NAL unit header semantics according to embodiments.

[0073] FIG. 45 is a diagram showing an example of the syntax structure of atlas_frame_rbsp() according to embodiments.

[0074] Figure 46 is a flowchart showing an example of a transmission method according to embodiments.

[0075] Figure 47 is a flowchart showing an example of a receiving method according to embodiments.

[0076] Preferred embodiments of the embodiments are described in detail, examples of which are illustrated in the accompanying drawings. The following detailed description, with reference to the accompanying drawings, is intended to illustrate preferred embodiments of the embodiments, rather than merely show embodiments that can be implemented according to the embodiments. The following detailed description includes details to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may be practiced without these details.

[0077] While most of the terms used in the examples are commonly used in the field, some terms were arbitrarily selected by the applicant, and their meanings are described in detail in the following descriptions as needed. Therefore, the examples should be understood based on the intended meaning of the terms, not simply their names or meanings.

[0078] Recently, with the development of 3D data modeling and rendering technology, research on creating and processing 3D data is being conducted in various fields such as Virtual Reality (VR), Augmented Reality (AR), autonomous driving, Computer-Aided Design (CAD) / Computer-Aided Manufacturing (CAM), and Geographic Information System (GIS). 3D data can be represented as a point cloud, mesh, etc., depending on the representation format. Among these, a mesh is composed of geometric information expressing the coordinate values ​​of each vertex (or point), connection information indicating the connection relationship between vertices, a texture map expressing the color information of the mesh surface as 2D image data, and texture coordinates indicating mapping information between the surface of the mesh and the texture map. In the present disclosure, a mesh is defined as a dynamic mesh if one or more of the elements constituting the mesh change over time, and a static mesh if they do not change.

[0079] Because dynamic mesh data has a large amount of data for elements that constitute the mesh compared to two-dimensional image data, technologies have been developed to efficiently compress this large amount of mesh data to store and transmit it.

[0080] FIG. 1 illustrates a system for providing dynamic mesh content according to embodiments.

[0081] The system of FIG. 1 includes a transmitting device (100) and a receiving device (110) according to embodiments. The transmitting device (100) may include a mesh video acquisition unit (101), a mesh video encoder (102), a file / segment encapsulator (103), and a transmitter (104). The receiving device (110) may include a receiving unit (111), a file / segment decapsulator (112), a mesh video decoder (113), and a renderer (114). Each component of FIG. 1 may correspond to hardware, software, a processor, and / or a combination thereof. Hereinafter, the mesh data transmitting device according to embodiments may be interpreted as a term referring to a 3D data transmitting device or transmitting device (100), or a mesh video encoder (hereinafter, referred to as an encoder) (102). The mesh data receiving device according to the embodiments may be interpreted as a term referring to a 3D data receiving device or receiving device (110), or a mesh video decoder (hereinafter, decoder) (113).

[0082] The system of FIG. 1 can perform video-based dynamic mesh compression and decompression.

[0083] Advances in 3D capture, modeling, and rendering have enabled users to consume diverse forms of 3D content, such as AR, XR, metaverse, and holograms, across multiple platforms and devices. 3D content increasingly represents objects with greater precision and realism, enabling users to enjoy immersive experiences. To achieve this, the creation and use of 3D models requires a significant amount of data. Among various types of 3D content, 3D meshes are widely used for efficient data utilization and realistic object representation. Embodiments include a series of processing steps in a system that utilizes such mesh content.

[0084] First, the method of compressing dynamic mesh data starts from the V-PCC (Video-based point cloud compression) standard technology for point cloud data. Point cloud data is data that has color information at the coordinates (X, Y, Z) of a vertex (or point). In the present disclosure, the coordinates (i.e., position information) of a vertex are referred to as geometry information, the color information of a vertex is referred to as attribute information, and the geometry information and attribute information are referred to as vertex information or point cloud data. The vertex information to which connectivity information between vertices is added is referred to as mesh data. When creating content, it can be created in the form of mesh data from the beginning. Alternatively, it can be used by converting it into mesh data by adding connectivity information to point cloud data.

[0085] Currently, the MPEG standards body defines two types of dynamic mesh data: Category 1: Mesh data with texture maps as color information. Category 2: Mesh data with vertex colors as color information.

[0086] Mesh coding standards for Category 1 data are currently under development, and work on Category 2 data standards is also planned for the future. The overall process for providing mesh content services may include acquisition, encoding, transmission, decoding, rendering, and / or feedback, as shown in Figure 1.

[0087] To provide mesh content services, 3D data acquired through multiple cameras or specialized cameras can be processed into mesh data types through a series of processes and then converted into video. The generated mesh video is then transmitted through a series of processes, and the receiving end can then reprocess the received data into mesh video and render it. This allows mesh video to be presented to users, who can then interact with the mesh content according to their intended intent.

[0088] A mesh compression system may include a transmitting device (100) and a receiving device (110) as shown in FIG. 1. The transmitting device (100) may encode mesh video to output a bitstream, and transmit the bitstream to the receiving device (110) in the form of a file or streaming (streaming segment) via a digital storage medium or a network. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD.

[0089] In the above transmitting device (100), the encoder may be called a mesh video / video / picture / frame encoding device, and in the receiving device (110), the decoder may be called a mesh video / video / picture / frame decoding device. The transmitter may be included in a mesh video encoder. The receiver may be included in a mesh video decoder. The renderer (114) may include a display unit, and the renderer and / or the display unit may be configured as separate devices or external components. The transmitting device (100) and the receiving device (110) may further include separate internal or external modules / units / components for a feedback process.

[0090] Mesh data represents the surface of an object as a number of polygons. Each polygon is defined by vertices in 3D space and connection information that describes how the vertices are connected. It can also contain vertex attributes such as vertex color and normal. Mapping information that allows the surface of the mesh to be mapped to a 2D planar area can also be included in the attributes of the mesh. The mapping can be described as a set of parameter coordinates, generally called UV coordinates or texture coordinates, associated with the mesh vertices. Meshes contain 2D attribute maps, which can be used to store high-resolution attribute information such as textures, normals, and displacement. Here, displacement can be used interchangeably with displacement, displacement information, or displacement vectors (i.e., displacement vectors).

[0091] The mesh video acquisition unit (101) may include processing 3D object data acquired through a camera, etc. into a mesh data type having the attributes described above through a series of processes and generating a video composed of such mesh data. The mesh video may have attributes of the mesh, such as vertices, polygons, connection information between vertices, colors, normals, etc., that may change over time. A mesh video having attributes and connection information that change over time in this way may be expressed as a dynamic mesh video.

[0092] A mesh video encoder (102) can encode an input mesh video into one or more video streams. One video can include multiple frames, and one frame can correspond to a still image / picture. In this document, a mesh video can include a mesh image / frame / picture, and a mesh video can be used interchangeably with a mesh image / frame / picture. The mesh video encoder (102) can perform a Video-based Dynamic Mesh (V-Mesh) Compression procedure. The mesh video encoder (102) can perform a series of procedures such as prediction, transformation, quantization, and entropy coding for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0093] The file / segment encapsulator (103) can encapsulate encoded mesh video data and / or mesh video-related metadata in the form of a file, etc. Here, the mesh video-related metadata may be received from a metadata processing unit, etc. The metadata processing unit may be included in the mesh video encoder (102) or may be configured as a separate component / module. The file / segment encapsulator (103) can encapsulate the corresponding data in a file format such as ISOBMFF, or process it in the form of other DASH segments, etc. The file / segment encapsulator (103) may include mesh video-related metadata in the file format according to an embodiment. The mesh video metadata may be included in boxes at various levels in the ISOBMFF file format, for example, or may be included as data in a separate track within the file. Depending on the embodiment, the file / segment encapsulator (103) may encapsulate the mesh video related metadata itself into a file.

[0094] The transmission processing unit can process encapsulated mesh video data for transmission according to the file format. The transmission processing unit can be included in the transmission unit (104) or can be configured as a separate component / module. The transmission processing unit can process mesh video data according to any transmission protocol. The processing for transmission can include processing for transmission through a broadcast network or processing for transmission through broadband. According to an embodiment, the transmission processing unit can receive not only mesh video data but also mesh video-related metadata from the metadata processing unit and process it for transmission.

[0095] The transmission unit (104) can transmit encoded video / image information or data output in the form of a bitstream to the reception unit (111) of the reception device (110) via a digital storage medium or network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmission unit (104) can include an element for generating a media file through a predetermined file format and can include an element for transmission via a broadcasting / communication network. The reception unit (111) can extract the bitstream and transmit it to a decoding device.

[0096] The receiving unit (111) can receive mesh video data transmitted by a mesh data transmission device. Depending on the channel through which it is transmitted, the receiving unit (111) can receive mesh video data through a broadcast network, through a broadband, or through a digital storage medium.

[0097] The receiving processing unit can perform processing according to the transmission protocol on the received mesh video data. The receiving processing unit can be included in the receiving unit (111) or can be configured as a separate component / module. In order to correspond to the processing performed for transmission on the transmitting side, the receiving processing unit can perform the reverse process of the aforementioned transmission processing unit. The receiving processing unit can transfer the acquired mesh video data to the file / segment decapsulator (112) and transfer the acquired mesh video-related metadata to the metadata parser. The mesh video-related metadata acquired by the receiving processing unit can be in the form of a signaling table.

[0098] The file / segment decapsulator (112) can decapsulate mesh video data in the form of a file received from a receiving processing unit. The file / segment decapsulator (112) can decapsulate files according to ISOBMFF, etc., to obtain a mesh video bitstream or mesh video-related metadata (metadata bitstream). The obtained mesh video bitstream can be transmitted to the mesh video decoder (113), and the obtained mesh video-related metadata (metadata bitstream) can be transmitted to the metadata processing unit. The mesh video bitstream may include metadata (metadata bitstream). The metadata processing unit may be included in the mesh video decoder (113) or may be configured as a separate component / module. The mesh video-related metadata obtained by the file / segment decapsulator (112) may be in the form of a box or track within a file format. The file / segment decapsulator (112) may receive metadata required for decapsulation from the metadata processing unit, if necessary. The mesh video related metadata may be passed to the mesh video decoder (113) and used in the mesh video decoding procedure, or may be passed to the renderer (114) and used in the mesh video rendering procedure.

[0099] The mesh video decoder (113) can receive a bitstream and perform a reverse operation corresponding to the operation of the mesh video encoder (102) to decode the video / image. The decoded mesh video / image can be displayed through the display unit of the renderer (114). The user can view all or part of the rendered result through a VR / AR display or a general display.

[0100] The feedback process may include a process of transmitting various feedback information that may be acquired during the rendering / display process to the transmitter or to the decoder on the receiver. Interactivity may be provided in mesh video consumption through the feedback process. Depending on the embodiment, head orientation information, viewport information indicating the area that the user is currently viewing, etc. may be transmitted during the feedback process. Depending on the embodiment, the user may interact with things implemented in the VR / AR / MR / autonomous driving environment, in which case information related to the interaction may be transmitted to the transmitter or the service provider during the feedback process. Depending on the embodiment, the feedback process may not be performed.

[0101] Head orientation information can refer to information about the user's head position, angle, and movement. Based on this information, information about the area the user is currently viewing within the mesh video, i.e. viewport information, can be calculated.

[0102] Viewport information can be information about the area the user is currently viewing in the mesh video. This can be used to perform gaze analysis to determine how the user consumes the mesh video, which area of ​​the mesh video they are gazing at, and for how long. Gaze analysis can be performed on the receiving side and transmitted to the transmitting side through a feedback channel. Devices such as VR / AR / MR displays can extract the viewport area based on the user's head position / orientation, the vertical or horizontal FOV supported by the device, etc.

[0103] Depending on the embodiment, the aforementioned feedback information may not only be transmitted to the transmitter but may also be consumed by the receiver. That is, the aforementioned feedback information may be utilized to perform decoding, rendering, and other processes on the receiver. For example, head orientation information and / or viewport information may be utilized to preferentially decode and render only the mesh video for the area currently being viewed by the user.

[0104] This document relates to embodiments of dynamic mesh video compression as described above. The method / embodiment disclosed in this document can be applied to the Video-based Dynamic Mesh Compression (V-Mesh) standard of the Moving Picture Experts Group (MPEG) or the next-generation video / image coding standard. Dynamic mesh video compression is a method for processing mesh connection information and attributes that change over time, and it can perform lossy and lossless compression for various applications such as real-time communication, storage, free-viewpoint video, and AR / VR.

[0105] The dynamic mesh video compression method described below is based on MPEG's V-Mesh method.

[0106] In this document, picture / frame can generally mean a unit representing one video of a specific time period.

[0107] A pixel or pel can refer to the smallest unit that constitutes a picture (or image). Additionally, the term "sample" can be used as a counterpart to a pixel. A sample can generally represent a pixel or a pixel value, and can represent only the pixel / pixel value of the luma component, only the pixel / pixel value of the chroma component, or only the pixel / pixel value of the depth component.

[0108] A unit may represent a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may include a set (or array) of samples (or sample array) or transform coefficients consisting of M columns and N rows.

[0109] As described above, the encoding process of Fig. 1 is as follows.

[0110] That is, the video-based dynamic mesh compression (V-Mesh) compression method can provide a method of compressing dynamic mesh video data based on 2D video codecs such as HEVC (High Efficiency Video Coding) and VVC (Versatile Video Coding). The V-Mesh compression process receives the following data as input and performs compression.

[0111] Input mesh: Contains the 3D coordinates of the vertices that make up the mesh, normal information for each vertex, mapping information that maps the mesh surface to a 2D plane, and connection information between the vertices that make up the surface. The mesh surface can be expressed as triangles or more polygons, and connection information between the vertices that make up each surface is stored according to a set shape. The input mesh can be saved in the OBJ file format.

[0112] Attribute map: (Hereinafter, texture map is also used in the same meaning): Contains information about the attributes of the mesh (color, normal, displacement, etc.), and stores data in the form of a mapping of the surface of the mesh onto a 2D image. Mapping which part (surface or vertex) of the mesh each data of this attribute map corresponds to is based on the mapping information contained in the input mesh. Since the attribute map has data for each frame of the mesh video, it can also be expressed as an attribute map video. The attribute map in the V-Mesh compression method mainly contains the color information of the mesh, and is saved in an image file format (PNG, BMP, etc.).

[0113] Material Library File: Contains information about the material attributes used in a mesh, and in particular, information that links the input mesh to its corresponding attribute map. It is saved in the Wavefront Material Template Library (MTL) file format.

[0114] In the V-Mesh compression method, the following data and information can be generated through the compression process.

[0115] Base mesh: The input mesh is simplified (decimated) through a pre-processing process, thereby expressing the objects of the input mesh using the minimum number of vertices determined by the user's standards.

[0116] Displacement: Displacement information used to express the input mesh as similarly as possible to the base mesh, and is expressed in the form of 3D coordinates.

[0117] Atlas information: This is the metadata required to reconstruct a mesh using base mesh, displacement, and attribute map information. It can be created and utilized as sub-mesh units (such as patches) that make up the mesh.

[0118] Referring to FIGS. 2 to 7, a method for encoding mesh position information (or vertex position information) is described, and referring to FIGS. 6 to 10, etc., a method for encoding attribute information (attribute map) by restoring mesh position information is described.

[0119] Figure 2 illustrates a V-MESH compression method according to embodiments.

[0120] Fig. 2 illustrates the encoding process of Fig. 1, and the encoding process may include a pre-processing process and an encoding process. The mesh video encoder (102) of Fig. 1 may include a pre-processor (200) and an encoder (201) as in Fig. 2. In addition, the transmitting device of Fig. 1 may be broadly referred to as an encoder, and the mesh video encoder (102) of Fig. 1 may be referred to as an encoder. The V-Mesh compression method may include a pre-processing process (Pre-processing, 200) and an encoding process (Encoding, 201) as in Fig. 2. The pre-processor (200) of Fig. 2 may be located in front of the encoder (201) of Fig. 2. The pre-processor (200) and the encoder (201) of Fig. 2 may be referred to as a single encoder.

[0121] The pre-processor (200) can receive a static of a dynamic mesh (M(i)) and / or an attribute map (A(i)). The pre-processor (200) can generate a base mesh (m(i)) and / or a displacement (or displacement) (d(i)) through pre-processing. The pre-processor (200) can receive feedback information from the encoder (201) and generate the base mesh and / or the displacement based on the feedback information.

[0122] The encoder (201) can receive a base mesh (m(i)), a displacement (d(i)), a static of a dynamic mesh (M(i)), and / or an attribute map (A(i)). In the present disclosure, at least one of the base mesh (m(i)), the displacement (d(i)), the static of a dynamic mesh (M(i)), and / or the attribute map (A(i)) can be referred to as mesh-related data. The encoder (201) can encode the mesh-related data to generate a compressed bitstream.

[0123] Figure 3 illustrates a pre-processing process of V-MESH compression according to embodiments.

[0124] Fig. 3 illustrates the configuration and operation of the preprocessor of Fig. 2. In Fig. 3, the input mesh may include a static of a dynamic mesh (M(i)) and / or an attribute map (A(i)). In addition, the input mesh may include three-dimensional coordinates of vertices constituting the mesh, normal information of each vertex, mapping information for mapping the mesh surface to a 2D plane, connection information between vertices constituting the surface, etc.

[0125] Fig. 3 shows a process of performing pre-processing on an input mesh. The pre-processing process (200) may largely include four steps: 1) GoF (Group of Frame) generation, 2) Mesh Decimation, 3) UV parameterization, and 4) Fitting subdivision surface (300). According to embodiments, GoF generation may be referred to as a GoF generation process or a GoF generation unit, mesh simplification may be referred to as a mesh simplification process or a mesh simplification unit, UV parameterization may be referred to as a UV parameterization process or a UV parameterization unit, and the fitting subdivision surface may be referred to as a fitting subdivision surface process or a fitting subdivision surface unit. The pre-processor (200) can generate displacement and / or base meshes from the received input mesh and transmit them to the encoder (201). The pre-processor (200) can transmit GoF information associated with GoF generation to the encoder (201).

[0126] Below, each step of Fig. 3 is described.

[0127] GoF Generation: This is the process of generating a reference structure for mesh data. If the number of vertices, the number of texture coordinates, the vertex connection information, and the texture coordinate connection information of the mesh of the previous frame and the current mesh are all the same, the previous frame can be set as the reference frame. That is, if only the vertex coordinate values ​​are different between the current input mesh and the reference input mesh, the encoder (201) can perform inter frame encoding. Otherwise, intra frame encoding is performed for the corresponding frame.

[0128] Mesh Decimation: This process simplifies the input mesh to create a simplified mesh, or base mesh. Vertices to be removed from the original mesh are selected based on user-defined criteria, and the selected vertices and the triangles connected to them can be removed.

[0129] In the process of performing mesh simplification (Mesh decimation), the input mesh (voxelized), target triangle ratio (TTR), and minimum triangle component (CCCount) information are passed as input, and the simplified mesh (decimated mesh) can be obtained as output. In this process, connected triangle components smaller than the set minimum triangle component (CCCount) can be removed.

[0130] UV parameterization: This is the process of mapping a 3D surface of a decimated mesh into a texture domain. Parameterization can be performed using the UVAtlas tool. This process generates mapping information, which indicates where each vertex of the decimated mesh can be mapped to on a 2D image. This mapping information is expressed and stored as texture coordinates, and through this process, the final base mesh is created.

[0131] Fitting subdivision surface (300): This is a process of performing subdivision on a decimated mesh (i.e., a simplified mesh having texture coordinates). The displacement and base mesh generated through this process are output to the encoder (201). A user-defined method, such as a mid-edge method, may be applied as the subdivision method. A fitting process is performed so that the input mesh and the mesh on which the subdivision has been performed become similar to each other. In the present disclosure, the mesh on which the fitting process has been performed is referred to as a fitted subdivision mesh (or fitted subdivision mesh).

[0132] Figure 4 illustrates a mid-edge subdivision method according to embodiments.

[0133] Figure 4 illustrates the mid-edge method of the fitting subdivision surface described in Figure 3. Referring to Figure 4, an original mesh containing four vertices is subdivided to generate a sub-mesh. A sub-mesh can be generated by creating a new vertex in the middle of the edge between the vertices. Then, a fitting process is performed so that the input mesh and the sub-mesh become similar to each other, thereby generating a fitted sub-division mesh.

[0134] When a fitted subdivided mesh (hereinafter referred to as a fitted subdivided mesh) is generated, displacement is calculated using this result and a pre-compressed and decoded base mesh (hereinafter referred to as a reconstructed base mesh). That is, the reconstructed base mesh is subdivided in the same way as the fitting subdivision surface. The difference in position of each vertex between this result and the fitted subdivided mesh is the displacement for each vertex. Since displacement represents the position difference in three-dimensional space, it is also expressed as a value in the (x, y, z) space of a Cartesian coordinate system. Depending on the user input parameters, the (x, y, z) coordinate values ​​can be converted to (normal, tangential, bi-tangential) coordinate values ​​of the local coordinate system.

[0135] Fig. 5 illustrates a displacement generation process according to embodiments. The displacement generation process of Fig. 5 may be performed in a pre-processor (200) or in an encoder (201).

[0136] FIG. 5 illustrates in detail the displacement calculation method of the fitting subdivision surface (300) as described in FIG. 4.

[0137] An encoder and / or pre-processor according to embodiments may include 1) a subdivision unit, 2) a local coordinate system calculation unit, and 3) a displacement calculation unit. The subdivision unit may perform subdivision on a reconstructed base mesh to generate a subdivided reconstructed base mesh. Here, the restoration of the base mesh may be performed in the pre-processor (200) or in the encoder (201). The local coordinate system calculation unit may receive a fitted subdivision mesh and a subdivided reconstructed base mesh, and may convert a coordinate system of the mesh into a local coordinate system based on the fitted subdivision mesh and the subdivided reconstructed base mesh. The local coordinate system calculation operation may be optional. The displacement calculation unit may calculate a positional difference between the fitted subdivision mesh and the subdivided reconstructed base mesh. For example, a positional difference value between vertices of two input meshes may be generated. The vertex positional difference value becomes a displacement.

[0138] The mesh data transmission method and device according to the embodiments can encode mesh data as follows. Mesh data is a term including point cloud data. Point cloud data (which may be referred to as point cloud for short) according to the embodiments can refer to data including vertex coordinates (or geometry information) and color information (or attribute information). In addition, geometry images, attribute images, occupancy maps, and additional information (or patch information) generated through patch generation and packing based on vertex coordinates and color information are also referred to as point cloud data. Therefore, point cloud data including connection information can be referred to as mesh data. In this document, point cloud and mesh data can be used interchangeably.

[0139] The V-Mesh compression (reconstruction) method according to the embodiments may include intra frame encoding (Fig. 6) and inter frame encoding (Fig. 7).

[0140] Based on the results of the GoF generation described above, intra-frame encoding or inter-frame encoding is performed. In the case of intra-encoding, the data to be compressed may be a base mesh, displacement, attribute map, etc. In the case of inter-encoding, the data to be compressed may be a displacement, attribute map, and a motion field between a reference base mesh and the current base mesh.

[0141] Fig. 6 illustrates an intra-frame encoding process of a V-MESH compression method according to embodiments. Each component for the intra-frame encoding process of Fig. 6 corresponds to hardware, software, a processor, and / or a combination thereof.

[0142] The encoding process of FIG. 6 details the encoding of the mesh video encoder (102) of FIG. 1. That is, it shows the configuration of the mesh video encoder (102) when the encoding of FIG. 1 is an intra-frame method. The encoder of FIG. 6 may include a pre-processor (200) and / or an encoder (201). The pre-processor (200) and encoder (201) of FIG. 6 may correspond to the pre-processor (200) and encoder (201) of FIG. 3.

[0143] The preprocessor (200) can receive an input mesh and perform the preprocessing described above. The preprocessing can generate a base mesh and / or a fitted subdivision mesh.

[0144] The quantizer (411) of the encoder (201) can quantize the base mesh and / or the fitted subdivided mesh. The static mesh encoder (412) can encode the static mesh (i.e., the quantized base mesh) and generate a bitstream (i.e., a compressed base mesh bitstream) including the encoded base mesh. The static mesh decoder (413) can decode the encoded static mesh (i.e., the encoded base mesh). The inverse quantizer (414) can inversely quantize the quantized static mesh (i.e., the base mesh) to output a reconstructed (or restored) base mesh. The displacement calculation unit (415) can generate displacements (or displacements) based on the reconstructed static mesh (i.e., the base mesh) and the fitted subdivided mesh. According to embodiments, the displacement calculation unit (415) calculates displacement, which is the position difference between each vertex of the subdivided base mesh and the fitted subdivided mesh after subdividing (or refining) the restored base mesh. In other words, the displacement is a displacement vector, which is the position difference between the vertices of the two meshes so that the fitted subdivided (or refining) mesh becomes similar to the original mesh. The forward linear lifting unit (416) can perform lifting transformation on the input displacement to generate lifting coefficients (or transform coefficients). The quantizer (417) can quantize the lifting coefficients. The image packing unit (418) can pack an image based on the quantized lifting coefficients. The video encoder (419) can encode the packed image. That is, the quantized lifting coefficients are packed into one frame as a 2D image by the image packing unit (418), compressed through the video encoder (419), and output as a displacement bitstream (i.e., compressed displacement bitstream).

[0145] A video decoder (420) decodes a compressed displacement bitstream. An image unpacking unit (421) can perform unpacking on the decoded displacement frame to output quantized lifting coefficients. A dequantizer (422) can dequantize the quantized lifting coefficients. An inverse linear lifting unit (423) applies inverse lifting to the inverse quantized lifting coefficients to generate restored displacement. A mesh restoration unit (424) reconstructs and deforms a mesh using the restored displacement output from the inverse linear lifting unit (423) and the restored base mesh (or subdivided restored base mesh) output from the inverse quantization unit (414). The present disclosure refers to the reconstructed and deformed mesh as a restored deformed mesh.

[0146] The attribute transfer (425) receives an input mesh and / or an input attribute map, and regenerates an attribute map based on the restored deformed mesh. The attribute map refers to a texture map corresponding to attribute information among mesh data components, and in the present disclosure, the attribute map and the texture map may be used interchangeably. The push-pull padding (426) may pad data in the attribute map based on the push-pull method. The color space conversion unit (427) may convert the space of the color component of the attribute map. For example, the attribute map may be converted from an RGB color space to a YUV color space. The video encoder (428) may encode the attribute map and output it as a compressed attribute bitstream.

[0147] A multiplexer (430) can generate a compressed bitstream by multiplexing a compressed base mesh bitstream, a compressed displacement bitstream, and a compressed attribute bitstream.

[0148] In Fig. 6, the displacement calculation unit (415) may be included in the pre-processor (200). In addition, at least one of the quantizer (411), the static mesh encoder (412), the static mesh decoder (413), and the inverse quantizer (414) may be included in the pre-processor (200).

[0149] As described in FIG. 6, the intra-frame encoding method includes base mesh encoding (also called static mesh encoding). That is, when performing intra-frame encoding on the current input mesh frame, the base mesh generated in the pre-processing process of the pre-processor (200) can be encoded using a static mesh compression technology in a static mesh encoder (412) after undergoing a quantization process in a quantizer (411). In the V-Mesh compression method, for example, Draco technology is applied to base mesh encoding, and vertex position information, mapping information (texture coordinates), vertex connection information, etc. of the base mesh become compression targets.

[0150] The encoder of Fig. 6 generates a bitstream by compressing the base mesh, displacement, and attributes within the frame, and the encoder of Fig. 7 generates a bitstream by compressing the motion, displacement, and attributes between the current frame and the reference frame.

[0151] Fig. 7 illustrates an inter-frame encoding process of a V-MESH compression method according to embodiments. Each component for the inter-frame encoding process of Fig. 7 corresponds to hardware, software, a processor, and / or a combination thereof.

[0152] The encoding process of Fig. 7 details the encoding of Fig. 1. That is, it shows the configuration of an encoder when the encoding of Fig. 1 is an inter-frame method. The encoder of Fig. 7 may include a pre-processor (200) and / or an encoder (201). The pre-processor (200) and encoder (201) of Fig. 7 may correspond to the pre-processor (200) and encoder (201) of Fig. 3.

[0153] For a description of the components corresponding to the encoding operation of FIG. 6 among the encoding operations of FIG. 7, refer to the description of FIG. 6. That is, the operation of the quantizer (511), displacement calculation unit (515), wavelet transformer (516), quantizer (517), image packing unit (518), video encoder (519), video decoder (520), image unpacking unit (521), inverse quantizer (522), inverse wavelet transformer (523), mesh restoration unit (524), attribute transfer (525), push-pull padding (526), ​​color space conversion unit (527), video encoder (528), and multiplexer (530) of FIG. 7 is similar to that of the quantizer (411), static mesh encoder (412), static mesh decoder (413), inverse quantizer (414), displacement calculation unit (415), forward linear lifting unit (416), quantizer (417), image Since the operations described in the packing unit (418), video encoder (419), video decoder (420), image unpacking unit (421), inverse quantizer (422), inverse linear lifting unit (423), mesh restoration unit (424), attribute transfer (425), push-pull padding (426), color space conversion unit (427), video encoder (428), and multiplexer (430) are the same or similar, a detailed description thereof is omitted in FIG. 7 to avoid redundant description.

[0154] In Fig. 7, for inter-frame based encoding, the motion encoder (512) can obtain a motion vector between the two base meshes based on the restored quantized reference base mesh and the quantized current base mesh, and then encode the motion vector to output a compressed motion bitstream. The motion encoder (512) can be referred to as a motion vector encoder. The base mesh restoration unit (513) can restore the base mesh based on the restored quantized reference base mesh and the encoded motion vector. The restored base mesh is dequantized in the dequantizer (514) and then output to the displacement calculation unit (515).

[0155] In Fig. 7, the displacement calculation unit (515) may be included in the pre-processor (200). In addition, at least one of the quantizer (511), the motion encoder (512), the base mesh restoration unit (513), and the inverse quantizer (514) may be included in the pre-processor (200).

[0156] As described in Fig. 7, the inter-frame encoding method may include motion field encoding (also called motion vector encoding). Inter-frame encoding may be performed when a one-to-one correspondence of vertices is established between a reference mesh and a current input mesh, and only the position information of the vertices is different. When performing inter-frame encoding, instead of compressing the base mesh, the difference between the vertices of the reference base mesh and the current base mesh, i.e., the motion field (also called motion vector), may be calculated and encoded to encode this information. The reference base mesh is the result of quantizing the already decoded base mesh data and is determined according to the reference frame index determined in the GoF generation. The motion field may also be encoded as a value. Alternatively, the predicted motion field can be calculated by averaging the motion fields of the restored vertices among the vertices connected to the current vertex, and the residual motion field, which is the difference between the predicted motion field value and the motion field value of the current vertex, can be encoded. This residual motion field value can be encoded using entropy coding.The process of encoding displacement and attribute maps, excluding the motion field encoding process of inter frame encoding, is the same as the structure of the intra frame encoding method except for the base mesh encoding.

[0157] Figure 8 illustrates a lifting conversion process for displacement according to embodiments.

[0158] Figure 9 illustrates a process of packing transformation coefficients (or lifting coefficients) according to embodiments into a 2D image.

[0159] Figures 8 and 9 illustrate the process of transforming displacement and packing transform coefficients of the encoding process of Figures 6 and 7, respectively.

[0160] The encoding method according to the embodiments includes displacement encoding.

[0161] After base mesh encoding and / or motion field encoding, a reconstructed base mesh is generated through restoration and dequantization, and the displacement between the result of performing subdivision on the reconstructed base mesh and the fitted subdivided mesh generated through the fitting subdivision surface can be calculated (see 415 in FIG. 6 or 515 in FIG. 7). For effective encoding, a data transform process such as wavelet transform can be applied to the displacement information (see 416 in FIG. 6 or 516 in FIG. 7).

[0162] FIG. 8 shows a process of transforming displacement information using a lifting transform in the forward linear lifting unit (416) of FIG. 6 or the wavelet transformer (516) of FIG. 7. For example, a linear wavelet-based lifting transform may be performed. The transform coefficients generated through the transform process are quantized in a quantizer (417 or 517) and then packed into a 2D image through an image packing unit (418 or 518) as in FIG. 9. The transform coefficients are configured as one block for every 256 (= 16×16) units, and each block can be packed in a z-scan order. The horizontal number of blocks is fixed to 16, but the vertical number of blocks can be determined according to the number of vertices of the subdivided base mesh. Transform coefficients can be packed by aligning them with Morton codes within a single block. The packed images generate displacement videos for each GoF unit, and these displacement videos can be encoded using a conventional video compression codec in a video encoder (419 or 519).

[0163] Referring to FIG. 8, the base mesh (original) may include vertices and edges for LoD0. A first subdivision mesh generated by dividing (or subdividing) the base mesh includes vertices generated by further dividing (or subdividing) edges of the base mesh. The first subdivision mesh includes vertices for LoD0 and vertices for LoD1. LoD1 includes the subdivided vertices and the vertices of the base mesh (LoD0). The first subdivision mesh may be further divided (or subdivided) to generate a second subdivision mesh. The second subdivision mesh includes LoD2. LoD2 includes base mesh vertices (LoD0), LoD1 including vertices further divided (or subdivided) from LoD0, and vertices further divided (or subdivided) from LoD1. LoD is a level of detail that indicates the degree of detail of mesh data content. As the level index increases, the distance between vertices becomes closer and the level of detail increases. In other words, the smaller the LoD value, the lower the detail of the mesh data content, and the larger the LoD value, the higher the detail of the mesh data content. LoD N contains the vertices included in the previous LoDN-1 as is. When a mesh (or vertex) is further divided through subdivision, the mesh can be encoded based on a prediction and / or update method by considering the previous vertices v1, v2 and the subdivided vertex v. Instead of directly encoding information about the current LoD N, a residual value between the previous LoD N-1 can be generated and the mesh can be encoded using the residual value to reduce the size of the bitstream. The prediction process means the operation of predicting the current vertex v using the previous vertices v1 and v2. Since adjacent subdivision meshes have similar data, this property can be utilized for efficient encoding.Current vertex position information is predicted as a residual of previous vertex position information, and the previous vertex position information is updated through the residual. In the present disclosure, vertex, apex, and point may be used with the same meaning. In addition, LoDs may be defined during the subdivision process of the base mesh. According to embodiments, the subdivision process of the base mesh may be performed in the pre-processor (200) or in a separate component / module.

[0164] Referring to FIG. 9, a vertex has a transform coefficient (also called a lifting coefficient) generated through a lifting transformation. The transform coefficient of a vertex related to a lifting transformation can be packed into an image by an image packing unit (418 or 518) and then encoded by a video encoder (419 or 519).

[0165] Figure 10 illustrates an attribute transfer process of a V-MESH compression method according to embodiments.

[0166] According to the embodiments, FIG. 10 shows the detailed operation of the attribute transfer (425 or 525) of the encoding of FIG. 6, FIG. 7, etc.

[0167] Encoding according to embodiments includes attribute map encoding. According to embodiments, attribute map encoding may be performed in the video encoder (428) of FIG. 6 or the video encoder (528) of FIG. 7.

[0168] According to embodiments, in the present disclosure, the encoder compresses information about the input mesh through base mesh encoding (i.e., intra encoding), motion field encoding (i.e., inter encoding), and displacement encoding. In the encoding process, the compressed input mesh is restored through base mesh decoding (intra frame), motion field decoding (inter frame), and displacement video decoding processes, and the restored result, the reconstructed deformed mesh (hereinafter referred to as Recon. deformed mesh), is used to compress the input attribute map as shown in FIGS. 6 and 7. The reconstructed deformed mesh (Recon. deformed mesh) has position information of vertices, texture coordinates, and corresponding connection information, but does not have color information corresponding to the texture coordinates. Therefore, as shown in Fig. 10, in the V-Mesh compression method, a new attribute map having color information corresponding to the texture coordinates of the reconstructed deformed mesh is regenerated through the attribute transfer process of attribute transfer (425 or 525).

[0169] According to embodiments, attribute transfer (425 or 525) first checks whether each point P(u, v) of a 2D texture domain belongs to a texture triangle of a reconstructed deformed mesh, and if it exists in a texture triangle T, the barycentric coordinate of P(u, v) according to the triangle T ( , , ) is calculated. And the 3D vertex positions of triangle T and ( , , ) is used to compute the 3D coordinates M(x, y, z) of P(u, v). Find the vertex coordinates M'(x', y', z') and the triangle T' containing this vertex that corresponds to the most similar position to the calculated M(x, y, z) in the input mesh domain. Then, the center of mass coordinates of M'(x', y', z') in this triangle T' ( ', ', ') is calculated. The texture coordinates corresponding to the three vertices of Triangle T' and ( ', ', ') is used to calculate the texture coordinates (u', v'), and the color information corresponding to these coordinates is found in the input attribute map. The color information found in this way is immediately assigned to the pixel location (u, v) of the new attribute map. If P(u, v) does not belong to any triangle, the pixel at that location in the new attribute map can be filled with a color value using a padding algorithm, such as the push-pull algorithm of push-pull padding (426 or 526).

[0170] The new attribute map generated through attribute transfer (425 or 525) is grouped into GoF units to form an attribute map video, which is compressed using the video codec of the video encoder (428 or 528).

[0171] Referring to Figure 10, the reference relationship between the input mesh, the input attribute map, the reconstructed deformed mesh, and the regenerated attribute map can be seen.

[0172] The decoding process of Fig. 1 can perform the reverse process of the corresponding process of the encoding process of Fig. 1. The specific decoding process is as follows.

[0173] FIG. 11 illustrates an intra-frame decoding (or intra-decoding) process of V-Mesh technology according to embodiments.

[0174] Fig. 11 illustrates the configuration and operation of the mesh video decoder (113) of the receiving device of Fig. 1. In addition, Fig. 11 can restore mesh data by performing the reverse process of the intra-frame encoding process of Fig. 6. Each component for the intra-frame decoding process of Fig. 11 corresponds to hardware, software, and / or a combination thereof.

[0175] First, the bitstream (i.e., compressed bitstream) received and input to the demultiplexer (611) of the intra frame decoding unit (610) can be separated into a mesh substream, a displacement substream, an attribute map substream, and a substream containing patch information of the mesh, such as V-PCC / V3C. The term V-PCC (Video-based Point Cloud Compression) used in this document can be used with the same meaning as V3C (Visual Volumetric Video-based Coding), and the two terms can be used interchangeably. Therefore, the term V-PCC in this document can be interpreted as the term V3C.

[0176] According to embodiments, the mesh sub-stream may be input to a static mesh decoder (612) and decoded, the displacement sub-stream may be input to a video decoder (613) and decoded, and the attribute map sub-stream may be input to a video decoder (617) and decoded.

[0177] According to embodiments, the mesh sub-stream is decoded through a decoder (612) of a static mesh codec used in encoding, such as Google Draco, and as a result, a reconstructed quantized base mesh, for example, connection information, vertex geometry information, vertex texture coordinates, etc. of the base mesh can be reconstructed.

[0178] According to embodiments, the displacement sub-stream is decoded into displacement video through a decoder (613) of a video compression codec used in encoding, and is restored as displacement information for each vertex (i.e., Recon. displacements) through an image unpacking process of an image unpacking unit (614), an inverse quantization process of an inverse quantizer (615), and an inverse transform process of an inverse linear lifting unit (616).

[0179] According to embodiments, the base mesh restored by the static mesh decoder (612) is inverse quantized by the inverse quantizer (620) and then output to the mesh restoration unit (630). The mesh restoration unit (630) reconstructs and restores the deformed mesh (i.e., decoded mesh) through the restored displacement output from the inverse linear lifting unit (616) and the restored base mesh output from the inverse quantizer (620). That is, the inverse quantized restored base mesh is combined with the restored displacement information to generate the final decoded mesh. In the present disclosure, the final decoded mesh is referred to as a reconstructed deformed mesh.

[0180] According to embodiments, an attribute map sub-stream is decoded through a decoder (617) corresponding to a video compression codec used in encoding, and then restored to a final attribute map (i.e., decoded attribute map) through a color conversion unit (640) through processes such as color format conversion and color space conversion.

[0181] According to embodiments, the restored decoded mesh and decoded attribute map can be utilized by the receiver as final mesh data that can be utilized by the user.

[0182] Referring to FIG. 11, the received compressed bitstream includes patch information, a mesh substream, a displacement substream, and an attribute map substream. A substream is interpreted as a term referring to a part of a bitstream included in a bitstream. The bitstream includes patch information (data), mesh information (data), displacement information (data), and attribute map information (data).

[0183] As described above, the decoder of FIG. 11 performs the following intra-frame decoding operations. The static mesh decoder (612) decodes the mesh sub-stream to generate a reconstructed quantized base mesh, and the inverse quantizer (620) applies the quantization parameters of the quantizer inversely to generate the reconstructed base mesh. The video decoder (613) decodes the displacement sub-stream, the image unpacking unit (614) unpacks the images of the decoded displacement video, and the inverse quantizer (615) inversely quantizes the quantized images. The inverse linear lifting unit (616) applies a lifting transform in the reverse process of the encoder to generate the reconstructed displacement. The mesh restoration unit (630) generates a reconstructed deformed mesh based on the reconstructed base mesh and the reconstructed displacement. The video decoder (617) decodes the attribute map sub-stream, and the color conversion unit (640) converts the color format and / or space of the decoded attribute map to generate a decoded attribute map.

[0184] Figure 12 illustrates the inter-frame decoding (or inter-decoding) process of V-Mesh technology.

[0185] Fig. 12 illustrates the configuration and operation of the mesh video decoder (113) of the receiving device of Fig. 1. In addition, Fig. 12 can restore mesh data by performing the reverse process of the inter-frame encoding process of Fig. 7. Each component for the inter-frame decoding process of Fig. 12 corresponds to hardware, software, and / or a combination thereof.

[0186] First, the bitstream received and input to the demultiplexer (711) of the intra frame decoding unit (710) can be separated into a motion sub-stream (also called a motion sub-stream or motion vector sub-stream), a displacement sub-stream, an attribute map sub-stream, and a sub-stream including patch information of a mesh such as V3C / V-PCC.

[0187] According to embodiments, a motion sub-stream may be input to a motion decoder (712) and decoded, a displacement sub-stream may be input to a video decoder (713) and decoded, and an attribute map sub-stream may be input to a video decoder (717) and decoded.

[0188] According to embodiments, a motion sub-stream is decoded through entropy decoding and inverse prediction processes in a motion decoder (712) and restored into motion information (or motion vector information). A base mesh restoration unit (718) combines the restored motion information with a reference base mesh that has already been restored and stored to generate a reconstructed quantized base mesh for the current frame. An inverse quantizer (720) applies inverse quantization to the restored quantized base mesh to generate a reconstructed base mesh. A video decoder (713) decodes a displacement sub-stream, an image unpacking unit (714) unpacks an image of the decoded displacement video, and an inverse quantizer (715) inversely quantizes a quantized image. The reverse linear lifting unit (716) applies a lifting transformation in the reverse process of the encoder to generate a restored displacement. The mesh restoration unit (730) generates a reconstructed deformed mesh, i.e., a final decoded mesh, based on the restored base mesh and the restored displacement.

[0189] According to embodiments, the video decoder (717) decodes the attribute map sub-stream in the same manner as intra decoding, and the color conversion unit (740) converts the color format and / or space of the decoded attribute map to generate a decoded attribute map. The decoded mesh and the decoded attribute map can be utilized by the receiver as final mesh data that can be utilized by the user.

[0190] Referring to Fig. 12, the bitstream includes motion information (also called motion vectors), displacement, and an attribute map. Since Fig. 12 performs inter-frame decoding, it further includes a process of decoding inter-frame motion information. The motion information is decoded, and a restored quantized base mesh for the motion information is generated based on the reference base mesh, thereby generating a restored base mesh. For a description of the operation of Fig. 12, which is identical to that of Fig. 11, refer to the description of Fig. 11.

[0191] Fig. 13 illustrates a mesh data transmission device according to embodiments.

[0192] FIG. 13 corresponds to the transmitting device (100) or mesh video encoder (102) of FIG. 1, the encoder (preprocessor and encoder) of FIG. 2, FIG. 6, or FIG. 7, and / or a transmitting encoding device corresponding thereto. Each component of FIG. 13 corresponds to hardware, software, a processor, and / or a combination thereof.

[0193] The operation process of a transmitter for compressing and transmitting dynamic mesh data using V-Mesh compression technology may be as shown in Fig. 13. The transmitter of Fig. 13 may perform an intra-frame encoding (or intra-encoding or intra-screen encoding) process and / or an inter-frame encoding (or inter-encoding or inter-screen encoding) process.

[0194] The pre-processor (811) receives the original mesh as input and generates a simplified mesh (decimated mesh) (or base mesh) and a fitted decimated mesh (or subdivision). Simplification can be performed based on the target number of vertices or target number of polygons that constitute the mesh. Parameterization, which generates texture coordinates and texture connection information per vertex, can be performed on the simplified mesh. For example, parameterization is a process of mapping a 3D surface to a texture domain for the decimated mesh. If parameterization is performed using the UVAtlas tool, mapping information is generated that can identify where each vertex of the decimated mesh can be mapped on a 2D image. The mapping information is expressed and stored as texture coordinates, and the final base mesh is generated through this process. In addition, the work of quantizing the mesh information in floating-point form into fixed-point form can be performed. This result can be output as a base mesh to a motion vector encoder (813) or a static mesh encoder (814) through a switching unit (812). The pre-processor (811) can perform mesh subdivision on the base mesh to generate additional vertices. Depending on the subdivision method, vertex connection information, texture coordinates, and texture coordinate connection information including the added vertices can be generated. The pre-processor (811) can generate a fitted subdivided mesh by adjusting the vertex positions so that the subdivided mesh becomes similar to the original mesh.

[0195] According to embodiments, the base mesh is output to a motion vector encoder (813) via a switching unit (812) when performing inter-encoding for the corresponding mesh frame, and is output to a static mesh encoder (814) via a switching unit (812) when performing intra-encoding for the corresponding mesh frame. The motion vector encoder (813) may be referred to as a motion encoder.

[0196] For example, when performing intra-encoding (or intra-frame encoding) on ​​the corresponding mesh frame, the base mesh can be compressed through a static mesh encoder (814). In this case, encoding can be performed on connection information, vertex geometry information, vertex texture information, normal information, etc. of the base mesh. The base mesh bitstream generated through encoding is transmitted to a multiplexer (823).

[0197] As another example, when performing inter-encoding (or inter-frame encoding) for the corresponding mesh frame, the motion vector encoder (813) can receive a base mesh and a reference reconstructed base mesh (or a reconstructed quantized reference base mesh) as input, calculate a motion vector between the two meshes, and encode the value. In addition, the motion vector encoder (813) can perform prediction based on connection information using a previously encoded / decoded motion vector as a predictor, and encode a residual motion vector obtained by subtracting the predicted motion vector from the current motion vector. The motion vector bitstream generated through encoding is transmitted to the multiplexer (823).

[0198] The base mesh restoration unit (815) can receive the base mesh encoded by the static mesh encoder (814) or the motion vector encoded by the motion vector encoder (813) and generate a reconstructed base mesh. For example, the base mesh restoration unit (815) can perform static mesh decoding on the base mesh encoded by the static mesh encoder (814) to restore the base mesh. At this time, quantization can be applied before the static mesh decoding, and inverse quantization can be applied after the static mesh decoding. As another example, the base mesh restoration unit (815) can restore the base mesh based on the reconstructed quantized reference base mesh and the motion vector encoded by the motion vector encoder (813). The reconstructed base mesh is output to the displacement calculation unit (816) and the mesh restoration unit (820).

[0199] The displacement calculation unit (816) can perform mesh refinement on the restored base mesh. The displacement calculation unit (816) can calculate a displacement vector, which is a difference value between the vertex positions of the restored base mesh and the fitted subdivision (or refined) mesh generated by the pre-processor (811). At this time, the displacement vector can be calculated as many times as the number of vertices of the refined mesh. The displacement calculation unit (816) can convert the displacement vector calculated in the 3D Cartesian coordinate system into a local coordinate system based on the normal vector of each vertex.

[0200] The displacement vector video generation unit (817) may include a linear lifting unit, a quantizer, and an image packing unit. That is, in the displacement vector video generation unit (817), the linear lifting unit may transform the displacement vector for effective encoding. The transformation may be performed by a lifting transformation, a wavelet transformation, etc., according to embodiments. In addition, quantization may be performed in a quantizer on the transformed displacement vector value, i.e., the transform coefficient. At this time, a different quantization parameter may be applied to each axis of the transform coefficient, and the quantization parameter may be derived according to an encoder / decoder agreement. The transformed and quantized displacement vector information may be packed into a 2D image in the image packing unit. The displacement vector video generation unit (817) may generate a displacement vector video by bundling packed 2D images for each frame, and the displacement vector video may be generated for each GoF (Group of Frame) unit of the input mesh.

[0201] The displacement vector video encoder (818) can encode the generated displacement vector video using a video compression codec. The generated displacement vector video bitstream is transmitted to a multiplexer (823).

[0202] The displacement vector restoration unit (819) may include a video decoder, an image unpacking unit, an inverse quantizer, and an inverse linear lifting unit. That is, the displacement vector restoration unit (819) performs decoding on an encoded displacement vector in the video decoder, performs image unpacking in the image unpacking unit, performs inverse quantization in the inverse quantizer, and then performs inverse transformation in the inverse linear lifting unit to restore the displacement vector. The restored displacement vector is output to the mesh restoration unit (820). The mesh restoration unit (820) restores a deformed mesh based on the base mesh restored by the base mesh restoration unit (815) and the displacement vector restored by the displacement vector restoration unit (819). The restored mesh (or referred to as a restored deformed mesh) has restored vertices, connection information between vertices, texture coordinates, and connection information between texture coordinates.

[0203] The texture map video generation unit (821) can regenerate a texture map based on the texture map (or attribute map) of the original mesh and the restored deformed mesh output from the mesh restoration unit (820). According to embodiments, the texture map video generation unit (821) can assign color information per vertex of the texture map of the original mesh to the texture coordinates of the restored deformed mesh. According to embodiments, the texture map video generation unit (821) can generate a texture map video by grouping the regenerated texture maps by GoF unit for each frame.

[0204] The generated texture map video can be encoded using a video compression codec of the texture map video encoder (822). The texture map video bitstream generated through encoding is transmitted to a multiplexer (823).

[0205] A multiplexer (823) multiplexes a motion vector bitstream (e.g., in case of inter encoding), a base mesh bitstream (e.g., in case of intra encoding), a displacement vector bitstream, and a texture map bitstream into a single bitstream. The single bitstream can be transmitted to a receiver via a transmitter (824). Alternatively, the motion vector bitstream, the base mesh bitstream, the displacement vector bitstream, and the texture map bitstream can be generated as a file with one or more track data or encapsulated into segments and transmitted to a receiver via the transmitter (824).

[0206] Referring to FIG. 13, a transmitting device (encoder) can encode a mesh in an intra-frame or inter-frame manner. A transmitting device according to intra-encoding can generate a base mesh, a displacement vector (or referred to as displacement), and a texture map (or referred to as attribute map). A transmitting device according to inter-encoding can generate a motion vector (or referred to as motion), a displacement vector (or referred to as displacement), and a texture map (or referred to as attribute map). The texture map obtained from the data input unit is generated and encoded based on the restored mesh. Displacement is generated and encoded through the difference in vertex positions between the base mesh and the divided (or subdivided or subdivided) mesh. More specifically, the displacement is the difference in position between the fitted sub-divided mesh and the sub-divided restored base mesh, i.e., the difference in vertex positions between the two meshes. In addition, the base mesh is generated by simplifying and encoding the original mesh through pre-processing. Motion is generated as motion vectors for the mesh of the current frame based on the reference base mesh of the previous frame.

[0207] Fig. 14 illustrates a mesh data receiving device according to embodiments.

[0208] Fig. 14 corresponds to the receiving device (110) or mesh video decoder (113) of Fig. 1, the decoder of Fig. 11 or Fig. 12, and / or the receiving decoding device corresponding thereto. Each component of Fig. 14 corresponds to hardware, software, a processor, and / or a combination thereof. The receiving (decoding) operation of Fig. 14 may follow the reverse process of the corresponding process of the transmitting (encoding) operation of Fig. 13.

[0209] The bitstream of the mesh data received by the receiver (910) is demultiplexed into a compressed motion vector bitstream (e.g., inter decoding) or a base mesh bitstream (e.g., intra decoding), a displacement vector bitstream, and a texture map bitstream after file / segment decapsulation in the demultiplexer (911). For example, if the current mesh has inter-screen encoding (i.e., inter encoding) applied, the motion vector bitstream is received, demultiplexed, and then output to the motion vector decoder (913) via the switching unit (912). As another example, if the current mesh has intra-screen encoding (i.e., intra encoding) applied, the base mesh bitstream is received, demultiplexed, and then output to the static mesh decoder (914) via the switching unit (912). Here, the motion vector decoder (913) may be referred to as a motion decoder.

[0210] According to embodiments, if the current mesh has inter-screen encoding applied according to frame header information, the motion vector decoder (913) can perform decoding on the motion vector bitstream. According to embodiments, the motion vector decoder (913) can reconstruct the final motion vector by adding the previously decoded motion vector as a predictor to the residual motion vector decoded from the bitstream.

[0211] According to embodiments, if the current mesh has been subjected to in-screen encoding according to frame header information, the static mesh decoder (914) can decode the base mesh bitstream to restore connection information, vertex geometry information, texture coordinates, normal information, etc. of the base mesh.

[0212] According to embodiments, the base mesh restoration unit (915) can restore the current base mesh based on the decoded motion vector or the decoded base mesh. For example, if the current mesh has inter-screen encoding applied, the base mesh restoration unit (915) can generate a restored base mesh by adding the decoded motion vector to the reference base mesh and then performing inverse quantization. As another example, if the current mesh has intra-screen encoding applied, the base mesh restoration unit (915) can generate a restored base mesh by performing inverse quantization on the base mesh decoded through the static mesh decoder (914).

[0213] According to embodiments, the displacement vector video decoder (917) can decode the displacement vector bitstream as a video bitstream using a video codec.

[0214] According to embodiments, the displacement vector restoration unit (918) extracts displacement vector transform coefficients from the decoded displacement vector video, and restores the displacement vector by applying inverse quantization and inverse transformation processes to the extracted displacement vector transform coefficients. To this end, the displacement vector restoration unit (918) may include an image unpacking unit, an inverse quantizer, and an inverse linear lifting unit. If the restored displacement vector is a value in a local coordinate system, a process of inversely transforming it into a Cartesian coordinate system may be performed.

[0215] The mesh restoration unit (916) can generate additional vertices by performing subdivision on the restored base mesh. Through subdivision, vertex connection information including the added vertices, texture coordinates, and texture coordinate connection information can be generated. At this time, the mesh restoration unit (916) can generate a final restored mesh (or a restored deformed mesh) by combining the subdivided restored base mesh with the restored displacement vector.

[0216] According to embodiments, the texture map video decoder (919) can decode the texture map bitstream as a video bitstream using a video codec to restore the texture map. The restored texture map has color information for each vertex contained in the restored mesh, and the color value of each vertex can be obtained from the texture map using the texture coordinates of each vertex.

[0217] According to embodiments, the mesh restored by the mesh restoration unit (916) and the texture map restored by the texture map video decoder (919) are shown to the user through a rendering process in the mesh data renderer (920).

[0218] Referring to FIG. 14, a receiving device (decoder) can decode a mesh in an intra-frame or inter-frame manner. A receiving device according to intra-decoding can receive a base mesh, a displacement vector (or referred to as displacement), a texture map (or referred to as attribute map), and render mesh data based on the restored mesh and the restored texture map. A receiving device according to inter-decoding can receive a motion vector (or referred to as motion), a displacement vector (or referred to as displacement), a texture map (or referred to as attribute map), and render mesh data based on the restored mesh and the restored texture map.

[0219] A mesh data transmission device and method according to embodiments may pre-process mesh data, encode the pre-processed mesh data, and transmit a bitstream including the encoded mesh data. A point mesh data reception device and method according to embodiments may receive a bitstream including mesh data and decode the mesh data. The mesh data transmission and reception method / device according to embodiments may be abbreviated as method / device according to embodiments. The mesh data transmission and reception method / device according to embodiments may also be referred to as 3D data transmission and reception method / device or point cloud data transmission and reception method / device.

[0220] As mentioned above, the V-Mesh method converts the displacement information generated during the encoding process into a video format and then compresses it using a conventional 2D video codec. In addition, the texture map (same meaning as the attribute map) information of the input mesh data is also processed as a video and compressed using a conventional 2D video codec. At this time, the V-DMC encoder / decoder can perform inter-screen prediction or intra-screen prediction during encoding and / or decoding. Here, inter-screen prediction is called inter-prediction, and intra-screen prediction is called intra-prediction. If the V-DMC encoder / decoder performs inter-screen prediction, currently, inter-screen prediction is performed only for the base mesh, and inter-screen prediction is not performed for displacement information (or displacements) and texture maps. For example, in the case of inter-screen prediction of the base mesh, prediction is performed using the base mesh and motion vector of the reference frame. In contrast, encoding / decoding is performed for displacement information and texture maps using intra-screen prediction, i.e., intra-coding.

[0221] However, performing inter-screen prediction only on the base mesh may be a less efficient method. Therefore, in order to perform more efficient compression, especially in inter-screen prediction, the present disclosure proposes a method of omitting the current texture map coding when the similarity between the reference texture map and the current texture map is high. In addition, when the texture map coding omitting is performed only on a frame basis, there may be cases where most of the texture maps are similar but some areas are different and thus the texture map coding omitting is not performed. In such cases, in order to perform efficient compression through texture map coding omitting, the present disclosure proposes a method of omitting the texture map even on a submesh, patch, tile, etc. basis.

[0222] In this way, the present disclosure can skip the current texture map coding when the similarity between the reference texture map of the dynamic mesh and the current texture map is high. At this time, the unit for skipping the texture map coding may be a frame, a submesh, a patch, a tile, etc. That is, the present disclosure can obtain the effect of reducing the bits of the transmitted texture map and the encoding / decoding complexity by omitting the texture map that occupies a considerably large proportion of the V-DMC bitstream data. In other words, the present disclosure can increase the compression efficiency of the V-DMC codec by increasing the compression efficiency of the texture map that occupies the largest bit proportion among the base mesh, displacement vector, and texture map, which are the main components of the V-DMC data.

[0223] In addition, the present disclosure proposes a high-level syntax signaling method based on an atlas stream to apply a texture map coding skip method in a V-DMC codec. In particular, the present disclosure proposes a method for applying a texture map skip method in units of Frame, Submesh, Patch, and Tile in a V-DMC codec, and a signaling method and syntax and semantics based on an atlas stream. According to embodiments, the present disclosure can signal a flag for whether to use a texture map skipping technique through an Atlas Sequence Parameter Set (ASPS) and a flag for whether an omitted texture map exists in a sequence. In addition, the present disclosure can signal a frame-by-frame texture map skipping flag and reference information for deriving an omitted texture map by syntax transmission position based on an atlas stream (Atlas Frame Parameter Set, Tile header, Patch data unit, NAL unit header).

[0224] Fig. 15 illustrates a transmitting device according to embodiments. The transmitting device of Fig. 15 may be referred to as a mesh data transmitting device or an encoder or an encoder of a transmitting device or a V-Mesh encoder or a dynamic mesh encoder.

[0225] FIG. 15 corresponds to the transmitting device (100) or mesh video encoder (102) of FIG. 1, the encoder (preprocessor and encoder) of FIG. 2, FIG. 6, or FIG. 7, the transmitting device of FIG. 13, and / or the transmitting encoding device corresponding thereto. Each component of FIG. 15 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in FIG. 15 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0226] In the present disclosure, the operation process of a transmitter for compressing and transmitting dynamic mesh data using V-Mesh compression technology may be as shown in FIG. 15. The transmitter of FIG. 15 may support both an intra-frame encoding (or intra-encoding or intra-screen encoding) process and / or an inter-frame encoding (or inter-encoding or inter-screen encoding) process.

[0227] In Fig. 15, the mesh simplification unit (11011) simplifies the input original mesh through a mesh simplification algorithm to generate a base mesh (or simplified base mesh or simplified mesh). At this time, mesh simplification can be performed based on the number of target vertices or target polygons constituting the mesh. For example, a method such as decimation can be used as a mesh simplification algorithm that simplifies the original mesh. That is, the decimation method can be a process of selecting vertices to be removed from the original mesh using a certain reference point, and then removing the selected vertices and the triangles connected to the selected vertices.

[0228] That is, the mesh simplification unit (11011) can simplify the input mesh by the target number of vertices or the target number of faces. At this time, the simplification process can be performed through various methods such as triangle collapse and edge collapse.

[0229] According to embodiments, the base mesh simplified in the mesh simplification unit (11011) is provided to the mesh parameterization unit (11012) and the mesh refinement unit (11018).

[0230] The mesh parameterization unit (11012) performs a process of mapping a 3D surface to a texture domain for a simplified mesh (decimated mesh). That is, the mesh parameterization unit (11012) generates texture coordinates and texture connection information of the input mesh. In one embodiment, the mesh parameterization unit (11012) may perform parameterization using a UV Atlas tool. Through this process, mapping information is generated regarding which location on a 2D image each vertex of the simplified mesh (decimated mesh) can be mapped to. The mapping information is expressed and stored as texture coordinates, and through this process, the final base mesh is generated. That is, the mesh parameterization unit (11012) performs parameterization to generate texture coordinates (UV coordinates) and texture connection information per vertex of the input mesh (i.e., simplified mesh or simplified base mesh).

[0231] The final base mesh (or base mesh with texture map) generated in the above parameterization unit (11012) is input to the mesh quantization unit (11013) and quantized.

[0232] According to embodiments, the mesh quantization unit (11013) may perform a task of quantizing floating-point type mesh information (e.g., geometry information (x, y, z) or / and texture coordinates (u, v), normal information (nx, ny, nz), etc.) into fixed-point type. That is, the mesh quantization unit (11013) may quantize vertex coordinates and texture coordinates of the base mesh. According to embodiments, quantization for specific components may be omitted.

[0233] The above mesh subdivision unit (11018) subdivides the base mesh simplified by the mesh simplification unit (11011). That is, the mesh subdivision unit (11018) can perform mesh subdivision on the base mesh to generate additional vertices. Depending on the subdivision method, vertex connection information, texture coordinates, and texture coordinate connection information including the added vertices can be generated. At this time, depending on the subdivision method, geometry information connection information, texture coordinate connection information, and texture coordinates can be implicitly derived and generated. According to embodiments, the mesh subdivision unit (11018) can perform subdivision through a method such as mid-edge, Loop, or Catmul&Clark.

[0234] In addition, mesh subdivision in the mesh subdivision unit (11018) can be performed n times by user parameters or an agreement between the encoder / decoder, and according to embodiments, the vertices of the base mesh are R9, the vertices newly generated by performing subdivision 1 are R1, …, the vertices generated by performing subdivision n are R n LOD when defined as n can be defined as follows:

[0235] LOD n = R9 R9 , ..., R n

[0236] According to embodiments, the mesh fitting unit (11019) may perform fitting by adjusting vertex positions so that the mesh subdivided by the mesh subdivision unit (11018) becomes similar to the original mesh, thereby generating a fitted subdivided mesh. That is, the mesh fitting unit (11019) performs vertex position adjustment so that the subdivided mesh becomes similar to the original mesh. According to embodiments, the mesh simplification unit (11011), the mesh parameterization unit (11012), the mesh subdivision unit (11018), and the mesh fitting unit (11019) may be omitted, and when the corresponding processes are omitted, the original mesh may be applied as an input to the mesh quantization unit (11013).

[0237] At this time, coordinate information of the original mesh can be applied as input to the displacement vector calculation unit (11020), and according to embodiments, the displacement vector encoding process (i.e., displacement vector calculation unit (11020), displacement vector coordinate system conversion unit (11021), displacement vector encoder (11022)) can be omitted.

[0238] The present disclosure may be referred to as a pre-processor, including a mesh simplification unit (11011), a mesh parameterization unit (11012), a mesh refinement unit (11018), and a mesh fitting unit (11019). According to embodiments, the pre-processor may further include a displacement vector calculation unit (11020).

[0239] According to embodiments, the quantized base mesh in the mesh quantization unit (11013) can be output to a motion vector encoder (11015) or a static mesh encoder (11016) through a switching unit (11014).

[0240] According to embodiments, the base mesh is output to a motion vector encoder (11015) through a switching unit (11014) when performing inter-encoding for the corresponding mesh frame, and is output to a static mesh encoder (11016) through a switching unit (11014) when performing intra-encoding for the corresponding mesh frame. The motion vector encoder (11015) may be referred to as a motion encoder.

[0241] For example, when performing intra encoding or intra frame encoding for the corresponding mesh frame, the base mesh can be compressed through a static mesh encoder (11016). In this case, encoding can be performed on connection information, vertex geometry information, vertex texture information, normal information, etc. of the base mesh. That is, vertex coordinates, vertex connection information, texture coordinates, texture connection information, etc. of the mesh can be encoded in the static mesh encoder (11016). The base mesh bitstream generated through encoding is transmitted to a multiplexer (not shown).

[0242] As another example, when performing inter-encoding (or inter-frame encoding) on ​​the corresponding mesh frame, the motion vector encoder (11015) can receive a base mesh and a reference reconstructed base mesh (or a reconstructed quantized reference base mesh) as input, calculate a motion vector between the two meshes, and encode the value. In addition, the motion vector encoder (11015) can perform prediction based on connection information using a previously encoded / decoded motion vector as a predictor, and entropy-encode a differential motion vector (or residual motion vector) obtained by subtracting the predicted motion vector from the current motion vector. The motion vector bitstream generated through the encoding is transmitted to a multiplexer (not shown) as a base mesh bitstream. That is, in the case of intra-frame encoding, the static mesh bitstream is input to the multiplexer as a base mesh bitstream, and in the case of inter-frame encoding, the motion vector bitstream is input to the multiplexer as a base mesh bitstream.

[0243] In FIG. 15, the base mesh decoder (11017, or base mesh restoration unit) can receive a base mesh encoded by a static mesh encoder (11016) or a motion vector encoded by a motion vector encoder (11015) and generate a reconstructed base mesh. The base mesh decoder (11017) performs restoration of the base mesh according to the encoding type (inter-screen encoding or intra-screen encoding) of the current mesh. For example, the base mesh decoder (11017) can perform static mesh decoding on the base mesh encoded by the static mesh encoder (11016) to restore the base mesh. At this time, quantization can be applied before static mesh decoding, and inverse quantization can be applied after static mesh decoding. That is, when intra-screen encoding is performed, the current base mesh can be restored by performing dequantization on the quantized base mesh through the mesh quantization unit (11013). As another example, the base mesh decoder (11017) can restore the base mesh based on the restored quantized reference base mesh and the motion vector encoded by the motion vector encoder (11015). That is, when inter-screen encoding is performed, the current base mesh can be generated by decoding the motion vector using the motion vector decoding method and then applying (i.e., adding) the decoded motion vector to the reference restored base mesh. At this time, when the motion vector is not quantized, the motion vector restoration process is omitted and the current base mesh can be restored using the motion vector calculated by the motion vector encoder (11015). The restored base mesh is output to the displacement vector calculation unit (11020) and the mesh dequantization unit (11024).

[0244] According to embodiments, the displacement vector calculation unit (11020) can perform mesh refinement on the restored base mesh. In addition, the displacement vector calculation unit (11020) can calculate a displacement vector, which is a difference value of vertex positions between the restored base mesh that has been refined and the fitted subdivision (or refined) mesh generated by the mesh fitting unit (11019). At this time, the displacement vector can be calculated as many times as the number of vertices of the refined mesh. That is, the displacement vector of the number of vertices of the refined mesh can be calculated through the displacement vector calculation unit (11020).

[0245] According to embodiments, the displacement vector coordinate system conversion unit (11021) may output the vertex displacement vector calculated in the 3D Cartesian coordinate system (i.e., (x, y, z) space) (or canonical coordinate system) as it is, or may convert it into a local coordinate system (i.e., normal, tangential, bi-tangential coordinate system) based on the normal vector of each vertex. At this time, the normal vector may be calculated for each subdivided vertex based on the geometry information and connection information of the surrounding vertices. According to embodiments, among the normal, tangential, and bi-tangential components of the local coordinate system, only the normal component may be encoded. When the coordinate system conversion is applied according to the agreement between the encoder / decoder, encoding of only the normal component is always performed, or the encoder may decide to signal a 1-bit flag. At this time, the normal vector can be calculated for each subdivided vertex based on the geometric information and / or connection information of the surrounding vertices.

[0246] And, in the displacement vector coordinate system conversion unit (11021), whether or not to perform displacement vector coordinate system conversion is determined by an agreement between the encoder (i.e., transmitting device) / decoder (i.e., receiving device), or a coordinate system conversion status flag (e.g., (asps_vmc_ext_displacement_coordinate_system)) may be transmitted in units such as sequence, GOF (Group of frame), frame, and sub-mesh to determine whether or not to perform coordinate system conversion. As an example, the displacement vector coordinate system conversion status flag (asps_vmc_ext_displacement_coordinate_system), which is information that can identify whether or not to perform displacement vector coordinate system conversion, may be signaled in signaling information (e.g., atlas sequence parameter set, ASPS) and transmitted to the receiving device. For example, if the value of the displacement vector coordinate system transformation flag (asps_vmc_ext_displacement_coordinate_system) syntax (or field) is 0, the canonical coordinate system is used as is, and if it is 1, it indicates that the transformation is performed to the local coordinate system.

[0247] According to embodiments, the displacement vector of the cartesian coordinate system or the displacement vector converted to the local coordinate system in the displacement vector coordinate system conversion unit (11021) is encoded into a displacement vector bitstream (or displacement vector video bitstream) in the displacement vector encoder (11022). In one embodiment, the displacement vector encoder (11022) can encode the displacement vector of the cartesian coordinate system or the displacement vector converted to the local coordinate system into the displacement vector bitstream using a 2D video encoder such as H.264, HEVC, or VVC.

[0248] That is, the displacement vector encoder (11022) can perform encoding on a displacement vector or a displacement vector coefficient (or displacement vector transform coefficient). In the present disclosure, the displacement vector encoder (11022) can perform encoding through a video codec-based encoder, a zero run-length encoder, an arithmetic encoder, etc. For example, when the encoding method is video codec-based encoding, the displacement vector encoder (11022) can encode the displacement vector or the displacement vector coefficient by packing it into a frame. That is, in the displacement vector encoder (11022), the displacement vector coefficients can be packed into a 2D image and then encoded using a 2D video codec (i.e., a video compression codec), or zero run-length encoded, or arithmetic encoded to generate a displacement vector video bitstream.

[0249] According to embodiments, a displacement vector video bitstream encoded and generated by a displacement vector encoder (11022) is transmitted to a multiplexer (not shown). According to embodiments, a method for selecting encoding of the displacement vector encoder (11022) may use a displacement vector encoder promised in an encoder (i.e., a transmitting side) / decoder (i.e., a receiving side), or may analyze the characteristics of a displacement vector in an encoder on the transmitting side and transmit the type of a selected displacement vector encoder to a decoder on the receiving side.

[0250] According to embodiments, the displacement vector restoration unit (11023) can restore the displacement vector by performing the reverse process of displacement vector encoding on the displacement vector or displacement vector coefficient encoded by the displacement vector encoder (11022). That is, the displacement vector restoration unit (11023) can perform displacement vector depacking (or unpacking) depending on the method of encoding the displacement vector, for example, when encoded based on a video codec. More specifically, the displacement vector restoration unit (11023) decodes a bitstream encoded through a 2D video encoder by being packed into a 2D image / video using a 2D video decoder, and performs depacking. Then, dequantization is performed on the quantized transform coefficients on which depacking was performed, and inverse transformation is performed to calculate the restored displacement vector. That is, the displacement vector restoration unit (11023) can additionally perform inverse quantization, inverse transformation, etc. processes depending on whether quantization and transformation processes are performed during the displacement vector encoding process.

[0251] According to embodiments, the mesh dequantization unit (11024) can dequantize vertex coordinates or texture coordinates of the restored base mesh using the reverse process of quantization. If the quantization process is omitted in the mesh quantization unit (11013), the dequantization process is also omitted in the mesh dequantization unit (11024).

[0252] According to embodiments, the mesh restoration unit (11025) can restore a mesh based on a restored displacement vector output from the displacement vector restoration unit (11023) and a restored base mesh (or a dequantized restored base mesh) output from the mesh dequantization unit (11024). More specifically, the mesh restoration unit (11025) can perform subdivision on the restored base mesh output from the mesh dequantization unit (11024) and add the restored displacement vector from the displacement vector restoration unit (11023) to generate a reconstructed deformed mesh. According to embodiments, the mesh restored by the mesh restoration unit (11025) (or referred to as a restored mesh or a restored deformed mesh) has restored vertices, connection information between vertices, texture coordinates, and connection information between texture coordinates. The restored mesh (or restored mesh or restored deformed mesh) generated in the above mesh restoration unit (11025) is provided to the texture map generation unit (11026).

[0253] According to embodiments, the texture map generation unit (11026) can regenerate the texture map of the current mesh based on the texture map (or attribute map) of the original mesh and the mesh restored by the mesh restoration unit (11025). That is, the texture map generation unit (11026) can generate the texture map of the restored mesh through the texture map of the original mesh and the relationship between the original mesh and the restored mesh. In other words, the texture map generation unit (11026) generates the texture map of the restored mesh through the texture coordinates and connection information of the restored mesh and the relationship between the original mesh and the texture map of the original mesh.

[0254] According to embodiments, the texture map generation unit (11026) can assign color information per vertex of the texture map of the original mesh to the texture coordinates of the restored base mesh (or the restored deformed mesh). According to embodiments, the texture map generation unit (11026) can generate a texture map (or texture map video) by grouping the regenerated texture maps by GoF unit for each frame.

[0255] According to embodiments, the texture map omission decision unit (11027) determines whether to omit encoding of the current texture map. In one embodiment of the present disclosure, if the similarity between the reference texture map and the current texture map is high, encoding of the current texture map is omitted. Here, omitting encoding of the current texture map includes not only not encoding the current texture map but also not transmitting the current texture map to the receiving device. A method for determining whether to omit encoding of the current texture map will be described in detail later.

[0256] A texture map that is determined not to be omitted in the texture map omission decision unit (11027) may be encoded in a texture map encoder (11028). For example, the texture map encoder (11028) may encode the texture map using a 2D video codec-based encoder, a zero run length encoder, an entropy coding-based arithmetic encoder, etc. In addition, the texture map encoder (11028) may further perform color space conversion of the texture map. Then, a texture map substream (or referred to as a texture map video bitstream) generated through texture map encoding is transmitted to a multiplexer (not shown).

[0257] According to embodiments, the type of texture map encoder (11028) may include a video encoder (e.g., VVC, HEVC, etc.), an entropy coding-based encoder, etc. In addition, a method for selecting a texture map encoder (11028) may use a texture map encoder promised in an encoder (i.e., a transmitting side) / decoder (i.e., a receiving side), or may transmit the type of texture map encoder selected by the encoder on the transmitting side to the decoder on the receiving side.

[0258] According to embodiments, a multiplexer (not shown) may multiplex an input base mesh bitstream, a displacement vector bitstream, and a texture map bitstream into a single bitstream and then transmit the multiple bitstreams to a receiving device. Alternatively, the base mesh bitstream, the displacement vector bitstream, and the texture map bitstream may be encapsulated into a file / segment and transmitted to the receiving device.

[0259] According to embodiments, the bitstream multiplexed in the multiplexer may be transmitted over a network or stored in a digital storage medium. Here, the network may include a broadcasting network and / or a communication network, and the digital storage medium may include various storage media such as a USB, SD, CD, DVD, Blu-ray, HDD, or SSD.

[0260] The following describes a method for determining whether to omit texture map coding in a texture map omission decision unit (11027) and a method for signaling related information.

[0261] According to embodiments, the texture map omission decision unit (11027) can determine whether to omit coding of a texture map generated by the texture map generation unit (11026) through the following processes.

[0262] According to embodiments, whether to skip encoding / decoding of a texture map in a texture map skip decision unit (11027) may be determined by explicitly signaling a texture map skip decision flag (texture_skip_flag), or the encoder / decoder may implicitly derive whether to skip according to conditions defined in the same manner.

[0263] In addition, the present disclosure can transmit information on whether a texture map skipping technique (i.e., method) is applied to a corresponding content or sequence to a receiving device by signaling a flag (asve_textureSkip_enable_flag) through an Atlas sequence parameter set (ASPS) indicating whether a texture map skipping technique (i.e., method) is used in a V-DMC codec. In the present disclosure, the signaling may be performed in the texture map skipping decision unit (11027) or may be performed in a separate block or module such as a signaling processing unit (not shown).

[0264] For example, if the asve_textutreSkip_eanble_flag value is 1, the texture map skipping related parameters can be parsed. At this time, if there is a skipped texture map in the sequence, the asve_textureSkip_present_flag value can be signaled as 1, and if there is not, the value can be signaled as 0.

[0265] According to embodiments, the criteria for determining whether to omit a texture map in the texture map omission decision unit (11027) may include the similarity between the current texture map and the reference texture map, the amount of prediction bits generated, the distance between the current texture map and the reference texture map, etc.

[0266] And, the omission unit of texture information (i.e., texture map) may be a unit such as a frame, patch, sub-mesh, tile, slice, etc. That is, a texture map omission flag (texture_skip_flag) indicating whether to omit the texture map may be determined according to the omission unit of the texture map, such as a frame unit, a patch unit, a slice unit, a sub-mesh unit, or a tile unit, determined by the texture map omission determination unit (11027).

[0267] Fig. 16 is a flowchart showing an example of a method for determining whether to omit texture map coding according to embodiments. That is, Fig. 16 is an example of a method for determining whether to omit texture map coding in a texture map omission decision unit (11027), and some of the processes may be omitted or some processes may be added.

[0268] According to embodiments, if it is confirmed in step 12011 that curUintIdx is smaller than lastUnitSize, the process proceeds to step 12012 to determine whether to omit texture map coding, and if not, the process of determining whether to omit texture map coding is terminated. That is, comparing curUintIdx and lastUnitSize in step 12011 is to perform whether to omit texture map coding in units of GoF (Group Of Frame). For example, if 30 frames are included in GoF, curUintIdx may be an index that increases from 0, 1, 2, ... 29, and lastUnitSize may mean the number of frames in GoF, that is, 30. In other words, this is an example of determining whether to omit texture map coding in units of frames in GoF.

[0269] If it is confirmed in step 12011 that curUintIdx is smaller than lastUnitSize, it is checked whether the base mesh type is I (step 12012). If the base mesh type is I, the process of determining whether to omit the texture map of the current frame is not performed (i.e., the coding of the texture map is not omitted), otherwise (e.g., the base mesh type is P or skipped), it proceeds to step 12013 to determine whether to omit the texture map coding of the current frame. Step 12013 may be referred to as a texture map reference information and omission determination unit. Although the present disclosure states that the coding of the texture map is not omitted if the base mesh type is I, this is an embodiment. That is, even if the base mesh type is I, the coding of the texture map may be omitted.

[0270] According to embodiments, in the present disclosure, the texture map skip flag (texture_skip_flag) may be transmitted only when the type of the current mesh or base mesh is a type for which inter-screen prediction is performed, and when the type of the current mesh is an intra-screen prediction type, transmission of the texture map skip flag (texture_skip_flag) may be omitted and the texture map skip flag (texture_skip_flag) may be implicitly derived to 0.

[0271] According to embodiments, if the type of the current mesh is a mesh type for which inter-screen prediction is performed, the texture map reference information and whether to omit the texture map can be determined through the texture map reference information and whether to omit the texture map (i.e., step 12013).

[0272] According to embodiments, if the current mesh satisfies the flag transfer condition, whether to omit texture map coding can be determined by comparing various metrics based on the current texture map and the reference texture map.

[0273] According to embodiments, the texture map reference information and omission decision unit (i.e., step 12013) determines whether the difference between the PSNR (Peak-Signal-to-Noise Ratio) value calculated through the current texture map and the PSNR value calculated through the reference texture map is a threshold value (TH). PSNR ) is less than the threshold (TH), the texture map skip flag is determined as 1 (i.e., texture_skip_flag=1) and the encoding of the current texture map can be skipped. That is, the difference between the PSNR value of the current texture map and the PSNR value of the reference texture map is less than the threshold (TH PSNR ) means that the similarity between the current texture map and the reference texture map is high. In other words, the current texture map is sufficiently similar to the reference texture map, so there is no need to encode the current texture map.

[0274] At this time, PSNR may be used, such as point cloud-based luminance and / or chrominance PSNR between the original mesh and the mesh composed of the current or reference texture map and restored geometry information.

[0275] Additionally, the texture map reference information and omission decision unit (i.e., step 12013) may determine whether to omit texture map coding by measuring Weighted PSNR by applying weights to units such as sub-meshes, tiles, etc. or some areas.

[0276] The following is a description of the texture map coding omission units.

[0277] According to embodiments, texture map omission information may be determined by signaling it on a frame-by-frame basis, a patch-by-patch basis, a sub-mesh basis, a tile-by-tile basis, a slice-by-slice basis, etc., or may be implicitly derived by the same rule in the encoder / decoder.

[0278] According to embodiments, a texture map skip flag (texture_skip_flag) determined on a frame basis, patch basis, sub-mesh basis, tile basis, slice basis, etc. may be transmitted to the receiving device for all encoding / decoding types, or may be transmitted to the receiving device only for a specific type.

[0279] For example, if the texture map coding skip unit is a frame unit, a frame unit texture map skip flag (afps_vdmc_texture_skip_flag) may be transmitted. That is, the frame unit texture map skip flag (afps_vdmc_texture_skip_flag) may be a flag indicating whether to skip the texture map on a frame unit basis.

[0280] For example, if the texture map coding skip unit is a patch unit, a patch unit texture map skip flag (atdu_texture_skip_flag) may be transmitted. That is, the patch unit texture map skip flag (atdu_texture_skip_flag) may be a flag indicating whether to skip the texture map on a patch unit basis.

[0281] For example, if the texture map coding skip unit is a sub-mesh unit, a sub-mesh unit texture map skip flag (smh_texture_skip_flag) may be transmitted. That is, the sub-mesh unit texture map skip flag (smh_texture_skip_flag) may be a flag indicating whether to skip the texture map in the sub-mesh unit.

[0282] For example, if the texture map coding skip unit is a tile unit, a tile unit texture map skip flag (ath_texture_skip_flag) may be transmitted. That is, the tile unit texture map skip flag (ath_texture_skip_flag) may be a flag indicating whether to skip the texture map on a tile unit basis.

[0283] According to embodiments, the texture map reference information and omission determination unit (i.e., step 12013) may determine information about a texture map referenced to generate an omitted texture map as follows.

[0284] According to embodiments, information about a referenced texture map may be an index to the referenced texture map, or may be a list of reference text maps, or may be referenced rule information, etc. Information about a referenced texture map in the present disclosure may be used interchangeably with reference texture map information with the same meaning.

[0285] According to embodiments, the reference texture map information may be determined by explicitly signaling the same reference texture map information in the encoder / decoder and / or may be implicitly induced to be omitted based on conditions defined in the same manner in the encoder / decoder.

[0286] If reference texture map information (or referred to as reference information) is explicitly signaled, a flag indicating reference texture direction information (past or future) can be determined.

[0287] After selecting the direction of the texture map to be referenced according to the embodiments, information about the direction of the texture map to be referenced may be transmitted to a decoder (i.e., a receiving device), or the encoder / decoder may be guided to the same direction.

[0288] At this time, the direction of the texture map to be referenced may be the direction of the previous texture map on the time axis (if smaller than the current texture map index) or the direction of the next texture map (if larger than the current texture map index) based on the current texture map index.

[0289] According to embodiments, if the POC (Picture order Count) gap between the reference texture map of the previous texture map direction and the current texture map is greater than a specific threshold, it is possible to signal reference texture direction information after first determining whether to refer to the reference texture map of the subsequent texture map direction, or if the POC gap is greater than a specific threshold by the same rule in the encoder / decoder, it is possible to implicitly derive reference texture direction information after determining whether to refer to the future texture map.

[0290] According to embodiments, when a unit such as a frame, sub-mesh, tile, slice, or patch is of type P, the texture map to be referenced may use previous texture maps, and when it is of type B, previous texture maps or future texture maps may be selected.

[0291] According to embodiments, after comparing the texture maps of both directions, the direction of the texture map to be referenced can be selected, and the information about the direction can be determined by explicitly signaling, or the direction information can be implicitly derived according to the same conditions in the encoder / decoder.

[0292] According to embodiments, when information about a referenced texture map is an index for the referenced texture map, the similarity between the current texture map and the reference texture map may be measured to determine a texture map with a high similarity as the reference texture map, or the reference texture map index may be determined by considering the order of the texture maps.

[0293] According to embodiments, when information about a referenced texture map is referenced rule information, a texture map located at a certain interval among texture maps whose encoding order is omnidirectional can be used as a reference texture map, and information about a specific interval can be determined by explicitly signaling, or the reference information can be implicitly derived by using the same interval in the encoder / decoder.

[0294] According to embodiments, the encoder / decoder may use the same texture map maximum distance parameter (max_skip_texturemap_distance) to determine whether to skip textures, and the texture map maximum distance parameter (max_skip_texturemap_distance) may be transmitted to the decoder (i.e., the receiving device), or may be derived in the decoder using the same texture map maximum distance parameter in the encoder / decoder.

[0295] In the present disclosure, the texture map maximum distance parameter (max_skip_texturemap_distance) may be a parameter that induces whether to skip the texture map.

[0296] According to embodiments, if the coding of the Nth texture map is omitted, if all texture maps located within a maximum distance from the omitted texture map on the time axis are consecutively omitted, the omission flag (texture_skip_flag) of the (N + max_skip_texturemap_distance)th texture map can be determined as 0.

[0297] Fig. 17 is a diagram showing an example of determining whether to skip a texture map using a texture map maximum distance parameter according to embodiments. Fig. 17 is an embodiment of a process for determining a texture map skip flag when the value of the texture map maximum distance parameter (max_skip_texturemap_distance) is 3.

[0298] Taking Fig. 17 as an example, if the first, second, and third texture maps are omitted in the encoding order, the fourth texture map, which is the texture map after the maximum distance parameter interval of the texture map on the time axis based on the first omitted texture map, may not be omitted, and the texture map omission flag (texture_skip_flag) may be determined to be 0. In this case, the texture map omission flag (texture_skip_flag) may not be transmitted separately.

[0299] For example, if the unit for determining whether to omit a texture map is a sub-mesh, the texture map of a sub-mesh that satisfies the texture map omission condition on a frame-by-frame basis may be omitted, and if the unit is two or more frames, such as a GOP (or GOF) unit, if the total number of sub-meshes per frame in the group is the same, and the texture map height and width of each sub-mesh are the same, the texture map of a sub-mesh that satisfies the texture map omission condition may be omitted.

[0300] And, the method of omitting the texture map of the sub-mesh can fill the omitted texture map area with a specific value according to embodiments.

[0301] Additionally, when omitting texture maps on a sub-mesh basis, a flag (smh_texture_skip_flag) indicating whether to omit texture maps on a sub-mesh basis can be signaled, or the encoder / decoder can derive it by the same rules.

[0302] If the flag indicating whether to skip the texture map on a frame-by-frame basis (afps_vdmc_texture_skip_flag) is 0, you can determine whether to skip the texture map on a sub-mesh basis through afps_vdmc_sm_skip_enable_flag.

[0303] For example, if afps_vdmc_sm_skip_enable_flag is 0, the texture map skipping flag (smh_texture_skip_flag) of the sub-mesh unit of the frame may be set to 0 and not signaled.

[0304] For example, if afps_vdmc_sm_skip_enable_flag is 1, it is possible to signal index information indicating the direction of the reference texture map (e.g., refDirection_idx) and index information indicating the index of the reference texture map (e.g., ref_texture_idx).

[0305] If the coding of the texture map is omitted for each sub-mesh unit, the texture maps referenced by the texture maps of the omitted sub-meshes within the same frame may reference sub-mesh texture maps of different frame indices.

[0306] Fig. 18 is a diagram showing an example of whether a texture map is omitted and a reference frame index in units of sub-mesh according to embodiments. In particular, Fig. 18 shows an example in which the frame indices of texture maps referenced in units of sub-mesh are different. More specifically, Fig. 18 shows an example in which each frame includes 5 sub-meshes, and the texture maps of sub-mesh indices 0 and 4 are omitted for a texture map with a frame index of 2 (i.e., smh_texture_skip_flag of sub-mesh indices 0 and 4 is 1). In this case, the frame index of the texture map referenced by sub-mesh index 0 is 0, and the frame index of the texture map referenced by sub-mesh index 4 is 1, so that texture maps of different frames can be referenced.

[0307] Fig. 19 is a diagram showing an example of a frame relationship of a texture map to be referenced when texture map coding is omitted for each sub-mesh unit according to embodiments. Fig. 19 is an example visualizing Fig. 18. Therefore, as in Fig. 18, this is a case where the coding of the texture map of the sub-meshes corresponding to sub-mesh indices 0 and 4 among the sub-meshes included in frame index 2 is omitted.

[0308] That is, among the five sub-meshes included in the frame of frame index 2, the texture map referenced by the texture map (texture 0) of the sub-mesh corresponding to sub-mesh index 0 is the texture map (texture 0) of the sub-mesh index 0 of the frame of frame index 0, and the texture map referenced by the texture map (texture 4) of the sub-mesh corresponding to sub-mesh index 4 is the texture map (texture 4) of the sub-mesh index 4 of the frame of frame index 1. In other words, when the reference frame is determined, the texture map of the sub-mesh at the same location within the reference frame is referenced.

[0309] As explained so far, when texture map coding and signaling are determined in the texture map omission decision unit (11027) among the texture maps generated through the texture map generation unit (11026), the texture map encoder (11028) omits encoding of the texture map with texture_skip_flag equal to 1. In other words, encoding and transmission of the texture map with texture_skip_flag equal to 1 are omitted.

[0310] And, a texture map with texture_skip_flag set to 0 can be encoded through a 2D video encoder, etc. in a texture map encoder (11028).

[0311] According to embodiments, when the color space of the texture map to be encoded is RGB 4:4:4, encoding can be performed after conversion to a color space such as YUV 4:2:0, YUV 4:4:4, YUV 4:0:0, YCgCo, etc.

[0312] Fig. 20 illustrates a receiving device according to embodiments. In the present disclosure, the receiving device of Fig. 20 may be referred to as a mesh data receiving device or decoder or decoder of a receiving device or a V-Mesh decoder or a dynamic mesh decoder.

[0313] FIG. 20 corresponds to the receiving device (110) or mesh video decoder (113) of FIG. 1, the decoder of FIG. 11 or FIG. 12, the receiving device of FIG. 14, and / or the receiving decoding device corresponding thereto. Each component of FIG. 20 corresponds to hardware, software, a processor, and / or a combination thereof. The receiving (decoding) operation of FIG. 20 may follow the reverse process of the corresponding process of the transmitting (encoding) operation of FIG. 15. The execution order of each block in FIG. 20 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0314] FIG. 20 may largely include a base mesh decoding unit, a displacement information decoding unit, and a texture map decoding unit. According to embodiments, the base mesh decoding unit may include a switching unit (15011), a motion vector decoder (15012), a static mesh decoder (15013), a base mesh restoration unit (15014), a mesh refinement unit (15015), and a mesh restoration unit (15016). According to embodiments, the displacement information decoding unit may include a displacement vector decoder (15017), a displacement vector inverse quantization unit (15018), a displacement vector inverse transformation unit (15019), and a displacement vector coordinate system inverse transformation unit (15020).

[0315] According to embodiments, a bitstream of mesh data received by a receiver (not shown) may be demultiplexed into a base mesh bitstream, a displacement vector bitstream, and a texture map bitstream after file / segment decapsulation in a demultiplexer (not shown). If the current mesh has inter-screen encoding (i.e., inter-encoding) applied, the base mesh bitstream may be a motion vector bitstream.

[0316] According to embodiments, the base mesh bitstream is provided to a motion vector decoder (15012) via a switching unit (15011) or to a static mesh decoder (15013).

[0317] For example, if the current mesh has inter-screen encoding (i.e., inter encoding) applied, the base mesh bitstream, i.e., the motion vector bitstream, is received, demultiplexed, and then output to the motion vector decoder (15012) through the switching unit (15011). As another example, if the current mesh has intra-screen encoding (i.e., intra encoding) applied, the base mesh bitstream is received, demultiplexed, and then output to the static mesh decoder (15013) through the switching unit (15011). Here, the motion vector decoder (15012) may be referred to as a motion decoder.

[0318] According to embodiments, the motion vector decoder (15012) can perform decoding on a motion vector bitstream on a vertex-by-vertex basis or a subgroup basis.

[0319] According to embodiments, the motion vector decoder (15012) can reconstruct a final motion vector by adding a differential motion vector (i.e., a residual motion vector) decoded from a bitstream using a previously decoded motion vector as a predictor. That is, the motion vector decoder (15012) can decode a differential motion vector (or a residual motion vector) in units of vertices or subgroups (or subblocks) through a motion vector bitstream, and perform prediction based on connection information using a previously decoded motion vector as a predictor to decode the motion vector by adding it to the residual motion vector.

[0320] According to embodiments, the static mesh decoder (15013) can decode the base mesh bitstream to restore connection information, vertex geometry information, texture coordinates (i.e., attribute geometry information), normal information, etc. of the base mesh.

[0321] According to embodiments, the base mesh restoration unit (15014) can restore the current base mesh based on the decoded motion vector or the decoded base mesh. For example, if the current mesh has inter-screen encoding applied, the base mesh restoration unit (15014) can add the decoded (or restored) motion vector to the reference base mesh and then perform inverse quantization to generate a restored base mesh (i.e., the current base mesh). As another example, if the current mesh has intra-screen encoding applied, the base mesh restoration unit (15014) can perform inverse quantization on the base mesh decoded (or restored) through the static mesh decoder (15012) to generate a restored base mesh (i.e., the current base mesh).

[0322] According to embodiments, the mesh subdivision unit (15015) can generate additional vertices by performing subdivision on the base mesh. The present disclosure can implicitly derive and generate geometry information connection information, texture coordinate connection information, and texture coordinates according to the subdivision method.

[0323] According to embodiments, the mesh refinement unit (15015) can perform refinement through methods such as mid-edge, Loop, and Catmul&Clark.

[0324] According to embodiments, mesh subdivision in the mesh subdivision unit (15015) may be performed n times by user parameters or encoder / decoder promises. According to embodiments, the vertices of the base mesh are R9, the newly generated vertices are R1 by performing subdivision 1, …, the vertices are R generated by performing subdivision n times. n LOD when defined as n can be defined as follows:

[0325] LOD n = R9 R9 , ..., R n

[0326] According to embodiments, the displacement vector decoder (15017) may perform video codec-based decoding on the demultiplexed displacement vector bitstream as a video bitstream, or perform zero run-length decoding, or perform arithmetic decoding. In the present disclosure, the displacement vector decoder may be used interchangeably with the displacement vector transform decoder.

[0327] According to embodiments, the displacement vector decoder (15017) can restore the displacement vector by decoding the displacement vector in a reverse process of the displacement vector encoding method of the transmitting side.

[0328] According to embodiments, the displacement vector coefficient inverse quantization unit (15018) can inverse quantize the displacement vector coefficients restored by the displacement vector decoder (15017). According to embodiments, the displacement vector transform coefficients can be quantized through different quantization parameters for each axis, and in this case, the displacement vector coefficient inverse quantization unit (15018) can determine the quantization rate for each LoD level by deriving a quantization parameter or a scaling parameter by an encoder / decoder agreement.

[0329] According to embodiments, the displacement vector inverse transform unit (15019) performs an inverse transform of the transform performed in the encoder of the transmitting device on the inverse quantized displacement vector coefficients to output displacement vectors. According to embodiments, a lifting inverse transform, a wavelet inverse transform, etc. may be performed.

[0330] If the lifting inverse transformation is performed, the vertex R of the kth subdivision level k R as a predictor when performing prediction t (t <k 또는 t<=k)의 세분화 정점 변위 벡터를 통해 k번째 세분화 레벨의 변위벡터 예측을 수행할 수 있다.

[0331] According to embodiments, when performing prediction of a displacement vector, an average or distance-based weighted average prediction can be performed on n points near the current vertex based on connection information among vertices with a lower level of detail than the current vertex.

[0332] According to embodiments, prediction can be performed based on the displacement vectors of n vertices used to generate the current vertex in the mesh refinement step.

[0333] Additionally, when the lifting inverse transformation is performed, a process of updating the displacement vector of the vertex used for prediction in the encoder can be performed through the parsed residual signal.

[0334] According to embodiments, the displacement vector coordinate system inverse transformation unit (15020) parses the coordinate system transformation flag (asps_vmc_ext_displacement_coordinate_system or applyLocalCoord) included in the signaling information in units of sequences, GOFs (Group of Frames), frames, or submesh, and if the value is 1, the inverse quantized (or inversely transformed) restored displacement vector can be inversely transformed from the local coordinate system (n, t, b) to the canonical coordinate system (x, y, z).

[0335] In addition, a normal vector per vertex is calculated based on the restoration vertex position information of the restoration base mesh, and the normal value of the newly created vertex can be assigned by interpolating the vertex normal vector of the calculated restoration base mesh for the vertex additionally created through the subdivision process.

[0336] At this time, in the case of interpolation, interpolation can be performed by averaging or distance-based weighting the normal information of the base mesh used for subdivision. Alternatively, the normal information of the base mesh can be used as is for subdivided vertices on the same plane.

[0337] And, by calculating the tangential and bi-tangential vectors orthogonal to the normal vector through the calculated normal vector per vertex, the displacement vector coordinate system inverse transformation can be performed through the following mathematical expression 1. The disp of the following mathematical expression 1 n [0], disp n [1], disp n[2] refers to the results of normal, tangential, and bi-tangential components obtained by performing inverse transformation and inverse quantization. At this time, since the normal, tangential, and bi-tangential components are all orthogonal to each other, the displacement vector of the bi-tangential component can be calculated through the cross product of the final displacement vectors of the normal and tangential components. Alternatively, the displacement vector of the bi-tangential component can be derived and calculated using methods such as linear regression or multiple regression.

[0338] [Mathematical Formula 1]

[0339]

[0340] According to embodiments, coordinate system inversion can always be performed without flag transmission.

[0341] The output of the displacement vector coordinate system inverse transformation unit (15020) is provided to the mesh restoration unit (15016).

[0342] That is, in the encoder of the transmitting device, the vertex displacement vector calculated in the (x,y,z) space can be converted to a (normal, tangential, bi-tangential) coordinate system (or local coordinate system) based on the normal vector of each vertex. At this time, the normal vector can be calculated for each subdivided vertex based on the geometric information and connection information of the surrounding vertices.

[0343] According to embodiments, the mesh restoration unit (15016) restores the mesh based on the mesh refined in the mesh refinement unit (15015) and the restored displacement vector output from the displacement vector coordinate system inverse transformation unit (15020).

[0344] According to embodiments, the received and demultiplexed texture map bitstream is input to a texture map decoder (15021). According to embodiments, the texture map decoder (15021) can decode the texture map through a video codec (e.g., a 2D scalable decoder). That is, the texture map decoder (15021) can restore the texture map by applying 2D scalable decoding to the texture map. In addition, the texture map decoder (15021) can generate a texture map that is omitted from the transmitting side based on reference text map information received and included in the signaling information.

[0345] FIG. 21 shows an example of a detailed block diagram of a texture map decoder according to embodiments.

[0346] In Fig. 21, the texture map decoder (15021) may include a texture map omission flag determination unit (16011), a reference texture map information determination unit (16012), a texture map decoding unit (16013), and a texture map derivation unit (16014). Each component of Fig. 21 corresponds to hardware, software, a processor, and / or a combination thereof. The execution order of each block in Fig. 21 may be changed, some blocks may be omitted, and some blocks may be newly added.

[0347] According to embodiments, the texture map decoder (15021) may parse asve_textureSkip_enable_flag signaled and received in ASPS (Atlas sequence parameter set), and if its value is 1, it may know that a texture map skipping method (or technique) is applied to the corresponding content or sequence, and may parse texture map skipping related parameters.

[0348] According to embodiments, the texture map decoder (15021) can determine whether there is a skipped texture map in the current sequence by parsing the asve_textureSkip_present_flag signaled and received in the ASPS. If the value of asve_textureSkip_present_flag is 1, it can be known that a restoration process should be performed as there is a skipped texture map.

[0349] In Fig. 21, the texture map omission flag determination unit (16011) can parse and determine a 1-bit flag (texture_skip_flag) indicating whether to omit each unit of texture (or texture map) determined by the encoder.

[0350] According to embodiments, the texture map decoding unit and the texture map omission unit may be a frame unit, a patch unit, a sub-mesh unit, a tile unit, a slice unit, etc.

[0351] In Fig. 21, the reference texture map information determination unit (16012) is a step for determining reference texture map information for a texture map that has been determined to be omitted (i.e., coding and transmission have been omitted) in the encoder of the transmitting device. At this time, the reference structure may be determined explicitly or implicitly.

[0352] If the reference structure is determined by explicit parsing, the reference information can be determined through a 1-bit flag indicating the direction information (past or future) of the reference texture and a flag or index indicating the POC (Picture order count) difference between the current texture map and the reference texture map.

[0353] According to embodiments, a flag or index indicating reference texture direction information may be parsed from signaling information about whether the reference texture map is a past texture map or a future texture map based on the current texture map, and / or may be parsed from signaling information about whether a bidirectional texture map is used.

[0354] According to embodiments, if the type of the current mesh is a P-type mesh for which unidirectional prediction is performed, a flag indicating direction information of the reference texture may be omitted.

[0355] According to embodiments, if texture map omission is performed by always referring to the texture map with the smallest POC difference from the current texture map according to a promise between the encoder / decoder, a flag or index indicating the POC difference may be omitted.

[0356] According to embodiments, when the POC gap between the reference texture map of the previous texture map direction and the current texture map is greater than a certain threshold, the reference texture direction information can be implicitly derived in the future direction by the same rule between the encoder / decoder.

[0357] According to embodiments, if the frame type of the current mesh is an I-type mesh that does not reference a mesh of a temporally different frame, texture map omission is not always performed, so the texture map omission flag determination unit (16011), the reference texture map information determination unit (16012), and the texture map derivation unit (16014) may be omitted, and the texture map omission flag (texture_skip_flag) may be derived as 0.

[0358] According to embodiments, the texture map decoding unit (16013) can restore the texture map by decoding the received texture map bitstream as a video bitstream using a video codec (e.g., a 2D scalable decoder). At this time, the texture map omitted on the transmitting side is not included in the received texture map bitstream and is therefore not restored by the texture map decoding unit (16013).

[0359] A texture map that is not restored by the texture map decoding unit (16013), i.e., a texture map that is not transmitted because coding was omitted on the transmitting side, is generated by the texture map derivation unit (16014) based on a reference texture map and / or signaling information.

[0360] That is, for a texture map in which the texture map skip flag (texture_skip_flag) is 1, the process of generating the current texture map based on the reference texture map can be performed through the texture map derivation unit (16014).

[0361] At this time, when referencing one texture map, the reference texture map can be used as the current texture map based on the reference information (or referred to as reference text map information or information about the referenced text map). That is, one reference texture map is derived as the current text map omitted from the texture map encoder of the transmitting device.

[0362] If the current texture map is derived by referencing multiple texture maps, the current texture map can be derived by weighting the multiple texture maps based on an average or distance (POC).

[0363] In the present disclosure, flags according to texture map skipping units may include afps_vdmc_texture_skip_flag, atdu_texture_skip_flag, smh_texture_skip_flag, ath_texture_skip_flag, etc.

[0364] afps_vdmc_texture_skip_flag can be a flag indicating whether to skip texture maps on a per-frame basis (i.e., whether to skip texture map coding).

[0365] atdu_texture_skip_flag can be a flag indicating whether to skip texture maps on a patch basis.

[0366] smh_texture_skip_flag can be a flag indicating whether to skip texture maps on a submesh basis.

[0367] ath_texture_skip_flag can be a flag indicating whether to skip texture maps on a tile-by-tile basis.

[0368] The following is a detailed description of the texture map omission flag determination unit (16011), the reference texture map information determination unit (16012), and the texture map derivation unit (16014) included in the texture map decoder (1521) of FIG. 21.

[0369] Flag determining whether to omit texture map (1611)

[0370] The encoder / decoder can use the same texture map maximum distance parameter (max_skip_texturemap_distance) to determine whether to skip texture maps.

[0371] The above texture map maximum distance parameter (max_skip_texturemap_distance) is a parameter that induces whether to skip the texture map, and by omitting the texture map skip flag (texture_skip_flag), bits that need to be signaled can be saved.

[0372] In the case where the Nth texture map is omitted according to embodiments, if all texture maps located within the maximum distance between frames based on the omitted texture map on the time axis are omitted, the omission flag (texture_skip_flag) of the (N + max_skip_texturemap_distance)th texture map can be determined as 0.

[0373] Fig. 22 is a diagram showing an example of inducing a texture map omission status flag to 0 according to embodiments. That is, Fig. 22 is an embodiment of inducing a texture map omission status flag to 0 according to a texture map maximum distance parameter. For example, when the texture map maximum distance parameter (max_skip_texturemap_distance) is 2, if the first texture map and the second texture map are omitted in the encoding order, the omission status flag (texture_skip_flag) of the third texture map can be induced to 0 by the texture map maximum distance parameter (max_skip_texturemap_distance).

[0374] Fig. 23 is a flowchart showing an example of deriving a texture map omission flag in a texture map omission flag determination unit according to embodiments. That is, Fig. 23 is an embodiment of implicitly deriving whether to omit a texture map by a texture map maximum distance parameter (max_skip_texturemap_distance), and specific processes (or steps) may be added or omitted.

[0375] First, the number of texture maps to be skipped consecutively (curSkipCount) is initialized to 0 (step 17011).

[0376] Then, it is checked whether curUintIdx is smaller than lastUnitSize (step 17012). If it is confirmed that curUintIdx is smaller than lastUnitSize, the process proceeds to step 17013 to determine whether to omit the texture map flag, and if not, the process of determining whether to omit the texture map flag ends. That is, the reason curUintIdx and lastUnitSize are compared in step 17012 is to determine whether to omit the texture map flag in units of GoF. For example, if 30 frames are included in GoF, curUintIdx is an index that increases from 0, 1, 2, … 29, and lastUnitSize can mean the number of frames in GoF, that is, 30.

[0377] If curUintIdx is determined to be less than lastUnitSize in step 17013, the process proceeds to step 17014 to determine whether the base mesh type is I (step 17013). If the base mesh type is I, the process of determining whether to skip texture maps is omitted. Otherwise (e.g., the base mesh type is P or skip), the process proceeds to step 17014 to determine whether to skip texture maps is omitted.

[0378] Step 17014 checks whether the number of texture maps to be skipped consecutively (curSkipCount) is greater than or equal to the maximum texture map distance parameter (max_skip_texturemap_distance).

[0379] If the number of texture maps to be skipped consecutively (curSkipCount) is greater than or equal to the maximum texture map distance parameter (max_skip_texturemap_distance), the texture map skip flag of the corresponding texture map is set to 0 (step 17015).

[0380] If the number of texture maps to be skipped consecutively (curSkipCount) is less than the maximum texture map distance parameter (max_skip_texturemap_distance), the texture skip flag (texture_skip_flag) can be obtained through the texture skip flag parsing unit (not shown) or the texture skip flag derivation unit (not shown) (step 17016).

[0381] And, by checking whether texture_skip_flag is 1 (step 17017), if texture_skip_flag is 1, curSkipCount can be increased by 1 (step 17019), and if texture_skip_flag is 0, curSkipCount can be initialized to 0 (step 17018).

[0382] Texture map derivation unit (16014)

[0383] According to embodiments, when the unit for determining whether to omit a texture map is a sub-mesh, the texture map derivation unit (16014) may derive the texture map of the omitted sub-mesh on a frame-by-frame basis, and when omission is performed on a frame-by-frame basis, such as a GOP unit, the texture map of the omitted sub-mesh may be derived when the total number of sub-meshes per frame in the group is the same and the texture map height and width of each sub-mesh are the same.

[0384] At this time, the texture map of the sub-mesh omitted from the transmitting side can be derived as the texture map information of the sub-mesh being referenced using the reference information (i.e., reference text map information) determined through the reference texture map information determination unit (16012).

[0385] According to embodiments, when reference direction information is used and the reference direction information is past or future, the closest past or future texture map based on the current texture map can be used as a reference texture map to derive the current texture map to the same value as the reference texture map.

[0386] According to embodiments, when the reference direction information is bidirectional, the closest past and future texture maps based on the current texture map may be used as reference texture maps to derive the current texture map by averaging the two texture maps, or the current texture map may be derived by weighting the POC difference between the past texture map and the future texture map based on the current texture map.

[0387] According to embodiments, when using reference texture map index information, a texture map corresponding to the reference texture map index can be used as a reference texture map to derive the current texture map to have the same value as the reference texture map.

[0388] And, in case the texture map is omitted in units of sub-mesh on the transmitting side, the texture map derivation unit (16014) may parse a flag (smh_texture_skip_flag) indicating whether to omit the texture map in units of sub-mesh, or may derive it by the same rule between the encoder / decoder.

[0389] Additionally, if the flag afps_vdmc_texture_skip_flag indicating whether to skip the texture map on a frame-by-frame basis is 0, the texture map derivation unit (16014) can parse afps_vdmc_sm_skip_enable_flag to determine whether to parse the skip flag on a sub-mesh basis.

[0390] At this time, if afps_vdmc_sm_skip_enable_flag is 0, the texture map skipping flag (smh_texture_skip_flag) of the sub-mesh unit of the corresponding frame is set to 0 and may not be parsed.

[0391] And, the index of the sub-mesh being referenced may be derived as the same index as the index of the omitted sub-mesh, or may be derived as the same as the reference structure of the base mesh, or may be parsed.

[0392] If smh_texture_skip_flag is 1, the refDirection_idx index indicating the direction of the reference texture map may be parsed, and / or the ref_texture_idx index indicating the index of the reference texture map may be parsed.

[0393] In cases where texture maps are omitted in units of sub-mesh according to embodiments, texture maps referenced by texture maps of omitted sub-meshes within the same frame may reference sub-mesh texture maps of different frame indices.

[0394] Fig. 24 is a diagram showing an example of whether a texture map is omitted and a reference frame index in a sub-mesh unit according to embodiments. In particular, Fig. 24 shows an example in which the frame indices of texture maps referenced in a sub-mesh unit are different. More specifically, Fig. 24 shows an example in which each frame includes 5 sub-meshes, and the texture maps of sub-mesh indices 0 and 4 are omitted for a texture map with a frame index of 2 (i.e., smh_texture_skip_flag of sub-mesh indices 0 and 4 is 1). In this case, the frame index of the texture map referenced by sub-mesh index 0 is 0, and the frame index of the texture map referenced by sub-mesh index 4 is 1, so that texture maps of different frames can be referenced.

[0395] Fig. 25 is a diagram showing an example of a frame relationship of a texture map to be referenced when texture map coding is omitted for each sub-mesh unit according to embodiments. Fig. 25 is an example visualizing Fig. 24. Therefore, as in Fig. 24, this is a case where the coding of the texture map of the sub-meshes corresponding to sub-mesh indices 0 and 4 among the sub-meshes included in frame index 2 is omitted.

[0396] That is, among the five sub-meshes included in the frame of frame index 2, the texture map referenced by the texture map (texture 0) of the sub-mesh corresponding to sub-mesh index 0 is the texture map (texture 0) of the sub-mesh index 0 of the frame of frame index 0, and the texture map referenced by the texture map (texture 4) of the sub-mesh corresponding to sub-mesh index 4 is the texture map (texture 4) of the sub-mesh index 4 of the frame of frame index 1. In other words, when the reference frame is determined, the texture map of the sub-mesh at the same location within the reference frame is referenced.

[0397] According to embodiments, signaling information in a transmitting device (or an encoder of the transmitting device) may be generated by a metadata processing unit (not shown, may be referred to as a metadata generator, etc.) and provided to corresponding blocks in the transmitting device and / or a receiving device (or a decoder of the receiving device), and a metadata parser (not shown) of the receiving device may parse the received signaling information and provide it to the corresponding blocks. According to embodiments, each block of the receiving device may perform each operation based on the signaling information.

[0398] As described above, the omission unit of texture information (i.e., texture map) may be a unit such as a frame, patch, sub-mesh, tile, slice, etc. That is, the position at which the texture map omission flag (texture_skip_flag) indicating whether to omit the texture map is signaled may be determined depending on the omission unit of the texture map, such as a frame unit, a patch unit, a slice unit, a sub-mesh unit, or a tile unit.

[0399] Fig. 26 is a diagram showing an example of a syntax structure of an atlas frame parameter set (AFPS) among signaling information in a bitstream according to embodiments. In particular, Fig. 26 is a diagram showing an example of an atlas frame parameter set vdmc extension RBSP syntax and semantics structure. That is, afps_vdmc_extension() of Fig. 26 may be extended from the atlas frame parameter set to include parameters (i.e., information) regarding texture map omission when the texture map omission unit is a frame.

[0400] At this time, the texture map decoder (1521) of the receiving device may parse the texture map omission and reference information on a frame-by-frame basis, or may parse them as a single index.

[0401] As an example, if parsing with one index, afps_texture_skip_refDirection_idx can be parsed.

[0402] As another example, when each is parsed, the afps_vdmc_texture_skip_flag flag, which indicates whether to skip texture maps on a per-frame basis, may be parsed, and if the flag is 0, the afps_vdmc_sm_skip_enable_flag flag, which indicates whether to perform texture map skipping on a per-sub-mesh basis within the frame, may be parsed. In addition, if afps_vdmc_texture_skip_flag is 1, the refDirection_idx index, which indicates the direction of the reference texture map, may be parsed, and / or the ref_texture_idx index, which indicates the index of the reference texture map, may be parsed.

[0403] More specifically, in Fig. 26, afps_texture_skip_refDirection_idx is an index indicating whether to skip the texture map and the reference texture map direction information (i.e., on a frame basis).

[0404] Fig. 27 is a table showing examples of whether or not to omit a texture map and the reference direction of afps_texture_skip_refDirection_idx according to embodiments. That is, Fig. 27 may be an example of a code word and the meaning of the code word for signaling information on whether or not to omit a texture map and the reference direction with a 2-bit index (texture_skip_refDirection_idx).

[0405] For example, if the value of afps_texture_skip_refDirection_idx is 00, the texture map is not skipped and there is no reference direction.

[0406] As another example, if the value of afps_texture_skip_refDirection_idx is 01, it indicates that the texture map can be skipped and the past frame can be referenced relative to the current frame.

[0407] As another example, if the value of afps_texture_skip_refDirection_idx is 10, it indicates that the texture map can be skipped and references to future frames relative to the current frame.

[0408] As another example, if the value of afps_texture_skip_refDirection_idx is 11, it indicates that the texture map can be skipped and the bidirectional frame relative to the current frame can be referenced.

[0409] afps_vdmc_texture_skip_flag is a flag that indicates whether to skip texture maps on a frame-by-frame basis.

[0410] afps_vdmc_sm_skip_enable_flag is a flag that indicates whether to parse the texture map skipping flag for each submesh unit.

[0411] refDirection_idx is an index indicating reference texture map direction information.

[0412] If the reference texture map direction information according to the embodiments is information indicating whether the current texture map is a past texture map or a future texture map, refDirection_idx may be u(1), and / or if it includes information on a bidirectional case that uses both the past texture map and the future texture map, refDirection_idx may be u(v). Here, u(1) means a value expressed with 1 bit, and u(v) means a value expressed with v bits. At this time, v may be any number greater than 1.

[0413] ref_texture_idx represents the reference texture frame index.

[0414] max_skip_texturemap_distance indicates the maximum distance between the current texture map and the skipped texture map when determining whether to skip the texture map.

[0415] Fig. 28 is a diagram showing an example of a syntax structure of an atlas tile header according to embodiments. That is, atla_tile_header() of Fig. 28 may include parameters (i.e., information) regarding texture map omission when the texture map omission unit is a tile.

[0416] At this time, the texture map decoder (1521) of the receiving device may parse the texture map omission and reference information in tile units, or may parse them as a single index.

[0417] As an example, if parsing by one index, ath_texture_skip_refDirection_idx can be parsed.

[0418] As another example, when each is parsed, after parsing the ath_texture_skip_flag flag indicating whether to skip the texture map on a tile basis, if the flag is 1, the refDirection_idx index indicating the direction of the reference texture map may be parsed, and / or the ref_texture_idx index indicating the index of the reference texture map may be parsed.

[0419] More specifically, in Fig. 28, ath_type indicates the coding type of the current atlas tile. For example, if the value of ath_type is 0, it may indicate P_TILE, if it is 1, it may indicate I_TILE, and if it is 2, it may indicate SKIP_TILE.

[0420] ath_texture_skip_refDirection_idx is an index indicating whether the texture map is skipped and the reference texture map direction information (i.e., in atlas tile units). At this time, whether the texture map is skipped and the reference texture map direction information refer to the description in Fig. 27, and are omitted here to avoid redundant description.

[0421] ath_texture_skip_flag is a flag that indicates whether to skip texture maps on a tile-by-tile basis.

[0422] refDirection_idx is an index indicating reference texture map direction information.

[0423] ref_texture_idx is the reference texture frame index.

[0424] Fig. 29 is a diagram showing an example of a syntax structure of an atlas tile data unit according to embodiments. That is, atlas_tile_data_unit(tileID) of Fig. 29 may include parameters (i.e., information) regarding texture map omission when the texture map omission unit is a patch.

[0425] At this time, the texture map decoder (1521) of the receiving device may parse the texture map omission and reference information for each patch, or may parse them as a single index.

[0426] As an example, if parsing by one index, atdu_texture_skip_refDirection_idx can be parsed.

[0427] As another example, when each is parsed, after parsing the atdu_texture_skip_flag flag indicating whether to skip the texture map on a patch basis, if the flag is 1, the refDirection_idx index indicating the direction of the reference texture map may be parsed, and / or the ref_texture_idx index indicating the index of the reference texture map may be parsed.

[0428] More specifically, in Fig. 29, atdu_patch_mode[tileID][p] represents the patch mode of the patch with index p in the current tileID.

[0429] Fig. 30 is a table showing examples of determining a patch mode according to an identifier when the coding type of the current atlas tile according to embodiments is I_TILE. For example, if the value of atdu_patch_mode[tileID][p] is 0, it indicates I_INTRA mode (i.e., non-predicted patch mode), if it is 1, it indicates I_RAW mode (i.e., RAW point patch mode), and if it is 2, it indicates I_EOM mode (i.e., EOM point patch mode).

[0430] Fig. 31 is a table showing examples of determining a patch mode according to a delimiter when the coding type of the current atlas tile is P_TILE according to embodiments. For example, if the value of atdu_patch_mode[tileID][p] is 0, it indicates P_SKIP mode (i.e., patch skip mode), if it is 1, it indicates P_MERGE mode (i.e., patch merge mode), and if it is 2, it indicates P_INTER mode (i.e., inter predicted patch mode).

[0431] And, if the coding type of the current atlas tile is not I_TILE, the atlas tile data unit may include information about texture map skipping (e.g., atdu_texture_skip_flag, refDirection_idx, ref_texture_idx).

[0432] atdu_texture_skip_flag is a flag indicating whether to skip texture maps on a patch basis.

[0433] refDirection_idx is an index indicating reference texture map direction information.

[0434] ref_texture_idx is the reference texture frame index.

[0435] Fig. 32 is a diagram showing an example of the syntax structure of a basemesh submesh header according to embodiments. That is, submesh_header() of Fig. 32 may include parameters (i.e., information) regarding texture map omission when the texture map omission unit is a submesh.

[0436] At this time, the texture map decoder (1521) of the receiving device may parse the texture map omission and reference information for each sub-mesh unit, or may parse them as a single index.

[0437] As an example, if parsing with one index, smh_texture_skip_refDirection_idx can be parsed.

[0438] As another example, when each is parsed, the smh_texture_skip_flag flag indicating whether to skip the texture map for each sub-mesh may be parsed, and if the flag is 1, the refDirection_idx index indicating the direction of the reference texture map may be parsed, and / or the ref_texture_idx index indicating the index of the reference texture map may be parsed.

[0439] More specifically, in Fig. 32, smh_texture_skip_refDirection_idx is an index (i.e., submesh unit) indicating whether a texture map is skipped and information on the reference texture map direction. At this time, whether a texture map is skipped and information on the reference texture map direction will refer to the description in Fig. 27, and will be omitted here to avoid redundant explanation.

[0440] smh_texture_skip _flag is a flag indicating whether to skip texture maps per submesh.

[0441] refDirection_idx is an index indicating reference texture map direction information.

[0442] ref_texture_idx is the reference texture frame index.

[0443] Fig. 33 is a diagram showing another example of the syntax structure of a basemesh submesh header according to embodiments. That is, submesh_header() of Fig. 33 may include parameters (i.e., information) regarding texture map omission when the texture map omission unit is a submesh.

[0444] In Figure 33, afps_vdmc_texture_skip_flag is a flag indicating whether to skip texture maps on a frame-by-frame basis.

[0445] afps_vdmc_sm_skip_enable_flag is a flag that indicates whether to parse the texture map skip flag per sub-mesh unit.

[0446] sm_index is the submesh index.

[0447] smh_texture_skip_refDirection_idx is an index (i.e., submesh unit) indicating whether the texture map is skipped and the reference texture map direction information. At this time, whether the texture map is skipped and the reference texture map direction information refer to the description in Fig. 27, and are omitted here to avoid redundant description.

[0448] smh_texture_skip _flag is a flag indicating whether to skip texture maps per submesh.

[0449] refDirection_idx is an index indicating reference texture map direction information.

[0450] ref_texture_idx is the reference texture frame index.

[0451] The following describes the operation of a transmitting device and the operation of a receiving device when information regarding texture map omission is signaled in units of frames, tiles, patches, and sub-mesh as in FIGS. 26 to 33. Here, the transmitting device may be any one of the transmitting device (100) or mesh video encoder (102) of FIG. 1, the encoder (pre-processor and encoder) of FIG. 2, FIG. 6, or FIG. 7, the transmitting device of FIG. 13, or the transmitting device of FIG. 15. The present disclosure will describe the transmitting device of FIG. 15 as an example. And, the receiving device may be any one of the receiving device (110) or mesh video decoder (113) of FIG. 1, the decoder of FIG. 11 or FIG. 12, the receiving device of FIG. 14, or the receiving device of FIG. 20. The present disclosure will describe the receiving device of FIG. 20 as an example.

[0452] That is, in the transmitting device, the original mesh to be transmitted is generated as a base mesh through the mesh simplification unit (11011) and the mesh parameterization unit (11012). The generated base mesh is quantized in the mesh quantization unit (11013). Then, in the case of an inter-frame, a motion vector is calculated from the previously referenced restored base mesh in the motion vector encoder (11015) and the motion vector is encoded, and in the case of an intra-frame, the motion vector is transmitted as a base mesh bitstream through the static mesh encoder (11016). Then, the mesh simplified through the mesh simplification unit is refined and fitted through the mesh refinement unit (11018) and the mesh fitting unit (11019). The displacement vector calculation unit (11020) calculates a displacement vector between the refined and fitted mesh data and the mesh data restored from the previously encoded base mesh. In order to efficiently encode the calculated displacement vector, the displacement vector coordinate system transformation unit (11021) can transform the displacement vector coordinate system into a local coordinate system. Then, the displacement vector is transformed and quantized into a displacement vector coefficient in the displacement vector transformation unit within the displacement vector encoder (11022), and the 2D image packed through the displacement vector coefficient packing unit is encoded through a 2D video encoder such as H.264, HEVC, or VVC to generate a displacement vector bitstream.

[0453] The texture map generation unit (11026) generates a new texture map having color information corresponding to the texture coordinates of the restored mesh, and encodes the texture map through a 2D video encoder and transmits it as a texture bitstream.

[0454] In the present disclosure, in the process of generating a texture map and encoding it into a texture map bitstream, whether to omit coding of the texture map is determined through a texture map omission decision unit (11027), and the remaining texture maps except for the texture maps determined to be omitted coding can be encoded. Here, the texture maps determined to be omitted coding are not coded or transmitted.

[0455] The present disclosure can determine whether to encode / decode a texture map in a texture map skipping determination unit (11027) after generating a texture map using restored mesh data. At this time, whether to encode / decode the texture map can be determined by explicitly signaling a texture map skipping determination flag (texture_skip_flag), or whether to skip can be implicitly derived according to a condition defined in the same manner in an encoder / decoder. In addition, when explicitly signaling reference texture map information (i.e., reference information), refDirection_idx indicating reference texture direction information (past or future) can be determined and signaled, and / or ref_texture_idx indicating an index of a reference texture map can be signaled. And, the encoder / decoder can use the same texture map maximum distance parameter (max_skip_texturemap_distance) to decide whether to skip the texture map, and the texture map maximum distance parameter can be transmitted to the decoder, or it can be derived in the decoder (i.e., the texture map decoder) using the same texture map maximum distance parameter between the encoder / decoder.

[0456] According to embodiments, the criteria for determining whether to omit a texture map may include the similarity between the current texture map and the reference texture map, the amount of prediction bits generated, the distance between the current texture map and the reference texture map, and the like. In addition, the omission unit of texture information (i.e., the texture map) may be a frame, a submesh, a patch, a tile, a slice, and the like. Depending on the texture map omission unit, the texture map omission flag may be signaled as afps_vdmc_texture_skip_flag (frame unit), smh_texture_skip_flag (submesh unit), atdu_texture_skip_flag (patch unit), and ath_texture_skip_flag (tile unit).

[0457] As described above, among the texture maps generated through the texture map generation unit (11026), texture maps with texture_skip_flag set to 1 may be skipped from encoding through the texture map omission determination unit (11027). In contrast, texture maps with texture_skip_flag set to 0 may be encoded through a 2D video encoder or the like in the texture map encoder (11028).

[0458] Then, the base mesh bitstream, displacement vector bitstream, and texture bitstream generated through the entire process described above in the transmitting device are generated as one bitstream through a multiplexer and transmitted to the receiving device through the transmitting unit.

[0459] Meanwhile, the receiving device receives the bitstream transmitted from the transmitting device and performs a process of decoding the base mesh bitstream, displacement vector bitstream, and texture map bitstream, respectively, through a demultiplexer.

[0460] First, the base mesh bitstream is decoded through a motion vector decoder (15012) for inter-frames and a static mesh decoder (15013) for intra-frames. The decoded base mesh is then processed through a base mesh restoration unit (15014) and a mesh refinement unit (15015) for mesh refinement.

[0461] The displacement vector bitstream is decoded in the reverse order of encoding in the displacement vector decoder (15017) and restored to displacement vector coefficients, and the restored displacement vector coefficients are inversely quantized and inversely transformed through the displacement vector inverse quantization unit (15018) and the displacement vector inverse transformation unit (15019). Then, the displacement vector coordinate system inverse transformation unit (15020) inversely transforms the coordinate system of the inversely transformed displacement vector from the local coordinate system to the Cartesian coordinate system to restore the mesh geometry information together with the base mesh data.

[0462] In the mesh restoration unit (15016), the vertex geometry information of the restoration mesh is calculated by adding a restoration displacement vector to the vertices generated through the subdivision process in the mesh refinement unit (15015), thereby restoring the final geometry information.

[0463] The texture map bitstream is restored into a texture map through a texture map decoder (15021). The operation process of the texture map decoder (15021) in the present disclosure is described in more detail below.

[0464] That is, when a texture map bitstream is input to a texture map decoder (15021), first, a texture skip flag determination unit (16011) parses and determines texture_skip_flag, a 1-bit flag indicating whether to skip texture maps of each unit determined by the transmission device. At this time, depending on the texture skip unit, the texture map skip flag can be parsed as afps_vdmc_texture_skip_flag (frame unit), smh_texture_skip_flag (submesh unit), atdu_texture_skip_flag (patch unit), and ath_texture_skip_flag (tile unit).

[0465] In the case where the texture map omission unit according to the embodiments is a frame unit, whether or not to omit the texture map and reference information may be parsed separately for each frame, or may be parsed as a single index. When parsed as a single index, afps_texture_skip_refDirection_idx may be parsed. When parsed separately, after parsing the afps_vdmc_texture_skip_flag flag indicating whether or not to omit the texture map for each frame, if the flag is 0, the afps_vdmc_sm_skip_enable_flag flag indicating whether or not to perform texture map omission for each sub-mesh within the frame may be parsed. If afps_vdmc_texture_skip_flag is 1, the refDirection_idx index indicating the direction of the reference texture map may be parsed, and / or the ref_texture_idx index indicating the index of the reference texture map may be parsed.

[0466] In the case where the texture map omission unit according to the embodiments is a tile, whether or not to omit the texture map and the reference information may be parsed separately for each tile, or may be parsed as a single index. When parsed as a single index, ath_texture_skip_refDirection_idx may be parsed. When parsed separately, after parsing the ath_texture_skip_flag flag indicating whether or not to omit the texture map for each tile, if the flag is 1, the refDirection_idx index indicating the direction of the reference texture map may be parsed, and / or the ref_texture_idx index indicating the index of the reference texture map may be parsed.

[0467] In the case where the texture map omission unit according to the embodiments is a patch, whether or not to omit the texture map and the reference information may be parsed separately for each patch, or may be parsed as a single index. Whether or not to omit the texture map and the reference information may be parsed separately for each patch, or may be parsed as a single index. In the case of parsing separately, after parsing the atdu_texture_skip_flag flag indicating whether or not to omit the texture map for each patch, if the flag is 1, the refDirecion_idx index indicating the direction of the reference texture map may be parsed, and / or the ref_texture_idx index indicating the index of the reference texture map may be parsed.

[0468] In the case where the texture map omission unit according to the embodiments is a submesh, whether or not to omit the texture map and the reference information may be parsed separately for each submesh unit, or may be parsed as a single index. In the case of parsing as a single index, smh_texture_skip_refDirection_idx may be parsed. In the case of parsing separately, after parsing the smh_texture_skip_flag flag indicating whether or not to omit the texture map for each submesh unit, if the flag is 1, the refDirection_idx index indicating the direction of the reference texture map may be parsed, and / or the ref_texture_idx index indicating the index of the reference texture map may be parsed. Alternatively, in each case of parsing, after parsing the afps_vdmc_texture_skip_flag flag from the Atlas frame parameter set (AFPS), if the flag is 0, the afps_vdmc_sm_skip_enable_flag flag, which indicates whether to perform texture map skipping for each sub-mesh unit within the frame, can be parsed. If afps_vdmc_sm_skip_enable_flag is 1, the smh_texture_skip_flag flag, which indicates whether to skip texture maps for each sub-mesh unit within the frame, can be parsed. If smh_texture_skip_flag is 1, the refDirection_idx index indicating the direction of the reference texture map may be parsed, and / or the ref_texture_idx index indicating the index of the reference texture map may be parsed.

[0469] In addition, the encoder / decoder can use the same texture map maximum distance parameter (max_skip_texturemap_distance) to derive whether to skip the texture map. The texture map maximum distance parameter (max_skip_texturemap_distance) is a parameter that derives whether to skip the texture map, and by omitting the texture map skipping flag, the bits that need to be signaled can be saved. If the number of texture maps that are skipped consecutively (curSkipCount) is greater than or equal to the texture map maximum distance parameter (max_skip_texturemap_distance), the texture map skipping flag of the corresponding texture map can be derived as 0. If the number of texture maps that are skipped consecutively (curSkipCount) is less than the texture map maximum distance parameter (max_skip_texturemap_distance), the texture_skip_flag can be obtained through the texture skipping flag parsing unit or the texture skipping flag derivation unit. And, if texture_skip_flag is 1, curSkipCount can be increased by 1, and if texture_skip_flag is 0, curSkipCount can be initialized to 0.

[0470] After restoring the texture map through the above process, the final restoration mesh is created along with the previously restored geometry information.

[0471] In this way, in the present disclosure, if the similarity between the reference texture map of the texture map and the current texture map is high, the current texture map coding can be omitted. At this time, the texture map coding omission unit can be a frame, a submesh, a patch, a tile, a slice, etc. In other words, the texture map coding can be omitted by judging the similarity by the omission unit. In general, among the base mesh, displacement vector, and texture map, which are the main components of mesh data, the size of the bit capacity occupied by the texture map is large, accounting for about 60 to 80%, although the size may vary depending on the mesh content and transmission conditions. The present disclosure can obtain a significant effect in terms of bit savings and transmission speed by omitting the coding and transmission for a part of the texture map with a large proportion.

[0472] FIG. 34 is a diagram showing another example of the syntax structure of an atlas sequence parameter set (ASPS) among signaling information in a bitstream according to embodiments. In particular, FIG. 34 is a diagram showing an example of the structure of an atlas sequence parameter set vdmc extension RBSP syntax and semantics. That is, asps_vdmc_extension() of FIG. 34 may include a flag indicating whether to use a texture map omission method technique for each sequence and whether there are frames to be omitted within the sequence. That is, the present disclosure may transmit to a receiving device a flag indicating whether to use a texture map omission method technique for each sequence and whether there are frames to be omitted within the sequence by signaling it through the Atlas Sequence Parameter set vdmc extension.

[0473] In Fig. 34, asve_textureSkip_enable_flag is a flag that determines whether the texture map skipping method is available for each sequence. If the value of asve_textureSkip_enable_flag is 1, the texture map skipping technique can be performed, and if the value of asve_textureSkip_enable_flag is 0, the texture map skipping technique is not used.

[0474] asve_textureSkip_present_flag is a flag indicating whether there are frames to be skipped within the sequence. If the value of asve_textureSkip_present_flag is 1, it means that there are frames to be skipped within the sequence, and if the value of asve_textureSkip_present_flag is 0, it means that there are no frames to be skipped within the sequence.

[0475] According to embodiments, the present disclosure may signal and transmit to a receiving device information about a flag (texturemap_skip_flag) indicating that a texture map is skipped on a frame-by-frame basis and a reference direction (ref_direction_idx) of the skipped texture map through one of AFPS, tile header, patch unit header, and Nal unit header. At this time, textureSkip_present_flag may have the same meaning as texture_skip_flag of FIGS. 26 to 33. And, ref_direction_idx may have the same meaning as refDirction_idx of FIGS. 26 to 33.

[0476] FIG. 35 is a diagram showing another example of the syntax structure of an atlas frame parameter set (AFPS) among signaling information in a bitstream according to embodiments. In particular, FIG. 35 is a diagram showing an example of the structure of an atlas frame parameter set vdmc extension RBSP syntax and semantics. That is, afps_vdmc_extension() of FIG. 35 may include a flag indicating that a texture map is omitted for each frame (texturemap_skip_flag) and information on the reference direction of the omitted texture map (ref_direction_idx). In other words, a flag indicating that a texture map referencing the atlas Frame Parameter set V-DMC extension is omitted and information on the reference direction of the omitted texture map may be signaled.

[0477] In Fig. 35, afve_texturemap_skip_flag is a flag indicating that the texture map of the current frame is skipped. If the value of afve_texturemap_skip_flag is 1, it means that the frame has skipped the texture map, and if the value of afve_texturemap_skip_flag is 0, it means that the frame does not have skipped the texture map.

[0478] afve_ref_direction_idx represents the reference direction information of the omitted texture map.

[0479] Figure 36 is a table showing the definition of afve_texturemap_skip_flag and afve_ref_direction_idx according to embodiments.

[0480] For example, if the value of afve_ref_direction_idx is 0, it refers only to past frames, if the value of afve_ref_direction_idx is 1, it refers only to future frames, and if the value of afve_ref_direction_idx is 2, it refers to frames in both directions.

[0481] Also, when the value of afve_texturemap_skip_flag is 0, the value of afve_ref_direction_idx may be absent or filled in with a promised value.

[0482] Fig. 37 is a diagram showing an example of texture map omission according to embodiments. That is, it is an example showing syntax information when related information is signaled via AFPS when texture map coding is omitted.

[0483] Taking Fig. 37 as an example, frame indices 0, 2, 3, and 5 are indices of non-skip frames, and frame indices 1 and 4 are indices of skipped frames. In this case, Fig. 37 shows that when generating a frame with frame index 1 in the texture map decoder of the receiving device, a past frame (frame index 0) is referenced, and when generating a frame with frame index 4, a future frame (frame index 5) is referenced.

[0484] FIG. 38(a) and FIG. 38(b) are diagrams showing an example of determining whether to omit a texture map and a reference information index on a frame-by-frame basis according to embodiments. In particular, FIG. 38(a) is a table showing an example of whether to omit a texture map and a reference information determined on a frame-by-frame basis, and FIG. 38(b) is a diagram showing an example of syntax information according to an atlas frame parameter set identifier (atlas_frame_parameter_set_id).

[0485] Taking Fig. 38(a) as an example, if the 1st frame and the 4th frame are omitted (the values ​​of frameIdx are 1 and 4), afve_texturemap_skip_flag can be determined as 1, and if the 1st frame refers to a past frame, afve_ref_direction_idx can be determined as 0, and if the 4th frame refers to a future frame, afve_ref_direction_idx can be determined as 1.

[0486] Also, since the 0th, 2nd, 3rd, and 5th frames are not skipped, afve_texturemap_skip_flag can be determined as 0.

[0487] Figure 38(b) shows an example of an AFPS added depending on whether the texture map is omitted and the combination of cases selected from the reference information.

[0488] For example, if the texture map is not skipped (afve_texturemap_skip_flag=0), it can be mapped to AFPS ID 0. If the texture map is skipped (afve_texturemap_skip_flag=1) and the reference index is 0, it can be mapped to AFPS ID 1. Also, if the texture map is skipped (afve_texturemap_skip_flag=1) and the reference index is 1, it can be mapped to AFPS ID 2.

[0489] As shown in FIG. 37, FIG. 38(a), and FIG. 38(b), syntax information changes depending on whether texture map coding is omitted and the texture map reference direction. Whenever syntax information changes, the value of afps_atlas_frame_parameter_set_id can be increased by one to signal new syntax information for distinction. For example, if there is no matching set among the AFPS sets already created in the afps_frame_tile_information syntax, a new AFPS can be added.

[0490] According to embodiments, whether to omit texture maps and reference information for frame derivation can be signaled and transmitted on a tile-by-tile basis through an atlas tile header.

[0491] FIG. 39 is a diagram showing an example of a syntax structure of an atlas tile layer according to embodiments. According to embodiments, an atlas tile layer may include an atlas tile header (atlas_tile_header()) and an atlas tile data unit (atlas_tile_data_unit()).

[0492] FIG. 40 is a diagram showing an example of the syntax structure of an atlas tile header (atlas_tile_header()) according to embodiments.

[0493] In Fig. 40, ath_texture_skip_flag is a flag indicating whether a texture map is omitted. For example, if the value of ath_texture_skip_flag is 1, it indicates that the corresponding texture map is omitted, and if the value of ath_texture_skip_flag is 0, it indicates that the corresponding texture map is not omitted.

[0494] ath_ref_direction_idx is an index indicating the reference direction in the process of deriving an omitted texture map. For example, if the value of ath_ref_direction_idx is 0, it indicates that the texture map in the past direction is referenced, if the value of ath_ref_direction_idx is 1, it indicates that the texture map in the future direction is referenced, and if the value of ath_ref_direction_idx is 2, it indicates that the texture map in both directions is referenced.

[0495] According to embodiments, whether to omit a texture map and reference information for frame derivation may be signaled and transmitted on a patch basis via a patch data unit and / or an inter-patch data unit.

[0496] FIG. 41 is a diagram showing an example of a syntax structure of patch information data (patch_information_data(tileID, patchIdx, patchMode)) according to embodiments.

[0497] The patch information data (patch_information_data(tileID, patchIdx, patchMode)) of FIG. 41 may include a patch data unit (patch_data_unit(tileID, patchIdx)) if ath_type indicates P_TILE and patchMode indicates P_INTRA. In addition, if ath_type indicates P_TILE and patchMode indicates P_INTER, it may include an inter patch data unit (inter_patch_data_unit(tileID, patchIdx)). In addition, if ath_type indicates I_TILE and patchMode indicates I_INTRA, it may include a patch data unit (patch_data_unit(tileID, patchIdx)).

[0498] Fig. 42 is a diagram showing an example of a syntax structure of a patch data unit according to embodiments. That is, when the tile type is P_TILE and the patch mode is P_INTRA, or when the tile type is I_TILE and the patch mode is I_INTRA, the patch mode data unit syntax as in Fig. 42 is transmitted to the receiving device. In Fig. 42, when asve_textureSkip_enable_flag and asve_textureSkip_present_flag are 1 and the tile type is not I_TILE, the texture map decoder can parse pdu_texture_skip_flag.

[0499] In Fig. 42, pdu_texture_skip_flag is a flag indicating whether a texture map is omitted. For example, if the value of pdu_texture_skip_flag is 1, it indicates that the corresponding texture map is omitted, and if the value of pdu_texture_skip_flag is 0, it indicates that the corresponding texture map is not omitted.

[0500] pdu_ ref_direction_idx is an index indicating the reference direction in the process of deriving an omitted texture map. For example, if the value of pdu_ ref_direction_idx is 0, it indicates that the texture map in the past direction is referenced, if the value of pdu_ ref_direction_idx is 1, it indicates that the texture map in the future direction is referenced, and if the value of pdu_ ref_direction_idx is 2, it indicates that the texture map in both directions is referenced.

[0501] Fig. 43 is a diagram showing an example of a syntax structure of an inter-patch data unit according to embodiments. That is, when the tile type is P_TILE and the patch mode is I_INTER, an inter-patch data unit syntax as in Fig. 43 is transmitted to a receiving device. In Fig. 42, when asve_textureSkip_enable_flag and asve_textureSkip_present_flag are 1, the texture map decoder can parse ipdu_texture_skip_flag.

[0502] In Fig. 43, ipdu_texture_skip_flag is a flag indicating whether a texture map is omitted. For example, if the value of ipdu_texture_skip_flag is 1, it indicates that the corresponding texture map is omitted, and if the value of ipdu_texture_skip_flag is 0, it indicates that the corresponding texture map is not omitted.

[0503] ipdu_ref_direction_idx is an index indicating the reference direction in the process of deriving an omitted texture map. For example, if the value of ipdu_ref_direction_idx is 0, it indicates that the texture map in the past direction is referenced, if the value of ipdu_ref_direction_idx is 1, it indicates that the texture map in the future direction is referenced, and if the value of ipdu_ref_direction_idx is 2, it indicates that the texture map in both directions is referenced.

[0504] According to embodiments, by adding atlas_frame_rbsp(), a structure that transmits frame unit information to an existing NAL unit parsing structure, a flag for whether a texture map is omitted and a reference information index for deriving an omitted frame can be transmitted to a receiving device.

[0505] Figure 44 is a diagram showing an example of NAL unit header semantics according to embodiments. That is, examples are shown where atlas_frame_rbsp() is included in the NAL unit header.

[0506] FIG. 45 is a diagram showing an example of the syntax structure of atlas_frame_rbsp() according to embodiments. According to embodiments, when asve_textureSkip_enable_flag and asve_textureSkip_present_flag are 1, the texture map decoder can parse af_texture_skip_flag.

[0507] In Fig. 45, af_texture_skip_flag is a flag indicating whether a texture map is omitted. For example, if the value of af_texture_skip_flag is 1, it indicates that the corresponding texture map is omitted, and if the value of af_texture_skip_flag is 0, it indicates that the corresponding texture map is not omitted.

[0508] af_ref_direction_idx is an index indicating the reference direction in the process of deriving an omitted texture map. For example, if the value of af_ref_direction_idx is 0, it indicates that the texture map in the past direction is referenced, if the value of af_ref_direction_idx is 1, it indicates that the texture map in the future direction is referenced, and if the value of af_ref_direction_idx is 2, it indicates that the texture map in both directions is referenced.

[0509] At this time, textureSkip_present_flag of FIGS. 34 to 45 may have the same meaning as texture_skip_flag of FIGS. 26 to 33. And, ref_direction_idx of FIGS. 34 to 45 may have the same meaning as refDirction_idx of FIGS. 26 to 33.

[0510] The following describes the operation of a transmitting device and the operation of a receiving device when a texture map omission flag and a reference information index for deriving an omitted frame are signaled to at least one of ASPS, AFPS, atlas tile header, patch data unit, inter patch data unit, and atlas_frame_rbsp as shown in FIGS. 34 to 45. Here, the transmitting device may be any one of the transmitting device (100) or mesh video encoder (102) of FIG. 1, the encoder (pre-processor and encoder) of FIG. 2, FIG. 6, or FIG. 7, the transmitting device of FIG. 13, or the transmitting device of FIG. 15. The present disclosure will describe the transmitting device of FIG. 15 as an example. And, the receiving device may be any one of the receiving device (110) or mesh video decoder (113) of FIG. 1, the decoder of FIG. 11 or FIG. 12, the receiving device of FIG. 14, or the receiving device of FIG. 20. The present disclosure will be described using the receiving device of FIG. 20 as an example.

[0511] That is, in the transmitting device, the original mesh to be transmitted is generated as a base mesh through the mesh simplification unit (11011) and the mesh parameterization unit (11012). The generated base mesh is quantized in the mesh quantization unit (11013). Then, in the case of an inter-frame, a motion vector is calculated from the previously referenced restored base mesh in the motion vector encoder (11015) and the motion vector is encoded, and in the case of an intra-frame, the motion vector is transmitted as a base mesh bitstream through the static mesh encoder (11016). Then, the mesh simplified through the mesh simplification unit is refined and fitted through the mesh refinement unit (11018) and the mesh fitting unit (11019). The displacement vector calculation unit (11020) calculates a displacement vector between the refined and fitted mesh data and the mesh data restored from the previously encoded base mesh. In order to efficiently encode the calculated displacement vector, the displacement vector coordinate system transformation unit (11021) can transform the displacement vector coordinate system into a local coordinate system. Then, the displacement vector is transformed and quantized into a displacement vector coefficient in the displacement vector transformation unit within the displacement vector encoder (11022), and the 2D image packed through the displacement vector coefficient packing unit is encoded through a 2D video encoder such as H.264, HEVC, or VVC to generate a displacement vector bitstream.

[0512] The texture map generation unit (11026) generates a new texture map having color information corresponding to the texture coordinates of the restored mesh.

[0513] Afterwards, it is possible to determine whether to apply the texture map omission technology to the current content or sequence unit, and accordingly, it is determined whether the texture map omission decision unit (11027) will be performed. If the texture map omission technology is applied, the value of asve_textureSkip_enable_flag can be signaled as 1 through the ASPS V-DMC extension to transmit whether the texture map omission technology is applied and the texture map-related parameters to the receiving device.

[0514] The texture map omission decision unit (11027) determines whether to omit the coding of the texture map of the current frame. When the difference between the result of calculating the PSNR using the metrics of the original mesh and the current mesh and the result of calculating the PSNR using the metrics of the original mesh and the reference mesh (i.e., the difference between the two PSNRs) is less than or equal to a certain threshold value, the texture map omission can be determined. If the texture map omission exists in the current sequence, the value of asve_textureSkip_present_flag can be signaled as 1 to transmit information to the receiving device that the omitted texture exists in the current sequence and that restoration of the omitted texture map is necessary. In addition, when the omission of the texture map coding of the current frame is determined, the texture map coding omission flag and reference direction information, etc. can be signaled. The present disclosure proposes four signaling methods for the syntax (i.e., information) related to the texture map coding omission on a frame-by-frame basis, as follows.

[0515] In the first method, if it is determined that the texture map coding is omitted, it can be signaled through the V-DMC extension of AFPS as shown in FIG. 34. Information on whether the texture map of the current frame is omitted (afve_texturemap_skip_flag) can be signaled. At this time, if the value of afve_texturemap_skip_flag is 1, it means that the texture map of the current frame is omitted, and if the value of afve_texturemap_skip_flag is 0, it means that it is not omitted. In addition, if the texture map is omitted, the reference direction (afve_ref_direction_idx) can be signaled. At this time, if the value of afve_ref_direction_idx is 0, it can refer to a past frame, if the value of afve_ref_direction_idx is 1, it can refer to a future frame, and if the value of afve_ref_direction_idx is 2, it can refer to frames in both directions. If the texture map is not skipped, the reference direction information (afve_ref_direction_idx) may not have a value and may be assigned or filled with a promised value (e.g. afve_ref_direction_idx=3). In addition, if the afve_texturemap_skip_flag and afve_ref_direction_idx information or values ​​in the AFPS V-DMC extension are different, afps_atlas_frame_parameter_set_id may be assigned and signaled differently, such as 0, 1, or 2.

[0516] As a second method, parameters related to texture map omission can be signaled and transmitted to the receiving device through atlas_tile_header as in FIG. 40. If the tile type is not I type and the texture map is omitted, information on whether the texture map of the current frame is omitted (ath_texture_skip_flag) can be signaled. At this time, if the value of ath_texture_skip_flag is 1, it means that the texture map of the current frame is omitted, and if the value of ath_texture_skip_flag is 0, it means that it is not omitted. In addition, if the texture map is omitted, the reference direction (ath_ref_direction_idx) can be signaled. At this time, if the value of ath_ref_direction_idx is 0, it can refer to a past frame, if the value of ath_ref_direction_idx is 1, it can refer to a future frame, and if the value of ath_ref_direction_idx is 2, it can refer to frames in both directions. If the texture map is not omitted, the reference direction information (ath_ref_direction_idx) may not have a value and may be assigned or filled with a promised value (e.g. ath_ref_direction_idx=3).

[0517] The third way is to signal and transmit texture map skip related parameters through patch_data_unit. When the tile type is P_TILE and the patch mode is P_INTRA, or when the tile type is I_TILE and the patch mode is I_INTRA, the texture map skip information (pdu_texture_skip_flag) and reference direction information (pdu_ref_direction_idx) of the current frame can be transmitted to the receiving device through patch data unit() as shown in FIG. 42. When the tile type is P_TILE and the patch mode is I_INTER, the texture map skip information (ipdu_texture_skip_flag) and reference direction information (ipdu_ref_direction_idx) of the current frame can be transmitted through inter patch data unit() as shown in FIG. 43.

[0518] As a fourth method, parameters related to texture map omission can be signaled and transmitted through Nal_unit_header as shown in FIGS. 44 and 45. The present disclosure newly adds a structure atlas_frame_rbsp() for transmitting frame unit information as shown in FIG. 44 to the existing nal unit parsing structure, and through this, a texture map omission flag (af_texture_skip_flag) and reference direction information (af_ref_direction_flag) can be signaled and transmitted to a receiving device as shown in FIG. 45. Similarly, if the value of af_texture_skip_flag is 1, it may mean that the texture map of the current frame is omitted, and if the value of af_texture_skip_flag is 0, it may mean that it is not omitted. Additionally, if the value of af_ref_direction_flag is 0, it can mean that it refers to a past frame, if the value of af_ref_direction_flag is 1, it can mean that it refers to a future frame, and if the value of af_ref_direction_flag is 2, it can mean that it refers to a bidirectional frame. If the texture map is not omitted, the reference direction information (af_ref_direction_idx) may not have a value, and may be assigned or filled with a promised value (e.g. af_ref_direction_idx=3).

[0519] As described above, among the texture maps generated through the texture map generation unit (11026), a texture map with texture_skip_flag set to 1 may be omitted from encoding and transmission through the texture map omission determination unit (11027). In contrast, a texture map with texture_skip_flag set to 0 may be encoded through a 2D video encoder or the like in the texture map encoder (11028).

[0520] Then, the base mesh sub-bitstream, displacement vector sub-bitstream, and texture map sub-bitstream generated through the entire process described above in the transmitting device are generated as one bitstream through a multiplexer and transmitted to the receiving device through the transmitting unit.

[0521] Meanwhile, the receiving device receives the bitstream transmitted from the transmitting device and performs a process of decoding the base mesh bitstream, displacement vector bitstream, and texture map bitstream, respectively, through a demultiplexer.

[0522] First, the base mesh bitstream is decoded through a motion vector decoder (15012) for inter-frames and a static mesh decoder (15013) for intra-frames. The decoded base mesh is then processed through a base mesh restoration unit (15014) and a mesh refinement unit (15015) for mesh refinement.

[0523] The displacement vector bitstream is decoded in the reverse order of encoding in the displacement vector decoder (15017) and restored to displacement vector coefficients, and the restored displacement vector coefficients are inversely quantized and inversely transformed through the displacement vector inverse quantization unit (15018) and the displacement vector inverse transformation unit (15019). Then, the displacement vector coordinate system inverse transformation unit (15020) inversely transforms the coordinate system of the inversely transformed displacement vector from the local coordinate system to the Cartesian coordinate system to restore the mesh geometry information together with the base mesh data.

[0524] In the mesh restoration unit (15016), the vertex geometry information of the restoration mesh is calculated by adding a restoration displacement vector to the vertices generated through the subdivision process in the mesh refinement unit (15015), thereby restoring the final geometry information.

[0525] The texture map bitstream is restored into a texture map through a texture map decoder (15021). The operation process of the texture map decoder (15021) in the present disclosure is described in more detail below.

[0526] First, the texture map decoder (15021) can determine whether the texture map skipping technique is applied to the current content or sequence by parsing asve_textureSkip_enable_flag from the ASPS. If the value of asve_textureSkip_enable_flag is 1 and the texture map skipping technique is applied, a process of restoring the omitted texture map is required, and for this purpose, the texture map decoder (15021) can receive parameters related to texture map skipping. In addition, if asve_texture_present_flag is parsed and its value is 1, it can be determined that some texture maps are omitted in the texture map bitstream transmitted in units of sequences, and a process of restoring the omitted texture maps can be performed. In the present disclosure, a method of parsing parameters required for restoration to perform restoration of omitted texture maps in units of frames is as follows.

[0527] First, by commonly parsing asve_textureSkip_enable_flag of ASPS, if its value is 1, we can know that the texture map coding skipping technology is applied to the current content or sequence. Then, by parsing asve_texture_present_flag, if its value is 1, we can know that an omitted texture map exists in the corresponding sequence. Afterwards, the related parameters for restoring the omitted texture map (or related information for restoring the omitted texture map) can be parsed according to each signaling method to perform texture map restoration.

[0528] First, when the relevant information for restoring the omitted texture map is transmitted via AFPS as shown in Fig. 35, afve_texturemap_skip_flag of the atlas frame parameter set V-DMC extension is parsed. Then, when the value of the parsed afve_texturemap_skip_flag is 1, it can be known that the texture map of the current frame is omitted. When the value of afve_texturemap_skip_flag is 0, it means that the texture map of the current frame is not omitted. If afve_texturemap_skip_flag is 1, the reference direction for restoring the omitted texture map can be known by parsing avfe_ref_direction_idx. If the value of avfe_ref_direction_idx is 0, it means to restore by referencing a past frame, if the value of avfe_ref_direction_idx is 1, it means to restore by referencing a future frame, and if the value of avfe_ref_direction_idx is 2, it means to restore by referencing frames in both directions. If the value of afve_texturemap_skip_flag is 0, the value of avfe_ref_direction_idx may be absent, or may be assigned a specific value or a value promised to the encoder. In addition, in the case of afps_vdmc_extension, if the configuration or value of the syntax (or field) is different, the syntax information can be signaled by assigning each set id.

[0529] Second, when the relevant information for restoring the omitted texture map is transmitted through the atlas tile header as in FIG. 40, when the type of the tile from atlas_tile_header() is not I_TILE, ath_texture_skip_flag is parsed, and if its value is 1, it can be known that the texture map of the current frame is omitted. If the value of ath_texture_skip_flag is 0, it means that the texture map of the current frame is not omitted. If ath_texturemap_skip_flag is 1, ath_ref_direction_idx is parsed to know the reference direction for restoring the omitted texture map. If the value of ath_ref_direction_idx is 0, it means that restoration is performed by referencing a past frame, if the value of ath_ref_direction_idx is 1, it means that restoration is performed by referencing a future frame, and if the value of ath_ref_direction_idx is 2, it means that restoration is performed by referencing frames in both directions. If the value of ath_texturemap_skip_flag is 0, the value of ath_ref_direction_idx may be absent, or may be assigned a specific value or a value promised to the encoder.

[0530] Third, when the relevant information for restoring the omitted texture map is transmitted via the patch data unit, it may differ depending on the types of the atlas and the patch in patch_information_data() as shown in FIG. 41. When the tile type is P type and the patch mode is P_INTRA, or when the atlas type is I type and the patch mode is I_INTRA, the pdu_texture_skip_flag of patch_data_unit() of FIG. 42 is parsed, and when its value is 1, it can be known that the texture map of the current frame is omitted. If the value of pdu_texture_skip_flag is 0, it means that the texture map of the current frame is not omitted. If the value of pdu_texture_skip_flag is 1, the reference direction for restoring the omitted texture map can be known by parsing pdu_ref_direction_idx. If the value of pdu_ref_direction_idx is 0, it means to restore by referencing a past frame, if the value of pdu_ref_direction_idx is 1, it means to restore by referencing a future frame, and if the value of pdu_ref_direction_idx is 2, it means to restore by referencing bidirectional frames. If the value of pdu_texture_skip_flag is 0, the value of pdu_ref_direction_idx may be absent, or may be assigned a specific value or a value promised to the encoder. When the tile type is P type and the patch mode is P_INTER, you can find out whether the texture map is skipped by parsing ipdu_texture_skip_flag of inter_patch_data_unit() in FIG. 43, and you can find out the texture map reference direction by parsing ipdu_ref_direction_idx. The meaning of each syntax is the same as that of pdu.

[0531] Fourth, when the relevant information for restoring the omitted texture map is transmitted through the NAL unit header, the existing NAL unit parsing structure parses the newly added atlas_frame_rbsp() as shown in Fig. 44. If the value of af_texture_skip_flag is 1, it can be known that the texture map of the current frame is omitted. If the value of af_texture_skip_flag is 0, it means that the texture map of the current frame is not omitted. If the value of af_texture_skip_flag is 1, the reference direction for restoring the omitted texture map can be known by parsing af_ref_direction_idx. If the value of af_ref_direction_idx is 0, it means that restoration is performed by referencing a past frame. If the value of af_ref_direction_idx is 1, it means that restoration is performed by referencing a future frame. If the value of af_ref_direction_idx is 2, it means that restoration is performed by referencing frames in both directions. If the value of af_texture_skip_flag is 0, the value of af_ref_direction_idx may be null, or may be assigned a specific value or a value promised to the encoder.

[0532] In this way, the texture map derivation unit (16014) of the texture map decoder (15021) can recognize the frame where the texture map is omitted based on the above methods, that is, based on texture_skip_flag and ref_direction_idx, obtain information about the reference frame, and perform restoration by copying the texture map of the omitted frame from the reference frame. In this case, in the case of bidirectional frame reference, restoration can be performed using the average value of the texture maps of the past frame and the future frame.

[0533] After restoring the texture map through the above process, the final restoration mesh is generated along with the previously restored final geometry information.

[0534] In this way, in the present disclosure, if the similarity between the reference texture map of the texture map and the current texture map is high, the current texture map coding is omitted. In addition, when the texture map decoder of the receiving device receives a texture map bitstream of an omitted texture map (i.e., a texture map whose coding and transmission are omitted), the transmitting device signals and transmits information related to the omitted texture map to the receiving device so that restoration of the omitted texture map can be performed. For example, in order to efficiently transmit syntax and drive the V-DMC codec in accordance with the existing V-DMC bitstream structure, a flag indicating whether to use texture map omission technology through ASPS and a flag indicating whether an omitted texture map exists in the sequence are signaled and transmitted. In addition, a frame-by-frame texture map omission status flag and reference information for deriving an omitted texture map are signaled for each syntax transmission position (Atlas Frame Parameter Set, Tile header, Patch data unit, NAL unit header) based on the atlas stream. Therefore, if texture map omission technology is applied to the V-DMC codec and a signaling method for omitted texture map information is provided, significant bit savings and fast transmission speed can be achieved.

[0535] Figure 46 is a flowchart illustrating an example of a transmission method according to embodiments. The transmission method according to embodiments may include a step of encoding mesh data (21011) and a step of transmitting a bitstream including the encoded mesh data (21012).

[0536] According to embodiments, the step (21011) of encoding mesh data omits coding of the current texture map when the similarity between the reference texture map of the texture map and the current texture map is high. At this time, the method of omitting the texture map by checking the similarity between the reference texture map of the texture map and the current texture map and the signaling of information related to the omitted texture map according to the method will refer to the descriptions of FIGS. 15 to 19 and FIGS. 26 to 45, and are omitted here to avoid redundant description.

[0537] Figure 47 is a flowchart illustrating an example of a receiving method according to embodiments. The receiving method according to embodiments may include a step (22011) of receiving a bitstream containing mesh data and a step (22012) of decoding the mesh data contained in the bitstream.

[0538] In the present disclosure, the step (22012) of decoding mesh data included in a bitstream performs decoding based on signaling information on a demultiplexed texture map bitstream to restore a texture map. At this time, some texture maps are omitted in the transmitted texture map bitstream, and the step (22012) of decoding mesh data performs a process of restoring the omitted texture maps based on the signaling information. At this time, the method of restoring the omitted texture map and the signaling of the omitted texture map-related information according to the method will refer to the descriptions of FIGS. 20 to 45, and are omitted here to avoid redundant description.

[0539] Each of the parts, modules, or units described above may be software, processors, or hardware parts that execute sequential execution processes stored in memory (or storage units). Each of the steps described in the embodiments described above may be performed by processors, software, or hardware parts. Each of the modules / blocks / units described in the embodiments described above may operate as a processor, software, or hardware. In addition, the methods presented in the embodiments may be implemented as code. This code may be written on a processor-readable storage medium and thus may be read by a processor provided by an apparatus.

[0540] Furthermore, throughout the specification, when a part is said to "include" a component, this does not exclude other components, unless otherwise specifically stated, but rather implies the inclusion of other components. Furthermore, terms such as "part" described in the specification mean a unit that processes at least one function or operation, which may be implemented using hardware, software, or a combination of hardware and software.

[0541] For convenience of explanation, this specification has been described separately in each drawing. However, it is also possible to design new embodiments by combining the embodiments described in each drawing. Furthermore, designing a computer-readable recording medium containing a program for executing the previously described embodiments, as required by those skilled in the art, is also within the scope of the embodiments.

[0542] The devices and methods according to the embodiments are not limited to the configurations and methods of the embodiments described above, but the embodiments may be configured by selectively combining all or part of each embodiment so that various modifications can be made.

[0543] Although preferred embodiments of the embodiments have been illustrated and described, the embodiments are not limited to the specific embodiments described above, and various modifications may be made by those skilled in the art to which the invention pertains without departing from the spirit or scope of the embodiments claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the embodiments.

[0544] The various components of the devices of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. The various components of the embodiments may be implemented by a single chip, for example, a single hardware circuit. The components according to the embodiments may be implemented by separate chips. At least one of the components of the devices of the embodiments may be configured with one or more processors capable of executing one or more programs, and the one or more programs may perform, or include instructions for performing, one or more of the operations / methods according to the embodiments. The executable instructions for performing the methods / operations of the devices of the embodiments may be stored in non-transitory CRMs or other computer program products configured to be executed by one or more processors, or may be stored in temporary CRMs or other computer program products configured to be executed by one or more processors. In addition, the memory according to the embodiments may be used as a concept including not only volatile memory (e.g., RAM, etc.), but also non-volatile memory, flash memory, PROM, etc. Additionally, it may include implementations in the form of carrier waves, such as transmissions via the Internet. Furthermore, processor-readable recording media may be distributed across network-connected computer systems, allowing processor-readable code to be stored and executed in a distributed manner.

[0545] In this document, " / " and "," are interpreted as "and / or". For example, "A / B" is interpreted as "A and / or B", and "A, B" is interpreted as "A and / or B". Additionally, "A / B / C" means "at least one of A, B, and / or C". Also, "A, B, C" means "at least one of A, B, and / or C". Additionally, "or" in this document is interpreted as "and / or". For example, "A or B" can mean 1) "A" only, 2) "B" only, or 3) "A and B". In other words, "or" in this document can mean "additionally or alternatively".

[0546] Various elements of the embodiments may be implemented by hardware, software, firmware, or a combination thereof. Various elements of the embodiments may be implemented on a single chip, such as a hardware circuit. In some embodiments, the embodiments may optionally be implemented on separate chips. In some embodiments, at least one of the elements of the embodiments may be implemented within one or more processors that include instructions for performing operations according to the embodiments.

[0547] Additionally, the operations according to the embodiments described in this document may be performed by a transceiver device including one or more memories and / or one or more processors according to the embodiments. One or more memories may store programs for processing / controlling the operations according to the embodiments, and one or more processors may control various operations described in this document. One or more processors may be referred to as a controller, etc. The operations according to the embodiments may be performed by firmware, software, and / or a combination thereof, and the firmware, software, and / or a combination thereof may be stored in a processor or a memory.

[0548] Terms such as "first" and "second" may be used to describe various components of the embodiments. However, the various components according to the embodiments should not be interpreted in a limited manner by these terms. These terms are merely used to distinguish one component from another. For example, a first user input signal may be referred to as a "second user input signal." Similarly, a second user input signal may be referred to as a "first user input signal." The use of these terms should be interpreted as not departing from the scope of the various embodiments. Although "first user input signal" and "second user input signal" are both user input signals, they do not necessarily mean the same user input signals unless the context clearly indicates otherwise.

[0549] The terminology used to describe the embodiments is for the purpose of describing particular embodiments and is not intended to be limiting of the embodiments. As used in the description of the embodiments and in the claims, the singular is intended to include the plural unless the context clearly dictates otherwise. The expressions “and / or” are used to mean all possible combinations of the terms. The expression “comprises” or “includes” describes the presence of features, numbers, steps, elements, and / or components, but does not mean that additional features, numbers, steps, elements, and / or components are not included. Conditional expressions such as “if” or “when” used to describe the embodiments are not intended to be limited to only optional cases. When a specific condition is satisfied, a related action is performed in response to a specific condition, or a related definition is intended to be interpreted.

[0550] As described above, the relevant contents have been described in the best form for carrying out the embodiments.

[0551] As described above, the embodiments may be applied, in whole or in part, to 3D data transmission and reception devices and systems. Those skilled in the art may variously modify or alter the embodiments within the scope of the embodiments. The embodiments may include modifications / alterations, and the modifications / alterations do not depart from the scope of the claims and their equivalents.

Claims

1. A step of receiving a base mesh bitstream, a displacement vector bitstream, a texture map bitstream, and signaling information; A base mesh processing step for restoring a base mesh from the above base mesh bitstream; A displacement information processing step for restoring displacement information from the displacement vector bitstream; A restoration step for restoring a mesh based on the base mesh and the displacement information; and A mesh data decoding method comprising a texture map processing step of restoring texture maps from the above texture map bitstream.

2. In the first paragraph, the texture map processing step A step of restoring texture maps by decoding the texture map bitstream based on the signaling information; A step of checking whether at least one texture map is omitted from the texture map bitstream based on the signaling information; and A mesh data decoding method comprising the step of generating the at least one omitted texture map based on the signaling information and at least one reference frame, if it is determined that there is at least one omitted texture map.

3. In paragraph 2, A mesh data decoding method wherein the omission unit of at least one texture map is at least one of a frame unit, a patch unit, a sub-mesh unit, a tile unit, and a slice unit.

4. In the second paragraph, the signaling information is A method for decoding mesh data, comprising information for identifying whether at least one texture map is omitted and reference texture map information related to at least one reference frame.

5. A receiving unit for receiving a base mesh bitstream, a displacement vector bitstream, a texture map bitstream, and signaling information; A base mesh processing unit for restoring a base mesh from the base mesh bitstream; A displacement information processing unit that restores displacement information from the displacement vector bitstream; A restoration unit that restores a mesh based on the base mesh and the displacement information; and A mesh data decoding device including a texture map processing unit for restoring a texture map from the above texture map bitstream.

6. In the fifth paragraph, the texture map processing unit A mesh data decoding device that restores texture maps by decoding the texture map bitstream based on the signaling information, determines whether at least one texture map is omitted from the texture map bitstream based on the signaling information, and, if it is determined that there is at least one omitted texture map, generates the at least one omitted texture map based on the signaling information and at least one reference frame.

7. In paragraph 6, A mesh data decoding device in which the omission unit of at least one texture map is at least one of a frame unit, a patch unit, a sub-mesh unit, a tile unit, and a slice unit.

8. In paragraph 6, the signaling information is A mesh data decoding device comprising information for identifying whether at least one texture map is omitted and reference texture map information related to at least one reference frame.

9. Step of encoding the original mesh; and A mesh data encoding method comprising the step of transmitting a bitstream including the encoded mesh and signaling information.

10. In the 9th paragraph, the encoding step A base mesh processing step for generating a base mesh bitstream by encoding a base mesh generated by simplifying the original mesh; A displacement information processing step of generating a displacement vector bitstream by encoding displacement information generated based on the above base mesh; A mesh restoration step for restoring a mesh based on the encoded base mesh and the encoded displacement information; and A mesh data encoding method comprising a texture map processing step of determining whether to omit at least one texture map among texture maps generated based on the original mesh and the restored mesh, and encoding texture maps that are not omitted to generate a texture map bitstream.

11. In the 10th paragraph, the texture map processing step A mesh data encoding method for determining whether to omit the current texture map by comparing the similarity between the current texture map and the reference texture map before encoding the generated texture maps.

12. In the 11th paragraph, the texture map processing step A mesh data encoding method that skips coding and transmission of the current texture map if the difference between the signal-to-noise ratio (PSNR) value calculated through the current texture map and the PSNR value calculated through the reference texture map is less than a preset threshold value.

13. In paragraph 10, A mesh data encoding method in which the omission unit of at least one texture map is at least one of a frame unit, a patch unit, a sub-mesh unit, a tile unit, and a slice unit.

14. In paragraph 13, the signaling information is A mesh data encoding method comprising information for identifying whether at least one texture map is omitted and reference texture map information related to at least one reference frame for generating the omitted texture map.

15. In paragraph 14, A mesh data encoding method wherein the above reference texture map information includes information for identifying the direction of at least one reference frame.

Images