Dynamic grid compression method and device

Abstract

A mesh compression method and apparatus according to an embodiment of the present invention may reconstruct a base mesh based on geometric information obtained from a base mesh bit stream, perform surface division on the reconstructed base mesh, obtain a displacement vector of at least one of a vertex used for the reconstruction of the base mesh and a sub-vertex generated by the surface division, and compress the base mesh based on the displacement vector. Reconstructing an enhancement grid based on the reconstructed base grid and the obtained displacement vector, and obtaining a texture image associated with a surface of the reconstructed enhancement grid.

Description

Technical Field

[0001] The present invention relates to methods and apparatus for video compression encoding and decoding, and more particularly, to methods and apparatus for video compression encoding and decoding using dynamic grid compression. Background Art

[0002] The video image is compressed and encoded by removing space-time redundancy and inter-viewpoint redundancy, and may be transmitted through a communication line or stored on a storage medium in an appropriate form. Summary of the invention

[0003] [Technical issues]

[0004] The present invention proposes a displacement vector prediction method and device for dynamic grid encoding and decoding.

[0005] In this case, the displacement vector prediction method proposed in the present invention is to predict the displacement vector by using the motion vector for inter-frame reference, thereby providing high encoding efficiency.

[0006] In addition, the present invention proposes a dynamic grid coding and decoding method and device based on frame groups for dynamic grid coding and decoding.

[0007] In this case, the dynamic mesh coding and decoding method in the frame group unit proposed in the present invention provides high coding efficiency by omitting encoding / decoding of the attribute map of the current frame and using the reconstructed attribute map within the same GOF.

[0008] [Technical solution]

[0009] In order to solve this problem, a dynamic mesh compression method and device using displacement vector prediction are proposed. In addition, a dynamic mesh compression method and device based on frame groups are proposed to solve this problem.

[0010] According to an embodiment of the present invention, a mesh compression method and apparatus can reconstruct a base mesh based on geometric information obtained from a base mesh bitstream, perform surface partitioning on the reconstructed base mesh, obtain displacement vectors of at least one of the vertices of the reconstructed base mesh and the sub-vertices generated by the surface partitioning, reconstruct an enhanced mesh based on the reconstructed base mesh and the obtained displacement vectors, and obtain a texture image associated with the surface of the reconstructed enhanced mesh.

[0011] In the grid compression method and apparatus according to an embodiment of the present invention, geometric information can be encoded and decoded based on a first prediction mode, displacement vectors can be encoded and decoded based on a second prediction mode, and texture images can be encoded and decoded based on a third prediction mode.

[0012] In the grid compression method and apparatus according to the embodiment of the present invention, the first prediction mode, the second prediction mode, and the third prediction mode may be one of an inter-frame prediction mode or an intra-frame prediction method.

[0013] In the grid compression method and apparatus according to the embodiment of the present invention, at least one of the first prediction mode, the second prediction mode, or the third prediction mode may be determined according to a predetermined coding unit.

[0014] In the mesh compression method and apparatus according to the embodiments of the present invention, the encoding and decoding unit may be at least one of a frame, a patch, a surface, or a vertex.

[0015] The mesh compression method and apparatus according to the embodiments of the present invention can reconstruct the current frame based on the reconstructed enhanced mesh and texture image.

[0016] The mesh compression method and apparatus according to the embodiment of the present invention can obtain a first flag indicating whether to use texture image compression for the current frame.

[0017] In the mesh compression method and apparatus according to the embodiment of the present invention, the first flag may be signaled in units of a frame group including at least one frame.

[0018] In the mesh compression method and apparatus according to the embodiment of the present invention, when texture image compression is used for the current frame, the texture image may be obtained by referring to the texture image of a frame encoded and decoded before the current frame.

[0019] The grid compression method and apparatus according to the embodiment of the present invention can obtain a second flag indicating whether to encode or decode the texture image of the current frame.

[0020] In the mesh compression method and apparatus according to the embodiment of the present invention, the second flag may be signaled in units of frames.

[0021] In a grid compression method and apparatus according to an embodiment of the present invention, when encoding and decoding of a texture image of a current frame is omitted, a texture image can be obtained by referring to a texture image of a frame whose picture order count (POC) value is closest to the current frame in a frame encoded and decoded before the current frame.

[0022] The mesh compression method and apparatus according to the embodiment of the present invention can obtain a texture index indicating a texture image in a texture video including a plurality of texture images.

[0023] According to an embodiment of the present invention, a mesh compression device may include a processor and a memory, wherein the processor controls the mesh compression device, the memory is combined with the processor and stores data, and the processor may: reconstruct a base mesh based on geometric information obtained from a base mesh bitstream, perform surface partitioning on the reconstructed base mesh, obtain displacement vectors of at least one of the vertices of the reconstructed base mesh and the sub-vertices generated by the surface partitioning, reconstruct an enhanced mesh based on the reconstructed base mesh and the obtained displacement vectors, and obtain a texture image associated with the surface of the reconstructed enhanced mesh.

[0024] [Technical Effect]

[0025] According to the displacement vector prediction method proposed in the present invention, high encoding efficiency can be provided by using a motion vector for inter-frame reference to predict a displacement vector.

[0026] According to the dynamic grid coding and decoding method in the frame group unit proposed in the present invention, high coding efficiency can be provided by omitting encoding / decoding of the attribute map of the current frame and using the reconstructed attribute map within the same group of frames (GOF). BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Examples of trellis encoders and decoders according to embodiments of the present disclosure are shown.

[0028] Figure 2 An example of a base mesh geometry information encoding unit according to an embodiment of the present disclosure is shown.

[0029] Figure 3 An example of an enhanced mesh geometry information encoding unit according to an embodiment of the present disclosure is shown.

[0030] Figure 4 An example of a mesh attribute information encoding unit according to an embodiment of the present disclosure is shown.

[0031] Figure 5 An example of a base mesh geometry information decoding unit according to an embodiment of the present disclosure is shown.

[0032] Figure 6 An example of an enhanced mesh geometry information decoding unit according to an embodiment of the present disclosure is shown.

[0033] Figure 7 An example of a mesh attribute information decoding unit according to an embodiment of the present disclosure is shown.

[0034] Figure 8 is a diagram illustrating a midpoint-based surface partitioning method according to an embodiment of the present disclosure.

[0035] Fig. 9 is a diagram for describing a displacement vector according to an embodiment of the present disclosure.

[0036] Fig.10 is a diagram illustrating a coefficient scanning method of an image packing unit according to an embodiment of the present disclosure.

[0037] Fig.11 is a diagram showing an enhanced mesh geometry information encoding unit according to an embodiment of the present disclosure.

[0038] Fig.12 is a diagram showing an enhanced mesh geometry information decoding unit according to an embodiment of the present disclosure.

[0039] Fig.13 is a diagram illustrating a displacement vector prediction method according to an embodiment of the present disclosure.

[0040] Fig.14 is a diagram showing an encoding / decoding order and a reference structure of basic mesh geometry information, a displacement vector image, and a texture image according to an embodiment of the present disclosure.

[0041] Fig.15 is a diagram illustrating an encoding / decoding process of a texture video according to an embodiment of the present disclosure.

[0042] Fig.16 is a diagram illustrating an encoding / decoding process of a texture video according to an embodiment of the present disclosure.

[0043] Fig.17 is a diagram illustrating an encoding / decoding process of a texture video according to an embodiment of the present disclosure.

[0044] Fig.18 is a flow chart illustrating a mesh compression process according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] In order to solve this problem, a dynamic mesh compression method and device using displacement vector prediction are proposed. In addition, a dynamic mesh compression method and device based on frame groups are proposed to solve this problem.

[0046] According to an embodiment of the present invention, a mesh compression method and apparatus can: reconstruct a base mesh based on geometric information obtained from a base mesh bitstream, perform surface partitioning on the reconstructed base mesh, obtain displacement vectors of at least one of the vertices of the reconstructed base mesh and the sub-vertices generated by the surface partitioning, reconstruct an enhanced mesh based on the reconstructed base mesh and the obtained displacement vectors, and obtain a texture image associated with the surface of the reconstructed enhanced mesh.

[0047] In the grid compression method and apparatus according to an embodiment of the present invention, geometric information can be encoded and decoded based on a first prediction mode, displacement vectors can be encoded and decoded based on a second prediction mode, and texture images can be encoded and decoded based on a third prediction mode.

[0048] In the grid compression method and apparatus according to the embodiment of the present invention, the first prediction mode, the second prediction mode, and the third prediction mode may be one of an inter-frame prediction mode or an intra-frame prediction method.

[0049] In the grid compression method and apparatus according to the embodiment of the present invention, at least one of the first prediction mode, the second prediction mode, or the third prediction mode may be determined according to a predetermined coding unit.

[0050] In the mesh compression method and apparatus according to the embodiments of the present invention, the encoding and decoding unit may be at least one of a frame, a patch, a surface, or a vertex.

[0051] The mesh compression method and apparatus according to the embodiments of the present invention can reconstruct the current frame based on the reconstructed enhanced mesh and texture image.

[0052] The mesh compression method and apparatus according to the embodiment of the present invention can obtain a first flag indicating whether to use texture image compression for the current frame.

[0053] In the mesh compression method and apparatus according to the embodiment of the present invention, the first flag may be signaled in units of a frame group including at least one frame.

[0054] In the mesh compression method and apparatus according to the embodiment of the present invention, when texture image compression is used for the current frame, the texture image may be obtained by referring to the texture image of a frame encoded and decoded before the current frame.

[0055] The grid compression method and apparatus according to the embodiment of the present invention can obtain a second flag indicating whether to encode or decode the texture image of the current frame.

[0056] In the mesh compression method and apparatus according to the embodiment of the present invention, the second flag may be signaled in units of frames.

[0057] In a grid compression method and apparatus according to an embodiment of the present invention, when encoding and decoding of a texture image of a current frame is omitted, a texture image can be obtained by referring to a texture image of a frame whose picture order count (POC) value is closest to the current frame in a frame encoded and decoded before the current frame.

[0058] The mesh compression method and apparatus according to the embodiment of the present invention can obtain a texture index indicating a texture image in a texture video including a plurality of texture images.

[0059] According to an embodiment of the present invention, a mesh compression device may include a processor and a memory, wherein the processor controls the mesh compression device, the memory is combined with the processor and stores data, and the processor may: reconstruct a base mesh based on geometric information obtained from a base mesh bitstream, perform surface partitioning on the reconstructed base mesh, obtain displacement vectors of at least one of the vertices of the reconstructed base mesh and the sub-vertices generated by the surface partitioning, reconstruct an enhanced mesh based on the reconstructed base mesh and the obtained displacement vectors, and obtain a texture image associated with the surface of the reconstructed enhanced mesh.

[0060] [model]

[0061] The embodiments of the present disclosure will be described in detail so that those skilled in the art can easily implement the embodiments by referring to the drawings attached to this specification. However, the present disclosure can be implemented in different forms, and the present disclosure is not limited to the embodiments described herein. Moreover, in order to clearly describe the present disclosure in the figures, parts not related to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

[0062] Throughout the specification, when a part is referred to as being “connected” to another part, it can include an electrical connection as well as a direct connection with other elements present therebetween.

[0063] In addition, throughout the specification, when a part is referred to as “comprising” components, it means that other components may also be included without excluding other components, unless otherwise contrary.

[0064] In addition, terms such as first, second, etc. may be used to describe various components, but the components should not be limited by these terms. The terms are only used to distinguish one component from other components.

[0065] In addition, for the embodiments of the device and method described in this specification, some configurations of the device or some steps of the method may be omitted. In addition, the order of some configurations of the device or some steps of the method may be changed. In addition, other configurations or other steps may be inserted into some configurations of the device or some steps of the method.

[0066] In addition, some configurations or some steps of the first embodiment of the present disclosure may be added to the second embodiment of the present disclosure, or may replace some configurations or some steps of the second embodiment.

[0067] In addition, the construction units shown in the embodiments of the present disclosure are shown separately to represent different characteristic functions, and they do not mean that each construction unit is configured with a separate hardware or software construction unit. In other words, for ease of description, each construction unit is described by being listed as each construction unit, and at least two construction units in each construction unit can be combined to form a construction unit, or a construction unit can be divided into multiple construction units to perform functions. The integrated implementation and separation implementation of each construction unit are also included in the scope of the rights of the present disclosure, unless they depart from the spirit of the present disclosure.

[0068] First, the terms used in this application can be briefly described as follows.

[0069] The decoding device (video decoding equipment) to be described later may be a device included in a server terminal such as a civilian security camera device, a civilian security system, a military security camera device, a military security system, a personal computer (PC), a notebook computer, a portable multimedia player (PMP), a wireless communication terminal, a smart phone, a TV application server, and a service server, and the decoding device (video decoding equipment) to be described later may refer to a variety of devices, the various devices being equipped with a user terminal including various equipment, a communication device including a communication modem for communicating using a wired / wireless communication network, a memory for storing various programs and data for decoding images or performing intra-frame prediction or inter-frame prediction for decoding, a microprocessor for executing programs and performing operations and controls, and others.

[0070] In addition, the image encoded as a bit stream by the encoder can be transmitted to the image decoding device, decoded and reconstructed and reproduced as an image in real time or non-real time through various communication interfaces such as cables, universal serial buses (USB), etc., or through wired or wireless communication networks such as the Internet, wireless local area networks, wireless LAN networks, Wi-Bro networks, mobile communication networks, etc. Alternatively, the bit stream generated by the encoder can be stored in a memory. The memory can include both volatile memory and non-volatile memory. In this specification, the memory can be represented as a recording medium storing a bit stream.

[0071] Generally, a video may be configured with a series of pictures, and each picture may be divided into coding units such as blocks. In addition, a person skilled in the art to which the present embodiment belongs may understand that the term "picture" input below may be used by being replaced with other terms having the same meaning as image, frame, etc. Also, a person skilled in the art to which the present embodiment belongs may understand that the term "coding unit" may be used by being replaced with other terms having the same meaning as unit block, block, etc.

[0072] Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure are described in more detail. In describing the present disclosure, repeated descriptions of the same components are omitted.

[0073] Figure 1 Examples of trellis encoders and decoders according to embodiments of the present disclosure are shown.

[0074] Reference Figure 1 The mesh encoder may receive the original mesh geometry information and the original mesh attribute information to generate a bitstream. The mesh decoder may receive the bitstream to reconstruct the mesh geometry information and the attribute information.

[0075] Here, the mesh geometry information may include three-dimensional coordinate information of mesh vertices, two-dimensional coordinates of mesh texture vertices, and connection information of vertices. In addition, the mesh attribute information may be a texture map. The texture map may include multiple channels. As an example, the texture map may be an image with at least three two-dimensional channels. In this case, the channel may mean a color space among RGB, YUV, YCbCr, YCoCg, and YCgCo. Alternatively, the channel may mean a color space of RGBA. In other words, the mesh attribute information may include color information of the outer surface of the three-dimensional object.

[0076] The mesh encoder may include a basic mesh geometry information encoding unit, an enhanced mesh geometry information encoding unit, a mesh attribute information encoding unit, and a multiplexer (MUX).

[0077] The base mesh geometry information encoding unit may receive the original mesh geometry information to generate the base mesh geometry information bitstream and the reconstructed base mesh geometry information, or may receive the base mesh geometry information bitstream and the reconstructed base mesh geometry information and the motion vector. The generated base mesh geometry information bitstream may be sent to the MUX. In addition, the generated reconstructed base mesh geometry information may be sent to the enhanced mesh geometry information encoding unit. In addition, the generated reconstructed motion vector may be sent to the enhanced mesh geometry information encoding unit.

[0078] The enhanced mesh geometry information encoding unit may receive the reconstructed base mesh geometry information and the original mesh geometry information, and perform encoding to generate a bitstream and reconstructed mesh geometry information. The generated enhanced mesh bitstream may be sent to the MUX. In addition, the reconstructed mesh geometry information may be sent to the mesh attribute information encoding unit.

[0079] The mesh attribute information encoding unit may receive and encode the original mesh geometry information, the original mesh attribute information, and the reconstructed mesh geometry information to generate a mesh attribute information bit stream. The generated mesh attribute information bit stream may be sent to the MUX.

[0080] A MUX can concatenate all the input bit streams and output them as a single bit stream.

[0081] The mesh decoder may include a demultiplexer (DEMUX), a basic mesh geometry information decoding unit, an enhanced mesh geometry information decoding unit, and a mesh attribute information decoding unit.

[0082] DEMUX can separate the input bit stream into a basic mesh geometry information bit stream, an enhanced mesh geometry information bit stream and a mesh attribute information bit stream, and send them to a basic mesh geometry information decoding unit, an enhanced mesh geometry information decoding unit and a mesh attribute information decoding unit respectively.

[0083] The basic mesh geometry information decoding unit may decode the input basic mesh geometry information decoding unit to reconstruct the basic mesh geometry information. The reconstructed basic mesh geometry information may be sent to the enhanced mesh geometry information decoding unit.

[0084] The enhanced mesh geometry information decoding unit may receive and decode the enhanced mesh geometry information bit stream and the reconstructed basic mesh geometry information, and reconstruct the mesh geometry information to output the mesh geometry information from the mesh decoding unit.

[0085] The mesh attribute information decoding unit may receive and decode the mesh attribute information bit stream, and reconstruct the mesh attribute information to output the mesh attribute information from the mesh decoding unit.

[0086] Figure 2 An example of a base mesh geometry information encoding unit according to an embodiment of the present disclosure is shown.

[0087] Reference Figure 2 The base mesh geometry information encoding unit may receive original mesh geometry information, perform encoding and output a base mesh bitstream, reconstructed base mesh geometry information and a reconstructed motion vector.

[0088] The basic grid geometry information encoding unit may include a grid downsampling unit, a geometry information encoding unit, a basic grid geometry information storage unit, a motion prediction and compensation unit, and a motion vector encoding unit.

[0089] The mesh downsampling unit may receive original mesh geometry information, perform downsampling of the geometry information and generate downsampled mesh geometry information. Here, the downsampled mesh may be referred to as a base mesh. In addition, when mesh downsampling is performed and mesh surface partitioning is performed thereafter, the input original mesh geometry information may be preprocessed to have the same number of vertices and connections. The generated base mesh geometry information may be sent to a geometry information encoding unit and a motion prediction and compensation unit. Alternatively, the generated base mesh geometry information may be sent to a geometry information encoding unit or a motion prediction and compensation unit based on a prediction mode. Here, the prediction mode may mean intra-frame prediction or inter-frame prediction, and the corresponding information may be sent to a decoder by entropy coding.

[0090] The geometry information encoding unit may receive the base mesh geometry information and perform encoding to generate a base mesh bitstream. In addition, reconstructed base mesh geometry information may be generated. In this case, the geometry information encoding unit may encode the three-dimensional vertex coordinates of the mesh, the two-dimensional texture vertex coordinates, and the connectivity between the vertices. In addition, the geometry information encoding unit may use international standard mesh geometry information compression methods such as MPEG-AFX / TFAN and Google Draco. Therefore, information about the mesh compression method used in the geometry information encoding unit may be sent to the decoder via entropy coding. The generated base mesh bitstream may be output from the base mesh geometry information encoding unit. The reconstructed base mesh geometry information may be sent to the base mesh geometry information storage unit.

[0091] The motion prediction and compensation unit may receive the base mesh geometry information and the reconstructed base mesh geometry information, and perform motion prediction to generate a motion vector. In addition, the reconstructed base mesh geometry information of the previous frame may be motion compensated by using the motion vector to generate the reconstructed base mesh geometry information of the current frame. The generated motion vector may be sent to the motion vector encoding unit. In addition, the reconstructed base mesh geometry information of the current frame may be sent to the base mesh geometry information storage unit and the motion vector encoding unit.

[0092] The motion vector encoding unit may generate a base mesh bitstream from the input motion vector by using the reconstructed base mesh geometry information. Here, one vertex may have at least one motion vector, and each motion vector may be a vector having at least one dimension. In this case, the size value of each dimension of the motion vector may be encoded in sequence. For example, the size values ​​of the first dimension of the motion vectors of all vertices of the base mesh may be entropy encoded in sequence. In this case, the order may be a traversal order of the vertices determined based on the connectivity of the vertex and the base mesh. After the motion vectors of all vertices are encoded in the first dimension, the size value of the second dimension of the motion vector may be encoded. In this way, the motion vectors of all dimensions may be encoded in sequence while increasing the dimension index. As another example, the motion vector encoding unit may encode the motion vectors in the vertex units of the reconstructed base mesh. According to the vertex traversal order, encoding may be performed in the order of the first dimension size, the second dimension size, and the third dimension size of the motion vector of each vertex, and then, encoding may be performed on the motion vector of the next vertex. Here, as a method for encoding a motion vector, when the motion vector corresponding to the current vertex is encoded, the reconstructed motion vector corresponding to the reconstructed adjacent vertex may be used to predict the motion vector and encode the size value of each dimension of the difference vector. The generated base mesh bitstream may be output from the base mesh geometry information encoding unit. In addition, the reconstructed base mesh geometry information may be output from the base mesh geometry information encoding unit.

[0093] Figure 3 An example of an enhanced mesh geometry information encoding unit according to an embodiment of the present disclosure is shown.

[0094] Reference Figure 3 , the enhanced mesh geometry information encoding unit can receive the original mesh geometry information and the reconstructed base mesh geometry information, and perform encoding to generate an enhanced mesh bitstream and the reconstructed mesh geometry information.

[0095] The enhanced mesh geometry information encoding unit may have a mesh surface division unit, a displacement vector transformation unit, a displacement vector quantization unit, a displacement vector image packing unit, a geometry video encoding unit, a displacement vector image unpacking unit, a displacement vector dequantization unit and a displacement vector inverse transformation unit.

[0096] The mesh surface division unit may receive the reconstructed basic mesh geometry information and divide the mesh surface to generate divided mesh geometry information. Here, the midpoint division method may be used as one of the surface division methods. The surface division method may be performed multiple times, and the number of divisions may be sent to the decoder by entropy coding. Figure 8 The relevant implementation methods are described in detail in FIG.

[0097] The divided mesh geometry information can be input to the displacement vector transform unit in the form of a displacement vector by being subtracted from the input original mesh geometry information. In addition, the divided mesh geometry information can be output from the enhanced mesh geometry information encoding unit in the form of reconstructed mesh geometry information by combining with the reconstructed displacement vector generated from the displacement vector inverse transform unit.

[0098] The displacement vector transformation unit can transform the input displacement vector to generate a displacement vector transformation coefficient. In this case, the transformation can be repeatedly performed multiple times. Alternatively, the transformation can be performed hierarchically. Wavelet transform can be used as an example of hierarchical transformation. Alternatively, graph-based wavelet transform can be used as a kind of wavelet transform. Graph-based wavelet transform is also called lifting transform, and can be a method of performing prediction and updating by using vertices connected to the current vertex when performing a transformation on the current vertex. In addition, the transformation can be performed by the dimension of the displacement vector. Alternatively, the transformation can be performed after the three-dimensional displacement vector is transformed into one dimension in dimension. The generated displacement vector transformation coefficient can be sent to the displacement vector quantization unit.

[0099] The displacement vector quantization unit may receive the transform coefficients and perform quantization to generate quantized transform coefficients. In this case, if a layered transform method is used in the displacement vector transform unit, different quantization parameters may be used for each layer. The quantization parameters used for quantization may be sent to a decoder through entropy coding. The generated quantized transform coefficients may be sent to a displacement vector image packing unit.

[0100] The displacement vector image packing unit may generate a displacement vector image by packing the quantized displacement vector transform coefficients sent in the form of an image. The method for packing the transform coefficients into the image may sequentially pack the displacement vector transform coefficients corresponding to the corresponding vertices into the image while traversing the vertices. In this case, the method for traversing the vertices may use the same traversal order as the method used in the geometry information encoding unit of the base mesh encoding unit. Alternatively, information about the traversal method used may be sent to the decoder by entropy coding.

[0101] As an embodiment, a depth-first traversal and a predicted degree traversal method can be used as a traversal method. The depth-first traversal method can be a method for traversing the vertices on the right side based on the edge of a specific grid and then traversing the vertices on the left side. The predicted degree traversal method is a method for determining the traversal order based on the number of reconstructed adjacent vertices connected to a specific vertex. For example, because not all initial vertices have reconstructed adjacent vertices, a specific vertex can be encoded / decoded. Afterwards, the vertex adjacent to the reconstructed vertex becomes the next traversal order because the number of reconstructed adjacent vertices is 1. Afterwards, all vertices can be traversed by repeatedly checking the number of vertices and increasing it from 1. And, the order of packing into the image can be packed in a raster scan order. In this case, the size information of the image to be packed can be sent to the decoder by entropy coding. In this case, the size information of the image can have at least one of the width, height and number of channels of the image. Alternatively, the size information of the image can be packed in units of blocks in a Z scan order. In this case, the size information of the block can be sent to the decoder by entropy coding. Here, the size information of the block can have at least one of the width and height. The generated displacement vector image may be sent to a geometry video coding unit.

[0102] The geometric video encoding unit can collect the input displacement vector image, receive it in the form of a video, perform encoding and generate an enhanced grid bitstream. In addition, a reconstructed displacement vector image can be generated. Here, the geometric video encoding unit can use international standard video compression technologies such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP8, VP9, ​​AV1, etc., and send information about the video compression technology used to the decoder through entropy coding. The generated enhanced grid bitstream can be output from the enhanced grid geometry information encoding unit. In addition, the generated reconstructed displacement vector image can be sent to the displacement vector image unpacking unit.

[0103] The displacement vector image unpacking unit may unpack the input displacement vector image and reconstruct the quantized displacement vector transform coefficients. In this case, the displacement vector image unpacking unit may use the same method of vertex traversal information and image scanning method as used in the displacement vector image packing unit. The generated quantized displacement vector transform coefficients may be sent to the displacement vector dequantization unit.

[0104] The displacement vector dequantization unit may perform dequantization by receiving the input quantized displacement vector transform coefficient to reconstruct the displacement vector transform coefficient. In this case, the dequantization may use the same quantization parameter as the quantization parameter used in the displacement vector quantization unit. Alternatively, it may be transformed into a dequantization parameter corresponding to the quantization parameter used in the displacement vector quantization unit and used. Typically, the quantization parameter and the dequantization parameter may be inversely related. Alternatively, the dequantization may be performed by using a value scaled based on the quantization parameter. The reconstructed displacement vector transform coefficient may be sent to the displacement vector inverse transform unit.

[0105] The displacement vector inverse transform unit may reconstruct the displacement vector by inversely transforming the input reconstructed displacement vector transform coefficient. In this case, the transform method used in the displacement vector inverse transform may be a corresponding inverse transform in the transform method used in the displacement vector transform unit. The reconstructed displacement vector may be combined with the divided mesh geometry information generated from the mesh surface division unit to reconstruct the mesh geometry information, and may be output from the enhanced mesh geometry information encoding unit.

[0106] Figure 4 An example of a mesh attribute information encoding unit according to an embodiment of the present disclosure is shown.

[0107] Reference Figure 4 The mesh attribute information encoding unit may receive the reconstructed mesh geometry information, the original mesh geometry information, and the original mesh attribute information, and encode the original mesh attribute information to generate an attribute bitstream.

[0108] The mesh attribute information encoding unit may include an attribute information mapping unit, a padding unit, a color space transform unit, and an attribute video encoding unit.

[0109] The attribute information mapping unit may receive the reconstructed mesh geometry information, the original mesh geometry information, and the original mesh attribute information, and map the attribute information of the original mesh to the reconstructed mesh geometry information based on the geometry information of the corresponding original mesh to generate a texture map. In other words, the geometry information of the reconstructed mesh and the geometry information of the original mesh represent the same three-dimensional object, but differences may occur due to geometry information encoding / decoding. Therefore, the attribute information mapping unit may be a process of mapping the texture map to match the reconstructed mesh geometry information. For example, a two-dimensional texture vertex corresponding to the three-dimensional coordinates of the triangular face of the original mesh closest to the vertex of the reconstructed mesh geometry information may be obtained, and the attribute information of the corresponding texture vertex may be mapped to the texture map of the reconstructed mesh. Alternatively, three texture vertices of the vertex of the triangular face of the original mesh closest to the reconstructed mesh geometry information may be used to map the triangular attribute information of the corresponding texture map to the texture map of the reconstructed mesh. The generated texture map may be sent to the filling unit.

[0110] The filling unit may perform filling on texture map areas in texture vertices and triangles that are not included in the reconstructed geometric information of the texture map for input. In this case, a fixed filling method for filling all contents with a specific value may be used as the filling method. For example, an intermediate value of the bit depth of the texture map may be used. Another method is a push-pull filling method. The push-pull filling method may be a method for downsampling the texture map and then upsampling it to fill the blank space by synthesizing it with the original image. In this case, the number of downsampling and upsampling may be greater than or equal to 1. Another method may be a nearest point pixel filling method. It may be a method for filling using the value of the texture vertex at each pixel position closest to the blank space. The filled texture map may be sent to the color space conversion unit.

[0111] The color space conversion unit may perform color space conversion on the input filled texture map. In this case, the color space conversion may be RGB-YUV, RGB-YCbCr, RGB-YCoCg, or RGB-YCgCo conversion. In this case, when the texture map has 4 channels, the color space conversion may not be performed, so the color space conversion may be performed on the remaining channels except for the channel representing transparency. In addition, the conversion method for the color space conversion may be sent to the decoder by entropy coding. The color space converted texture map may be sent to the attribute video encoding unit.

[0112] The attribute video encoding unit may generate an attribute bitstream by collecting the input color space transformed texture maps and encoding them in the form of a video. Here, the attribute video encoding unit may use international standard video compression technologies such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP8, VP9, ​​AV1, etc., and send information about the used video compression technology to the decoder through entropy coding. The generated attribute bitstream may be output from the grid attribute encoding unit.

[0113] Figure 5 An example of a base mesh geometry information decoding unit according to an embodiment of the present disclosure is shown.

[0114] Reference Figure 5 , the base grid geometry information decoding unit can receive the base grid bit stream and decode it to generate reconstructed base grid geometry information and reconstructed motion vectors.

[0115] The basic mesh geometry information decoding unit may include a geometry information decoding unit, a basic mesh geometry information storage unit, a motion vector decoding unit, and a motion compensation unit.

[0116] The base grid bitstream as an input of the base grid geometry information decoding unit may be sent to the geometry information decoding unit or the motion vector decoding unit according to the prediction mode sent as the advanced information from the base grid geometry information encoding unit. For example, when the prediction mode is intra prediction, the base grid bitstream may be sent to the geometry information decoding unit. In the opposite case, the base grid bitstream may be sent to the motion vector decoding unit.

[0117] The geometry information decoding unit may reconstruct the input base mesh bitstream to reconstruct the base mesh geometry information. The reconstructed base mesh geometry information may be sent to the base mesh geometry information storage unit. In this case, the geometry information decoding unit may use a decoding method corresponding to the method used in the geometry information encoding unit of the base mesh geometry information encoding unit.

[0118] The base mesh geometry information storage unit may output the reconstructed base mesh geometry information of the input current frame and store it for decoding subsequent frames. The stored base mesh geometry information may be output from the base mesh geometry information decoding unit. In addition, it may be sent to the motion vector decoding unit.

[0119] The motion vector decoding unit may reconstruct the motion vector by using the input base mesh bitstream and the reconstructed base mesh geometry information. The motion vector may be reconstructed based on the traversal order used when encoding / decoding the reconstructed base mesh geometry information. Alternatively, the base mesh bitstream may be decoded to reconstruct the residual motion vector, and the motion vector may be reconstructed by adding the reconstructed motion vectors of the adjacent vertices to the predicted motion vector. The reconstructed motion vector and the reconstructed base mesh geometry information may be sent to the motion compensation unit.

[0120] The motion compensation unit may reconstruct the base mesh geometry information by performing motion compensation on the input base mesh geometry information by using the input motion vector. In this case, motion compensation may be performed on the three-dimensional coordinates of the vertices of the reconstructed mesh geometry information. The reconstructed base mesh geometry information may be sent to the base mesh geometry information storage unit. In addition, the reconstructed motion vector may be output from the base mesh geometry information decoding unit.

[0121] Figure 6 An example of an enhanced mesh geometry information decoding unit according to an embodiment of the present disclosure is shown.

[0122] Reference Figure 6 , the enhanced mesh geometry information decoding unit can receive the reconstructed basic mesh geometry information and the enhanced mesh bit stream to reconstruct the mesh geometry information.

[0123] The enhanced mesh geometry information decoding unit may include a mesh surface partitioning unit, a geometry video decoding unit, an image unpacking unit, a displacement vector dequantization unit, and a displacement vector inverse transformation unit.

[0124] The mesh surface division unit may receive the reconstructed basic mesh geometry information and divide the mesh surface to generate divided mesh geometry information. The basic mesh division method may be a method received from the enhanced mesh geometry information encoding unit. In this case, when the division method is performed hierarchically, the number of repetitions may be additionally received from the encoder to perform the corresponding number of divisions. The divided mesh geometry information may be combined with the reconstructed displacement vector generated from the reconstructed vector inverse transform unit and output from the enhanced mesh geometry information decoding unit in the form of reconstructed mesh geometry information.

[0125] The geometry video decoding unit may receive the enhanced mesh bitstream, reconstruct the displacement vector video and output the displacement vector video frame by frame in the form of a displacement vector image. In this case, video decoding may be performed based on information about the geometry video compression method received from the enhanced mesh geometry information decoding unit. The geometry video decoding unit may use international standard video compression technologies such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP8, VP9, ​​AV1, etc. The reconstructed displacement vector image may be sent to the displacement vector image unpacking unit.

[0126] The displacement vector image unpacking unit may unpack the input displacement vector image and reconstruct the quantized displacement vector transform coefficients. In this case, the displacement vector image unpacking unit may use the same method as the vertex traversal information and image scanning method used in the displacement vector image packing unit of the enhanced mesh geometry information encoding unit. Alternatively, the vertex traversal information and image scanning method received from the enhanced mesh geometry information encoding unit may be used. The generated quantized displacement vector transform coefficients may be sent to the displacement vector dequantization unit.

[0127] The displacement vector dequantization unit may perform dequantization by receiving the input quantized displacement vector transform coefficient to reconstruct the displacement vector transform coefficient. In this case, the dequantization may use the same quantization parameter as the quantization parameter used in the displacement vector quantization unit of the enhanced mesh geometry information encoding unit. Alternatively, the quantization parameter received from the enhanced mesh geometry information encoding unit may be used. In this case, instead of using the quantization parameter directly, the quantization parameter may be used by transforming into a dequantization parameter corresponding to the input quantization parameter. Alternatively, dequantization may be performed by using a value scaled based on the quantization parameter. The reconstructed displacement vector transform coefficient may be sent to the displacement vector inverse transform unit.

[0128] The displacement vector inverse transform unit can reconstruct the displacement vector by inversely transforming the input reconstructed displacement vector transform coefficient. In this case, the transform method used for the displacement vector inverse transform may be the corresponding inverse transform of the transform method used in the displacement vector transform unit of the enhanced mesh geometry information encoding unit. Alternatively, an inverse transform method received from the enhanced mesh encoding unit may be used. Alternatively, an inverse transform method corresponding to the received transform method may be used. The reconstructed displacement vector may be combined with the divided mesh geometry information generated from the mesh surface division unit to reconstruct the mesh geometry information, and may be output from the enhanced mesh geometry information decoding unit.

[0129] Figure 7 An example of a mesh attribute information decoding unit according to an embodiment of the present disclosure is shown.

[0130] Reference Figure 7 , the mesh attribute information decoding unit may receive the attribute bit stream and decode it to reconstruct the mesh attribute information.

[0131] The mesh attribute information decoding unit may include an attribute information decoding unit and a color space inverse transform unit.

[0132] The attribute video decoding unit can reconstruct the attribute bitstream to generate the texture video, output one frame at a time, and send the reconstructed texture image to the color space inverse transform unit. In this case, the attribute video decoding unit can perform video decoding by using the video compression method received from the attribute video encoding unit.

[0133] The color space inverse transform unit may perform color space inverse transform on the input reconstructed texture image to reconstruct the mesh attribute information. In this case, for the color space inverse transform method, an inverse transform method corresponding to the color space transform method used in the color space transform unit of the mesh attribute information encoding unit may be used. Alternatively, an inverse transform method received from the mesh attribute information encoding unit may be used. Alternatively, an inverse transform method corresponding to the transform method received from the mesh attribute information encoding unit may be used.

[0134] Figure 8 is a diagram illustrating a midpoint-based surface partitioning method according to an embodiment of the present disclosure.

[0135] According to an embodiment of the present disclosure, a sub-vertex may be generated at the center of an edge connecting a vertex and a vertex, and surface partitioning may be performed by connecting the generated sub-vertices.

[0136] Reference Figure 8 , the reconstructed basic mesh geometry information can be expressed as Figure 8The triangular mesh shown on the left side of . In this case, when midpoint partitioning is performed, a sub-vertex can be generated at the center of each edge of the triangle. By connecting the generated vertices to each other, a triangle can be divided into four small triangles (or sub-triangles).

[0137] In an embodiment, the midpoint-based surface partitioning method may be performed hierarchically (or recursively), and the number of partitions or repetitions may be transmitted to a decoder through entropy coding.

[0138] Fig. 9 is a diagram for describing a displacement vector according to an embodiment of the present disclosure.

[0139] Reference Fig. 9 , the sub-vertices generated by the mesh surface division unit can exist on the edge connecting two reconstructed base vertices. In this regard, the above can be applied Figure 8 The midpoint-based surface partitioning method described in, but not limited to.

[0140] The reconstructed base vertex including the sub-vertices may be mapped one-to-one to the original vertex. As an example, a pre-processing step may be processed before encoding / decoding such that the original vertex and the reconstructed vertex are mapped one-to-one.

[0141] As an embodiment, the displacement vector may be a three-dimensional vector representing the difference between the original vertex and the reconstructed base vertex and / or sub-vertex. The geometric information of the mesh may be reconstructed by encoding and decoding the displacement vector and adding it to the reconstructed base vertex and / or sub-vertex.

[0142] Fig.10 is a diagram illustrating a coefficient scanning method of an image packing unit according to an embodiment of the present disclosure.

[0143] Reference Fig.10 ,like Fig.10 As shown on the left side of , the reconstructed base vertices and the sub-vertices of the surface partitioning can have quantized displacement vector transformation coefficients for each vertex.

[0144] As an implementation, the quantized displacement vector transform coefficients of each vertex may be packed according to a predefined scanning order to generate a displacement vector image. Fig.10 In the example of the displacement vector image in the middle and the packing order on the right, the quantized displacement vector transform coefficients of each vertex can be packed in a Z-scan order in units of 2×2 sub-regions to generate a displacement vector image. Alternatively, the displacement vector transform coefficients can be packed in the reverse order of the Z-scan order to generate a displacement vector image.

[0145] Here, the empty pixels of the displacement vector image may be filled with a specific value through a filling process in a filling unit. The specific value may be set to zero.

[0146] Fig.11 is a diagram showing an enhanced mesh geometry information encoding unit according to an embodiment of the present disclosure.

[0147] Reference Fig.11 The enhanced mesh geometry information encoding unit may have a mesh surface division unit, a displacement vector storage unit, a displacement vector prediction unit, a displacement vector transformation unit, a displacement vector quantization unit, a displacement vector image packing unit, a geometry video encoding unit, a displacement vector image unpacking unit, a displacement vector dequantization unit, and a displacement vector inverse transformation unit.

[0148] According to an embodiment of the present disclosure, the enhanced mesh geometry information encoding unit may receive original mesh geometry information, reconstructed base mesh geometry information, and reconstructed motion vectors, and perform encoding to generate an enhanced mesh bitstream and reconstructed mesh geometry information.

[0149] The mesh surface division unit may receive the reconstructed basic mesh geometry information and divide the mesh surface to generate the divided mesh geometry information. In this regard, the above method may be applied. Figure 8 The midpoint-based surface partitioning method described in, but not limited to.

[0150] The divided mesh geometry information may be subtracted from the input original mesh geometry information to generate a displacement vector, and the divided mesh geometry information may be subtracted from the predicted displacement vector generated from the displacement vector prediction unit and input to the displacement vector transformation unit in the form of a differential displacement vector. As an embodiment, a displacement vector may be generated based on the difference between the vertices of the divided mesh and the vertices of the original mesh. Then, a prediction of the displacement vector may be performed. Encoding may be performed on the differential displacement vector representing the difference between the generated displacement vector and the predicted displacement vector. The encoding unit and the decoding unit may derive (or generate) the predicted displacement vector in the same manner. The decoding unit may obtain the displacement vector of the vertex by combining the predicted displacement vector and the differential displacement vector.

[0151] In addition, the divided mesh geometry information may be combined with the reconstructed differential displacement vector generated from the displacement vector inverse transform unit and the predicted displacement vector generated from the displacement vector prediction unit, and output from the enhanced mesh geometry information encoding unit in the form of reconstructed mesh geometry information. In the present disclosure, the difference between the divided mesh geometry information and the original mesh geometry information may be referred to as the original displacement vector, and the difference between the original displacement vector and the predicted displacement vector may be referred to as the differential displacement vector. In the present disclosure, the differential displacement vector may also be referred to as the residual displacement vector.

[0152] The displacement vector transform unit may transform the input displacement vector to generate a displacement vector transform coefficient. The generated displacement vector transform coefficient may be sent to the displacement vector quantization unit.

[0153] The displacement vector quantization unit may receive the transform coefficients and perform quantization to generate quantized transform coefficients. The generated quantized transform coefficients may be sent to the displacement vector image packing unit.

[0154] The displacement vector image packing unit may generate a displacement vector image by packing the quantized displacement vector transform coefficients sent in the form of an image. The generated displacement vector image may be sent to the geometry video encoding unit.

[0155] The geometry video encoding unit may receive a combination of displacement vector images of an input in the form of a video and perform encoding to generate an enhanced mesh bitstream. The generated enhanced mesh bitstream may be output from the enhanced mesh geometry information encoding unit. Additionally, the generated reconstructed displacement vector image may be sent to a displacement vector image unpacking unit.

[0156] The displacement vector image unpacking unit may unpack the input displacement vector image and reconstruct quantized displacement vector transform coefficients. The generated quantized displacement vector transform coefficients may be sent to the displacement vector dequantization unit.

[0157] The displacement vector dequantization unit may perform dequantization by receiving the input quantized displacement vector transform coefficient to reconstruct the displacement vector transform coefficient. The reconstructed displacement vector transform coefficient may be sent to the displacement vector inverse transform unit.

[0158] The displacement vector inverse transform unit can reconstruct the displacement vector by inversely transforming the input reconstructed displacement vector transform coefficient. The reconstructed displacement vector can be combined with the divided mesh geometry information generated from the mesh surface division unit to reconstruct the mesh geometry information, and can be output from the enhanced mesh geometry information encoding unit. In addition, the reconstructed displacement vector can also be stored in the displacement vector storage unit.

[0159] The displacement vector storage unit may store the displacement vector of the current frame, and send the stored displacement vector to the displacement vector prediction unit for displacement vector prediction of subsequent frames.

[0160] The displacement vector prediction unit may receive the reconstructed motion vector and the reconstructed displacement vector to generate a predicted displacement vector of the current frame. The generated predicted displacement vector may be subtracted from the original displacement vector and sent to the displacement vector transformation unit in the form of a differential displacement vector. In addition, the displacement vector may be reconstructed by combining with the reconstructed differential displacement vector generated from the displacement vector inverse transformation unit.

[0161] Fig.12 is a diagram showing an enhanced mesh geometry information decoding unit according to an embodiment of the present disclosure.

[0162] Reference Fig.12 The enhanced mesh geometry information decoding unit may include a mesh surface division unit, a displacement vector prediction unit, a displacement vector storage unit, a geometry video decoding unit, an image unpacking unit, a displacement vector dequantization unit, and a displacement vector inverse transformation unit.

[0163] The enhanced mesh geometry information decoding unit may receive the reconstructed basic mesh geometry information, the reconstructed motion vector and the enhanced mesh bitstream to reconstruct the mesh geometry information.

[0164] The mesh surface division unit may receive the reconstructed basic mesh geometry information and divide the mesh surface to generate the divided mesh geometry information. The divided mesh geometry information may be subtracted from the input original mesh geometry information to generate the displacement vector, and the divided mesh geometry information may be subtracted from the predicted displacement vector generated from the displacement vector prediction unit, and input to the displacement vector transformation unit in the form of a differential displacement vector. In addition, the divided mesh geometry information may be output from the enhanced mesh geometry information encoding unit in the form of reconstructed mesh geometry information by combining with the reconstructed differential displacement vector generated from the displacement vector inverse transformation unit and the predicted displacement vector generated from the displacement vector prediction unit.

[0165] The geometric video decoding unit may receive the enhanced grid bitstream, reconstruct the displacement vector video and output the displacement vector video frame by frame in the form of a displacement vector image. The reconstructed displacement vector image may be sent to the displacement vector image unpacking unit.

[0166] The displacement vector image unpacking unit may unpack the input displacement vector image and reconstruct quantized displacement vector transform coefficients. The generated quantized displacement vector transform coefficients may be sent to the displacement vector dequantization unit.

[0167] The displacement vector dequantization unit may perform dequantization by receiving the input quantized displacement vector transform coefficient to reconstruct the displacement vector transform coefficient. The reconstructed displacement vector transform coefficient may be sent to the displacement vector inverse transform unit.

[0168] The displacement vector inverse transform unit can reconstruct the displacement vector by inversely transforming the input reconstructed displacement vector transform coefficient. The reconstructed displacement vector can be combined with the divided mesh geometry information generated from the mesh surface division unit to reconstruct the mesh geometry information, and can be output from the enhanced mesh geometry information encoding unit. In addition, the reconstructed displacement vector can be stored in the displacement vector storage unit.

[0169] The displacement vector storage unit may store the displacement vector of the current frame, and send the stored displacement vector to the displacement vector prediction unit for displacement vector prediction of subsequent frames.

[0170] The displacement vector prediction unit may receive the reconstructed motion vector and the reconstructed displacement vector to generate a predicted displacement vector of the current frame. In addition, the generated predicted displacement vector may be combined with the reconstructed differential displacement vector generated from the displacement vector inverse transform unit to reconstruct the displacement vector.

[0171] Fig.13 is a diagram illustrating a displacement vector prediction method according to an embodiment of the present disclosure.

[0172] Fig.13 (A) shows a displacement vector rotation method based on a motion vector, and Fig.13 (B) shows the displacement vector rotation method based on the normal vector.

[0173] Reference Fig.13 , a predicted displacement vector may be generated by receiving a reconstructed motion vector and a displacement vector of a previous frame. In other words, the displacement vector prediction unit may generate a predicted displacement vector by using a motion vector and / or a displacement vector of a previous frame.

[0174] According to an embodiment of the present disclosure, the predicted displacement vector may be a vector in which the displacement vector of the previous frame is rotated. In other words, the predicted displacement vector of the current vertex may be a vector in which the displacement vector of the vertex corresponding to the current vertex in the previous frame is rotated based on the motion vector. In the present disclosure, the motion vector may be used to obtain (or derive) geometric information (i.e., the position of the vertex) of the base mesh based on inter-frame prediction (or inter-frame prediction). In other words, the position of the corresponding vertex of the previous frame (i.e., the vertex corresponding to the current vertex) and the motion vector may be used to determine the position of the current vertex in the current frame.

[0175] As an embodiment, a rotation transformation using Euler angles, a rotation transformation based on quaternions, etc. may be used as a rotation transformation method of a displacement vector. Fig.13 As shown in (A), the transformation parameters required for the rotation transformation can be calculated based on the motion vector. In addition, as another embodiment, the rotation transformation parameters can be calculated (or obtained) based on the normal vector.

[0176] As an embodiment, a method for obtaining rotation transformation parameters may be selectively used depending on whether the current vertex is a base vertex. In the present disclosure, a base vertex may refer to a vertex of a base mesh. If the current vertex is a vertex of a base mesh, it is not on the same three-dimensional plane as the adjacent vertices, so a rotation based on a normal vector (or normal vector) may be used. If the current vertex is a sub-vertex generated by performing surface division on a base mesh, it is on the same three-dimensional plane as the adjacent base vertices, so a rotation transformation parameter may be obtained based on the motion vector of the adjacent base vertices. In this case, the adjacent base vertices may refer to the three base vertices used to generate the current sub-vertex. Rotation transformation parameters for predicting the displacement vector of the current sub-vertex may be obtained based on the motion vector of the adjacent base vertices. Alternatively, the rotation transformation parameters may be obtained by using the motion vector of the adjacent sub-vertex calculated based on the motion vector of the base vertex.

[0177] As another embodiment, the rotation transformation may be selectively performed based on the magnitude value of the motion vector. For example, when the magnitude value of the motion vector is less than or less than or equal to a specific value, the rotation transformation may not be performed. On the contrary, when the magnitude value of the motion vector exceeds or is greater than or equal to a specific value, the rotation transformation may be performed. Alternatively, the rotation transformation may be selectively performed based on the difference between the motion vectors of adjacent vertices. In other words, the rotation transformation may be performed only when the difference between the motion vectors of adjacent vertices is greater than or greater than or equal to a predefined value.

[0178] Fig.14 is a diagram showing an encoding / decoding order and a reference structure of basic mesh geometry information, a displacement vector image, and a texture image according to an embodiment of the present disclosure.

[0179] As in Fig.14 In the example of , mesh geometry information, displacement vector video (or image) and texture video (or image) can be encoded / decoded in sequence. In this case, for inter-frame prediction, encoding / decoding can be performed using pre-reconstructed geometry information, displacement vector and attribute information, respectively.

[0180] According to an embodiment of the present disclosure, geometric information, displacement vector video, and texture video may use different predictions for each codec unit. For example, displacement vector video and texture video may perform intra-frame prediction and / or inter-frame prediction in a specific processing unit according to the codec unit and mode of the video compression method. In the present disclosure, a codec unit may be a processing unit in which a prediction method (or prediction mode) is determined. As an example, a codec unit of geometric information may be at least one of a frame, a patch, a triangle (or a surface), and a vertex.

[0181] For example, when the coding unit is a frame, reference frame information may be sent from the encoder to the decoder. Alternatively, when the coding unit is a patch, reference frame information and reference patch information may be sent from the encoder to the decoder. Alternatively, when the coding unit is a triangle, reference frame information, reference patch information, reference triangle information, etc. may be sent from the encoder to the decoder. Alternatively, when the coding unit is a vertex, reference frame information, reference patch information, reference triangle information, and reference vertex information may be sent from the encoder to the decoder.

[0182] As an embodiment, the displacement vector video and the texture video may have different reference structures because they are encoded / decoded by using an existing video compression standard technology. Fig.14 The example in is an example of a structure in which both videos refer to a single frame that was just reconstructed before to achieve low-latency encoding / decoding.

[0183] As an example, the base mesh geometry information may have a different reference structure than the displacement vector video and / or texture video. For example, for the inter prediction mode, the base mesh geometry information may perform inter prediction on the entire frame geometry information. On the other hand, the displacement vector and texture video may perform intra or inter prediction in a codec block unit within a video encoding / decoding device.

[0184] Fig.15 is a diagram illustrating an encoding / decoding process of a texture video according to an embodiment of the present disclosure.

[0185] Reference Fig.15 , the texture image of the mesh can exist in all mesh picture order counts (POC). In other words, each mesh can be assigned a POC value, and each mesh can have a texture image.

[0186] In this case, even when the three-dimensional object moves, the attribute information of the outer surface of the object may be similar. Therefore, according to an embodiment of the present invention, considering this point, compression encoding may be performed on the texture image in units of frames or in units of frame groups.

[0187] As an implementation, a predetermined group of frames GOF (group of frames) may be defined. With GOF as a unit, a texture video may be generated using a texture image smaller than the size of the GOF, and based on this, encoding / decoding may be performed. As an example, information about the size of the GOF may be sent to a decoder as high-level information through entropy coding.

[0188] As an embodiment, in units of GOF, a texture video may be generated by using only the first texture image. The decoder may obtain the GOF size from the high-level information and use one transmitted texture image in all frames of the corresponding GOF.

[0189] As an example, in units of GOF, a method for encoding / decoding a texture smaller than the GOF size may be selectively used. In other words, in units of GOF, it may be determined whether to apply the texture image compression method according to the present embodiment. For example, a flag LC_FLAG related to the use of a method for encoding / decoding a texture smaller than the GOF size may be defined. In other words, a flag indicating whether to use GOF unit texture image compression may be defined, and in the present disclosure, the corresponding flag may be referred to as LC_FLAG. LC_FLAG may be sent to a decoder through entropy coding.

[0190] For example, when the LC_FLAG value is 1, GOF unit texture image compression may be used, and when the LC_FLAG value is 0, GOF unit texture image compression may not be used. Fig.15 As shown, a mesh included in a GOF having an LC_FLAG value of 1 can be reconstructed by using a texture image having a predetermined POC value in a texture video. For a mesh included in a GOF having an LC_FLAG value of 0, a corresponding texture image can be included in the texture video and reconstructed by using it. In other words, the decoder can reconstruct a mesh in units of GOFs by partially referring to a texture image from a texture video received by checking a corresponding flag.

[0191] Fig.16 is a diagram illustrating an encoding / decoding process of a texture video according to an embodiment of the present disclosure.

[0192] Reference Fig.16 , each frame of a particular GOF (i.e., grid, grid frame) may define a flag indicating whether the texture image of the current frame is coded or decoded. In the present disclosure, the corresponding flag may be referred to as a texture coding flag or a texture_coded flag. In other words, in Fig.16 In the texture_coded, it indicates the texture encoding and decoding flag.

[0193] As an embodiment, when the texture codec flag is 1, the texture image of the current mesh may be encoded by being included in the texture video. Conversely, when the texture codec flag is 0, encoding of the texture image of the current mesh may be omitted. When the texture codec flag is 0, encoding of the texture image of the current mesh may be omitted, and the texture image of the previous mesh may be referenced. In other words, when the texture codec flag is 0, encoding of the texture image of the current mesh may be omitted, and the current mesh may be encoded / decoded by using the reference texture image.

[0194] Texture encoding / decoding information may be included in the high-level information on a grid frame basis and signaled to the decoder by entropy encoding. The decoder may check the corresponding information on a grid frame basis to reconstruct the texture image.

[0195] As an example, when the texture encoding / decoding flag value is 0, the texture image of the grid frame in the pre-reconstructed grid frames that has a texture encoding / decoding flag value of 1 and a POC value closest to the POC of the current grid frame may be used as the current texture image. In other words, when the texture encoding / decoding flag value is 0, the texture image of the grid frame with the closest POC value in the pre-reconstructed grid frames may be referred to.

[0196] In addition, in an embodiment, the texture encoding / decoding flag may be used (or signaled) only when the LC_FLAG of the current GOF is 1. As another example, when the prediction mode of the current grid frame is an intra prediction mode, the encoding / decoding of the texture encoding / decoding flag may be omitted and its value may be derived (or inferred) as 1.

[0197] Fig.17 is a diagram showing an encoding / decoding process of a texture video according to an embodiment of the present disclosure.

[0198] Referring to Fig.17 , a reference texture image may be determined on a grid frame basis. In an embodiment of the present disclosure, a group of pictures (GOP) may be defined as a predetermined group of texture images. The texture video may include a GOP. In other words, the texture image (i.e., the reference texture image) of the current grid may be specified within the GOP of the texture video on a per grid frame basis. As an embodiment, the reference texture image within the GOP of the texture video may be specified by a frame index. In the present disclosure, the frame index specifying the reference texture image within the GOP of the texture video may be referred to as a frame index, a texture index, or a texture image index.

[0199] For example, when the GOP size of the texture video is 2, the texture index ( Fig.17 texture_index in) may have values 0 and 1. When it is 0, it may mean that the first frame within the GOP is used as the texture image for the current grid POC in the texture video. Conversely, when it is 1, it may mean that the second frame within the GOP is used as the texture image for the current grid POC in the texture video.

[0200] In an embodiment, one texture video frame may be generated by collecting original texture images with the same texture index. Alternatively, an image with a minimum grid POC value in the same original texture image may be used as a texture video frame. The texture index may be included in the advanced information per grid frame and sent to the decoder by entropy coding. In addition, in an embodiment, the texture index may be used (or signaled) only when the LC_FLAG of the current GOF is 1.

[0201] Fig.18 is a flow chart illustrating a mesh compression process according to an embodiment of the present disclosure.

[0202] Reference Fig.18 , when describing the grid compression method, the method performed by the grid decoder is mainly described, but it is not limited thereto, and the method described in this embodiment can be performed by the grid encoder in substantially the same manner. Figures 1 to 17 The described method can be applied to this embodiment.

[0203] The mesh decoder may reconstruct the base mesh based on geometric information obtained from the base mesh bitstream ( S1800 ).

[0204] The mesh decoder may perform surface partitioning on the reconstructed base mesh ( S1810 ).

[0205] The mesh decoder may obtain a displacement vector of at least one of a vertex of a base mesh for reconstruction and a sub-vertices generated by surface partitioning ( S1820 ).

[0206] The mesh decoder may reconstruct an enhanced mesh based on the reconstructed base mesh and the obtained displacement vector (S1830).

[0207] The mesh decoder may obtain a texture image associated with a surface of the reconstructed enhanced mesh (S1840). The mesh decoder may reconstruct a current frame based on the reconstructed enhanced mesh and the texture image.

[0208] As described above, the geometric information can be encoded and decoded based on the first prediction mode, the displacement vector can be encoded and decoded based on the second prediction mode, and the texture image can be encoded and decoded based on the third prediction mode. The first prediction mode, the second prediction mode, and the third prediction mode can be one of an inter-frame prediction mode or an intra-frame prediction method.

[0209] In addition, as described above, at least one of the first prediction mode, the second prediction mode, or the third prediction mode may be determined according to a predetermined coding unit. As an example, the coding unit may be at least one of a frame, a patch, a surface, or a vertex.

[0210] In addition, as described above, the mesh decoder may obtain a first flag indicating whether texture image compression is used for the current frame. Here, the first flag may be signaled in units of a frame group including at least one frame. When texture image compression is used for the current frame, the texture image obtained in S1840 may be obtained by referring to a texture image of a frame encoded and decoded before the current frame. In this regard, the above in S1840 may be applied. Fig.15 The contents described in , where repeated descriptions are omitted.

[0211] In addition, as described above, the grid decoder may obtain a second flag indicating whether to encode or decode the texture image of the current frame. The second flag may be signaled in units of frames. When encoding and decoding the texture image of the current frame is omitted, the texture image may be obtained by referring to a texture image of a frame whose picture order count (POC) value is closest to the current frame among frames encoded and decoded before the current frame. In this regard, the above-mentioned Fig.16 The contents described in , where repeated descriptions are omitted.

[0212] In addition, as described above, in the step of obtaining a texture image, the mesh decoder may obtain a texture index indicating a texture image in a texture video including a plurality of texture images. In this regard, the above Fig.17 The contents described in , where repeated descriptions are omitted.

[0213] The above-mentioned embodiments may be a combination of components and features of the present disclosure in a predetermined form. Unless otherwise expressly stated, each component or feature should be regarded as optional. Each component or feature may be implemented in a form not combined with other components or features. In addition, some components and / or features may be combined to configure the embodiments of the present disclosure. The order of the operations described in the embodiments of the present disclosure may be changed. Some configurations or features of the embodiments may be included in other embodiments, or may be replaced with configurations or features corresponding to other embodiments. It is obvious that claims without a clear reference relationship within the scope of the claims may be combined to configure the embodiments, or may be included as new claims by modification after the application.

[0214] The embodiments according to the present disclosure may be implemented by various means such as hardware, firmware, software or a combination thereof. For implementation by hardware, the embodiments of the present disclosure may be implemented by one or more ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, microcontrollers, microprocessors, etc.

[0215] In addition, for implementation by firmware or software, the embodiments of the present disclosure may be implemented in the form of modules, programs, functions, etc. that perform the above functions or operations, and may be recorded in a readable recording medium by various computer means. Here, the recording medium may include program instructions, data files, data structures, etc., alone or in combination. The program instructions recorded in the recording medium may be those program instructions specially designed and configured for the present disclosure, or those program instructions available by being notified to a computer software technician. For example, the recording medium includes: magnetic media such as hard disks, floppy disks, and tapes; optical media such as CD-ROMs (Compact Disc Read Only Memory) and DVDs (Digital Video Discs); magneto-optical media such as optical magnetic disks; and hardware devices such as ROMs, RAMs, flash memories, etc. that are specifically configured to store and execute program instructions. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter, etc., and machine language codes such as those generated by a compiler. Such a hardware device may be configured to operate as at least one software module for performing the operations of the present disclosure, and vice versa.

[0216] In addition, the device or terminal according to the present disclosure can be driven by a command that causes at least one processor to perform the above functions and processes. For example, such a command may include, for example, an interpreted command, such as a script command such as a JavaScript or ECMAScript command, or other commands stored in a computer-readable medium readable or executable code. In addition, the device according to the present disclosure can be implemented in a distributed manner across a network such as a server group, or can be implemented in a single computer device.

[0217] In addition, a computer program (also referred to as a program, software, software application, script or code) provided together with an apparatus according to the present disclosure and executing a method according to the present disclosure can be written in a programming language including a compiled or interpreted language or any form of a priori or procedural language, and can be deployed in any form including independent programs, modules, components or subroutines or other units suitable for use in a computer environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a single file provided for a requested program, or in a plurality of interactive files (e.g., a file storing a portion of at least one module, subroutine or code), or in a portion of a file having other programs or data (e.g., at least one script stored in a markup language document). A computer program may be located in a site or distributed across multiple sites, and may be deployed to be executed on a single computer or on multiple computers interconnected by a communication network.

[0218] It is obvious to those skilled in the art that the present disclosure may be implemented in other specific forms without departing from the basic features of the present disclosure. Therefore, the detailed description described above should not be interpreted restrictively in all aspects, but should be regarded as illustrative. The scope of the present disclosure should be determined by a reasonable interpretation of the attached claims, and all changes within the equivalent range of the present disclosure are included within the scope of the present disclosure.

[0219] [Industrial Applicability]

[0220] The present invention can be used for a grid compression method and device.

Claims

1. A grid compression method, include: reconstructing a base mesh based on geometric information obtained from a base mesh bitstream; performing surface partitioning on the reconstructed base mesh; Obtaining a displacement vector of at least one of a vertex of a base mesh for the reconstruction and a sub-vertex generated by the surface partitioning; reconstructing an enhanced mesh based on the reconstructed base mesh and the obtained displacement vectors; as well as A texture image associated with the surface of the reconstructed augmented mesh is obtained.

2. The method according to claim 1, in, The geometric information is encoded and decoded based on a first prediction mode, the displacement vector is encoded and decoded based on a second prediction mode, and the texture image is encoded and decoded based on a third prediction mode.

3. The method according to claim 2, in, The first prediction mode, the second prediction mode, and the third prediction mode are one of an inter prediction mode or an intra prediction method.

4. The method according to claim 2, in, At least one of the first prediction mode, the second prediction mode, or the third prediction mode is determined according to a predetermined coding unit.

5. The method according to claim 4, in, The coding unit is at least one of a frame, a patch, a surface or a vertex.

6. The method according to claim 1, further comprising: include: A current frame is reconstructed based on the reconstructed enhanced mesh and the texture image.

7. The method according to claim 6, further comprising: include: A first flag indicating whether to use texture image compression for the current frame is obtained.

8. The method according to claim 7, in, The first flag is signaled in units of a frame group including at least one frame.

9. The method according to claim 7, in, When the texture image compression is used for the current frame, the texture image is obtained by referring to a texture image of a frame encoded and decoded before the current frame.

10. The method according to claim 6, further comprising: include: A second flag indicating whether to encode and decode the texture image of the current frame is obtained.

11. The method according to claim 10, in, The second flag is signaled in frames.

12. The method according to claim 10, in, When encoding and decoding of the texture image of the current frame is omitted, the texture image is obtained by referring to a texture image of a frame having a picture order count (POC) value closest to the current frame among frames encoded and decoded before the current frame.

13. The method according to claim 1, in, Obtaining the texture image includes: obtaining a texture index indicating a texture image in a texture video including a plurality of texture images.

14. A grid compression device, include: a processor, the processor controlling the grid compression device; as well as a memory connected to the processor and storing data, Wherein, the processor: Reconstructing the base mesh based on the geometric information obtained from the base mesh bitstream, Perform surface partitioning on the reconstructed base mesh, obtaining a displacement vector of at least one of a vertex of a base mesh for the reconstruction and a sub-vertices generated by the surface partitioning, reconstructing an enhanced mesh based on the reconstructed base mesh and the obtained displacement vectors, and A texture image associated with the surface of the reconstructed augmented mesh is obtained.

Images