Method and apparatus for patch cell grid coding and decoding

Abstract

The invention discloses patch unit grid coding and decoding methods and devices. In this embodiment, a mesh decoding device decodes a bitstream to reconstruct patch information and a patch-based base mesh. A mesh decoding device reconstructs base mesh vertices and connectivity by using patch information and a patch-based base mesh. The mesh decoding device generates predicted vertices and connectivity on the basis of the reconstructed base mesh vertices and connectivity. A mesh decoding device decodes a bitstream to reconstruct a transform coefficient image, and then unpacks, inversely quantizes, and inversely transforms the transform coefficient image to reconstruct a vector difference of vertices. The mesh decoding device reconstructs the vertices and connectivity of the mesh by summing the predicted vertices and the vector differences.

Description

Technical Field

[0001] The invention relates to a patch-based mesh encoding and decoding method and device. Background Art

[0002] The statements in this section merely provide background information related to the present invention and may not necessarily constitute prior art.

[0003] In the April 2022 MPEG meeting, in response to the Call for Proposal (CfP) for V-Computer Graphics (CG), Apple's proposal showed the highest objective / subjective performance. Accordingly, MPEG 3D Graphic Compression (3DGC) decided to determine the test model for V-CG based on Apple's proposal at the next meeting.

[0004] The basic idea of ​​Apple's proposal is scalable mesh codec. Scalable mesh codec downsamples dense meshes to become coarse, compresses the coarse mesh (base mesh) by utilizing static mesh compression technology (e.g., Google Draco and MPEG-AFX), and encodes the difference between the dense mesh and the base mesh based on the reconstructed base mesh. In addition, in MPEG-3DGC, the V-CG standard considers sharing the Volumetric Visual Video based Coding (V3C) standard. Therefore, Apple's proposal also converts most of the information of the mesh into a 2D UV domain image by utilizing mapping, and then compresses the mesh by utilizing a video codec.

[0005] In the existing V-PCC / V3C, the encoder divides the 3D point cloud into multiple patches, maps each patch to the UV domain, and transmits the mapped information to the decoder. In Apple's proposal, the encoder generates patches and maps the generated patches to the UV domain, but since the corresponding information includes UV coordinate information for each vertex, the encoder does not transmit the patch information separately. However, in order to share the V3C standard, patch information is expected to be added in the future standardization process. Therefore, when the mesh is encoded / decoded in patches, it is necessary to consider a method of utilizing patch information. Summary of the invention

[0006] Technical issues

[0007] The present invention is directed to providing a mesh coding method and apparatus for segmenting a base mesh into a plurality of patches, and performing compression on a patch basis by utilizing static mesh coding to improve coding efficiency for a 3D mesh.

[0008] In addition, the present invention is directed to providing a mesh encoding and decoding method and apparatus for converting vertex coordinates and UV coordinates of a base mesh into vertex coordinates and UV coordinates within a patch by utilizing bounding box (i.e., start and end coordinates) information in 3D space and UV space included in patch information.

[0009] Technical Solution

[0010] At least one aspect of the present invention provides a method for decoding a mesh performed by a mesh decoding device. The method includes separating a bitstream into a base mesh bitstream, a transform coefficient bitstream, and an attribute mapping bitstream. Here, the base mesh bitstream includes a patch bitstream and a sub-base mesh bitstream. The method also includes decoding the base mesh bitstream to reconstruct a base mesh including base mesh vertices and connectivity. The method also includes decoding the patch bitstream to reconstruct patch information. The patch information includes an offset and a size of each patch. The method also includes decoding the sub-base mesh bitstream to reconstruct a patch-based base mesh. The method also includes reconstructing the base mesh vertices and connectivity by utilizing the patch information and the patch-based base mesh.

[0011] Another aspect of the present invention provides a method for encoding a mesh performed by a mesh encoding device. The method includes obtaining original vertices and connectivity and an original attribute map for the mesh. The method also includes encoding the original vertices and connectivity to generate a base mesh bitstream. The method also includes generating a reconstructed base mesh including reconstructed base mesh vertices and connectivity from the base mesh bitstream. The method also includes downsampling the original vertices and connectivity to generate a base mesh. Here, the base mesh includes base mesh vertices and connectivity. The method also includes classifying the base mesh into multiple patches by using the base mesh vertices and connectivity, and generating patch information for each patch. The patch information includes an offset and a size of each patch. The method also includes subtracting the offset from the position coordinate value of the base mesh vertex to generate a differential base mesh vertex and connectivity.

[0012] Another aspect of the present invention provides a computer-readable recording medium that stores a bitstream generated by a mesh coding method. The mesh coding method includes obtaining original vertices and connectivity and an original attribute map for a mesh. The mesh coding method also includes encoding the original vertices and connectivity to generate a base mesh bitstream. The mesh coding method also includes generating a reconstructed base mesh including reconstructed base mesh vertices and connectivity from the base mesh bitstream. The mesh coding method also includes downsampling the original vertices and connectivity to generate a base mesh. Here, the base mesh includes base mesh vertices and connectivity. The mesh coding method also includes classifying the base mesh into multiple patches using the base mesh vertices and connectivity, and generating patch information for each patch. The patch information includes an offset and a size of each patch. The mesh coding method also includes subtracting the offset from the position coordinate value of the base mesh vertex to generate a differential base mesh vertex and connectivity.

[0013] Beneficial Effects

[0014] As described above, the present invention provides a mesh encoding and decoding method and apparatus for segmenting a base mesh into multiple patches and performing compression on a patch basis by utilizing static mesh encoding and decoding. Therefore, the mesh encoding and decoding method and apparatus improves the encoding and decoding efficiency for 3D meshes.

[0015] In addition, according to the present embodiment, the present invention provides a mesh encoding and decoding method and apparatus for converting vertex coordinates and UV coordinates of a base mesh into vertex coordinates and UV coordinates within a patch by using bounding box (i.e., start and end coordinates) information in 3D space and UV space included in patch information. Therefore, the mesh encoding and decoding method and apparatus improves encoding and decoding efficiency for 3D meshes according to the reduction of the dynamic range of coordinate values. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram illustrating a grid coding device according to an embodiment of the present invention.

[0017] Figure 2 is a block diagram illustrating a basic grid encoder according to an embodiment of the present invention.

[0018] Figure 3 is a schematic diagram illustrating the operation of a mesh surface partitioner according to an embodiment of the present invention.

[0019] Figure 4 is a schematic diagram illustrating the operation of a vertex subtractor according to an embodiment of the present invention.

[0020] Figure 5 is a schematic diagram illustrating the operation of an image packager according to an embodiment of the present invention.

[0021] Figure 6 is a schematic diagram showing a basic grid encoder according to another embodiment of the present invention.

[0022] Figure 7 is a schematic diagram illustrating the operation of a patch generator according to an embodiment of the present invention.

[0023] Figure 8 is a schematic diagram illustrating the operation of an image packager according to an embodiment of the present invention.

[0024] Fig. 9 is a schematic diagram illustrating image packing on a patch basis according to an embodiment of the present invention.

[0025] Fig.10 is a schematic diagram showing a trellis decoding apparatus according to an embodiment of the present invention.

[0026] Fig.11 is a block diagram illustrating a base grid decoder according to an embodiment of the present invention.

[0027] Fig.12 is a block diagram illustrating a base grid decoder according to another embodiment of the present invention.

[0028] Fig.13 and Fig.14 is a flowchart illustrating a method for encoding a grid by a grid encoding device according to an embodiment of the present invention.

[0029] Fig.15 and Fig.16 is a flowchart illustrating a method for decoding a trellis by a trellis decoding device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, the same reference numerals represent the same elements, although the elements are shown in different drawings. In addition, in the following description of some embodiments, when it is considered to be obscure the subject of the present invention, the specific description of the related known components and functions may be omitted for the sake of clarity and brevity.

[0031] Figure 1 is a block diagram of a video encoding device that can implement the technology of the present invention. Figure 1 A video encoding device and components of the device are described in detail.

[0032] The encoding device may include: an image divider 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a rearrangement unit 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filtering unit 180 and a memory 190.

[0033] The present embodiment discloses a method and apparatus for patch-based mesh encoding and decoding. More specifically, the present embodiment provides a mesh encoding and decoding method and apparatus for segmenting a base mesh into multiple patches, and performing compression on a patch basis by utilizing static mesh encoding and decoding to improve encoding and decoding efficiency for 3D meshes. In addition, the present embodiment provides a mesh encoding and decoding method and apparatus for converting vertex coordinates and UV coordinates of a base mesh into vertex coordinates and UV coordinates within a patch by utilizing bounding box (i.e., start and end coordinates) information in 3D space and UV space included in patch information.

[0034] Figure 1 is a block diagram illustrating a grid coding device according to an embodiment of the present invention.

[0035] A mesh encoding device (hereinafter, used interchangeably with "encoding device") encodes original vertices and connectivity and an original attribute map for a mesh to generate a bitstream. Here, the vertices of the mesh may include coordinate values ​​in a three-dimensional (3D) space and coordinate values ​​in a two-dimensional (2D) UV domain. In addition, the connectivity of the mesh may be information representing a surface of the mesh as a set of vertices in a 3D space. In addition, the attribute map may be information representing an attribute value of a mesh surface.

[0036] The encoding device may include all or some of a base mesh encoder 102, a mesh surface segmenter 104, a vertex subtractor 106, a transformer 108, a quantizer 110, an image packer 112, a video encoder 114 or 130, an image unpacker 116, an inverse quantizer 118, an inverse transformer 120, a vertex adder 122, an attribute information mapper 124, a filler 126, a color space converter 128 and a bitstream synthesizer 132.

[0037] Figure 2 is a block diagram illustrating a basic grid encoder according to an embodiment of the present invention.

[0038] The base mesh encoder 102 may encode the received original vertices and connectivity to generate a base mesh bitstream and a reconstructed base mesh. The reconstructed base mesh represents the reconstructed base mesh vertices and connectivity. The base mesh bitstream may be transmitted to the bitstream synthesizer 132. In addition, the reconstructed base mesh vertices and connectivity may be transmitted to the mesh surface segmenter 104. In addition, the reconstructed base mesh vertices and connectivity may be transmitted to the attribute information mapper 124 together with the child vertices generated by the vertex adder 122.

[0039] On the other hand, the base mesh vertices and connectivity correspond to the geometric information of the base mesh. However, the base mesh encoder 102 is only for the geometric information of the base mesh. Hereinafter, encoding or decoding the base mesh means encoding or decoding the base mesh vertices and connectivity.

[0040] The base grid encoder 102 may include all or some of a grid downsampler 202, a static grid encoder 204, a motion predictor 206, a motion vector encoder 208, a static grid decoder 210, a motion vector decoder 212, a motion compensator 214, and a memory 216. Among the components of the base grid encoder 102, the grid downsampler 202, the static grid encoder 204, and the static grid decoder 210 directly encode or decode the base grid. In addition, the motion predictor 206, the motion vector encoder 208, the motion vector decoder 212, and the motion compensator 214 encode or decode the base grid using motion vectors.

[0041] The mesh downsampler 202 may receive the original vertices and connectivity and may downsample the mesh to generate base mesh vertices and connectivity. The generated base mesh vertices and connectivity may be transmitted to the static mesh encoder 204.

[0042] The static mesh encoder 204 may encode the input base mesh vertices and connectivity to generate a base mesh bitstream. The static mesh encoder 204 may encode the base mesh by utilizing a static mesh encoding method (e.g., Google Draco or MPEG-AFX). The generated bitstream may be output from the base mesh encoder 102. In addition, the bitstream may be transmitted to a static mesh decoder 210.

[0043] The static mesh decoder 210 may decode the input base mesh bitstream to reconstruct the base mesh vertices and connectivity. The reconstructed base mesh vertices and connectivity may be output from the base mesh encoder 102. In addition, the reconstructed base mesh vertices and connectivity may be transferred to the memory 216 for subsequent frame encoding.

[0044] The motion predictor 206 may perform motion prediction to generate motion vectors based on the input original vertices and connectivity and the reconstructed base mesh vertices and connectivity of the previous frame stored in the memory 216. The generated motion vectors may be transmitted to the motion vector encoder 208.

[0045] The motion vector encoder 208 may encode the input motion vector to generate a base grid bitstream. The generated bitstream may be output from the base grid encoder 102. In addition, the bitstream may be transmitted to the motion vector decoder 212.

[0046] The motion vector decoder 212 may decode the input base grid bitstream to reconstruct the motion vector. The reconstructed motion vector may be transmitted to the motion compensator 214.

[0047] The motion compensator 214 may generate a base mesh of the current frame by compensating for motion of base mesh vertices of a previous frame received from the memory 216 using the motion vectors received from the motion vector decoder 212. The reconstructed base mesh may be output from the base mesh encoder 102. In addition, the reconstructed base mesh vertices and connectivity may be transmitted to the memory 216 for subsequent frame encoding.

[0048] The memory 216 may store the input reconstructed base grid. The memory 216 may transmit the reconstructed base grid to the motion predictor 206 for the next frame encoding. In addition, the memory 216 may transmit the reconstructed base grid to the motion compensator 214 for the next frame decoding.

[0049] Figure 3 is a schematic diagram illustrating the operation of a mesh surface partitioner according to an embodiment of the present invention.

[0050] The mesh surface segmenter 104 may perform surface segmentation by receiving the reconstructed base mesh vertices and connectivity to generate sub-vertices. In addition, connectivity may be generated to connect the generated sub-vertices to the existing base mesh vertices. As an example, as in Figure 3 In an example of , the surface segmentation method can generate a sub-vertex in the middle of two reconstructed base vertices connected to each other to split a triangular face into four triangular faces. As another example, the surface segmentation method can generate a vertex at one of three edges to split a triangular face into two triangular faces. The generated sub-vertex can become a predicted vertex together with the reconstructed base mesh vertex. The predicted vertex and connectivity can be transmitted to the vertex subtractor 106 and the vertex adder 122.

[0051] Figure 4 is a schematic diagram illustrating the operation of a vertex subtractor according to an embodiment of the present invention.

[0052] Vertex subtractor 106 may receive the original vertices and connectivity and the predicted vertices and connectivity and subtract the predicted vertices from the original vertices to generate a vector difference, as in Figure 4 The generated vector difference and connectivity may be transmitted to the transformer 108 .

[0053] The transformer 108 may perform a transformation on the input vector difference to generate a transformation coefficient, and may transmit the generated transformation coefficient to the quantizer 110. Here, a wavelet transform may be used as a transformation method. In addition, a lifting transform as a type of wavelet transform may be used. Here, the connectivity of the vector difference may be used for the wavelet transform.

[0054] The quantizer 110 may quantize the input transform coefficient to generate a quantized transform coefficient, and may transmit the generated quantized transform coefficient to the image packer 112 .

[0055] Figure 5 is a schematic diagram illustrating the operation of an image packager according to an embodiment of the present invention.

[0056] The image packer 112 may pack the input transform coefficients into a 2D space to generate a transform coefficient image. For example, the transform coefficients may be packed on an M×N block basis according to a raster scan order. Figure 5 In the example of , the transform coefficients are packed in a raster scan order on a 2×2 block basis. Alternatively, various scanning methods may be used, such as vertical scanning, horizontal scanning, diagonal zig-zag scanning, etc. The block size and the scanning method may be transmitted to a trellis decoding device (hereinafter, used interchangeably with a “decoding device”) while being included in the bitstream. On the other hand, the generated transform coefficient image may be transmitted to the video encoder 114.

[0057] The video encoder 114 may encode the input transform coefficient image to generate a transform coefficient bitstream. In addition, the video encoder 114 may generate a reconstructed transform coefficient image from the transform coefficient bitstream. Here, the video encoder 114 may use various video coding techniques of related technologies, such as H.264 / Advanced Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), H.266 / Versatile Video Coding (VVC), EVC, VP9, ​​AV1, AVS2, etc. The generated bitstream may be transmitted to the bitstream synthesizer 132. In addition, the reconstructed transform coefficient image may be transmitted to the image depacketizer 116.

[0058] The image unpacker 116 may obtain quantized transform coefficients of vertices from the input transformed coefficient image. Here, the block size and scanning method used in the image packer 112 may be used. The quantized transform coefficients may be transmitted to the inverse quantizer 118.

[0059] The inverse quantizer 118 may inversely quantize the input quantized transform coefficient to reconstruct the transform coefficient. The reconstructed transform coefficient may be transmitted to the inverse transformer 120.

[0060] The inverse transformer 120 may inversely transform the input transformation coefficient to reconstruct the vector difference of the vertex. The reconstructed vector difference may be transmitted to the vertex adder 122.

[0061] The vertex adder 122 may reconstruct the vertex by adding the input predicted vertex and the reconstructed vector difference. The reconstructed vertex and connectivity may be transmitted to the attribute information mapper 124. As described above, since the predicted vertex includes the reconstructed base mesh vertex and sub-vertices, the reconstructed vertex may also correspond to the reconstructed base mesh vertex and sub-vertices.

[0062] The attribute information mapper 124 may receive the original vertices and connectivity; the reconstructed vertices and connectivity; and the original attribute map, and may correct the attribute information. Thereafter, the attribute information mapper 124 may map the corrected attribute information to the 2D UV space to generate a corrected attribute map. The corrected attribute map may be transmitted to the filler 126.

[0063] The filler 126 may perform padding on the input corrected attribute map. Here, a push-pull padding method may be used as the padding method. Alternatively, a nearest padding method may be used. The padded attribute map may be transmitted to the color space converter 128.

[0064] Here, the push-pull filling method hierarchically performs downsampling on the target frame (attribute map in the present invention), hierarchically performs upsampling, and then combines the foreground area of ​​the same layer and the upsampled background area. The push-pull filling method can improve video encoding and decoding efficiency by smoothing the edge area caused by the foreground texture packed on a patch basis.

[0065] The color space converter 128 may perform color space conversion on the input filled attribute map. Here, information on the color space conversion may be transmitted to the decoding device while being included in the bitstream. The attribute map whose color space has been converted may be transmitted to the video encoder 130.

[0066] The video encoder 130 may encode the input property map to generate a property map bitstream. The generated bitstream may be transmitted to the bitstream synthesizer 132.

[0067] The bitstream synthesizer 132 connects all input bitstreams (ie, the base grid bitstream, the transform coefficient bitstream, and the attribute map bitstream) to generate one bitstream. The synthesized bitstream may be output from the encoding apparatus.

[0068] Figure 6 is a schematic diagram showing a basic grid encoder according to another embodiment of the present invention.

[0069] As described above, the base mesh encoder 102 can encode the input original vertices and connectivity to generate a base mesh bitstream and a reconstructed base mesh. The base mesh encoder 102 can include all or some of the mesh downsampler 202, the patch generator 602, the patch offset subtractor 604, the static mesh encoder 204, the patch information encoder 606, the motion predictor 206, the motion vector encoder 208, the patch offset adder 608, the motion vector decoder 212, the motion compensator 214, the memory 216, and the bitstream synthesizer 610.

[0070] Among the components of the base grid encoder 102, the grid downsampler 202, the patch generator 602, the patch offset subtractor 604, the static grid encoder 204, the static grid decoder 210, the patch information encoder 606, and the patch offset adder 608 encode or decode the base grid by using the patch. In addition, the motion predictor 206, the motion vector encoder 208, the motion vector decoder 212, and the motion compensator 214 encode or decode the base grid by using the motion vector.

[0071] The mesh downsampler 202 may receive the original vertices and connectivity and may downsample the mesh to generate base mesh vertices and connectivity. The generated base mesh vertices and connectivity may be transmitted to the patch generator 602.

[0072] Figure 7 is a schematic diagram illustrating the operation of a patch generator according to an embodiment of the present invention.

[0073] The patch generator 602 can classify the base mesh into multiple patches by utilizing the input base mesh vertices and connectivity. Figure 7 In the example of , the patch generator 602 can generate patch information including the starting position (ie, patch offset) and size of each patch by using the generated patches. Figure 7In the example of , the patch information may include 2D space (UV space) patch information and 3D space patch information. The generated patch information may be transmitted to the patch offset subtractor 604, the patch information encoder 606, and the patch offset adder 608.

[0074] The patch information encoder 606 may encode the input patch information to generate a patch bitstream. The generated patch bitstream may be transmitted to the bitstream synthesizer 610.

[0075] The patch offset subtractor 604 can correct the coordinate values ​​of the base mesh vertices by subtracting the patch offset from the position coordinate values ​​of the base mesh vertices using the input patch information. Here, the patch offset can be the coordinate value of the starting position of the patch bounding box. The differential base mesh vertices and connectivity can be transmitted to the static mesh encoder 204.

[0076] The static mesh encoder 204 can encode the input differential base mesh vertices and connectivity to generate a sub-base mesh bitstream. Here, the static mesh encoder 204 can encode the differential base mesh by utilizing a static mesh encoding method (e.g., Google Draco or MPEG-AFX). The generated bitstream can be output from the base mesh encoder 102. In addition, the bitstream can be transmitted to the static mesh decoder 210.

[0077] The static mesh decoder 210 may decode the input sub-base mesh bitstream to reconstruct the differential base mesh vertices and connectivity. The reconstructed differential base mesh vertices and connectivity may be transmitted to the patch offset adder 608.

[0078] The patch offset adder 608 may receive the reconstructed differential base mesh vertices and connectivity and the patch information. The patch offset adder 608 may reconstruct the base mesh vertices by adding the input reconstructed differential base mesh vertices and offsets. Here, the offset may be a coordinate value of the start position of the patch included in the patch information. The reconstructed base mesh vertices and connectivity may be output from the base mesh encoder 102. In addition, the reconstructed base mesh vertices and connectivity may be transmitted to the memory 216 for subsequent frame encoding.

[0079] The motion predictor 206 may perform motion prediction to generate motion vectors based on the input original vertices and connectivity and the reconstructed base mesh vertices and connectivity of the previous frame stored in the memory 216. The generated motion vectors may be transmitted to the motion vector encoder 208.

[0080] The motion vector encoder 208 may encode the input motion vector to generate a sub-basis grid bitstream. The generated bitstream may be transmitted to the bitstream synthesizer 610. In addition, the bitstream may be transmitted to the motion vector decoder 212.

[0081] The motion vector decoder 212 may decode the input bitstream to reconstruct the motion vector. The reconstructed motion vector may be transmitted to the motion compensator 214.

[0082] The motion compensator 214 may generate a base mesh of a current frame by compensating for motion of base mesh vertices of a previous frame transmitted from the memory 216 using the motion vector transmitted from the motion vector decoder 214. The reconstructed base mesh may be output from the base mesh encoder 102. In addition, the reconstructed base mesh vertices and connectivity may be transmitted to the memory 216 for next frame encoding.

[0083] The memory 216 may store the input reconstructed base grid. The memory 216 may transmit the reconstructed base grid to the motion predictor 206 for subsequent frame encoding. In addition, the memory 216 may transmit the reconstructed base grid to the motion compensator 214 for subsequent frame decoding.

[0084] The bitstream synthesizer 610 may connect all received bitstreams (ie, patch bitstreams and sub base grid bitstreams) to generate a base grid bitstream. The generated base grid bitstream may be output from the base grid encoder 102.

[0085] Figure 8 is a schematic diagram illustrating the operation of an image packager according to an embodiment of the present invention.

[0086] The image packer 112 in the encoding device may pack the transform coefficients received on a patch basis in the order of 2D blocks. Here, the block size may be determined based on preset horizontal and vertical sizes. Alternatively, the block size may be determined based on the number of transform coefficients included in the patch. Information related to the block may be entropy encoded and then may be transmitted to the decoding device.

[0087] exist Figure 8 In the example of , the image packer 112 determines the size of the square block based on the number of transform coefficients included in each patch. The image packer 112 can pack the transform coefficients into the determined blocks in a scan order. If the number of transform coefficients packed into one block is less than the internal area of ​​the block, the image packer 112 can fill all empty spaces with zeros. Alternatively, the empty spaces can be filled with an intermediate value of the bit depth used by the video encoder 114. Alternatively, the empty spaces can be filled with the most recent transform coefficients.

[0088] exist Figure 8 In the example of , the image packer 112 packs the transform coefficients according to the raster scan order on a block basis. Various methods can be used for the scan order, and the scan order information can be entropy encoded and then transmitted to the decoding device. As described above, in Figure 8 In the example of , blocks of different sizes may be determined based on the number of transform coefficients included in the patch. In order to efficiently pack blocks of different sizes into a 2D image, the transform coefficients may be packed in the form of a quadtree, as in Figure 8 In the example on the right. Alternatively, various tree methods may be used, such as a binary tree, a ternary tree, etc. The tree information used may be included in the bitstream and may be transmitted to the decoding device.

[0089] In addition, if Fig. 9 When generating multiple blocks, the image packer 112 can pack the multiple blocks in a packing order that changes according to the block size. The image packer 112 can first pack blocks of the same size in order, increase the quadtree depth by one, and then pack the next smaller size block.

[0090] In addition, the quantizer 112 can adjust the quantization degree of the transform coefficient by using different quantization parameters on a path basis. In addition, the encoding device can transmit the corresponding quantization parameter information to the video encoder 114, and can encode the transform coefficient image based on the transmitted quantization parameter, thereby performing video encoding more efficiently. Generally, the video encoder 114 transmits the quantization parameter on a block basis. Therefore, the encoding device can perform more accurate bit rate adjustment by using the quantization parameter information on a patch basis. In addition, by performing block segmentation at a high speed using patch information in the video encoder 114, the complexity of the encoding device can be reduced. As described above, the generated transform coefficient image can be transmitted to the video encoder 114.

[0091] The trellis encoding device may store the bit stream of the encoded trellis data in a non-volatile storage medium or transmit the bit stream to the trellis decoding device by utilizing a communication network.

[0092] Fig.10 is a schematic diagram showing a trellis decoding apparatus according to an embodiment of the present invention.

[0093] A mesh decoding device (i.e., a "decoding device") can decode an input bitstream to generate reconstructed vertices, connectivity, and attribute maps to reconstruct a mesh. The decoding device may include all or some of a bitstream separator 1002, a base mesh decoder 1004, a mesh surface segmenter 1006, video decoders 1008 and 1018, an image unpacker 1010, an inverse quantizer 1012, an inverse transformer 1014, a vertex adder 1016, and a color space converter 1020.

[0094] The bitstream separator 1002 can separate the input bitstream to generate a base grid bitstream, a transform coefficient bitstream, and an attribute map bitstream. The base grid bitstream can be transmitted to the base grid decoder 1004. The transform coefficient bitstream and the attribute map bitstream can be transmitted to the video decoder 1006 and 1018 respectively. Here, each bitstream can be input to a separate video decoder. Or, each bitstream can be continuously input to a video decoder.

[0095] Fig.11 is a block diagram illustrating a base grid decoder according to an embodiment of the present invention.

[0096] The base mesh decoder 1004 can decode the input base mesh bitstream to generate a reconstructed base mesh. The reconstructed base mesh represents the reconstructed base mesh vertices and connectivity. The base mesh decoder 1004 can include all or some of the static mesh decoder 1102, the motion vector decoder 1104, the motion compensator 1106, and the memory 1108.

[0097] On the other hand, the base mesh vertices and connectivity correspond to the geometric information of the base mesh. However, the base mesh decoder 1004 is only for the geometric information of the base mesh. In the following, decoding the base mesh refers to decoding the base mesh vertices and connectivity.

[0098] Among the components of the base mesh decoder 1004, the static mesh decoder 1102 directly decodes the base mesh. In addition, the motion vector decoder 1104 and the motion compensator 1106 decode the base mesh by using the motion vector.

[0099] The static mesh decoder 1102 can decode the input bitstream to reconstruct the base mesh vertices and connectivity. The reconstructed base mesh vertices and connectivity can be transmitted from the base mesh decoder 1004 to the mesh surface segmenter 1006. In addition, the reconstructed base mesh vertices and connectivity can be transmitted to the memory 1108 for subsequent frame encoding.

[0100] The motion vector decoder 1104 may decode the input bitstream to reconstruct the motion vector. The reconstructed motion vector may be transmitted to the motion compensator 1106.

[0101] The motion compensator 1106 may generate a base mesh of the current frame by compensating for motion of base mesh vertices of a previous frame received from the memory 1108 using the motion vectors received from the motion vector decoder 1104. The reconstructed base mesh may be output from the base mesh decoder 1104. In addition, the reconstructed base mesh vertices and connectivity may be transmitted to the memory 1108 for encoding a next frame.

[0102] The memory 1108 may store the input reconstructed base mesh. In addition, the memory 1108 may transmit the reconstructed base mesh to the motion compensator 1106 for subsequent frame decoding.

[0103] Fig.12 is a block diagram illustrating a base grid decoder according to another embodiment of the present invention.

[0104] As described above, the base grid decoder 1004 can decode the input base grid bitstream to generate a reconstructed base grid. The base grid decoder 1004 may include all or some of the bitstream separator 1202, the patch information decoder 1204, the static grid decoder 1102, the patch synthesizer 1206, the motion vector decoder 1104, the motion compensator 1106, and the memory 1108. Among the components of the base grid decoder 1004, the patch information decoder 1204, the static grid decoder 1102, and the patch synthesizer 1206 decode the base grid by using the patch. In addition, the motion vector decoder 1104 and the motion compensator 1106 decode the base grid by using the motion vector.

[0105] The bitstream separator 1202 may separate the input base grid bitstream to generate a patch bitstream and a sub base grid bitstream. The patch bitstream may be transmitted to the patch information decoder 1204. The sub base grid bitstream may be transmitted to the static grid decoder 1102 or the motion vector decoder 1102.

[0106] The patch information decoder 1204 can decode the input patch bitstream to reconstruct the patch information. As described above, the patch information can include 2D space (UV space) patch information and 3D space patch information. In addition, the patch information includes an offset (i.e., a starting position) and a size of each patch. The reconstructed patch information can be transmitted to the patch synthesizer 1206.

[0107] The static mesh decoder 1102 may decode the input sub-base mesh bitstream to reconstruct the patch-based base mesh. The reconstructed patch-based base mesh may be transmitted to the patch synthesizer 1206.

[0108] The patch synthesizer 1206 may generate a reconstructed base mesh by utilizing the input patch information and the reconstructed patch-based base mesh. The patch synthesizer 1206 may first correct the mesh vertex information by adding the offset reconstructed on the basis of the path to the vertex information of the base mesh by utilizing the patch information. Thereafter, the patch synthesizer 1206 may reconstruct the base mesh by collecting the corrected patches in the same space. The reconstructed base mesh vertices and connectivity may be output from the base mesh decoder 1004. In addition, the reconstructed base mesh vertices and connectivity may be transmitted to the memory 1108 for subsequent frame encoding.

[0109] The motion vector decoder 1104 may decode the input sub-basis grid bitstream to reconstruct the motion vector. The reconstructed motion vector may be transmitted to the motion compensator 1106.

[0110] The motion compensator 1106 may generate a base mesh of the current frame by compensating for motion of base mesh vertices of a previous frame received from the memory 1108 using the motion vectors received from the motion vector decoder 1104. The reconstructed base mesh may be output from the base mesh decoder 1104. In addition, the reconstructed base mesh vertices and connectivity may be transmitted to the memory 1108 for encoding a next frame.

[0111] The memory 1108 may store the input reconstructed base mesh. In addition, the memory 1108 may transmit the reconstructed base mesh to the motion compensator 1106 for subsequent frame decoding.

[0112] exist Fig.10 In the example of , the mesh surface segmenter 1006 can receive the reconstructed base mesh vertices and connectivity and perform surface segmentation to generate child vertices. In addition, the mesh surface segmenter 1006 can generate connectivity connecting the generated child vertices and the existing base mesh vertices. The generated child vertices can become predicted vertices together with the reconstructed base mesh vertices. The predicted vertices and connectivity can be transmitted to the vertex adder 1016.

[0113] The video decoder 1008 may decode the input transform coefficient bitstream to reconstruct the transform coefficient image. The reconstructed transform coefficient image may be transmitted to the image depacketizer 1010.

[0114] The image unpacker 1010 may generate quantized transform coefficients of vertices from the input transform coefficient image. Here, the block size and scanning method used in the image packer 112 may be used. The quantized transform coefficients may be transmitted to the inverse quantizer 1012.

[0115] The inverse quantizer 1012 may inversely quantize the input quantized transform coefficient to reconstruct the transform coefficient. The reconstructed transform coefficient may be transmitted to the inverse transformer 1014.

[0116] The inverse transformer 1014 may inversely transform the input transformation coefficients to reconstruct the vector difference of the vertex. The reconstructed vector difference may be transmitted to the vertex adder 1016.

[0117] The vertex adder 1016 may add the input predicted vertex and the reconstructed vector difference to reconstruct the vertex. The reconstructed vertex and connectivity may be output from the decoding device. As described above, since the predicted vertex includes the reconstructed base mesh vertex and sub-vertices, the reconstructed vertex may also correspond to the reconstructed base mesh vertex and sub-vertices.

[0118] The video decoder 1018 may decode the input property map bitstream to reconstruct the property map. The reconstructed property map may be transmitted to the color space converter 1020.

[0119] The color space converter 1020 may convert the input reconstructed attribute map into a color space of the attribute map that is the same as the original mesh. The attribute map whose color space is converted may be output from the decoding device.

[0120] In the following, using Figures 13 to 16 A diagram is used to describe the method of encoding and decoding the grid.

[0121] Fig.13 and Fig.14 is a flowchart illustrating a method of encoding a grid by a grid encoding device according to an embodiment of the present invention.

[0122] The encoding apparatus obtains original vertices and connectivity and an original attribute map for a mesh (S1300).

[0123] The encoding device encodes the original vertices and connectivity to generate a base mesh bitstream and a reconstructed base mesh (S1302). Here, the reconstructed base mesh includes the reconstructed base mesh vertices and connectivity.

[0124] The encoding apparatus generates predicted vertices and connectivity based on the reconstructed base mesh vertices and connectivity (S1304). Here, the predicted vertices include the reconstructed base mesh vertices and sub-vertices.

[0125] The encoding device may perform surface segmentation based on the reconstructed base mesh vertices and connectivity to generate child vertices. In addition, the encoding device may generate connectivity that connects the generated child vertices to the existing base mesh vertices. The generated child vertices may become predicted vertices together with the reconstructed base mesh vertices.

[0126] The encoding apparatus subtracts the predicted vertex from the original vertex to generate a vector difference based on the original vertex and connectivity and the predicted vertex and connectivity (S1306).

[0127] The encoding apparatus transforms, quantizes, and packs the vector difference to generate a transform coefficient image (S1308).

[0128] The encoding apparatus encodes a transform coefficient image by using a video encoding method to generate a transform coefficient bitstream and generates a reconstructed transform coefficient image ( S1310 ).

[0129] The encoding apparatus unpacks, dequantizes, and inversely transforms the reconstructed transform coefficient image to reconstruct the vector difference (S1312).

[0130] The encoding apparatus adds the predicted vertices and the reconstructed vector differences to reconstruct the vertices and connectivity for the mesh (S1314).

[0131] The encoding apparatus generates a corrected attribute map by using the original vertices and connectivity, the original attribute map, and the reconstructed vertices and connectivity (S1316).

[0132] Additionally, the encoding apparatus may apply padding and color space conversion to the corrected attribute map.

[0133] The encoding apparatus encodes the corrected property map by using a video encoding method to generate a property map bitstream (S1318).

[0134] The encoding apparatus combines a base grid bitstream, a transform coefficient bitstream, and a property map bitstream (S1320).

[0135] On the other hand, the operation of generating a base grid bitstream and a reconstructed base grid (S1302) may include the following operations.

[0136] The encoding apparatus downsamples the original vertices and connectivity to generate a base mesh (S1400). Here, the base mesh includes base mesh vertices and connectivity.

[0137] The encoding apparatus classifies the base mesh into a plurality of patches by using base mesh vertices and connectivity, and generates patch information for each patch (S1402). Here, the patch information includes an offset and a size of each patch.

[0138] The encoding apparatus encodes the patch information to generate a patch bitstream (S1404).

[0139] The encoding apparatus subtracts the offset from the position coordinate value of the base mesh vertex to generate a differential base mesh vertex and connectivity (S1406).

[0140] The encoding apparatus encodes the differential base mesh vertices and connectivity based on a static mesh encoding method to generate a sub-base mesh bitstream (S1408).

[0141] The encoding apparatus reconstructs differential base mesh vertices and connectivity from the sub-base mesh bitstream (S1410).

[0142] The encoding apparatus adds the reconstructed differential base mesh vertices and offsets to reconstruct the base mesh vertices and connectivity (S1412).

[0143] The encoding apparatus performs motion prediction based on the original vertices and connectivity and the reconstructed base mesh vertices and connectivity of the previous frame to generate a motion vector (S1414).

[0144] The encoding device encodes the motion vector to generate a sub-base grid bitstream (S1416).

[0145] The encoding apparatus decodes the sub base grid bitstream to reconstruct the motion vector (S1418).

[0146] The encoding apparatus reconstructs base mesh vertices and connectivity by compensating for motions of reconstructed base mesh vertices of a previous frame using the reconstructed motion vectors ( S1420 ).

[0147] The encoding apparatus stores the reconstructed base mesh vertices and connectivity (S1422).

[0148] The encoding apparatus combines the patch bitstream and the sub base grid bitstream to generate a base grid bitstream (S1424).

[0149] Fig.15 and Fig.16 is a flowchart illustrating a method of decoding a trellis by a trellis decoding apparatus according to an embodiment of the present invention.

[0150] The decoding apparatus separates a bit stream into a base grid bit stream, a transform coefficient bit stream, and an attribute map bit stream (S1500).

[0151] The decoding device decodes the base mesh bitstream to reconstruct the base mesh (S1502). Here, the reconstructed base mesh includes reconstructed base mesh vertices and connectivity.

[0152] The decoding apparatus generates predicted vertices and connectivity based on the reconstructed base mesh vertices and connectivity (S1504). Here, the predicted vertices include the reconstructed base mesh vertices and sub-vertices.

[0153] The decoding device may perform surface segmentation based on the reconstructed base mesh vertices and connectivity to generate sub-vertices. In addition, the decoding device may generate connectivity connecting the generated sub-vertices and the existing base mesh vertices. The generated sub-vertices may become predicted vertices together with the reconstructed base mesh vertices.

[0154] The decoding apparatus decodes the transform coefficient bit stream to reconstruct the transform coefficient image (S1506).

[0155] The decoding apparatus unpacks, dequantizes, and inversely transforms the transform coefficient image to reconstruct vector differences of vertices (S1508).

[0156] The decoding apparatus adds the predicted vertices and vector differences to reconstruct the vertices and connectivity (S1510).

[0157] The decoding apparatus decodes the property map bitstream based on the video decoding method to reconstruct the property map (S1512).

[0158] Additionally, the decoding apparatus may apply a color space conversion to the reconstructed attribute map.

[0159] On the other hand, the operation of reconstructing the basic mesh (S1502) may include the following operations.

[0160] The decoding apparatus separates a base grid bitstream into a patch bitstream and a sub-base grid bitstream (S1600).

[0161] The decoding apparatus decodes the patch bitstream to reconstruct patch information (S1602). Here, the patch information includes an offset and a size of each patch.

[0162] The decoding apparatus decodes the sub-base grid bitstream to reconstruct the patch-based base grid ( S1604 ).

[0163] The decoding apparatus reconstructs base mesh vertices and connectivity by using the patch information and the patch-based base mesh ( S1606 ).

[0164] The decoding device corrects the patch-based base mesh by adding the offset to the patch-based base mesh. Thereafter, the decoding device may collect the corrected patch-based base meshes in the same space to generate base mesh vertices and connectivity.

[0165] The decoding apparatus decodes the sub base grid bitstream to reconstruct a motion vector (S1608).

[0166] The decoding apparatus reconstructs base mesh vertices and connectivity by compensating for motions of base mesh vertices of a previous frame using motion vectors ( S1610 ).

[0167] The decoding apparatus stores the reconstructed base mesh vertices and connectivity (S1612).

[0168] Although the steps in the various flow charts are described as being performed sequentially, these steps merely illustrate the technical concepts of some embodiments of the present invention. Therefore, a person of ordinary skill in the art to which the present invention belongs can perform the steps by changing the order described in the various figures or by performing two or more steps in parallel. Therefore, the steps in the various flow charts are not limited to the time sequence shown.

[0169] It should be understood that the above description presents illustrative embodiments that can be implemented in various other ways. The functions described in some embodiments can be implemented by hardware, software, firmware and / or their combination. It should also be understood that the functional components described in the present invention are marked as "... units" to highlight the possibility of their independent implementation.

[0170] On the other hand, the various methods or functions described in some embodiments can be implemented as instructions stored in a non-volatile recording medium, which can be read and executed by one or more processors. The non-volatile recording medium may include, for example, various types of recording devices that store data in a form that is readable by a computer system. For example, the non-volatile recording medium may include storage media, such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs), etc.

[0171] Although embodiments of the present invention are described for illustrative purposes, it will be appreciated by those skilled in the art that the present invention belongs to that various modifications, additions and substitutions may be made without departing from the spirit and scope of the present invention. Therefore, for the sake of brevity and clarity, embodiments of the present invention are described. The scope of the technical ideas of the embodiments of the present invention are not limited by the examples. Accordingly, it will be appreciated by those skilled in the art that the scope of the present invention should not be limited by the embodiments clearly described above, but by the claims and their equivalents.

[0172] Reference numerals:

[0173] 1002: Bitstream separator

[0174] 1004: Basic Grid Decoder

[0175] 1006: Mesh Surface Splitter

[0176] 1008, 1018: Video decoder

[0177] 1010: Image unpacker

[0178] 1012: Inverse Quantizer

[0179] 1014: Inverse Converter

[0180] 1016: Vertex Adder

[0181] 1020: Color space converter.

[0182] CROSS-REFERENCE TO RELATED APPLICATIONS

[0183] This application claims priority to and the benefit of Korean Patent Application No. 10-2022-0059436, filed on May 16, 2022, and Korean Patent Application No. 10-2023-0057829, filed on May 3, 2023, the entire contents of each of which are incorporated herein by reference.

Claims

1. A method for decoding a grid performed by a grid decoding device, the method comprising: Separating a bitstream into a base grid bitstream, a transform coefficient bitstream, and an attribute map bitstream, wherein the base grid bitstream includes a patch bitstream and a sub-base grid bitstream; and decoding the base mesh bitstream to reconstruct the base mesh including base mesh vertices and connectivity, Among them, reconstructing the basic grid includes: Decoding the patch bitstream to reconstruct patch information, wherein the patch information includes an offset and a size of each patch; decoding the sub-base grid bitstreams to reconstruct the patch-based base grid; and The base mesh vertices and connectivity are reconstructed by utilizing the patch information and the patch-based base mesh.

2. The method according to claim 1, further comprising: performing surface segmentation based on the reconstructed base mesh vertices and connectivity to generate predicted vertices and connectivity, wherein the predicted vertices include the reconstructed base mesh vertices and sub-vertices; decoding a transform coefficient bitstream based on a video decoding method to reconstruct vector differences for vertices of a mesh; Adding the predicted vertices and vector differences to reconstruct vertices and connectivity; and The attribute map bitstream is decoded based on a video decoding method to reconstruct the attribute map.

3. The method according to claim 1, wherein: Reconstructing the base mesh includes: separating a base grid bitstream into a patch bitstream and a sub-base grid bitstream; and Stores the reconstructed base mesh vertices and connectivity.

4. The method according to claim 1, wherein: Reconstructing base mesh vertices and connectivity includes: adding an offset to the patch-based base mesh to correct the patch-based base mesh; and The corrected patch-based base meshes in the same space are collected to generate base mesh vertices and connectivity.

5. The method according to claim 1, wherein: Reconstructing the base mesh includes: decoding the sub-basis grid bitstream to reconstruct motion vectors; reconstructing base mesh vertices and connectivity by compensating for motion of base mesh vertices of a previous frame using motion vectors; and Stores the reconstructed base mesh vertices and connectivity.

6. The method according to claim 2, wherein: The vertices and connectivity for generating predictions include: performing surface segmentation based on base mesh vertices and connectivity to generate child vertices; and Generates connectivity connecting child vertices to base mesh vertices.

7. The method according to claim 2, wherein: The vector difference of the reconstructed vertex includes: decoding a transform coefficient bitstream to reconstruct a transform coefficient image; and Generates the transform coefficients for the vertices from the transform coefficient image.

8. The method according to claim 7, wherein: The vector difference of the reconstructed vertex includes: inverse quantizing transform coefficients; and The inverse quantized transform coefficients are inversely transformed to reconstruct the vector differences of the vertices.

9. A method for encoding a grid performed by a grid encoding device, the method comprising: Get the original vertices and connectivity and the original attribute map for the mesh; Encode the original vertices and connectivity to generate the base mesh bitstream; as well as generating a reconstructed base mesh including reconstructed base mesh vertices and connectivity from the base mesh bitstream, The step of generating a basic grid bitstream includes: downsampling the original vertices and connectivity to generate a base mesh, wherein the base mesh includes base mesh vertices and connectivity; Classifying a base mesh into a plurality of patches by using base mesh vertices and connectivity, and generating patch information for each patch, wherein the patch information includes an offset and a size of each patch; and The offset is subtracted from the position coordinate values ​​of the base mesh vertices to generate the differential base mesh vertices and connectivity.

10. The method according to claim 9, further comprising: performing surface segmentation based on the reconstructed base mesh vertices and connectivity to generate predicted vertices and connectivity, wherein the predicted vertices include the reconstructed base mesh vertices and sub-vertices; and The predicted vertices are subtracted from the original vertices based on the original vertices and connectivity and the predicted vertices and connectivity to generate a vector difference.

11. The method according to claim 10, further comprising: transforming, quantizing, and packing the vector differences to generate a transform coefficient image; as well as The transform coefficient image is encoded by using a video encoding method to generate a transform coefficient bitstream.

12. The method according to claim 11, further comprising: generating a reconstructed transform coefficient image from a transform coefficient bitstream; unpacking, inverse quantizing, and inverse transforming the reconstructed transform coefficient image to reconstruct the vector difference; as well as The predicted vertices and reconstructed vector differences are added to reconstruct the vertices and connectivity for the mesh.

13. The method according to claim 12, further comprising: generating a corrected attribute map by utilizing the original vertices and connectivity, the original attribute map, and the reconstructed vertices and connectivity; encoding the corrected property map by utilizing a video coding method to generate a property map bitstream; as well as The base grid bitstream, transform coefficient bitstream and attribute map bitstream are combined.

14. The method according to claim 9, wherein: Generating the base grid bitstream includes: encoding patch information to generate a patch bitstream; encoding the differential base mesh vertices and connectivity based on a static mesh encoding method to generate a sub-base mesh bitstream; and The patch bitstream and the sub-base mesh bitstream are combined to generate a base mesh bitstream.

15. The method according to claim 9, wherein: Generating the reconstructed base mesh includes: reconstructing differential base mesh vertices and connectivity from the sub-base mesh bitstream; adding the reconstructed differential base mesh vertices and offsets to generate reconstructed base mesh vertices and connectivity; and Stores the reconstructed base mesh vertices and connectivity.

16. The method according to claim 9, wherein: Generating the base grid bitstream includes: performing motion prediction based on the original vertices and connectivity and the reconstructed base mesh vertices and connectivity of a previous frame to generate motion vectors; and The motion vectors are encoded to generate a sub-basis grid bitstream.

17. The method according to claim 16, wherein: Generating the reconstructed base mesh includes: decoding the sub-basis grid bitstream to reconstruct motion vectors; Generating reconstructed base mesh vertices and connectivity by compensating for motion of reconstructed base mesh vertices of a previous frame using the reconstructed motion vectors; and Stores the reconstructed base mesh vertices and connectivity.

18. A computer-readable recording medium storing a bit stream generated by a trellis coding method, the trellis coding method comprising: Get the original vertices and connectivity and the original attribute map for the mesh; Encode the original vertices and connectivity to generate the base mesh bitstream; as well as generating a reconstructed base mesh including reconstructed base mesh vertices and connectivity from the base mesh bitstream, The step of generating a basic grid bitstream includes: downsampling the original vertices and connectivity to generate a base mesh, wherein the base mesh includes base mesh vertices and connectivity; Classifying the base mesh into a plurality of patches using base mesh vertices and connectivity, and generating patch information for each patch, wherein the patch information includes an offset and a size of each patch; and The offset is subtracted from the position coordinate values ​​of the base mesh vertices to generate the differential base mesh vertices and connectivity.

Images