Method and apparatus for patch unit mesh coding

Abstract

Disclosed are a patch unit mesh coding method and apparatus. In this embodiment, a mesh decoding apparatus decodes a bitstream to restore patch information and a patch unit basic mesh. The mesh decoding apparatus restores basic mesh vertices and connectivity using the patch information and the patch unit basic mesh. The mesh decoding apparatus generates predicted vertices and connectivity based on the restored basic mesh vertices and connectivity. After the mesh decoding apparatus decodes the bitstream to restore a transform coefficient image, it unpacks, inverse quantizes, and inverse transforms the transform coefficient image to restore vertex difference vectors. The mesh decoding apparatus adds the predicted vertices and the difference vectors to restore the vertices and connectivity of the mesh.

Description

Technical Field

[0001] The present disclosure relates to a patch unit mesh coding method and apparatus.

Background Art

[0002] The content described below merely provides background information related to the present invention and does not constitute prior art.

[0003] In the MPEG meeting in April 22, among the responses to the CfP (Call for Proposal) for V-CG (Computer Graphics), the proposal from Apple showed the highest objective / subjective performance. Therefore, MPEG 3DGC (3D Graphic Compression) decided to determine the test model for V-CG in the next meeting based on the proposal from Apple.

[0004] The basic concept of Apple's proposal is scalable mesh coding. Scalable mesh coding downsamples a dense mesh to make it coarse, and then compresses the coarse mesh (hereinafter referred to as the base mesh) using static mesh compression technologies such as Google Draco and MPEG-AFX. Based on the restored base mesh, the difference between the dense mesh and the base mesh is encoded. Furthermore, in MPEG-3DGC, considering that the V-CG standard shares the V3C (Volumetric Visual Video based Coding) standard, Apple's proposal also converts most of the mesh information into an image in the 2D UV domain using mapping and then compresses the mesh using a video codec.

[0005] In the existing V-PCC / V3C, after the encoder divides the 3D point cloud into multiple patches, each patch is mapped to the UV domain, and the mapped information is transmitted to the decoder. In the proposal of Apple Inc., the encoder generates patches and maps the generated patches to the UV domain. Since the relevant information includes the UV coordinate information for each vertex, the encoder does not transmit the patch information separately. However, in order to share the V3C standard, it is expected that patch information will be added in the future standardization process. Therefore, when encoding / decoding the mesh in patch units, it is necessary to consider a solution that uses patch information.

Summary of the Invention

Problems to be Solved by the Invention

[0006] The present disclosure aims to provide a mesh coding method and apparatus that divide a base mesh into a number of patches and perform compression in patch units using static mesh coding in order to improve the coding efficiency for 3D meshes.

[0007] Furthermore, the present disclosure aims to provide a mesh coding method and apparatus that convert the vertex coordinates and UV coordinates of the base mesh into the vertex coordinates and UV coordinates inside the patch using the bounding box (i.e., start and end coordinates) information in the 3D space and UV space included in the patch information.

Means for Solving the Problems

[0008] According to an embodiment of the present disclosure, in a method for decrypting a mesh performed by a mesh decryption device, the method includes separating a bitstream into a base mesh bitstream, a transform coefficient bitstream, and an attribute map bitstream, where the base mesh bitstream includes a patch bitstream and a lower-level base mesh bitstream; decrypting the base mesh bitstream to restore a base mesh including base mesh vertices and connectivity; and restoring the base mesh includes decrypting the patch bitstream to restore patch information, where the patch information includes an offset and a size of each patch unit, decrypting the lower-level base mesh bitstream to restore a patch unit base mesh, and restoring the base mesh vertices and connectivity using the patch information and the patch unit base mesh. A method is provided, which is characterized by including the above steps.

[0009] According to another embodiment of the present disclosure, in a method for encoding a mesh performed by a mesh encoding device, the method includes obtaining connectivity with original vertices and an original attribute map for the mesh; encoding the original vertices and connectivity to generate a base mesh bitstream; and generating a restored base mesh including restored base mesh vertices and connectivity from the base mesh bitstream. The step of generating the base mesh bitstream includes downsampling the original vertices and connectivity to generate a base mesh, where the base mesh includes base mesh vertices and connectivity; classifying the base mesh into a plurality of patches using the base mesh vertices and connectivity, and generating patch information for each patch, where the patch information includes an offset and a size of each patch; and generating differential base mesh vertices and connectivity by differentiating the offset from the position coordinate value of the base mesh vertices. A method is provided, which is characterized by including the above steps.

[0010] According to another embodiment of the present disclosure, a computer-readable recording medium for storing a bitstream generated by a mesh encoding method, wherein the mesh encoding method includes: obtaining an original vertex, connectivity, and an original attribute map for a mesh; encoding the original vertex and connectivity to generate a base mesh bitstream; and generating a restored base mesh including restored base mesh vertices and connectivity from the base mesh bitstream. The step of generating the base mesh bitstream includes: downsampling the original vertex and connectivity to generate a base mesh, the base mesh including base mesh vertices and connectivity; classifying the base mesh into a plurality of patches using the base mesh vertices and connectivity, and generating patch information for each patch, the patch information including an offset and a size of each patch; and generating differential base mesh vertices and connectivity by differentiating the offset from the position coordinate values of the base mesh vertices. A recording medium is provided, characterized in that it includes the above steps.

Advantages of the Invention

[0011] As described above, according to this embodiment, by providing a mesh encoding method and apparatus that divide a base mesh into a plurality of patches and perform compression in units of patches using static mesh coding, it is possible to improve the encoding efficiency for 3D meshes.

[0012] Also, according to this embodiment, by providing a mesh encoding method and apparatus that convert the vertex coordinates and UV coordinates of a base mesh into vertex coordinates and UV coordinates inside a patch using the bounding box (i.e., start and end coordinates) information in the 3D space and UV space included in the patch information, it is possible to improve the encoding efficiency for 3D meshes by reducing the dynamic range of the coordinate values.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. Note that when adding reference numerals to the components of each drawing, it should be noted that the same components have the same numerals as much as possible even if they are shown on other drawings. In addition, when describing the present embodiment, if it is determined that a detailed description of related known configurations or functions obscures the gist of the present embodiment, the detailed description thereof will be omitted.

[0015] FIG. 1 is an exemplary block diagram of a video encoding device embodying the technology of the present disclosure. Hereinafter, the video encoding device and the subordinate configurations of this device will be described with reference to the illustration of FIG. 1.

[0016] The video encoding device is configured to include a picture splitting unit 110, a prediction unit 120, a subtractor 130, a conversion unit 140, a quantization unit 145, a rearrangement unit 150, an entropy encoding unit 155, an inverse quantization unit 160, an inverse conversion unit 165, an adder 170, a loop filter unit 180, and a memory 190.

[0017] This embodiment discloses content related to a patch unit mesh coding method and apparatus. More specifically, in order to improve the coding efficiency related to a 3D mesh, a base mesh is divided into a plurality of patches, and a mesh coding method and apparatus are provided that perform compression in units of patches using static mesh coding. Further, this embodiment provides a mesh coding method and apparatus that convert the vertex coordinates and UV coordinates of the base mesh into vertex coordinates and UV coordinates inside the patch using the bounding box information (i.e., start and end coordinates) in the 3D space and UV space included in the patch information.

[0018] FIG. 1 is a block diagram showing a mesh encoding apparatus according to an embodiment of the present disclosure.

[0019] The mesh encoding apparatus (hereinafter used interchangeably with the "encoding apparatus") encodes the original vertices, connectivity, and original attribute map of the mesh to generate a bitstream. At this time, the vertices of the mesh include coordinate values in the 3D space and coordinate values in the 2D UV domain. Also, the connectivity of the mesh is information representing the faces of the mesh as a set of vertices in the 3D space. Also, the attribute map is information representing the attribute values of the mesh faces.

[0020] The encoding apparatus includes all or part of a basic mesh encoding unit 102, a mesh surface division unit 104, a vertex difference unit 106, a conversion unit 108, a quantization unit 110, an image packing unit 112, video encoding units 114, 130, an image unpacking unit 116, an inverse quantization unit 118, an inverse conversion unit 120, a vertex summation unit 122, an attribute information mapping unit 124, a padding unit 126, a color space conversion unit 128, and a bitstream synthesis unit 132.

[0021] FIG. 2 is a block diagram showing a basic mesh encoding unit according to an embodiment of the present disclosure.

[0022] The basic mesh encoding unit 102 encodes the input original vertices and connectivity to generate a basic mesh bitstream and a restored basic mesh. The restored basic mesh represents the vertices and connectivity of the restored basic mesh. The basic mesh bitstream is transmitted to the bitstream synthesis unit 132. Also, the restored basic mesh vertices and connectivity are transmitted to the mesh surface division unit 104. Also, the restored basic mesh vertices and connectivity are transmitted to the attribute information mapping unit 124 together with the sub-vertices generated by the vertex summation unit 122.

[0023] On the other hand, the basic mesh vertices and connectivity correspond to the geometric information of the basic mesh. However, the basic mesh encoding unit 102 targets only the geometric information of the basic mesh. Hereinafter, encoding or decoding the basic mesh indicates encoding or decoding the basic mesh vertices and connectivity.

[0024] The basic mesh encoding unit 102 includes all or part of the mesh downsampling unit 202, the static mesh encoding unit 204, the motion prediction unit 206, the motion vector encoding unit 208, the static mesh decoding unit 210, the motion vector decoding unit 212, the motion compensation unit 214, and the memory 216. Among the components of the basic mesh encoding unit 102, the mesh downsampling unit 202, the static mesh encoding unit 204, and the static mesh decoding unit 210 directly encode or decode the basic mesh. Also, the motion prediction unit 206, the motion vector encoding unit 208, the motion vector decoding unit 212, and the motion compensation unit 214 encode or decode the basic mesh using motion vectors.

[0025] The mesh downsampling unit 202 receives the original vertices and connectivity, downsamples the mesh, and generates basic mesh vertices and connectivity. The generated basic mesh vertices and connectivity are transmitted to the static mesh encoding unit 204.

[0026] The static mesh encoding unit 204 encodes the input basic mesh vertices and connectivity to generate a basic mesh bitstream. The static mesh encoding unit 204 encodes the basic mesh using a static mesh encoding method such as Google Draco or MPEG-AFX. The generated bitstream is output from the basic mesh encoding unit 102. Further, the bitstream is transmitted to the static mesh decoding unit 210.

[0027] The static mesh decoding unit 210 decodes the input basic mesh bitstream to restore the basic mesh vertices and connectivity. The restored basic mesh vertices and connectivity are output from the basic mesh encoding unit 102. Further, the restored basic mesh vertices and connectivity are transmitted to the memory 216 for subsequent frame encoding.

[0028] The motion prediction unit 206 performs motion prediction based on the input original vertices and connectivity, and the restored basic mesh vertices and connectivity of the previous frame stored in the memory 216 to generate motion vectors. The generated motion vectors are transmitted to the motion vector encoding unit 208.

[0029] The motion vector encoding unit 208 encodes the input motion vectors to generate a basic mesh bitstream. The generated bitstream is output from the basic mesh encoding unit 102. Also, the bitstream is transmitted to the motion vector decoding unit 212.

[0030] The motion vector decoding unit 212 decodes the input basic mesh bitstream to restore the motion vectors. The restored motion vectors are transmitted to the motion compensation unit 214.

[0031] The motion compensation unit 214 generates the base mesh of the current frame by compensating for the motion of the base mesh vertices of the previous frame transmitted from the memory 216 using the motion vectors transmitted from the motion vector decoding unit 212. The restored base mesh is output from the base mesh encoding unit 102. Further, the restored base mesh vertices and connectivity are transmitted to the memory 216 for the encoding of the next frame.

[0032] The memory 216 stores the input restored base mesh. The memory 216 transmits the restored base mesh to the motion prediction unit 206 for the encoding of subsequent frames. Further, the memory 216 transmits the restored base mesh to the motion compensation unit 214 for the decoding of subsequent frames.

[0033] FIG. 3 is an exemplary diagram showing the operation of the mesh surface division unit according to an embodiment of the present disclosure.

[0034] The mesh surface division unit 104 performs surface division by inputting the restored base mesh vertices and connectivity to generate sub-vertices. Also, connectivity is generated to connect the generated sub-vertices and the existing base mesh vertices. As an example, as shown in the example of FIG. 3, the surface division method generates sub-vertices in the middle of two restored base vertices connected to each other and divides one triangular face into four triangular faces. As another example, the surface division method generates vertices on one of the three edges and divides one triangular face into two triangular faces. The generated sub-vertices, together with the restored base mesh vertices, become prediction vertices. The prediction vertices and connectivity are transmitted to the vertex difference unit 106 and the vertex summation unit 122.

[0035] FIG. 4 is an exemplary diagram showing the operation of the vertex difference unit according to an embodiment of the present disclosure.

[0036] The vertex difference unit 106 receives the original vertices and connectivity, and the prediction vertices and connectivity, and generates a difference vector by taking the difference between the prediction vertices and the original vertices as shown in the example of FIG. 4. The generated difference vector and connectivity are transmitted to the conversion unit 108.

[0037] The conversion unit 108 performs a conversion on the input difference vector to generate conversion coefficients, and transmits the generated conversion coefficients to the quantization unit 110. At this time, wavelet transform is used as the conversion method. Further, a lifting transform, which is a type of wavelet transform, is used. At this time, the connectivity of the difference vector is used for the wavelet transform.

[0038] The quantization unit 110 quantizes the input conversion coefficients to generate quantized conversion coefficients, and transmits the generated quantized conversion coefficients to the image packing unit 112.

[0039] FIG. 5 is an exemplary diagram showing the operation of the image packing unit according to an embodiment of the present disclosure.

[0040] The image packing unit 112 packs the input conversion coefficients in a two-dimensional space to generate a conversion coefficient image. For example, the conversion coefficients are packed in raster-scan order in units of M×N blocks. In the example of FIG. 5, the conversion coefficients are packed in raster-scan order in units of 2×2 blocks. Alternatively, various scan methods such as vertical scan, horizontal scan, diagonal zig-zag scan, etc. are used. The block size and the scan method are transmitted to the mesh decoding device (hereinafter, used interchangeably with the "decoding device") while being included in the bit stream. On the other hand, the generated conversion coefficient image is transmitted to the video encoding unit 114.

[0041] The video encoding unit 114 encodes the input transform coefficient image to generate a transform coefficient bitstream. Further, a transform coefficient image restored from the transform coefficient bitstream is generated. At this time, in the video encoding unit 114, various conventional video coding technologies such as H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), H.266 / VVC (Versatile Video Coding), EVC, VP9, AV1, AVS2, etc. are used. The generated bitstream is transmitted to the bitstream synthesis unit 132. Also, the restored transform coefficient image is transmitted to the image unpacking unit 116.

[0042] The image unpacking unit 116 obtains the quantized transform coefficients of the vertices from the input transform coefficient image. At this time, the block size and scan method used in the image packing unit 112 are used. The quantized transform coefficients are transmitted to the inverse quantization unit 118.

[0043] The inverse quantization unit 118 inverse quantizes the input quantized transform coefficients to restore the transform coefficients. The restored transform coefficients are transmitted to the inverse transform unit 120.

[0044] The inverse transform unit 120 inverse transforms the input transform coefficients to restore the vertex difference vectors. The restored difference vectors are transmitted to the vertex summation unit 122.

[0045] The vertex summation unit 122 sums the input predicted vertices and the restored difference vectors to restore the vertices. The restored vertices and connectivity are transmitted to the attribute information mapping unit 124. As described above, since the predicted vertices include the restored basic mesh vertices and sub-vertices, the restored vertices also correspond to the restored basic mesh vertices and sub-vertices.

[0046] The attribute information mapping unit 124 corrects the attribute information by inputting the original vertices and connectivity, the restored vertices and connectivity, and the original attribute map. Subsequently, the attribute information mapping unit 124 maps the corrected attribute information to the two-dimensional UV space to generate a corrected attribute map. The corrected attribute map is transmitted to the padding unit 126.

[0047] The padding unit 126 performs padding on the input corrected attribute map. At this time, the push-pull padding method is used as the padding method. Alternatively, the nearest neighbor padding method is used. The padded attribute map is transmitted to the color space conversion unit 128.

[0048] Here, the push-pull padding method hierarchically performs down-sampling on the target frame (the attribute map in this disclosure), hierarchically performs up-sampling, and then combines the foreground region and the up-sampled background region of the same layer. The push-pull padding method can improve the video encoding efficiency by smoothing the edge regions caused by the foreground texture packed in patch units.

[0049] The color space conversion unit 128 performs color space conversion on the input padded attribute map. At this time, the information regarding the color space conversion remains included in the bitstream and is transmitted to the decoding device. The attribute map with the color space converted is transmitted to the video encoding unit 130.

[0050] The video encoding unit 130 encodes the input attribute map to generate an attribute map bitstream. The generated bitstream is transmitted to the bitstream synthesis unit 132.

[0051] The bitstream synthesizing unit 132 concatenates all the input bitstreams (i.e., the basic mesh bitstream, the transform coefficient bitstream, and the attribute map bitstream) to generate one bitstream. The synthesized bitstream is output from the encoding device.

[0052] FIG. 6 is an exemplary diagram showing a basic mesh encoding unit according to another embodiment of the present disclosure.

[0053] As described above, the basic mesh encoding unit 102 encodes the input original vertices and connectivity to generate a basic mesh bitstream and a restored basic mesh. The basic mesh encoding unit 102 includes all or part of a mesh downsampling unit 202, a patch generation unit 602, a patch offset difference unit 604, a static mesh encoding unit 204, a patch information encoding unit 606, a motion prediction unit 206, a motion vector encoding unit 208, a patch offset summing unit 608, a motion vector decoding unit 212, a motion compensation unit 214, a memory 216, and a bitstream synthesizing unit 610.

[0054] Among the components of the basic mesh encoding unit 102, the mesh downsampling unit 202, the patch generation unit 602, the patch offset difference unit 604, the static mesh encoding unit 204, the static mesh decoding unit 210, the patch information encoding unit 606, and the patch offset summing unit 608 encode or decode the basic mesh using patches. Also, the motion prediction unit 206, the motion vector encoding unit 208, the motion vector decoding unit 212, and the motion compensation unit 214 encode or decode the basic mesh using motion vectors.

[0055] The mesh downsampling unit 202 receives the original vertices and connectivity, downsamples the mesh, and generates basic mesh vertices and connectivity. The generated basic mesh vertices and connectivity are transmitted to the patch generation unit 602.

[0056] FIG. 7 is an exemplary diagram showing the operation of the patch generation unit according to an embodiment of the present disclosure.

[0057] The patch generation unit 602 classifies the input basic mesh into a number of patches using the input basic mesh vertices and connectivity. As illustrated in FIG. 7, the patch generation unit 602 generates patch information including the start position (i.e., patch offset) and size of each patch using the generated patches. Also, as illustrated in FIG. 7, the patch information includes two-dimensional space (UV space) patch information and three-dimensional space patch information. The generated patch information is transmitted to the patch offset difference unit 604, the patch information encoding unit 606, and the patch offset summation unit 608.

[0058] The patch information encoding unit 606 encodes the input patch information to generate a patch bit stream. The generated patch bit stream is transmitted to the bit stream synthesis unit 610.

[0059] The patch offset difference unit 604 corrects the coordinate values of the basic mesh vertices by differentiating the patch offset from the position coordinate values of the basic mesh vertices using the input patch information. At this time, the patch offset is the coordinate value of the start position of the patch bounding box. The differentiated basic mesh vertices and connectivity are transmitted to the static mesh encoding unit 204.

[0060] The static mesh encoding unit 204 encodes the input differentiated basic mesh vertices and connectivity to generate a lower-level basic mesh bit stream. Here, the static mesh encoding unit 204 encodes the differentiated basic mesh using a static mesh encoding method such as Google Draco or MPEG-AFX. The generated bit stream is output from the basic mesh encoding unit 102. Further, the bit stream is transmitted to the static mesh decoding unit 210.

[0061] The static mesh decoding unit 210 decodes the input lower-level basic mesh bit stream to restore the differentiated basic mesh vertices and connectivity. The restored differentiated basic mesh vertices and connectivity are transmitted to the patch offset summation unit 608.

[0062] The patch offset summing unit 608 receives the restored differential basic mesh vertices and connectivity, and the patch information. The patch offset summing unit 608 sums the vertices of the input restored differential basic mesh with the offset to restore the basic mesh vertices. At this time, the offset is the coordinate value of the start position of the patch included in the patch information. The restored basic mesh vertices and connectivity are output from the basic mesh encoding unit 102. Further, the restored basic mesh vertices and connectivity are transmitted to the memory 216 for subsequent frame encoding.

[0063] The motion prediction unit 206 performs motion prediction based on the input original vertices and connectivity, and the restored basic mesh vertices and connectivity of the previous frame stored in the memory 216 to generate a motion vector. The generated motion vector is transmitted to the motion vector encoding unit 208.

[0064] The motion vector encoding unit 208 encodes the input motion vector to generate a lower-level basic mesh bitstream. The generated bitstream is transmitted to the bitstream combining unit 610. Also, the bitstream is transmitted to the motion vector decoding unit 212.

[0065] The motion vector decoding unit 212 decodes the input bitstream to restore the motion vector. The restored motion vector is transmitted to the motion compensation unit 214.

[0066] The motion compensation unit 214 can generate the basic mesh of the current frame by compensating the motion of the basic mesh vertices of the previous frame transmitted from the memory 216 using the motion vector transmitted from the motion vector decoding unit 214. The restored basic mesh is output from the basic mesh encoding unit 102. Further, the restored basic mesh vertices and connectivity are transmitted to the memory 216 for the next frame encoding.

[0067] Memory 216 stores the input restored basic mesh. Memory 216 transmits the restored basic mesh to the motion prediction unit 206 for subsequent frame encoding. Further, Memory 216 transmits the restored basic mesh to the motion compensation unit 214 for subsequent frame decoding.

[0068] The bitstream synthesis unit 610 concatenates all the transmitted bitstreams (i.e., the patch bitstream and the lower basic mesh bitstream) to generate a basic mesh bitstream. The generated basic mesh bitstream is output from the basic mesh encoding unit 102.

[0069] FIG. 8 is an exemplary diagram showing the operation of the image packing unit according to an embodiment of the present disclosure.

[0070] The in-encoder image packing unit 112 packs the input conversion coefficients in units of patches in the order of two-dimensional blocks. At this time, the block size is determined based on the preset horizontal and vertical sizes. Alternatively, the block size is determined based on the number of conversion coefficients included in the patch. Information related to the block is entropy encoded and then transmitted to the decoder.

[0071] In the example of FIG. 8, the image packing unit 112 determines the size of a square block according to the number of conversion coefficients included in each patch. The image packing unit 112 packs the conversion coefficients in scan order into the determined block. If the number of conversion coefficients packed in one block is smaller than the internal area of the block, the image packing unit 112 fills all the empty spaces with 0. Alternatively, the empty space is padded with the median value of the bit depth used by the video encoding unit 114. Alternatively, the empty space is padded with the nearest conversion coefficient.

[0072] In the example of FIG. 8, the image packing unit 112 packs the transform coefficients according to the raster scan order in block units. Various methods are used for the scan order, and the scan order information is entropy-coded and then transmitted to the decoding device. As described above, blocks of different sizes are determined according to the number of transform coefficients included in the patch as in the example of FIG. 8. In order to effectively pack blocks of different sizes into a two-dimensional image, the transform coefficients are packed in the form of a quad tree as in the example on the right of FIG. 8. Alternatively, various tree methods such as binary trees and trimmed trees are used. The tree information used is transmitted to the decoding device while remaining included in the bitstream.

[0073] Also, when a large number of blocks are generated as in FIG. 9, the image packing unit 112 packs a large number of blocks while the packing order is changed according to the block size. The image packing unit 112 first packs blocks of the same size in order, then increases the depth of the quad tree by one, and packs the next smaller-sized blocks.

[0074] Also, the quantization unit 112 adjusts the quantization degree of the transform coefficients using different quantization parameters in patch units. Also, the encoding device transmits the corresponding quantization parameter information to the video encoding unit 114, and performs video encoding more efficiently by coding the transform coefficient image based on the transmitted quantization parameter. Generally, the video encoding unit 114 transmits the quantization parameter in block units. Therefore, the encoding device performs more accurate bitrate adjustment using the patch unit quantization parameter information. Also, by performing block division at high speed using patch information in the video encoding unit 114, the complexity of the encoding device is reduced. As described above, the generated transform coefficient image is transmitted to the video encoding unit 114.

[0075] The mesh encoding device stores the bitstream of the encoded mesh data in a non-temporary recording medium or transmits it to the mesh decoding device using a communication network.

[0076] Figure 10 is an exemplary diagram showing a mesh decoding device according to an embodiment of the present disclosure.

[0077] The mesh decoding device (i.e., the "decoding device") can restore the mesh by decoding the input bitstream to generate the restored vertices, connectivity, and attribute maps. The decoding device includes all or part of a bitstream separation unit 1002, a basic mesh decoding unit 1004, a mesh surface division unit 1006, video decoding units 1008, 1018, an image unpacking unit 1010, an inverse quantization unit 1012, an inverse transformation unit 1014, a vertex summation unit 1016, and a color space conversion unit 1020.

[0078] The bitstream separation unit 1002 separates the input bitstream to generate a basic mesh bitstream, a transform coefficient bitstream, and an attribute map bitstream. The basic mesh bitstream is transmitted to the basic mesh decoding unit 1004. The transform coefficient bitstream and the attribute map bitstream are transmitted to the video decoding units 1006, 1018, respectively. At this time, each bitstream is input to a separate video decoding unit. Alternatively, they are continuously input to one video decoding unit.

[0079] Figure 11 is a block diagram showing a basic mesh decoding unit according to an embodiment of the present disclosure.

[0080] The basic mesh decoding unit 1004 decodes the input basic mesh bitstream to generate the restored basic mesh. The restored basic mesh indicates the restored basic mesh vertices and connectivity. The basic mesh decoding unit 1004 includes all or part of a static mesh decoding unit 1102, a motion vector decoding unit 1104, a motion compensation unit 1106, and a memory 1108.

[0081] On the one hand, the basic mesh vertices and connectivity correspond to the geometric information of the basic mesh. However, the basic mesh decoding unit 1004 targets only the geometric information of the basic mesh. Hereinafter, decoding the basic mesh means decoding the basic mesh vertices and connectivity.

[0082] Among the components of the basic mesh decoding unit 1004, the static mesh decoding unit 1102 immediately decodes the basic mesh. Also, the motion vector decoding unit 1104 and the motion compensation unit 1106 decode the basic mesh using the motion vector.

[0083] The static mesh decoding unit 1102 decodes the input bitstream to restore the basic mesh vertices and connectivity. The restored basic mesh vertices and connectivity are transmitted from the basic mesh decoding unit 1004 to the mesh surface division unit 1006. Also, the restored basic mesh vertices and connectivity are transmitted to the memory 1108 for subsequent frame encoding.

[0084] The motion vector decoding unit 1104 decodes the input bitstream to restore the motion vector. The restored motion vector is transmitted to the motion compensation unit 1106.

[0085] The motion compensation unit 1106 uses the motion vector transmitted from the motion vector decoding unit 1104 to compensate for the motion of the basic mesh vertices of the previous frame transmitted from the memory 1108, thereby generating the basic mesh of the current frame. The restored basic mesh is output from the basic mesh decoding unit 1004. Also, the restored basic mesh vertices and connectivity are transmitted to the memory 1108 for the next frame encoding.

[0086] The memory 1108 stores the input restored basic mesh. Also, the memory 1108 transmits the restored basic mesh to the motion compensation unit 1106 for subsequent frame decoding.

[0087] FIG. 12 is a block diagram showing a basic mesh decoding unit according to another embodiment of the present disclosure.

[0088] As described above, the basic mesh decoding unit 1004 decodes the input basic mesh bitstream to generate a restored basic mesh. The basic mesh decoding unit 1004 includes all or part of a bitstream separation unit 1202, a patch information decoding unit 1204, a static mesh decoding unit 1102, a patch synthesis unit 1206, a motion vector decoding unit 1104, a motion compensation unit 1106, and a memory 1108. Among the components of the basic mesh decoding unit 1004, the patch information decoding unit 1204, the static mesh decoding unit 1102, and the patch synthesis unit 1206 decode the basic mesh using patches. Also, the motion vector decoding unit 1104 and the motion compensation unit 1106 decode the basic mesh using motion vectors.

[0089] The bitstream separation unit 1202 separates the input basic mesh bitstream to generate a patch bitstream and a lower-level basic mesh bitstream. The patch bitstream is transmitted to the patch information decoding unit 1204. The lower-level basic mesh bitstream is transmitted to the static mesh decoding unit 1102 or the motion vector decoding unit 1102.

[0090] The patch information decoding unit 1204 decodes the input patch bitstream to restore patch information. As described above, the patch information includes two-dimensional space (UV space) patch information and three-dimensional space patch information. Also, the patch information includes the offset (i.e., start position) and size of each patch unit. The restored patch information is transmitted to the patch synthesis unit 1206.

[0091] The static mesh decoding unit 1102 decodes the input lower-level basic mesh bitstream to restore a patch-unit basic mesh. The restored patch-unit basic mesh is transmitted to the patch synthesis unit 1206.

[0092] The patch synthesis unit 1206 generates a restored basic mesh using the input patch information and the restored patch-unit basic mesh. First, the patch synthesis unit 1206 corrects the mesh vertex information by adding the offsets restored in patch units to the vertex information of the basic mesh using the patch information. Thereafter, the patch synthesis unit 1206 collects the corrected patches in the same space to restore the basic mesh. The restored basic mesh vertices and connectivity are output from the basic mesh decoding unit 1004. Further, the restored basic mesh vertices and connectivity are transmitted to the memory 1108 for subsequent frame encoding.

[0093] The motion vector decoding unit 1104 decodes the input lower-level basic mesh bitstream to restore the motion vector. The restored motion vector is transmitted to the motion compensation unit 1106.

[0094] The motion compensation unit 1106 generates the basic mesh of the current frame by compensating for the motion of the basic mesh vertices of the previous frame transmitted from the memory 1108 using the motion vector transmitted from the motion vector decoding unit 1104. The restored basic mesh is output from the basic mesh decoding unit 1004. Also, the restored basic mesh vertices and connectivity are transmitted to the memory 1108 for the next frame encoding.

[0095] The memory 1108 stores the input restored basic mesh. Also, the memory 1108 transmits the restored basic mesh to the motion compensation unit 1106 for subsequent frame decoding.

[0096] In the example of FIG. 10, the mesh surface division unit 1006 receives the restored basic mesh vertices and connectivity, performs surface division to generate sub-vertices. Also, the mesh surface division unit 1006 generates the connectivity connecting the generated sub-vertices and the existing basic mesh vertices. The generated sub-vertices, together with the restored basic mesh vertices, become prediction vertices. The prediction vertices and connectivity are transmitted to the vertex summation unit 1016.

[0097] The video decoding unit 1008 decodes the input conversion coefficient bitstream to restore the conversion coefficient image. The restored conversion coefficient image is transmitted to the image unpacking unit 1010.

[0098] The image unpacking unit 1010 generates the quantized conversion coefficients of the vertices from the input conversion coefficient image. At this time, the block size and scan method used in the image packing unit 112 are used. The quantized conversion coefficients are transmitted to the inverse quantization unit 1012.

[0099] The inverse quantization unit 1012 inverse quantizes the input quantized conversion coefficients to restore the conversion coefficients. The restored conversion coefficients are transmitted to the inverse conversion unit 1014.

[0100] The inverse conversion unit 1014 inverse converts the input conversion coefficients to restore the vertex difference vectors. The restored difference vectors are transmitted to the vertex summation unit 1016.

[0101] The vertex summation unit 1016 sums the input predicted vertices and the restored difference vectors to restore the vertices. The restored vertices and connectivity are output from the decoding device. As described above, since the predicted vertices include the restored basic mesh vertices and sub-vertices, the restored vertices also correspond to the restored basic mesh vertices and sub-vertices.

[0102] The video decoding unit 1018 decodes the input attribute map bitstream to restore the attribute map. The restored attribute map is transmitted to the color space conversion unit 1020.

[0103] The color space conversion unit 1020 converts the input restored attribute map to the color space of the same attribute map as the original mesh. The attribute map with the converted color space is output from the decoding device.

[0104] Hereinafter, a method for encoding and decoding a mesh will be described with reference to the illustrations of FIGS. 13 to 16.

[0105] FIG. 13 and FIG. 14 are flowcharts showing a method for a mesh encoding device to encode a mesh according to an embodiment of the present disclosure.

[0106] The encoding device obtains the original vertices, connectivity, and original attribute map for the mesh (S1300).

[0107] The encoding device encodes the original vertices and connectivity to generate a basic mesh bitstream and a restored basic mesh. Here, the restored basic mesh includes restored basic mesh vertices and connectivity.

[0108] The encoding device generates predicted vertices and connectivity based on the restored basic mesh vertices and connectivity (S1304). Here, the predicted vertices include restored basic mesh vertices and sub-vertices.

[0109] The encoding device performs surface splitting based on the restored basic mesh vertices and connectivity to generate sub-vertices. Also, the encoding device generates connectivity connecting the generated sub-vertices and the existing basic mesh vertices. The generated sub-vertices, together with the restored basic mesh vertices, become predicted vertices.

[0110] The encoding device generates a difference vector by differencing the predicted vertices from the original vertices based on the original vertices and connectivity and the predicted vertices and connectivity (S1306).

[0111] The encoding device converts, quantizes, and packs the difference vector to generate a transform coefficient image (S1308).

[0112] The encoding device encodes the transform coefficient image using a video encoding method to generate a transform coefficient bitstream and a restored transform coefficient image (S1310).

[0113] The encoding device unpacks, inverse quantizes, and inverse transforms the restored transform coefficient image to restore the difference vector (S1312).

[0114] The symbolization device sums up the predicted vertices and the restored difference vectors to restore the vertices and connectivity for the mesh (S1314).

[0115] The symbolization device generates an attribute map corrected using the original vertices and connectivity, the original attribute map, and the restored vertices and connectivity (S1316).

[0116] Furthermore, the symbolization device applies padding and color space conversion to the corrected attribute map.

[0117] The symbolization device generates an attribute map bitstream by symbolizing the corrected attribute map using a video symbolization method (S1318).

[0118] The symbolization device combines the basic mesh bitstream, the transform coefficient bitstream, and the attribute map bitstream (S1320).

[0119] On the other hand, step S1302 of generating the basic mesh bitstream and the restored basic mesh includes the following steps.

[0120] The symbolization device downsamples the original vertices and connectivity to generate a basic mesh. Here, the basic mesh includes basic mesh vertices and connectivity (S1400).

[0121] The symbolization device classifies the basic mesh into a number of patches using the basic mesh vertices and connectivity, and generates patch information for each patch. Here, the patch information includes the offset and size of each patch (S1402).

[0122] The symbolization device symbolizes the patch information to generate a patch bitstream (S1404).

[0123] The symbolization device generates differential basic mesh vertices and connectivity by differencing the offset from the position coordinate values of the basic mesh vertices (S1406).

[0124] The symbolization device encodes the differential basic mesh vertices and connectivity based on the static mesh symbolization method to generate a lower-level basic mesh bitstream (S1408).

[0125] The symbolization device restores the differential basic mesh vertices and connectivity from the lower-level basic mesh bitstream (S1410).

[0126] The symbolization device adds the restored differential basic mesh vertices and the offset to restore the basic mesh vertices and connectivity (S1412).

[0127] The symbolization device performs motion prediction based on the original vertices and connectivity, and the restored basic mesh vertices and connectivity of the previous frame to generate a motion vector (S1414).

[0128] The symbolization device encodes the motion vector to generate a lower-level basic mesh bitstream (S1416).

[0129] The symbolization device decodes the lower-level basic mesh bitstream to restore the motion vector (S1418).

[0130] The symbolization device uses the restored motion vector to compensate with the motion of the restored basic mesh vertices of the previous frame to restore the basic mesh vertices and connectivity (S1420).

[0131] The symbolization device stores the restored basic mesh vertices and connectivity (S1422)

[0132] The symbolization device combines the patch bitstream and the lower-level basic mesh bitstream to generate a basic mesh bitstream (S1424).

[0133] FIG. 15 and FIG. 16 are flowcharts showing a method by which a mesh decoding device decodes a mesh according to an embodiment of the present disclosure.

[0134] The decoding device separates the bitstream into a basic mesh bitstream, a transform coefficient bitstream, and an attribute map bitstream (S1500).

[0135] The decoding device decodes the basic mesh bitstream to restore the basic mesh (S1502). Here, the restored basic mesh includes restored basic mesh vertices and connectivity.

[0136] The decoding device generates predicted vertices and connectivity based on the restored basic mesh vertices and connectivity (S1504). Here, the predicted vertices include restored basic mesh vertices and sub-vertices.

[0137] The decoding device performs surface division based on the restored basic mesh vertices and connectivity to generate sub-vertices. Further, the decoding device generates connectivity that connects the generated sub-vertices and the existing basic mesh vertices. The generated sub-vertices, together with the restored basic mesh vertices, become predicted vertices.

[0138] The decoding device decodes the transform coefficient bitstream to restore the transform coefficient image (S1506).

[0139] The decoding device unpacks, inverse quantizes, and inverse-transforms the transform coefficient image to restore the vertex difference vector (S1508).

[0140] The decoding device adds the predicted vertices and the difference vector to restore the vertices and connectivity (S1510).

[0141] The decoding device decodes the attribute map bitstream based on the video decoding method to restore the attribute map (S1512).

[0142] Also, the decoding device applies a color space conversion to the restored attribute map.

[0143] On the other hand, step S1502 of restoring the basic mesh includes the following steps.

[0144] The decoding device separates the basic mesh bit stream into a patch bit stream and a lower-level basic mesh bit stream (S1600).

[0145] The decoding device decodes the patch bit stream to restore patch information (S1602). Here, the patch information includes the offset and size of each patch unit.

[0146] The decoding device decodes the lower-level basic mesh bit stream to restore the basic mesh per patch unit (S1604).

[0147] The decoding device restores the basic mesh vertices and connectivity using the patch information and the basic mesh per patch unit (S1606).

[0148] The decoding device adds the offset to the basic mesh per patch unit to correct the basic mesh per patch unit. Thereafter, the decoding device gathers the corrected basic mesh per patch unit in the same space to generate the basic mesh vertices and connectivity.

[0149] The decoding device decodes the lower-level basic mesh bit stream to restore the motion vector (S1608).

[0150] The decoding device compensates for the motion of the basic mesh vertices in the previous frame using the motion vector, thereby restoring the basic mesh vertices and connectivity (S1610).

[0151] The decoding device stores the restored basic mesh vertices and connectivity (S1612).

[0152] In the flowcharts / timing diagrams of this specification, it is described that each process is executed sequentially, but this is only an exemplary explanation of the technical idea of an embodiment of the present disclosure. In other words, those having ordinary knowledge in the technical field to which an embodiment of the present disclosure belongs can modify and apply it in various ways by changing the order described in the flowchart / timing diagram or executing one or more of the processes in parallel without departing from the essential characteristics of an embodiment of the present disclosure. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0153] It should be understood that the exemplary embodiments in the above description can be implemented in many different ways. The functions or methods described in one or more of the examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described in this specification are labeled as “… unit” to particularly emphasize their implementation independence.

[0154] On the other hand, the various functions or methods described in this embodiment may also be implemented by instruction words stored in a non - transitory recording medium that is read and executed by one or more processors. The non - transitory recording medium includes, for example, any type of recording device in which data is stored in a form readable by a computer system. For example, the non - transitory recording medium includes storage media such as EPROM (erasable programmable read only memory), flash drive, optical drive, magnetic hard drive, solid - state drive (SSD).

[0155] The above description merely exemplarily explains the technical idea of this embodiment. Those with ordinary knowledge in the technical field to which this embodiment belongs can make various modifications and variations without departing from the essential characteristics of this embodiment. Therefore, this embodiment is not for limiting the technical idea of this embodiment but for explanation, and the scope of the technical idea of this embodiment is not limited by such an embodiment. The protection scope of this embodiment should be interpreted according to the scope of the claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this embodiment.

[0156] CROSS-REFERENCE TO RELATED APPLICATION This patent application claims priority to Patent Application No. 10-2022-0059436 filed in Korea on May 16, 2022, and Patent Application No. 10-2023-0057829 filed in Korea on May 3, 2023, and all of their contents are incorporated herein by reference.

Description of Reference Numerals

[0157] 1002 Bitstream Separation Unit 1004 Basic Mesh Decryption Unit 1006 Mesh Surface Division Unit 1008, 1018 Video Decryption Unit 1010 Image Unpacking Unit 1012 Inverse Quantization Unit 1014 Inverse Transformation Unit 1016 Vertex Summation Unit 1020 Color Space Conversion Unit

Claims

1. A method for decrypting a mesh, performed by a mesh decryption device, comprising: separating a bitstream into a base mesh bitstream, a transform coefficient bitstream, and an attribute map bitstream, wherein the base mesh bitstream includes a patch bitstream and a lower-level base mesh bitstream; decrypting the base mesh bitstream to restore a base mesh including base mesh vertices and connectivity; wherein the step of restoring the base mesh includes: decrypting the patch bitstream to restore patch information, wherein the patch information includes an offset and a size for each patch unit; decrypting the lower-level base mesh bitstream to restore a patch-unit base mesh; restoring the base mesh vertices and connectivity using the patch information and the patch-unit base mesh; A method characterized by including the above steps.

2. Further comprising: performing surface division based on the restored base mesh vertices and connectivity to generate predicted vertices and connectivity, wherein the predicted vertices include the restored base mesh vertices and sub-vertices; decrypting the transform coefficient bitstream based on a video decryption method to restore vertex difference vectors for the mesh; adding the predicted vertices and the difference vectors to restore the vertices and connectivity; decrypting the attribute map bitstream based on the video decryption method to restore an attribute map; A method according to claim 1, characterized by including the above steps.

3. The step of restoring the base mesh further includes: separating the base mesh bitstream into the patch bitstream and the lower-level base mesh bitstream; saving the restored base mesh vertices and connectivity; A method according to claim 1, characterized by including the above steps.

4. The step of restoring the base mesh vertices and connectivity includes: adding the offset to the patch-unit base mesh to correct the patch-unit base mesh; collecting the corrected patch-unit base meshes in the same space to generate the base mesh vertices and connectivity; The method according to claim 1, characterized by comprising

5. The step of restoring the basic mesh further comprises decoding the sub-basic mesh bitstream to restore the motion vector; compensating the motion of the basic mesh vertices in the previous frame using the motion vector to restore the basic mesh vertices and connectivity; storing the restored basic mesh vertices and connectivity; The method according to claim 1, characterized by comprising

6. The step of generating the predicted vertices and connectivity comprises performing the surface segmentation based on the basic mesh vertices and connectivity to generate sub-vertices; generating the connectivity connecting the sub-vertices and the basic mesh vertices; The method according to claim 2, characterized by comprising

7. The step of restoring the difference vector of the vertices comprises decoding the transform coefficient bitstream to restore the transform coefficient image; generating the transform coefficients of the vertices from the transform coefficient image; The method according to claim 2, characterized by comprising

8. The step of restoring the difference vector of the vertices comprises inverse quantizing the transform coefficients; inverse transforming the inverse quantized transform coefficients to restore the difference vector of the vertices; The method according to claim 7, characterized by comprising

9. In a method for encoding a mesh performed by a mesh encoding device, obtaining the original vertices, connectivity, and original attribute map for the mesh; encoding the original vertices and connectivity to generate a base mesh bitstream; generating a restored base mesh including restored base mesh vertices and connectivity from the base mesh bitstream; comprising The step of generating the base mesh bitstream comprises downsampling the original vertices and connectivity to generate a base mesh, the base mesh including base mesh vertices and connectivity; classifying the base mesh into a plurality of patches using the base mesh vertices and connectivity, and generating patch information for each patch, the patch information including the offset and size of each patch; A step of generating differential basic mesh vertices and connectivity by differentiating the offset from the position coordinate values of the basic mesh vertices; A method, characterized by including this.

10. Furthermore, a step of performing surface division based on the restored basic mesh vertices and connectivity to generate predicted vertices and connectivity, wherein the predicted vertices include the restored basic mesh vertices and sub-vertices; A step of generating a differential vector by differentiating the predicted vertices from the original vertices based on the connectivity between the original vertices and the connectivity between the predicted vertices; The method according to claim 9, characterized by including this.

11. Furthermore, a step of converting, quantizing, and packing the differential vector to generate a transform coefficient image; A step of generating a transform coefficient bitstream by encoding the transform coefficient image using a video encoding method; The method according to claim 10, characterized by including this.

12. Furthermore, a step of generating a restored transform coefficient image from the transform coefficient bitstream; A step of unpacking, inverse quantizing, and inverse transforming the restored transform coefficient image to restore the differential vector; A step of adding the predicted vertices and the restored differential vector to restore the vertices and connectivity for the mesh; The method according to claim 11, characterized by including this.

13. A step of generating a corrected attribute map using the connectivity between the original vertices, the original attribute map, and the connectivity between the restored vertices; A step of generating an attribute map bitstream by encoding the corrected attribute map using the video encoding method; A step of combining the basic mesh bitstream, the transform coefficient bitstream, and the attribute map bitstream; The method according to claim 12, characterized by including this.

14. The step of generating the basic mesh bitstream further includes: A step of encoding the patch information to generate a patch bitstream; A step of encoding the differential basic mesh vertices and connectivity based on a static mesh encoding method to generate a lower-level basic mesh bitstream; A step of combining the patch bitstream and the lower-level basic mesh bitstream to generate the basic mesh bitstream; The method according to claim 9, characterized by comprising

15. The step of generating the restored basic mesh comprises: Restoring the differential basic mesh vertices and connectivity from the lower basic mesh bitstream; Adding the restored differential basic mesh vertices and the offset to generate the restored basic mesh vertices and connectivity; Saving the restored basic mesh vertices and connectivity; The method according to claim 9, characterized by comprising

16. The step of generating the basic mesh bitstream further comprises: Performing motion prediction based on the original vertices and connectivity and the restored basic mesh vertices and connectivity of the previous frame to generate motion vectors; Encoding the motion vectors to generate a lower basic mesh bitstream; The method according to claim 9, characterized by comprising

17. The step of generating the restored basic mesh further comprises: Decoding the lower basic mesh bitstream to restore the motion vectors; Using the restored motion vectors to compensate for the motion of the restored basic mesh vertices of the previous frame to generate the restored basic mesh vertices and connectivity; Saving the restored basic mesh vertices and connectivity; The method according to claim 16, characterized by comprising

18. A computer-readable recording medium for storing a bitstream generated by a mesh encoding method, wherein the mesh encoding method comprises: Obtaining original vertices, connectivity, and an original attribute map for a mesh; Encoding the original vertices and connectivity to generate a base mesh bitstream; Generating a restored basic mesh including restored basic mesh vertices and connectivity from the base mesh bitstream; including The step of generating the base mesh bitstream is: Downsampling the original vertices and connectivity to generate a base mesh, wherein the base mesh includes base mesh vertices and connectivity A step of classifying the basic mesh into a number of patches using the basic mesh vertices and connectivity, and generating patch information for each patch, wherein the patch information includes the offset and size of each patch. A step of generating differential basic mesh vertices and connectivity by differentiating the offset from the position coordinate values of the basic mesh vertices. A recording medium, characterized by including the above.

Images