Method and apparatus for grid and point cloud coding

Abstract

A method and apparatus for grid and point cloud coding are disclosed. The present embodiment provides a method and apparatus for grid and point cloud encoding, which predicts with reference to a reconstructed point cloud when encoding / decoding a grid, or predicts with reference to a reconstructed grid when encoding / decoding a point cloud to improve the encoding efficiency of a three-dimensional grid and point cloud.

Description

Technical Field

[0001] The present disclosure relates to methods and apparatus for mesh and point cloud encoding. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] Conventionally, mesh compression technology and point cloud compression technology are used separately, and the mesh and point cloud are encoded / decoded by using separate compression technologies.

[0004] A mesh is a collection of multiple faces in a three-dimensional space. A mesh includes geometric information (such as the position coordinates of vertices in the three-dimensional space and the connectivity information of polygons between vertices), and a mesh can use the position coordinates of the vertices and the connectivity information to represent a three-dimensional body of a specific object in the three-dimensional space. A mesh can also include attribute information of the faces of the polygons surrounding the three-dimensional body. A mesh can also include attribute information of each vertex. Alternatively, a mesh can include attribute information in the form of a texture map, which is generated by projecting the faces of the polygons into a two-dimensional space. If a mesh includes a texture map, the mesh can further include coordinates on the texture map for each vertex of the polygon. A mesh can be composed of vertex coordinates, connectivity information, and attribute information, or a mesh can be composed of vertex coordinates, connectivity information, texture map coordinates, and a texture map. Here, the attribute information can represent a color representation such as RGB, YCbCr, etc., or can represent information such as the material, reflectivity, etc. of the surface.

[0005] A point cloud is a collection of points in a three-dimensional space and can represent the volume of the three-dimensional space. A point cloud may include coordinate values ​​and attribute values ​​of the positions of relevant points in the three-dimensional space. Here, the attribute value may represent a color representation such as RGB, YCbCr, etc., or may represent information such as the material, reflectivity, etc. of a surface.

[0006] Both meshes and point clouds are ways to represent 3D space, where dense point clouds can contain richer information than meshes, and sparse meshes including connectivity information can contain richer information in different aspects than point clouds. Therefore, a method that exploits these complementary properties of meshes and point clouds is needed when meshes and point clouds are encoded / decoded. Summary of the invention

[0007]

Technical issues

[0008] The present disclosure attempts to provide a mesh and point cloud encoding method and device that references a reconstructed point cloud for prediction when encoding / decoding the mesh to improve the encoding efficiency of the three-dimensional mesh and point cloud. Or the mesh and point cloud encoding method and device references a reconstructed mesh for prediction when encoding / decoding the point cloud.

[0009]

Technical solution

[0010] At least one aspect of the present disclosure provides a method for decoding a point cloud and a mesh performed by a point cloud mesh decoding device. The method includes dividing a bit stream into a first bit stream, a second bit stream, a third bit stream, a fourth bit stream, and a fifth bit stream. The method also includes reconstructing slice information by decoding the first bit stream. The method also includes reconstructing a geometric image by decoding the second bit stream. The method also includes reconstructing a placeholder image by decoding the third bit stream. The method also includes reconstructing an attribute image or a mesh texture map by decoding the fourth bit stream. The method also includes reconstructing the geometric information of the mesh by decoding the fifth bit stream. Here, the geometric information of the mesh includes the vertices of the mesh, the connectivity of the mesh, and the vertices of the mesh texture map.

[0011] Another aspect of the present disclosure provides a method performed by a point cloud mesh encoding device for encoding point clouds and meshes. The method includes obtaining a point cloud and a mesh. The method also includes generating a point cloud image from the point cloud. Here, the point cloud image includes patch information, a geometric image, a placeholder image, and an attribute image. The method also includes encoding the patch information, encoding the geometric image, encoding the placeholder image, encoding the attribute image or mesh texture map, and encoding the geometric information of the mesh. Here, the geometric information of the mesh includes the vertices of the mesh, the connectivity of the mesh, and the vertices of the mesh texture map.

[0012] Another aspect of the present invention provides a computer-readable recording medium storing a bit stream generated by a point cloud and mesh coding method. The point cloud and mesh coding method includes obtaining a point cloud and a mesh. The point cloud and mesh coding method also includes generating a point cloud image from the point cloud. Here, the point cloud image includes patch information, a geometric image, a placeholder image, and an attribute image. The point cloud and mesh coding method also includes encoding the patch information, encoding the geometric image, encoding the placeholder image, encoding the attribute image or mesh texture map, and encoding the geometric information of the mesh. Here, the geometric information of the mesh includes the vertices of the mesh, the connectivity of the mesh, and the vertices of the mesh texture map.

[0013]

Beneficial Effects

[0014] As described above, the present disclosure provides a mesh and point cloud encoding method and device for making predictions with reference to a reconstructed point cloud when encoding / decoding a mesh. Alternatively, the mesh and point cloud encoding method and device make predictions with reference to a reconstructed mesh when encoding / decoding a point cloud. Thus, the mesh and point cloud encoding method and device improve the encoding efficiency of three-dimensional meshes and point clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of a point cloud mesh encoding device according to at least one embodiment of the present disclosure.

[0016] Figure 2 is a diagram showing a form of a reference texture video and a reference structure of a mesh texture map according to at least one embodiment of the present disclosure.

[0017] Figure 3 is a diagram showing a reference structure of a form of a reference texture video and an attribute image of a point cloud according to at least one embodiment of the present disclosure.

[0018] Figure 4 is a block diagram of a point cloud mesh decoding device according to at least one embodiment of the present disclosure.

[0019] Figure 5 is a block diagram of a point cloud mesh encoding device according to another embodiment of the present disclosure.

[0020] Figure 6 is a detailed block diagram of a point cloud encoder according to at least one embodiment of the present disclosure.

[0021] Figure 7 is a detailed block diagram of a trellis encoder according to at least one embodiment of the present disclosure.

[0022] Figure 8 is a block diagram of a point cloud mesh decoding device according to another embodiment of the present disclosure.

[0023] Fig. 9 is a detailed block diagram of a point cloud decoder according to at least one embodiment of the present disclosure.

[0024] Fig.10 is a detailed block diagram of a trellis decoder according to at least one embodiment of the present disclosure.

[0025] Fig.11 is a diagram illustrating a method of predicting geometric information and attribute information of a point cloud by using a reconstructed mesh.

[0026] Fig.12 is a diagram showing a method of predicting geometric information and attribute information of a point cloud by using a reconstructed mesh with a texture map.

[0027] Fig.13 is a diagram showing a method of predicting a mesh texture map by using a reconstructed point cloud of a previous frame.

[0028] Fig.14 is a flow chart of a method of encoding point clouds and meshes according to at least one embodiment of the present disclosure.

[0029] Fig.15 is a flow chart of a method of decoding a point cloud and a mesh according to at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Hereinafter, some embodiments of the present disclosure are described in detail with reference to the accompanying drawings. In the following description, the same reference numerals represent the same elements, although the elements are shown in different drawings. In addition, in the following description of some embodiments, for the purpose of clarity and brevity, detailed descriptions of related known components and functions may be omitted when it is considered to obscure the subject matter of the present disclosure.

[0031] The present disclosure relates to a method and apparatus for encoding a mesh and a point cloud. More specifically, the present disclosure provides a mesh and a point cloud encoding method and apparatus for predicting with reference to a reconstructed point cloud when a mesh is encoded / decoded, or predicting with reference to a reconstructed mesh when a point cloud is encoded / decoded.

[0032] Figure 1 is a block diagram of a point cloud mesh encoding device according to at least one embodiment of the present disclosure.

[0033] The point cloud mesh encoding device (hereinafter used interchangeably with "encoding device") encodes an input three-dimensional point cloud or mesh received as input to generate a bitstream. Alternatively, the encoding device encodes an input three-dimensional point cloud and mesh received as input to generate a bitstream. The encoding device may include all or part of the point cloud image generator 110, the slice information encoder 120, the geometry video encoder 130, the placeholder video encoder 140, the attribute video encoder 150, the mesh geometry information encoder 160 and the bitstream synthesizer 170.

[0034] The input of the encoding device may include both the three-dimensional point cloud and the mesh. Further, the output of the encoding device may include both a bit stream for the three-dimensional point cloud and a bit stream for the three-dimensional mesh.

[0035] The point cloud image generator 110 receives a point cloud as input and generates patch information, a geometric image, a placeholder image, and an attribute image. Here, the patch information, the geometric image, and the placeholder image can all be generated from the geometric information of the point cloud.

[0036] The point cloud image generator 110 uses the patch information when mapping or projecting a plurality of patches classified in the three-dimensional space onto a two-dimensional projection plane. Thus, the patch information may include the coordinate values ​​of each patch in the three-dimensional space and information such as width, length, depth, etc. in the three-dimensional space. It may also include information such as the coordinate values ​​in the two-dimensional projection plane and information such as the horizontal length and the vertical length.

[0037] The geometric image is an image that two-dimensionally maps the position of a point in the three-dimensional space to the distance between the projection plane by using the generated slice information. In other words, the geometric image may be a graph of the depth between a point and the two-dimensional plane when the three-dimensional space is projected onto the two-dimensional plane. In this case, the two-dimensional plane may be one of the xy plane, the yz plane, and the xz plane.

[0038] On the other hand, if the 3D slice has a volume in 3D space, multiple 3D points can be projected to a single 2D location. In this case, multiple depths may exist, and multiple geometric images can be generated. Typically, a total of two geometric images can be generated, one for the closest 3D point and one for the farthest 3D point.

[0039] The placeholder image is an image indicating the position where the point cloud image generator 110 projects the plurality of slices classified in the three-dimensional space as an image on a two-dimensional projection plane. The placeholder image may be a binary image that indicates whether a point is projected to the position of each pixel by 0 or 1. The generated slice information, the generated geometric image, and the generated placeholder image may be transmitted to the slice information encoder 120, the geometric video encoder 130, and the placeholder video encoder 140, respectively.

[0040] At the same time, an attribute image may be generated from the attribute information of the point cloud. The attribute image is a two-dimensional image generated by projecting the attribute value of a point in a three-dimensional space onto a projection plane. The generated attribute image may be transmitted to the attribute video encoder 150.

[0041] The slice information encoder 120 may encode the slice information to generate a bitstream. At this time, the conventional entropy-encoded bitstream may be transmitted to the bitstream synthesizer 170.

[0042] The geometry video encoder 130 encodes one or more geometry images to generate a bitstream. At this time, the geometry video encoder 130 may use a video encoding technology such as H.264 / AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding), VVC (Versatile Video Coding), VP8, VP9, ​​AV1, etc. The generated bitstream may be transmitted to the bitstream synthesizer 170.

[0043] The placeholder video encoder 140 encodes the placeholder image to generate a bitstream. At this time, the placeholder video encoder 140 may use a video encoding technology such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP8, VP9, ​​AV1, etc. The generated bitstream may be transmitted to the bitstream synthesizer 170.

[0044] The attribute video encoder 150 receives and then encodes the attribute image or the mesh texture map to generate a bitstream. At this time, the attribute video encoder 150 may use a video encoding technology such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP8, VP9, ​​AV1, etc. The generated bitstream may be transmitted to the bitstream synthesizer 170. The encoding device may determine information indicating whether the image to be encoded is an attribute image of a point cloud or a texture map of a mesh frame by frame, and may transmit the information to the decoding device.

[0045] At the same time, attribute images and mesh texture maps can be used as Figure 2 The order shown in the top example of is input to the attribute video encoder 150. In addition, the attribute video encoder 150 can be input to the attribute video encoder 150 by using a reference structure (e.g., Figure 2 The bottom example of FIG. 10 shows an example of a video encoder 150 for encoding an attribute image of a point cloud. For example, the attribute video encoder 150 for encoding an attribute image of a point cloud can perform inter-frame prediction by referring to a grid texture map, thereby effectively encoding the attribute image. Figure 2 In the example, PCC indicates point cloud coding.

[0046] As another example, attribute images and mesh texture maps can be Figure 3 The order in the top example of is input to the attribute video encoder 150. In addition, the attribute video encoder 150 can be configured by using a reference structure (e.g., Figure 3 For example, the attribute video encoder 150 for encoding the grid texture map can perform inter-frame prediction by referring to the attribute image of the point cloud, thereby effectively encoding the grid texture map.

[0047] The mesh geometry information encoder 160 encodes mesh vertices, connectivity, texture map vertices, etc. to generate a bitstream. In this case, the mesh vertices, connectivity, and texture map vertices may be generated by a mesh geometry information compression technique such as TFAN (Triangle-FAN) or DRACO as a mesh compression method. The generated bitstream may be transmitted to the bitstream synthesizer 170.

[0048] In the following, a mesh vertex may implicitly represent the vertex coordinates of the mesh. Further, a texture map vertex may implicitly represent the texture map vertex coordinates.

[0049] The bitstream synthesizer 170 concatenates all received bitstreams to generate a single bitstream. The order of each bitstream may be determined by the arrangement between the encoding device and the decoding device. Alternatively, the bitstreams may be concatenated in any order. Alternatively, each bitstream may include a symbol representing the type of the bitstream at the beginning of the bitstream. In this case, the bitstream synthesizer 170 may determine the type of each bitstream and concatenate the bitstreams in a preset order. Alternatively, the encoding device may concatenate the bitstreams in any order, and the decoding device may decode their symbols and may separate the bitstreams according to these symbols.

[0050] Figure 4 is a block diagram of a point cloud mesh decoding device according to at least one embodiment of the present disclosure.

[0051] The point cloud mesh decoding device (hereinafter used interchangeably with the "decoding device") reconstructs the three-dimensional point cloud and mesh from the bitstream. Alternatively, the decoding device may reconstruct the three-dimensional point cloud or mesh from the bitstream. The decoding device may include all or part of the bitstream separator 410, the slice information decoder 420, the geometry video decoder 430, the placeholder video decoder 440, the attribute video decoder 450, the mesh geometry information decoder 460 and the point cloud image synthesizer 470.

[0052] The input to the decoding device may include both a bitstream for a three-dimensional point cloud and a bitstream for a three-dimensional mesh. The output of the decoding device may include both a three-dimensional point cloud and a mesh.

[0053] The bitstream separator 410 receives a bitstream and then separates the bitstream into a plurality of bitstreams. The separated bitstreams may be delivered to a slice information decoder 420, a geometry video decoder 430, a placeholder video decoder 440, an attribute video decoder 450, and a mesh geometry information decoder 460.

[0054] The slice information decoder 420 decodes the input bit stream to reconstruct the slice information. The reconstructed slice information may be delivered to the point cloud image synthesizer 470.

[0055] The geometry video decoder 430 decodes the received bit stream to reconstruct the geometry image. At this time, the geometry video decoder 430 may use a video decoding technique. The reconstructed geometry image may be transmitted to the point cloud image synthesizer 470.

[0056] The placeholder video decoder 440 decodes the input bitstream to reconstruct the placeholder image. At this time, the placeholder video decoder 440 can use video decoding technology. The reconstructed placeholder image can be transmitted to the point cloud image synthesizer 470.

[0057] The attribute video decoder 450 can decode the input bitstream to reconstruct the attribute image or the mesh texture map. At this time, the attribute video decoder 450 can use video decoding technology. At the same time, the decoding device can receive information indicating whether the image to be reconstructed is the attribute image of the point cloud or the mesh texture map frame by frame from the encoding device. If the information indicates the attribute image of the point cloud, the reconstructed image can be transmitted to the point cloud image synthesizer 470. If the information indicates the mesh texture map, the reconstructed mesh texture map can be combined with the result generated by the mesh geometry information decoder 460 to generate a reconstructed mesh.

[0058] At the same time, the attribute video decoder 450 can perform the following Figure 2 or Figure 3 As shown in Figure 2 In the example of , the attribute video decoder 450 can perform inter-frame prediction by referencing the grid texture map for decoding the attribute image of the point cloud. Figure 3 In the example of , the attribute video decoder 450 can perform inter-frame prediction by referring to the attribute image of the point cloud for decoding the grid texture map.

[0059] The mesh geometry information decoder 460 may receive the bitstream as input to reconstruct the geometry information of the mesh. In this case, the geometry information of the mesh may include information of mesh vertices, mesh connectivity, and texture vertices of the mesh. The reconstructed geometry information may be combined with the reconstructed mesh texture map generated by the attribute video decoder 450, and the combination may be output in the form of a reconstructed mesh.

[0060] The point cloud image synthesizer 470 combines the reconstructed patch information, the geometry image, the placeholder image, and the attribute image to reconstruct the point cloud.

[0061] Reference now Figures 5 to 7 , a point cloud mesh encoding device utilizing a reconstructed point cloud or a reconstructed mesh is described.

[0062] Figure 5 It is a block diagram of a point cloud mesh encoding device according to another embodiment of the present disclosure.

[0063] The encoding device encodes the input point cloud and mesh to generate a bitstream. Alternatively, the encoding device may encode the input point cloud or mesh to generate a bitstream. The encoding device may include all or part of a coordinate system converter 510, a color space converter 520, a point cloud encoder 530, a storage unit 540, a mesh encoder 550, and a bitstream synthesizer 560.

[0064] The coordinate system converter 510 receives the point cloud and the mesh as input and converts their coordinate systems to generate a converted point cloud and / or mesh. At this time, the coordinate system conversion can be used to convert the Cartesian coordinate system into a cylindrical coordinate system. Alternatively, the coordinate system conversion can be used to convert the spherical coordinate system into a cylindrical coordinate system. Alternatively, the coordinate system conversion can be used to convert the cylindrical coordinate system into a spherical coordinate system. In other words, mutual conversion between the Cartesian coordinate system, the cylindrical coordinate system and the spherical coordinate system is feasible, and information about the determined conversion can be transmitted to the decoding device. The decoding device can perform a coordinate system inversion based on the information.

[0065] On the other hand, if the three-dimensional coordinates of the point cloud and the mesh are in the world standard coordinate system, the coordinate system conversion may include converting the world standard coordinate system into the internal coordinate system of the encoding device. Here, if the input to the coordinate system converter 510 is a point cloud, the coordinate system converter 510 may convert the coordinate system for the coordinate values ​​that are the geometric information of each point. Alternatively, if the input is a mesh, the coordinate system converter 510 may convert the coordinate system for the three-dimensional coordinate values ​​of the mesh vertices.

[0066] The color space converter 520 receives the input of the point cloud and / or mesh and performs color space conversion on the point cloud and / or mesh to generate a converted point cloud and / or mesh. At this time, if the input to the color space converter 520 is a point cloud, the color space converter 520 can convert the color space of the attribute information for each point. At this time, by using the information received from the higher layer, the encoding device can determine the number of channels and the type of attribute information. For example, the attribute information can have three color channels, such as RGB, YUV, YCbCr, etc. Alternatively, the attribute information can contain a reflectance coefficient, such as a single reflectance. Alternatively, the attribute information can include four channels, such as RGB+reflectance. Thus, the color space converter 520 can convert the color space for the remaining color channels other than the reflectance. If the attribute information only includes the reflectance, the color space conversion can be omitted. The encoding device can transmit the information items used for the color space conversion to the decoding device. The decoding device can perform the inverse process of the color space conversion based on the information.

[0067] The point cloud encoder 530 receives input of a point cloud, a reconstructed point cloud, and / or a reconstructed mesh, and encodes the point cloud to generate a point cloud bitstream and a reconstructed point cloud. The reconstructed point cloud may be delivered to a storage unit 540, and the point cloud bitstream may be delivered to a bitstream synthesizer 560.

[0068] The mesh encoder 550 receives an input of a mesh, a reconstructed mesh, and / or a reconstructed point cloud, and encodes the mesh to generate a mesh bitstream and a reconstructed mesh. The reconstructed mesh may be transmitted to the storage unit 540, and the mesh bitstream may be transmitted to the bitstream synthesizer 560.

[0069] The storage unit 540 receives and stores the reconstructed point cloud or the reconstructed mesh when input. Then, the reconstructed point cloud and the reconstructed mesh can be used for point cloud or mesh encoding / decoding.

[0070] The bitstream combiner 560 concatenates all the input bitstreams to generate a single bitstream.

[0071] Figure 6 is a detailed block diagram of a point cloud encoder according to at least one embodiment of the present disclosure.

[0072] As described above, upon receiving input of an original point cloud, a reconstructed point cloud, and / or a reconstructed mesh, the point cloud encoder 530 may encode the original point cloud to generate a point cloud bitstream, and may generate a reconstructed point cloud. The point cloud encoder 530 may include all or part of the following: a first point partitioning unit 602, a geometry information prediction encoder 604, a geometry information entropy encoder 606, a point global motion compensator 608, a second point partitioning unit 610, a point local motion compensator 612, a mesh vertex global motion compensator 614, and a mesh vertex partitioning unit 616, a mesh vertex local motion compensator 618, a geometry information prediction decoder 620, an attribute information compensator 622, a level of detail (LOD) generator 624, an attribute information prediction encoder 626, an attribute information quantizer 628, an attribute information entropy encoder 630, and a bitstream synthesizer 632.

[0073] The first point division unit 602 divides points in the three-dimensional space according to a tree-structured division method (e.g., a binary tree, a quadtree, an octree, or a K-dimensional tree, etc.) using the geometric information of the input original point cloud. The divided points of the point cloud can be transmitted to the geometric information prediction encoder 604.

[0074] The point global motion compensator 608 receives the reconstructed point cloud and the original point cloud as input, and performs global motion compensation on the geometric information of the reconstructed point cloud. Global motion compensation can be performed by using parameters representing the global motion between the original point cloud and the reconstructed point cloud. In this case, global motion compensation uses the same parameters for all points to compensate for the motion. Global motion parameters can be generated by least squares using the geometric information of the original point cloud and the reconstructed point cloud. At this time, the generated parameters can be entropy encoded and transmitted to the decoding device. The reconstructed point cloud after global motion compensation can be transmitted to the second point segmentation unit 610.

[0075] The second point cloud division unit 610 receives the reconstructed point cloud after global motion compensation and divides the reconstructed point cloud after global motion compensation into smaller units. At this time, the division method used may be a tree structure-based division method such as a binary tree, a quadtree, an octree, a KD tree, etc. The divided point cloud may be transmitted to the point local motion compensator 612.

[0076] The point local motion compensator 612 performs local motion compensation on the divided point cloud using the input geometric information of the divided point cloud and the geometric information of the original point cloud. The point local motion compensator 612 can calculate local motion compensation parameters by using the original point cloud and the reconstructed point cloud, and then use the local motion compensation parameters to perform local motion compensation. A difference can be determined between the currently divided point cloud unit and the point cloud unit with the smallest error in the original point cloud, and the difference can be represented by a three-dimensional vector assigned to the motion compensation parameter. The generated parameters can be entropy encoded and then transferred to a decoding device. The motion compensated point cloud can be transmitted to the geometric information prediction encoder 604 and the attribute information prediction encoder 626.

[0077] The mesh vertex global motion compensator 614 receives the reconstructed mesh and the original point cloud as input and performs global motion compensation on the mesh vertices. The globally motion compensated mesh may be transmitted to the mesh vertex partitioning unit 616. Parameters for motion compensation may be entropy encoded and may be transmitted to a decoding device.

[0078] The mesh vertex division unit 616 divides the mesh vertices into smaller units. In this case, a tree structure-based division method such as a binary tree, a quad tree, an octree, a KD tree, etc. may be used as a division method. The divided mesh vertices may be transmitted to the mesh vertex local motion compensator 618.

[0079] The mesh local motion compensator 618 performs motion compensation on the divided mesh vertices respectively by using the divided mesh vertices and the original point cloud. The mesh local motion compensator 618 can perform motion compensation by using the current point cloud and the reconstructed mesh vertices to calculate the parameters required for motion compensation. Here, the parameters required for motion compensation are motion vectors, which can be three-dimensional vectors. Alternatively, the mesh local motion compensator 618 can perform coordinate conversion for each vertex by using a matrix with 6 or 9 elements (such as an affine transformation) as a parameter. The parameters for motion compensation can be entropy encoded and can be transmitted to a decoding device. The motion compensated mesh can be transmitted to the geometry information prediction encoder 604 and the attribute information prediction encoder 626.

[0080] The geometric information prediction encoder 604 receives the divided current point cloud and the motion compensated reconstructed point cloud, or the divided current point cloud and the motion compensated reconstructed mesh, and performs prediction encoding on the geometric information of the current point cloud. Symbols associated with the geometric information generated by the prediction encoding can be transmitted to the geometric information entropy encoder 606 and the geometric information prediction decoding unit 620.

[0081] The geometry information entropy encoder 606 performs entropy encoding on the input geometry information symbol to generate a bit stream. The generated bit stream can be transmitted to the bit stream synthesizer 632.

[0082] The geometry information prediction decoder 620 reconstructs the current point cloud by using the input geometry information symbols. The reconstructed point cloud is transmitted to the attribute information compensator 622 and can be used for the next point cloud or mesh encoding.

[0083] The attribute information compensator 622 receives the attribute information of the current point cloud and the reconstructed geometric information of the current point cloud to correct the attribute information of the current point cloud. The attribute information compensation can use the attribute value of the nearest original point cloud based on the reconstructed geometric information. Optionally, the attribute information of the current point can be determined based on the attribute information of the three nearest points based on the reconstructed geometric information. In this case, the weights of the three points can be determined based on the distance. The attribute-corrected point cloud can be transmitted to the LOD generator 624.

[0084] LOD generator 624 receives the point cloud of attribute correction and divides the point cloud of attribute correction into multiple levels to form a level of detail (LOD). When the LOD level increases, all points are included, and a lower LOD may only include a small amount of points, such as only enough to maintain the shape of a three-dimensional object. Here, the points included in each LOD level can be classified based on distance. The point cloud divided into multiple levels can be transmitted to the attribute information prediction encoder 626.

[0085] The attribute information prediction encoder 626 receives the LOD-divided point cloud, the motion-compensated reconstructed point cloud, and / or the motion-compensated reconstructed mesh as input, and performs prediction coding on each point. The residual attribute information generated by the prediction coding can be transmitted to the attribute information quantizer 628. In addition, the prediction information of the point can be transmitted to the attribute information quantizer 628.

[0086] The attribute information quantizer 628 receives the residual attribute information and performs quantization to generate attribute information symbols. The generated attribute information symbols can be transmitted to the attribute information entropy encoder 630. In addition, the attribute information quantizer 628 can dequantize the symbols according to the quantization, and can reconstruct the attribute information of the point cloud by using the received prediction information. The reconstructed point cloud attribute information can be used to encode the next frame.

[0087] The attribute information entropy encoder 630 entropy encodes the input attribute information symbol to generate a bit stream. The generated bit stream can be transmitted to the bit stream synthesizer 632.

[0088] The bitstream synthesizer 632 concatenates all received bitstreams to generate a point cloud bitstream.

[0089] Figure 7 is a detailed block diagram of a trellis encoder according to at least one embodiment of the present disclosure.

[0090] As described above, the mesh encoder 550 receives input of the original mesh, the reconstructed point cloud, and / or the original mesh and the reconstructed mesh of the previous frame, encodes the original mesh to generate a mesh bitstream, and generates a reconstructed mesh of the current frame. The mesh encoder 550 may include all or part of a point cloud global motion compensator 702, a point cloud partitioning unit 704, a point cloud local motion compensator 706, a texture map predictor 708, a video encoder 710, a mesh geometry information encoder 712, and a bitstream synthesizer 714.

[0091] The point cloud global motion compensator 702 receives the reconstructed point cloud and the original mesh vertices as input to perform global motion compensation. Global motion compensation can be performed by using parameters representing the global motion between the original mesh and the reconstructed point cloud. In this case, global motion compensation uses the same parameters for all points to perform motion compensation. Global motion parameters can be generated by least squares using the geometric information of the original mesh and the reconstructed point cloud. The generated parameters can be entropy encoded and then transferred to a decoding device. The reconstructed point cloud after global motion compensation can be transmitted to the point cloud segmentation unit 704.

[0092] The point cloud division unit 704 receives the reconstructed point cloud of global motion compensation and divides it into smaller units. In this case, a tree structure-based division method such as a binary tree, a quadtree, an octree, a KD tree, etc. can be used as a division method. The divided point cloud can be transmitted to the point cloud local motion compensator 706.

[0093] The point cloud local motion compensator 706 receives the partitioned point cloud and the original mesh vertices as input and performs local motion compensation on the partitioned point cloud. At this time, the information for motion compensation can be entropy encoded and then transmitted to the decoding device. The local motion compensated point cloud can be transmitted to the texture map predictor 708.

[0094] The mesh geometry information encoder 712 receives the input original mesh vertex position information and vertex connectivity information, and encodes the input information to generate a mesh geometry information bitstream, as well as reconstructed mesh vertices and connectivity information. The generated mesh geometry information bitstream can be transmitted to the bitstream synthesizer 714, and the reconstructed mesh vertices and connectivity information can be combined with the reconstructed mesh texture. Figure 1 The reconstructed mesh texture map can be used for the next point cloud or mesh encoding. The reconstructed mesh vertex and connectivity information can also be transmitted to the texture map predictor 708.

[0095] The texture map predictor 708 generates a predicted texture map using the motion compensated point cloud, and the reconstructed mesh vertices and connectivity information. The texture map predictor 708 may subtract the predicted texture map from the original texture map to generate a differential texture map, which may then be transmitted to the video encoder 710. Here, if the reconstructed point cloud is not input to the mesh encoder 550, but the reconstructed mesh of the previous frame has been input, the differential texture map obtained by subtracting the texture map of the reconstructed mesh of the previous frame from the original mesh texture map may be transmitted to the video encoder 710.

[0096] The video encoder 710 encodes the input differential texture map to generate a grid texture map bitstream. The video encoder 710 can use image compression technology such as portable network graphics (PNG), joint image coding experts group (JPEG), JPEG2000, high efficiency image file format (HEIF), etc. Alternatively, the video encoder 710 can use video coding technology such as MPEG-2, H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP8, VP9, ​​AV1, etc. The grid texture map bitstream can be transmitted to the bitstream synthesizer 714.

[0097] The bitstream synthesizer 714 concatenates all input bitstreams to generate a trellis bitstream.

[0098] Reference now Figures 8 to 10 , the following describes a point cloud mesh decoding device using a reconstructed point cloud or a reconstructed mesh.

[0099] Figure 8 It is a block diagram of a point cloud mesh decoding device according to another embodiment of the present disclosure.

[0100] The decoding device receives a bitstream as input to generate a reconstructed point cloud and / or a reconstructed mesh. The decoding device may include all or part of a bitstream separator 810, a point cloud decoder 820, a storage unit 830, a mesh decoder 840, a color space inverse converter 850, and a coordinate system inverse converter 860.

[0101] The bitstream separator 810 separates the input bitstream into a point cloud bitstream and a mesh bitstream. The separated point cloud bitstream may be transmitted to the point cloud decoder 820. The separated mesh bitstream may be transmitted to the mesh decoder 840.

[0102] The point cloud decoder 820 receives the point cloud bitstream, the reconstructed point cloud of the previous frame, and / or the reconstructed mesh to reconstruct the point cloud of the current frame. The reconstructed point cloud of the current frame may be transmitted to the storage unit 830.

[0103] The mesh decoder 840 may reconstruct the mesh of the current frame by receiving the mesh bitstream, the reconstructed point cloud and / or the reconstructed mesh. The reconstructed mesh of the current frame may be transmitted to the storage unit 830.

[0104] The storage unit 830 may store the reconstructed point cloud and the reconstructed mesh of the current frame, and may transmit the reconstructed point cloud and the reconstructed mesh to the point cloud decoder 820 and the mesh decoder 840 for decoding subsequent frames. In addition, the storage unit 830 may transmit the reconstructed point cloud and the mesh to the color space inverse converter 850.

[0105] The color space inverse converter 850 may receive the reconstructed point cloud and mesh, may perform color space inversion on the attribute information of the point cloud and the mesh texture map, and may transmit the inverted attribute information and mesh texture map to the coordinate system inverse converter 860 .

[0106] The coordinate system inverse converter 860 receives the color space converted point cloud and mesh, and applies the coordinate system inversely to the geometric information of the point cloud and mesh vertices to generate a reconstructed point cloud and a reconstructed mesh.

[0107] Fig. 9 is a detailed block diagram of a point cloud decoder according to at least one embodiment of the present disclosure.

[0108] The point cloud decoder 820 generates a reconstructed point cloud by inputting a bitstream, a reconstructed point cloud of a previous frame, and a reconstructed mesh. The point cloud decoder 820 may include all or part of the following: a bitstream separator 902, a geometry information entropy decoder 904, a point cloud global motion compensator 906, a point partitioning unit 908, a point cloud local motion compensator 910, a mesh vertex global motion compensator 912 and a mesh vertex partitioning unit 914, a mesh vertex local motion compensator 916, a geometry information prediction decoder 918, an LOD generator 20, an attribute information entropy decoder 922, an attribute information dequantizer 924, and an attribute information prediction decoder 926.

[0109] The bitstream separator 902 receives the bitstream and separates the bitstream into a point cloud geometry information bitstream and a point cloud attribute information bitstream. The point cloud geometry information bitstream may be transmitted to the geometry information entropy decoder 904, and the point cloud attribute information bitstream may be transmitted to the attribute information entropy decoder 922.

[0110] The geometry information entropy decoder 904 decodes the point cloud geometry information bitstream to generate geometry information symbols and parameters required for motion compensation. The generated geometry information symbols can be transmitted to the geometry information prediction decoder 918.

[0111] The point global motion compensator 906 receives the reconstructed point cloud input and performs global motion compensation on the geometric information of the reconstructed point cloud. Global motion compensation can be performed by using parameters representing the global motion between the original point cloud and the reconstructed point cloud. In this case, the global motion compensation uses the same parameters to perform motion compensation for all points. The global motion parameters can be obtained from the geometric information entropy decoder 904. The reconstructed point cloud after global motion compensation can be transmitted to the point segmentation unit 908.

[0112] The point partitioning unit 908 receives the globally motion compensated reconstructed point cloud and partitions it into smaller units. In this case, a tree structure-based partitioning method such as a binary tree, a quadtree, an octree, a KD tree, etc. can be used as a partitioning method. The partitioned point cloud can be transmitted to the point local motion compensator 910.

[0113] The point local motion compensator 910 performs local motion compensation on the geometric information of the input divided point cloud by using parameters related to local motion compensation. Here, the parameter may be a motion vector represented as a three-dimensional vector. Further, the parameter may be obtained from the geometric information entropy decoder 904. The motion compensated point cloud may be transmitted to the geometric information prediction decoder 918 and the attribute information prediction decoder 926.

[0114] The mesh vertex global motion compensator 912 receives the reconstructed mesh as input and performs global motion compensation on the mesh vertices. At this time, the global motion parameters for global motion compensation can be obtained from the geometric information entropy decoder 904. The globally motion compensated mesh can be transmitted to the mesh vertex partitioning unit 914.

[0115] The mesh vertex division unit 914 divides the mesh vertices into smaller units. In this case, a tree structure-based division method such as a binary tree, a quad tree, an octree, a KD tree, etc. may be used as a division method. The divided mesh vertices may be transmitted to the mesh vertex local motion compensator 916.

[0116] The mesh local motion compensator 916 performs motion compensation on each divided mesh vertex. At this time, motion compensation can utilize parameters related to local motion compensation sent from the encoding device. Here, the parameters required for motion compensation can be motion vectors. Motion vectors can mean three-dimensional vectors. Alternatively, the mesh local motion compensator 916 can perform coordinate conversion for each vertex by using a matrix with 6 or 9 elements (such as affine transformation) as a parameter. Parameters related to motion compensation can be obtained from the geometric information entropy decoder 904. The motion compensated mesh can be transmitted to the geometric information prediction decoder 918 and the attribute information prediction decoder 926.

[0117] The geometry information prediction decoder 918 receives the geometry information symbol, the motion compensated point cloud of the previous frame, and the motion compensated mesh of the previous frame to reconstruct the geometry information of the point cloud of the current frame. The reconstructed point cloud geometry information can be output together with the reconstructed point cloud attribute information. Further, the reconstructed point cloud geometry information can be transmitted to the LOD generator 920.

[0118] The LOD generator 920 divides the point cloud by constructing a plurality of LODs using the input point cloud geometry information. The point cloud divided according to the LOD may be transmitted to the attribute information prediction decoder 926.

[0119] The attribute information entropy decoder 922 decodes the input point cloud attribute information bitstream to reconstruct the quantized residual attribute information of the point cloud. The quantized residual attribute information can be transmitted to the attribute information dequantizer 924.

[0120] The attribute information dequantizer 924 dequantizes the input quantized residual attribute information to reconstruct the residual attribute information. The reconstructed residual attribute information may be transmitted to the attribute information prediction decoder 926.

[0121] The attribute information prediction decoder 926 reconstructs the attribute information of the current frame by using the input reconstructed residual attribute information, LOD partition point cloud, motion compensation point cloud of the previous frame and / or motion compensation mesh of the previous frame. The reconstructed point cloud attribute information can be output together with the reconstructed point cloud geometric information.

[0122] Fig.10 is a detailed block diagram of a trellis decoder according to at least one embodiment of the present disclosure.

[0123] The mesh decoder 840 receives the mesh bitstream, the reconstructed point cloud and the reconstructed mesh as input to reconstruct the mesh of the current frame. The mesh decoder 840 may include all or part of a bitstream separator 1002, a video decoder 1004, a mesh geometry information decoder 1006, a point cloud global motion compensator 1008, a point cloud partitioning unit 1010, a point cloud local motion compensator 1012 and a texture map predictor 1014.

[0124] The bitstream separator 1002 receives the mesh bitstream as input and separates the mesh bitstream into a texture map bitstream and a mesh geometry information bitstream. The texture map bitstream may be transmitted to the video decoder 1004. The mesh geometry information bitstream may be transmitted to the mesh geometry information decoder 1006.

[0125] The video decoder 1004 can decode the input texture map bitstream to reconstruct a residual texture map. The residual texture map can be summed with the predicted texture map and can be output together with the reconstructed mesh vertices and connectivity information.

[0126] The mesh geometry information decoder 1006 decodes the input mesh geometry information bitstream to reconstruct mesh vertices and connectivity information. The reconstructed mesh vertices and connectivity information may be transmitted to the texture map predictor 1014. Further, the reconstructed mesh vertices and connectivity information may be combined with the reconstructed mesh texture. Figure 1 The output is in the form of a reconstructed mesh.

[0127] The point cloud global motion compensator 1008 receives the reconstructed point cloud as input and performs global motion compensation on the geometric information of the reconstructed point cloud. Motion compensation can be performed by using parameters representing the global motion between the original mesh vertices and the reconstructed point cloud. In this case, the global motion compensation uses the same parameters for all points to perform motion compensation. The global motion parameters can be obtained from the geometric information entropy decoder 904. The globally motion compensated reconstructed point cloud can be transmitted to the point cloud segmentation unit 1010.

[0128] The point cloud division unit 1010 receives the reconstructed point cloud of global motion compensation and divides it into smaller units. In this case, a tree structure-based division method such as a binary tree, a quadtree, an octree, a KD tree, etc. can be used as a division method. The divided point cloud can be transmitted to the point local motion compensator 1012.

[0129] The point local motion compensator 1012 performs local motion compensation on the geometric information of the input segmented point cloud by using parameters related to local motion compensation. Here, the parameter can be a motion vector represented as a three-dimensional vector. Further, the parameter can be obtained from the geometric information entropy decoder 904. The motion compensated point cloud can be transmitted to the texture map predictor 1014.

[0130] The texture map predictor 1014 generates a predicted texture map by using the motion compensated point cloud, the reconstructed mesh vertices and the connectivity information. The texture map can be reconstructed by summing the predicted texture map and the differential texture map. In addition, if the reconstructed point cloud is not input to the mesh decoder 840 and the reconstructed mesh of the previous frame has been input to the mesh decoder 840, the differential texture map can be summed with the texture map of the reconstructed mesh of the previous frame to generate a reconstructed mesh texture map. The reconstructed mesh texture map can be output in the form of a reconstructed mesh together with the reconstructed mesh vertices and the connectivity information.

[0131] See now Fig.11 and Fig.12 , the following describes a method of utilizing a reconstructed grid when a current point is predicted by a geometry information prediction encoder 604 and an attribute information prediction encoder 626 of a point cloud encoder 530 and a geometry information prediction decoder 918 and an attribute information prediction decoder 926 of a point cloud decoder 820.

[0132] Fig.11 is a diagram illustrating a method of predicting geometric information and attribute information of a point cloud by using a reconstructed mesh.

[0133] like Fig.11 As shown, for the reconstructed mesh, each vertex may include geometric information and attribute information, and one surface of the triangle mesh may include three vertices in total.

[0134] In order to predict the current point, the face of the mesh having the face closest to the current point can be selected, and the information of the vertices included in the face can be used. At this time, the encoding device can find the face closest to the current point based on the distance. The decoding device can select a specific mesh face by receiving information such as the index of the face of the reconstructed mesh and the three-dimensional motion vector from the encoding device.

[0135] Then, a representative point can be generated from the selected face, and the representative point can be used as a prediction point. At this time, the geometric information and attribute information of the representative point can be calculated from the geometric information of the three vertices included in the mesh face. For example, the geometric information and attribute information of the representative point can be the weighted sum of the geometric information and attribute information of the three vertices included in the selected mesh face. In this case, a fixed value can be used as a weight according to the agreement between the encoding device and the decoding device. Optionally, the geometric information of the representative point can be the position of the center of gravity of the three vertices.

[0136] Fig.12 is a diagram showing a method of predicting geometric information and attribute information of a point cloud by using a reconstructed mesh with a texture map.

[0137] like Fig.12As shown, for the reconstructed mesh, each vertex may include geometric information and may include attribute information in the form of a texture map.

[0138] In order to predict the current point, the face of the mesh having the face closest to the current point can be selected, and the attribute information of the texture map can be used based on the information of the vertices included in the face and the coordinates of the texture map included in the vertices. At this time, the encoding device can find the face closest to the current point based on the distance. The decoding device can select a specific mesh face by receiving information such as the index of the face of the reconstructed mesh and the three-dimensional motion vector from the encoding device.

[0139] A representative point can then be generated from the selected face, and the representative point can be used as a prediction point. In this case, the geometric information of the representative point can be calculated from the geometric information of the three vertices included in the mesh face. For example, the geometric information of the representative point can be a weighted sum of the geometric information of the three vertices included in the selected mesh face. In this case, the weight can be a fixed value based on the agreement between the encoding device and the decoding device. Optionally, the geometric information of the representative point can be the position of the center of gravity of the three vertices.

[0140] Then, the coordinates of the texture map at the relevant position can be generated based on the coordinates of the vertex, the coordinates of the texture map of the vertex and the geometric information of the representative point. By using the coordinates of the texture map of the representative point, the attribute information of the texture map at the relevant position can be determined as the attribute information of the representative point.

[0141] Reference now Fig.13 , the following describes a method performed by the texture map predictor 708 of the mesh encoder 550 and the texture map predictor 1014 of the mesh decoder 840 for generating a predicted texture map based on the motion compensated point cloud and the reconstructed geometric information of the current mesh.

[0142] Fig.13 is a diagram showing a method of predicting a mesh texture map by using a reconstructed point cloud of a previous frame.

[0143] The geometric information of the reconstructed mesh of the current frame may include three-dimensional coordinates of vertices, connectivity of vertices, and two-dimensional coordinates of a texture map.

[0144] like Fig.13 As shown in the example of , the face of the mesh can be obtained based on the three-dimensional coordinates and connectivity information of the vertices, and the attribute information of the reconstructed point cloud can be projected onto the face. In addition, the projected attribute information can be projected onto the texture map again based on the texture map coordinates of the vertices. The predicted texture map can be an image obtained by projecting the attribute information of the point cloud into the UV domain based on the coordinates of the vertices and the coordinates of the texture map.

[0145] In the following, by using Fig.14 and Fig.15 The diagram describes a method for encoding a point cloud and a mesh and a method for decoding a point cloud and a mesh.

[0146] Fig.14 is a flow chart of a method of encoding point clouds and meshes according to at least one embodiment of the present disclosure.

[0147] The encoding device obtains a point cloud and a mesh ( S1400 ).

[0148] The encoding device generates a point cloud image from the point cloud (S1402). Here, the point cloud image includes patch information, a geometric image, a placeholder image, and an attribute image.

[0149] The slice information is information used when a plurality of slices classified in a three-dimensional space are mapped to a two-dimensional projection plane. Thus, the slice information may include a coordinate value of each slice in the three-dimensional space and information such as width, length, depth, etc. in the three-dimensional space. The slice information may also include a coordinate value in a two-dimensional projection plane and information such as horizontal and vertical lengths, etc.

[0150] The geometric image is an image that two-dimensionally maps the position of a point in a three-dimensional space to the distance between the projection plane using patch information. In other words, the geometric image can be a map of the depth between a point and a two-dimensional plane when the three-dimensional space is projected onto the two-dimensional plane.

[0151] The placeholder image is an image indicating the positions where a plurality of patches classified in a three-dimensional space are projected as images onto a two-dimensional projection plane. In other words, the placeholder image may be a binary image with 0 or 1 indicating whether a point is projected for each pixel position.

[0152] An attribute image can be generated from the attribute information of the point cloud. An attribute image is a two-dimensional image generated by projecting the attribute values ​​of points in a three-dimensional space onto a projection plane.

[0153] The encoding apparatus encodes the slice information to generate a first bitstream (S1404).

[0154] The encoding device encodes the geometric image to generate a second bit stream (S1406).

[0155] The encoding device encodes the placeholder image to generate a third bitstream ( S1408 ).

[0156] The encoding apparatus encodes the attribute image or the mesh texture map to generate a fourth bitstream (S1410).

[0157] The encoding device determines information indicating whether an image to be encoded is an attribute image of a point cloud or a mesh texture map on a frame-by-frame basis.

[0158] The encoding device may encode the attribute image by performing inter-frame prediction with reference to the grid texture map, such as in Figure 2 Alternatively, as in Figure 3 In the example, the encoding device can encode the grid texture map by performing inter-frame prediction on the attribute image of the reference point cloud.

[0159] The encoding device may generate the second to fourth bitstreams using video encoding technology such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP8, VP9, ​​AV1, etc.

[0160] The encoding device encodes the geometric information of the mesh to generate a fifth bitstream (S1412). Here, the geometric information of the mesh includes mesh vertices, mesh connectivity, and texture map vertices of the mesh.

[0161] The encoding apparatus combines the first to fifth bitstreams ( S1414 ).

[0162] The encoding apparatus may include a symbol indicating a type of each bit stream at the beginning of each bit stream, and may combine each bit stream according to the symbol and a preset order.

[0163] Fig.15 is a flowchart of a method of decoding point clouds and meshes according to at least one embodiment of the present disclosure.

[0164] The decoding apparatus separates a bit stream into a first bit stream, a second bit stream, a third bit stream, a fourth bit stream, and a fifth bit stream (S1500).

[0165] The decoding apparatus may decode a symbol indicating the type of each bit stream included at the beginning of each bit stream, and may separate each bit stream according to the decoded symbol.

[0166] The decoding apparatus decodes the first bit stream to reconstruct slice information (S1502).

[0167] The decoding device decodes the second bit stream to reconstruct a geometric image (S1504).

[0168] The decoding device decodes the third bitstream to reconstruct the placeholder image (S1506).

[0169] The decoding device decodes the fourth bit stream to reconstruct the attribute image or the grid texture map (S1508).

[0170] The decoding device may decode information indicating whether the image to be reconstructed is an attribute image of a point cloud or a mesh texture map, and decode the fourth bit stream according to the decoded information.

[0171] The decoding device may decode the attribute image by performing inter-frame prediction with reference to the grid texture map, such as Figure 2 Alternatively, as in Figure 3 In the example, the decoding device can decode the grid texture map by performing inter-frame prediction on the attribute image of the reference point cloud.

[0172] In order to decode the second bit stream to the fourth bit stream, the decoding device can use video decoding technology, such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, VP8, VP9, ​​AVl, etc.

[0173] The decoding device decodes the fifth bitstream to reconstruct geometric information of the mesh (S1510). Here, the geometric information of the mesh includes mesh vertices, mesh connectivity, and texture map vertices of the mesh.

[0174] The decoding device combines the geometric information of the mesh and the mesh texture map to generate a reconstructed mesh (S1512).

[0175] The decoding device combines the tile information, the geometric image, the placeholder image, and the attribute image to generate a reconstructed point cloud ( S1508 ).

[0176] Although the steps in each flowchart are described as being performed sequentially, these steps are merely examples of the technical concepts of some embodiments of the present disclosure. Therefore, a person of ordinary skill in the art to which the present disclosure belongs can perform the steps by changing the order described in each figure or by performing two or more steps in parallel. Therefore, the steps in each flowchart are not limited to the time series order shown.

[0177] It should be understood that the above description presents illustrative embodiments that can be implemented in various other ways. The functions described in some embodiments can be implemented by hardware, software, firmware and / or a combination thereof. It should also be understood that the functional components described in this disclosure are marked with "... units" to strongly emphasize the possibility of their independent implementation.

[0178] At the same time, the various methods or functions described in some embodiments may be implemented as instructions stored in a non-transitory recording medium that can be read and executed by one or more processors. For example, the non-transitory recording medium may include various types of recording devices in which data is stored in a form readable by a computer system. For example, the non-transitory recording medium may include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).

[0179] Although the embodiments of the present disclosure have been described for illustrative purposes, however, those of ordinary skill in the art to which the present disclosure belongs should recognize that various modifications, additions, and substitutions are possible without departing from the concept and scope of the present disclosure. Therefore, for the sake of brevity and clarity, the embodiments of the present disclosure have been described. The scope of the technical concept of the embodiments of the present disclosure is not limited by the diagrams. Therefore, those of ordinary skill in the art to which the present disclosure belongs should understand that the scope of the present disclosure should not be limited by the embodiments explicitly described above, but by the claims and their equivalents.

[0180] (reference numerals)

[0181] 110 Point Cloud Image Generator

[0182] 120-chip information encoder

[0183] 130 Geometry Video Encoder

[0184] 140 Placeholder Video Encoder

[0185] 150 Attributes Video Encoder

[0186] 160 Mesh geometry information encoder

[0187] 420-chip information decoder

[0188] 430 Geometry Video Decoder

[0189] 440 Placeholder Video Decoder

[0190] 450 Attribute Video Decoder

[0191] 460 Mesh geometry information decoder

[0192] 470 Point Cloud Image Synthesizer

[0193] Cross-reference to related applications

[0194] This application claims priority to and the benefit of Korean Patent Application No. 10-2022-0035266, filed on March 22, 2022, and Korean Patent Application No. 10-2023-0020172, filed on February 15, 2023, the entire contents of each of which are incorporated herein by reference.

Claims

1. A method for decoding a point cloud and a mesh performed by a point cloud mesh decoding device, the method comprising: separating the bit stream into a first bit stream, a second bit stream, a third bit stream, a fourth bit stream, and a fifth bit stream; reconstructing slice information by decoding the first bit stream; reconstructing a geometric image by decoding the second bit stream; reconstructing a placeholder image by decoding the third bit stream; reconstructing an attribute image or a grid texture map by decoding the fourth bit stream; and reconstructing geometric information of the mesh by decoding the fifth bit stream, The geometric information of the mesh includes vertices of the mesh, connectivity of the mesh, and vertices of the mesh texture map.

2. The method according to claim 1, further comprising: A reconstructed mesh is generated by combining the geometric information of the mesh with the mesh texture map.

3. The method according to claim 1, further comprising: A reconstructed point cloud is generated by combining the patch information, the geometric image, the placeholder image, and the attribute image.

4. The method according to claim 1, wherein: The separated bitstream includes: A symbol indicating a type of each bit stream and included in a leading portion of each bit stream is decoded, and each bit stream is separated according to the symbol.

5. The method according to claim 1, wherein: Reconstructing the geometric image, reconstructing the placeholder image, and reconstructing the attribute image or the grid texture map includes utilizing video decoding technology.

6. The method according to claim 1, wherein: Reconstructing the attribute image or the grid texture map comprises: Indication information indicating whether the image to be reconstructed is the attribute image in the point cloud or the grid texture map is decoded, and the fourth bit stream is decoded according to the indication information.

7. The method according to claim 1, wherein: The patch information is used when mapping a plurality of patches classified in a three-dimensional space onto a two-dimensional projection plane.

8. The method according to claim 7, wherein: The geometric image includes an image into which a distance between a position of a point in the three-dimensional space and the projection plane is two-dimensionally mapped by using the patch information.

9. The method according to claim 7, wherein: The placeholder image includes an image indicating positions where the plurality of tiles are projected as images on the two-dimensional projection plane.

10. The method according to claim 7, wherein: The attribute image includes a two-dimensional image generated by projecting attribute values ​​of points in the three-dimensional space onto the projection plane.

11. The method according to claim 1, wherein: Reconstructing the attribute image or the grid texture map comprises: Inter-frame prediction is performed with reference to the grid texture map to reconstruct the attribute image.

12. The method according to claim 1, wherein: Reconstructing the attribute image or the grid texture map comprises: Inter-frame prediction is performed with reference to the attribute image of the point cloud to reconstruct the mesh texture map.

13. A method for encoding a point cloud and a mesh performed by a point cloud mesh encoding device, the method comprising: Obtaining the point cloud and the grid; generating a point cloud image from the point cloud, wherein the point cloud image includes patch information, a geometric image, a placeholder image, and an attribute image; encoding the slice information; Encoding the geometric image; Encoding the placeholder image; encoding the attribute image or the grid texture map; and encoding geometric information of the mesh, The geometric information of the mesh includes vertices of the mesh, connectivity of the mesh, and vertices of the mesh texture map.

14. The method according to claim 13, further comprising: A first bit stream generated by encoding the slice information, a second bit stream generated by encoding the geometric image, a third bit stream generated by encoding the placeholder image, a fourth bit stream generated by encoding the attribute image or the mesh texture map, and a fifth bit stream generated by encoding the geometric information of the mesh are combined.

15. The method according to claim 14, wherein: The combination includes: A symbol indicating the type of each bit stream is included in a leading portion of each bit stream, and the bit streams are combined in a predetermined order according to the symbol.

16. The method according to claim 11, wherein: Encoding the geometry image, encoding the placeholder image, and encoding the attribute image or the mesh texture map includes utilizing video coding techniques.

17. The method according to claim 11, wherein: Encoding the attribute image or the grid texture map includes: Information indicating whether an image to be encoded is the attribute image or the mesh texture map in the point cloud is determined frame by frame.

18. A computer-readable recording medium storing a bit stream generated by a point cloud and mesh coding method, the point cloud and mesh coding method comprising: Get point cloud and mesh; generating a point cloud image from the point cloud, wherein the point cloud image includes patch information, a geometric image, a placeholder image, and an attribute image; encoding the slice information; Encoding the geometric image; Encoding the placeholder image; encoding the attribute image or the grid texture map; and encoding geometric information of the mesh, The geometric information of the mesh includes vertices of the mesh, connectivity of the mesh, and vertices of the mesh texture map.

Images