Patch mesh connection coding

Abstract

The connectivity and mapping information of mesh surface patches can be encoded after projection into 2D. Regarding the connectivity information, the projection operation does not change the connections between vertices, so the same list of connected vertices can be carried in the atlas data. Similarly, the mapping information does not change after projection and can be carried in the atlas data. We disclose two methods for encoding the connectivity and mapping information. For the connectivity information, the video-based method uses adjacent color coding. For the mapping coordinates, the method uses the projected vertex positions. The connectivity and mapping can also be processed by an external mesh encoder. The newly proposed mapping information can be utilized to perform temporal compression.

Description

Technical Field

[0001] [Cross - reference to Related Applications] This application claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 269,905, filed on March 25, 2022, entitled "PATCH MESH CONNECTIVITY CODING", which is hereby incorporated by reference in its entirety for all purposes.

[0002] The present invention relates to 3D graphics. More specifically, the present invention relates to the coding of 3D graphics.

Background Art

[0003] In recent years, new methods for compressing volumetric content such as point clouds based on projections from 3D to 2D are being standardized. This method, also known as V3C (Visual Volume Video - based Compression), maps 3D volume data to several 2D patches, further arranges the patches in an atlas image, and then encodes them with a video encoder. The atlas image corresponds to the geometry of the points, their respective textures, and an occupancy map indicating which positions should be considered for point cloud reconstruction.

[0004] In 2017, MPEG issued a Call for Proposals (CfP) for point cloud compression. After evaluating several proposals, MPEG is currently considering two different techniques for point cloud compression: 3D native coding techniques (based on octree and similar coding methods), or conventional video coding after 3D-to-2D projection. For dynamic 3D scenes, MPEG uses Test Model Software (TMC2) based on patch surface modeling, projection of patches from 3D to 2D images, and coding of the 2D images by a video encoder such as HEVC. This method has proven to be more efficient than native 3D coding and can achieve competitive bitrates with acceptable quality.

[0005] Since coding 3D point clouds using projection-based methods (also known as video-based methods or V-PCC) has been successful, future versions of this standard are expected to include further 3D data, such as 3D meshes. However, the current version of this standard is only suitable for transmitting sets of unconnected points and lacks a mechanism for transmitting point connections, as required for 3D mesh compression.

[0006] Methods have also been proposed to extend the functionality of V-PCC to meshes. One possible method is to encode vertices using V-PCC and then encode connections using a mesh compression method such as TFAN or Edgebreaker. A limitation of this method is that the original mesh must be dense enough so that the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection. Furthermore, since the order of vertices affects the coding of connections, different methods have been proposed for reorganizing the connections of the mesh. An alternative method for encoding sparse meshes is to encode the vertex positions in 3D using RAW patch data. In this method, all vertices are encoded as RAW data, as RAW patches directly encode (x,y,z), while connections are encoded by a similar mesh compression method as described above. Note that in RAW patches, vertices can be sent in any preferred order, so the order generated from connection encoding can be used. While this method can encode sparse point clouds, RAW patches are not efficient for encoding 3D data, and further data such as triangular face attributes may be missing from this method. [Overview of the project] [Problems that the invention aims to solve]

[0007] The connection and mapping information of mesh surface patches can be encoded after projection into 2D. Regarding connection information, since the projection operation does not alter the connections between vertices, the same list of connected vertices can be carried in the atlas data. Similarly, the mapping information remains unchanged after projection and can be carried in the atlas data. Two methods for encoding connection and mapping information are disclosed. For connection information, the video-based method uses adjacent color coding. For mapping coordinates, the method uses the positions of the projected vertices. Connections and mapping can also be processed by an external mesh encoder. Temporal compression can be performed using the newly proposed mapping information. [Means for solving the problem]

[0008] In one embodiment, a method for encoding connection and mapping information includes the steps of: encoding vertex mapping information including delta information for geometry correction; and encoding patch connection information, which includes implementing mesh simplification by fixing the positions of vertices in time. The method further includes transmitting a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly. The steps of encoding the vertex mapping information and encoding the patch connection information are performed by an external encoder. The step of encoding the patch connection information includes using color coding on the occupancy map. The use of color coding on the occupancy map is limited to a maximum of four colors. The step of encoding the vertex mapping information further includes using rate distortion face transmission. Implementing mesh simplification includes transmitting only boundary vertices and not internal vertices. The internal vertices are determined based on a previous set of internal vertices from the previous frame.

[0009] In another embodiment, the device includes non-temporary memory for storing an application, the application for encoding patch connection information, which includes encoding vertex mapping information including delta information for geometry correction and implementing mesh simplification by fixing the positions of vertices in time; and a processor coupled to the memory and configured to process the application. The application is further configured to transmit a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly. Encoding the vertex mapping information and encoding the patch connection information are performed by an external encoder. Encoding the patch connection information includes using color coding in the occupancy map. The use of color coding in the occupancy map is limited to a maximum of four colors. Encoding the vertex mapping information further includes using rate-distorted face transmission. Implementing mesh simplification includes transmitting only boundary vertices and not internal vertices. The internal vertices are determined based on a previous set of internal vertices from the previous frame.

[0010] In another embodiment, the system includes one or more cameras for acquiring 3D content; an encoder for encoding the 3D content, wherein the encoding includes encoding vertex mapping information including delta information for geometry correction and encoding patch connection information, which includes implementing mesh simplification by fixing the positions of vertices in time; and a decoder for decoding the encoded 3D content, wherein the decoding includes adjusting the mesh using the delta information and determining the internal vertices of the patch connection information from previous internal vertices of the previous frame. The encoder is further configured to transmit a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly. Encoding the vertex mapping information and encoding the patch connection information are performed by an external encoder. Encoding the patch connection information includes using color coding in the occupancy map. The use of color coding in the occupancy map is limited to a maximum of four colors. Encoding the vertex mapping information further includes using rate-distorted face transmission. Implementing mesh simplification includes transmitting only boundary vertices and not internal vertices. [Brief explanation of the drawing]

[0011] [Figure 1] This is a diagram illustrating binary coding in several embodiments. [Figure 2] This is a diagram of color coding using an occupancy map according to several embodiments. [Figure 3] This diagram shows RD plane transmission according to several embodiments. [Figure 4] This diagram shows the temporal stability of mesh connections in several embodiments. [Figure 5] This is a flowchart of a patch mesh connection coding method according to several embodiments. [Figure 6]This is a block diagram of an exemplary computer device configured to implement a patch mesh connection coding method according to several embodiments. [Modes for carrying out the invention]

[0012] The connection and mapping information of mesh surface patches can be encoded after projection into 2D. Regarding connection information, since the projection operation does not alter the connections between vertices, the same list of connected vertices can be carried in the atlas data. Similarly, the mapping information remains unchanged after projection and can be carried in the atlas data. Two methods for encoding connection and mapping information are disclosed. For connection information, the video-based method uses adjacent color coding. For mapping coordinates, the method uses the positions of the projected vertices. Connections and mapping can also be processed by an external mesh encoder. Temporal compression can be performed using the newly proposed mapping information.

[0013] Connection information indicates which pixels are connected. A triangle has an information set. One of the information sets is the position of the triangle on the texture map. Each triangle on the texture map has two information sets: 1) how the vertices are connected in 3D (e.g., a connection list), and 2) vertex mapping information.

[0014] There are three ways to encode vertex mapping information: implicit, explicit, and binary. In an implicit implementation, when projected onto a 2D surface, the projection is the same as the mapping. For example, the location hit when projecting onto a projection surface is the UV coordinates. In an explicit implementation, even though projection is performed, different coordinates are sent to the texture. In a binary implementation, explicit information is encoded by an external encoder (e.g., Draco or AFX).

[0015] This shows the updated syntax for explicit implementation. AtlasPatch2dPosX = mpdu_vertex_pos_x[tileID][patchIdx][I] AtlasPatch2dPosY = mpdu_vertex_pos_y[tileID][patchIdx][I] if (asps_mesh_uv_coordinates_present_flag) mappingU = (mpdu_vertex_u_coord[tileID][patchIdx][i]) / 2 asps_mesh_coordinates_bit_depth_minus1+1 -1 mappingV = (mpdu_vertex_v_coord[tileID][patchIdx][i]) / 2 asps_mesh_coordinates_bit_depth_minus1+1 -1 else mappingU = AtlasPatch2dPosX mappingV = AtlasPatch2dPosY A flag can be sent to indicate whether vertex mapping information is sent implicitly or explicitly. If the information is sent explicitly, the value is scaled by the bit depth.

[0016] If binary coding is implemented, an external mesh encoder can be used to encode patch mesh information. U and V are added to the ply, and vertex mapping information is encoded along with the ply. In some embodiments, delta information for the z coordinate is added. The delta information can be used for geometry correction.

[0017] There are numerous ways to encode patch connection information. In an explicit implementation, since which pixels are connected is indicated by the syntax, a list of pixel connections is transmitted in the patch. In a binary implementation, an external encoder can be utilized. In another implementation, mesh simplification can be performed by fixing the positions of vertices over time. In a color-coding implementation, color-coding using an occupancy map is implemented. Triangles can be mapped using only four colors. In another implementation, rate distortion (RD) surface transmission is utilized.

[0018] Figure 1 shows a diagram of binary encoding according to some embodiments. Using a geometry image, map 100 is generated. There are still some information that can be modified by video encoding. By explicitly transmitting delta information to an external encoder, a binary image can be generated while correcting errors from video transmission. UV coordinates useful for video compression are also transmitted. In some embodiments, an external mesh encoder is used to encode patch mesh information.

[0019] Figure 2 shows a color-coding diagram using an occupancy map according to some embodiments. The mapping of the triangular indices can use only four colors. Since the figure is in grayscale, it may be difficult to distinguish some colors, edges, and vertices, but only four colors are used, and triangles of one color do not share edges with triangles of the same color. In the occupancy map, connection information is added using only the luminance channel (4 colors -> (0,0,0), (64,0,0), (128,0,0), (255,0,0)). Further details regarding color-coding and mesh compression can be found in U.S. Patent Application No. 17 / 322,662, titled "VIDEO BASED MESH COMPRESSION," filed on May 17, 2021, U.S. Provisional Patent Application No. 63 / 088,705, titled "VIDEO BASED MESH COMPRESSION," filed on October 7, 2020, and U.S. Provisional Patent Application No. 63 / 087,958, titled "VIDEO BASED MESH COMPRESSION," filed on October 6, 2020, all of which are hereby incorporated by reference in their entirety for all purposes. The connection of the 3D mesh or 2D patch mesh can be encoded using an occupancy map and exploiting temporal correlation using video-based mesh compression. Additionally, vertices can be detected using the color map 200, and the intersection of triangles can be detected based on color.

[0020] Figure 3 shows a diagram of RD plane transmission according to some embodiments. A mesh is received / obtained (300). The mesh connection points are encoded by an encoder (302). On the decoder side, the mesh is reconstructed (304), but the position of the points may be slightly different from the original mesh. Since correction information (e.g., delta information) is sent to the decoder, the decoder can adjust the mesh (306) to be more accurate compared to the original mesh.

[0021] The vertices of the input mesh are encoded with V-PCC and decoded locally. The encoder generates a mesh from the decoded point cloud. The encoder compares the generated face / connection information with the original information. The encoder reports mismatched faces incorporating rate / distortion tradeoffs. The decoder decodes the mesh vertices using V-PCC and generates a mesh from the decoded vertices. The decoder corrects the mesh using the reported mismatched faces. In some embodiments, a similar technique can also be applied to encode UV coordinates and their connections using 2D triangulation instead of 3D.

[0022] Figure 4 shows the temporal stability of mesh connections in several embodiments. When a patch is sent, only boundary vertices are sent. Internal vertices are not sent. The decoder determines the internal vertices (for example, based on the previous or subsequent frame). For example, the internal vertices of patch 400 in frame 1 are used as the internal vertices of patch 402 in frame 2. In some embodiments, a first set of internal vertices is sent (for example, for frame 0 or frame 1), and future frames will use the internal vertices of the previous frame. The decoder can regenerate triangles from boundary vertices and internal vertices. The same internal points can be used even if the patch rotates from frame to frame. Rotation may result in slightly different internal triangles, but this is acceptable. Not sending internal vertices reduces the number of bits sent.

[0023] Figure 5 shows flowcharts of patch mesh connection coding methods according to several embodiments. In step 500, vertex mapping information is encoded. Delta information may be included in the vertex mapping information for geometry correction. A flag may be sent indicating whether the vertex mapping information is sent implicitly or explicitly. In some embodiments, encoding of vertex mapping information uses RD face transmission. In step 502, patch connection information is encoded. Encoding of patch connection information includes implementing mesh simplification by fixing the vertex positions in time. In some embodiments, mesh simplification includes sending only boundary vertices and not internal vertices. Patch connection information may include color coding in the occupancy map. Color coding is limited to a maximum of four colors. In some embodiments, vertex information and patch connection information are performed by an external encoder. In some embodiments, the order of steps is changed. In some embodiments, fewer or additional steps are implemented. For example, the encoded information can be used to adjust the mesh using delta information. In another example, internal vertices of a patch connection can be determined from previous internal vertices in the previous frame.

[0024] Figure 6 shows a block diagram of an exemplary computer device configured to implement a patch-mesh connection coding method according to several embodiments. The computer device 600 can be used to acquire, store, compute, process, communicate, and / or display information such as images and videos, including 3D content. The computer device 600 can implement any of the encoding / decoding modes. Generally, a suitable hardware structure for implementing the computer device 600 includes a network interface 602, memory 604, a processor 606, (one or more) I / O devices 608, a bus 610, and a storage device 612. The choice of processor is not critical as long as a suitable processor of sufficient speed is selected. The memory 604 can be any conventional computer memory known in the art. The storage device 612 can include a hard drive, CD-ROM, CDRW, DVD, DVDRW, high-definition disk / drive, ultra-high-definition drive, flash memory card, or any other storage device. The computer device 600 can include one or more network interfaces 602. Examples of network interfaces include a network card connected to Ethernet or other types of LANs. The (single or multiple) I / O devices 608 may include one or more of the following: a keyboard, mouse, monitor, screen, printer, modem, touchscreen, button interface, and other devices. The storage devices 612 and memory 604 store the (single or multiple) patch mesh connection coding applications 630 used to perform the implementation of patch mesh connection coding, and are likely to be processed as the applications normally would. The computer device 600 may also include more or fewer components than those shown in Figure 6. In some embodiments, patch mesh connection coding hardware 620 is included.The computer device 600 in Figure 6 includes an application 630 and hardware 620 for implementing patch mesh connection coding, but the patch mesh connection coding method can also be implemented on the computer device in hardware, firmware, software, or a combination thereof. For example, in some embodiments, the patch mesh connection coding application 630 is programmed in memory and executed using a processor. In another example, in some embodiments, the patch mesh connection coding hardware 620 is programmed hardware logic including gates specifically designed to implement the patch mesh connection coding method.

[0025] In some embodiments, the (single or multiple) patch mesh connection coding application 630 includes multiple applications and / or modules. In some embodiments, a module also includes one or more submodules. In some embodiments, it may include fewer or more modules.

[0026] Examples of suitable computer devices include personal computers, laptop computers, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants (PDAs), cellular / mobile phones, smart home appliances, game consoles, digital cameras, digital camcorders, camera phones, smartphones, portable music players, tablet computers, mobile devices, video players, video disc writers / players (e.g., DVD writers / players, high-definition disc writers / players, ultra-high-definition disc writers / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smartwatches), vehicles (e.g., autonomous vehicles), or any other suitable computer device.

[0027] To utilize the patch-mesh connection coding method, the device acquires or receives 3D content (e.g., point cloud content). The patch-mesh connection coding method can be implemented with or without user assistance.

[0028] During operation, the patch mesh connection coding method enables more efficient and accurate 3D content encoding compared to conventional implementations.

[0029] Several embodiments of patch mesh connection coding 1. A method for encoding connection information and mapping information, A step of encoding vertex mapping information including delta information for geometry correction, The process includes a step of encoding patch connection information, which involves implementing mesh simplification by fixing the vertex positions in time, A method that includes this.

[0030] 2. The method of paragraph 1, further comprising the step of sending a flag indicating whether the vertex mapping information is sent implicitly or explicitly.

[0031] 3. The method according to paragraph 1, wherein the steps of encoding the vertex mapping information and encoding the patch connection information are performed by an external encoder.

[0032] 4. The method according to paragraph 1, wherein the step of encoding the patch connection information includes using color coding in the occupancy map.

[0033] 5. The use of color coding in the occupancy map is limited to a maximum of four colors, as described in Section 4.

[0034] 6. The method according to paragraph 1, further comprising the step of encoding the vertex mapping information using rate-distortion surface transmission.

[0035] 7. Implementing mesh simplification includes the method described in Section 1, which involves sending only boundary vertices and not sending internal vertices.

[0036] 8. The method according to paragraph 7, wherein the internal vertices are determined based on the previous set of internal vertices from the previous frame.

[0037] 9. A device, Non-temporary memory for storing applications, wherein the applications are Encoding vertex mapping information including delta information for geometry correction, This includes encoding patch connection information, which involves implementing mesh simplification by fixing the vertex positions in time, Non-temporary memory is used to perform this task, A processor coupled to the memory and configured to process the application, A device that includes this.

[0038] 10. The apparatus according to paragraph 9, wherein the application is further configured to transmit a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly.

[0039] 11. The apparatus described in paragraph 9, wherein encoding the vertex mapping information and encoding the patch connection information are performed by an external encoder.

[0040] 12. The apparatus described in paragraph 9, which encodes the patch connection information, including using color coding in the occupancy map.

[0041] 13. The use of color coding in the occupancy map is limited to a maximum of four colors, as described in paragraph 12.

[0042] 14. The apparatus described in paragraph 9, further comprising using rate-distortion surface transmission for encoding the vertex mapping information.

[0043] 15. Implementing mesh simplification includes the device described in Section 9, which transmits only boundary vertices and not internal vertices.

[0044] 16. The apparatus according to paragraph 15, wherein the internal vertices are determined based on a previous set of internal vertices from the previous frame.

[0045] 17. It is a system, One or more cameras for acquiring 3D content, An encoder for encoding the three-dimensional content, wherein the encoding is Encoding vertex mapping information including delta information for geometry correction, This includes encoding patch connection information, which involves implementing mesh simplification by fixing the vertex positions in time, Encoders, including, A decoder for decoding the encoded 3D content, wherein the decoding is performed Adjusting the mesh using the aforementioned delta information, Determining the internal vertices of the patch connection information from the previous internal vertices of the previous frame, A decoder, including A system that includes this.

[0046] 18. The system according to paragraph 17, wherein the encoder is further configured to transmit a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly.

[0047] 19. The system described in paragraph 17, wherein encoding the vertex mapping information and encoding the patch connection information are performed by an external encoder.

[0048] 20. The system described in paragraph 17, which encodes the patch connection information, including using color coding in the occupancy map.

[0049] 21. The use of color coding in the aforementioned occupancy map is limited to a maximum of four colors, as described in Section 20.

[0050] 22. The system described in paragraph 17, further comprising encoding the vertex mapping information using rate-distorted surface transmission.

[0051] 23. Implementing mesh simplification includes the system described in Section 17, which transmits only boundary vertices and not internal vertices.

[0052] The present invention has been described in relation to specific embodiments, including details, to facilitate understanding of its structure and operating principles. Such references to specific embodiments and their details herein are not intended to limit the claims appended herein. Those skilled in the art will readily see that various other modifications can be made to the embodiments selected for illustrative purposes without departing from the spirit and scope of the invention as defined by the claims. [Explanation of Symbols]

[0053] 100 Maps 200 Color Map Receive / acquire 300 mesh Encoding 302 mesh connection points Reconstruct the 304 mesh. Adjust the 306 mesh. 400 patches 402 patch Encode 500 vertex mapping information. 502 Encode patch connection information 600 Computer devices 602 Network Interface 604 memory 606 Processor 608 I / O devices 610 Bus 612 Storage device 620 Patch Mesh Connectivity Coding Hardware 630 Patch Mesh Connection Coding Application

Claims

1. A method for encoding connection information and mapping information, A step of encoding vertex mapping information including delta information for geometry correction, The process includes a step of encoding patch connection information, which involves implementing mesh simplification by fixing the vertex positions in time, A method characterized by including the following.

2. The method according to claim 1, further comprising the step of sending a flag indicating whether the vertex mapping information is sent implicitly or explicitly.

3. The method according to claim 1, characterized in that the steps of encoding the vertex mapping information and encoding the patch connection information are performed by an external encoder.

4. The method according to claim 1, characterized in that the step of encoding the patch connection information includes using color coding in the occupancy map.

5. The method according to claim 4, characterized in that the use of color coding in the occupancy map is limited to a maximum of four colors.

6. The method according to claim 1, further comprising the step of encoding the vertex mapping information using rate distortion face transmission.

7. The method according to claim 1, characterized in that implementing mesh simplification includes transmitting only boundary vertices and not transmitting internal vertices.

8. The method according to claim 7, characterized in that the internal vertices are determined based on a previous set of internal vertices from the previous frame.

9. It is a device, Non-temporary memory for storing applications, wherein the applications are Encoding vertex mapping information including delta information for geometry correction, This includes encoding patch connection information, which involves implementing mesh simplification by fixing the vertex positions in time, Non-temporary memory is used to perform this task, A processor coupled to the non-temporary memory and configured to process the application, An apparatus characterized by including

10. The apparatus according to claim 9, further characterized in that the application is configured to transmit a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly.

11. The apparatus according to claim 9, characterized in that encoding the vertex mapping information and encoding the patch connection information are performed by an external encoder.

12. The apparatus according to claim 9, characterized in that encoding the patch connection information includes using color coding in the occupancy map.

13. The apparatus according to claim 12, characterized in that the use of color coding in the occupancy map is limited to a maximum of four colors.

14. The apparatus according to claim 9, further comprising using rate-distortion surface transmission to encode the vertex mapping information.

15. The apparatus according to claim 9, characterized in that implementing mesh simplification includes transmitting only boundary vertices and not transmitting internal vertices.

16. The apparatus according to claim 15, characterized in that the internal vertices are determined based on a previous set of internal vertices from the previous frame.

17. It is a system, One or more cameras for acquiring 3D content, An encoder for encoding the three-dimensional content, wherein the encoding is Encoding vertex mapping information including delta information for geometry correction, This includes encoding patch connection information, which involves implementing mesh simplification by fixing the vertex positions in time, Encoders, including, A decoder for decoding the encoded three-dimensional content, wherein the decoding is performed Adjusting the mesh using the aforementioned delta information, Determining the internal vertices of the patch connection information from the previous internal vertices of the previous frame, A decoder, including A system characterized by including

18. The system according to claim 17, further characterized in that the encoder is configured to transmit a flag indicating whether the vertex mapping information is transmitted implicitly or explicitly.

19. The system according to claim 17, characterized in that encoding the vertex mapping information and encoding the patch connection information are performed by an external encoder.

20. The system according to claim 17, characterized in that encoding the patch connection information includes using color coding in the occupancy map.

21. The system according to claim 20, characterized in that the use of color coding in the occupancy map is limited to a maximum of four colors.

22. The system according to claim 17, further comprising encoding the vertex mapping information using rate-distortion surface transmission.

23. The system according to claim 17, characterized in that implementing mesh simplification includes transmitting only boundary vertices and not transmitting internal vertices.

Images