Mesh Geometry Coding

Abstract

The generation of the depth image is improved by more efficient encoding using video codecs. The mapping of depth to the luminance channel is performed without using all available bits, and the remaining bits are used to generate a depth scaling factor, which is incorporated into the bilinear interpolation algorithm used during rasterization. A normal filtering procedure is described. In this procedure, the positions of the vertices are adjusted according to the normals estimated from the surface pixels. After decoding the depth image, the pixels related to the surface of a triangle are collected and used to estimate a plane and the normal of the plane. The normal is compared with the normal obtained from the plane defined by the three vertices of the triangle. If there is no match, the position of the vertex is adjusted to match the normal estimated from the surface of the pixel. The adjustment can follow an iterative minimization process.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 269,915, entitled “MESH GEOMETRY CODING,” filed March 25, 2022, which is hereby incorporated by reference in its entirety for all purposes.

[0002] The present invention relates to three-dimensional graphics, and more particularly to coding of three-dimensional graphics. [Background technology]

[0003] In recent years, a new method for compressing volumetric content such as point clouds is becoming standardized, based on a 3D to 2D projection. Also known as V3C (visual volumetric video-based compression), this method maps the 3D volume data into a number of 2D patches, which are then placed into an atlas image, which is then encoded by a video encoder. The atlas image corresponds to the geometry of the points, their respective textures, and an occupancy map that indicates which locations should be considered for point cloud reconstruction.

[0004] In 2017, MPEG issued a Call for Proposals (CfP) for the compression of point clouds. After evaluation of several proposals, MPEG is currently considering two different techniques for point cloud compression: 3D native coding techniques (based on octrees and similar coding methods) or 3D to 2D projection followed by traditional video coding. For dynamic 3D scenes, MPEG is using Test Model Software (TMC2), based on patch surface modeling, projection of the patches from 3D to a 2D image, and coding of the 2D image by a video encoder such as HEVC. This method has proven to be more efficient than native 3D coding and can achieve competitive bitrates with acceptable quality.

[0005] Due to the success of coding 3D point clouds with projection-based methods (also known as video-based methods, or V-PCC), future versions of this standard are expected to include additional 3D data, such as 3D meshes. However, the current version of this standard is only suitable for transmitting a set of unconnected points, and therefore has no mechanism for transmitting point connectivity as required for 3D mesh compression.

[0006] Methods have also been proposed to extend the functionality of V-PCC to meshes. One possible method is to use V-PCC to encode the vertices and then use a mesh compression method such as TFAN or Edgebreaker to encode the connectivity. A limitation of this method is that the point cloud generated from the vertices is not sparse, and the original mesh must be dense so that it can be efficiently encoded after projection. Furthermore, the order of the vertices affects the coding of connectivity, so different methods have been proposed to reorganize the connectivity of the mesh. An alternative method for encoding sparse meshes is to use the raw patch data to encode the 3D vertex positions. Since raw patches directly encode (x,y,z), in this method all vertices are encoded as raw data, while the connectivity is encoded by a similar mesh compression method, as mentioned before. It is noted that in raw patches, the vertices can be sent in any preferred order, so the order generated from the connectivity encoding can be used. While this method can encode sparse point clouds, raw patches are not efficient for encoding 3D data, and further data, such as triangular face attributes, may be missing from this method. Summary of the Invention [Problem to be solved by the invention]

[0007] The generation of the depth image is improved by more efficient encoding using video codecs. The mapping of depth to the luminance channel is performed without using all available bits, and the remaining bits are used to generate a depth scaling factor, which is incorporated into the bilinear interpolation algorithm used during rasterization. A normal filtering procedure is described. In this procedure, the positions of the vertices are adjusted according to the normals estimated from the surface pixels. After decoding the depth image, the pixels related to the surface of a triangle are collected and used to estimate a plane and the normal of the plane. The normal is compared with the normal obtained from the plane defined by the three vertices of the triangle. If there is no match, the position of the vertex is adjusted to match the normal estimated from the surface of the pixel. The adjustment can follow an iterative minimization process. [Means for solving the problem]

[0008] In one aspect, a method of mesh geometry coding includes mapping depth information to a luminance channel using less than all available bits, generating a depth scaling factor that is incorporated into a bilinear interpolation algorithm used during rasterization, and performing normal filtering including adjusting vertex positions according to normals estimated from surface pixels. The mapping of depth information to the luminance channel includes using M bits of the available N bits, where M is less than N. The generating of the depth scaling factor utilizes the remaining 2 bits of the available N bits. The remaining 2 bits allow for multiplying the data by 4 such that the last 2 bits of the data are 0, allowing most significant bit alignment to be used. The method includes applying the depth scaling factor to data, the data including floating point values. The normal filtering includes performing a plane fitting using points within a triangle. The normal filtering includes using 1-ring, 2-ring, or 3-ring face neighborhoods to minimize a total normal angle.

[0009] In another aspect, an apparatus includes a non-transitory memory for storing an application for mapping depth information to a luminance channel using less than all available bits, generating a depth scaling factor incorporated into a bilinear interpolation algorithm used during rasterization, and performing normal filtering including adjusting vertex positions according to normals estimated from surface pixels; and a processor coupled to the memory and configured to process the application. Mapping the depth information to the luminance channel includes using M bits of the available N bits, where M is less than N. Generating the depth scaling factor utilizes the remaining 2 bits of the available N bits. The remaining 2 bits allow for multiplying the data by 4 such that the last 2 bits of the data are 0 and for using most significant bit alignment. The application is configured to apply the depth scaling factor to data, the data including floating point values. The normal filtering includes performing a plane fitting using points within a triangle. The normal filtering includes using a 1-ring, 2-ring, or 3-ring surface neighborhood to minimize a total normal angle.

[0010] In another aspect, a system includes one or more cameras for acquiring three-dimensional content, and an encoder for encoding the three-dimensional content, the encoder using less than all available bits to map depth information to a luminance channel, generating a depth scaling factor that is incorporated into a bilinear interpolation algorithm used during rasterization, and performing normal filtering including adjusting vertex positions according to normals estimated from surface pixels. Mapping the depth information to the luminance channel includes using M bits of the available N bits, where M is less than N. Generating the depth scaling factor utilizes the remaining two bits of the available N bits. The remaining two bits allow for multiplying the data by 4 such that the last two bits of the data are 0, and for using most significant bit alignment. The encoder is configured to apply the depth scaling factor to data, the data including floating point values. The normal filtering includes performing a plane fitting using points within a triangle. The normal filtering includes using a one-ring, two-ring, or three-ring surface neighborhood to minimize a total normal angle. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 illustrates encoding depth information using MSB alignment according to some embodiments. [Diagram 2] FIG. 1 is an illustration of normal filtering in accordance with some embodiments. [Diagram 3] 14A-14C are diagrams of 1-ring, 2-ring and 3-ring face neighborhoods for minimizing the total normal angle, according to some embodiments. [Figure 4] FIG. 1 illustrates a flowchart of a method for mesh geometry coding according to some embodiments. [Diagram 5] FIG. 1 is a block diagram of an example computing device configured to implement a mesh geometry coding method according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The generation of the depth image can be improved by more efficient encoding using video codecs. The mapping of depth to the luminance channel is performed by not using all available bits (e.g. using only 6 bits instead of 8 bits), and the remaining bits are used to generate a depth scaling factor that is incorporated into the bilinear interpolation algorithm used during rasterization. In this way, the depth scaling factor allows the use of more precise values ​​when rasterizing the triangular surface. A normal filtering procedure is also described. In this procedure, the positions of the vertices are adjusted according to the normals estimated from the surface pixels. After decoding the depth image, the pixels related to the triangular surface are collected and used to estimate the plane and, consequently, the normal of the plane. The normal is then compared to the normal obtained from the plane defined by the three vertices of the triangle. If it does not match a certain threshold, the position of the vertex is adjusted to match the normal estimated from the surface of the pixel. The adjustment can also follow an iterative minimization process that finds the total minimum deviation of the normal angle within a 1-ring, 2-ring or 3-ring surface neighborhood of a given surface.

[0013] FIG. 1 illustrates a diagram of encoding depth information using MSB alignment, according to some embodiments. A triangle is projected. Once the triangle is projected, the depth image (the distance between the triangle and the projection surface) produces the image (e.g., gray area) shown in image 100. For example, a patch of the image is projected onto the surface of the image. The range (e.g., 8-bit vs. 6-bit) determines how much of the image is involved. 8-bit allows values ​​from 0 to 256, while 6-bit allows values ​​from 0 to 64.

[0014] The advantage of using N bits (e.g. 8 bits) is that more triangles can be packed together in a patch. With M bits (e.g. 6 bits), the patch is segmented because not all triangles can be packed together. However, if you use M bits (e.g. 6 bits) to put information in a video that uses N bits (e.g. 8 bits) for the luminous channel, there are NM bits (e.g. 2 bits) that are not used. The NM bits (e.g. 2 bits) can be used for video scaling (e.g. all values ​​are multiplied by 4). For example, 32x4=128, 31x4=124, 30x4=120. Relatively speaking, the values ​​are the same, but the last 2 bits are always zero. By setting the MSB value to true (most significant bit alignment), the video encoder performs video scaling. On the decoder side, the value can be divided by 4 to get it back to the original value. By performing video scaling, banding effects due to quantization occur (e.g. the change from 128 to 124 is a big enough difference to generate banding effects).

[0015] Instead of implementing video scaling, patch scaling can be performed: when the image is rasterized, it can be rasterized to floating point values, then multiplied by 4, the values ​​become 32x4=128, 31.5x4=126, 30.75x4=123, the values ​​are closer together, which makes the transitions smoother and reduces the banding effect.

[0016] Figure 2 shows a diagram of normal filtering according to some embodiments. When a depth image is decoded, points (e.g., diagram 200) are sampled. However, some distortion may occur after video compression (e.g., slightly different colors, as shown in figure 202). Since the points are sampling the surface of a triangle, they should be more or less around the surface, as shown in figure 206. Patch scaling, which multiplies floating point numbers, can cause small quantization errors that make the points slightly above or below the surface. In addition, there are video errors (e.g., 128 becomes 129). These errors are shown in figure 202, where some values ​​are above or below the proper value.

[0017] In the case of figure 200, the normals are determined by taking the cross product of the triangle's vectors (e.g., the vector from the top vertex point to the bottom left vertex point, and the vector from the top vertex point to the bottom right vertex point). In the case of figure 202, the normals are determined in the same way, but the points have moved and changed slightly, so the normals are subject to distortion of the points. Thus, the normals of the triangles in figure 202 are slightly different (e.g., in a different direction) than the normals of the triangles in figure 200.

[0018] Another way to calculate the normal is to perform a plane fitting as shown in Figure 204. Instead of using the points at the vertices of the triangle, all the points (in the triangle containing the vertex) are used to find a plane that intersects the points and minimizes the error between the points and the plane. The normal of the plane can then be calculated. Since more points are used, the resulting normal will usually be closer to the original normal.

[0019] If the normal calculated by plane fitting is close to the original normal, the coordinates of the vertex can be adjusted to be closer to the fitted normal. In some embodiments, the normal calculated by plane fitting is compared to the original normal, and if the calculated normal is within a threshold amount of the original normal, the normal calculated by plane fitting is used. To fix the position of the vertex, the coordinate position perpendicular to the projection plane (the tangent and bitangent coordinates of the normal coordinates are losslessly coded) is adjusted. The adjustment of the vertex by normal filtering can be performed simultaneously by considering all triangles connected to the vertex. The multidimensional problem becomes an optimization problem that can be solved with linear equations.

[0020] FIG. 3 illustrates a diagram of 1-ring, 2-ring, and 3-ring face neighborhoods for minimizing the total normal angle, according to some embodiments. Each set of triangles around a particular triangle is considered a ring. For example, for triangle 300, the first ring 302 of triangles is the 12 triangles surrounding triangle 300. The second ring 304 of triangles is the 24 triangles surrounding first ring 302. The third ring 306 of triangles is the 36 triangles surrounding second ring 304. The normals of each ring can be used to perform normal analysis to determine better, more reliable normal values. The normal values ​​determined using 1-ring, 2-ring, or 3-ring can be used to adjust the normal values ​​of a triangle (e.g., triangle 300) and / or the positions of the triangle vertices to generate normals that match the improved normal values.

[0021] FIG. 4 shows a flow chart of a method for mesh geometry coding according to some embodiments. In step 400, depth information is mapped to a luminance channel using less than all available bits. Mapping depth information to a luminance channel involves using M bits of the available N bits, where M is less than N (e.g., 6 bits of 8 bits, 8 bits of 10 bits, or 14 bits of 16 bits). The remaining NM bits (e.g., 8-6=2) are used for precision improvement (scaling factor). In step 402, a depth scaling factor is generated that is incorporated into a bilinear interpolation algorithm used during rasterization. Generating the depth scaling factor utilizes the remaining 2 bits of the available N bits (e.g., 8 bits). The remaining 2 bits allow multiplying the data by 4 so that the last 2 bits of the data are 0, and allows most significant bit alignment to be used. In some embodiments, the depth scaling factor is applied to a floating point value. In step 404, normal filtering is performed, which includes adjusting the positions of vertices according to normals estimated from surface pixels. Normal filtering involves performing a plane fitting using points within the triangle. Normal filtering involves using a 1-ring, 2-ring, or 3-ring neighborhood of faces to minimize the total normal angle. In some embodiments, fewer or additional steps are implemented. In some embodiments, the order of the steps is changed.

[0022] FIG. 5 illustrates a block diagram of an exemplary computing device configured to implement a mesh geometry coding method according to some embodiments. The computing device 500 can be used to acquire, store, compute, process, communicate and / or display information, such as images and videos, including 3D content. The computing device 500 can implement any of the encoding / decoding aspects. In general, a hardware structure suitable for implementing the computing device 500 includes a network interface 502, a memory 504, a processor 506, an I / O device(s) 508, a bus 510 and a storage device 512. The choice of processor is not critical as long as a suitable processor of sufficient speed is selected. The memory 504 can be any conventional computer memory known in the art. The storage device 512 can include a hard drive, a CDROM, a CDRW, a DVD, a DVDRW, a high definition disk / drive, an ultra high definition drive, a flash memory card, or any other storage device. The computing device 500 can include one or more network interfaces 502. Examples of network interfaces include a network card connected to an Ethernet or other type of LAN. The I / O device(s) 508 may include one or more of a keyboard, mouse, monitor, screen, printer, modem, touch screen, button interface, and other devices. The storage 512 and memory 504 likely store a mesh geometry coding application(s) 530 used to execute the mesh geometry coding implementation and are processed as an application would normally be processed. The computing device 500 may also include more or fewer components than those shown in FIG. 5. In some embodiments, mesh geometry coding hardware 520 is included.5 includes application 530 and hardware 520 for implementing mesh geometry coding, the mesh geometry coding method may also be implemented on the computing device in hardware, firmware, software, or any combination thereof. For example, in some embodiments, mesh geometry coding application 530 is programmed into memory and executed using a processor. As another example, in some embodiments, mesh geometry coding hardware 520 is programmed hardware logic that includes gates specifically designed to implement the mesh geometry coding method.

[0023] In some embodiments, the mesh geometry coding application(s) 530 include multiple applications and / or modules. In some embodiments, a module also includes one or more sub-modules. In some embodiments, fewer or additional modules may be included.

[0024] Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular / mobile phone, a smart appliance, a game console, a digital camera, a digital camcorder, a camera phone, a smartphone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer / player (e.g., DVD writer / player, high definition disc writer / player, ultra high definition disc writer / player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., a smart watch), a vehicle (e.g., a self-driving vehicle), or any other suitable computing device.

[0025] To utilize the mesh geometry coding method, a device acquires or receives 3D content (e.g., point cloud content). The mesh geometry coding method can be implemented with user assistance or automatically without user involvement.

[0026] In operation, the mesh geometry coding method allows for more efficient and accurate 3D content encoding compared to conventional implementations. By using the depth scaling factor in the encoder, the video image becomes smoother and easier to code. On the decoder side, when the video is reconstructed, discrepancies between normal values ​​obtained from surface pixels and normal values ​​obtained from a surface defined by only three vertices can be checked. Normal filtering allows the vertices to be repositioned to align normals and improve reconstruction of the mesh. The method described herein makes the image suitable for the codec.

[0027] Some embodiments of mesh geometry coding 1. A method of mesh geometry coding, comprising: mapping the depth information to a luminance channel using less than all available bits; generating a depth scaling factor that is incorporated into a bilinear interpolation algorithm used during rasterization; performing normal filtering, which includes adjusting the positions of the vertices according to normals estimated from the surface pixels; The method includes:

[0028] 2. The method of claim 1, wherein the step of mapping the depth information to the luminance channel includes using M bits out of the available N bits, where M is less than N.

[0029] 3. The method of claim 2, wherein the step of generating the depth scaling factor utilizes the remaining 2 bits of the available N bits.

[0030] 4. The method of claim 3, wherein the remaining two bits allow the data to be multiplied by four such that the last two bits of the data are 0, and most significant bit alignment is used.

[0031] 5. The method of claim 1, further comprising the step of applying the depth scaling factor to data, the data comprising floating point values.

[0032] 6. The method of claim 1, wherein the normal filtering includes performing plane fitting using points within a triangle.

[0033] 7. The method of claim 1, wherein the normal filtering includes using a 1-ring, 2-ring, or 3-ring face neighborhood to minimize the total normal angle.

[0034] 8. An apparatus comprising: A non-transitory memory for storing an application, the application comprising: Mapping the depth information to a luminance channel using less than all of the available bits; generating a depth scaling factor that is incorporated into a bilinear interpolation algorithm used during rasterization; performing normal filtering, which includes adjusting the positions of the vertices according to normals estimated from the surface pixels; a non-transient memory for performing a processor coupled to the memory and configured to process the application; An apparatus comprising:

[0035] 9. The apparatus of claim 8, wherein mapping the depth information to the luma channel includes using M bits of the available N bits, where M is less than N.

[0036] 10. The apparatus of claim 9, wherein generating the depth scaling factor utilizes the remaining 2 bits of the available N bits.

[0037] 11. The apparatus of claim 10, wherein the remaining two bits allow the data to be multiplied by four such that the last two bits of the data are 0, allowing most significant bit alignment to be used.

[0038] 12. The apparatus of claim 8, wherein the application is configured to apply the depth scaling factor to data, the data including floating point values.

[0039] 13. The apparatus of claim 8, wherein the normal filtering includes performing plane fitting using points within a triangle.

[0040] 14. The apparatus of claim 8, wherein the normal filtering includes using a one-ring, two-ring, or three-ring surface neighborhood to minimize the total normal angle.

[0041] 15. A system comprising: one or more cameras for acquiring three-dimensional content; An encoder for encoding the three-dimensional content, comprising: Mapping the depth information to a luminance channel using fewer than all of the available bits; generating a depth scaling factor that is incorporated into a bilinear interpolation algorithm used during rasterization; performing normal filtering, which includes adjusting the positions of the vertices according to normals estimated from the surface pixels; An encoder; A system including:

[0042] 16. The system of claim 15, wherein mapping the depth information to the luminance channel includes using M bits of the available N bits, where M is less than N.

[0043] 17. The system of claim 16, wherein generating the depth scaling factor utilizes the remaining 2 bits of the available N bits.

[0044] 18. The system of claim 17, wherein the remaining two bits allow the data to be multiplied by four such that the last two bits of the data are 0, allowing most significant bit alignment to be used.

[0045] 19. The system of claim 15, wherein the encoder is configured to apply the depth scaling factor to data, the data including floating point values.

[0046] 20. The system of claim 15, wherein the normal filtering includes performing plane fitting using points within a triangle.

[0047] 21. The system of claim 15, wherein the normal filtering includes using a 1-ring, 2-ring, or 3-ring surface neighborhood to minimize the total normal angle.

[0048] The present invention has been described with respect to specific embodiments containing details so as to facilitate an understanding of the principles of construction and operation of the invention. Reference herein to such specific embodiments and their details is not intended to limit the scope of the claims appended hereto. It will be readily apparent to those skilled in the art that various other modifications can be made in the embodiments selected for illustration without departing from the spirit and scope of the invention as defined by the claims. [Explanation of symbols]

[0049] 100 images 200 Figures 202 Figure 204 Figure 206 Figures 300 triangles 302 First Ring 304 Second Ring 306 The Third Ring 400 Maps depth information to the luma channel using fewer than all available bits 402 Generates a depth scaling factor that is incorporated into the bilinear interpolation algorithm used during rasterization. 404 Perform normal filtering 500 Computer equipment 502 Network Interface 504 Memory 506 Processor 508 I / O device 510 Bus 512 Storage device 520 Mesh Geometry Coding Hardware 530 Mesh Geometry Coding Applications

Claims

1. 1. A method of mesh geometry coding, comprising: mapping the depth information to a luminance channel using less than all available bits; generating a depth scaling factor that is incorporated into a bilinear interpolation algorithm used during rasterization; performing normal filtering, which includes adjusting the positions of the vertices according to normals estimated from the surface pixels; The method according to claim 1, further comprising:

2. 2. The method of claim 1, wherein the step of mapping the depth information to the luminance channel includes using M bits of available N bits, where M is less than N.

3. 3. The method of claim 2, wherein the step of generating the depth scaling factor utilizes the remaining 2 bits of the available N bits.

4. 4. The method of claim 3, wherein the remaining two bits allow the data to be multiplied by four such that the last two bits of the data are zero, allowing most significant bit alignment to be used.

5. 2. The method of claim 1, further comprising the step of applying the depth scaling factor to data, the data comprising floating point values.

6. The method of claim 1 , wherein the normal filtering comprises performing plane fitting using points within a triangle.

7. 2. The method of claim 1, wherein the normal filtering includes using 1-ring, 2-ring, or 3-ring face neighborhoods to minimize a total normal angle.

8. An apparatus comprising: A non-transitory memory for storing an application, the application comprising: Mapping the depth information to a luminance channel using less than all of the available bits; generating a depth scaling factor that is incorporated into a bilinear interpolation algorithm used during rasterization; performing normal filtering, which includes adjusting the positions of the vertices according to normals estimated from the surface pixels; a non-transient memory for performing a processor coupled to the memory and configured to process the application; An apparatus comprising:

9. 9. The apparatus of claim 8, wherein mapping the depth information to the luminance channel includes using M bits of available N bits, where M is less than N.

10. 10. The apparatus of claim 9, wherein generating the depth scaling factor utilizes the remaining 2 bits of the available N bits.

11. 11. The apparatus of claim 10, wherein the remaining two bits allow the data to be multiplied by four such that the last two bits of the data are zero, allowing most significant bit alignment to be used.

12. The apparatus of claim 8 , wherein the application is configured to apply the depth scaling factor to data, the data comprising floating point values.

13. The apparatus of claim 8 , wherein the normal filtering comprises performing plane fitting using points within a triangle.

14. 9. The apparatus of claim 8, wherein the normal filtering comprises using a one-ring, two-ring, or three-ring surface neighborhood to minimize a total normal angle.

15. 1. A system comprising: one or more cameras for acquiring three-dimensional content; An encoder for encoding the three-dimensional content, comprising: Mapping the depth information to a luminance channel using fewer than all of the available bits; generating a depth scaling factor that is incorporated into a bilinear interpolation algorithm used during rasterization; performing normal filtering, which includes adjusting the positions of the vertices according to normals estimated from the surface pixels; An encoder; A system comprising:

16. 16. The system of claim 15, wherein mapping the depth information to the luminance channel includes using M bits of available N bits, where M is less than N.

17. 17. The system of claim 16, wherein generating the depth scaling factor utilizes the remaining 2 bits of the available N bits.

18. 20. The system of claim 17, wherein the remaining two bits allow the data to be multiplied by four such that the last two bits of the data are zero, allowing most significant bit alignment to be used.

19. 16. The system of claim 15, wherein the encoder is configured to apply the depth scaling factor to data, the data comprising floating point values.

20. 16. The system of claim 15, wherein the normal filtering comprises performing plane fitting using points within a triangle.

21. 16. The system of claim 15, wherein the normal filtering includes using a 1-ring, 2-ring, or 3-ring face neighborhood to minimize a total normal angle.

Images