Inter-frame attribute coding in geometry-based dynamic point cloud compression

Abstract

This specification describes attribute prediction and compensation schemes for geometry-based dynamic point cloud compression. A combination of multiple reference frames can be used as predictors for the current frame. The methods described herein improve efficiency and accuracy.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 496,969, filed on April 19, 2023, entitled "INTER - FRAME ATTRIBUTE CODING IN GEOMETRY - BASED DYNAMIC POINT CLOUDS COMPRESSION", which is hereby incorporated by reference in its entirety for all purposes.

[0002] The present invention relates to 3D graphics. More specifically, the present invention relates to dynamic mesh coding.

Background Art

[0003] Geometry - based Point Cloud Compression (G - PCC) has been developed under the Collaborative Test Model (TMC13) research. G - PCC supports the compression of both Category 1 (static point clouds) and Category 3 (dynamically acquired point clouds). Motion - compensated inter - frame prediction is introduced into the Region - Adaptive Hierarchical Transform (RAHT) coding scheme.

[0004] In MPEG142, in G - PCCv2 and TMC13, inter - prediction for RAHT attribute coding is adopted. In this method, in the coding of the target node, the coefficient residual value is calculated using the coefficients of the reference frame and the current frame, and the residual value is coded. Inter - prediction is applied only when the current point cloud and the reference point cloud have the same position nodes. If there is no predicted coefficient value in the reference buffer, inter - prediction is not applied.

[0005] In MPEG, a method based on RAHT is being discussed. This is limited to RAHT and is performed in the transformed region. In summary, it encodes the residual of the transformation coefficients between each node of the reference frame and the current frame. The limitations are as follows: (1) It applies only to RAHT attribute coding. (2) It is limited to the transformed coefficient region. (3) Geometry slicing used before attribute transformation may cause unexpected misalignment between nodes, resulting in residuals that are not suitable for compression. [Overview of the project] [Problems that the invention aims to solve]

[0006] This specification describes attribute prediction and compensation schemes for geometry-based dynamic point cloud compression. A combination of multiple reference frames can be used as predictors for the current frame. The methods described herein improve efficiency and accuracy. [Means for solving the problem]

[0007] In one embodiment, a method programmed in the device's non-temporary memory includes the steps of: encoding and decoding a first frame to generate a reconstructed first frame; and encoding a second frame based on the reconstructed first frame. The reconstructed first frame includes reconstructed attribute information and reconstructed location information. The method includes the step of predicting the attribute information of the second frame from the reconstructed first frame. The method includes the step of generating a predictor which includes the same location information as the second frame and the same number of points as the second frame, the attribute information which is based on a color transfer performed from the reconstructed first frame to the location of the second frame. The method includes the step of calculating a residual by subtracting the attribute information of the predictor from the attribute information of the second frame. The method includes the step of encoding and decoding the residual to generate a reconstructed residual which includes the attribute information and the location information. The method includes the step of generating a reconstructed predictor which includes a color transfer from the first frame to the reconstructed predictor. The method includes the step of adding the reconstructed residual to the reconstructed predictor to generate new attribute information and new location information.

[0008] In another embodiment, the device includes non-temporary memory for storing an application, the application for encoding and decoding a first frame to generate a reconstructed first frame, and encoding a second frame based on the reconstructed first frame; and a processor coupled to the memory and configured to process the application. The reconstructed first frame includes reconstructed attribute information and reconstructed location information. The application is further configured to predict the attribute information of the second frame from the reconstructed first frame. The application is further configured to generate a predictor which includes the same location information as the second frame and the same number of points as the second frame, the attribute information which is based on a color transfer performed from the reconstructed first frame to the location of the second frame. The application is further configured to calculate a residual by subtracting the attribute information of the predictor from the attribute information of the second frame. The application is further configured to encode and decode the residual to generate a reconstructed residual which includes the attribute information and the location information. The application is further configured to generate a reconstructed predictor, to which the color from the first frame is transferred. The application is further configured to add the reconstructed residual to the reconstructed predictor to generate new attribute information and new location information.

[0009] In another embodiment, a method programmed into the device's non-temporary memory includes the step of encoding the current frame based on one or more previous frames or one or more future frames, wherein the attribute information of the current frame is based on the attribute information of the one or more previous frames or one or more future frames. The attribute information is based on color transfers performed from the one or more previous frames or one or more future frames. The method includes the step of calculating a residual by subtracting the attribute information of a predictor from the attribute information of the current frame. The method includes the step of encoding and decoding the residual to generate a reconstructed residual including the attribute information and location information. The method includes the step of generating a reconstructed predictor, wherein the colors from the one or more previous frames or one or more future frames are transferred to the reconstructed predictor. The method includes the step of adding the reconstructed residual to the reconstructed predictor to generate new attribute information and new location information. [Brief explanation of the drawing]

[0010] [Figure 1] This is an illustrative diagram of inter-frame attribute prediction in geometry-based point cloud compression (G-PCC) according to several embodiments. [Figure 2] This is a diagram illustrating an exemplary prediction method according to several embodiments. [Figure 3] This is a flowchart of a method for inter-frame attribute coding according to several embodiments. [Figure 4] This is a block diagram of an exemplary computer device configured to implement an interframe attribute coding method according to several embodiments. [Modes for carrying out the invention]

[0011] This specification describes attribute prediction and compensation methods for geometry-based dynamic point cloud compression. While a combination of multiple reference frames can be used as predictors for the current frame, for simplicity, the method is demonstrated using an example with a single reference frame.

[0012] On the encoder side, we consider the reference frame F0 and the current frame F1. First, the reference frame F0 is intra-encoded using an arbitrary attribute coding scheme such as G-PCC's RAHT or predictive lifting transform, and sent to the decoder. The reference frame F0 is reconstructed on the encoder side to obtain F'0. The reconstructed attributes of F'0 are transferred to the geometry of the current frame F1 to generate the predictor P1. By subtracting the attributes of P1 from the attributes of F1 at the same coordinate positions, the residual point cloud R1 is generated. The residual R1 is also encoded and sent to the decoder. The decoder decodes F'0 and R'1. The attributes attached to F'0 are transferred to the geometry of R'1 to generate P'1. The attributes of R'1 and P'1 at the same coordinate positions are added together to obtain the decoded current frame F'1.

[0013] Prediction schemes can use groups of frames containing varying numbers of frames, and one or more past and future frames. Different global and local point cloud matching strategies can be used to improve attribute transfer, optimize predictor generation, and thus provide residuals that are more suitable for compression.

[0014] This method enables prediction-based inter-frame attribute coding in geometry-based dynamic point cloud compression. Attribute prediction is performed in the attribute domain and can be used as input to arbitrary attribute transformations such as G-PCC's RAHT and predictive lifting transformations. The predictor is based on the entire point cloud frame and is not affected by positional shifts caused by point cloud slicing, which is implemented by some codecs such as G-PCC.

[0015] FIG. 1 shows an exemplary diagram of inter-frame attribute prediction in geometry-based point cloud compression (G-PCC) according to some embodiments. Frame 0 (F0) and Frame 1 (F1) (e.g., two consecutive frames) are shown. Frame 0 includes the attribute information of Frame 0, the position information of Frame 0, and N0 points. Using the information from Frame 0, a prediction of Frame 1 can be performed, and only the residuals between the two frames can be encoded.

[0016] Encode and decode Frame 0 to generate a reconstructed Frame 0 (F’0). The reconstructed Frame 0 has reconstructed attribute information (a’0), reconstructed position information (p’0), and M0 points (M0 < N0).

[0017] After encoding and reconstructing Frame 0, encode Frame 1. Frame 1 has different attribute information (a1), position information (p1), and a different number of points (N1). Use the reconstructed Frame 0 to predict the attribute information of Frame 1. The predictor (P1) has the same position information (p1) as Frame 1 and the same number of points (N1) as Frame 1, but for the attribute information (a”0), color transfer is performed from the reconstructed Frame 0 to the position of Frame 1. In other words, Frame 1 is recolored based on the color of the reconstructed Frame 0. Calculate the residual (R1) by subtracting the attribute information (a”0) of the predictor from the attribute information (a1) of Frame 1 (e.g., a1 - a”0). For the residual, the position information (p1) and the number of points (N1) remain the same. Next, encode and decode the residual (R1) to generate a reconstructed residual (R’1) that includes attribute information ([a1 - a”0]’), position information (p’1), and M1 points (M1 < N1).

[0018] Next, the color from frame 0 is transferred to the estimated P1 (also called the reconstructed predictor (P'1)). The reconstructed residual (R'1) is added to the reconstructed predictor (P'1) to generate attribute information ([a1-a”0]'+a”'0) and location information (p'1). This is an approximation of the encoded and decoded frame 1 (F'1), including the attribute information a'1 and location information p'1.

[0019] In other words, the color of the reconstructed frame 0 is transferred to frame 1, then the residual is determined and encoded on behalf of frame 1. Once the residual is recovered, the color is transferred from the reconstructed frame 0 to the residual geometry to reconstruct and recover the point cloud (frame 1).

[0020] Any recoloring method can be used. For example, nearest neighbor coloring can be implemented, and the nearest neighbor's color is used in the target point cloud and used as a predictor. G-PCC includes a recoloring implementation that includes neighborhood analysis / exploration and additional information for performing recoloring. Colors can be obtained from previous point clouds using other nearest neighbor analysis / exploration and motion estimation.

[0021] Figure 2 shows diagrams of exemplary prediction schemes according to several embodiments. For the current frame, predictors can be generated using any past or future frames. For example, instead of using one frame to generate a predictor, M past frames, N future frames, or any combination thereof can be used. Taking this example further, the attributes of the predictor can be based on average attributes based on past and future frames.

[0022] Figure 3 shows flowcharts of interframe attribute coding methods according to several embodiments. In step 300, the first frame is encoded and decoded to generate a reconstructed frame. The reconstructed first frame has reconstructed attribute information and reconstructed position information.

[0023] In step 302, the second frame is encoded. Using the reconstructed first frame, the attribute information of the second frame is predicted. The predictor has the same position information as the second frame and the same number of points as the second frame, but for the attribute information, color transfer is performed from the reconstructed first frame to the position of the second frame. A residual is calculated by subtracting the attribute information of the predictor from the attribute information of the second frame. For the residual, the position information and the number of points remain the same. Next, the residual is encoded and decoded to generate a reconstructed residual including the attribute information and the position information. Next, the color from the first frame is transferred to the reconstructed predictor. The reconstructed residual is added to the reconstructed predictor to generate the attribute information and the position information. This is an approximation of the encoded and decoded second frame including the attribute information and the position information. This process is continued for additional frames (e.g., the third frame, the fourth frame, etc.). In some embodiments, instead of using the first frame for recoloring and prediction of the second frame, other frames (e.g., the previous frame, the future frame, or a combination thereof) are utilized. In some embodiments, additional steps or fewer steps are implemented. In some embodiments, the order of the steps is changed.

[0024] Figure 4 shows a block diagram of an exemplary computer device configured to implement an interframe attribute coding method according to several embodiments. The computer device 400 can be used to acquire, store, compute, process, communicate, and / or display information such as images and videos, including 3D content. The computer device 400 can implement any form of encoding / decoding. Generally, a suitable hardware structure for implementing the computer device 400 includes a network interface 402, memory 404, a processor 406, (one or multiple) I / O devices 408, a bus 410, and a storage device 412. The choice of processor is not critical as long as a suitable processor with sufficient speed is selected. The memory 404 can be any conventional computer memory known in the art. The storage device 412 can include a hard drive, CD-ROM, CDRW, DVD, DVDRW, high-definition disk / drive, ultra-high-definition drive, flash memory card, or any other storage device. The computer device 400 can include one or more network interfaces 402. An example of a network interface includes a network card connected to Ethernet or another type of LAN. The (single or multiple) I / O devices 408 may include one or more of the following: keyboard, mouse, monitor, screen, printer, modem, touchscreen, button interface, and other devices. The (single or multiple) interframe attribute coding application 430 used to implement the interframe attribute coding method is likely to be stored in the storage device 412 and memory 404 and processed as the application normally would. The computer device 400 may include more or fewer components than those shown in Figure 4. In some embodiments, interframe attribute coding hardware 420 is included. The computer device 400 in Figure 4 includes the application 430 and hardware 420 for the interframe attribute coding method, but the interframe attribute coding method may be implemented in the computer device as hardware, firmware, software, or any combination thereof.For example, in some embodiments, the inter-frame attribute coding application 430 is programmed in memory and executed using a processor. In another example, in some embodiments, the inter-frame attribute coding hardware 420 is programmed hardware logic that includes gates specifically designed to implement an inter-frame attribute coding method.

[0025] In some embodiments, the inter-frame attribute coding application 430 (singular or plural) includes several applications and / or modules. In some embodiments, a module also includes one or two or more sub-modules. In some embodiments, fewer or additional modules can be included.

[0026] Examples of suitable computer devices include personal computers, laptop computers, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants, cellular phones / mobile phones, smart home appliances, game consoles, digital cameras, digital camcorders, camera phones, smartphones, portable music players, tablet computers, mobile devices, video players, video disc recorders / players (e.g., DVD recorders / players, high-definition disc recorders / players, ultra-high-definition disc recorders / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smartwatches), vehicles (e.g., autonomous vehicles), or any other suitable computer device.

[0027] To utilize the inter-frame attribute coding method, the device acquires or receives 3D content (e.g., point cloud content). The inter-frame attribute coding method can be implemented automatically with or without user assistance.

[0028] During operation, the inter-frame attribute coding method improves efficiency by coding residuals instead of frames, and is more accurate compared to other efficient implementations. Furthermore, the inter-frame attribute coding method does not operate in the transformation region, thus avoiding other potential problems.

[0029] When G-PCC coding is performed, a large number of points exist, and the codec slices the point cloud into subpoints. Each slice has a certain number of subpoints. An optree is constructed independently for each slice. If there are two consecutive frames, and an optree is constructed for the slice of the reference point cloud and an optree is constructed for the slice of the current frame, the two slices may not correspond spatially. Slice 0 from frame 0 does not necessarily correspond to slice 0 from frame 1. Therefore, the optree may have no correspondence at all, and thus attribute transformations to these two structures may have no predictive power. This leads to poor predictions with very high residuals. However, by recoloring the point cloud before any transformations, optree construction, and slicing, the colors move to the appropriate positions.

[0030] Several embodiments of inter-frame attribute coding in geometry-based dynamic point cloud compression 1. A method programmed into the non-temporary memory of a device, The steps include encoding and decoding the first frame to generate a reconstructed first frame, The steps include encoding a second frame based on the reconstructed first frame, A method that includes this.

[0031] 2. The method according to paragraph 1, wherein the reconstructed first frame includes reconstructed attribute information and reconstructed location information.

[0032] 3. The method according to the first paragraph, further comprising the step of predicting attribute information of a second frame from the reconstructed first frame.

[0033] 4. Further comprising the step of generating a predictor that includes the same position information as the second frame and the same number of points as the second frame, Attribute information is based on color transfer performed from the reconstructed first frame to the second frame, The method described in paragraph 1.

[0034] 5. The method of paragraph 4, further comprising the step of calculating a residual by subtracting the attribute information of the predictor from the attribute information of the second frame.

[0035] 6. The method of paragraph 5, further comprising the step of encoding and decoding the residuals to generate reconstructed residuals including the attribute information and the location information.

[0036] 7. Further includes the step of generating a reconstructed predictor, The color from the first frame is transferred to the reconstructed predictor. The method described in paragraph 6.

[0037] 8. The method according to paragraph 7, further comprising the step of adding the reconstructed residual to the reconstructed predictor to generate new attribute information and new location information.

[0038] 9. A device, Non-temporary memory for storing applications, wherein the applications are The first frame is encoded and decoded to generate a reconstructed first frame. Based on the reconstructed first frame, encode the second frame. Non-temporary memory is intended for this purpose, A processor coupled to the memory and configured to process the application, A device that includes this.

[0039] 10. The apparatus according to paragraph 9, wherein the reconstructed first frame includes reconstructed attribute information and reconstructed location information.

[0040] 11. The apparatus according to paragraph 9, wherein the application is further configured to predict attribute information of a second frame from the reconstructed first frame.

[0041] 12. The application is further configured to generate a predictor that includes the same position information as the second frame and the same number of points as the second frame, and the attribute information is based on a color transfer performed from the reconstructed first frame to the position of the second frame, as described in paragraph 9.

[0042] 13. The apparatus according to paragraph 12, wherein the application is further configured to calculate a residual by subtracting the attribute information of the predictor from the attribute information of the second frame.

[0043] 14. The apparatus according to paragraph 13, wherein the application is further configured to encode and decode the residuals to generate reconstructed residuals including the attribute information and the location information.

[0044] 15. The apparatus according to paragraph 14, wherein the application is further configured to generate a reconstructed predictor, and the color from the first frame is transferred to the reconstructed predictor.

[0045] 16. The apparatus according to paragraph 15, wherein the application is further configured to add the reconstructed residual to the reconstructed predictor to generate new attribute information and new location information.

[0046] 17. A method programmed into the non-temporary memory of a device, The process includes the step of encoding the current frame based on one or more previous frames or one or more future frames, The attribute information of the current frame is based on the attribute information of one or more previous frames or one or more future frames. method.

[0047] 18. The method described in paragraph 17, wherein the attribute information is based on a color transfer performed from one or more previous frames or one or more future frames.

[0048] 19. The method of paragraph 17, further comprising the step of calculating a residual by subtracting the attribute information of a predictor from the attribute information of the current frame.

[0049] 20. The method according to paragraph 19, further comprising the step of encoding and decoding the residuals to generate reconstructed residuals including attribute information and location information.

[0050] 21. Further includes the step of generating a reconstructed predictor, The colors from one or more previous frames or one or more future frames are transferred to the reconstructed predictor. The method described in paragraph 20.

[0051] 22. The method according to paragraph 21, further comprising the step of adding the reconstructed residual to the reconstructed predictor to generate new attribute information and new location information.

[0052] To facilitate understanding of the structure and operating principles of the present invention, the invention has been described in relation to specific embodiments incorporating detailed features. Such references to specific embodiments and their details in this specification are not intended to limit the claims appended herein. It will be readily apparent to those skilled in the art that various other modifications can be made to the embodiments selected for illustrative purposes without departing from the spirit and scope of the invention as defined by the claims. [Explanation of Symbols]

[0053] 300 Encode the first frame 302 Encode the second frame 400 Computer devices 402 Network Interface 404 memory 406 Processors 408 I / O devices 410 Bus 412 Storage device 420 Inter-frame attribute coding hardware 430 Inter-frame attribute coding application

Claims

1. A method programmed into the device's non-temporary memory, The steps include encoding and decoding the first frame to generate a reconstructed first frame, The steps include encoding a second frame based on the reconstructed first frame, A method characterized by including the following.

2. The method according to claim 1, characterized in that the reconstructed first frame includes reconstructed attribute information and reconstructed location information.

3. The method according to claim 1, further comprising the step of predicting attribute information of the second frame from the reconstructed first frame.

4. The process further includes the step of generating a predictor that includes the same position information as the second frame and the same number of points as the second frame, The attribute information is based on a color transfer performed from the reconstructed first frame to the second frame. The method according to claim 1, characterized in that

5. The method according to claim 4, further comprising the step of calculating a residual by subtracting the attribute information of the predictor from the attribute information of the second frame.

6. The method according to claim 5, further comprising the step of encoding and decoding the residuals to generate reconstructed residuals including the attribute information and the location information.

7. The process further includes the step of generating a reconstructed predictor, The color from the first frame is transferred to the reconstructed predictor. The method according to claim 6, characterized in that

8. The method according to claim 7, further comprising the step of adding the reconstructed residual to the reconstructed predictor to generate new attribute information and new location information.

9. It is a device, Non-temporary memory for storing applications, wherein the applications are The first frame is encoded and decoded to generate a reconstructed first frame. Based on the reconstructed first frame, the second frame is encoded. Non-temporary memory is intended for this purpose, A processor coupled to the memory and configured to process the application, An apparatus characterized by including

10. The apparatus according to claim 9, characterized in that the reconstructed first frame includes reconstructed attribute information and reconstructed position information.

11. The apparatus according to claim 9, wherein the application is further configured to predict attribute information of the second frame from the reconstructed first frame.

12. The apparatus according to claim 9, wherein the application is further configured to generate a predictor that includes the same position information as the second frame and the same number of points as the second frame, and the attribute information is based on a color transfer performed from the reconstructed first frame to the position of the second frame.

13. The apparatus according to claim 12, wherein the application is further configured to calculate a residual by subtracting the attribute information of the predictor from the attribute information of the second frame.

14. The apparatus according to claim 13, wherein the application is further configured to encode and decode the residuals to generate reconstructed residuals including the attribute information and the location information.

15. The apparatus according to claim 14, wherein the application is further configured to generate a reconstructed predictor, and the color from the first frame is transferred to the reconstructed predictor.

16. The apparatus according to claim 15, wherein the application is further configured to add the reconstructed residual to the reconstructed predictor to generate new attribute information and new location information.

17. A method programmed into the device's non-temporary memory, The process includes the step of encoding the current frame based on one or more previous frames or one or more future frames, The attribute information of the current frame is based on the attribute information of one or more previous frames or one or more future frames. A method characterized by the following:

18. The method according to claim 17, characterized in that the attribute information is based on a color transfer performed from one or more previous frames or one or more future frames.

19. The method according to claim 17, further comprising the step of calculating a residual by subtracting the attribute information of a predictor from the attribute information of the current frame.

20. The method according to claim 19, further comprising the step of encoding and decoding the residuals to generate reconstructed residuals including attribute information and location information.

21. The process further includes the step of generating a reconstructed predictor, The colors from one or more previous frames or one or more future frames are transferred to the reconstructed predictor. The method according to claim 20, characterized in that

22. The method according to claim 21, further comprising the step of adding the reconstructed residual to the reconstructed predictor to generate new attribute information and new location information.

Images