Displacement packing using a single LoD per block

Abstract

A single level of detail (LoD) displacement packing method per block enables the implementation of slice decoding, scalability, and other video processing. By using a per-block LoD packing implementation, the sequence displacement bitrate changes minimally.

Description

Technical Field

[0001] 〔Cross - Reference to Related Applications〕 This application claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application No. 63 / 496,968, entitled “DISPLACEMENT PACKING USING SINGLE LoD PER BLOCK,” filed on April 19, 2023, which is hereby incorporated by reference in its entirety for all purposes.

[0002] The present invention relates to 3D graphics. More specifically, the present invention relates to dynamic mesh coding.

Background Art

[0003] In recent years, new methods for compressing volumetric content such as point clouds based on 3D - to - 2D projections are being standardized. This method, also known as V3C (Visual Volume Video - based Compression), maps 3D volume data into several 2D patches, further arranges the patches in an atlas image, and then encodes them with a video encoder. The atlas image corresponds to the geometry of the points, their respective textures, and an occupancy map indicating which positions should be considered for point cloud reconstruction.

[0004] In 2017, MPEG issued a Call for Proposals (CfP) for point cloud compression. After evaluating several proposals, MPEG is currently considering two different techniques for point cloud compression: 3D native coding techniques (based on octree and similar coding methods), or conventional video coding after 3D-to-2D projection. For dynamic 3D scenes, MPEG uses Test Model Software (TMC2) based on patch surface modeling, projection of patches from 3D to 2D images, and coding of the 2D images by a video encoder such as HEVC. This method has proven to be more efficient than native 3D coding and can achieve competitive bitrates with acceptable quality.

[0005] Since coding 3D point clouds using projection-based methods (also known as video-based methods or V-PCC) has been successful, future versions of this standard are expected to include further 3D data, such as 3D meshes. However, the current version of this standard is only suitable for transmitting sets of unconnected points and lacks a mechanism for transmitting point connectivity, as required for 3D mesh compression.

[0006] Methods have also been proposed to extend the functionality of V-PCC to meshes. One possible method is to encode vertices using V-PCC and then encode connectivity using a mesh compression method such as TFAN or Edgebreaker. A limitation of this method is that the original mesh must be dense so that the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection. Furthermore, since the order of vertices affects the coding of connectivity, different methods have been proposed for reorganizing the connectivity of the mesh. An alternative method for encoding sparse meshes is to encode the positions of 3D vertices using RAW patch data. Since RAW patches directly encode (x,y,z), in this method all vertices are encoded as RAW data, while connectivity is encoded by a similar mesh compression method as described above. Note that in RAW patches, vertices can be sent in any preferred order, so the order generated from connectivity coding can be used. While this method can encode sparse point clouds, RAW patches are not efficient for encoding 3D data, and further data such as triangular face attributes may be missing from this method.

[0007] Microsoft's UVAtlas is a state-of-the-art automated texture map generator, but it takes a considerable amount of time and only optimizes for local frames. V-PCC generates patches using orthographic projection, but it only targets point clouds and does not support mesh patch generation. [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] A single level of detail (LoD) displacement packing method per block enables the implementation of slice decoding, scalability, and other video processing. By using a per-block LoD packing implementation, the sequence displacement bitrate changes minimally. [Means for solving the problem]

[0009] In one embodiment, a method programmed into the device's non-temporary memory includes the steps of: packing displacement information of a single level of detail (LoD) into one or more blocks; and packing filler bits up to the end of the current block among the one or more blocks. The method further includes the steps of: packing the displacement information of the single LoD into one or more blocks; and packing the filler bits up to the end of the current block among the one or more blocks; for subsequent LoDs until all the displacement information is packed. The method further includes the step of receiving position count information indicating the start and end positions of each LoD. Each subsequent LoD starts from the beginning of the block. The method further includes the step of triggering block-by-block LoD displacement packing using a flag. The method further includes the step of dividing the mesh information into different LoDs. The method further includes the step of performing slice decoding. The method further includes the step of decoding the displacement information.

[0010] In another embodiment, the device includes non-temporary memory for storing an application, the application being for packing displacement information of a single level of detail (LoD) into one or more blocks and packing filler bits up to the end of the current block among the one or more blocks; and a processor coupled to the memory and configured to process the application. The application is further configured to repeat packing the displacement information of the single LoD into the one or more blocks and packing the filler bits up to the end of the current block among the one or more blocks for subsequent LoDs until all the displacement information is packed. The application is further configured to receive position count information indicating the start and end positions of each LoD. Each subsequent LoD starts from the beginning of a block. The application is further configured to trigger block-by-block LoD displacement packing using a flag. The application is further configured to divide mesh information into different LoDs. The application is further configured to perform slice decoding. The application is further configured to decode the displacement information.

[0011] In another embodiment, the system includes an encoder configured to encode a 3D mesh, and a decoder configured to pack displacement information of a single level of detail (LoD) into one or more blocks and pack filler bits up to the end of the current block among the one or more blocks. The decoder is further configured to repeat packing the displacement information of the single LoD into the one or more blocks and packing the filler bits up to the end of the current block among the one or more blocks for subsequent LoDs until all the displacement information is packed. The decoder is further configured to receive position count information indicating the start and end positions of each LoD. Each subsequent LoD starts from the beginning of a block. The decoder is further configured to trigger block-by-block LoD displacement packing using a flag. The decoder is further configured to divide the mesh information into different LoDs. The decoder is further configured to perform slice decoding. The decoder is further configured to decode the displacement information. [Brief explanation of the drawing]

[0012] [Figure 1] This is a diagram of a V-DMC decoder according to several embodiments. [Figure 2] This diagram shows a conventional packing implementation and a block-by-block LoD packing implementation according to several embodiments. [Figure 3] This figure shows images of displacement packing according to several embodiments. [Figure 4] This is a flowchart of a method for single-lodge displacement packing per block according to several embodiments. [Figure 5] This is a block diagram of an exemplary computer device configured to implement a single LoD displacement packing method per block according to several embodiments. [Modes for carrying out the invention]

[0013] A mesh typically consists of a set of polygons representing the surface of a volume. An efficient method for encoding a dynamic mesh is to simplify the mesh to a base mesh, encode the base mesh with any mesh encoder, and then encode the displacement information (i.e., the difference between the subdivided base mesh and the original input mesh). Displacement information is provided per vertex and can be efficiently encoded in video. The displacement information is packed into video data by level of detail (LoD) in raster scan order. This prevents the use of video tools that allow for individual coding of blocks (such as slices), as a single block may contain two or more LoD data points together.

[0014] This specification describes a method for packing displacement information. First, the mesh information is divided into detail levels, and then each detail level of data is packed into separate blocks. This allows for slice decoding of separate LoDs and also enables scalability on the decoder side. Furthermore, a high-level syntax signal is provided that enables block packing of different LoDs. For example, the base mesh is level 0, each edge is divided in half, all vertices of the half edge are level 1, and as the edge is divided in half, a new level (e.g., level 2, 3, etc.) is generated. Other tricks can be performed based on different LoDs. If a certain LoD does not fill a block, the remaining bits of the block can be filled with zeros or another filler, and the next LoD starts from the beginning of the next block.

[0015] Figure 1 shows diagrams of V-DMC decoders according to several embodiments. The V-DMC decoder 100 includes many standard components. Displacement packing with a single LoD per block is implemented using the image unpacking component 102. After the subdivision component, positionCount1 and LoDs are generated and passed to the image unpacking component 102. The start and end positions of each LoD can be determined using positionCount1.

[0016] Figure 2 shows a conventional packing implementation and diagrams of block-by-block LoD packing implementations according to several embodiments. The displacement packing stream 200 indicates that each LoD is packed immediately after the others (e.g., LoD-0 is immediately followed by LoD-1). The displacement packing stream 202 indicates filler gaps between each LoD (e.g., LoD-0 is followed by zero, and then LoD-1).

[0017] Figure 202 illustrates a conventional packing implementation where each LoD is packed immediately after another without gaps, allowing different LoDs to exist in the same block. For example, blocks 204, 206, and 208 show that there are two different LoDs in the same block. Taking this example further, block 204 contains LoD-0 and LoD-1 in the same block, block 206 contains LoD-1 and LoD-2 in the same block, and block 208 contains LoD-2 and LoD-3 in the same block. Different LoDs have distinct shading / hatching / patterns to distinguish each other. For example, in the figure, LoD-0 has a cross-hatch pattern, and LoD-1 has lines running from the bottom left to the top right, and so on.

[0018] Figure 252 illustrates a block-by-block LoD packing implementation where LoDs cannot occupy the same block. For example, a filler bit 260 (e.g., a bit equal to 0) is used to reserve space to prevent multiple LoDs from occupying the same block. Continuing this example, LoD-0 ends in block 254, so filler bit 260 is used to fill block 254. Next, LoD-1 can be packed into the next block. LoD-1 ends in block 256, so filler bit 260 fills the rest of block 256. Then LoD-2 starts in the next block and ends in block 258, where block 258 is filled with filler bit 260. Then LoD-3 starts in the next block and continues until it ends, filling the remaining blocks with filler bit 260. While video images have many more "edges," having only one LoD per block allows for the implementation of slice decoding and other video processing techniques.

[0019] Figure 3 shows images of displacement packing according to several embodiments. Image 300 shows a video image with a conventional packing implementation. Image 350 shows a video image with a block-by-block LoD packing implementation. A gap 352 is shown where filler bits are used to ensure that no two different LoDs are present in the same block, as described herein. By using a block-by-block LoD packing implementation, the sequence displacement bitrate changes minimally. However, slice decoding is now possible, whereas conventional packing implementations could not. For example, if we want to decode the first level, we do not need to decode the entire video, but only the first two blocks, because the first level is contained only in the first two blocks.

[0020] An example syntax for LoD packing is as follows: JPEG2026511920000002.jpg248161 JPEG2026511920000003.jpg246170 JPEG2026511920000004.jpg248161 JPEG2026511920000005.jpg249161 JPEG2026511920000006.jpg144170

[0021] Atlas sequence parameter set extension RBSP syntax JPEG2026511920000007.jpg136125

[0022] As shown, the flag (asps_vmc_ext_lod_block_packing) can be set for LoD block packing. When asps_vmc_ext_lod_block_packing is equal to 1, it indicates that the displacement components are packed for each level of detail with respect to the block boundaries. When asps_vmc_ext_duplicate_vertex_removal_flag is equal to 0, it indicates that the displacements at all levels of detail are packed together.

[0023] Next, with the parameters of width, height, bitDepth, dispQuantCoeffFrame, blockSize, positionCount1, inverseDisplacementPacking, lodBlockPacking, subdivisionIterationCount, and levelOfDetailPositionCounts as inputs and the parameters of dispQuantCoeffArray as outputs, the inverse packing process is called. JPEG2026511920000008.jpg249159 JPEG2026511920000009.jpg163170

[0024] Figure 4 shows flowcharts of methods for single LoD displacement packing per block according to several embodiments. In step 400, position count information and LoD displacement information are received. For example, the position count information indicates the start and end positions of each LoD. In step 402, the displacement information of a single LoD (e.g., LoD-0) is packed (e.g., into a video bitstream). If the displacement information of a single LoD contains more bits than a single block in the bitstream, subsequent blocks are filled. When the end of the displacement information of a single LoD is reached, in step 404, filler bits (e.g., zeros) are packed until the current block is filled. If the displacement information of an LoD ends at the end of a block, no filler bits are used for that block. This process is repeated until the displacement information of each LoD is packed. Each LoD starts from the beginning of the block. For example, the displacement information for LoD-0 is packed, followed by the filler bits, then the displacement information for LoD-1 is packed, followed by the filler bits, and so on, until the last LoD (e.g., LoD-n) is packed. In a further example, the displacement information for LoD-0 fills a first block, then fills part of a second block, so the rest of the second block is filled with filler bits. Then the displacement information for LoD-1 starts from the beginning of a third block and continues as described above. Two LoDs do not exist in the same block, and a single block has only one LoD and possibly filler bits. In some embodiments, fewer or additional steps are implemented. For example, a step is performed to instruct and / or trigger block-by-block LoD displacement packing based on a flag. In another example, it may be implemented to split the mesh information into different LoDs. In yet another example, block-by-block LoD displacement packing enables further video processing such as slice decoding and scalability. In some embodiments, decoding the information is implemented. In some embodiments, the order of the steps is changed.

[0025] Figure 5 shows a block diagram of an exemplary computer device configured to implement a single LoD displacement packing method per block according to several embodiments. The computer device 500 can be used to acquire, store, compute, process, communicate, and / or display information such as images and videos, including 3D content. The computer device 500 can implement any form of encoding / decoding. Generally, a suitable hardware structure for implementing the computer device 500 includes a network interface 502, memory 504, a processor 506, (one or multiple) I / O devices 508, a bus 510, and a storage device 512. The choice of processor is not critical as long as a suitable processor with sufficient speed is selected. The memory 504 can be any conventional computer memory known in the art. The storage device 512 can include a hard drive, CD-ROM, CDRW, DVD, DVDRW, high-definition disk / drive, ultra-high-definition drive, flash memory card, or any other storage device. The computer device 500 can include one or more network interfaces 502. An example of a network interface includes a network card connected to Ethernet or another type of LAN. The (single or multiple) I / O devices 508 may include one or more of the following: keyboard, mouse, monitor, screen, printer, modem, touchscreen, button interface, and other devices. The (single or multiple) single-LoD displacement packing application 530 per block used to implement a single-LoD displacement packing method per block is likely to be stored in storage device 512 and memory 504 and processed as the application normally would. The computer device 500 may include more or fewer components than those shown in Figure 5. In some embodiments, single-LoD displacement packing hardware 520 per block is included.The computer device 500 in Figure 5 includes an application 530 and hardware 520 for a single-block-per-LoD displacement packing method, but the single-block-per-LoD displacement packing method can be implemented in the computer device as hardware, firmware, software, or any combination thereof. For example, in some embodiments, the single-block-per-LoD displacement packing application 530 is programmed in memory and executed using a processor. In another example, in some embodiments, the single-block-per-LoD displacement packing hardware 520 is programmed hardware logic including gates specifically designed to implement the single-block-per-LoD displacement packing method.

[0026] In some embodiments, a single LoD displacement packing application 530 per (single or multiple) block includes several applications and / or modules. In some embodiments, a module also includes one or more submodules. In some embodiments, fewer or additional modules may be included.

[0027] Examples of suitable computer devices include personal computers, laptop computers, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants, cellular phones / mobile phones, smart home appliances, game consoles, digital cameras, digital camcorders, camera phones, smartphones, portable music players, tablet computers, mobile devices, video players, video disc writers / players (e.g., DVD writers / players, high-definition disc writers / players, ultra-high-definition disc writers / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smartwatches), vehicles (e.g., autonomous vehicles), or any other suitable computer devices.

[0028] To utilize a single LoD displacement packing method per block, the device acquires or receives 3D content (e.g., point cloud content). This single LoD displacement packing method per block can be implemented with or without user assistance.

[0029] During operation, a single LoD displacement packing method per block enables the implementation of slice decoding and other video processing. By using a per-block LoD packing implementation, the sequence displacement bitrate changes minimally.

[0030] Several embodiments of displacement packing using a single LoD per block 1. A method programmed into the non-temporary memory of a device, A step of packing displacement information of a single level of detail (LoD) into one or more blocks, The steps include packing filler bits up to the end of the current block among the one or more blocks, A method that includes this.

[0031] 2. The method according to the first paragraph, further comprising the steps of packing the displacement information of a single LoD into one or more blocks and packing the filler bits up to the end of the current block among the one or more blocks, for subsequent LoDs until all the displacement information has been packed.

[0032] 3. The method according to paragraph 1, further comprising the step of receiving position count information indicating the start and end positions of each LoD.

[0033] 4. Each subsequent LoD begins from the beginning of the block, as described in paragraph 1.

[0034] 5. The method according to paragraph 1, further comprising the step of triggering block-by-block LoD displacement packing using a flag.

[0035] 6. The method according to paragraph 1, further comprising the step of dividing the mesh information into different LoDs.

[0036] 7. The method according to paragraph 1, further comprising the step of performing slice decoding.

[0037] 8. The method according to paragraph 1, further comprising the step of decoding the displacement information.

[0038] 9. A device, Non-temporary memory for storing applications, wherein the applications are Displacement information at a single level of detail (LoD) is packed into one or more blocks. Pack the filler bits up to the end of the current block among the one or more blocks mentioned above. Non-temporary memory is intended for this purpose, A processor coupled to the memory and configured to process the application, A device that includes this.

[0039] 10. The apparatus according to paragraph 9, wherein the application is further configured to pack the displacement information of a single LoD into one or more blocks and to pack the filler bits up to the end of the current block among the one or more blocks, for subsequent LoDs until all the displacement information has been packed.

[0040] 11. The apparatus according to paragraph 9, wherein the application is further configured to receive position count information indicating the start and end positions of each LoD.

[0041] 12. Each subsequent LoD is the device described in Section 9, starting from the beginning of the block.

[0042] 13. The apparatus described in Section 9, wherein the application is further configured to trigger block-by-block LoD displacement packing using a flag.

[0043] 14. The apparatus described in Section 9, wherein the application is further configured to divide the mesh information into different Levels of Data (LoDs).

[0044] 15. The apparatus described in paragraph 9, further configured to perform slice decoding.

[0045] 16. The apparatus according to paragraph 9, wherein the application is further configured to decode the displacement information.

[0046] 17. It is a system, An encoder configured to encode a 3D mesh, Displacement information at a single level of detail (LoD) is packed into one or more blocks. Pack the filler bits up to the end of the current block among the one or more blocks mentioned above. A decoder configured as follows, A system that includes this.

[0047] 18. The system according to paragraph 17, wherein the decoder is further configured to pack the displacement information of a single LoD into one or more blocks and to pack the filler bits up to the end of the current block among the one or more blocks, and repeat this for subsequent LoDs until all the displacement information has been packed.

[0048] 19. The system according to paragraph 17, wherein the decoder is further configured to receive position count information indicating the start and end positions of each LoD.

[0049] 20. Each subsequent LoD starts from the beginning of the block, as described in Section 17.

[0050] 21. The system according to Section 17, wherein the decoder is further configured to trigger block-by-block LoD displacement packing using a flag.

[0051] 22. The system according to Section 17, wherein the decoder is further configured to divide the mesh information into different Levels of Data (LoDs).

[0052] 23. The system according to paragraph 17, wherein the decoder is further configured to perform slice decoding.

[0053] 24. The system according to paragraph 17, wherein the decoder is further configured to decode the displacement information.

[0054] To facilitate understanding of the structure and operating principles of the present invention, the invention has been described in relation to specific embodiments incorporating detailed features. Such references to specific embodiments and their details in this specification are not intended to limit the claims appended herein. It will be readily apparent to those skilled in the art that various other modifications can be made to the embodiments selected for illustrative purposes without departing from the spirit and scope of the invention as defined by the claims. [Explanation of Symbols]

[0055] 100 V-DMC Decoder 102 Image Unpacking Components 200 Displacement Packing Stream 202 Displacement Packing Stream Blocks 204, 206, 208 252 images 254 blocks 256 blocks 258 blocks 260 Filler Bits 300 images 350 images 352 Gap 400 Receive position count information and LoD displacement information. Packing of 402 LoD displacement information 404 Pack the filler bits until the current block is filled. 500 computer devices 502 Network Interface 504 memory 506 Processors 508 I / O devices 510 Bus 512 Storage device Single LoD displacement packing hardware per 520 blocks 530 Single LoD displacement packing application per block

Claims

1. A method programmed into the device's non-temporary memory, A step of packing displacement information of a single level of detail (LoD) into one or more blocks, The steps include packing filler bits up to the end of the current block among the one or more blocks, A method characterized by including the following.

2. The method according to claim 1, further comprising the steps of packing the displacement information of a single LoD into one or more blocks, and packing the filler bits up to the end of the current block among the one or more blocks, repeating this for subsequent LoDs until all the displacement information has been packed.

3. The method according to claim 1, further comprising the step of receiving position count information indicating the start and end positions of each LoD.

4. The method according to claim 1, characterized in that each subsequent LoD starts from the beginning of the block.

5. The method according to claim 1, further comprising the step of triggering block-by-block LoD displacement packing using a flag.

6. The method according to claim 1, further comprising the step of dividing mesh information into different LoDs.

7. The method according to claim 1, further comprising the step of performing slice decoding.

8. The method according to claim 1, further comprising the step of decoding the displacement information.

9. It is a device, Non-temporary memory for storing applications, wherein the applications are Displacement information at a single level of detail (LoD) is packed into one or more blocks. Pack the filler bits up to the end of the current block among the one or more blocks mentioned above. Non-temporary memory is intended for this purpose, A processor coupled to the memory and configured to process the application, An apparatus characterized by including

10. The apparatus according to claim 9, wherein the application is further configured to pack the displacement information of a single LoD into one or more blocks, and to pack the filler bits up to the end of the current block among the one or more blocks, and repeat this for subsequent LoDs until all the displacement information has been packed.

11. The apparatus according to claim 9, wherein the application is further configured to receive position count information indicating the start and end positions of each LoD.

12. The apparatus according to claim 9, characterized in that each subsequent LoD starts from the beginning of the block.

13. The apparatus according to claim 9, wherein the application is further configured to trigger block-by-block LoD displacement packing using a flag.

14. The apparatus according to claim 9, wherein the application is further configured to divide the mesh information into different Levels of Disorder (LD).

15. The apparatus according to claim 9, wherein the application is further configured to perform slice decoding.

16. The apparatus according to claim 9, wherein the application is further configured to decode the displacement information.

17. It is a system, An encoder configured to encode a 3D mesh, Displacement information at a single level of detail (LoD) is packed into one or more blocks. Pack the filler bits up to the end of the current block among the one or more blocks mentioned above. A decoder configured as follows, A system characterized by including

18. The system according to claim 17, further configured to pack the displacement information of a single LoD into one or more blocks, and to pack the filler bits up to the end of the current block among the one or more blocks, for subsequent LoDs until all the displacement information has been packed.

19. The system according to claim 17, characterized in that the decoder is further configured to receive position count information indicating the start and end positions of each LoD.

20. The system according to claim 17, characterized in that each subsequent LoD starts from the beginning of the block.

21. The system according to claim 17, wherein the decoder is further configured to trigger block-by-block LoD displacement packing using a flag.

22. The system according to claim 17, characterized in that the decoder is further configured to divide the mesh information into different Lines of Data (LoDs).

23. The system according to claim 17, wherein the decoder is further configured to perform slice decoding.

24. The system according to claim 17, characterized in that the decoder is further configured to decode the displacement information.

Images