Dynamic block decimation in V-PCC decoders
By adjusting the tile resolution based on rendering information during point cloud encoding and decoding, the computationally intensive problem of point cloud rendering is solved, optimizing rendering performance and user experience, especially on resource-constrained devices.
Patent Information
- Application Number
- CN202380033397.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-13
- Filing Date
- 2023-03-23
- Publication Date
- 2024-11-26
AI Technical Summary
Existing technologies, when processing point cloud data, especially during the rendering process, suffer from performance bottlenecks due to computationally intensive requirements. This makes it difficult to achieve real-time rendering and a good user experience, particularly on mobile devices or devices with limited computing resources.
By dynamically adjusting the resolution of tiles based on the rendering information provided by the renderer during point cloud encoding and decoding, high-resolution reconstruction is performed only on visible parts, while the resolution of unseen or unimportant parts is reduced or not reconstructed, thus optimizing the rendering process and reducing computational load.
It improves the real-time performance and user experience quality of point cloud rendering under limited computing resources, reduces the computing load, and ensures high-resolution display of critical parts.
Smart Images

Figure CN119032381A_ABST
Abstract
Description
Technical Field
[0001] The present embodiments generally relate to methods and apparatus for video encoding and rendering of point clouds. Background Art
[0002] Point cloud (PC) data format is a common data format across several business fields, for example, from autonomous driving, robotics, augmented reality / virtual reality (AR / VR), civil engineering, computer graphics to animation / film industry. 3D LiDAR (Light Detection and Ranging) sensors have been deployed on autonomous vehicles, and affordable LiDAR sensors have been released from Velabit, Apple iPad Pro 2020, and Intel RealSense LiDAR Camera L515. With the advancement of sensing technology, 3D point cloud data has become more practical than ever and is expected to be the ultimate enabler of the applications discussed in this article. Summary of the invention
[0003] According to one embodiment, a method for processing point cloud data is provided, comprising: obtaining rendering information from a renderer; based on the rendering information, selecting a resolution at which a portion of a geometric image is to be reconstructed; reconstructing the portion at the selected resolution; and outputting the reconstructed portion to the renderer.
[0004] According to another embodiment, a device for processing point cloud data is provided, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to: obtain rendering information from a renderer; based on the rendering information, select a resolution at which a portion of a geometric image is to be reconstructed; reconstruct the portion at the selected resolution; and output the reconstructed portion to the renderer.
[0005] One or more embodiments also provide a computer program including instructions, which, when executed by one or more processors, cause the one or more processors to perform a method according to any embodiment described herein. One or more embodiments of the present invention also provide a computer-readable storage medium having stored thereon instructions for processing point cloud data according to the method described herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] Figure 1 A block diagram of a system is shown in which aspects of the present embodiments may be implemented.
[0007] Figure 2 A schematic block diagram of an example of an image-based point cloud encoder is shown.
[0008] Figure 3A shows an example of a point in 3D space, and Figure 3B The projected pixels in the 2D image are shown.
[0009] Figure 4A shows an example of clustering of 3D points, and Figure 4B The projection of each cluster to a 2D patch is shown.
[0010] Figure 5 An example of a canvas C comprising two tiles P1 and P2 and their associated 2D bounding boxes B1 and B2 is shown.
[0011] Fig. 6A shows an example of a geometry image, Figure 6B An example of a texture image is shown, and Figure 6C An example of an occupancy map is shown.
[0012] Figure 7 A schematic block diagram showing an example of an image-based point cloud decoder.
[0013] Figure 8 A schematic block diagram of a modified image-based point cloud decoder according to one embodiment is shown.
[0014] Fig. 9A , Fig. 9B and Fig. 9C Examples of point clouds rendered at various resolutions are shown.
[0015] Fig.10 A modified reconstruction process according to one embodiment is shown.
[0016] Fig.11 Shown are examples of transformation matrices and coordinate spaces used during the rendering process.
[0017] Fig. 12A The atlas coordinate system is shown. Fig. 12B shows the local 3D tile coordinate system, and Fig. 12C The final target 3D coordinate system is shown.
[0018] Fig.13A and Fig. 13B An example of a 3D bounding box for a tile of a V-PCC decoded point cloud is shown.
[0019] Fig.14 An example of a decoded point cloud with several levels of decimation is shown according to one embodiment. DETAILED DESCRIPTION
[0020] Figure 1A block diagram of an example of a system in which various aspects and embodiments can be implemented is shown. System 100 can be implemented as a device including various components described below, and is configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 100 can be embodied in a single integrated circuit, multiple ICs and / or discrete components, either alone or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed over multiple ICs and / or discrete components. In various embodiments, system 100 is coupled to other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 100 is configured to implement one or more aspects described in this application.
[0021] The system 100 includes at least one processor 110 configured to execute instructions loaded therein for implementing, for example, various aspects described in the present application. The processor 110 may include embedded memory, input-output interfaces, and various other circuits known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include a non-volatile memory and / or a volatile memory, including but not limited to an EEPROM, a ROM, a PROM, a RAM, a DRAM, an SRAM, a flash memory, a magnetic disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device.
[0022] The system 100 includes an encoder / decoder module 130, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents a (multiple) module that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 130 may be implemented as a separate element of the system 100, or may be incorporated into the processor 110 as a combination of hardware and software known to those skilled in the art.
[0023] Program code to be loaded to the processor 110 or the encoder / decoder 130 to perform various aspects described in this application may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during the execution of the processes described in this application. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.
[0024] In several embodiments, memory inside the processor 110 and / or the encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be a memory 120 and / or a storage device 140, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as for MPEG-2, JPEG Pleno, MPEG-1, HEVC, or VVC.
[0025] As indicated at block 105, input may be provided to the elements of system 100 via various input devices. Such input devices include, but are not limited to, (i) an RF section that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.
[0026] In various embodiments, the input device of block 105 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for the following operations: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a certain frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal band that (for example) may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the desired frequency band again to perform frequency selection by filtering, down-conversion and filtering. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements that perform similar or different functions. Adding element can include inserting element between existing elements, for example, inserting amplifier and analog-to-digital converter. In various embodiments, the RF part includes antenna.
[0027] In addition, the USB and / or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within the processor 110 as desired. Similarly, various aspects of USB or HDMI interface processing may be implemented within a separate interface IC or within the processor 110 as desired. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 110 and the encoder / decoder 130, which operate in conjunction with memory and storage elements to process the data streams as desired for presentation on an output device.
[0028] The various components of system 100 may be disposed within an integrated housing where the various components may be interconnected and data may be transmitted therebetween using suitable connection means 115, such as an internal bus known in the art, including an I2C bus, wiring, and a printed circuit board.
[0029] The system 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 190. The communication interface 150 may include, but is not limited to, a modem or a network card, and the communication channel 190 may be implemented in, for example, a wired and / or wireless medium.
[0030] In various embodiments, a Wi-Fi network such as IEEE 802.11 is used to stream data to the system 100. The Wi-Fi signals of these embodiments are received through a communication channel 190 and a communication interface 150 suitable for Wi-Fi communication. The communication channel 190 of these embodiments is typically connected to an access point or router that provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to the system 100, which passes data through the HDMI connection of the input block 105. Still other embodiments use the RF connection of the input block 105 to provide streaming data to the system 100.
[0031] The system 100 can provide output signals to various output devices, including a display 165, a speaker 175, and other peripherals 185. In various examples of embodiments, the other peripherals 185 include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functions based on the output of the system 100. In various embodiments, control signals are transmitted between the system 100 and the display 165, the speaker 175, or other peripherals 185 using signaling such as AV.Link, CEC, or other communication protocols that can achieve device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 100 via dedicated connections through their respective interfaces 160, 170, and 180. Alternatively, the output devices can be connected to the system 100 using a communication channel 190 via the communication interface 150. The display 165 and the speaker 175 can be integrated into a single unit with other components of the system 100 in an electronic device (e.g., a television). In various embodiments, the display interface 160 includes a display driver, such as a timing controller (T Con) chip.
[0032] For example, if the RF portion of input 105 is part of a separate set-top box, the display 165 and speaker 175 may alternatively be separate from one or more of the other components. In various embodiments where the display 165 and speaker 175 are external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.
[0033] Figure 2 Schematic block diagram of an example of an image-based point cloud encoder is shown. The encoder 200 may provide a geometry / texture image representing the geometry / attributes of the 3D samples of the input point cloud frame (IPCF). The bitstream generated by the encoder 200 may be generated by Figure 7 An image-based decoder 700 is shown decoding, which can provide a reconstructed point cloud frame.
[0034] The image-based point cloud encoder 200 utilizes existing video codecs to compress the geometric and texture (attribute) information of the dynamic point cloud. This is achieved by converting the point cloud data into a set of different video sequences.
[0035] In one embodiment, an existing video codec (eg, a HEVC Main Profile encoder / decoder) may be used to generate and compress two videos, one for capturing geometric information of the point cloud data and the other for capturing texture information.
[0036] Additional metadata used to interpret the two videos is also typically generated and compressed separately. Such additional metadata includes, for example, occupancy maps and / or auxiliary tile information (PI).
[0037] The resulting video bitstream and metadata may then be multiplexed together to generate a combined bitstream.
[0038] An example of such a point cloud encoding / decoding process is given in the text of ISO / IEC FDIS23090-5 Coding of visual volumetric video and compression of point clouds based on video, ISO / IEC JTC 1 / SC 29 / WG 11 / MPEG-I 3DG, w19579.
[0039] Specifically, the tile generator (210) may generate at least one tile by decomposing 3D samples of a data set representing an input point cloud frame (IPCF) into 2D samples on a projection plane, for example, using a strategy that provides optimal compression. A tile may be defined as a set of 2D samples.
[0040] For example, in V-PCC, the normal of each 3D sample is first estimated. Next, an initial clustering of the input point cloud frame is obtained by associating each 3D sample with one of the six oriented planes of the 3D bounding box containing the 3D sample of the input point cloud frame. More specifically, each 3D sample is clustered and associated with the oriented plane with the closest normal (i.e., maximizing the dot product of the point normal and the plane normal). The 3D samples are then projected to the planes to which they are associated, e.g., Figure 3A The points in the 3D space shown are projected onto Figure 3B Pixels in the 2D image shown.
[0041] A set of 3D samples that form a connected area in its plane is called a connected component. A connected component is a set of at least one 3D sample with similar normals and the same associated orientation plane. The initial clustering is then refined by iteratively updating the cluster associated with each 3D sample based on its normal and the cluster of its nearest neighboring samples. A tile is generated from each connected component by projecting the 3D samples of each connected component onto the orientation plane associated with the connected component. The tiles are associated with auxiliary tile information PI, which represents auxiliary tile information defined for each tile to interpret the projected 2D samples corresponding to geometry and / or attribute information. Figure 4A shows the clustering of 3D points, and Figure 4B The projection of each cluster to a 2D tile is shown.
[0042] For example, in V-PCC, the auxiliary tile information PI includes (1) information indicating one of six oriented planes of a 3D bounding box containing a 3D sample of a connected component; (2) information relative to the plane normal; (3) information determining the 3D position of the connected component relative to the tile expressed in terms of depth, tangential displacement, and bi-tangential displacement; and (4) information such as coordinates (u0, v0, u1, v1) in the projection plane defining a 2D bounding box enclosing the tile.
[0043] The tile packing module (220) can map (place) at least one generated tile onto a 2D grid (also called a canvas) in a manner that generally minimizes unused space, and can ensure that each TxT (e.g., 16×16) block of the 2D grid is associated with a unique tile. A given minimum block size TxT of the 2D grid can specify a minimum distance between different tiles placed on the 2D grid. The 2D grid resolution can depend on the size of the input point cloud and its width W and height H, and the block size T can be sent to the decoder as metadata.
[0044] The auxiliary tile information (PI) may also include information related to the association between the block of the 2D grid and the tile. In V-PCC, the auxiliary information PI may include block-to-tile index information (BlockToPatch) that determines the association between the block of the 2D grid and the tile index.
[0045] Figure 5 An example of a canvas C including two tiles P1 and P2 and their associated 2D bounding boxes B1 and B2 is shown. Note that Figure 5 As shown, two bounding boxes may overlap in canvas C. The 2D grid (partitioning of the canvas) is represented only inside the bounding boxes, but the partitioning of the canvas also occurs outside those bounding boxes. The bounding box associated with a tile may be partitioned into TxT blocks, typically T=16.
[0046] A TxT block containing 2D samples belonging to a tile can be considered an occupied block. Each occupied block of the canvas is represented by a specific pixel value (e.g. 1) in the occupancy map (three-channel image), and each unoccupied block of the canvas is represented by another specific value, e.g. 0. The pixel value of the occupancy map can then indicate whether the TxT block of the canvas is occupied, i.e., contains a 2D sample belonging to a tile.
[0047] exist Figure 5 In the image generation process, the geometry and texture of the input point cloud frame IPCF are stored as images using the mapping of the generated (multiple) tiles to the 2D grid calculated during tile packing. The geometry image can represent the geometry of the input point cloud frame IPCF, and can be, for example, a monochrome image of WxH pixels represented in YUV 420-8 bit format. The texture image is a three-channel image that can represent the texture of the input point cloud frame IPCF, and can be, for example, an image of WxH pixels represented in YUV 420-8 bit format or RGB444-8 bit format.
[0048] Specifically, the geometry image generator (230) can generate at least one geometry image from the input point cloud frame, the occupancy map and the auxiliary tile information. The geometry image generator (230) can use the occupancy map information to detect (locate) occupied tiles, thereby detecting (locating) non-empty pixels in the geometry image.
[0049] To better handle the case where multiple 3D samples are projected (mapped) to the same 2D sample of the projection plane (along the same projection direction (line)), multiple images called layers can be generated. Thus, different depth values D1, ..., Dn can be associated with the 2D samples of the tile, and then multiple geometric images can be generated.
[0050] In V-PCC, the 2D samples of a tile are projected onto two layers. The first layer, also called the near layer, can store, for example, depth values D0 associated with 2D samples with a smaller depth. The second layer, called the far layer, can store, for example, depth values D1 associated with 2D samples with a larger depth. Alternatively, the second layer can store the difference between the depth values D1 and D0.
[0051] The texture image generator (240) can generate at least one texture image from the input point cloud frame IPCF, the occupancy map, the auxiliary tile information PI and the geometry of the reconstructed point cloud frame derived from at least one decoded geometry image.
[0052] The texture image generator may utilize the occupancy map information in order to detect (locate) occupied blocks and thus detect (locate) non-empty pixels in the texture image.The texture image generator may be adapted to generate a texture image and associate the texture image with each geometry image / layer.
[0053] According to an embodiment, the texture image generator (240) may encode (store) the texture (attribute) value T0 associated with the 2D sample of the first layer as a pixel value of the first texture image TI0, and encode (store) the texture value T1 associated with the 2D sample of the second layer as a pixel value of the second texture image TI1.
[0054] Alternatively, the texture image generation module (240) may encode (store) the texture value T1 associated with the 2D sample of the second layer as the pixel value of the first texture image TI0, and encode (store) the texture value D0 associated with the 2D sample of the first layer as the pixel value of the second geometric image GI1. For example, Fig. 6A shows a geometric image, Figure 6B shows a texture image, and Figure 6C An occupancy map is shown.
[0055] According to one embodiment, a padding process may be applied to the geometry and / or texture images. The padding process may be used to fill in the empty spaces between tiles to generate piecewise smooth images suitable for video compression.
[0056] The video encoder (250) may encode the generated geometry image and texture image. According to one embodiment, the video encoder (250) and / or the occupancy map encoder (260) may be an HEVC based encoder. The occupancy map encoder (260) may encode the occupancy map as an image, as described in detail in, for example, Section H.9.3 Occupancy Video Decoding Process in w19579. Lossy or lossless encoding may be used.
[0057] The tile information encoder (270) may encode auxiliary tile information and possibly additional metadata such as block size T, width W and height H of the geometry / texture image. According to one embodiment, the auxiliary tile information may be differentially encoded (as defined, for example, in w19579, section H.9.6, sub-bitstream extraction process).
[0058] A multiplexer (280) may be applied to the generated outputs of the video encoder (250), the occupancy map encoder (260), and the tile information encoder (270), and as a result, these outputs may be multiplexed together to generate a bitstream. It should be noted that the metadata information typically represents a small portion of the entire bitstream, and the information blocks are compressed using the video codec.
[0059] Figure 7A schematic block diagram of an example of an image-based point cloud decoder 700 according to one embodiment is shown.
[0060] Specifically, a demultiplexer (710) may be applied to demultiplex coded information of a bitstream generated, for example, by an image-based point cloud encoder (200). A video decoder (720) may then decode the coded information to derive at least one decoded geometry image and at least one decoded texture image. An occupancy map decoder (730) may decode the coded information to derive a decoded occupancy map. According to one embodiment, the video decoder and / or the occupancy map decoder may be a HEVC-based decoder. A tile information decoder (740) may decode the coded information to derive auxiliary tile information. Possibly, metadata may also be derived from the bitstream.
[0061] The geometry reconstruction module (750) may derive the geometry of the reconstructed point cloud frame from at least one decoded geometry image, a decoded occupancy map, decoded auxiliary tile information, and possibly additional metadata. The geometry reconstruction module (750) may utilize the decoded occupancy map information in order to locate non-empty pixels in the at least one decoded geometry image.
[0062] The texture reconstruction module (760) may derive a texture of the reconstructed point cloud frame from the geometric shape and at least one decoded texture image. The texture reconstruction module (760) may derive a texture of non-empty pixels.
[0063] The decoded point cloud decoded by the geometry reconstruction module (750) and the texture reconstruction module (760) is transmitted to the rendering engine, which renders the point cloud onto the screen according to the user's position and the parameters of the camera and displays it to the user.
[0064] When rendering V-PCC point clouds, the number of points (or vertices) to render can greatly impact the performance of applications that are typically on less advanced mobile devices or on computationally intensive hardware such as mixed reality glasses. Reducing the number of points to render can ensure real-time execution, but at the cost of a loss in perceived quality.
[0065] Rendering large point clouds on limited memory devices can be performed in various ways by shrinking the point cloud to allow loading all points in memory and making rendering possible. Other approaches propose to split the large model into several 3D boxes / regions and store each sub-point cloud at various resolutions. This allows to choose which resolution to use for each region according to the user position to the camera during the rendering phase. These approaches optimize the rendering of the point cloud but do not change the decoding process.
[0066] To ensure real-time applications and a good user experience, we propose to decode and reconstruct the V-PCC bitstream in different ways based on the viewer's position. Blocks of the V-PCC tile will be decoded with variable resolution depending on the position of the camera responsible for rendering the decoded point cloud. This process will generate fewer points for less visible parts and keep all points of parts facing and close to the viewer. This process works for V-PCC bitstreams, but the same approach can be used to decode and render other types of bitstreams that describe 3D objects or 3D scenes.
[0067] In the V-PCC decoder, each block is decoded independently of the other blocks of the tile, but all pixels of the block are reconstructed in the same way because the tile information used for reconstruction is encoded per block. Figure 8 As shown, we propose to alter the geometry reconstruction process (850) by dynamically reducing the resolution of a patch given its reconstructed position in the projected space, directly allowing for less computationally intensive rendering of the point cloud.
[0068] like Figure 7 As shown, the V-PCC decoding and reconstruction process specified in the V-PCC standard is performed without any information about the use of the point cloud reconstructed during the rendering phase. Figure 8 As shown, we propose to modify the reconstruction process (850) by utilizing information about where the decoded point cloud will be rendered in the renderer (880). In this embodiment, the decoder (800) does not decode or generate all points if they are not present or visible in the rendered scene, or if decimation does not have a significant impact on the perceived visual quality of the rendering while reducing the computational load.
[0069] like Fig. 9A , Fig. 9B and Fig. 9C As shown, each block can be reconstructed at various resolutions or decimation levels. For example, for a 16x16 block at various decimation levels, the number of points is: - Decimation level 0: 16x16=256 - Extraction level 1: 8x8=64 - Extraction level 2: 4x4=16 - Extraction level 3: 2x2=4 - Extraction level 4: 1x1=1 -Decimation level 5: 0x0=0 (no reconstruction).
[0070] The texture reconstruction process (860) assigns a color to each reconstructed point. Depending on the selected decimation level, the geometry reconstruction process (850) will generate fewer points, and in this case, the texture reconstruction process (860) will color fewer points, but the process itself remains unchanged.
[0071] The decimation rates for the horizontal and vertical directions can also be different, and there can be separate decimation levels for the horizontal and vertical directions. For example, the number of dots can be controlled by the horizontal decimation level and the vertical decimation level.
[0072] Based on the rendering information transmitted by the renderer, it is possible to evaluate for each tile, for each block component of a tile, or for each pixel of a tile whether the corresponding reconstructed point will be viewed by the user after the rendering process and which decimation level can be selected without significantly compromising the rendering quality. Based on the evaluation result, it can be decided whether to render the current area and at what resolution. In the following, we propose to perform the proposed method on 16×16 blocks and set the decimation level at which the block must be rendered for each block. The use of 16x16 blocks was chosen because the V-PCC bitstream uses a 16x16 block size by default to describe tiles. The proposed process can be performed with other block sizes and the block size used can be taken according to the complexity of the process. For example, if the size is small, the process of selecting the decimation level will be performed several times and the fidelity of the reconstruction will be reduced; if the size of the block is too large, the selected decimation level will not be correct for all parts of the block.
[0073] Fig.10 A modified reconstruction process in a V-PCC decoder according to one embodiment is shown. In this embodiment, rendering information is transmitted to the V-PCC decoder, which can be used to calculate whether the point will be visible in screen space. At step 1010, a 3D bounding box for the tile is constructed. At step 1020, it is evaluated whether the tile is at least partially visible. If the tile will not be visible at all, it is not rendered (rendering is ignored). Otherwise, if the tile will be partially or completely visible, then at step 1030, each 16×16 block in the tile is considered.
[0074] Specifically, at step 1030, the reconstructed position of the block in screen space is calculated. At step 1040, it is evaluated whether the block will be at least partially visible. If the block will not be visible at all, it is not rendered. Otherwise, if the block will be partially or completely visible, at step 1050, a decimation level is selected for the block, and the block is reconstructed at the selected decimation level. The reconstructed point cloud is then rendered by the renderer (1060). In the following, the steps will be described in further detail.
[0075] Rendering Information
[0076] like Figure 8As shown, information for rendering the point cloud is transmitted to the V-PCC point cloud decoder. The transmitted rendering information may include information about matrices used to transform points from model coordinates to world coordinates, view coordinates, and clipping coordinates, namely, model matrix, view matrix, projection matrix, viewport matrix.
[0077] Fig.11 shows an example of a transformation matrix and coordinate space used during the rendering process. Fig.11 In , vertex coordinates start out as local coordinates in local space, then are processed into world coordinates in world space, view coordinates in view space, clip coordinates in clip space, and finally end up as screen coordinates in screen space, using the model matrix, view matrix, projection matrix, and viewport matrix, respectively. Local coordinates are the coordinates of an object relative to its local origin, and world space coordinates are coordinates relative to the larger world. These coordinates are relative to some global origin of the world, along with many other objects are also placed relative to this world's origin. In view space coordinates, each coordinate is seen from the perspective of the camera or viewer. Clip coordinates are in the range of -1.0 and 1.0, and are used to determine which vertices will end up on the screen.
[0078] As mentioned above, the rendering information is used by the renderer to project the points that need to be rendered onto the pixels of the display screen. They can be transmitted from the renderer to the decoder, for example, every frame. With this additional information, the decoder can estimate where the points will be displayed before reconstruction and modify the reconstruction process based on the rendering information.
[0079] For example, the coordinates (x, y, z) of a 3D reconstructed point can be transformed to local Transform to screen space coordinates (x', y', z') screen : (x′, y′, z′, w) screen =M viewport ·M projection ·M view ·M model (x, y, z, 1) local The points are represented by homogeneous coordinates, denoted as (x, y, z, w). Reconstructing the point (x, y, z) local It can be expressed in homogeneous coordinates: (x, y, z) local And it can be easily transformed into homogeneous coordinates (x′, y′, z′, w) by matrix multiplication or even projection transformation. screen The Cartesian coordinates can then be obtained by dividing the x and y components of the vector by w, and we get that the screen coordinates (x', y') are equal to (y' / w', y' / w) screen .
[0080] In screen space, (x', y') corresponds to the coordinates of a pixel of the screen in the range [0; screen width] [0; screen height] ([0; screen width] x [0; screen height]) where the 3D point will be displayed, and the z' coordinate contains a normalized depth value between [-1; 1] corresponding to the distance between the 3D point and the screen, normalized with the near and far plane values defined by the camera parameters. In screen space, if (x', y', z') screen If it is outside the range, the point will not be displayed on the screen.
[0081] Evaluation of tiles
[0082] A 3D bounding box of the tile is constructed (1010) and evaluated (1020) in screen space to determine whether the tile needs to be rendered. As shown in FIG. 12, which depicts an example of (a) an atlas coordinate system, (b) a local 3D tile coordinate system, and (c) a final target 3D coordinate system, the following decoded tile data is used to construct bounding box coordinates in model space (local space):
[0083] Fig.13A and Fig. 13B An example of a 3D bounding box for a tile of a V-PCC decoded point cloud is shown.
[0084] The 3D tile bounding box information can be used to decide whether to render the current tile. If the projected area of the tile does not intersect the screen, the projection of the eight 3D points defining the 3D bounding box into screen space can be used to cull the tile.
[0085] In addition to the process described in the following steps, the distances between the eight projected points of the tile's bounding box in screen space can give a first approximation of the decimation applied to render a tile block.
[0086] Evaluation of blocks
[0087] During the decoding process, each individual 16×16 pixel block in the tile area is considered. For each block, the tile metadata, occupancy map values, geometry values, and camera information are used to calculate (1030) the pixel value in screen space (Rec screen ) The approximation of the position of the reconstructed point(s) in screen space, and in particular the distance(s) to the screen, can be used to select the decimation level of the block.
[0088] The reconstructed point in screen space is defined as the coordinates (x, y, z) screen , where (x, y) are the coordinates of the pixel of the screen where the point will be displayed. The z coordinate corresponds to Reclocal The normalized distance between (a 3D reconstructed point in model space) and the camera.
[0089] Different approaches can be used to compute the (multiple) reconstructed positions. The simplest way (reduces computational load) is to consider only one point, ideally close to the center of the block, since in most cases there are no large variations within each block. A limitation of this approach is that a single point may not represent the entire block (local singularities, such as e.g. sharp spikes or holes). But this is a trade-off between complexity and quality. It is also possible to use an average of multiple positions to limit singularity effects. A subsampling of the values within the block can be defined, mainly weighing the number of points to consider and the computational load. The calculation can also be performed by considering all pixels within the block and then computing the exact bounding box of the block.
[0090] The 16×16 block is not fully occupied, and it is not guaranteed that the 4 corner pixels of the block are occupied. But when the geometry data is padded for better compression, even if these points are not actually part of the model, they are guaranteed to be in a somewhat consistent position relative to the rest of the block. Therefore, even if the block is not fully occupied, the above method can be used, and only using the occupied points to calculate the reconstructed position will get better results (but this requires obtaining the occupancy map data for each point to know whether these points are occupied).
[0091] Selection of extraction level
[0092] The (x, y, z) of the reconstructed point in screen space can be screen The coordinates are analyzed to detect whether the tile is to be rendered (1060), and if so, the resolution (decimation level) of the tile is selected (1050).
[0093] In the following, we propose to use the projection point coordinates (x, y, z) screen There are two ways to choose the decimation level. In the first method, we use the distance to the screen (z screen ) is compared with the range of the distance threshold. In the second method, we compare the (x, y) distance in screen coordinates of the projections of two neighboring points of the block.
[0094] According to the V-PCC specification, when the block-to-tile index of a block is not defined because the current block is not in the tile, the unused block is directly discarded and then shall not be rendered.
[0095] When the projected point is outside the viewport, that is, when (x, y) screen Not in [0; Width screen ]x[0;Height screen], we can cull the entire block. It improves rendering performance, but may cull partially displayed blocks. There are many ways to design to reduce the probability of culling partially visible blocks. In one example, we can test multiple points (in this case, ideally the four corners). In another example, we can add a margin (usually 10%) to the rendering block to ensure that every part that needs to be rendered is rendered. In this case, we can check (x, y) screen Is it in [-Width screen * N%; Width screen *(1+N%)]x[-Height screen *N%;Height screen *(1+N%)]) within.
[0096] According to the first method, the threshold is a set of distance ranges. If the distance (z screen ) is within the threshold range, the corresponding extraction level is selected for reconstructing the block. In the case of reconstructing multiple block points, the minimum z is compared. screen value.
[0097] The number of thresholds and their corresponding depth ranges can be user defined, developer defined (compiled in the application), or dynamically configured based on the distance of two neighboring pixels of a tile or on the camera's projection screen.
[0098] The threshold may be fixed, where a fixed distance range is defined for each decimation level. For example, for distances between 0 and 0.5, the decimation level is set to 0; for distances between 0.5 and 0.75, the decimation level is set to 1.
[0099] The threshold can also be adaptive, for example based on the size of the object. In this case, the bounding box of the object can be used to define the range of positions where the object will be rendered [min(z screen ); max(z screen )], and in particular the range of distances from the screen that the object will be at. For example, if the object is at min(z screen ) and max(z screen ), that is, between [0%, 25%]*[min(z screen ); max(z screen )], the decimation level is set to 0; if the object is located at min(z screen ) and max(z screen ), that is, at [25%, 50%]*[min(z screen ); max(z screen)], the decimation level will be set to 1.
[0100] These thresholds may also be dynamic, based on rendering parameters. In particular, the thresholds may be changed dynamically to adapt the reconstructed point cloud to the capabilities of the display and the type of rendering point process used.
[0101] If the rendering process uses splatting to enlarge the points on the screen display, or any other rendering process that increases the size of the area covered by the points on the screen after projection, the threshold given to the decoder must be adjusted to compensate. If the system performance is low (FPS, memory, GPU usage, etc.), or if the number of reconstructed points is too high, the threshold can be adapted to reconstruct fewer or more points.
[0102] In a second approach, the choice of the decimation level may be related to the distance in screen space between two projected points from two neighboring points in the geometric image.
[0103] For two adjacent points in the geometry image, labeled (u′, v′) and (u″, v″), the two 3D reconstructed points in the model space will be (x′ local , y′ local , z′ local ) and (x″ local , y″ local , z″ local ). After projection, these points will be (x′ screen , y′ screen , z′ screen ) and (x″ screen , y″ screen , z″ screen ).
[0104] The distance d(p′) can be calculated screen , p″ screen )=||(x′ screen -x″ screen )+(y′ screen -y″ screen ))||(Depending on the desired process complexity, distance L1 or L2 can be used). d(p′ screen , p″ screen ) represents the distance in screen space between two neighboring points of the block, and is a good approximation of the screen area that the block will cover. If this value is less than 1 (in which case 1 pixel per point or more, indicating full resolution), the decimation level is chosen using the following formula: From this we get: where α is a multiplication factor (α is typically 0.5 or 1, where 0.5 represents decimation that minimizes information loss and 1 represents decimation that minimizes aliasing).
[0105] For example, for two adjacent points A and B in the same row of a geometric image, the coordinates of these points are A = (u, v) and B = (u+1, v). The corresponding screen projection point is: A′ = (x′ A , y′ A ) and B′=(x′ B , y′ B ). If the distance d(A′, B′) is equal to 0.2, we can estimate that the entire line of 16 points will be projected on a plane of size: space. Therefore, these 16 points will be drawn on only 4 pixels. Based on this, we can choose the lowest decimation level. The previous formula suggests that if α is 0.5, a decimation level of 2 is used, and in this case, the line will be decimated twice and the line will create four projected points that can be stored nicely in the 3.2 pixels of the screen without much overlap. If α is set to 1, the previous formula suggests using a decimation level of 3 and the line will be reduced to two points, which ensures no aliasing at the cost of some information loss.
[0106] If d(p′ screen , p″ screen ) is lower than the pixel size, two adjacent points of the block will be projected onto the same pixel of the screen and the block can be rendered at a lower resolution.
[0107] This process must be adjusted depending on the rendering method used to display the point on the screen (point, point with OpenGL point size, circle, splat, quad, etc.) to adjust the alpha factor to the size of the area covered by the projection of a point.
[0108] For better accuracy, more than two neighboring points of a block can be used. If several points are projected, the decimation level can be calculated in two dimensions and two decimation levels can be sent to the reconstruction process (n u , n v ), and the block can be reconstructed using the two resolutions after u and v. For example, if (n u , n v ) is equal to (1, 2), the reconstruction process will not create 16×16 points, but only 8×4 points.
[0109] The information (whether to reconstruct each block and at what resolution) is sent to the actual model reconstruction algorithm, which generates output vertices accordingly, and each block is reconstructed according to its decimation level.
[0110] Fig.14 An example of a decoded point cloud with several decimation levels is shown. The portion outside the camera window has been removed, and the visible point cloud blocks have been reconstructed at different decimation levels. Specifically, portion 1410 is reconstructed at full resolution, portion 1420 is reconstructed at resolution 2 (for a 16×16 block, 8×8=64 points are created, where 256 points should be created without decimation), and portion 1430 is reconstructed at resolution 4 (for a 16×16 block, 4×4=16 points are created).
[0111] Various numerical values are used in this application. The specific values are for illustrative purposes, and the described aspects are not limited to these specific values.
[0112] Various methods are described herein, and each method includes one or more steps or actions for realizing the method.Unless the correct operation of the method requires a specific order of steps or actions, otherwise the order and / or use of specific steps and / or actions can be modified or combined.In addition, terms such as "first", "second" etc. can be used to modify elements, components, steps, operations, etc. in various embodiments, such as, for example, "first decoding" and "second decoding".Unless specifically required, the use of these terms does not mean the sequencing of the operation to modification.Therefore, in this example, the first decoding does not need to be performed before the second decoding, and can occur in, for example, before, during or in a time period overlapping with the second decoding.
[0113] The implementations and aspects described herein can be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if only discussed in the context of a single implementation (e.g., discussed only as a method), the implementations of the features discussed can also be implemented in other forms (e.g., devices or programs). Devices can be implemented in, for example, appropriate hardware, software, and firmware. The method can be implemented in, for example, a device, such as a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication equipment, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other equipment that facilitates the communication of information between end users.
[0114] References to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation", as well as any other variations appearing in various places throughout the application, do not necessarily all refer to the same embodiment.
[0115] Furthermore, the present application may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.
[0116] Furthermore, the present application may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.
[0117] Furthermore, the present application may involve "receiving" various pieces of information. Like "accessing," receiving is a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved in one way or another.
[0118] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," any of the following uses of " / ," "and / or," and "at least one of" are intended to include selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to many of the listed items, as will be apparent to one of ordinary skill in this and related arts.
[0119] It will be apparent to one of ordinary skill in the art that implementations may generate a variety of signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing the method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.
Claims
1. A method for processing point cloud data, comprising: Get rendering information from the renderer; Based on the rendering information, selecting a resolution at which a portion of the geometric image is to be reconstructed; reconstructing the portion at the selected resolution; as well as The reconstructed portion is output to the renderer.
2. The method according to claim 1, further comprising: One or more reconstructed positions of one or more corresponding points of the portion in screen space are obtained, wherein the resolution is selected based on the one or more reconstructed positions.
3. The method according to claim 2, further comprising: The one or more reconstructed positions are compared to a range of distance values, wherein the resolution is selected in response to the comparison.
4. The method according to claim 3, characterized in that The distance value range is adapted to the size of the 3D model contained in the point cloud data.
5. The method according to claim 3, characterized in that: The range of distance values is appropriate for at least one of a type of rendering process used by the renderer and a display.
6. The method according to any one of claims 1 to 5, further comprising: The reconstructed portion is rendered based on the rendering information.
7. The method according to any one of claims 1 to 6, wherein: The portion is reconstructed only within the viewport based on the one or more reconstruction positions.
8. The method according to claim 1, further comprising: Projection positions of two adjacent points of the portion in screen space are obtained, wherein the resolution is selected based on the projection positions.
9. The method according to claim 8, further comprising: The projected positions of the two adjacent points of the portion are compared, wherein the resolution is selected in response to the comparison.
10. The method according to claim 8, further comprising: The distance between the projected positions is compared to a second range of distance values, wherein the resolution is selected in response to the comparison. The method according to claim 10 , wherein the second distance value range is suitable for a rendering method used by the renderer.
12. An apparatus for processing point cloud data, comprising one or more processors and at least one memory coupled to the one or more processors, wherein the one or more processors are configured to: Get rendering information from the renderer; Based on the rendering information, selecting a resolution at which a portion of the geometric image is to be reconstructed; reconstructing the portion at the selected resolution; as well as The reconstructed portion is output to the renderer.
13. The device according to claim 12, wherein: The one or more processors are further configured to: One or more reconstructed positions of one or more corresponding points of the portion in screen space are obtained, wherein the resolution is selected based on the one or more reconstructed positions.
14. The device according to claim 13, wherein: The one or more processors are further configured to: The one or more reconstructed positions are compared to a range of distance values, wherein the resolution is selected in response to the comparison. 15 . The apparatus of claim 14 , wherein the distance value range is suitable for a size of a 3D model contained in the point cloud data.
16. The apparatus of claim 14, wherein the range of distance values is appropriate for at least one of a type of rendering process used by the renderer and a display.
17. The device according to any one of claims 12 to 16, wherein: The one or more processors are further configured to render the reconstructed portion based on the rendering information.
18. The device according to any one of claims 12 to 17, wherein: The portion is reconstructed based solely on the one or more reconstruction positions being within the viewport.
19. The device according to claim 12, wherein: The one or more processors are further configured to: Projection positions of two adjacent points of the portion in screen space are obtained, wherein the resolution is selected based on the projection positions.
20. The apparatus of claim 19, wherein the one or more processors are further configured to: The projected positions of the two adjacent points of the portion are compared, wherein the resolution is selected in response to the comparison.
21. The apparatus of claim 19, wherein the one or more processors are further configured to: The distance between the projected positions is compared to a second range of distance values, wherein the resolution is selected in response to the comparison.
22. The apparatus of claim 21, wherein the second distance value range is suitable for a rendering method used by the renderer.
23. A non-transitory computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1-11.
