Signaling point size information for v-PCC rendering

Abstract

In one implementation, we propose to generate point size information in response to a change of resolution in the reconstruction process of 3D point cloud. In one example, point size information is signaled to a renderer. Advantageously, point size information allows a renderer to increase the size of the area covered by a 3D point on the screen after projection. In one approach, the point size information is signaled in data structure shared with point cloud data. In another approach, the point size information is signaled in a metadata structure. In yet other approaches, the point size information may be signaled at 3D point level or a block level.

Description

SIGNALING POINT SIZE INFORMATION FOR V-PCC RENDERINGCROSS REFERENCE TO RELATED APPLICATION[1] This application claims the benefit of European Patent Application No. 23306560.6, filed on September 20, 2023, which is incorporated herein by reference in their entirety. TECHNICAL FIELD[2] The present embodiments generally relate to a method and an apparatus for video coding and rendering of point clouds.BACKGROUND[3] The Point Cloud (PC) data format is a universal data format across several business domains, e.g., from autonomous driving, robotics, augmented reality / virtual reality (AR / VR), civil engineering, computer graphics, to the animation / movie industry. 3D LiDAR (Light Detection and Ranging) sensors have been deployed in self-driving cars, and affordable LiDAR sensors are released from Velodyne Velabit, Apple iPad Pro 2020 and Intel RealSense LiDAR camera L515. With advances in sensing technologies, 3D point cloud data becomes more practical than ever and is expected to be an ultimate enabler in the applications discussed herein.SUMMARY[4] According to one embodiment, a method of processing point cloud data is provided, the method comprising obtaining a resolution used for reconstructing point cloud data from a part of a geometry image; determining point size information responsive to the resolution used for reconstructing point cloud data; and outputting point size information relative to reconstructed point cloud data to a renderer.[5] According to another embodiment, a method for rendering point cloud data is provided, the method comprising obtaining point cloud data from a decoder; obtaining point size information for rendering point cloud data; rendering, using point size information, the point cloud data; and outputting rendered point cloud data.[6] According to another embodiment, an apparatus for processing point cloud data is provided, comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configured to obtain a resolution used for reconstructing point cloud data from a part of a geometry image; determine point size information responsive to the resolution used for reconstructing point cloud data; and outputting point size information relative to reconstructed point cloud data to a renderer.[7] According to another embodiment, an apparatus for rendering point cloud data in media content is provided, comprising one or more processors and at least one memory coupled to said one or more processors, wherein said one or more processors are configuredto obtain point cloud data from a decoder; obtain point size information for rendering point cloud data; rendering, using point size information, the point cloud data in media content; and outputting media content with rendered point cloud data.[8] One or more embodiments also provide a computer program comprising instructions which when executed by one or more processors cause the one or more processors to perform the method according to any of the embodiments described herein. One or more of the present embodiments also provide a computer readable storage medium having stored thereon instructions for processing point cloud data according to the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS[9] FIG. 1 illustrates a block diagram of a system within which aspects of the present embodiments may be implemented.

[0010] FIG. 2 illustrates a schematic block diagram of a first example of an image-based point cloud decoder and renderer.

[0011] FIG. 3 illustrates a schematic block diagram of a second example of an image-based point cloud decoder and renderer.

[0012] FIG. 4 illustrates a schematic block diagram of a third example of an image-based point cloud decoder and renderer within which aspects of the present embodiments may be implemented.

[0013] FIG. 5A, FIG. 5B and FIG. 5C illustrate an example of point cloud rendering at various resolutions.

[0014] FIG. 6 illustrates a schematic block diagram of a modified image-based point cloud decoder, in accordance with an embodiment.

[0015] FIG. 7 illustrates a schematic block diagram of a method for processing point cloud data according to an embodiment.

[0016] FIG. 8 illustrates a schematic block diagram of a method for rendering point cloud data according to an embodiment.

[0017] FIG. 9 illustrates a scan order for 3D DCT coefficient as implemented in an embodiment.DETAILED DESCRIPTION

[0018] FIG. 1 illustrates a block diagram of an example of a system in which various aspects and embodiments can be implemented. System 100 may be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this application. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100, singlyor in combination, may be embodied in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 100 is communicatively coupled to other systems, or to other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 100 is configured to implement one or more of the aspects described in this application.

[0019] The system 100 includes at least one processor 1 10 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this application. Processor 110 may include embedded memory, input output interface, and various other circuitries as known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device, and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drive, and / or optical disk drive. The storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device, as non-limiting examples.

[0020] System 100 includes an encoder / decoder module 130 configured, for example, to process data to provide an encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents module(s) that may be included in a device to perform the encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 130 may be implemented as a separate element of system 100 or may be incorporated within processor 1 10 as a combination of hardware and software as known to those skilled in the art.

[0021] Program code to be loaded onto processor 110 or encoder / decoder 130 to perform the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 110. In accordance with various embodiments, one or more of processor 1 10, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of various items during the performance of the processes described in this application. Such stored items may include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0022] In several embodiments, memory inside of the processor 110 and / or the encoder / decoder module 130 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, amemory external to the processing device (for example, the processing device may be either the processor 1 10 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations, such as for MPEG-2, JPEG Pleno, MPEG-I, HEVC, or VVC.

[0023] The input to the elements of system 100 may be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to, (i) an RF portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0024] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) down converting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which may be referred to as a channel in certain embodiments, (iv) demodulating the down converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion may include a tuner that performs various of these functions, including, for example, down converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, down converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements may include inserting elements in between existing elements, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[0025] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example,Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 110 as necessary. Similarly, aspects of USB or HDMI interface processing may be implemented within separate interface ICs or within processor 1 10 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110, and encoder / decoder 130 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.

[0026] Various elements of system 100 may be provided within an integrated housing, Within the integrated housing, the various elements may be interconnected and transmit data therebetween using suitable connection arrangement 115, for example, an internal bus as known in the art, including the I2C bus, wiring, and printed circuit boards.

[0027] The system 100 includes communication interface 150 that enables communication with other devices via communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network card and the communication channel 190 may be implemented, for example, within a wired and / or a wireless medium.

[0028] Data is streamed to the system 100, in various embodiments, using a Wi-Fi network such as IEEE 802. 11. The Wi-Fi signal of these embodiments is received over the communications channel 190 and the communications interface 150 which are adapted for Wi-Fi communications. The communications channel 190 of these embodiments is typically connected to an access point or router that provides access to outside networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 100 using a set-top box that delivers the data over the HDMI connection of the input block 105. Still other embodiments provide streamed data to the system 100 using the RF connection of the input block 105.

[0029] The system 100 may provide an output signal to various output devices, including a display 165, speakers 175, and other peripheral devices 185. The other peripheral devices 185 include, in various examples of embodiments, one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide a function based on the output of the system 100. In various embodiments, control signals are communicated between the system 100 and the display 165, speakers 175, or other peripheral devices 185 using signaling such as AV. Link, CEC, or other communications protocols that enable device- to-device control with or without user intervention. The output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output devices may be connected to system100 using the communications channel 190 via the communications interface 150. The display 165 and speakers 175 may be integrated in a single unit with the other components of system 100 in an electronic device, for example, a television. In various embodiments, the display interface 160 includes a display driver, for example, a timing controller (T Con) chip.

[0030] The display 165 and speaker 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which the display 165 and speakers 175 are external components, the output signal may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0031] V-PCC Standard as defined in the document “ISO / IEC 23090-5, Information technology — Coded Representation of Immersive Media — Part 5: Visual Volumetric Videobased Coding (V3C) and Video-based Point Cloud Compression (V-PCC)” specifies the syntax, semantics, and decoding for visual volumetric media using video-based coding methods. Furthermore, the document V-PCC specifies processes that may be needed for video rendering of points clouds consisting of points in a 3D space with associated attributes (e.g. color). A V-PCC bitstream may comprise coded occupancy / geometry / texture / normal / reflectance images representing the occupancy / geometry / attributes of 3D samples of an input point cloud frame IPCF. The bitstream generated by an encoder may be decoded by an image-based decoder that may provide a reconstructed point cloud frame (or point cloud data) for a renderer. The imagebased point cloud encoder leverages existing video codecs to compress the occupancy, geometry and texture (attribute) information of a dynamic point cloud. This is accomplished by essentially converting the point cloud data into a set of different video sequences. For instance, three videos, one for capturing the geometry information of the input point cloud frame and another for capturing the texture information of the input point cloud frame, may be generated and compressed using existing video codecs, for example, an HEVC Main profile encoder / decoder. Additional metadata that are used to interpret the three videos are typically also generated and compressed separately. Such additional metadata includes, for example, auxiliary patch information. The generated video bitstreams and the metadata may then be multiplexed together so as to generate a combined bitstream, called V3C bitstream.

[0032] FIG. 2 illustrates a schematic block diagram of an example of an image-based point cloud decoder 200 and renderer 210 according to a first embodiment.

[0033] In particular, the V-PCC decoder 200 may receive a combined bitstream including video bitstreams and the metadata representative of the input point cloud frame. A demultiplexer 220 may be applied to demultiplex the encoded information of the bitstream, for example, generated by an image-based point cloud encoder. Then, a 2D geometry videodecoder 230 may decode encoded information to derive at least one decoded geometry image. A 2D attribute video decoder 240 may decode encoded information to derive at least one decoded texture image. An occupancy map decoder 250 may decode encoded information to derive a decoded occupancy map. According to an embodiment, the 2D geometry video decoder, and / or 2D attribute video decoder and / or the occupancy map decoder may be a HEVC-based decoder. Possibly, metadata may also be derived 260 from the bitstream.

[0034] According to the first embodiment of FIG.2, the 3D reconstruction 270 and rendering 275 are let to the 3D engine (for instance Unity). The decoded geometry image, the at least one decoded texture image, the decoded occupancy map and the additional metadata may be synchronized and transmitted to the reconstructing and rendering engine which, depending on the position of the user and the parameters of the camera, renders the point cloud onto the screen and displays it to the user.

[0035] Accordingly, a 3D reconstruction module 270 in the 3D engine may derive the geometry of a reconstructed point cloud frame from the at least one decoded geometry image, the decoded occupancy map, and possible additional metadata. The 3D reconstruction module 270 may exploit the decoded occupancy map information in order to locate the nonempty pixels in the at least one decoded geometry image. Besides, the 3D reconstruction module 270 may derive the texture of the reconstructed point cloud frame from the geometry and the at least one decoded texture image. The rendering engine 275, depending on the position of the viewer and the parameters of the camera renders the reconstructed point cloud to the screen and displays it to the user.

[0036] When rendering a V-PCC point cloud, the number of points (or vertices) to render can dramatically impact the performance of the application, typically on less than high-end mobile devices or on computationally intensive hardware like Mixed Reality glasses. Reducing the number of points to render ensures a real time execution but generates a loss in the perceptual quality. The PCT application number PCT / EP2023 / 057567 by the same applicant, discloses a decimation method that allows reconstructing the V-PCC bitstreams by different ways based on the position of the viewer. More specifically, PCT application number PCT / EP2023 / 057567 discloses that blocks (typically a region of 16*16 pixels used for patch metadata coding in V- PCC) are reconstructed at variable resolutions according to the position of the camera in charge of rendering the decoded point cloud. This process will generate fewer points for the parts that are less visible and keep all the point for the parts facing and close to the viewer. Advantageously this method optimizes the complexity of both the reconstructing and the rendering of the point cloud while limiting the impact on perceptual quality of the rendered point cloud.

[0037] As shown in FIG. 2, the 3D reconstruction process 270 includes a block decimationprocess 280 that exploits viewport information 290 about where the decoded point cloud will be rendered at the renderer on a display screen. In this embodiment, the renderer 210 does not generate all points if the points are not present or visible in the rendered scene, or if the decimation does not have a significant impact on the perceived visual quality of rendering. Each block could be reconstructed at various resolutions, or decimation levels, as shown in FIG. 5A, FIG. 5B and FIG. 5C. For example, for a 16x16 block at various levels of decimation, the number of points will be:Decimation level 0: 16x16 = 256Decimation level 1 : 8x8 = 64 Decimation level 2: 4x4 = 16 Decimation level 3: 2x2 = 4 Decimation level 4: 1x1 = 1 Decimation level 5: 0x0 = 0 (no reconstruction).

[0038] The decimation ratios for the horizontal and vertical directions can also be different, and there can be separate decimation levels for the horizontal and vertical directions. For example, the number of points can be controlled by a horizonal decimation level and a vertical decimation level.

[0039] Based on the rendering information (viewport) transmitted by the renderer, it is possible to evaluate for each patch, for each block component of patches, or for each pixel of the patches, if the corresponding reconstructed point will be viewed by the user after the rendering process and which decimation level could be selected without significantly impairing the rendering quality. Based on the evaluation results, it is possible to decide whether to render or not the current area and at which resolution. The disclosed decimation process operates at 16x16 block and sets for each block a decimation level at which the block must be rendered. The choice to use 16x16 blocks is adapted to V-PCC bitstreams that uses 16x16 block sizes by default to describe patches. The disclosed decimation process may be executed with other block sizes and the choice of the block size used may be taken according to the complexity of the process. This process must be tuned according to the rendering method used to display the points on the screen (point, point with OpenGL point size, circle, splat, quad, ...) to adjust the decimation level to the size of the area covered by the projection of one point. However, it may happen that after decimation, one 3D point project on more than one pixel on the screen. The decimated point cloud may be rendered with holes if the size of the area covered by the projection of one point is not adapted in the renderer.

[0040] In the first embodiment of FIG. 2, the renderer 210 is a device with limited processing capabilities. As shown on FIG. 2, the at least one 2D texture images (for V-PCC decoded videos) and a buffer (for V-PCC metadata) are outputted synchronized inside a native plugin,the 3D reconstruction and rendering are let to the 3D engine (for instance Unity). In that case, if the block decimation is applied, the 3D engine is able to directly adapt the size of points of the rendered point cloud (as it knows how decimation has been done during the 3D reconstruction) and the display buffer view of the device.

[0041] FIG. 3 illustrates a schematic block diagram of an example of an image-based point cloud decoder 300 including a renderer 320, in connection with an application (i.e. a 3D engine or a display application) 310 that displays the output of the rendering process done in the decoder 300 according to a second embodiment.

[0042] In the second embodiment of FIG. 3, the decoder 300 oversees every step from the V- PCC decoding, synchronization of buffers to the 3D reconstruction 370. The 3D reconstruction 370 may implement the block decimation and the rendering. The decoder 300 directly provides the 3D engine with the 2D texture images (one for depth and one for attribute) for render on the device’s screen. This embodiment is particularly well adapted to MIV use case (whose concept is to render a 3D scene directly on the display) and to the willingness to keep shader code internal so a 3D engine can do a naive rendering with this 2D textures using straightforward shaders. In that case, if the block decimation process 380 is applied, the rendering 320 is able to adapt the point size as it internally knows how decimation has been done regarding the viewport information 390 given by the 3D engine 310.

[0043] However, the second embodiment raises some issue while integrating Augmented Reality AR into the final scene. To integrate AR into a final scene, the camera texture of a capturing device is made available to the 3D engine 310. For instance, Unity proposes a framework through AR Foundation package were the camera texture, name ARCameraBackground, is provided into the 3D engine 310. To be able to render the point cloud data (decoded in 300) merged into the AR camera background coming from 310, a copy and a blit (a merge) operations are needed inside the native plugin 300 to be able to prepare the 2D render textures ready for the display in the 3D engine 310. This process is heavy and impact real time performances of the V-PCC decoder platform.

[0044] FIG. 4 illustrates a schematic block diagram of an example of an image-based point cloud decoder 400 in connection with a renderer / 3D engine 410 according to a third embodiment. The decoder 400 comprises 2D decoders, synchronize and reconstruct the 3D model regarding the user viewpoint with the possibility to use or not the block decimation process. The decoder 400 directly outputs points of the point cloud with associated depth and attribute information, shared through 2D textures. The 3D engine 410 can then use its own shaders to render the point cloud textures (depth and attributes) to the display with the AR camera background. The shader is typically a program in GPU that feeds an 3D display engine with 3D model vertices and textures. Advantageously, the third embodiment no longer requiretransmitting data texture of camera background between the decoder 400 and the 3D engine 410. Render textures are handled into the 3D engine itself as data buffer and optimized graphics tools can be used inside the 3D engine itself to efficiently merge the point cloud data with the AR camera background (which keep the interoperability when available inside the 3D engine). In a nutshell, the shader merges, pixel by pixel, two 2D texture data structures representing the point cloud with the two 2D texture data representing the AR camera background. This process is very efficient as highly parallelizable with the GPU.

[0045] However, if the embodiment properly handles rendering of AR scene in device with limited computational power, since the 3D engine 410 has no information on whether block decimation process 480 has been performed during the reconstruction, the 3D engine 410 is no longer able to adapt the size of points of the rendered point cloud. This degrades the visual perception of the rendered point cloud in the final AR scene as hole may appear in the rendered point cloud with lower resolution.

[0046] At least one embodiment proposes transmitting information on point size to render for a 3D point of the point cloud according to whether or not block decimation process was applied and which resolution level was used during the 3D reconstruction.

[0047] According to at least one embodiment, a method for processing point cloud data comprising obtaining a resolution used for reconstructing point cloud data from a part of a geometry image; determining point size information responsive to the resolution used for reconstructing point cloud data and outputting point size information to a renderer.

[0048] FIG. 6 illustrates a schematic block diagram of a modified image-based point cloud decoder, in accordance with an embodiment. A point cloud decoder 600 may implement a block decimation 680, i.e. a change of resolution during the 3D reconstruction 670 of a point cloud PC 630. Point cloud data 630 typically comprises two 2D texture data structures, one for the depth and one for the attributes (color in Y, U, V format). Metadata buffer 640 may comprises additional information useful for the renderer. As explained above, for a 16x16 block at various levels of decimation, the number of points will be:Decimation level 0: 16x16 = 256, meaning that no decimation has been performed and all points within the block are reconstructed,Decimation level 1 : 8x8 = 64, meaning that a quarter of the points are reconstructed, Decimation level 2: 4x4 = 16 Decimation level 3: 2x2 = 4 Decimation level 4: 1 x1 = 1Decimation level 5: 0x0 = 0 (no reconstruction).

[0049] According to other variant embodiment, non-isomorphic decimation (ex: from 16x16 to 8x2), or even non power of two factors decimation (ex: from 16x16 to 5x9) may beimplemented in the block decimation 680. The decimation level is therefore representative of a resolution used for reconstructing point cloud data from at least a part of a geometry image in the decoder 600.

[0050] According to at least one embodiment, point size information is generated by the decoder 600 responsive to the resolution used for reconstructing point cloud data. Point size information may be transmitted to the renderer 620 with point cloud data 630 or with metadata 640 as non-limiting examples. Advantageously, the present principles allow improving the point cloud rendering quality by providing to the renderer the size of the area covered by the projection of 3D one point. According to an embodiment, point size information is generated at the block level, for instance for 16x16 block as the decimation process operates at 16x16 block and sets for each block a decimation level at which the block must be rendered. As the decimation process may be executed with other block sizes, the choice of the block size for point size information may be set accordingly.

[0051] According to a particular embodiment, point size information comprises a value of a size of a block of rendered pixels from a 3D point of a point cloud. In a variant, a value for the point size may be explicitly coded on 8 bits value per 3D point therefore allowing any value of 2D rendered pixels in the range [1 -256] per 3D point. According to another variant, a value for the point size may be coded as a power of 2 rather than the integer value. A value for the point size may be “3” instead of “8” as 8=2A3 for example whereAis the power function. For instance, this variant allows coding values 1 , 2, 4, 8, 16, 32, 64, 128, 256 on 3 bits. For instance, this variant allows coding point size values 0, 1 , 4, 32, 64, 256 on 3 bits. This variant is particularly adapted to the case where the decimation level is limited to powers of 2, each coded value of the point size information corresponding to a decimation level from 0 to 5. In another variant of the point size value, a default value may be defined, for instance the default value is set to 1 , i.e. one pixel is rendered per 3D point of a point cloud. Advantageously, in case no change of resolution was applied during 3D reconstruction, the point size value may be set to the default value meaning that all points within the block are reconstructed.

[0052] According to a particular embodiment, point size information may be signaled in a various data structure between the decoder and the 3D engine. From an implementation point of view, the point size information may be sent in any type of data container, such as buffers 640 or textures 630, indifferently. In a variant, point size information is signaled in a data structure (texture data structures) shared with point cloud data 630. For instance, the point size information may be sent in a shared data structure with the rest of the point cloud data 630 (e.g. color and position), or even in an unused texture color channel (for instance the alpha channel) if the point cloud data is stored in that way. This embodiment is well adapted to data organization in the 3D engine (as it uses shader programs based on vertexcomputation). Besides, this embodiment is well adapted to AR application where the 3D engine may obtain video data (AR camera background video) and wherein the rendering further comprises merging point cloud data with video data using the shaders to generate video data with rendered point cloud data for a display screen. In another variant, the point size information is sent in a metadata buffer 640, for instance the point size information may be signaled as supplemental enhancement information (SEI) message.

[0053] According to a particular embodiment, an indication that a change of resolution was applied during 3D reconstruction of the block or not is further generated by the decoder 600 and transmitted to the renderer 620. For instance, a flag indicating that a block decimation process has been performed on the point cloud to render is determined by the decoder 600. In that case, a flag, block_decimation_info_flag, may be delivered with the list of points sent to the renderer. In yet another variant, point size information is conditionally sent according to the flag. If the flag is set to true, the flag indicates that point size information is present for the point cloud. If the flag is false, the flag may further indicate that no point size information is present for the corresponding 3D point and the rendering is done with the default value (for instance with a value for point size set to 1 ). This flag may be set when a buffer / texture in charge to transmit point information are exchanged between the native application and the 3D engine.

[0054] FIG. 7 illustrates a schematic block diagram of a generic method 700 for processing point cloud data according to an embodiment. In a preliminary step 710, a reconstructed point cloud PC is obtained for instance from the 3D reconstruction module 670 of the decoder of FIG. 6. The 3D reconstruction module 670 may determine at least one part of point cloud data from a part of a geometry image with at least one of a part of an occupancy map, a part of an attribute image (i.e. texture) and possible additional metadata as previously detailed. The 3D reconstruction module may exploit the occupancy map in order to locate the non-empty pixels in the at least one geometry image. Besides, the 3D reconstruction module may derive the texture of the reconstructed point cloud from the geometry image and the at least one texture image. Besides, according to a particular feature, a block decimation module 680 may be applied during the 3D reconstruction in order to output 3D point cloud data at a resolution lower than the resolution of input geometry image. The 3D reconstruction and the decimation process are out of the scope of the present principles, only the output 3D reconstructed point cloud along with an information on the resolution used for reconstructing point cloud data are input of the processing method 700.

[0055] In a step 720, a resolution used for reconstructing point cloud data is obtained from the 3D reconstruction module. In a step 730, the 3D reconstructed point cloud is output to the renderer 620 on FIG. 6. Advantageously, in a step 740, point size information is generatedresponsive to the resolution used for reconstructing point cloud data. Advantageously, point size information provides to the renderer the size of the area covered by the projection of one 3D point of the reconstructed point cloud. As explained above, point size information may comprise a value of a size of a block of rendered pixels from a 3D point of a point cloud. In an optional step 750, an indication may be generated responsive to determining that a change of a resolution occurred during reconstruction of point cloud data. The indication that a change of resolution is used for reconstructing point cloud data may be a flag set to 0 in case no decimation process is applied and set to 1 in case a decimation process is applied. According to a particular variant, point size information is generated on condition that a change of resolution is used for reconstructing point cloud data. Advantageously, this variant avoids transmitting point size information which may be set to the default value in case of no resolution change.

[0056] FIG. 8 illustrates a schematic block diagram of a generic method 800 for rendering point cloud data according to an embodiment. In a first step 810, a reconstructed point cloud PC is obtained for instance from the 3D reconstruction module 670 of the decoder of FIG. 6. In a step 820, point size information for rendering point cloud data is obtained. In a step 840, the point cloud data is rendered using point size information. Advantageously, the rendering process may use splatting to enlarge points on the screen display or any other rendering processes that increase the sizes of the areas covered by the points on the screen after projection. In a final, step (not shown on FIG. 8), the rendered point cloud data is output to a screen display. Although, not described, the method 800 may implement any of the variants described for signaling point size information, in particular the flag. In another variant, in a step 830, video data in which the point cloud may be rendered is obtained, for instance from the camera of the device which implement the method. Then, the rendering 840 may further comprise merging the rendered point cloud with the video data, for example, by using the shaders of the rendering. This embodiment is particularly well adapted to AR application.

[0057] Several embodiments are now described for point size information used in the generic method for processing point cloud data, the embodiments may be arranged together according to any combination of the several embodiments. In the following, various examples of syntax and semantic for point size information are disclosed.

[0058] According to a first embodiment, the point size is explicitly defined for each reconstructed point after block decimation. According to another variant, point size information is generated on condition that a change of resolution is used for reconstructing point cloud data. For instance, the syntax and semantic for point size information structure may be defined as follow:Table 1 : Point size information syntax

[0059] Where:The syntax element block_decimation_info_flag, if true, indicates that point size information structure is present for the block. If false, it indicates that no block information is present for the corresponding block and the rendering is done, for instance using the default value of the point size.The value point_size indicates the size of a point to render. Default value of point size is 1 .

[0060] According to a second embodiment, instead transmitting the explicit value of the point size, the point size may be inferred as a power of two from the block decimation level applied during the reconstruction process as follows: point_size = 2An with n C [0;4] where n is the decimation level.

[0061] Therefore, the value of the size of block of pixels for a 3D point of the point cloud may be defined as in table 2 below:Table 2: Point_size values for implicit mode based on decimation level

[0062] In that case, point size information is signaled as the block decimation level. The skilled in the art will appreciate that for decimation level set to 5, as no reconstruction is performed, it is not possible to define a size of a blocks for a non-existing 3D point cloud.

[0063] According to yet another variant, a flag may indicate a mode used to code the point size information that is, whether the point size value is explicitly coded or implicitly coded and derived from the decimation level. For instance, the syntax element point_size_information_mode may indicate the mode with which the point size may bedetermined. Default value is 0. The possible values are described in table 3 below:Table 3 : Point_size_information_mode values

[0064] For instance, syntax and semantics for point size information structure may be defined as follow:Table 4 : Point size information syntax

[0065] Where:The syntax element block_decimation_info_flag, if true, indicates that a point size information structure is present for the block. If false, it indicates that no block information is present for the corresponding block and the rendering is done as well as possible in a blind mode by the 3D engine. point_size_information_mode indicates the mode with which the point size may be determined. Default value is 0. The possible values are described in Table 3. point_size indicates the size of a point. Default value of point size is 1 . decimation level indicates the level of decimation inside the decimated block as described in Table 2. Unsigned integer using n bits, the number of bits varies in a manner dependent on the value of the element. Default value of point size is 0.

[0066] According to a third embodiment, instead of signaling the point size for all 3D points ina block, the point size information may be signaled per decimated block. In yet another variant, the point size information may be signaled for a group of decimated blocks. Then, for each decimated block, the decoder defines a function that will allow a renderer to know if block information is present to improve the point cloud quality. To indicate if point size information is sent or not to the rendering per block, a flag, block_decimation_info_flag, may be delivered with a list of points sent to the renderer.

[0067] For instance, syntax and semantic for point size information structure may be defined as follow:Table 5 : Point size information syntax

[0068] Where:The syntax element block_decimation_info_flag, if true, indicates that block information structure is present for the block. If false, it indicates that no block information is present for the corresponding block and the rendering is done as well as possible in a blind mode by the 3D engine. When block information is present:height indicates the height of the decimated block, width indicates the width of the decimated block, x specifies the quantized x coordinate value of the origin position of the 3D bounding box of the decimated block, y specifies the quantized y coordinate value of the origin position of the 3D bounding box of the decimated block, z specifies the quantized z coordinate value of the origin position of the 3D bounding box of the decimated block, point_size_information_mode indicates the mode with which the point size may be determined. Default value is 0. The possible values are described in Table 3. point_size indicates the size of the point. Default value of point size is 1 . decimation level indicates the level of decimation inside the decimated block as described in Table 2. Unsigned integer using n bits, the number of bits varies in a manner dependent on the value of the element. Default value of point size is 0.

[0069] According to a fourth embodiment, the size of the decimated block may be implicit. Accordingly, a default value is defined for the size of the decimated block. For instance, the default value may be 16*16 (as it is a typical unit size used in the V-PCC decoder) and shared by configuration or initialization. In that case, syntax and semantic for point size information structure may be defined as follow:

[0070] Where:The syntax element block_decimation_info_flag, if true, indicates that block information structure is present for the block. If false, it indicates that no block information is present for the corresponding block and the rendering is done as well as possible by the 3D engine, height indicates the height of each decimated block, width indicates the width of each decimated block,Each decimated block is defined as follow: x specifies the quantized x coordinate value of the origin position of the 3D bounding box of the decimated block, y specifies the quantized y coordinate value of the origin position of the 3D bounding box of the decimated block, z specifies the quantized z coordinate value of the origin position of the 3D bounding box of the decimated block, point_size_information_mode indicates the mode with which the point size may de dertermined. Default value is 0. The possible values are described in table 3. point_size indicates the size of the point. Default value of point size is 1 . decimation level indicates the level of decimation inside the decimated block as described in Table 2. Unsigned integer using n bits, the number of bits varies in a manner dependent on the value of the element. Default value of point size is 0.

[0071] According to a fifth embodiment, the origin of the decimated block may be implicit. Accordingly, a default origin (x,y,z) is set from a 3D scanning pattern as an extension of the 2D scanning pattern (e.g. raster scan order). A possible 3D scanning pattern is for instance described by Fryza, Tomas in the document “Scan order and huffman coding of 3D DCT coefficients." (2005). FIG. 8 illustrates a possible scan order for the 3D reconstructed model to render by a modified zig-zag method. In that case, syntax and semantic for point size information structure may be defined as follow:Table 7: Point size information syntax

[0072] Where:The syntax element block_decimation_info_flag, if true, indicates that block information structure is present for the block. If false, it indicates that no block information is present for the corresponding block and the rendering is done as well as possible by the 3D engine, height indicates the height of each decimated block. width indicates the width of each decimated block. scan_pattern_id is an id of possible 3D scanning patterns as the one shown in FIG. 8. xO specifies the initialization of the quantized x coordinate value where the scanning starts, yO specifies the initialization of the quantized y coordinate value where the scanning starts, zO specifies the initialization of the quantized z coordinate value where the scanning starts, Each decimated block is defined as follow: point_size_information_mode indicates the mode with which the point size can be computed. Default value is 0. The possible values are described in table 3. point_size indicates the size of the point. Default value of point size is 1 .decimation level indicates the level of decimation inside the decimated block as described in Table 2. Unsigned integer using n bits, the number of bits varies in a manner dependent on the value of the element. Default value of point size is 0.

[0073] Various numeric values are used in the present application. The specific values are for example purposes and the aspects described are not limited to these specific values.

[0074] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., such as, for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0075] The implementations and aspects described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed may also be implemented in other forms (for example, an apparatus or program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, for example, computers, cell phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0076] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0077] Additionally, this application may refer to “determining” various pieces of information. Determining the information may include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0078] Further, this application may refer to “accessing” various pieces of information. Accessing the information may include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0079] Additionally, this application may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information may include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0080] It is to be appreciated that the use of any of the following“and / or”, and “at least one of’, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.

[0081] As will be evident to one of ordinary skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry the bitstream of a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

Claims

CLAIMS1 . A method for processing point cloud data, comprising: obtaining a resolution used for reconstructing point cloud data from a part of a geometry image; determining point size information responsive to the resolution used for reconstructing point cloud data; and outputting point size information to a renderer.

2. The method of claim 1 , wherein the point size information is output in data structure shared with point cloud data.

3. The method of claim 1 , wherein the point size information is output in a metadata structure.

4. The method of claim 1 or 2-3, wherein the point size information is output at any level among a 3D point level or a block level.

5. The method of claim 1 or 2-4, wherein the point size information comprises a value of a size of a block of rendered pixels from a 3D point of a point cloud.

6. The method of claim 1 or 2-4, wherein the point size information is derived from a level of the resolution used for reconstructing point cloud data.

7. The method of claim 1 or 2-6, further comprising: determining a change of a resolution used for reconstructing point cloud data from a part of a geometry image; outputting an indication that a change of resolution was used for reconstructing point cloud data; and wherein responsive to the indication that a change of resolution is used for reconstructing point cloud data, outputting a point size information to a renderer.

8. An apparatus for processing point cloud data, comprising one or more processors, wherein the one or more processors are configured to: obtain a resolution used for reconstructing point cloud data from a part of a geometry image; determine point size information responsive to the resolution used for reconstructingpoint cloud data; and output point size information to a Tenderer.

9. A method for rendering point cloud data, comprising: obtaining point cloud data from a decoder; obtaining point size information for rendering point cloud data; rendering, using point size information, the point cloud data; and outputting rendered point cloud data.

10. The method of claim 9, wherein the point size information is output in data structure shared with point cloud data.1 1 . The method of claim 9, wherein the point size information is output in a metadata structure.

12. The method of claim 9 or 10-11 , wherein the point size information is output at any level among a 3D point level or a block level.

13. The method of claim 9 or 10-12, wherein the point size information comprises a value of a size of a block of rendered pixels from a 3D point of a point cloud.

14. The method of claim 9 or 10-13, wherein the point size information is derived from a level of a resolution used for reconstructing point cloud data.

15. The method of claim 9 or 10-14, further comprising: obtaining an indication that a change of resolution is used for reconstructing point cloud data; and wherein responsive to the indication that a change of resolution was used for reconstructing point cloud data, decoding point size information.

16. The method of claim 9 or 10-15, wherein rendering the point cloud data comprising increasing a size of an area covered by a 3D point on a screen after projection according to point size information.

17. The method of claim 9 or 10-16, further comprising: obtaining video data; wherein rendering, using point size information, the point cloud data further comprisingshading point cloud data with video data; and outputting video data with rendered point cloud data.

18. An apparatus for rendering point cloud data in media content, comprising one or more processors, wherein the one or more processors are configured to obtain point cloud data from a decoder; obtain point size information for rendering point cloud data; render, using point size information, the point cloud data; and output rendered point cloud data.

19. A non-transitory computer readable medium comprising instructions which, when the instructions are executed by a computer, cause the computer to perform the method of any of claims 1 -7 or 9-17.

Images