Processing point cloud

Abstract

To provide a method, a device, and a non-transitory computer readable medium for using point clouds for various purposes such as culture heritage / buildings.SOLUTION: A method for adding at least one 3D sample to a point cloud frame and for assigning a color-coding mode to the at least one 3D sample, includes: assigning, when a stuffing 3D sample is to be added, the color-coding mode to the stuffing 3D sample, the color-coding mode indicating if color information associated with the at least one 3D sample is explicitly encoded in a bitstream or if the color information is implicit; and encoding, if an explicit color-coding mode is associated with the stuffing 3D sample, the color information in the bitstream.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] At least one of the present embodiments generally relates to processing point clouds. [Background technology]

[0002] SUMMARY OF THE DISCLOSURE This section is intended to introduce the reader to various aspects of art that may be related to various aspects of at least one of the present embodiments described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of at least one embodiment.

[0003] Point clouds can be used for various purposes such as cultural heritage / architecture, to scan objects like statues or buildings in 3D and share the spatial configuration of the object without sending or visiting the object. Also, in case the object may be destroyed, for example a temple may be destroyed by an earthquake, the point cloud is a way to ensure that the knowledge of the object is preserved. Such point clouds are usually static, color-coded and huge.

[0004] Another use case is in topography and mapmaking, where the use of 3D representations allows maps that are not limited to flat surfaces and may include relief. Google Maps is currently a good example of a 3D map, but it uses meshes rather than point clouds. Nevertheless, point clouds may be the preferred data format for 3D maps, and such point clouds are usually static, color-coded, and large.

[0005] The automotive industry and autonomous vehicles are also areas where point clouds can be used. Autonomous vehicles need to be able to "explore" their environment and make good driving decisions based on the reality of their immediate neighborhood. Typical sensors like LIDAR (Light Detection and Ranging) generate dynamic point clouds that are used by decision engines. These point clouds are not intended for humans to see, they are usually small, not necessarily color-coded, and dynamic with a high capture frequency. These point clouds can have other attributes like reflectivity provided by LIDAR, when this attribute provides good information about the material of the detected object, which can help make decisions.

[0006] Virtual reality and immersive worlds have been a hot topic lately and are predicted by many to be the future of 2D flat video. The basic idea is to immerse the viewer in the environment that surrounds them, as opposed to standard TV, where the viewer can only see the virtual world in front of him or her. There are several degrees of immersion, depending on the viewer's freedom within the environment. Point clouds are a good candidate format for delivering virtual reality (VR) worlds.

[0007] In many applications, it is important to be able to deliver dynamic point clouds to end users (or store them in a server) while consuming only a reasonable amount of bitrate (or storage space for storage applications) while maintaining an acceptable (or preferably very good) quality of experience. Efficient compression of these dynamic point clouds is a key point for the practical implementation of many immersive world delivery networks.

[0008] At least one embodiment has been devised with the above in mind. Summary of the Invention

[0009] The following presents a simplified summary of at least one of the present embodiments to provide a basic understanding of some aspects of the present disclosure. This summary is not an extensive overview of the embodiments. It is not intended to identify key or essential elements of the embodiments. The following summary merely presents some aspects of at least one of the present embodiments in a simplified form as a prelude to the more detailed description provided elsewhere in this document.

[0010] According to a general aspect of at least one embodiment, a method is provided for adding at least one 3D sample to a point cloud and assigning a coloring mode to the at least one 3D sample, the coloring mode indicating whether color information associated with the at least one 3D sample is explicitly coded in a bitstream or whether the color information is implicit.

[0011] One or more of at least one of the present embodiments also provide a device, a computer program, a computer-readable storage medium, and a signal.

[0012] The particular nature of at least one of the present embodiments, as well as other objects, advantages, features, and applications of the at least one of the present embodiments, will become apparent from the following description of examples taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0013] In the drawings, some example embodiments are illustrated.

[0014] [Figure 1] 1 shows a schematic block diagram of an example of a two-layer based point cloud coding structure according to at least one of the present embodiments. [Diagram 2] 2 shows a schematic block diagram of an example of a two-layer based point cloud decoding structure according to at least one of the present embodiments; [Diagram 3]1 shows a schematic block diagram of an example image-based point cloud encoder according to at least one of the present embodiments. [Figure 3a] 3 illustrates a schematic of an exemplary patch generation according to step 3100. [Figure 4] 40 shows a schematic block diagram of an example image-based point cloud decoder 4000 according to at least one of the present embodiments. [Diagram 5] 13 illustrates a schematic example of a syntax for a bitstream representing a base layer BL according to at least one of the present embodiments. [Figure 6] 1 shows a schematic block diagram of an example system in which various aspects and embodiments may be implemented. [Figure 7] 1 shows a schematic block diagram of an example of a method for adding at least one stuffing 3D sample to a point cloud frame PCF according to at least one of the present embodiments. [Figure 7a] 8 shows a schematic block diagram of an example embodiment of the method of FIG. 7. [Figure 7b] 1 shows a schematic block diagram of an example of a method for adding at least one stuffing 3D sample to a point cloud frame PCF according to at least one of the present embodiments. [Figure 8] 8A-8C show schematic diagrams of examples of the connected components of FIG. 3A in accordance with at least one embodiment of the method of FIG. 7; [Figure 9] An example of sub-step 7110 is shown diagrammatically. [Figure 10] An example of sub-step 7110 is shown diagrammatically. [Figure 11] An example of a variation of the embodiment of step 7200 is shown diagrammatically. [Figure 12] An example of a variation of the embodiment of step 7200 is shown diagrammatically. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] At least one of the present embodiments will be described more fully hereinafter with reference to the attached figures, in which at least one example of the present embodiments is shown. However, an embodiment may be embodied in many alternative forms and should not be construed as being limited to the examples set forth herein. It should therefore be understood that there is no intention to limit the embodiments to the particular forms disclosed. On the contrary, the present disclosure is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure.

[0016] Where a figure is presented as a flow diagram, it will be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it will be understood that the figure also provides a flow diagram of the corresponding method / process. Similar or identical elements are referred to using the same reference numerals.

[0017] The aspects described and contemplated below can be implemented in many different forms. Figures 1-12 below provide some embodiments, but other embodiments are contemplated and discussion of Figures 1-12 is not intended to limit the scope of the implementations.

[0018] At least one of the aspects relates generally to encoding and decoding point clouds, and at least one other aspect relates generally to transmitting the generated or encoded bitstream.

[0019] More precisely, the modules can be modified using various methods and other aspects described herein. These modifications can be performed immediately after reconstructing the geometry of the point cloud or immediately after reconstructing the geometry and assigning color information. The patch generation module PGM (step 3100 in FIG. 3), the texture image generator TIG (step 3400 in FIG. 3) and the geometry generation module GGM (step 4500 in FIG. 4) are examples of potential modification modules.

[0020] For example, the patch generation module PGM (step 3100 in FIG. 3) adds at least one stuffing 3D sample and explicitly encodes (or does not encode) color information associated with said at least one stuffing 3D sample. The texture image generator TIG (step 3400 in FIG. 3) or the geometry generation module GGM (step 4500 in FIG. 4) can assign the interpolated or decoded color information to the stuffing 3D sample.

[0021] Furthermore, the present aspects are not limited to MPEG standards such as MPEG-I Part 5 related to point cloud compression, but may be applied, for example, to other standards and recommendations, whether existing or developed in the future, and extensions of any related standards and recommendations, including MPEG-I Part 5. Unless otherwise indicated or technically excluded, the aspects described herein may be used individually or in combination.

[0022] In the following, image data refers to data, e.g. one or several arrays of 2D samples in a particular image / video format. A particular image / video format may specify information about pixel values ​​of an image (or video). A particular image / video format may also specify information that may be used by a display and / or any other device to, e.g., visualize and / or decode the image (or video). An image generally includes a first component, in the form of a first array of 2D samples, usually representing the luminance (or luma) of the image. An image may also include a second and a third component, in the form of another array of 2D samples, usually representing the chromaticity (or chroma) of the image. Some embodiments represent the same information using a set of 2D arrays of color samples, such as the conventional three-color RGB representation.

[0023] A pixel value is represented in one or more embodiments by a vector of C values, where C is the number of components. Each value in the vector is typically represented using a number of bits that can define the dynamic range of the pixel value.

[0024] An image block refers to a set of pixels that belong to an image. The pixel values ​​of an image block (or image block data) refer to the values ​​of the pixels that belong to this image block. An image block may have any shape, but is typically rectangular.

[0025] A point cloud may be represented by a dataset of 3D samples in a 3D volumetric space, each having unique coordinates and may also have one or more attributes.

[0026] A 3D sample of this dataset may be defined by its spatial location (X, Y, and Z coordinates in 3D space) and possibly by one or more associated attributes, such as, for example, color expressed in RGB or YUV color space, transparency, reflectance, a two-component normal vector, or any feature that represents a characteristic of this sample. For example, a 3D sample may be defined by six components (X,Y,Z,R,G,B) or in other words (X,Y,Z,y,U,V), where (X,Y,Z) define the coordinates of a point in 3D space and (R,G,B) or (y,U,V) define the color of this 3D sample. Attributes of the same type may be present multiple times. For example, multiple color attributes may provide color information from different viewpoints.

[0027] A point cloud can be static or dynamic, depending on whether the cloud changes over time. An instance of a static or dynamic point cloud is commonly denoted as a point cloud frame. Note that for a dynamic point cloud, the number of 3D samples is typically not constant, but in contrast typically changes over time. More generally, a point cloud can be considered dynamic if something changes over time, such as the number of points, the location of one or more points, or any attribute of any point.

[0028] As an example, a 2D sample may be defined by six components (u,v,Z,R,G,B), or equivalently (u,v,Z,y,U,V). (u,v) define the coordinates of the 2D sample in the 2D space of the projection plane. Z is the depth value of the 3D sample projected onto this projection plane. (R,G,B) or (y,U,V) define the color of this 3D sample.

[0029] FIG. 1 shows a schematic block diagram of an example of a two-layer based point cloud encoding structure 1000 according to at least one of the present embodiments.

[0030] The two-layer based point cloud encoding structure 1000 may provide a bitstream B representing an input point cloud frame IPCF, possibly representing a frame of a dynamic point cloud, which may then be encoded by the two-layer based point cloud encoding structure 1000 independently of other frames.

[0031] Essentially, the two-layer based point cloud coding structure 1000 may provide the ability to structure the bitstream B as a base layer BL and an enhancement layer EL. The base layer BL may provide a lossy representation of the input point cloud frame IPCF, and the enhancement layer EL may provide a lossless representation by encoding isolated points not represented by the base layer BL. As a possible option, the enhancement layer EL may provide a higher quality (but lossy) representation of the input point cloud frame IPCF by encoding additional points not represented by the base layer BL.

[0032] The base layer BL may be provided by an image-based encoder 3000, as shown in Figure 3, which may provide geometry / texture images representing the geometry / attributes of the 3D samples of the input point cloud frame IPCF and may enable discarding orphaned 3D samples. The base layer BL may be decoded by an image-based decoder 4000, as shown in Figure 4, which may provide an intermediate reconstructed point cloud frame IRPCF.

[0033] Returning now to the two-layer based point cloud encoding 1000 of FIG. 1, a comparator COMP may compare the 3D samples of the input point cloud frame IPCF with the intermediate reconstructed point cloud frame IRPCF to detect / locate the missed / orphaned 3D samples. Then, an encoder ENC may encode the missed 3D samples and provide an enhancement layer EL. Finally, the base layer BL and the enhancement layer EL may be multiplexed together by a multiplexing device MUX to generate a bitstream B.

[0034] According to an embodiment, the encoder ENC may include a detector capable of detecting 3D reference samples of the intermediate reconstructed point cloud frame IRPCF and associating them with the missed 3D samples M. The encoder ENC may also be an image-based encoder.

[0035] For example, the 3D reference sample R associated with a missed 3D sample M may be the closest neighbor of M according to a given meter.

[0036] According to an embodiment, the encoder ENC can then encode the spatial positions of the missed 3D samples M, and their attributes, as differences determined according to the spatial positions and attributes of said 3D reference sample R.

[0037] In a variant, the differences may be coded separately.

[0038] For example, for a missed 3D sample M, using spatial coordinates x(M), y(M), and z(M), the x-coordinate difference Dx(M), the y-coordinate position difference Dy(M), the z-difference Dz(M), the R-attribute component difference Dr(M), the G-attribute component difference Dg(M), and the B-attribute component difference Db(M) can be calculated as follows: Dx(M)=x(M)-x(R), where x(M) is the x-coordinate of the 3D sample M in the geometry image given by FIG. 3, and similarly for R, respectively. Dy(M) = y(M) - y(R) where y(M) is the y-coordinate of the 3D sample M in the geometry image given by FIG. 3, and similarly for R, respectively. Dz(M) = z(M) - z(R) where z(M) is the z-coordinate of the 3D sample M in the geometry image given by FIG. 3, and similarly for R, respectively. Dr(M) = R(M) - R(R). where R(M) and R(R) are the r-color components of the color attributes of 3D samples M and R, respectively. Dg(M) = G(M) - G(R). where G(M) and G(R) are the g-color components of the color attributes of 3D samples M and R, respectively. Db(M) = B(M) - B(R). where B(M), B(R) are the b-color components of the color attributes of 3D samples M, R, respectively.

[0039] FIG. 2 shows a schematic block diagram of an example of a two-layer based point cloud decoding structure 2000 according to at least one of the present embodiments.

[0040] The operation of the two-layer based point cloud decoding structure 2000 depends on its capabilities.

[0041] A two-layer based point cloud decoding structure 2000 with limited capabilities can access only the base layer BL from the bitstream B by using a demultiplexing device DMUX, and then provide a faithful (but lossy) version RPCF of the input point cloud frame IPCF by decoding the base layer BL by a point cloud decoder 4000, as shown in FIG. 4.

[0042] The fully capable two-layer based point cloud decoding structure 2000 can access both the base layer BL and the enhancement layer EL from the bit stream B by using a demultiplexing device DMUX. As shown in Fig. 4, the point cloud decoder 4000 can determine a reconstructed point cloud frame RPCF from the base layer BL. The decoder DEC can determine a complementary point cloud frame CPCF from the enhancement layer EL. The combiner COM can then combine the reconstructed point cloud frame RPCF and the complementary point cloud frame CPCF together, thus providing a lossless (or higher quality) representation (reconstruction) CRPCF of the input point cloud frame IPCF.

[0043] FIG. 3 shows a schematic block diagram of an example image-based point cloud encoder 3000 according to at least one of the present embodiments.

[0044] The image-based point cloud encoder 3000 leverages existing video codecs to compress the geometric and texture (attribute) information of dynamic point clouds, by essentially converting the point cloud data into a set of distinct video sequences.

[0045] In certain embodiments, two videos, one for capturing the geometric information of the point cloud data and another for capturing the texture information, may be generated and compressed using an existing video codec, such as the HEVC Main Profile Encoder / Decoder (ITU-T H.265 ITU Telecommunication Standardization Sector (02 / 2018), Series H, i.e., Audiovisual and Multimedia Systems, Infrastructure for Audiovisual Services - Coding of Video Moving Pictures, High Efficiency Video Coding, Recommendation ITU-T H.265).

[0046] Additional metadata used to interpret the two videos is also typically generated and compressed separately, such additional metadata including, for example, an occupancy map OM and / or auxiliary patch information PI.

[0047] The generated video bitstream and metadata can then be multiplexed together to generate a combined bitstream.

[0048] Note that metadata typically represents a small amount of the overall information; the majority of the information is in the video bitstream.

[0049] An example of such a point cloud encoding / decoding process is given by the Test Model Category 2 Algorithm (also denoted as V-PCC), as specified in ISO / IEC JTC1 / SC29 / WG11 MPEG2018 / N18030, and N17996 (October 2018, Macau).

[0050] In step 3100, the patch generation module PGM may generate at least one patch by decomposing a dataset of 3D samples representing the input point cloud frame IPCF into 2D samples on a given projection plane using a strategy that provides the best compression.

[0051] A patch may be defined as a set of 2D samples.

[0052] For example, in V-PCC, normals for each 3D sample are first estimated, as described for example in Hoppe et al. (Hugues Hoppe, Tony DeRose, Tom Duchamp, John McDonald, Werner Stuetzle, Surface reconstruction from unorganized points, ACM SIGGRAPH 1992 Proceedings, 71-78). Then, an initial clustering of the input point cloud frame IPCF is obtained by associating each 3D sample with one of six orientation planes of a 3D bounding box that encloses the 3D samples of the input point cloud frame IPCF. More precisely, each 3D sample is clustered and associated with the orientation plane that has the closest normal (maximizing the dot product of the point normal and the surface normal). The 3D samples are then projected onto their associated planes. A set of 3D samples that form a connected region in their planes is called a connected component. A connected component is a set of at least one 3D sample with similar normals and the same associated orientation plane. The initial clustering is then refined by iteratively updating the cluster associated with each 3D sample based on its normal and the clusters of its nearest neighbors. The final step consists of generating one patch from each connected component, which is performed by projecting the 3D sample of each connected component onto the orientation plane associated with that connected component.

[0053] FIG. 3a shows a schematic example of patch generation according to an embodiment of step 3100, where a point cloud frame (here a 2D point cloud frame) is segmented into four connected components: a first connected component constituting a light gray rectangle (samples), a second connected component constituting a dark gray rectangle, a third connected component constituting a rectangle of diagonal lines, and a fourth connected component constituting a rectangle of points. As shown, the first connected component is projected onto a vertical "line" (in the illustrated example, a 2D sample (and not a 3D sample at the same time) is projected along a line (and not a plane at the same time)), so that some spatial locations within the projection line have more than one depth value, in this example up to three depth values. Then, three patches #1, #2, and #3 are generated to capture all the projected information (depth) for the first connected component.

[0054] The patches are associated with auxiliary patch information PI, which represents auxiliary patch information defined such that each patch interprets a projected 2D sample corresponding to geometric shape and / or attribute information.

[0055] In V-PCC, for example, the auxiliary patch information PI includes information such as: 1) information indicating one of six orientation planes of a 3D bounding box that encloses the 3D sample of the connected component; 2) information about the plane normal; 3) information determining the 3D position of the connected component with respect to the patch expressed in terms of depth, tangent shift, and both tangent shifts; and 4) coordinates (u0,v0,u1,v1) in the projection plane that define a 2D bounding box that encloses the patch.

[0056] In step 3200, the patch packing module PPM can map (place) at least one generated patch onto a 2D grid (also called canvas) without any overlap in a way that minimizes unused space and can ensure that every block of TxT (e.g., 16x16) of the 2D grid is associated with a unique patch. A given minimum block size TxT of the 2D grid can specify the minimum distance between distinct patches when placed on this 2D grid. The resolution of the 2D grid can depend on the input point cloud size, and its width W and height H as well as the block size T can be transmitted to the decoder as metadata.

[0057] The auxiliary patch information PI may further include information regarding the association between blocks of the 2D grid and patches.

[0058] In the V-PCC, the auxiliary information PI includes block to patch index information that determines the association between blocks of the 2D grid and patch indices.

[0059] A TxT block containing 2D samples belonging to a patch may be considered as an occupied block in the corresponding occupancy map OM. A block of the occupancy map OM may then indicate whether the block is occupied or not, i.e. whether it contains 2D samples belonging to a patch or not.

[0060] The image generation process (steps 3300 and 3400) exploits the mapping of at least one generated patch onto the 2D grid computed during step 3200 to store the geometry and texture of the input point cloud frame IPCF as an image.

[0061] In step 3300, the geometric image generator GIG can generate at least one geometric image GI from the input point cloud frame IPCF, the occupancy map OM and the auxiliary patch information PI. The geometric image generator GIG can exploit the occupancy map information to detect (locate) occupied blocks and thus non-empty pixels in the geometric image GI.

[0062] The geometry image GI may represent the geometry of the input point cloud frame IPCF, and may be, for example, a monochrome image of WxH pixels represented in YUV420-8 bit format.

[0063] To better handle the case where multiple 3D samples are projected (mapped) (along the same projection direction (line)) onto the same 2D sample in the projection plane, multiple images, called layers, can be generated. Thus, the 2D samples of a patch can have different depth values ​​D1,...,Dn associated with them, and multiple geometry images are generated.

[0064] In V-PCC, the 2D samples of a patch are projected onto two layers. The first layer, also called the near layer, may for example store the depth value D0 associated with the 2D sample with the lowest depth. The second layer, also called the far layer, may for example store the difference between the depth value D1 associated with the 2D sample with the highest depth and D0. The information stored by the second depth image is thus in the interval [0,Δ], which corresponds to depth values ​​in the range [D0,D0+Δ], where Δ is a user-defined parameter representing the surface thickness.

[0065] In this way, the second layer may contain high frequency features of significant relief, so it is very difficult for the second depth image to be encoded by using a traditional video coder, and therefore the depth values ​​are not well reconstructed from the decoded second depth image, resulting in a clearly visible poor quality of the geometry of the reconstructed point cloud frame.

[0066] According to an embodiment, the geometric image generation module GIG is able to encode (derive) depth values ​​associated with the 2D samples of the first and second layers by using the auxiliary patch information PI.

[0067] In V-PCC, the position of a 3D sample within a patch with a corresponding connected component can be expressed in terms of depth δ(u,v), tangent shift s(u,v), and bi-tangent shift r(u,v) as follows: δ(u,v)=δ0+g(u,v) s(u,v)=s0-u0+u r(u,v)=r0-v0+v where g(u,v) is the luma component of the geometry image, (u,v) is the pixel associated with the 3D sample on the projection plane, (δ0,s0,r0) is the 3D position of the corresponding patch of the connected component to which the 3D sample belongs, and (u0,v0,u1,v1) are the coordinates in that projection plane that define a 2D bounding box that contains the projection of the patch associated with that connected component.

[0068] Thus, the geometric image generation module GIG can encode (derive) the depth values ​​associated with the 2D samples of a layer (first and / or second) as luma components g(u,v) given by g(u,v)=δ(u,v)−δ0. Note that using this relationship, the 3D sample positions (δ0,s0,r0) can be reconstructed from the reconstructed geometric image g(u,v) using the accompanying auxiliary patch information PI.

[0069] According to an embodiment, the projection mode can be used to indicate whether the first geometric image GI0 may store depth values ​​of 2D samples of either the first or second layer, and whether the second geometric image GI1 may store depth values ​​associated with 2D samples of either the second or first layer.

[0070] For example, when the projection mode is equal to 0, the first geometric image GI0 may store depth values ​​of the 2D samples of the first layer, and the second geometric image GI1 may store depth values ​​associated with the 2D samples of the second layer. Conversely, when the projection mode is equal to 1, the first geometric image GI0 may store depth values ​​of the second 2D samples, and the second geometric image GI1 may store depth values ​​associated with the 2D samples of the first layer.

[0071] According to an embodiment, the frame projection mode can be used to indicate whether a fixed projection mode is used for all patches, or whether a variable projection mode is used, where each patch may use a different projection mode.

[0072] The projection mode and / or the frame projection mode may be transmitted as metadata.

[0073] A frame projection mode decision algorithm may be provided, for example, in section 2.2.1.3.1 of the V-PCC.

[0074] According to an embodiment, when the frame projection indicates that a variable projection mode may be used, the patch projection mode may be used to indicate the appropriate mode to use for projecting (reconstructing) the patch.

[0075] The patch projection mode may be transmitted as metadata and may possibly be information contained within the auxiliary patch information PI.

[0076] A patch projection mode decision algorithm is provided as an example in section 2.2.1.3.2 of the V-PCC.

[0077] The use of several geometry layers may still not adequately capture the geometry of the point cloud frame if the given number of layers is not sufficient, but increasing the number of layers increases the overall bit rate and therefore reduces the compression efficiency.

[0078] In step 3400, a texture image generator TIG may generate at least one texture image TI from an input point cloud frame IPCF, an occupancy map OM, auxiliary patch information PI, and at least one decoded geometry image DGI, i.e., the geometry of a reconstructed point cloud frame derived from the output of the video decoder VDEC (step 4200 in FIG. 4).

[0079] The texture image TI may represent the texture of the input point cloud frame IPCF and may be, for example, an image of WxH pixels represented in YUV420-8bit format.

[0080] The texture image generator TG can exploit the occupancy map information to detect (locate) occupied blocks and thus non-empty pixels in the texture image.

[0081] The texture image generator TIG may be adapted to generate a texture image TI and to associate said texture image with each geometry image / layer DGI.

[0082] According to an embodiment, the texture image generator TIG is able to encode (store) texture (attribute) values ​​T0 associated with the 2D samples of the first layer as pixel values ​​of a first texture image TI0 and texture values ​​T1 associated with the 2D samples of the second layer as pixel values ​​of a second texture image TI1.

[0083] Alternatively, the texture image generation module TIG can encode (store) the texture values ​​T1 associated with the 2D samples of the second layer as pixel values ​​of the first texture image TI0 and the texture values ​​D0 associated with the 2D samples of the first layer as pixel values ​​of the second geometric image GI1.

[0084] For example, the color of the 3D sample may be obtained as described in sections 2.2.3, 2.2.4, 2.2.5, 2.2.8, or 2.5 of the V-PCC.

[0085] According to an embodiment, a padding process may be applied on the geometry and / or texture images. The purpose of filling the white space between patches is to generate piecewise smooth images suitable for video compression.

[0086] Examples of image padding are provided in sections 2.2.6 and 2.2.7 of the V-PCC.

[0087] In step 3500, the video encoder VENC may encode the generated images / layers TI and GI.

[0088] In step 3600, the encoder OMENC may encode the occupancy map as an image, for example as detailed in section 2.2.2 of V-PCC. Lossy or lossless encoding may be used.

[0089] According to an embodiment, the video encoder ENC and / or OMENC may be a HEVC-based encoder.

[0090] In step 3700, the encoder PIENC may encode the auxiliary patch information PI and possibly additional metadata, such as the block size T, width W and height H of the geometry / texture image.

[0091] According to an embodiment, the auxiliary patch information may be differentially encoded (eg, as defined in section 2.4.1 of the V-PCC).

[0092] In step 3800, a multiplexing device may multiplex together the generated video bitstreams that are the output of steps 3500, 3600, and 3700 to generate a bitstream represented by the base layer BL. Note that the metadata information represents a small percentage of the entire bitstream; most of the information is compressed using a video codec.

[0093] FIG. 4 shows a schematic block diagram of an example image-based point cloud decoder 4000 according to at least one of the present embodiments.

[0094] In step 4100, the demultiplexing device DMUX has access to the coded information of the bitstream representing the base layer BL.

[0095] In step 4200, the video decoder VDEC can decode the encoded information to derive at least one decoded geometry image DGI and at least one decoded texture image DTI.

[0096] In step 4300, a decoder OMDEC can decode the encoded information to derive a decoded occupancy map DOM.

[0097] According to an embodiment, the video decoders VDEC and / or OMDEC may be HEVC-based decoders.

[0098] In step 4400, a decoder PIDEC can decode the encoded information to derive auxiliary patch information DPI.

[0099] In some cases, metadata may also be derived from the bitstream BL.

[0100] In step 4500, the geometry generation module GGM can derive the geometry RG of the reconstructed point cloud frame RPCF (or IRPCF) from at least one decoded geometry image DGI, the decoded occupancy map DOM, the decoded auxiliary patch information DPI and possibly additional metadata.

[0101] The geometry generation module GGM can exploit the decoded occupancy map information DOM to find non-empty pixels in at least one decoded geometric image DGI. The 3D coordinates of a reconstructed 3D sample associated with a non-empty pixel can then be derived from the coordinates of said non-empty pixel and from the values ​​of said reconstructed 2D sample.

[0102] According to an embodiment, the geometry generation module GGM is able to derive the 3D coordinates of the reconstructed 3D samples from the coordinates of the non-empty pixels.

[0103] According to an embodiment, the geometry generation module GGM can derive the 3D coordinates of the reconstructed 3D sample from the coordinates of the non-empty pixels, the values ​​of said non-empty pixels of one of the at least one decoded geometry images DG, the decoded auxiliary patch information and possibly from additional metadata.

[0104] The use of non-empty pixels is based on the relationship of the 2D pixels to the 3D samples. For example, using the projection in the V-PCC, the 3D coordinates of the reconstructed 3D sample in terms of the depth δ(u,v), tangent shift s(u,v), and bi-tangent shift r(u,v) can be expressed as follows: δ(u,v)=δ0+g(u,v) s(u,v)=s0-u0+u r(u,v)=r0-v0+v where g(u,v) is the luma component of the decoded geometry image DGI, (u,v) is the pixel associated with the reconstructed 3D sample, (δ0,s0,r0) is the 3D position of the connected component to which the reconstructed 3D sample belongs, and (u0,v0,u1,v1) are the coordinates in the projection plane that define the 2D bounding box that contains the projection of the patch associated with that connected component.

[0105] In step 4600, the texture generation module TGM may derive the texture of the reconstructed point cloud frame RPCF (or IRPCF) from the geometry RG and at least one decoded texture image DTI.

[0106] FIG. 5 illustrates generally an example syntax of a bitstream representing a base layer BL according to at least one of the present embodiments.

[0107] The bitstream includes a bitstream header BSH and at least one frame stream group GOFS.

[0108] The frame stream group GOFS includes a header HS, at least one syntax element OMS representing an occupancy map OM, at least one syntax element GVS representing at least one geometric image (or video), at least one syntax element TVS representing at least one texture image (or video), and at least one syntax element PIS representing auxiliary patch information, as well as other additional metadata.

[0109] In a variant, the frame stream group GOFS comprises at least one frame stream.

[0110] FIG. 6 shows a schematic block diagram illustrating an example of a system in which various aspects and embodiments may be implemented.

[0111] System 6000 may be embodied as one or more devices including various components described below and configured to perform one or more of the aspects described in this document. Examples of equipment that may form all or part of system 6000 include personal computers, laptops, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, head-mounted display devices (HMDs, see-through glasses), projectors (beamers), "immersive virtual reality caves" (systems including multiple displays), servers, video encoders, video decoders, post-processing output from video decoders, pre-processors providing input to video encoders, web servers, set-top boxes, and any other devices for processing point clouds, videos, or images, or other communication devices. Elements of system 6000 may be embodied, alone or in combination, in a single integrated circuit, multiple ICs, and / or separate components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 6000 may be distributed across multiple ICs and / or separate components. In various embodiments, the system 6000 may be communicatively coupled to other similar systems or other electronic devices, for example, via a communication bus or through dedicated input or output ports. In various embodiments, the system 6000 may be configured to implement one or more of the aspects described in this document.

[0112] The system 6000 may include at least one processor 6010 configured to execute instructions loaded therein, for example, to implement various aspects described herein. The processor 6010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 6000 may include at least one memory 6020 (e.g., volatile and / or non-volatile memory devices). The system 6000 may include a storage 6040, which may include non-volatile and / or volatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash, magnetic disk devices, and / or optical disk devices. The storage device 6040 may include, as non-limiting examples, internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0113] The system 6000 may include an encoder / decoder module 6030 configured to process data to provide encoded or decoded data, for example, and the encoder / decoder module 6030 may include its own processor and memory. The encoder / decoder module 6030 may represent a module or modules that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Furthermore, the encoder / decoder module 6030 may be implemented as a separate element of the system 6000 or may be incorporated into the processor 6010 as a combination of hardware and software, as is known to those skilled in the art.

[0114] Program code loaded into the processor 6010 or the encoder / decoder 6030 for performing various aspects described herein may be stored in the storage device 6040 and then loaded onto the memory 6020 for execution by the processor 6010. According to various embodiments, one or more of the processor 6010, the memory 6020, the storage device 6040, and the encoder / decoder module 6030 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, point cloud frames, encoded / decoded geometry / texture videos / images or portions of encoded / decoded geometry / texture videos / images, bitstreams, matrices, variables, and intermediate or final results from the processing of mathematical expressions, formulas, operations, and computational logic.

[0115] In some embodiments, memory within the processor 6010 and / or the encoder / decoder module 6030 may be used to store instructions and provide working memory for processes that may be performed during encoding or decoding.

[0116] However, in other embodiments, memory external to the processing device (e.g., the processing device can be either the processor 6010 or the encoder / decoder module 6030) may be used for one or more of these functions. The external memory may be the memory 6020 and / or the storage device 6040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory may be used to store the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory such as RAM may be used as working memory for video encoding and decoding operations, such as for MPEG-2 Part 2 (also known as ITU-T Recommendation H.262 and ISO / IEC 13818-2, also known as MPEG-2 Video), HEVC (High Efficiency Video Coding), or VVC (Versatile Video Coding).

[0117] Input to the elements of system 6000 may be provided via various input devices, as shown in block 6130. Such input devices include, but are not limited to, (i) an RF section that may receive RF signals transmitted over the air, for example by a broadcast station, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0118] In various embodiments, the input devices of block 6130 may have associated respective input processing elements known in the art. For example, the RF section may be associated with elements necessary to (i) select a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) downconvert the selected signal, (iii) band-limit again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to (for example) as a channel, (iv) demodulate the downconverted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired stream of data packets. The RF section of various embodiments may include one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexing device. The RF section may include a tuner that performs these various functions, including, for example, downconverting a received signal to a lower frequency (e.g., an intermediate frequency or a nearby baseband frequency) or to baseband.

[0119] In one set-top box embodiment, the RF section and its associated input processing elements can receive RF signals transmitted over a wired (e.g., cable) medium. The RF section can then perform frequency selection by filtering, downconverting, and filtering again to a desired frequency band.

[0120] Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions.

[0121] Adding elements may include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters, etc. In various embodiments, the RF section may include an antenna.

[0122] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 6000 to other electronic devices across USB and / or HDMI connections. It should be appreciated that various aspects of input processing, e.g., Reed-Solomon error correction, may be implemented within a separate input processing IC, e.g., or within the processor 6010, as appropriate. Similarly, aspects of USB or HDMI interface processing may be implemented within a separate interface IC, e.g., or within the processor 6010, as appropriate. The demodulated, error corrected, and demultiplexed stream may be provided to various processing elements, including, e.g., the processor 6010, as well as an encoder / decoder 6030 operating in combination with memory and storage elements to process the data stream for presentation to an output device, as appropriate.

[0123] The various elements of the system 6000 may be provided in a unified housing in which the various elements may be interconnected and transmit data between each other using suitable connection arrangements 6140, such as internal buses known in the art, including an I2C bus, wiring, and printed circuit boards.

[0124] The system 6000 may include a communication interface 6050, which enables communication with other devices over a communication channel 6060. The communication interface 6050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 6060. The communication interface 6050 may include, but is not limited to, a modem or a network card, and the communication channel 6060 may be implemented in a wired and / or wireless medium, for example.

[0125] In various embodiments, data may be streamed to the system 6000 using a Wi-Fi network, such as IEEE 802.11. The Wi-Fi signal of these embodiments may be received via a communication channel 6060 and a communication interface 6050 adapted for Wi-Fi communication. The communication channel 6060 of these embodiments may typically be connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications.

[0126] Other embodiments may provide stream data to the system 6000 using a set-top box that delivers data via an HDMI connection in the input block 6130.

[0127] Yet other embodiments may use the RF connection of input block 6130 to provide stream data to system 6000 .

[0128] It should be appreciated that signaling may be accomplished in a variety of ways, for example, in various embodiments, information may be signaled to a corresponding decoder using one or more syntax elements, flags, etc.

[0129] System 6000 can provide output signals to a variety of output devices, including a display 6100, speakers 6110, and other peripheral devices 6120, which in various example embodiments may include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 3000.

[0130] In various embodiments, control signals may be communicated between the system 3000 and the display 6100, speaker 6110, or other peripheral device 6120 using signaling methods such as an AV link (audio / video link), CEC (customer electronics control), or other communication protocols that enable inter-device control with or without user involvement.

[0131] Output devices may be communicatively coupled to system 6000 via dedicated connections through respective interfaces 6070, 6080, and 6090.

[0132] Alternatively, an output device may be connected to the system 3000 via the communication interface 6050 using a communication channel 6060. The display 6100 and speakers 6110 may be integrated into a single unit along with other components of the system 6000 in an electronic device such as a television.

[0133] In various embodiments, the display interface 6070 may include a display driver, such as, for example, a timing controller (TCon) chip.

[0134] The display 6100 and speakers 6110 may alternatively be separate from one or more of the other components, for example, if the RF portion of the input 6130 is part of a separate set-top box. In various embodiments in which the display 6100 and speakers 6110 may be external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0135] FIG. 7 shows a schematic block diagram of an example of a method for adding at least one 3D sample (called stuffing 3D sample) to a point cloud frame PCF according to at least one of the present embodiments.

[0136] The point cloud frame PCF is, for example, a reconstructed point cloud frame derived from at least one decoded geometry image DGI, for example an intermediate reconstructed point cloud IRPCF, a reconstructed point cloud frame RPCF, or a reconstructed point cloud frame derived from at least one decoded geometry image DGI of FIG. 3.

[0137] In step 7100, the module determines whether at least one stuffing 3D sample needs to be added to the point cloud frame PCF.

[0138] According to an embodiment of step 7100, at least one stuffing 3D sample may be added from a current 3D sample of the point cloud frame PCF according to a difference between depth values ​​of adjacent 3D samples of the current 3D sample and a depth value of the current 3D sample.

[0139] If a stuffing 3D sample needs to be added in step 7200, a coloring mode is assigned to the stuffing 3D sample. The coloring mode indicates whether color information associated with the 3D sample is explicitly coded in the bitstream (explicit coloring mode) or whether the color information is implicit (implicit coloring mode), i.e., whether the color information is derived from color information associated with other 3D samples, including 3D samples of the point cloud frame PCT, and possibly other stuffing 3D samples.

[0140] Deciding whether the color information assigned to the stuffing 3D samples is explicitly coded or not is a trade-off between quality (if explicitly coded) and bitrate reduction (if implicit).

[0141] In step 7300, color information associated with a stuffing 3D sample is coded in the bitstream if an explicit coloring mode is associated with the stuffing 3D sample.

[0142] At step 7400, optionally, the color-coding mode is encoded in the bitstream.

[0143] The method of Figure 7 is particularly advantageous in terms of bit rate because the use of stuffing 3D samples reduces the number of geometry images (layers) required to adequately capture the geometry of the point cloud frame, as shown in Figure 8. Thus, encoding a very limited number of geometry images and stuffing them into 3D samples reduces the overall bit rate compared to methods that use only geometry images to capture the geometry of the point cloud frame.

[0144] FIG. 8 illustrates, in a schematic manner, an example of the connected components of FIG. 3 a processed according to at least one embodiment of the method of FIG. 7 .

[0145] For the first connected component represented by a light grey rectangle (FIG. 3a), instead of keeping (maximum) three patches to capture all projected information, a single layer is kept according to the method of FIG. 7 (patch #1), e.g., the patch corresponds to the first layer, which is the layer that stores the depth value D0 associated with the 2D sample with the lowest depth. Then, the projection of said patch #1 is restored, and the restored connected component is analyzed to detect potential holes that are then filled by at least one stuffing 3D sample. According to this example, three stuffing 3D samples P1, P2, and P3 are added.

[0146] As shown in FIG. 7a, according to an embodiment of step 7100, adding stuffing 3D samples includes the following sub-steps. In sub-step 7110, for at least one current 3D sample Cur of the point cloud PCF, the maximum depth difference (distance) deltaMax(Cur) between the depth value Depth(Cur) of the current 3D sample and the depth value Depth(Ni) of an adjacent 3D sample Ni of the current 3D sample is calculated. deltaMax(Cur)=max(Depth(Ni)-Depth(Cur)) where Ni denotes the i-th neighboring 3D sample of the current 3D sample Cur, and max(a,b) is the ordinary maximum function between two values ​​a and b.

[0147] According to an embodiment of sub-step 7110, the neighboring 3D samples Ni belong to an NxN neighborhood centered on the current 3D sample.

[0148] According to a variant of sub-step 7110, the projections of the neighbouring 3D samples Ni onto a geometry image plane and the projection of the current 3D sample Cur onto said geometry image plane belong to the same patch of the same geometry image (same layer).

[0149] According to a variant of sub-step 7110, the projections of the neighbouring 3D samples Ni onto a geometry image plane and the projection of the current 3D sample Cur onto said geometry image plane belong to different patches of the same geometry image (same layer).

[0150] According to a variant of sub-step 7110, the projection of the adjacent 3D samples Ni onto a first geometric image plane and the projection of the current 3D sample Cur onto a second geometric image plane, said first and second geometric image planes correspond to different geometric images (different layers), for example one of said geometric images stores the depth value D0 associated with the 2D sample having the lowest depth (i.e. the first layer in the V-PCC) and another of said geometric images stores the depth value D1 associated with the 2D sample having the highest depth (i.e. the second layer in the V-PCC).

[0151] According to a variant of sub-step 7110, the difference between the depth value Depth(Cur) of the current 3D sample and the depth value Depth(Ni) of an adjacent 3D sample Ni is ignored if the difference does not belong to a given range of values ​​defined by a lower bound LB and an upper bound UB.

[0152] For example, LB=0 and UB=4.

[0153] The maximum depth difference deltaMax(current) may be set to zero if the difference does not fall quite within a given range of values.

[0154] In sub-step 7120, at least one stuffing 3D sample is added at depth values ​​ranging from depth(Cur) to depth(Cur)+deltaMax(Cur)−1.

[0155] According to an embodiment of sub-step 7120, at least one stuffing 3D sample is added at the same 2D position as the 2D sample representing the projection of the current 3D sample onto the image plane, up to a maximum depth difference deltaMax(Cur). Thus, (deltMax(Cur)-1) stuffing 3D samples are added at depth values ​​ranging from Depth(Cur) to Depth(Cur)+deltaMax(Cur)-1.

[0156] When deltMax(Cur)=0 or =1, a stuffing 3D sample is also added at that 2D position.

[0157] FIG. 9 illustrates diagrammatically examples of sub-step 7110 for cases where deltaMax(Cur) is equal to 0, 1, 2, 3, and 4.

[0158] The light grey rectangle represents the current 3D sample Cur, the block rectangles represent the neighboring 3D samples Ni, and the crosses are the cases of added stuffing 3D samples. We assume that the current 3D sample Cur has a single neighboring 3D sample Ni for a simpler illustrative example.

[0159] In the first example EX1 of Fig. 9, the difference between depth Depth(Cur) and depth Depth(Ni) is equal to 5, which is greater than the upper bound UP (= 4). Then deltMax(Cur) = 0 and a single stuffing 3D sample is added as the 2D position of the 2D sample representing the projection of the current 3D sample.

[0160] In the second example EX2 of Fig. 9, the difference between depth Depth(Cur) and depth Depth(Ni) is equal to 4. Therefore, deltMax(Cur) = 4, and 3 stuffing 3D samples are added (up to the maximum depth difference deltaMax(Cur) - 1) at the same 2D position as the current 3D sample Cur.

[0161] In the third example EX3 of Fig. 9, the difference between depth Depth(Cur) and depth Depth(Ni) is equal to 3. Therefore, deltMax(Cur) = 3, and two stuffing 3D samples are added (up to the maximum depth difference deltaMax(Cur) - 1) at the same 2D position as the current 3D sample Cur.

[0162] In the fourth example EX4 of Fig. 9, the difference between depth Depth(Cur) and depth Depth(Ni) is equal to 2. Therefore, deltMax(Cur) = 2, and one stuffing 3D sample is added (up to the maximum depth difference deltaMax(Cur) - 1) at the same 2D position as the current 3D sample Cur.

[0163] In the fifth example EX5 of Fig. 9, the difference between depth Depth(Cur) and depth Depth(Ni) is equal to 1. Therefore, deltMax(Cur) = 1, and a single stuffing 3D sample is added as the 2D position of the 2D sample representing the projection of the current 3D sample.

[0164] In the sixth example EX6 of Fig. 9, the difference between depth Depth(Cur) and depth Depth(Ni) is equal to 0. Therefore, deltMax(Cur) = 0, and a single stuffing 3D sample is added as the 2D position of the 2D sample representing the projection of the current 3D sample.

[0165] In the seventh example EX7 of Fig. 9, the lower bound LB is equal to 0 and the upper bound is equal to 4. The difference between depth Depth(Cur) and depth Depth(Ni) is equal to -1. Therefore, deltMax(Cur) = 0, and a single stuffing 3D sample is added as the 2D position of the 2D sample representing the projection of the current 3D sample.

[0166] According to another embodiment of sub-step 7120, at least one stuffing 3D sample is added at the same 2D position as the 2D sample representing the projection of the current 3D sample onto the image plane up to a maximum depth difference deltaMax(Cur)-1, and another stuffing 3D sample is added at a 2D position as the 2D sample representing the projection of the adjacent 3D sample at a depth value equal to deltMax(Cur)-1. Thus, (deltMax(Cur)-1) stuffing 3D samples are added at depth values ​​ranging from Depth(Cur) to Depth(Cur)+deltaMax(Cur)-1.

[0167] FIG. 10 illustrates diagrammatically examples of sub-step 7110 for cases where deltaMax(Cur) is equal to 0, 1, 2, 3, and 4.

[0168] The light grey rectangle represents the current 3D sample Cur, the block rectangles represent the neighboring 3D samples Ni, and the crosses are the cases of added stuffing 3D samples. We assume that the current 3D sample Cur has a single neighboring 3D sample Ni for a simpler illustrative example.

[0169] In the second example EX2 of Fig. 10, the difference between depth Depth(Cur) and depth Depth(Ni) is equal to 4. Thus, deltMax(Cur) = 4, two stuffing 3D samples are added at the same 2D position as the current 3D sample Cur (up to a maximum depth difference deltaMax(Cur) - 2) and another stuffing 3D sample is added at a 2D position as the 2D sample representing the projection of the adjacent 3D sample set as deltMax(Cur) - 1.

[0170] In the third example EX3 of Fig. 10, the difference between depth Depth(Cur) and depth Depth(Ni) is equal to 3. Thus, deltMax(Cur) = 3, one stuffing 3D sample is added at the same 2D position as the current 3D sample Cur (up to the maximum depth difference deltaMax(Cur) - 2) and another stuffing 3D sample is added at the 2D position as the 2D sample representing the projection of the adjacent 3D sample set as deltMax(Cur) - 1.

[0171] The other examples remain the same as those described in FIG.

[0172] According to a variant of sub-step 7120, the maximum depth difference deltaMax(Cur) is at least equal to the following minimum value minDepth. deltaMax(Cur)=max((Depth(Ni)-Depth(Cur)),minDepth)

[0173] For example, minDepth may be equal to 1 or 2.

[0174] This variant ensures a minimum thickness of the reconstructed point cloud frame by adding at least one stuffing 3D sample, if necessary.

[0175] According to a variant of sub-step 7120, the value of minDepth may be defined by a rate / distortion optimization loop by estimating the cost of encoding at least one additional stuffing 3D sample.

[0176] According to an embodiment of sub-step 7120, the minimum value MinDepth is transmitted (signaled) in the bitstream.

[0177] According to an embodiment of step 7300, if an explicit coloring mode is assigned to the stuffing 3D sample, the color information is coded in the texture image TI.

[0178] This maximizes fidelity and reduces decoding complexity.

[0179] According to an embodiment of step 7200, a coloring mode value is assigned to a stuffing 3D sample Cur according to the maximum depth difference deltaMax(Cur) associated with that stuffing 3D sample Cur.

[0180] According to an embodiment of step 7200, if the maximum depth difference deltaMax(Cur) is equal to 0 or 1, assign the explicit coloring mode to the added stuffing 3D sample. If the maximum depth difference deltaMax(Cur) is greater than 1, assign the explicit coloring mode to the added stuffing 3D sample with the highest depth value. The added stuffing 3D sample with the highest depth value is the added stuffing 3D sample with a depth value equal to Depth(Cur)+deltaMax(Cur)-1. Then assign the implicit coloring mode to the remaining added stuffing 3D samples (if any).

[0181] FIG. 11 illustrates generally an example of an embodiment of step 7200 for the cases where deltaMax(Cur) is equal to 0, 1, 2, 3, and 4.

[0182] The light grey rectangle represents the current 3D sample Cur, the block rectangles represent the neighboring 3D samples Ni, the crosses are for added stuffing 3D samples, and the hashed rectangles represent added stuffing 3D samples with explicit color coding. We assume that the current 3D sample Cur has a single neighboring 3D sample Ni for a simpler illustrative example.

[0183] The seven examples of FIG. 9 are used here as non-limiting example cases.

[0184] In the first, sixth and seventh examples (EX1, EX6 and EX7) of FIG. 11, the maximum depth difference deltaMax(Cur) is equal to 0, then the explicit coloring mode is assigned to the stuffing 3D sample placed at the 2D position of the 2D sample representing the projection of the current 3D sample Cur.

[0185] In the second example EX2 of Fig. 11, the maximum depth difference deltaMax(Cur) is equal to 4, then the explicit coloring mode is assigned to the added stuffing 3D sample with the highest depth value, i.e. the added stuffing 3D sample with a depth value equal to Depth(Cur) + 3. The implicit coloring mode is assigned to the added stuffing 3D sample with a depth value equal to Depth(Cur) + 2 and to the added stuffing 3D sample with a depth value equal to Depth(Cur) + 1.

[0186] In the third example EX3 of Fig. 11, the maximum depth difference deltaMax(Cur) is equal to 3, then the explicit coloring mode is assigned to the added stuffing 3D sample with the highest depth value, i.e. the added stuffing 3D sample with a depth value equal to Depth(Cur)+2. The implicit coloring mode is assigned to the added stuffing 3D sample with a depth value equal to Depth(Cur)+1.

[0187] In the fourth example EX4 of Figure 11, the maximum depth difference deltaMax(Cur) is equal to 2, then the explicit coloring mode is assigned to the added stuffing 3D sample with the highest depth value, i.e. the added stuffing 3D sample with a depth value equal to Depth(Cur) + 1.

[0188] In the fifth example EX5 of Figure 11, the maximum depth difference deltaMax(Cur) is equal to 1, then the explicit coloring mode is assigned to the added stuffing 3D sample with the highest depth value, i.e. the added stuffing 3D sample with a depth value equal to Depth(Cur).

[0189] According to an embodiment of the variant of step 7200 shown in FIG. 12, if the maximum depth difference deltaMax(Cur) is equal to 0 or 1, an explicit coloring mode is assigned to the added stuffing 3D samples. If the maximum depth difference deltaMax(Cur) is greater than 1, an explicit coloring mode is assigned to the added stuffing 3D samples with intermediate depth values. The added stuffing 3D samples with intermediate depth values ​​are the added stuffing 3D samples with depth values ​​equal to ceil(Depth(Curr)+deltaMax(Curr) / 2). Then, an implicit coloring mode is assigned to the remaining added stuffing 3D samples (if any).

[0190] According to an embodiment of step 7300, if the color coding mode associated with a stuffing 3D sample does not indicate that color information is associated with the stuffing 3D sample, color information associated with the stuffing 3D sample is obtained by interpolating color information associated with neighboring 3D samples Ni of the stuffing 3D sample.

[0191] This embodiment reduces the bit rate.

[0192] FIG. 7b shows a schematic block diagram of an example of a method for adding at least one 3D sample, called stuffing 3D sample, to a point cloud frame PCF according to at least one of the present embodiments.

[0193] In step 7100 described above, the module determines whether at least one stuffing 3D sample needs to be added to the point cloud frame PCF.

[0194] If a stuffing 3D sample needs to be added, then in step 7500 the decoder decodes the bitstream to provide a color-coding mode associated with the at least one stuffing 3D sample.

[0195] If the color coding mode indicates that color information associated with the stuffing 3D sample is explicitly coded, then in step 7600, the decoder decodes the bitstream to provide color information associated with the at least one stuffing 3D sample.

[0196] If the color coding mode is implicit, then in step 7700 the module derives color information from color information associated with other 3D samples, including 3D samples of the point cloud frame PCT, and possibly other stuffing 3D samples.

[0197] The embodiments and variations of step 7100 shown in FIG. 7a apply to the method of FIG. 7b.

[0198] Additionally, the method of FIG. 7b includes decoding the bitstream to provide a minimum value, minDepth.

[0199] Any well-known interpolation function may be used to interpolate the color information associated with the neighboring 3D samples of the stuffing 3D sample.

[0200] For example, the color interpolation function can be the maximum, minimum, median or average value of the color information associated with the neighboring 3D samples Ni. A weighted average may be used with weights defined from the neighboring distances.

[0201] Another example of a color interpolation function is given by the following pseudocode: [Table 1] [Table 2] [Table 3]

[0202] The algorithm, presented above as pseudocode, performs the following steps for each current 3D sample from the reconstructed point cloud that requires color interpolation: -Finding the color and location of the closest colored 3D sample in the reconstructed point cloud frame, e.g., IRPCF; -Find the color and position of the closest colored 3D sample in the original point cloud frame, e.g. IPCF; - averaging the colors of all 3D samples based on their distance to the centroid (calculated by averaging the geometric coordinates of the 3D samples); - assigning such average color to the current 3D sample; and Correcting the color of the current 3D sample so that its deviation from the original 3D sample is minimized (within ±2 difference per color component).

[0203] The methods of FIG. 7 (encoding) and FIG. 7b (decoding) can be used in different use cases.

[0204] In a first use case, a single geometry image (layer) can be used to encode the geometry of the point cloud frame PCF. For example, the single geometry image can store a depth value D0 associated with the 2D sample with the lowest depth. Then, at least one stuffing 3D sample can be added according to the method of FIG. 7 or FIG. 7b. Finally, color information associated with the 3D samples of the point cloud PCF can be encoded as a first texture image TI0, and color information associated with the at least one stuffing sample is encoded as a second texture image TI1 when an explicit coloring mode is assigned to the at least one stuffing 3D sample.

[0205] In a second use case, two geometry images GI0 and GI1 (two layers) may be used to encode the geometry of the point cloud frame PCF. For example, the first geometry image may store a depth value D0 associated with the 2D sample with the lowest depth (first layer), and the second geometry image GI1 may store a depth value D1 associated with the 2D sample with the highest depth (second layer). Then, at least one stuffing 3D sample may be added according to the method of FIG. 7 or FIG. 7b. Finally, color information associated with the 3D samples of the point cloud PCF may be encoded as two texture images TI0 and TI1. The first texture image TI1 encodes color information for the first geometry image GI0, and the second texture image TI1 encodes color information for the second geometry image GI1. The third texture image TI2 may encode color information associated with at least one stuffing sample when an explicit coloring mode is assigned to the at least one stuffing 3D sample.

[0206] 1-12, various methods are described herein, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for the proper operation of the method, the order and / or use of specific steps and / or actions can be varied or combined.

[0207] Some examples are described with reference to block diagrams and operational flow charts. Each block represents a circuit element, module, or portion of code that includes one or more executable instructions for implementing a particular logical function(s). It should also be noted that in other implementations, the function(s) described in the blocks may be performed out of the order shown. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in the reverse order, depending on the functionality necessarily involved.

[0208] The embodiments and aspects described herein may be embodied in, for example, a method or process, an apparatus, a computer program, a data stream, a bit stream, or a signal. Even if discussed only in the context of a single form of embodiment (e.g., discussed only as a method), the discussed feature implementations may also be embodied in other forms (e.g., an apparatus or computer program).

[0209] The methods may be implemented, for example, in a processor, which generally refers to a processing device and includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communications devices.

[0210] Furthermore, the method may be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the implementation) may be stored on a computer-readable storage medium. The computer-readable storage medium may take the form of a computer-readable program product, which has computer-readable program code embodied thereon that is embodied in one or more computer-readable medium(s) and executable by a computer. A computer-readable storage medium as used herein may be considered a non-transitory storage medium given an inherent capability for storing information therein, as well as an inherent capability for retrieving information therefrom. The computer-readable storage medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. We will provide more specific examples of computer-readable storage media to which the present embodiment may be applied, but as will be readily understood by those skilled in the art, it should be understood that the following is merely an exemplary and non-exhaustive list. That is, a portable computer diskette, a hard disk, a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0211] The instructions may form an application program tangibly embodied on a processor-readable medium.

[0212] The instructions may be, for example, hardware, firmware, software, or a combination. The instructions may be found, for example, in an operating system, a separate application, or a combination of the two. Thus, a processor may be characterized as both a device configured to execute a process and a device that includes a processor-readable medium (such as a storage device) having instructions for executing a process. Furthermore, a processor-readable medium may store data values ​​generated by an implementation in addition to or in lieu of instructions.

[0213] The apparatus may be implemented, for example, in suitable hardware, software, and firmware. Examples of such apparatus include personal computers, laptops, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, head-mounted display devices (HMDs, see-through glasses), projectors (beamers), "immersive virtual reality experience devices (caves)" (systems including multiple displays), servers, video encoders, video decoders, post-processing output from video decoders, pre-processors providing input to video encoders, web servers, set-top boxes, and any other device for processing point clouds, videos, or images, or other communication devices. As will be apparent, the equipment may be mobile, even installed in a moving vehicle.

[0214] The computer software may be implemented by the processor 6010 or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may also be implemented by one or more integrated circuits. The memory 6020 may be of any type suitable for the technical environment, and may be implemented using any suitable data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memories, and removable memories. The processor 6010 may be of any type suitable for the technical environment, and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0215] As will be apparent to one of ordinary skill in the art, the implementations may generate a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored in a processor-readable medium.

[0216] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" may be intended to include the plural unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," as used herein, may specify the presence of the described, e.g., features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as "responsive to" or "connected" to another element, the element may be directly responsive to or directly connected to the other element, or intervening elements may be present. In contrast, when an element is referred to as "directly responsive to" or "directly connected" to another element, there are no intervening elements.

[0217] It is to be understood that the use of any of the symbols / terms " / ", "and / or", and "at least one of" may be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B), for example, in the case of "A / B", "A and / or B", and "at least one of A and B". As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A and B and C). This can be expanded as many times as the items listed, as would be apparent to one of ordinary skill in this and related arts.

[0218] Various numerical values, such as minDepth or surface thickness, may be used in this application. Particular values ​​may be, for example, objectives, and the described embodiments are not limited to these particular values.

[0219] Terms such as first, second, etc. may be used herein to describe various elements, but it will be understood that these elements are not limited by these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the teachings of the present application. The ordering between a first element and a second element is not meaningful.

[0220] "In one embodiment" or "in an embodiment" or "in one embodiment" or "in an embodiment," as well as references to other variations thereof, are frequently used to convey that a particular feature, structure, characteristic, etc. (described in connection with the embodiment / embodiment) is included in at least one embodiment / embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one embodiment" or "in an embodiment," as well as any other variations, may appear in various places throughout this application and are not necessarily all referring to the same embodiment.

[0221] Similarly, references to "according to an embodiment / example / implementation" or "in an embodiment / example / implementation," as well as other variations thereof, are frequently used herein to convey that a particular feature, structure, or characteristic (described in connection with an embodiment / example / implementation) may be included in at least one embodiment / example / implementation. Thus, appearances of the phrases "according to an embodiment / example / implementation" or "in an embodiment / example / implementation" in various places throughout this specification do not necessarily all refer to the same embodiment / example / implementation, nor do separate or alternative embodiments / examples / implementations necessarily exclude other embodiments / examples / implementations from one another.

[0222] Reference numerals appearing in the claims are for illustrative purposes only and shall have no limiting effect on the scope of the claims. Although not explicitly described, the present embodiments / examples and variations can be used in any combination or subcombination.

[0223] It should be understood that when a diagram is presented as a flow diagram, it also provides a block diagram of the corresponding apparatus. Similarly, when a diagram is presented as a block diagram, it should be understood that it also provides a flow diagram of the corresponding method / process.

[0224] Some figures include arrows on communication paths to indicate a primary direction of communication, however, it should be understood that communication may occur in a direction opposite to that shown by the arrow.

[0225] Various implementations involve decoding. "Decoding," as used herein, can encompass, for example, all or part of the processes performed on a received point cloud frame to generate a final output suitable for display. In various embodiments, such processes generally include one or more of the processes performed by an image-based decoder.

[0226] As a further example, in one embodiment, "decoding" may refer only to entropy decoding, in another embodiment, "decoding" may refer only to differential decoding, and in another embodiment, "decoding" may refer to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" may be intended to refer specifically to a subset of operations or to the broader decoding process in general may be clear based on the context of the specific description and is believed to be well understood by one of ordinary skill in the art.

[0227] Various implementations involve encoding. Similar to the above discussion regarding "decoding," as used in this application, "encoding" may encompass all or part of the processes performed, for example, on an input point cloud frame to generate an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an image-based decoder.

[0228] As a further example, in one embodiment, "encoding" may refer only to entropy encoding, in another embodiment, "encoding" may refer only to differential encoding, and in another embodiment, "encoding" may refer to a combination of differential and entropy encoding. Whether the phrase "encoding process" may be intended to refer specifically to a subset of operations or to a broader encoding process in general may be clear based on the context of the specific description and is believed to be well understood by one of ordinary skill in the art.

[0229] In addition, the application may refer to "determining" various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0230] Additionally, the application may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, obtaining information (e.g., from a memory), storing information, moving information, replicating information, calculating information, determining information, predicting information, or estimating information.

[0231] Additionally, the term "adding a stuffing 3D sample" refers to including a 3D sample in a point cloud frame. For example, when the 3D samples of a point cloud are stored as elements of a table, array, adding a stuffing 3D sample means storing information defining the stuffing 3D sample in a new element of the table (array).

[0232] In addition, the term "stuffing" indicates that a 3D sample is created according to at least one of the present embodiments.

[0233] Additionally, the term "assigning" information (e.g., color information or color-coding mode) to a 3D sample means associating said information with said 3D sample, i.e., for example, in a table (array) or by any other means.

[0234] In addition, the application may refer to "receiving" various pieces of information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" typically involves in some way, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0235] Also, as used herein, the word "signal" refers to, among other things, indicating something to a corresponding decoder. For example, in a particular embodiment, an encoder signals a particular coloring mode to the at least one stuffing 3D sample, which indicates whether color information assigned to the at least one stuffing 3D sample is explicitly coded in the bitstream or whether the color information is implicit, and the color information is interpolated. In this way, in an embodiment, the same parameters may be used at both the encoder side and the decoder side. Thus, for example, an encoder may send a particular parameter to a decoder (explicit signaling), so that the decoder can use the same particular parameter. Conversely, if the decoder already has a particular parameter, as well as other parameters, signaling may be used without sending (implicit signaling), simply allowing the decoder to know and select the particular parameter. By avoiding the transmission of any actual function, bit savings are realized in various embodiments. It should be understood that signaling may be achieved in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. Although the above relates to the verb form of the word "signal," the word "signal" may also be used as a noun herein.

[0236] A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made. For example, elements of different embodiments may be combined, supplemented, modified, or deleted to produce other embodiments. Moreover, those skilled in the art will understand that other structures and processes may be substituted for those disclosed, with the resulting embodiments performing at least substantially the same function(s), and in at least substantially the same way(s), to achieve at least substantially the same result(s) as the disclosed embodiments. Accordingly, these and other embodiments are contemplated by the present application.

Claims

1. 1. A method comprising: reconstructing at least one 3D sample of the point cloud using at least one geometric image storing depth values ​​of the 3D sample of the point cloud; determining a first depth difference as a maximum depth difference between a depth value of a neighboring 3D sample of a current 3D sample and a depth value of the current 3D sample, the neighboring 3D sample and the current 3D sample being projected onto the geometry image and belonging to a same patch of the geometry image; adding at least one 3D sample to the point cloud at the same 2D position as a 2D sample representing a projection of the current 3D sample up to the first depth difference; The method includes:

2. The method of claim 1 , wherein the point cloud is reconstructed by recovering a projection of the at least one geometric image.

3. 3. The method of claim 1 or 2, wherein the at least one 3D sample is added using depth values ​​that span a range of depth values ​​from a depth value of a current 3D sample of the point cloud to a depth value equal to the sum of the depth value of the current 3D sample and the first depth difference minus 1.

4. A method according to claim 1 or 2, wherein the first depth difference is greater than or equal to a first value, and the first value is 1 or 2.

5. 5. The method according to claim 1, wherein when determining the maximum depth difference between a depth value of a neighboring 3D sample of the current 3D sample and a depth value of the current 3D sample, the difference is ignored if the difference does not belong to a first range of values.

6. 6. The method of claim 1 , further comprising obtaining color information associated with at least one additional 3D sample by interpolating color information associated with other 3D samples of the point cloud.

7. 6. The method of claim 1, further comprising: assigning a color coding mode to at least one added 3D sample, the color coding mode indicating whether or not color information associated with the at least one added 3D sample is coded in a bitstream, and in response to a determination that the color coding mode indicates that the color information is not coded in the bitstream, the color information associated with the at least one added 3D sample is derived from color information associated with at least one 3D sample of the point cloud.

8. The method of claim 7 , wherein the color-coding mode is assigned to the added 3D sample according to the first depth difference associated with the added 3D sample.

9. 1. An apparatus comprising at least one processor, The at least one processor reconstructing at least one 3D sample of the point cloud using at least one geometric image storing depth values ​​of the 3D sample of the point cloud; determining a first depth difference as a maximum depth difference between a depth value of a neighboring 3D sample of a current 3D sample and a depth value of the current 3D sample, the neighboring 3D sample and the current 3D sample being projected onto the geometry image and belonging to a same patch of the geometry image; adding at least one 3D sample to the point cloud at the same 2D position as a 2D sample representing a projection of the current 3D sample up to the first depth difference; The apparatus is configured to:

10. The apparatus of claim 9 , wherein the point cloud is reconstructed by recovering a projection of the at least one geometric image.

11. 11. The apparatus of claim 9 or 10, wherein the at least one 3D sample is added using depth values ​​spanning a range of depth values ​​from a depth value of a current 3D sample of the point cloud to a depth value equal to the sum of the depth value of the current 3D sample and the first depth difference minus 1.

12. An apparatus as described in any one of claims 9 to 10, wherein the first depth difference is greater than or equal to a first value, and the first value is 1 or 2.

13. 13. The apparatus of claim 9, wherein when determining a maximum depth difference between a depth value of an adjacent 3D sample of the current 3D sample and a depth value of the current 3D sample, the difference is ignored if the difference does not belong to a first range of values.

14. 14. The apparatus of claim 9, wherein the at least one processor is configured to obtain color information associated with the at least one additional 3D sample by interpolating color information associated with other 3D samples of the point cloud.

15. A non-transitory computer readable medium comprising instructions for causing one or more processors to perform the method of any one of claims 1 to 8.

16. The method of claim 1, wherein the number of at least one 3D sample added to the point cloud at the same 2D position as the 2D sample representing the projection of the current 3D sample is equal to the maximum depth difference -1.

17. The method of claim 1, wherein in response to a determination that the maximum depth difference is equal to 0 or 1, a single 3D sample is added at the same 2D position as the 2D sample representing the projection of the current 3D sample.

18. The apparatus of claim 9, wherein in response to a determination that the maximum depth difference is equal to 0 or 1, a single 3D sample is added at the same 2D position as the 2D sample representing the projection of the current 3D sample.

Images