Point Cloud Processing

Abstract

At least one embodiment relates to a method for signaling a syntax element representing a local point reconstruction mode, the local point reconstruction mode representing at least one parameter defining a mode for reconstructing at least one point of a point cloud frame.

Description

[Technical field]

[0001] Technical Field At least one of the present embodiments generally relates to processing point clouds. [Background technology]

[0002] background This section is intended to introduce the reader to various aspects of art that may be related to various aspects of at least one of the present embodiments described and / or claimed below. The following discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of at least one embodiment.

[0003] Point clouds can be used for various purposes, such as cultural heritage / architecture, to 3D scan objects such as statues or buildings to share their spatial configuration without transporting or visiting them. It is also a way to ensure that knowledge of objects is preserved in case they are at risk of being destroyed, such as a temple collapsing due to an earthquake. Such point clouds are typically static, colored and large.

[0004] Another use case is in topography and cartography, where 3D representations can allow maps that are not limited to flat surfaces and can include relief. Today, Google Maps is a good example of a 3D map, but it uses meshes instead of point clouds. However, point clouds can be a suitable data format for 3D maps, and such point clouds are typically static, painted, and large.

[0005] The automotive industry and autonomous vehicles are also areas where point clouds can be used. Autonomous vehicles must be able to "explore" their environment to make good driving decisions based on the reality of their immediate surroundings. Typical sensors such as LIDAR (Light Detection and Ranging) generate dynamic point clouds that are used by decision engines. These point clouds are not intended for human viewing, are typically small, not necessarily colored, and are dynamic, requiring high frequency imaging. These point clouds may have other attributes, such as reflectivity provided by LIDAR, because these attributes can provide good information about the material of the detected object to help in making decisions.

[0006] Virtual reality and immersive worlds have become a hot topic in recent years, and many see them as the future of 2D flat video. The basic idea is to immerse the viewer in the environment that surrounds them, as opposed to standard television, where the viewer can only see the virtual world in front of them. There are several shades of immersion, depending on the viewer's degrees of freedom within the environment. Point clouds are a good candidate format for delivering virtual reality (VR) worlds.

[0007] It is important in many applications to be able to deliver dynamic point clouds to end users (or store them on a server) while maintaining an acceptable (or preferably very good) quality of experience by consuming only a moderate amount of bitrate (or storage space, in the case of storage applications). Efficient compression of these dynamic point clouds is a key aspect to making many immersive world delivery chains practical. Summary of the Invention [Means for solving the problem]

[0008] At least one embodiment has been conceived with the above in mind.

[0009] overview SUMMARY OF THE DISCLOSURE The following presents a summary of at least one embodiment in order to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of an embodiment. It is not intended to identify key or critical elements of an embodiment. The following summary is merely intended to present at least one aspect of the embodiment in a simplified form as a prelude to the more detailed description presented elsewhere in this specification.

[0010] According to a general aspect of at least one embodiment, there is provided a method comprising signaling in a bitstream a first syntax element representing a local point reconstruction mode, the local point reconstruction mode representing at least one parameter defining a mode for reconstructing at least one point of a point cloud frame.

[0011] According to one embodiment, said local point reconstruction mode is an index value of an entry in a look-up table, the look-up table defining a relationship between said index value and said at least one parameter.

[0012] According to one embodiment, the first syntax element is signaled either block-wise or patch-wise, where a patch is a set of at least one block of 2D samples representing an orthogonal projection of at least one 3D sample of the point cloud frame onto a projection plane.

[0013] According to one embodiment, a patch is a set of at least one block of at least one 2D sample representing an orthogonal projection of at least one 3D sample of the point cloud frame onto a projection plane, and the method further comprises signaling for each patch a second syntax element indicating whether a single first syntax element is signaled once for all blocks of the patch or whether a first syntax element is signaled for each block of the patch.

[0014] According to one embodiment, said second syntax element is signaled only if the patch contains more than a given number of blocks.

[0015] According to one embodiment, the method further comprises signaling a third syntax element representing a default local point reconstruction mode.

[0016] According to one embodiment, the third syntax element is signaled per point cloud frame and / or per patch, a patch being a set of at least one block of 2D samples representing an orthogonal projection of at least one 3D sample of the point cloud frame onto a projection plane.

[0017] According to one embodiment, a patch is a set of at least one block of multiple 2D samples representing an orthogonal projection of at least one 3D sample of the point cloud frame onto a projection plane, and the method further comprises signaling a fourth syntax element used for signaling a granularity of the first syntax element per block or per patch and the use of the third syntax element per patch.

[0018] According to one embodiment, the first syntax element is coded differently than at least one previously signaled first syntax element.

[0019] One or more of at least one of the present embodiments also provide an apparatus, a computer program, a signal, and a non-transitory computer-readable storage medium.

[0020] At least one particular nature of the present embodiment and said at least one other objects, advantages, features and uses of the present embodiment will become apparent from the following description of the examples given in conjunction with the accompanying drawings.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS The drawings show some example embodiments. [Brief description of the drawings]

[0022] [Figure 1] FIG. 2 shows a schematic block diagram of an example of a two-layer point cloud coding structure according to at least one of the present embodiments. [Diagram 2]FIG. 2 shows a schematic block diagram of an example of a two-layer point cloud decoding structure according to at least one of the present embodiments. [Diagram 3] FIG. 2 shows a schematic block diagram of an example of an image analysis point cloud encoder according to at least one of the present embodiments. [Figure 3a] Here is an example of a canvas containing two patches and their 2D bounding boxes. [Figure 4] FIG. 2 shows a schematic block diagram of an example of an image analysis point cloud decoder according to at least one of the present embodiments. [Diagram 5] 1 illustrates a schematic diagram of an example of a syntax for a bitstream representing a base layer BL according to at least one of the present embodiments. [Figure 6] FIG. 1 illustrates a schematic block diagram of an example of a system in which various aspects and embodiments may be implemented. [Figure 7] 1 shows a schematic block diagram of an example of a method for signaling a local point reconstruction mode according to at least one of the present embodiments. [Figure 8a] 7 shows an example of a look-up table LUT according to an embodiment of step 710. [Figure 8b] 7 shows an example of a look-up table LUT according to an embodiment of step 710. [Figure 8c] 13 shows an example of a syntax for signaling a look-up table LUT according to an embodiment of step 710. [Figure 8d] An example of this embodiment of step 710 is shown below. [Figure 8e] An example of this embodiment of step 710 is shown below. [Figure 9] 8 illustrates an example of an embodiment of the method of FIG. 7. [Figure 10] 8 illustrates an example of a variation of an embodiment of the method of FIG. [Figure 11] An example of a variation of FIG. 8c is shown. [Figure 12a] 8 illustrates several examples of an embodiment of the method of FIG. 7. [Figure 12b] 8 illustrates several examples of an embodiment of the method of FIG. 7. [Figure 13] An example of a variation of the method of FIG. 12b is shown. [Figure 14] An example of a variation of the method of FIG. 7 is shown. [Figure 15] 8 illustrates an example of an embodiment of the method of FIG. 7. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Detailed Description of the Invention At least one of the present embodiments will be described in more detail below with reference to the accompanying drawings showing at least one example of the present embodiment. However, an embodiment may be embodied in many alternative forms and should not be construed as being limited to the examples disclosed herein. It should therefore be understood that there is no intention to limit the embodiments to the particular forms disclosed. On the contrary, the present disclosure is intended to cover all modifications, equivalents and alternatives falling within the scope of this application.

[0024] Where a diagram is presented as a flow diagram, it should be understood that a corresponding apparatus block diagram is also presented. Similarly, where a diagram is presented as a block diagram, it should be understood that a corresponding method / process flow diagram is also presented.

[0025] Similar or identical elements in the figures are referred to with the same reference numbers.

[0026] Some figures represent syntax tables widely used in V-PCC that define the structure of bitstreams conforming to V-PCC. In these syntax tables, the term "..." represents parts of the syntax that are invariant with respect to the original definition given in V-PCC but omitted in the figures for ease of reading. Terms in bold in the figures indicate that the value of the term is obtained by parsing the bitstream. The right column of the syntax table indicates the number of bits required to encode the data of the syntax element. For example, u(4) indicates that 4 bits are used to encode the data, u(8) indicates that 8 bits are used to encode the data, and ae(v) indicates the context-adaptive arithmetic entropy coding syntax element.

[0027] The aspects described and discussed below can be implemented in many different forms. Figures 1-15 below show some embodiments, but other embodiments are possible and the content of Figures 1-15 is not intended to limit the breadth of implementations.

[0028] At least one of the aspects relates generally to point cloud encoding and decoding, and at least one other aspect relates generally to transmission of the generated or encoded bitstream.

[0029] More precisely, the various methods and other aspects described herein may be used to modify modules, such as modules associated with encoding metadata, such as occurs in patch information encoder 3700 of FIG. 3, and modules associated with decoding metadata, such as occurs in the reconstruction process in patch information decoder 4400 of FIG. 4 or geometry generation module 4500 of FIG. 4).

[0030] Furthermore, the aspects are not limited to MPEG standards such as MPEG-I Part 5 for point cloud compression, but may be applied, for example, to other standards and recommendations established now or in the future, and any extensions of such standards and recommendations, including MPEG-I Part 5. Unless otherwise indicated or technically precluded, aspects described in this application may be used individually or in combination.

[0031] In the following, image data refers to data, e.g. one or more arrays of 2D samples of a particular image / video format. A particular image / video format may specify information about pixel values ​​of an image (or video). A particular image / video format may also specify information that can be utilized, e.g. to visualize and / or decode the image (or video) by a display and / or any other device. An image typically includes a first component, usually in the form of a first 2D array of samples representing the luminance (or luma) of the image. An image may also include a second component and a third component, usually in the form of another 2D array of samples representing the chrominance (or saturation) of the image. Some embodiments represent the same information using a set of 2D arrays of color samples, such as a conventional three-color RGB representation.

[0032] In one or more embodiments, a pixel value is represented by a vector of C values, where C is the number of components. Each value in the vector is typically represented by a number of bits that can define the dynamic range of the pixel value.

[0033] An image block refers to a set of pixels that belong to an image. A pixel value of an image block (or image block data) refers to the value of a pixel that belongs to the image block. An image block can have any shape, but a rectangle is common.

[0034] A point cloud can be represented by a data set of 3D samples in a 3D volumetric space, each having unique coordinates and possibly having one or more attributes.

[0035] A 3D sample of a dataset can be defined by a spatial location (X, Y and Z coordinates in 3D space) and possibly one or more associated attributes such as color expressed in e.g. RGB or YUV color space, transmittance, reflectance, a two-component normal vector or any feature that represents a characteristic of the sample. For example, a 3D sample can be defined by six components (X, Y, Z, R, G, B) or even (X, Y, Z, y, U, V), where (X, Y, Z) define the coordinates of a point in 3D space and (R, G, B) or (y, U, V) define the color of the 3D sample. The same type of attribute can be present multiple times. For example, multiple color attributes can provide color information from different viewpoints.

[0036] A point cloud can be static or dynamic depending on whether the cloud changes over time. An instance of a static or dynamic point cloud is usually denoted as a point cloud frame. Note that for a dynamic point cloud, the number of points is generally not constant, but rather generally changes over time. More generally, a point cloud is considered dynamic if some attribute, such as the number of points, the location of one or more points, or any attribute of any point, changes over time.

[0037] As an example, a 2D sample can be defined by six components (u,v,Z,R,G,B) or even equivalently (u,v,Z,y,U,V). (u,v) define the coordinates of the 2D sample in the 2D space of the projection plane. Z is the depth value of the 3D sample projected onto the projection plane. (R,G,B) or (y,U,V) define the color of the 3D sample.

[0038] FIG. 1 shows a schematic block diagram of an example of a two-layer point cloud coding structure 1000 according to at least one of the present embodiments.

[0039] The two-layer point cloud coding structure 1000 can generate a bitstream B representing an input point cloud frame IPCF, possibly representing a frame of a dynamic point cloud, which can then be coded by the two-layer point cloud coding structure 1000 independently of other frames.

[0040] Essentially, the two-layer point cloud coding structure 1000 may provide the ability to construct a bitstream B as a base layer BL and an enhancement layer EL. The base layer BL may provide a lossy representation of the input point cloud frame IPCF, and the enhancement layer EL may provide a higher quality (possibly lossless) representation by encoding isolated points not represented by the base layer BL.

[0041] The base layer BL may be provided by an image analysis encoder 3000 as shown in Fig. 3, which may provide geometry / texture images representing the geometry / attributes of the 3D samples of the input point cloud frame IPCF, thereby allowing for the removal of isolated 3D samples. The base layer BL may be decoded by an image analysis decoder 4000 as shown in Fig. 4, which may provide an intermediate reconstructed point cloud frame IRPCF.

[0042] Returning to the two-layer point cloud encoding 1000 of FIG. 1, a comparator COMP can then compare the 3D samples of the input point cloud frame IPCF with the 3D samples of the intermediate reconstructed point cloud frame IRPCF to detect / locate missing / orphan 3D samples. An encoder ENC can then encode the missing 3D samples to provide an enhancement layer EL. Finally, the base layer BL and the enhancement layer EL can be multiplexed by a multiplexer MUX to generate a bitstream B.

[0043] According to one embodiment, the encoder ENC may include a detector capable of detecting 3D reference samples R of the intermediate reconstructed point cloud frame IRPCF and relating them to the missing 3D samples M.

[0044] For example, the 3D reference sample R associated with the missing 3D sample M may be the nearest neighbor of M according to a given measure.

[0045] According to one embodiment, the encoder ENC can then encode the spatial positions of the missing 3D samples M and their attributes as differences determined according to the spatial positions and attributes of said 3D reference samples R.

[0046] In one variation, these differences can be coded separately.

[0047] For example, for a missing 3D sample M with spatial coordinates x(M), y(M), and z(M), the x-coordinate position difference Dx(M), the y-coordinate position difference Dy(M), the Z-coordinate position difference Dz(M), the R-attribute component difference Dr(M), the G-attribute component difference Dg(M), and the B-attribute component difference Db(M) can be calculated as follows: Dx(M)=x(M)-x(R) where x(M) is the x coordinate of the 3D sample M or R in the geometry image shown in FIG. Dy(M) = y(M) - y(R) where y(M) is the y coordinate of the 3D sample M or R in the geometry image shown in FIG. Dz(M) = z(M) - z(R) where z(M) is the z coordinate of the 3D sample M or R in the geometry image shown in FIG. Dr(M)=R(M)-R(R) where R(M) or R(R) is the r color component of the color attribute of the 3D sample M or R, Dg(M) = G(M) - G(R) where G(M) or G(R) is the g color component of the color attribute of the 3D sample M or R, Db(M)=B(M)-B(R) where B(M) or B(R) is the b color component of the color attribute of the 3D sample M or R.

[0048] FIG. 2 shows a schematic block diagram of an example of a two-layer point cloud decoding structure 2000 according to at least one of the present embodiments.

[0049] The behavior of the two-layer point cloud decoding structure 2000 is capability dependent.

[0050] A limited-capability two-layer point cloud decoding structure 2000 can use a demultiplexer DMUX to access only the base layer BL from the bitstream B, and then decode the base layer BL by a point cloud decoder 4000 to provide a faithful (but lossy) version IRPCF of the input point cloud frame IPCF, as shown in FIG. 4.

[0051] The full-featured two-layer point cloud decoding structure 2000 can access both the base layer BL and the enhancement layer EL from the bitstream B using a demultiplexer DMUX. The point cloud decoder 4000 can determine an intermediate reconstructed point cloud frame IRPCF from the base layer BL as shown in Fig. 4. The decoder DEC can determine a complementary point cloud frame CPCF from the enhancement layer EL. The combiner COM can then provide a higher quality (possibly lossless) representation (reconstruction) CRPCF of the input point cloud frame IPCF by combining the intermediate reconstructed point cloud frame IRPCF and the complementary point cloud frame CPCF.

[0052] FIG. 3 shows a schematic block diagram of an example image analysis point cloud encoder 3000 according to at least one of the present embodiments.

[0053] The Image Analysis Point Cloud Encoder 3000 utilizes existing video codecs to compress the geometry and texture (attribute) information of dynamic point clouds by essentially converting the point cloud data into a set of distinct video sequences.

[0054] In certain embodiments, an existing video codec can be used to generate and compress two videos, one capturing the geometry information of the point cloud data and the other capturing the texture information. One example of an existing video codec is the HEVC Main Profile Encoder / Decoder (ITU (02 / 2018), ITU-T H.265 Telecommunications Standardization Sector in Series H: Audiovisual and Multimedia Systems, Infrastructure for Audiovisual Services - Coding of Motion Picture Video, High Efficiency Video Coding, Recommendation ITU-T H.265).

[0055] Additional metadata used to interpret the two videos is also typically generated and compressed separately, such additional metadata including, for example, an occupancy map OM and / or auxiliary patch information PI.

[0056] The generated video bitstream and metadata may then be multiplexed to generate a combined bitstream.

[0057] Note that metadata typically represents only a small amount of the overall information: most of the information is present in the video bitstream.

[0058] An example of such a point cloud encoding / decoding process is given in the Test Model Category 2 Algorithm (also denoted V-PCC) implementing the MPEG draft standard defined in ISO / IEC JTC1 / SC29 / WG11 MPEG2019 / w18180 (January 2019, Marrakesh).

[0059] In step 3100, the module PGM may generate at least one patch by decomposing the 3D samples representing the data set of the input point cloud frame IPCF into 2D samples on a projection plane using a strategy that provides the best compression.

[0060] A patch can be defined as a set of 2D samples.

[0061] For example, in V-PCC, normals are first estimated for each 3D sample, as described for example in Hoppe et al. (Hugues Hoppe, Tony DeRose, Tom Duchamp, John McDonald, Werner Stuetzle, Surface reconstruction from unorganized points. ACM SIGGRAPH 1992 Proceedings, 71-78). Then, an initial clustering of the input point cloud frame IPCF is obtained by associating each 3D sample with one of six oriented planes of a 3D bounding box that contains the 3D samples of the input point cloud frame IPCF. More precisely, each 3D sample is clustered and associated with the oriented plane that has the closest normal (i.e., that maximizes the dot product of the point normal and the plane normal). The 3D sample is then projected onto the associated plane. A set of 3D samples that form a connected region in the plane is called a connected component. A connected component is a set of at least one 3D sample with similar normals and the same associated oriented plane. The initial clustering is then refined by iteratively updating the clusters associated with each 3D sample based on the normals and the clusters of the nearest neighbors. The final step consists of generating one patch from each connected component, by projecting the 3D samples of each connected component onto the oriented plane associated with said connected component. The patches are associated with auxiliary patch information PI representing auxiliary patch information defined for each patch to interpret the projected 2D samples corresponding to geometry and / or attribute information.

[0062] In a V-PCC, for example, the auxiliary patch information PI includes information such as: 1) information indicating one of six oriented planes of a 3D bounding box containing a 3D sample of the connected component; 2) information related to the plane normal; 3) information determining the 3D position of the connected component relative to the patch, expressed as depth, tangential displacement and bi-tangential displacement; and 4) coordinates (u0, v0, u1, v1) in the projection plane that defines the 2D bounding box containing the patch.

[0063] In step 3200, the patch packing module PPM may map (place) at least one generated patch onto a 2D grid (also called canvas) without any overlaps that typically minimize unused space, ensuring that every T×T (e.g., 16×16) block of the 2D grid is associated with a unique patch. A given minimum block size T×T of the 2D grid may specify the minimum distance between different patches placed in the 2D grid. The resolution of the 2D grid may depend on the input point cloud size and its width W and height H, and the block size T may be transmitted to the decoder as metadata.

[0064] The auxiliary patch information PI may further include information related to the association of blocks of the 2D grid with patches.

[0065] In V-PCC, the auxiliary information PI may include block-to-patch index information (BlockToPatch) that determines the association between blocks of a 2D lattice and patch indexes.

[0066] Figure 3a shows an example of a canvas C that includes two patches P1 and P2 and their associated 2D bounding boxes B1 and B2. Note that the two bounding boxes may overlap in the canvas C, as shown in Figure 3a. The 2D grid (canvas partitioning) is only represented within the bounding boxes, but canvas partitioning also occurs outside these bounding boxes. The bounding boxes associated with the patches can be divided into TxT blocks, with T = 16 being typical.

[0067] The T×T blocks that contain 2D samples that belong to a patch can be considered as occupied blocks in the corresponding occupancy map OM. A block in the occupancy map OM thus indicates whether the block is occupied or not, i.e. whether it contains 2D samples that belong to the patch or not.

[0068] In Fig. 3a, occupied blocks are represented by white blocks and light grey blocks are represented as vacant blocks. The image generation process (steps 3300 and 3400) uses a mapping of at least one generated patch onto the 2D grid computed in step 3200 to save the geometry and texture of the input point cloud frame IPCF as an image.

[0069] In step 3300, the geometry image generator GIG can generate at least one geometry image GI from the input point cloud frame IPCF, the occupancy map OM and the auxiliary patch information PI. The geometry image generator GIG can use the occupancy map information to detect (locate) occupied blocks and thus non-empty pixels in the geometry image GI.

[0070] The geometry image GI may represent the geometry of the input point cloud frame IPCF and may be, for example, a monochrome image of W×H pixels represented in YUV420-8bit format.

[0071] To better handle the case of multiple 3D samples that are projected (mapped) onto the same 2D sample in the projection plane (along the same projection direction (line)), multiple images, called layers, may be generated. Thus, different depth values ​​D1,...,Dn may be associated with the 2D samples of the patch, and then multiple geometry images may be generated.

[0072] In V-PCC, the 2D samples of a patch are projected into two layers. The first layer, also called the near layer, may for example store depth values ​​D0 associated with the 2D samples having a shallower depth. The second layer, also called the far layer, may for example store depth values ​​D1 associated with the 2D samples having a deeper depth. Alternatively, the second layer may store difference values ​​between the depth values ​​D1 and D0. For example, the information stored by the second depth image may be in the interval [0,Δ], corresponding to depth values ​​in the range [D0,D0+Δ], where Δ is a user-defined parameter describing the thickness of the surface.

[0073] Thus, the second layer may contain significant contour-like high frequency features. Therefore, encoding the second depth image using a conventional video coder proves to be difficult, and therefore depth values ​​may not be well reconstructed from the decoded second depth image, leading to poor quality geometry in the reconstructed point cloud frame.

[0074] According to one embodiment, the geometry image generation module GIG is able to use the auxiliary patch information PI to encode (derive) depth values ​​associated with the 2D samples of the first and second layers.

[0075] In V-PCC, the position of a 3D sample within a patch with a corresponding connected component can be expressed in terms of depth δ(u,v), tangent shift s(u,v) and bi-tangent shift r(u,v) as follows: δ(u,v)=δ0+g(u,v) s(u,v)=s0-u0+u r(u,v)=r0-v0+v where g(u,v) is the luma component of the geometry image, (u,v) is the pixel associated with the 3D sample on the projection plane, (δ0,s0,r0) is the 3D position of the corresponding patch of the connected component to which the 3D sample belongs, and (u0,v0,u1,v1) are the coordinates in said projection plane that define a 2D bounding box that contains the projection plane of the patch associated with said connected component.

[0076] Therefore, the geometry image generation module GIG can encode (derive) the depth values ​​associated with the 2D samples of the (first or second or both) layers as luma components g(u,v) given by g(u,v)=δ(u,v)−δ0. Note that this relationship can be exploited for reconstruction from the reconstructed geometry image g(u,v) with auxiliary patch information PI to the 3D sample positions (δ0,s0,r0).

[0077] According to one embodiment, the projection mode can be used to indicate whether the first geometry image GI0 can preserve depth values ​​of 2D samples of the first or second layer, and whether the second geometry image GI1 can preserve depth values ​​associated with 2D samples of the second or first layer.

[0078] For example, if the projection mode is equal to 0, the first geometry image GI0 may store depth values ​​of the 2D samples of the first layer and the second geometry image GI1 may store depth values ​​associated with the 2D samples of the second layer. Conversely, if the projection mode is equal to 1, the first geometry image GI0 may store depth values ​​of the 2D samples of the second layer and the second geometry image GI1 may store depth values ​​associated with the 2D samples of the first layer.

[0079] According to one embodiment, the frame projection mode can be used to indicate whether a fixed projection mode was used for all patches, or whether a variable projection mode was used, where each patch may use a different projection mode.

[0080] The projection mode and / or the frame projection mode may be transmitted as metadata.

[0081] The frame projection mode decision algorithm is shown, for example, in section 2.2.1.3.1 of the V-PCC.

[0082] According to one embodiment, if the frame projection indicates that a variable projection mode may be used, then the patch projection mode may be used to indicate the preferred mode to use for (un)projecting the patch.

[0083] The patch projection mode may be transmitted as metadata, possibly information included in the auxiliary patch information PI.

[0084] The patch projection mode decision algorithm is given, for example, in section 2.2.1.3.2 of the V-PCC.

[0085] According to an embodiment of step 3300, a pixel value of a first geometry image, e.g. GI0, corresponding to a 2D sample (u,v) of a patch may represent a depth value associated with at least one intermediate 3D sample defined along a projection line corresponding to said 2D sample (u,v). Said intermediate 3D sample lies along the projection line and shares the same coordinate of the 2D sample (u,v), whose depth value D1 is encoded in a second geometry image, e.g. GI1. Furthermore, said intermediate 3D sample may have a depth value from depth value D0 to depth value D1. A bit designated to be set to 1 if an intermediate 3D sample exists and to 0 otherwise may be associated with each of said intermediate 3D samples.

[0086] Then, all the designated bits along the projection line can be concatenated to form a codeword, hereafter called an Extended Delta Depth (EDD) code. Finally, all the EDD codes can be packed into an image, for example a first geometry image GI1 or an occupancy map OM.

[0087] In step 3400, a texture image generator TIG can generate at least one texture image TI from the geometry of a reconstructed point cloud frame derived from an input point cloud frame IPCF, an occupancy map OM, auxiliary patch information PI and at least one decoded geometry image DGI that is output of the video decoder VDEC (step 4200 in FIG. 4).

[0088] The texture image TI represents the texture of the input point cloud frame IPCF and may be, for example, an image of W×H pixels represented in YUV420-8bit format.

[0089] The texture image generator TG can use the occupancy map information to detect (locate) occupied blocks, and hence non-empty pixels, in the texture image.

[0090] The texture image generator TIG may be adapted to generate and associate a texture image TI with each geometry image / layer DGI.

[0091] According to one embodiment, the texture image generator TIG can encode (store) texture (attribute) values ​​T0 associated with the 2D samples of the first layer as pixel values ​​of a first texture image TI0 and texture values ​​T1 associated with the 2D samples of the second layer as pixel values ​​of a second texture image TI1.

[0092] Alternatively, the texture image generation module TIG can encode (store) the texture values ​​T1 associated with the 2D samples of the second layer as pixel values ​​of the first texture image TI0 and the texture values ​​D0 associated with the 2D samples of the first layer as pixel values ​​of the second geometry image GI1.

[0093] For example, the color of a 3D sample can be obtained as described in sections 2.2.3, 2.2.4, 2.2.5, 2.2.8 or 2.5 of the V-PCC.

[0094] According to one embodiment, a padding operation can be applied to geometry and / or texture images to fill empty spaces between patches to produce piecewise smooth images suitable for video compression.

[0095] Examples of image padding are given in sections 2.2.6 and 2.2.7 of the V-PCC.

[0096] In step 3500, the video encoder VENC may encode the generated images / layers TI and GI.

[0097] In step 3600, the encoder OMENC may encode the occupancy map as an image, for example as detailed in section 2.2.2 of V-PCC. Lossy or lossless encoding may be used.

[0098] According to one embodiment, the video encoders ENC and / or OMENC may be HEVC-based encoders.

[0099] In step 3700, the encoder PIENC may encode the auxiliary patch information PI and possibly additional metadata such as block size T, width W and height H of the geometry / texture image.

[0100] According to one embodiment, the auxiliary patch information may be encoded in different ways (eg, as defined in section 2.4.1 of the V-PCC).

[0101] In step 3800, a multiplexer is applied to the generated outputs of steps 3500, 3600 and 3700 so that these outputs can be multiplexed to generate a bitstream representing the base layer BL. Note that the metadata information represents a small portion of the overall bitstream; most of the information is compressed using a video codec.

[0102] FIG. 4 shows a schematic block diagram of an example image analysis point cloud decoder 4000 according to at least one of the present embodiments.

[0103] In step 4100, a demultiplexer DMUX may be applied to demultiplex the coded information of the bitstream representing the base layer BL.

[0104] In step 4200, the video decoder VDEC is able to decode the encoded information to derive at least one decoded geometry image DGI and at least one decoded texture image DTI.

[0105] In step 4300, the decoder OMDEC is able to decode the encoded information to derive a decoded occupancy map DOM.

[0106] According to one embodiment, the video decoders VDEC and / or OMDEC may be HEVC-based decoders.

[0107] In step 4400, the decoder PIDEC can decode the encoded information to derive the auxiliary patch information DPI.

[0108] Optionally, metadata can also be derived from the bitstream BL.

[0109] In step 4500, the geometry generation module GGM can derive a geometry RG of the reconstructed point cloud frame IRPCF from at least one decoded geometry image DGI, a decoded occupancy map DOM, decoded auxiliary patch information DPI and possible additional metadata.

[0110] The geometry generation module GGM can utilize the decoded occupancy map information DOM to identify the location of non-empty pixels in at least one decoded geometry image DGI. The 3D coordinates of a reconstructed 3D sample associated with a non-empty pixel can then be derived from the coordinates of said non-empty pixel and the values ​​of said reconstructed 2D sample.

[0111] According to one embodiment, the geometry generation module GGM is capable of deriving the 3D coordinates of the reconstructed 3D samples from the coordinates of the non-empty pixels.

[0112] According to one embodiment, the geometry generation module GGM is able to derive the 3D coordinates of the reconstructed 3D sample from the coordinates of the non-empty pixels, the value of one of said non-empty pixels of at least one decoded geometry image DGI, the decoded auxiliary patch information and possibly additional metadata.

[0113] The use of non-empty pixels is based on the relationship of 2D pixels with 3D samples. For example, with the above mentioned projection plane in V-PCC, the 3D coordinates of the reconstructed 3D sample can be expressed in terms of depth δ(u,v), tangent shift s(u,v) and bi-tangent shift r(u,v) as follows: δ(u,v)=δ0+g(u,v) s(u,v)=s0-u0+u r(u,v)=r0-v0+v where g(u,v) is the luma component of the decoded geometry image DGI, (u,v) is the pixel associated with the reconstructed 3D sample, (δ0,s0,r0) is the 3D position of the connected component to which the reconstructed 3D sample belongs, and (u0,v0,u1,v1) are the coordinates in the projection plane that define a 2D bounding box containing the projection of the patch associated with said connected component.

[0114] In step 4600, the texture generation module TGM is able to derive the texture of the reconstructed point cloud frame IRPCF from the geometry RG and at least one decoded texture image DTI.

[0115] FIG. 5 illustrates a schematic example of a syntax for a bitstream representing a base layer BL according to at least one of the present embodiments.

[0116] The bitstream includes a bitstream header SH and at least one frame stream group GOFS.

[0117] The frame stream group GOFS includes a header HS, at least one syntax element OMS representing an occupancy map OM, at least one syntax element GVS representing at least one geometry image (or video), at least one syntax element TVS representing at least one texture image (or video) and at least one syntax element PIS representing auxiliary patch information and other additional metadata.

[0118] In one variant, the frame stream group GOFS comprises at least one frame stream.

[0119] FIG. 6 illustrates a schematic block diagram of an example of a system in which various aspects and embodiments may be implemented.

[0120] System 6000 may be embodied as one or more devices including various components described below and configured to perform one or more aspects described herein. Examples of equipment that may form all or part of system 6000 include personal computers, laptops, smartphones, tablet computers, digital multimedia set-top boxes, digital video receivers, personal video recording systems, connected home appliances, connected vehicles and associated processing systems, head mounted display devices (HMDs, see-through glasses), projectors (beamers), "caves" (systems including multiple displays), servers, video encoders, video decoders, post-processors that process output from video decoders, pre-processors that provide input to video encoders, web servers, set-top boxes, and any other device that processes point clouds, videos, or images or other communications devices. The elements of system 6000, alone or in combination, may be embodied as a single integrated circuit, multiple ICs, and / or separate components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 6000 may be distributed across multiple ICs and / or separate components. In various embodiments, the system 6000 can be communicatively coupled to other similar systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, the system 6000 can be configured to implement one or more aspects described herein.

[0121] The system 6000 may include at least one processor 6010 configured to execute instructions loaded therein to perform various aspects described herein, for example. The processor 6010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 6000 may include at least one memory 6020 (e.g., volatile and / or non-volatile memory elements). The system 6000 may include storage 6040, which may include non-volatile and / or volatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash, magnetic disk drives, and / or optical disk drives. The storage 6040 may include, as non-limiting examples, internal storage, attached storage, and / or network-accessible storage.

[0122] The system 6000 may include, for example, an encoder / decoder module 6030 configured to process data to provide encoded or decoded data, which may include its own processor and memory. The encoder / decoder module 6030 may represent a module that may be included in an apparatus that performs encoding and / or decoding functions. As is known, an apparatus may include one or both of an encoding and decoding module. Additionally, the encoder / decoder module 6030 may be implemented as a separate element of the system 6000 or may be incorporated within the processor 6010 as a combination of hardware and software, as is known to those skilled in the art.

[0123] Program code to be loaded into the processor 6010 or the encoder / decoder 6030 to perform various aspects described herein may be stored in the storage device 6040 and then loaded into the memory 6020 for execution by the processor 6010. According to various embodiments, one or more of the processor 6010, memory 6020, storage device 6040 and encoder / decoder module 6030 may store one or more various items during the execution of the processes described herein. Such stored items include, but are not limited to, point cloud frames, encoded / decoded geometry / texture videos / images or portions of encoded / decoded geometry / texture videos / images, bitstreams, matrices, variables and equations, formulas, operations and intermediate or final results from the processing of operational logic.

[0124] In some embodiments, memory within the processor 6010 and / or the encoder / decoder module 6030 may be used to store instructions and provide working memory for processes that may be performed during encoding or decoding.

[0125] However, in other embodiments, memory external to the processing unit (e.g., the processing unit can be either the processor 6010 or the encoder / decoder module 6030) can be used for one or more of these functions. The external memory can be, for example, the memory 6020 and / or the storage 6040, a dynamic volatile memory and / or a non-volatile flash memory. In some embodiments, the external non-volatile flash memory can be used to store the operating system of the television. In at least one embodiment, a high-speed external dynamic volatile memory such as RAM can be used as working memory for video encoding and decoding operations such as MPEG-2 Part 2 (also known as ITU-T Recommendation H.262 and ISO / IEC 13818-2, also known as MPEG-2 Video), HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding).

[0126] Input to the elements of system 6000 may be provided via various input devices, as shown in block 6130. Such input devices may include, but are not limited to, (i) an RF section, which may receive an RF signal transmitted wirelessly, for example by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0127] In various embodiments, the input devices of block 6130 may be associated with each input processing element as known in the art. For example, the RF section may be associated with elements necessary to (i) select a desired frequency (also referred to as signal selection or bandlimiting a signal to a frequency band), (ii) downconvert the selected signal, (iii) bandlimit the frequency again to a narrower band to select a signal frequency band, which may be referred to as a channel in certain embodiments (for example), (iv) demodulate the downconverted bandlimited signal, (v) perform error correction, and (vi) demultiplex to select a desired stream of data packets. The RF section of various embodiments may include one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs these various functions, such as downconverting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency near baseband) or to baseband.

[0128] In one embodiment of a set-top box, the RF section and associated input processing elements can receive an RF signal transmitted over a wired (e.g., cable) medium. The RF section can then perform frequency selection by filtering, down-conversion, and re-filtering to a desired frequency band.

[0129] Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions.

[0130] Adding elements may include inserting elements between existing elements, such as inserting amplifiers, analog to digital converters, etc. In various embodiments, the RF section may include an antenna.

[0131] The USB and / or HDMI terminal may also include respective interface processors that connect the system 6000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of the input processing, such as Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 6010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 6010, as desired. The demodulated and error corrected, and demultiplexed stream may be provided to various processing elements, such as the processor 6010 and an encoder / decoder 6030, which operates in combination with memory and storage elements to process the data stream as desired for presentation on an output device.

[0132] The various elements of the system 6000 may be provided in a unified housing in which the various elements may be interconnected to transmit data to one another using any suitable connection arrangement 6140, such as an internal bus known in the art, including an I2C bus, wires, and printed circuit boards.

[0133] The system 6000 may include a communication interface 6050 that enables communication with other devices over a communication channel 6060. The communication interface 6050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 6060. The communication interface 6050 may include, but is not limited to, a modem or a network card, and the communication channel 6060 may be implemented in a wired and / or wireless medium, for example.

[0134] In various embodiments, a Wi-Fi network, such as IEEE 802.11, can be used to stream data to the system 6000. The Wi-Fi signal in these embodiments can be received via a communication channel 6060 and communication interface 6050 adapted for Wi-Fi communication. The communication channel 6060 in these embodiments can typically be connected to an access point or router that provides access to an outside network, including the Internet, enabling streaming applications and other over-the-top communications.

[0135] Other embodiments may provide stream data to the system 6000 using a set-top box that delivers data via an HDMI connection in the input block 6130.

[0136] According to yet another embodiment, the RF connection of input block 6130 can be used to provide stream data to system 6000 .

[0137] It should be appreciated that signaling may be done in a variety of ways, for example, one or more syntax elements, flags, etc. may be used to signal information to a corresponding decoder in various embodiments.

[0138] System 6000 can provide output signals to various output devices, including a display 6100, speakers 6110, and other peripheral devices 6120. Other peripheral devices 6120, in various example embodiments, can include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 3000.

[0139] In various embodiments, control signals can be exchanged between the system 6000 and the display 6100, speaker 6110, or other peripheral device 6120 using signaling such as AV.Link (audio / video link), CEC (consumer electronic control), or other communication protocols that allow inter-device control with or without user intervention.

[0140] Output devices may be communicatively coupled to system 6000 via dedicated connections through respective interfaces 6070, 6080 and 6090.

[0141] Alternatively, output devices may be connected to the system 6000 via the communication interface 6050 using a communication channel 6060. The display 6100 and speakers 6110 may be integrated into a single unit with other components of the system 6000 in an electronic device such as a television.

[0142] In various embodiments, the display interface 6070 may include a display driver, such as a timing controller (T Con) chip.

[0143] The display 6100 and speakers 6110 may alternatively be separate from one or more of the other components, for example if the RF portion of the input 6130 is part of a separate set-top box. In various embodiments in which the display 6100 and speakers 6110 may be external elements, the output signal may be output via a dedicated output connection, for example an HDMI port, a USB port, or a COMP including output.

[0144] PLR stands for local point reconstruction. PLR is a reconstruction method that can be used to generate additional 3D samples for a point cloud frame. PLR is typically applied immediately after 2D to 3D projection, before any other processing, such as geometry or texture smoothing.

[0145] PLR takes a layer of 3D samples as input and applies a set of filters driven by PLR metadata to generate additional 3D samples along with their geometry and textures.

[0146] Since the PLR ​​metadata can vary for every T×T blocks of the canvas, the PLR ​​can be local and the set of filters can use a small neighborhood to generate additional 3D samples. Note that the same PLR ​​metadata needs to be used for the encoder and decoder.

[0147] Under typical conditions, using a single layer of PLRed depth and texture images provides better BD-rate performance than using two layers of depth and texture images. The coding of PLR metadata uses fewer bits than conventional coding of 3D samples via projection onto depth and texture images, thus reducing the overall bitrate. At the same time, the additional 3D samples compensate for the quality loss caused by using fewer layers.

[0148] V-PCC includes an implementation of PLR, which is determined at the encoder side by RDO (Rate Distortion Optimization), which defines several modes, denoted PLRM (Local Reconstruction Mode), to reconstruct (generate) at least one 3D sample of a point cloud frame. Each PLRM is determined by a specific value of the PLRM metadata that defines the filter usage.

[0149] For example, in V-PCC section 9.4.4, multiple PLRMs are determined by four parameters described in V-PCC section 7.4.35 Local Point Reconstruction Semantics, which are transmitted as PLRM metadata in the bitstream. ·point_local_reconstruction_mode_interpolate_flag: This parameter is equal to 1 if it indicates that point-to-point interpolation is used during the execution of the reconstruction method, and this parameter is equal to 0 if it indicates that point-to-point interpolation is not used during the execution of the reconstruction method. ·point_local_reconstruction_mode_filling_flag: This parameter is equal to 1 if it indicates that the filling mode is to be used during the execution of the reconstruction method, and this parameter is equal to 0 if it indicates that the filling mode is not to be used during the execution of the reconstruction process. ·point_local_reconstruction_mode_minimum_depth_minus1: This parameter specifies the minimum depth value minus one to be used during the execution of the reconstruction method. ·point_local_reconstruction_mode_neighbour_minus1: This parameter specifies the size of the 2D neighborhood used in the reconstruction method minus one.

[0150] The syntax for encoding PLR is detailed in V-PCC section "7.3.35 Local Point Reconstruction Syntax". This syntax describes the PLRM metadata sent in TxT blocks (occupied packing block size) via "blockToPatch" information. For each block of TxT pixels, the blockToPatch structure indicates the patch to which the block belongs. The size of the block-to-patch block, 16x16 pixels, is a typical value used in V-PCC test model software. This specification (and at least one of the embodiments) can also support other block sizes of bloc-to-patch index and PLRM metadata.

[0151] The PLR ​​metadata can be obtained from the bitstream as follows: Loop over the T×T blocks of the canvas in scan order, e.g., raster scan order. For each block of the canvas, if the occupancy map OM indicates that the block is not occupied, proceed to the next block. Otherwise, i.e., if the block is occupied, the block-to-patch information BlockToPatch is obtained from the bitstream to obtain the patch ID of the block, and the PLR ​​parameters of the occupied block are obtained from the bitstream.

[0152] In this implementation of point cloud coding (V-PCC), decoding the PLR ​​metadata from the bitstream requires significant computational resources (CPU, GPU, memory) since four parameters need to be decoded for each of the T×T blocks.

[0153] According to a general aspect of at least one embodiment, there is provided a method for signaling a syntax element representing a local point reconstruction mode, the local point reconstruction mode representing at least one parameter defining a mode for reconstructing at least one point of a point cloud frame.

[0154] Signaling a syntax element instead of four parameters per block reduces the computational resources on the decoding side defined in V-PCC since the syntax element can be signaled per patch or per point cloud frame, which allows for flexibility in signaling PLRM metadata.

[0155] FIG. 7 shows a schematic block diagram of an example of a method for signaling a local point reconstruction mode according to at least one of the present embodiments.

[0156] In step 710, the module can add to the bitstream at least one syntax element SE1 representing a local point reconstruction mode, denoted PLRM, which represents at least one parameter defining a mode for reconstructing at least one 3D sample of a point cloud frame.

[0157] In step 740, the bitstream is transmitted.

[0158] In step 770, the module may obtain (read) from the bit stream (received bit stream) at least one first syntax element SE1 representing at least one PLRM used for reconstructing at least one 3D sample of the point cloud.

[0159] At least one parameter is obtained from the at least one syntax element SE1, and then at least one 3D sample of a point cloud is reconstructed using the at least one PLRM.

[0160] According to one embodiment of step 710, the PLRM may be an index value of an entry in a look-up table LUT, each entry of said LUT defining a relationship between an index value and at least one parameter defining a particular mode for reconstructing at least one point of said point cloud frame.

[0161] 8a-8b show an example of a look-up table LUT according to one embodiment of step 710. FIG.

[0162] The left column shows different values ​​of PLRM, each PLRM defining a combination of the four parameters I, F, D1min and N that determine the reconstruction method using the following combination of tools as explained above: point_local_reconstruction_mode_interpolate_flag, point_local_reconstruction_mode_filling_flag, point_local_reconstruction_mode_minimum_depth_minus1, and point_local_reconstruction_mode_neighbour_minus1

[0163] PLRM0 in FIG. 8a indicates the use of "no PLRM metadata", i.e. no PLR metadata is sent.

[0164] Such PLRM modes are removed from the look-up table LUT of Figure 8b, and the "no PLRM metadata" mode is then signaled elsewhere.

[0165] FIG. 8c shows an example of a syntax for signaling the look-up table LUT according to one embodiment of step 710.

[0166] According to one embodiment of step 710, the first syntax element SE1 may be signaled per block as shown in FIG. 8d, where point_local_reconstruction_mode[p][i] refers to the syntax element SE1 associated with one block i of patch p.

[0167] According to one embodiment of step 710, the first syntax element SE1 may be signaled per patch as shown in FIG. 8e, where point_local_reconstruction_mode[p] refers to the first syntax element SE1 associated with patch p.

[0168] In step 720 (optional), the module may add to the bitstream, for each patch, a second syntax element SE2 indicating whether a single first syntax element SE1 is signaled once for all blocks of the patch or whether a first syntax element SE1 is signaled for each block of the patch.

[0169] If the second syntax element SE2 indicates that a single first syntax element is signaled once for all blocks of a patch, then the single first syntax element is used in all blocks of the patch and is therefore signaled at the patch level.

[0170] As shown in FIG. 9, point_local_reconstruction_patch_level[p] refers to the second syntax element SE2, point_local_reconstruction_patch_level[p]=0 means that the first syntax element point_local_reconstruction_mode[p][i] is signaled for each block of the patch, and point_local_reconstruction_patch_level[p]=1 indicates that the first syntax element point_local_reconstruction_mode[p] is signaled once for all blocks of patch p.

[0171] According to a variant of step 720, the second syntax element SE2 may be signaled only if the patch contains more than a given number of blocks.

[0172] As shown in Figure 10, BlockCountThreshold refers to a given number of blocks. If the number of blocks in a patch is equal to or greater than BlockCountThreshold, the second syntax element point_local_reconstruction_patch_level[p] is set to 0 (signaling the first syntax element point_local_reconstruction_mode[p] per block), otherwise it is set to 1 (signaling the first syntax element point_local_reconstruction_mode[p] per patch).

[0173] In step 730 (optional), the module may add at least one third syntax element SE3 to the bitstream, representing a default local point reconstruction mode, also denoted as preferred local point reconstruction mode.

[0174] As shown in Figures 12a-b, point_local_reconstruction_preferred_mode refers to a third syntax element SE3.

[0175] FIG. 11 shows an embodiment of step 710 and an example of signaling a look-up table LUT (first syntax element SE1) according to point_local_reconstruction_preferred_mode.

[0176] According to one embodiment of step 730, the third syntax element SE3 may be signaled per point cloud frame and / or per patch.

[0177] As shown in FIG. 13, point_local_reconstruction_preferred_mode_patch[p] refers to a third syntax element SE3 that is signaled per patch, and point_local_reconstruction_preferred_mode refers to a third syntax element SE3 that is signaled per point cloud frame.

[0178] As shown in Figure 15, point_local_reconstruction_preferred_mode refers to the third syntax element SE3 that is signaled for each point cloud frame.

[0179] According to one embodiment of step 740, the module can add to the bitstream a fourth syntax element SE4 that is used to signal the granularity of the first signal element SE1 (PLRM per block or per patch) and the use of the third syntax element SE3 (default local point reconstruction mode signaled per patch).

[0180] As shown in Fig. 12a-b, point_local_reconstruction_patch_level[p] refers to the fourth syntax element. If point_local_reconstruction_patch_level[p]=0, the first syntax element SE1(point_local_reconstruction_mode[p][i]) is signaled for each block i, if point_local_reconstruction_patch_level[p]=1, the first syntax element SE1(point_local_reconstruction_mode[p]) is signaled for each patch p, otherwise the third syntax element SE3(point_local_reconstruction_preferred_mode) is signaled for all blocks of patch p.

[0181] In Figure 12a, point_local_reconstruction_preferred_mode is signaled per point cloud frame.

[0182] In Figure 12b, point_local_reconstruction_preferred_mode_patch[p] is signaled on a per-patch basis.

[0183] FIG. 13 shows a variant of FIG. 12b, which signals the second syntax element SE2 only if the patch contains more than a given number of blocks BlockCountThreshold.

[0184] Figure 14 shows this variant where the first syntax element point_local_reconstruction_mode[p][i] is signalled per block (Figure 8d), where point_local_reconstruction_mode_delta[p][i] replaces point_local_reconstruction_mode[p][i].

[0185] The variants can be applied to other examples for signalling a first syntax element SE1, as shown for example in Figs. 8e, 9, 10, 12a-b, 12a and 13.

[0186] Said variants may also be applied to code the second syntax element SE2 in different ways, for example as shown in FIGS.

[0187] 1-15, various methods are described herein, each of which includes one or more steps or actions to achieve the described method. Unless a specific order of steps or actions is required for the method to properly operate, the order and / or use of specific steps and / or actions can be varied or combined.

[0188] Some examples are described with reference to block diagrams and operational flow diagrams. Each block represents a circuit element, a module, or a portion of code that includes one or more executable instructions for performing a specified logical function. It should be noted that in other implementations, the functions noted in the blocks may occur out of the order specified. For example, two blocks shown in succession may in fact be executed substantially simultaneously or the blocks may sometimes be executed in the reverse order depending on the functionality involved.

[0189] Implementations and aspects described herein may be implemented, for example, as a method or process, an apparatus, a computer program, a data stream, a bit stream, or a signal. Even if discussed in the context of a single executable form (e.g., discussed only as a method), implementations of the discussed features may also be implemented in other forms (e.g., an apparatus or a computer program).

[0190] The methods may be implemented in, for example, a processor, which generally refers to a processing device including a computer, a microprocessor, an integrated circuit, or a programmable logic device. A processor also includes a communication device.

[0191] The method may also be implemented by instructions executed by a processor, and such instructions (and / or data values ​​resulting from the implementation) may be stored on a computer-readable storage medium. A computer-readable storage medium may take the form of a computer-readable program product embodied in one or more computer-readable media and embodied with computer-readable program code executable by a computer. As used herein, a computer-readable storage medium may be considered a non-transitory storage medium with the inherent ability to store information as well as the inherent ability to make information retrievable. For example, a computer-readable storage medium may be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. Below, more specific examples of computer-readable storage media to which the present embodiment can be applied are given, but it will be understood that these are merely illustrative and are not a comprehensive list that can be easily understood by those skilled in the art: a portable computer diskette, a hard disk, a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0192] The instructions may form an application program tangibly embodied on a processor-readable medium.

[0193] The instructions may be, for example, hardware, firmware, software, or a combination thereof. The instructions may be found, for example, in an operating system, a separate application, or a combination of both. Thus, a processor may be characterized as both a device configured to perform a process and a device (e.g., a storage device) that includes a processor-readable medium having instructions for performing a process. Additionally, a processor-readable medium may store data values ​​produced by an implementation in addition to or in lieu of instructions.

[0194] The devices may be implemented, for example, in suitable hardware, software, and firmware. Examples of such devices include personal computers, laptops, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, head mounted display devices (HMDs, see-through glasses), projectors (beamers), "caves" (systems including multiple displays), servers, video encoders, video decoders, post-processors that process output from video decoders, pre-processors that provide input to video encoders, web servers, set-top boxes, and any other device that processes point clouds, videos, or images, or other communications devices. As will become apparent, the equipment may be mobile and installed in a moving vehicle.

[0195] The computer software may be implemented by the processor 6010 or hardware or a combination of hardware and software. As a non-limiting example, these embodiments may also be implemented in one or more integrated circuits. The memory 6020 may be of any type suitable for the technology environment and may be implemented using any suitable data storage technology, such as, by way of non-limiting examples, optical memory devices, magnetic storage devices, semiconductor memory devices, fixed memories and removable memories. The processor 6010 may be of any type suitable for the technology environment and by way of non-limiting examples may include one or more of a microprocessor, a general purpose computer, a special purpose computer, and a processor based on a multi-core architecture.

[0196] As will be apparent to one of ordinary skill in the art, implementations can generate a variety of signals formatted to carry information that can be stored or transmitted, for example. The information can include, for example, instructions to perform a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bit stream of the described embodiments. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0197] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used herein, the singular forms "a," "an," and "the" may include the plural, unless the context clearly dictates otherwise. Furthermore, it should be understood that the terms "comprise" and / or "comprising," as used herein, may specify, for example, the presence of a referenced feature, integer, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as "responsive to" or "connected to" another element, it may be directly responsive or connected to the other element, or intervening elements may be present. In contrast, when an element is referred to as "directly responsive to" or "directly connected to" another element, there are no intervening elements present.

[0198] For example, it is to be understood that the use of the symbols / terms " / ", "and / or" and "at least one of" in the cases of "A / B", "A and / or B" and "at least one of A and B" may be intended to encompass the selection of only the first enumerated alternative (A) or the selection of only the second enumerated alternative (B) or the selection of both alternatives (A and B). As a further example, in the case of "A, B and / or C" and "at least one of A, B and C", such phrases are intended to encompass the selection of only the first enumerated alternative (A), or the selection of only the second enumerated alternative (B), or the selection of only the third enumerated alternative (C), or the selection of only the first and second enumerated alternatives (A and B), or the selection of the first and third enumerated alternatives (A and C), or the selection of the second and third enumerated alternatives (B and C), or the selection of all three alternatives (A and B and C). This can be extended to the number of items listed, as would be apparent to one of skill in the art.

[0199] Various numerical values ​​may be used in this application. The specific values ​​are for illustrative purposes only and the described aspects are not limited to these specific values.

[0200] In this specification, terms such as first, second, etc. may be used to describe various elements, but it should be understood that these elements are not limited by the terms. These terms are only used to distinguish one element from another. For example, a first element may be referred to as a second element, and a second element may be referred to as a first element, without departing from the teachings of the present application. No ordering between the first element and the second element is implied.

[0201] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof are frequently used to convey that a particular feature, structure, characteristic, etc. (described in connection with the embodiment / implementation) is included in at least one embodiment / implementation. Thus, the appearances of "in one embodiment" or "an embodiment" or "in one implementation" or "in an implementation" in various places throughout this application, as well as any other variations thereof, are not necessarily all referring to the same embodiment.

[0202] Similarly, references to "according to one embodiment / example / implementation" or "in one embodiment / example / implementation" and other variations thereof are frequently used to convey that a particular feature, structure, characteristic, etc. (described in connection with the embodiment / example / implementation) may be included in at least one embodiment / example / implementation. Thus, the appearances of "according to one embodiment / example / implementation" or "in one embodiment / example / implementation" in various places throughout this application do not necessarily all refer to the same embodiment / example / implementation, and separate or alternative embodiments / examples / implementations are not necessarily mutually exclusive of other embodiments / examples / implementations.

[0203] Reference numerals appearing in the claims are for illustrative purposes only and shall have no limiting effect on the scope of the claims. Although not expressly stated, the embodiments / examples and variations can be used in any combination or subcombination.

[0204] Where a diagram is presented as a flow diagram, it is understood that a corresponding apparatus block diagram is also provided. Similarly, where a diagram is presented as a block diagram, it is understood that a corresponding method / process flow diagram is also provided.

[0205] Some of the figures include arrows on communication paths to indicate a primary direction of communication, however, it should be understood that communication may occur in the opposite direction to that shown in the figure.

[0206] Various implementations include decoding. As used herein, "decoding" may include all or part of the processing performed on received point cloud frames (including, optionally, a received bitstream encoding one or more point cloud frames), for example, to generate a final output suitable for display or for further processing of a reconstructed point cloud region. In various embodiments, such processing includes one or more of the processing typically performed by an image analysis decoder.

[0207] As a further example, in one embodiment, "decoding" may refer only to entropy decoding, in another embodiment, "decoding" may refer only to differential decoding, and in another embodiment, "decoding" may refer to a combination of entropy and differential decoding. Whether the phrase "decoding process" refers specifically to a subset of operations or to the broader decoding process in general will be clear based on the context of the specific description and is believed to be well understood by one of ordinary skill in the art.

[0208] Various implementations include encoding. Similar to the above discussion of "decoding," "encoding" as used herein may include all or part of the processing performed on, for example, an input point cloud frame to generate an encoded bitstream. In various embodiments, such processing may include one or more of the processing typically performed by an image analysis decoder.

[0209] As a further example, in one embodiment, "encoding" may refer only to entropy encoding, in another embodiment, "encoding" may refer only to differential encoding, and in another embodiment, "encoding" may refer to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to the encoding process more generally will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0210] Please note that syntax elements as used herein, e.g. the flag point_local_reconstruction_mode_present_flag, are descriptive terms, and thus do not exclude the use of other syntax element names.

[0211] Various embodiments refer to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is usually considered, often subject to computational complexity constraints. Rate-distortion optimization can be formulated as the minimization of a rate-distortion function, which is usually a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on an extensive test of all encoding options, including all considered modes or encoding parameter values, with a full evaluation of the encoding cost and associated distortion after encoding and decoding of the reconstructed signal. In particular, faster approaches may be used to avoid the coding complexity of calculating the approximated distortion based on a prediction signal or a prediction residual signal, rather than on the reconstructed signal. A mixture of these two approaches may be used, for example, by using approximate distortion for only some of the possible encoding options and full distortion for others. Other approaches only evaluate a subset of the possible encoding options. More generally, many approaches perform the optimization using any of a variety of techniques, but the optimization does not necessarily evaluate the encoding cost and associated distortion completely.

[0212] The application may also refer to "determining" various items of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0213] Additionally, the application may refer to "accessing" various items of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, replicating information, calculating information, determining information, predicting information, or estimating information.

[0214] This application may also refer to "receiving" various items of information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" typically involves in some manner an operation, such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0215] The word "signal" as used herein refers specifically to indicating something to a corresponding decoder. For example, in this particular embodiment, the encoder signals a particular syntax element SE and / or PLR metadata. In this way, in one embodiment, the same parameters (PLR metadata) can be used at both the encoder and decoder sides. Thus, for example, the encoder can send a particular parameter to the decoder (explicit signaling) so that the decoder can use the same particular parameter. Conversely, if the decoder already has a particular parameter along with other parameters, the decoder can simply use signaling to recognize and select the particular parameter without sending the parameter (implicit signaling). By avoiding sending an actual function, bit savings are achieved in various embodiments. It will be appreciated that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder in various embodiments. Although the above relates to the verb form of the word "signaling", in this specification the word "signal" can also be used as a noun.

[0216] A number of implementations have been described. However, it will be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to form other implementations. Those skilled in the art will also understand that other structures and processes may be substituted for those disclosed, such that the resulting implementations perform at least substantially the same functions in at least substantially the same way as the disclosed implementations to achieve at least substantially the same results. Accordingly, these and other implementations are contemplated by this application.

Claims

1. generating at least one additional 3D sample from a layer storing depth values ​​of 3D samples of the reconstructed point cloud frame into the reconstructed point cloud frame using a local point reconstruction mode indicated by a first syntax element, the local point reconstruction mode including at least one parameter defining the local point reconstruction mode; signaling a second syntax element in the bitstream for each patch of the point cloud frame, the second syntax element indicating whether the first syntax element was signaled once per patch, whether the same first syntax element was used for all blocks of a patch, or whether the first syntax element was signaled for each block of a patch; and signaling in a bitstream the first syntax element that depends on the second syntax element; The method includes:

2. generating at least one additional 3D sample from a layer storing depth values ​​of 3D samples of the reconstructed point cloud frame into the reconstructed point cloud frame using a local point reconstruction mode indicated by a first syntax element, the local point reconstruction mode including at least one parameter defining the local point reconstruction mode; signaling a second syntax element in the bitstream for each patch of the point cloud frame, the second syntax element indicating whether the first syntax element was signaled once per patch, whether the same first syntax element was used for all blocks of a patch, or whether the first syntax element was signaled for each block of a patch; and signaling in a bitstream the first syntax element that depends on the second syntax element; 23. An apparatus comprising one or more processors configured to:

3. 2. The method of claim 1, wherein the first syntax element is an index value of an entry in a look-up table, the look-up table defining a relationship between the index value and a particular value for the at least one parameter of the local point reconstruction mode.

4. The method of claim 1 or 3, wherein a patch is a set of at least one block of at least one 2D sample representing an orthogonal projection of at least one 3D sample of the point cloud frame onto a projection plane.

5. 5. The method of claim 1, 3 or 4, wherein signaling the second syntax element is in response to determining that a patch includes more than a given number of blocks.

6. The method according to any one of claims 1, 3 to 5, further comprising signalling a third syntax element representing a default mode of the local point reconstruction mode.

7. generating at least one additional 3D sample from a layer storing depth values ​​of 3D samples of the reconstructed point cloud frame into the reconstructed point cloud frame using a local point reconstruction mode indicated by a first syntax element, the local point reconstruction mode including at least one parameter defining the local point reconstruction mode; decoding a second syntax element from the bitstream for each patch of the point cloud frame, the second syntax element indicating whether the first syntax element was decoded once per patch, whether the same first syntax element was used for all blocks of a patch, and whether the first syntax element was decoded for each block of a patch; decoding the first syntax element from a bitstream, the first syntax element depending on the second syntax element; The method includes:

8. generating at least one additional 3D sample from a layer storing depth values ​​of 3D samples of the reconstructed point cloud frame into the reconstructed point cloud frame using a local point reconstruction mode indicated by a first syntax element, the local point reconstruction mode including at least one parameter defining the local point reconstruction mode; decoding a second syntax element from the bitstream for each patch of the point cloud frame, the second syntax element indicating whether the first syntax element was decoded once per patch, whether the same first syntax element was used for all blocks of a patch, and whether the first syntax element was decoded for each block of a patch; decoding the first syntax element from a bitstream, the first syntax element depending on the second syntax element; 23. An apparatus comprising one or more processors configured to:

9. 8. The method of claim 7, wherein the first syntax element is an index value of an entry in a look-up table, the look-up table defining a relationship between the index value and a particular value for the at least one parameter of the local point reconstruction mode.

10. The method of claim 7 or 9, wherein a patch is a set of at least one block of at least one 2D sample representing an orthogonal projection of at least one 3D sample of the point cloud frame onto a projection plane.

11. 11. The method of claim 7, 9 or 10, wherein decoding the second syntax element is responsive to determining that the patch includes more than a given number of blocks.

12. The method according to any one of claims 7, 9 to 11, wherein the method further comprises decoding a third syntax element representing a default mode of the local point reconstruction mode.

13. A computer program product comprising instructions which, when executed by one or more processors, cause said one or more processors to perform a method according to any one of claims 1 or 3 to 7 or 9 to 12.

14. A non-transitory computer readable medium comprising instructions for causing one or more processors to perform the method of any one of claims 1 or 3-7 or 9-12.

15. The apparatus of claim 8 , wherein decoding the second syntax element is responsive to determining that a patch includes more than a given number of blocks.

16. The local point reconstruction mode includes four parameters, values ​​of the parameters define a mode of reconstructing at least one point of the point cloud frame; The four parameters are: a first flag indicating whether point interpolation was used to reconstruct the at least one point; a second flag indicating whether a fill mode was used to reconstruct the at least one point; and a first value indicating a minimum depth value used to reconstruct the at least one point minus one; and a second value indicating one less than the size of the 2D neighborhood used to reconstruct the at least one point; The apparatus of claim 8 , comprising:

17. The method of claim 9 , wherein the look-up table is signaled in a bitstream.

18. The apparatus of claim 8 , wherein the local point reconstruction mode takes a layer of 3D samples as input and applies a set of filters driven by the at least one parameter of the local point reconstruction mode.

19. The local point reconstruction mode includes four parameters, values ​​of the parameters define a mode of reconstructing at least one point of the point cloud frame; The four parameters are: a first flag indicating whether point interpolation was used to reconstruct the at least one point; a second flag indicating whether a fill mode was used to reconstruct the at least one point; and a first value indicating a minimum depth value used to reconstruct the at least one point minus one; and a second value indicating one less than the size of the 2D neighborhood used to reconstruct the at least one point; The method of claim 7, comprising:

20. The method of claim 3 , wherein the look-up table is signaled in a bitstream.

Images