Processing point clouds

Abstract

At least one embodiment relates to a method for smoothing (filtering) the geometry of a point cloud frame by performing analysis and filtering of the geometric point cloud in 2D space without reconstructing 3D samples in 3D space, and using a flexible filtering operator that can remove points or add new points in addition to moving existing points.

Description

Technical Field

[0001] At least one of the present embodiments generally relates to processing of point clouds. Background Art

[0002] This section is intended to introduce the reader to various aspects of the art, which may be related to various aspects of at least one embodiment of the present invention described and / or claimed below. It is believed that this discussion will help provide the reader with background information to facilitate a better understanding of various aspects of at least one embodiment.

[0003] Point clouds can be used for various purposes, such as cultural heritage / architecture, where objects such as statues or buildings are 3D scanned to share the object's spatial configuration without having to send or access it. Furthermore, they are a way to preserve knowledge of an object in the event it is destroyed; for example, a temple destroyed by an earthquake. These point clouds are typically static, colorful, and large.

[0004] Another use case is in topography and cartography, where using 3D representations allows maps to be more than just flat and include terrain. Google Maps is a good example of a 3D map that uses meshes instead of point clouds. However, point clouds can be a suitable data format for 3D maps, and such point clouds are often static, colorful, and large.

[0005] The automotive industry and autonomous vehicles are also areas where point clouds can be used. Autonomous vehicles should be able to "detect" their environment to make good driving decisions based on the reality of their immediate vicinity. Typical sensors like LIDAR (Light Detection and Ranging) produce dynamic point clouds that are used by the decision engine. These point clouds are not intended to be seen by humans, and they are usually small, not necessarily in color, and dynamic with a high capture frequency. These point clouds may have other properties such as reflectivity provided by LIDAR, as this property provides good information about the material of the sensed object and can help in making decisions.

[0006] Virtual reality and immersive worlds are a hot topic lately, foreseen by many as the future of 2D, flat video. The basic idea is to immerse the viewer in a surrounding environment, in contrast to standard television (TV) where viewers are limited to viewing a virtual world in front of them. There are several levels of immersion, depending on the viewer's degree of freedom within the environment. Point clouds are a promising format for distributing virtual reality (VR) worlds.

[0007] In many applications, it is important to be able to distribute dynamic point clouds to end users (or store them in a server) while maintaining an acceptable or improved quality of experience by consuming only a reasonable amount of bitrate (or storage space for the application). To make the distribution chain of many immersive worlds practical, it is key to effectively compress these dynamic point clouds.

[0008] With the foregoing in mind, at least one embodiment has been devised. Summary of the Invention

[0009] The following presents a simplified overview of at least one of the present embodiments to provide a basic understanding of some aspects of the present disclosure. This overview is not an extensive overview of the embodiments. It is not intended to identify key or important elements of the embodiments. The following overview merely presents some aspects of at least one of the present embodiments in a simplified form as a prelude to the more detailed description provided elsewhere in the document.

[0010] According to a general aspect of at least one embodiment, there is provided a method for filtering 2D samples of a patch, comprising: spatially modifying at least one of the 2D samples to fit a reference profile, the reference profile being obtained by reprojecting at least one 2D sample of at least one neighboring patch of the patch onto a projection plane onto which the 2D samples of the patch have been projected.

[0011] In one embodiment, the at least one neighboring patch is obtained from at least one candidate patch.

[0012] In one embodiment, when the 3D sample corresponding to the 2D sample of the candidate patch and the 3D sample corresponding to the neighboring patch are close to each other, the candidate patch is a neighboring patch of a patch.

[0013] In one embodiment, a candidate patch is a neighboring patch of a patch when the 3D bounding box of the candidate patch intersects the 3D bounding box of the patch.

[0014] In one embodiment, the 2D sample of the patch forms the outline of the patch.

[0015] In one embodiment, the outline of the patch is modified by adding and / or removing at least one 2D sample of the patch.

[0016] In one embodiment, a 2D sample of the contour of the patch is modified if the sample is inside or outside the reference contour.

[0017] In one embodiment, the contour of the patch is modified by repositioning at least one 2D sample of the patch along the projection axis of the patch.

[0018] In one embodiment, the contour of the patch is modified by adding at least one additional 3D sample between at least one 3D sample corresponding to a 2D sample of the patch and at least one 3D sample corresponding to a 2D sample of the reference contour.

[0019] In one embodiment, the method further comprises filtering properties of said 2D samples of the patch.

[0020] In one embodiment, the properties of the 2D samples of the patch include color information or depth information.

[0021] According to other general aspects of at least one embodiment, an apparatus, a computer program product, and a non-transitory computer-readable medium are provided.

[0022] One or more general aspects of at least one embodiment also provide a device, a computer program product, and a non-transitory computer-readable medium.

[0023] The specific nature of at least one of the present embodiments and other objects, advantages, features and uses of at least one of the present embodiments will become apparent from the following description of examples taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic block diagram of an example of a two-layer point cloud coding structure 1000 according to at least one of the present embodiments is shown.

[0025] Figure 2 A schematic block diagram of an example of a two-layer point cloud decoding structure 2000 according to at least one of the present embodiments is shown.

[0026] Figure 3 A schematic block diagram of an example of an image-based point cloud encoder 3000 according to at least one of the present embodiments is shown.

[0027] Figure 3a An example of a canvas C is shown, comprising two patches P1 and P2 and their associated 2D bounding boxes B1 and B2.

[0028] Figure 3b Two intermediate 3D samples P are shown located between two 3D samples P0 and P1 along the projection line PL. i1 and P i2 .

[0029] Figure 4 A schematic block diagram of an example of an image-based point cloud decoder 4000 according to at least one of the present embodiments is shown.

[0030] Figure 5 An example syntax representing a bitstream of a base layer BL according to at least one of the present embodiments is schematically shown.

[0031] Figure 6 A schematic block diagram illustrating an example of a system in which various aspects and embodiments are implemented is shown.

[0032] Figure 7 A schematic block diagram of an example method for filtering the geometry of a point cloud frame according to at least one embodiment is shown.

[0033] Figures 8a to 8d Some examples related to the current patch are shown.

[0034] Figure 9 An example of contour fitting by recursively eroding / dilating 2D samples of a contour C(p) based on its position relative to a reference contour RC(p) is shown.

[0035] Figure 10 A variation of an embodiment of sub-step 7031 is shown.

[0036] Figure 11 An example of an embodiment of sub-step 7032 is shown.

[0037] Figure 12 An example of an embodiment of sub-step 7033 is shown.

[0038] Figure 13 An example of a placeholder outline and a 2D pattern for extracting directions and corresponding normals is shown.

[0039] Figure 14 An example of 3D color grid geometric smoothing is shown.

[0040] Figure 15 Illustration showing the smoothing of color properties of a 3D mesh.

[0041] Figure 16 One or more aspects of the present color smoothing embodiments are shown, which provide a new approach to solving color problems at patch boundaries.

[0042] Figure 17A and Figure 17B An example of a frame for computing color centroids and updating attribute pixel values ​​using smoothed color centroids is shown.

[0043] Figure 18A 18 show examples of calculating a smoothed color centroid using all boundary points and updating a property value using the smoothed color centroid.

[0044] Figure 19 Shown Figure 18AA first variation of the smooth color module or process is shown.

[0045] Figure 20 Shown Figure 18A Another variation of the smooth color module or process shown.

[0046] Figure 21 An example of a process 2100 is shown in accordance with one or more aspects of the present embodiment. DETAILED DESCRIPTION

[0047] Hereinafter, at least one of the present embodiments will be described more fully with reference to the accompanying drawings, which illustrate examples of at least one of the present embodiments. However, the embodiments may be implemented in a variety of alternative forms and should not be construed as limited to the examples set forth herein. Therefore, it should be understood that the embodiments are not intended to be limited to the specific forms disclosed. On the contrary, this disclosure is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of this application.

[0048] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented in the form of a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0049] Similar or identical elements in the figures are denoted by the same reference numerals.

[0050] Figures 1 and 2 show syntax tables widely used in video-based point cloud compression (V-PCC) to define the bitstream structure of V-PCC. An example of V-PCC is the Test Model Category 2 (TMC2) algorithm, which implements the MPEG draft standard defined in ISO / IEC JTC1 / SC29 / WG11 MPEG2019 / w18180 (Marrakesh, January 2019).

[0051] In these syntax tables, the term "..." indicates unchanged portions of the syntax from the original definition given in V-PCC, and these portions are removed in the figures for easier reading. Terms in bold in the figures indicate that the value of the term is obtained by parsing the bitstream. The right column of the syntax table indicates the number of bits used to encode the data for the syntax element. For example, u(4) indicates that 4 bits are used to encode the data, u(8) indicates that 8 bits are used to encode the data, and ae(v) indicates a context-adaptive arithmetic entropy coded syntax element.

[0052] The aspects described and contemplated below can be implemented in many different forms. Figure 1-21 Some embodiments are provided, but other embodiments are also contemplated, and Figure 1-21 The discussion does not limit the breadth of implementations.

[0053] At least one of these aspects generally relates to point cloud encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream.

[0054] More specifically, the various methods and other aspects described herein may be used to modify modules, such as, for example Figure 3 The texture image generator TIG (3400) of the encoder 3000, Figure 4 The geometry generation module GGM (4500) and / or texture generation module (4600) of the decoder 4000 are shown.

[0055] Furthermore, the present aspects are not limited to MPEG standards, such as MPEG-I Part 5, which relates to point cloud compression, and may be applied, for example, to other standards and recommendations, whether existing or developed in the future, as well as extensions to any such standards and recommendations (including MPEG-I Part 5). Unless otherwise stated, or technically excluded, the aspects described in this application may be used alone or in combination.

[0056] Hereinafter, image data may refer to data representing one or more arrays of 2D samples of, for example, a specific image / video format. A specific image / video format may specify information related to the pixel values ​​of the image (or video). For example, a specific image / video format may also specify information that can be used by a display and / or any other device to visualize and / or decode the image (or video). An image typically includes a first component of a first 2D array shape of samples, which typically represents the brightness of the image. An image may also include second and third components of other 2D array shapes of samples, which typically represent the chrominance of the image. Some embodiments use a collection of 2D arrays of color samples to represent the same information, such as a traditional three-color RGB representation.

[0057] In one or more embodiments, a pixel value is represented by a vector of C values, where C is the number of components. Each value of the vector is typically represented by a number of bits, which can define the dynamic range of the pixel value.

[0058] An image block is a collection of pixels belonging to an image. The pixel values ​​of an image block (or image block data) refer to the values ​​of the pixels belonging to that image block. An image block can have any shape, although rectangular shapes are common.

[0059] A point cloud may be represented by a dataset of 3D samples within a 3D volumetric space, which has unique coordinates and may also have one or more attributes.

[0060] A 3D sample of the dataset can be defined by its spatial position (X, Y, and Z coordinates in 3D space) and possibly by one or more associated attributes, such as color expressed in RGB or YUV color space, for example, transparency, reflectivity, a two-component normal vector, or any feature that represents the characteristics of the sample. For example, a 3D sample can be defined by 6 components (X, Y, Z, R, G, B) or equivalently (X, Y, Z, y, U, V), where (X, Y, Z) defines the coordinates of a point in 3D space and (R, G, B) or (y, U, V) defines the color of the 3D sample. The same type of attribute may appear multiple times. For example, multiple color attributes can provide color information from different perspectives.

[0061] Point clouds can be static or dynamic, depending on whether the cloud changes with respect to time. Examples of static or dynamic point clouds are often represented as point cloud frames. It should be noted that in the case of dynamic point clouds, the number of points is typically not constant, but rather typically changes over time. More generally, a point cloud can be considered dynamic if anything changes over time, such as, for example, the number of points, the position of one or more points, or any attribute of any point.

[0062] As an example, a 2D sample can be defined by 6 components (u, v, Z, R, G, B) or equivalently (u, v, Z, y, U, V). (u, v) defines the coordinates of the 2D sample in the 2D space of the projection plane. Z is the depth value of the 3D sample projected onto the projection plane. (R, G, B) or (y, U, V) defines the color of the 3D sample.

[0063] Figure 1 A schematic block diagram of an example of a two-layer point cloud coding structure 1000 according to at least one of the present embodiments is shown.

[0064] The two-layer point cloud coding structure 1000 can provide a bitstream B representing an input point cloud frame IPCF. Possibly, the input point cloud frame IPCF represents a frame of a dynamic point cloud. Then, the frame of the dynamic point cloud can be encoded independently of the other frame by the two-layer point cloud coding structure 1000.

[0065] Basically, the two-layer based point cloud coding structure 1000 can provide the ability to construct a bitstream B into a base layer BL and an enhancement layer EL. The base layer BL can provide a lossy representation of the input point cloud frame IPCF, and the enhancement layer EL can provide a higher quality (possibly lossless) representation by encoding isolated points not represented by the base layer BL.

[0066] The base layer BL may be provided by an image-based encoder 3000, such as Figure 3The image-based encoder 3000 may provide a geometry / texture image representing the geometry / attributes of the 3D samples of the input point cloud frame IPCF. It may allow the discarding of isolated 3D samples. The base layer BL may be composed of Figure 4 The image-based decoder 4000 shown decodes and can provide an intermediate reconstructed point cloud frame IRPCF.

[0067] Then, return to Figure 1 In the two-layer point cloud coding 1000, the comparator COMP can compare the 3D samples of the input point cloud frame IPCF with the 3D samples of the intermediate reconstructed point cloud frame IRPCF to detect / locate missing / isolated 3D samples. Next, the encoder ENC can encode the missing 3D samples and provide an enhancement layer EL. Finally, the base layer BL and the enhancement layer EL can be multiplexed together by the multiplexer MUX to generate a bitstream B.

[0068] According to an embodiment, the encoder ENC may comprise a detector that may detect 3D reference samples R of the intermediate reconstructed point cloud frame IRPCF and associate them with the missing 3D samples M.

[0069] For example, the 3D reference sample R associated with a missing 3D sample M may be its nearest neighbor to M according to a given metric.

[0070] According to an embodiment, the encoder ENC may then encode the spatial position of the missing 3D sample M and its properties as a difference determined from the spatial position and properties of the 3D reference sample R.

[0071] In variants, these differences can be coded separately.

[0072] For example, for a missing 3D sample M, using the spatial coordinates x(M), y(M), and z(M), the x-coordinate position difference Dx(M), y-coordinate position difference Dy(M), z-coordinate position difference Dz(M), R attribute component difference Dr(M), G attribute component difference Dg(M), and B attribute component difference Db(M) can be calculated as follows:

[0073] Dx(M)=x(M)-x(R),

[0074] Where x(M) is Figure 3 The x-coordinate of the 3D sample M in the provided geometric image, and similarly for R,

[0075] Dy(M)=y(M)-y(R)

[0076] Where y(M) is Figure 3 The y coordinate of the 3D sample M in the provided geometric image, and the same applies to R,

[0077] Dz(M)=z(M)-z(R)

[0078] Where z(M) is Figure 3 The z coordinate of the 3D sample M in the provided geometric image, and the same applies to R,

[0079] Dr(M)=R(M)-R(R)

[0080] Among them, R(M) and R(R) are the r color components of the color attributes of 3D samples M and R respectively.

[0081] Dg(M)=G(M)-G(R)

[0082] Among them, G(M) and G(R) are the g color components of the color attributes of 3D samples M and R respectively.

[0083] Db(M)=B(M)-B(R)

[0084] Among them, B(M) and B(R) are the b color components of the color attributes of 3D samples M and R respectively.

[0085] Figure 2 A schematic block diagram of an example of a two-layer point cloud decoding structure 2000 according to at least one of the present embodiments is shown.

[0086] The behavior of the two-layer based point cloud decoding structure 2000 depends on its capabilities.

[0087] The two-layer based point cloud decoding structure 2000 with limited capabilities can access only the base layer BL from the bitstream B using the demultiplexer DMUX, and can then provide a faithful (but lossy) version IRPCF of the input point cloud frame IPCF by decoding the base layer BL by the point cloud decoder 4000, as shown in FIG. Figure 4 shown.

[0088] The fully capable two-layer based point cloud decoding structure 2000 can access the base layer BL and the enhancement layer EL from the bitstream B by using the demultiplexer DMUX. Figure 4 As shown, the point cloud decoder 4000 can determine an intermediate reconstructed point cloud frame IRPCF from the base layer BL. The decoder DEC can determine a complementary point cloud frame CPCF from the enhancement layer EL. The combiner COM can then combine the intermediate reconstructed point cloud frame IRPCF and the complementary point cloud frame CPCF together to provide a higher quality (possibly lossless) representation (reconstruction) CRPCF of the input point cloud frame IPCF.

[0089] Figure 3 A schematic block diagram of an example of an image-based point cloud encoder 3000 according to at least one of the present embodiments is shown.

[0090] The image-based point cloud encoder 3000 utilizes existing video codecs to compress the geometric and texture (attribute) information of dynamic point clouds. This is essentially achieved by converting the point cloud data into a collection of different video sequences.

[0091] In certain embodiments, two videos can be generated and compressed using an existing video codec, one for capturing geometric information of the point cloud data and the other for capturing texture (e.g., color / attribute) information. An example of an existing video codec is the HEVC Main Profile encoder / decoder (ITU-T H.265 ITU (02 / 2018) Telecommunication Standardization Sector, Series H: Audiovisual and Multimedia Systems, Infrastructure of Audiovisual Services - Coding of Motion Video, High Efficiency Video Coding, Recommendation ITU-T H.265).

[0092] Additional metadata used to explain the two videos is usually also generated and compressed separately. Such additional metadata includes, for example, a placeholder map OM and / or auxiliary patch information PI.

[0093] The resulting video bitstream and metadata may then be multiplexed together to generate a combined bitstream.

[0094] It should be noted that metadata generally represents a small amount of overall information. Most of the information is in the video bitstream.

[0095] In step 3100 , the patch generation module PGM may generate at least one patch by decomposing 3D samples of a dataset representing an input point cloud frame IPCF into 2D samples on a projection plane using a strategy that provides optimal compression.

[0096] A patch can be defined as a collection of 2D samples.

[0097] For example, in V-PCC, the normal of each 3D sample is first estimated as described in Hoppe et al. (Hugues Hoppe, Tony DeRose, Tom Duchamp, John McDonald, Werner Stuetzle. Surface reconstruction from unorganized points) (, Proceedings of the 1992 Conference on Signal Graphics, ACM, pp. 71-78). Next, an initial clustering of the input point cloud frame IPCF is obtained by associating each 3D sample with one of the six oriented planes of the 3D bounding box of the 3D sample containing the input point cloud frame IPCF. More precisely, each 3D sample is clustered and associated with the oriented plane with the closest normal (i.e., maximizing the dot product of the point normal and the plane normal). The 3D samples are then projected to the planes to which they are associated. The set of 3D samples that form a connected region within their planes is called a connected region. Connected components. A connected component is a set of 3D samples with similar normals and the same associated orientation plane that indicate a 3D sample. The initial clustering is then improved by iteratively updating the cluster associated with each 3D sample based on the clustering of its normal and its nearest neighboring samples. The last step involves generating a patch from each connected component, which is done by projecting the 3D samples of each connected component onto the orientation plane associated with the connected component. The patches are associated with auxiliary patch information PI, which represents auxiliary patch information defined for each patch to explain the projected 2D samples corresponding to geometric and / or attribute information.

[0098] For example, in V-PCC, the auxiliary patch information PI includes: 1) information indicating one of the six orientation planes of the 3D bounding box of the 3D sample containing the connected component; 2) information relative to the plane normal; 3) information determining the 3D position of the connected component relative to the patch expressed in depth, tangential offset and bi-tangential offset; and 4) information such as coordinates (u0, v0, u1, v1) in the projection plane defining the 2D bounding box containing the patch.

[0099] In step 3200, the patch packing module PPM may map (place) at least one generated patch onto a 2D grid (also referred to as a canvas) without any overlap in a manner that generally minimizes unused space and may ensure that each TxT (e.g., 16x16) block of the 2D grid is associated with a unique patch. A given minimum block size TxT of the 2D grid may specify a minimum distance between different patches placed on the 2D grid. The 2D grid resolution may depend on the input point cloud size and its width W and height H, and the block size T may be sent to the decoder as metadata.

[0100] The auxiliary patch information PI may also include information about the association between blocks and patches of the 2D grid.

[0101] In V-PCC, the auxiliary information PI may include block-to-patch index information (BlockToPatch), which determines the association between the block and patch index of the 2D grid.

[0102] Figure 3a An example of a canvas C is shown, which includes two patches P1 and P2 and their associated 2D bounding boxes B1 and B2. Note that Figure 3a As shown, two bounding boxes can overlap in canvas C. The 2D grid (partitioning of the canvas) is only represented inside the bounding boxes, but the partitioning of the canvas also occurs outside these bounding boxes. The bounding box associated with a patch can be partitioned into TxT blocks, typically T=16.

[0103] A TxT block containing 2D samples belonging to a patch can be considered an occupied block. Each occupied block of the canvas is represented by a specific pixel value (e.g., 1) in the placeholder map OM, and each unoccupied block of the canvas is represented by another specific value (e.g., 0). The pixel value of the placeholder map OM can then indicate whether the TxT block of the canvas is occupied, i.e., contains 2D samples belonging to a patch.

[0104] exist Figure 3a In FIG, occupied blocks are represented by white blocks, while light grey blocks represent unoccupied blocks. The image generation process (steps 3300 and 3400) stores the geometry and texture of the input point cloud frame IPCF as an image using a mapping of at least one generated patch onto the 2D mesh calculated during step 3200.

[0105] In step 3300, a geometry image generator GIG may generate at least one geometry image GI from the input point cloud frame IPCF, the placeholder map OM, and the auxiliary patch information PI. The geometry image generator GIG may utilize the placeholder map information to detect (locate) occupied blocks and, therefore, detect (locate) non-empty pixels in the geometry image GI.

[0106] The geometry image GI may represent the geometry of the input point cloud frame IPCF and may be, for example, a monochrome image of WxH pixels represented in a YV420-8 bit format.

[0107] To better handle the case where multiple 3D samples are projected (mapped) onto the same 2D sample of the projection plane (along the same projection direction (line)), multiple images called layers can be generated. Thus, different depth values ​​D1, ..., Dn can be associated with the 2D samples of the patch, and multiple geometric images can then be generated.

[0108] In V-PCC, the 2D samples of the patch are projected onto two layers. The first layer, also called the near layer, can store, for example, depth values ​​D0 associated with 2D samples with a smaller depth. The second layer, called the far layer, can store, for example, depth values ​​D1 associated with 2D samples with a larger depth. Alternatively, the second layer can store the difference between the depth values ​​D1 and D0. For example, the information stored by the second depth image can be in the interval [0, Δ] corresponding to depth values ​​in the range [D0, D0+Δ], where Δ is a user-defined parameter describing the thickness of the surface.

[0109] In this way, the second layer can contain significant contour-like high-frequency features. Therefore, it is obvious that the second depth image may be difficult to encode by using a traditional video encoder, and therefore, the depth value reconstructed from the decoded second depth image may be poor, which leads to poor geometric quality of the reconstructed point cloud frame.

[0110] According to an embodiment, the geometric image generation module GIG may encode (derive) depth values ​​associated with the 2D samples of the first and second layers by using the auxiliary patch information PI.

[0111] In V-PCC, the position of a 3D sample in a patch with a corresponding connected component can be expressed in terms of depth δ(u, V), tangential offset s(u, V), and bi-tangential offset r(u, V) as follows:

[0112] δ(u, v)=δ0+g(u, v)

[0113] s(u, v)=s0-u0+u

[0114] r(u,v)=r0-v0+v

[0115] where g(u, v) is the brightness component of the geometric image, (u, v) is the pixel associated with the 3D sample on the projection plane, (δ0, s0, r0) is the 3D position of the corresponding patch of the connected component to which the 3D sample belongs, and (u0, v0, u1, v1) are the coordinates in the projection plane that define the projected 2D bounding box containing the patch associated with the connected component.

[0116] Thus, the geometric image generation module GIG can encode (derive) the depth value associated with the 2D sample of the layer (the first layer or the second layer or both) as a luminance component g(u,v), given by: g(u,v)=δ(u,v)-δ0. Note that this relationship can be used to reconstruct the 3D sample position (δ0, s0, r0) from the reconstructed geometric image g(u,v) with accompanying auxiliary patch information PI.

[0117] According to an embodiment, the projection mode may be used to indicate whether the first geometric image GI0 may store depth values ​​of 2D samples of the first or second layer, and whether the second geometric image GI1 may store depth values ​​associated with 2D samples of the second or first layer.

[0118] For example, when the projection mode is equal to 0, the first geometric image GI0 may store the depth values ​​of the 2D samples of the first layer, and the second geometric image GI1 may store the depth values ​​associated with the 2D samples of the second layer. Conversely, when the projection mode is equal to 1, the first geometric image GI0 may store the depth values ​​of the 2D samples of the second layer, and the second geometric image GI1 may store the depth values ​​associated with the 2D samples of the first layer.

[0119] According to an embodiment, the frame projection mode may be used to indicate whether to use a fixed projection mode for all patches or whether to use a variable projection mode, where each patch may use a different projection mode in a variable projection module.

[0120] The projection mode and / or frame projection mode may be sent as metadata.

[0121] For example, the frame projection mode decision process may be provided in Section 2.2.1.3.1 of the V-PCC.

[0122] According to an embodiment, when the frame projection indicates that a variable projection mode may be used, the patch projection mode may be used to indicate an appropriate mode for (de)projecting the patch.

[0123] The patch projection mode may be sent as metadata and may be information included in the auxiliary patch information PI.

[0124] For example, the patch projection mode decision process is provided in Section 2.2.1.3.2 of V-PCC.

[0125] According to an embodiment of step 3300, a pixel value in a first geometric image (e.g., GI0) corresponding to a 2D sample (u, v) of the patch may represent a depth value of at least one intermediate 3D sample defined along a projection line corresponding to the 2D sample (u, v). More precisely, the intermediate 3D sample resides along the projection line and shares the same coordinates as the 2D sample (u, v), and the depth value D1 of the 2D sample (u, v) is encoded in a second geometric image, e.g., GI1. Furthermore, the intermediate 3D sample may have a depth value between the depth value D0 and the depth value D1. A designated bit may be associated with each intermediate 3D sample, the designated bit being set to 1 if the intermediate 3D sample exists and to 0 otherwise.

[0126] Figure 3bTwo intermediate 3D samples P are shown located between two 3D samples P0 and P1 along the projection line PL. i1 and P i2 . 3D samples P0 and P1 have depth values ​​equal to D0 and D1 respectively. The two intermediate 3D samples P i1 and P i2 The depth value D i1 and D i2 are respectively greater than D0 and less than D1.

[0127] All designated bits along the projected line can then be concatenated to form a codeword, denoted as an enhanced occupation map (EOM) codeword in the following. Figure 3b As shown, assuming an 8-bit length EOM codeword, 2 bits are equal to 1 to indicate two 3D samples P i1 and P i2 position. Finally, all EOM codewords can be packed into one image, such as a placeholder map OM. In this case, at least one patch of the canvas can contain at least one EOM codeword. Such a patch is represented as a reference patch, and a block of the reference patch is represented as an EOM reference block. Therefore, the pixel value of the placeholder map OM can be equal to a first value, such as 0, to indicate an unoccupied block of the canvas, or equal to another value, such as greater than 0, to indicate an occupied block of the canvas, such as when D1-D0<=1, or indicate an EOM reference block of the canvas, such as when D1-D0>1.

[0128] The positions of the pixels of the EOM reference block in the placeholder map OM and the values ​​of the bits of the EOM codeword obtained from the values ​​of these pixels indicate the 3D coordinates of the intermediate 3D samples.

[0129] In step 3400, a texture image generator TIG may generate at least one texture image TI from an input point cloud frame IPCF, an occupation map OM, auxiliary patch information PI, and a geometry of a reconstructed point cloud frame derived from at least one decoded geometry image DGI output from a video decoder VDEC. Figure 4 Step 4200 in ).

[0130] The texture image TI may represent the texture of the input point cloud frame IPCF and may be, for example, an image of WxH pixels represented in a YV420-8 bit format.

[0131] The texture image generator TG may utilize the occupancy map information in order to detect (locate) occupied blocks and, thus, detect (locate) non-empty pixels in the texture image.

[0132] The texture image generator TIG may be adapted to generate a texture image TI and associate it with each geometry image / layer DGI.

[0133] According to an embodiment, the texture image generator TIG may encode (store) the texture (attribute) value T0 associated with the 2D sample of the first layer as a pixel value of the first texture image TI0, and encode (store) the texture value T1 associated with the 2D sample of the second layer as a pixel value of the second texture image TI1.

[0134] Optionally, the texture image generation module TIG may encode (store) the texture value T1 associated with the 2D sample of the second layer as a pixel value of the first texture image TI0, and encode (store) the texture value D0 associated with the 2D sample of the first layer as a pixel value of the second geometric image GI1.

[0135] For example, the color of the 3D sample can be obtained as described in Section 2.2.3, 2.2.4, 2.2.5, 2.2.8, or 2.5 of V-PCC.

[0136] The texture values ​​of the two 3D samples are stored in the first or second texture image. However, the texture value of the intermediate 3D sample can be stored neither in the first texture image TI0 nor in the second texture image TI1, because the position of the projected intermediate 3D sample corresponds to an occupied block that is already used to store the texture value of another 3D sample (P0 or P1), as shown in Figure 3b As shown. Therefore, the texture values ​​of the intermediate 3D samples are stored in the EOM texture block, which is located elsewhere in the first or second texture image in a program-defined location (V-PCC Section 9.4.5). In short, the process determines the location of an unoccupied block in the texture image and stores the texture values ​​associated with the intermediate 3D samples as the pixel values ​​of the unoccupied block of the texture image, denoted as the EOM texture block.

[0137] According to an embodiment, the padding process may be applied to geometry and / or texture images. The padding process may be used to fill in the empty spaces between patches to generate piecewise smooth images suitable for video compression.

[0138] Sections 2.2.6 and 2.2.7 of V-PCC provide examples of image padding.

[0139] At step 3500 , the video encoder VENC may encode the generated pictures / layers TI and GI.

[0140] In step 3600, the encoder OMENC may encode the placeholder map into an image, as described in detail, for example, in V-PCC section 2.2.2. Lossy or lossless coding may be used.

[0141] According to an embodiment, the video encoder ENC and / or OMENC may be an HEVC-based encoder.

[0142] In step 3700 , the encoder PIENC may encode the auxiliary patch information PI and possible additional metadata such as the block size T, width W and height H of the geometry / texture image.

[0143] According to an embodiment, the auxiliary patch information may be differentially encoded (as defined, for example, in section 2.4.1 of V-PCC).

[0144] In step 3800, a multiplexer may be applied to the generated outputs of steps 3500, 3600, and 3700, and as a result, these outputs may be multiplexed together to generate a bitstream representing the base layer BL. It should be noted that the metadata information represents a small portion of the entire bitstream. A large amount of information is compressed using a video codec.

[0145] Figure 4 A schematic block diagram of an example of an image-based point cloud decoder 4000 according to at least one of the present embodiments is shown.

[0146] In step 4100 , a demultiplexer DMUX may be used to demultiplex information representing the encoding of a bitstream of a base layer BL.

[0147] In step 4200 , the video decoder VDEC may decode the encoded information to derive at least one decoded geometry image DGI and at least one decoded texture image DTI.

[0148] In step 4300 , the decoder OMDEC may decode the encoded information to derive a decoded placeholder DOM.

[0149] According to an embodiment, the video decoder VDEC and / or OMDEC may be a HEVC-based decoder.

[0150] In step 4400 , the decoder PIDEC may decode the encoded information to derive auxiliary patch information DPI.

[0151] Possibly, metadata can also be derived from the bitstream BL.

[0152] In step 4500 , the geometry generation module GGM may derive the geometry RG of the reconstructed point cloud frame IRPCF from at least one decoded geometry image DGI, a decoded placeholder map DOM, decoded auxiliary patch information DPI and possible additional metadata.

[0153] The geometry generation module GGM may utilize the decoded placeholder information DOM in order to locate non-empty pixels in at least one decoded geometry image DGI.

[0154] As described above, depending on the pixel value of the decoded occupancy information DOM and the value of D1-D0, the non-empty pixel belongs to an occupied block or an EOM reference block.

[0155] According to an embodiment of step 4500 , the geometry generation module GGM may derive two of the 3D coordinates of the intermediate 3D sample from the coordinates of the non-empty pixels.

[0156] According to an embodiment of step 4500 , when the non-empty pixel belongs to an EOM reference block, the geometry generation module GGM may derive the third one of the 3D coordinates of the intermediate 3D sample from the bit values ​​of the EOM codeword.

[0157] For example, according to Figure 3b For example, the EOM codeword EOMC is used to determine the intermediate 3D sample P i1 and P i2 The 3D coordinates of the intermediate 3D sample P i1 The third coordinate of can be obtained, for example, by D i1 = D0+3 is derived from D0, and the reconstructed 3D sample P i2 The third coordinate of can be obtained, for example, by D i2 = D0 + 5 is derived from D0. The offset value (3 or 5) is the number of intervals along the projection line between D0 and D1.

[0158] According to an embodiment, when a non-empty pixel belongs to an occupied block, the geometry generation module GGM can derive the 3D coordinates of the reconstructed 3D sample from the coordinates of the non-empty pixel, the value of the non-empty pixel of one of at least one decoded geometry images DGI, the decoded auxiliary patch information and possibly additional metadata.

[0159] The use of non-empty pixels is based on the 2D pixel relationship with the 3D sample. For example, in the case of projection in V-PCC, the 3D coordinates of the reconstructed 3D sample can be expressed in terms of depth δ(u, v), tangential offset s(u, v) and bi-tangential offset r(u, v) as follows:

[0160] δ(u, v)=δ0+g(u, v)

[0161] s(u, v)=s0-u0+u

[0162] r(u,v)=r0-v0+v

[0163] where g(u, v) is the luminance component of the decoded geometric image DGI, (u, v) is the pixel associated with the reconstructed 3D sample, (δ0, s0, r0) is the 3D position of the connected component to which the reconstructed 3D sample belongs, and (u0, v0, u1, v1) are the coordinates in the projection plane that define the projected 2D bounding box containing the patch associated with the connected component.

[0164] In step 4600 , the texture generation module TGM may derive the texture of the reconstructed point cloud frame IRPCF from the geometry RG and at least one decoded texture image DTI.

[0165] According to an embodiment of step 4600, the texture generation module TGM may derive the texture of non-null pixels belonging to the EOM reference block from the corresponding EOM texture block. The position of the EOM texture block in the texture image is defined procedurally (V-PCC Section 9.4.5).

[0166] According to an embodiment of step 4600 , the texture generation module TGM may directly derive the texture of the non-empty pixels belonging to the occupied block as pixel values ​​of the first or second texture image.

[0167] Figure 5 An example syntax representing a bitstream of a base layer BL according to at least one of the present embodiments is schematically shown.

[0168] The bitstream comprises a bitstream header SH and at least a group of frame streams GOFS.

[0169] A set of frame streams GOFS includes a header HS, at least one syntax element OMS representing a placeholder map OM, at least one syntax element GVS representing at least one geometry image (or video), at least one syntax element TVS representing at least one texture image (or video), and at least one syntax element PIS representing auxiliary patch information and other additional metadata.

[0170] In a variant, a group of frame flows GOFS comprises at least one frame flow.

[0171] Figure 6 A schematic block diagram illustrating an example of a system in which various aspects and embodiments are implemented is shown.

[0172] System 6000 can be implemented as one or more devices including the following various components and is configured to perform one or more aspects described in this document. Examples of equipment that can form all or part of system 6000 include personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, connected vehicles and their associated processing systems, head-mounted display devices (HMD, perspective glasses), projectors, "cave systems" (systems including multiple displays), servers, video encoders, video decoders, post-processors for processing outputs from video decoders, pre-processors for providing inputs to video encoders, network servers, set-top boxes, and any other equipment or other communication devices for processing point clouds, videos, or images. The elements of system 6000 can be embodied in a single integrated circuit, multiple integrated circuits, and / or discrete components, either alone or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 6000 can be distributed over multiple integrated circuits and / or discrete components. In various embodiments, system 6000 can be coupled to other similar systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports. In various embodiments, system 6000 can be configured to implement one or more aspects described in this document.

[0173] The system 6000 may include at least one processor 6010 configured to execute instructions loaded therein to implement, for example, the various aspects described in this document. The processor 6010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 6000 may include at least one memory 6020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 6000 may include a storage device 6040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drive, and / or optical disk drive. As non-limiting examples, the storage device 6040 may include an internal storage device, an attached storage device, and / or a network accessible storage device.

[0174] The system 6000 may include an encoder / decoder module 6030, which is configured to, for example, process data to provide encoded data or decoded data, and the encoder / decoder module 6030 may include its own processor and memory. The encoder / decoder module 6030 may represent a module that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 6030 may be implemented as a separate element of the system 6000, or may be incorporated into the processor 6010 as a combination of hardware and software known to those skilled in the art.

[0175] Program code to be loaded onto the processor 6010 or the encoder / decoder 6030 to perform various aspects described in this document may be stored in the storage device 6040 and subsequently loaded onto the memory 6020 for execution by the processor 6010. According to various embodiments, one or more of the processor 6010, the memory 6020, the storage device 6040, and the encoder / decoder module 6030 may store one or more of various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, point cloud frames, encoded / decoded geometry / texture video / images or portions of encoded / decoded geometry / texture video / images, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operation logic.

[0176] In several embodiments, memory internal to the processor 6010 and / or encoder / decoder module 6030 may be used to store instructions and provide working memory for processes that may be performed during encoding or decoding.

[0177] However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 6010 or the encoder / decoder module 6030) may be used for one or more of these functions. The external memory may be memory 6020 and / or storage device 6040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory may be used to store the operating system of the television. In at least one embodiment, fast external dynamic volatile memory, such as RAM, may be used as working memory for video encoding and decoding operations, such as for MPEG-2 Part 2 (also known as ITU-T Recommendation H.262 and ISO / IEC 13818-2, also known as MPEG-2 Video), HEVC (High Efficiency Video Coding), or VVC (Versatile Video Coding).

[0178] Input to the elements of system 6000 may be provided through various input devices, as shown in block 6130. Such input devices include, but are not limited to, (i) an RF section that may receive an RF signal, such as that transmitted over the air by a broadcaster; (ii) a composite input terminal; (iii) a USB input terminal; and / or (iv) an HDMI input terminal.

[0179] In various embodiments, the input device of block 6130 may have associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements required to: (i) select a desired frequency (also known as selecting a signal, or band-limiting a signal to a frequency band); (ii) down-convert the selected signal; (iii) again band-limit to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments; (iv) demodulate the down-converted and band-limited signal; (v) perform error correction; and (vi) demultiplex to select a desired data packet stream. The RF section of various embodiments may include one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband.

[0180] In a set-top box embodiment, the RF section and its associated input processing elements can receive RF signals sent through a wired (e.g., cable) medium. The RF section can then perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band.

[0181] Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions.

[0182] Adding an element may include inserting an element between existing elements, such as, for example, inserting an amplifier and an analog-to-digital converter.In various embodiments, the RF portion may include an antenna.

[0183] In addition, the USB and / or HDMI terminals can include respective interface processors for connecting the system 6000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, can be implemented as needed, for example, within a separate input processing integrated circuit or processor 6010. Similarly, various aspects of USB or HDMI processing can be implemented within a separate interface integrated circuit or within the processor 6010, if desired. The demodulated, error-corrected, and demultiplexed streams can be provided to various processing elements, including, for example, the processor 6010 and encoder / decoder 6030 operating in conjunction with memory and storage elements, to process the data streams as needed for presentation on an output device.

[0184] The various components of system 6000 can be arranged in an integrated housing. Within the integrated housing, the various components can be interconnected and transmit data between them using a suitable connection arrangement 6140 (e.g., an internal bus known in the art, including an I2C bus, wiring, and printed circuit boards).

[0185] System 6000 may include a communication interface 6050 that can communicate with other devices via a communication channel 6060. Communication interface 6050 may include, but is not limited to, a transceiver configured to send and receive data through communication channel 6060. Communication interface 6050 may include, but is not limited to, a modem or a network card, and communication channel 6060 may be implemented, for example, within a wired and / or wireless medium.

[0186] In various embodiments, a Wi-Fi network, such as IEEE 802.11, may be used to stream data to the system 6000. The Wi-Fi signals of these embodiments may be received via a communication channel 6060 adapted for Wi-Fi communications and the communication interface 6050. The communication channel 6060 of these embodiments may typically connect to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications.

[0187] Other embodiments may provide streaming data to the system 6000 using a set-top box that transmits data via an HDMI connection to input block 6130 .

[0188] Other embodiments may use the RF connection of input block 6130 to provide streaming data to the system 6000 .

[0189] It should be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc. can be used to signal information to a corresponding decoder.

[0190] The system 6000 can provide output signals to various output devices, including a display 6100, speakers 6110, and other peripheral devices 6120. In various examples of embodiments, the other peripheral devices 6120 can include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of the system 3000.

[0191] In various embodiments, control signals may be communicated between the system 6000 and the display 6100, speakers 6110, or other peripheral devices 6120 using signaling such as AV.Link (Audio / Video Link), CEC (Consumer Electronics Control), or other communication protocols that enable device-to-device control with or without user intervention.

[0192] Output devices may be communicatively coupled to the system 6000 through respective interfaces 6070 , 6080 , and 6090 via dedicated connections.

[0193] Optionally, output devices can be connected to system 6000 via communication interface 6050 using communication channel 6060. Display 6100 and speaker 6110 can be integrated into a single unit with other components of system 6000 in an electronic device, such as a television.

[0194] In various embodiments, the display interface 6070 may include a display driver such as, for example, a timing controller (TCon) chip.

[0195] For example, if the RF portion of input 6130 is part of a separate set-top box, the display 6100 and speaker 6110 may optionally be separate from one or more other components. In various embodiments where the display 6100 and speaker 6110 may be external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0196] In V-PCC, geometric smoothing (filtering) is used to remove aliasing effects caused by possible subsampling of the placeholder map (usually at a 4x4 pixel resolution) and to mitigate the effects of coding errors. More precisely, because aliasing effects and coding errors are most prominent at the boundaries (edges) of patches, the goal of geometric smoothing is to make the seam between two patches as smooth (seamless) as possible.

[0197] Two geometric smoothing methods are used in V-PCC. The first, based on 3D filtering, performs well in terms of quality metrics, but is computationally very expensive. The main idea behind 3D filtering for geometric smoothing is to move 3D samples (corresponding to 2D samples located close to the patch boundary) to the centroid of their nearest neighboring 3D samples. Therefore, 3D filtering is applied to 3D sample "p" (corresponding to 2D samples close to the patch boundary) as follows:

[0198] - Select the N nearest 3D samples, pn(i), i=0, ..., N-1, at a 3D distance from "p" less than the maximum distance Dmax;

[0199] - If N is less than the minimum value Nmin, move to the next 3D sample "p" (corresponding to the next 2D sample close to the patch boundary);

[0200] If all neighboring 3D samples pn(i) correspond to 2D samples belonging to the same patch as the current patch to be filtered, then:

[0201] - Calculate the centroid of the neighboring 3D samples pn(i); the centroid is the average of the 3D geometric coordinates of all neighboring 3D samples pn(i); and

[0202] - Move 3D sample "p" to the centroid

[0203] The second approach (Section 2.2.5 of ISO / IEC JTC1 / SC29 / WG11 MPEG2018 / N18017, October 2018, Macau, China) uses a hybrid of 2D analysis and 3D processing, which significantly reduces the computational complexity but also the quality metrics of the filtered point cloud geometry.

[0204] According to a general aspect of at least one embodiment, a method is provided for smoothing (filtering) the geometry of a point cloud frame by performing analysis and filtering of a geometric point cloud in 2D space without reconstructing 3D samples in 3D space, and by using a flexible filtering operator that can remove points or add new points in addition to moving existing points.

[0205] Performing 2D analysis and filtering instead of processing 3D samples greatly reduces computational complexity.

[0206] Improve filtering quality using flexible filtering.

[0207] Therefore, this method provides a better trade-off between complexity and filtering quality compared to existing methods that are either too complex or do not provide sufficient filtering capabilities.

[0208] Figure 7A schematic block diagram of an example method for filtering the geometry of a point cloud frame according to at least one embodiment is shown.

[0209] The geometry of the point cloud frame is represented by at least one geometric image GI, which is generated from the placeholder map and the auxiliary patch information PI as described above. Figure 3a As shown, the canvas, which identifies at least one patch, is also used to define the geometry of the point cloud frame. As described above, each patch represents a 2D projection of connected components along a projection axis. Modifying a patch on the canvas will modify the geometry of the point cloud frame because the coordinates of the 3D samples of the point cloud frame are obtained from either the modified or unmodified patch of the canvas, as described above.

[0210] The method applies steps 7010, 7020 and 7030 to each current patch p of the canvas.

[0211] In step 7010 , the module may obtain at least one neighboring patch Nei(p) of the current patch p from at least one candidate patch CAN(p).

[0212] exist Figure 8a In the example of , the current patch p is depicted in gray without an outline, while the candidate patch CAN(p) is marked with a black outline. Candidate patches adjacent to the current patch p are depicted in the same gray as the current patch p, and candidate patches not adjacent to the current patch are depicted in a lighter gray.

[0213] In step 7020 , the module may obtain a reference contour RC(p) of the current patch p by reprojecting the 2D samples of at least one neighboring patch Nei(p) to a projection plane onto which the 2D samples of the current patch p have been projected.

[0214] First, a 3D sample is obtained by back-projecting the 2D sample of the neighboring patch Nei(p) according to the projection axis associated with the neighboring patch Nei(p). The projection axis of the patch can be derived from the auxiliary patch information PI. For example, in V-PCC, the projection axis can be derived from: 1) information indicating one of the six orientation planes of the 3D bounding box containing the 3D sample of the connected component; 2) information relative to the plane normal.

[0215] Next, reprojected 2D samples are obtained by projecting the 3D samples along the projection axis associated with the current patch p.

[0216] The reference contour RC(p) is then the set of reprojected 2D samples.

[0217] Note that even though reprojecting the 2D samples to a different projection plane requires back-projecting the 2D samples to 3D space and projecting the 3D samples to another projection plane, step 7020 does not perform filtering operations in 3D space, thus limiting complexity.

[0218] According to an embodiment variant of step 7020, only 2D samples located along the border of the neighboring patch Nei(p) may be considered.

[0219] For example, a 2D sample of a patch is considered to be along the boundary of the patch when the 2D sample is closer to the edge of the patch than a given threshold.

[0220] According to a variant of the embodiment of step 7020, only one reprojected 2D sample is kept per position of the canvas.

[0221] For example, the reprojected 2D sample retained at a given position is the reprojected 2D sample whose depth is closest to the depth of the 2D sample of the current patch p at that position.

[0222] Therefore, by reprojecting the 2D samples of multiple neighboring patches Nei(p) onto the same projection plane, the reprojected 2D samples will tend to fall on true (i.e., original, encoding error-free) positions. When only the 2D samples of neighboring patches located along the boundary are projected, these positions may correspond to the true contour between the neighboring patch Nei(p) and the current patch p.

[0223] exist Figure 8b In the example of , the current patch p is shown (gray) together with the re-projected 2D samples of its four neighboring patches (black lines). Note that the re-projected 2D samples may not form a closed contour.

[0224] In step 7030, the module may modify the profile C(p) of the current patch p to fit the reference profile RC(p). The profile C(p) is a set of at least one 2D sample of the current patch p. For example, those 2D samples define the current patch p, i.e., are located along its borders (edges).

[0225] According to an embodiment of step 7010, when 3D samples corresponding to 2D samples of the candidate patch CAN(p) and 3D samples corresponding to 2D samples of the neighboring patch Nei(p) are close to each other, the candidate patch CAN(p) may be the neighboring patch Nei(p) of the current patch p.

[0226] According to a variation of this embodiment of step 7010, the module may obtain 3D samples by back-projecting the 2D samples of the current patch p and the 2D samples of each candidate patch CAN(p). The module then performs a 3D search among all 3D samples to find a pair of closest 3D samples formed by a 3D sample relative to the 2D sample of the current patch p and a 3D sample relative to the 2D sample of the candidate patch CAN(p).

[0227] If a pair of 3D samples meets the proximity criteria, such as the distance is lower than a threshold, the candidate patch CAN(p) is considered to be a neighbor Nei(p) of the current patch p. Otherwise, it is not.

[0228] While this variant is very accurate, it is very expensive in terms of computational complexity.

[0229] According to a variation of this embodiment of step 7010, the module may determine whether the 3D bounding box of the candidate patch CAN(p) intersects with the 3D bounding box of the current patch p: if so, the candidate patch CAN(p) is considered to be a neighbor Nei(p) of the current patch p. Otherwise, it is not.

[0230] This variant is inexpensive in terms of computational complexity and is very easy to use, since the 3D bounding box of a patch is defined by the 3D coordinates of its origin (usually corresponding to the bottom, left, front corners) and its width, height, and depth. This information can be used as part of the auxiliary patch information PI.

[0231] According to an embodiment of step 7010, candidate patches may be obtained as follows:

[0232]

[0233] enter:

[0234] -patch list {p}

[0235] -OMI (Operational Miscellaneous Image)

[0236] -Geometry image for map 0 (GI0)

[0237] Output:

[0238] - For each patch p, we get CAN(p) with the following information appended

[0239] The expanded geometric image 0 (UGI0) of all points in p

[0240] Occupancy map (OM) of all points in p

[0241] 3D boundary points

[0242] 3D bounding box of 3D boundary points

[0243] deal with:

[0244] - As input, only the depth of points occupied in GI0 is used for optimization purposes, but both geometry GI0 and GI1 can be used. Using only GI0 is an optimization in the process, as it avoids the dilation stage on GI1 which can be time-consuming.

[0245] - Dilate Depth Map → Propagate the depth value of the current point to the “empty space”, so we can know the depth of the nearest point of any boundary patch.

[0246] -identifyBoundaryPoints → This phase can be iterative (usually the number of iterations is equal to 2),

[0247] And allows to identify whether a point must be considered a boundary patch point.

[0248] 1- For each point of the patch, if there is almost one neighbor pixel (top right, bottom, lower neighbor pixel) that is unoccupied, then the point is marked as a boundary point. We have a list of identified boundary points in the outer ring of the patch boundary, called IBP1.

[0249] 2- Repeat the same operation for all IBP1 points to obtain more points within the second ring of the patch boundary. The result is a list of second identified boundary points, called IPB2.

[0250] -For all points identified as IBP1 and IBP2, reconstruct the 3D points and compute the corresponding 3D bounding boxes.

[0251] IBP1 and BP2 are our CAN(p) lists

[0252] According to an embodiment of step 7020, at least one neighboring patch Nei(p) of the current patch p can be obtained from at least one candidate patch CAN(p) as follows:

[0253]

[0254] enter:

[0255] -Patch list

[0256] -Previous step candidate patch Can(p)

[0257] Output:

[0258] -Neighbor patch list →Nei(p)

[0259] deal with:

[0260] If the bounding boxes of p and CAN(p) intersect, then CAN(p) is set to Nei(p)

[0261] According to an embodiment of step 7030 , in sub-step 7031 , the contour C(p) may be modified by adding and / or removing at least one 2D sample of the current patch p.

[0262] like Figure 8c As shown, Figure 8b Compared to the patch of , some regions (at least one 2D sample) of the current patch p can then be removed (top right), while other regions can be added (bottom left). Figure 8d The modified current patch p is shown.

[0263] According to an embodiment of sub-step 7031 , the module processes all 2D samples of the contour C(p) individually.

[0264] Therefore, for each 2D sample of the contour C(p):

[0265] - if a 2D sample of contour C(p) is outside the reference contour RC(p), the module erodes such a 2D sample, removes it from the current patch p, then marks its neighboring 2D samples as 2D samples of contour C(p), and processes them recursively; and

[0266] - If a 2D sample of contour C(p) is inside the reference contour RC(p), the module dilates such 2D samples, then marks the newly appended 2D samples as 2D samples of contour C(p), and processes them recursively.

[0267] Figure 9 An example of contour fitting by recursively eroding / dilating 2D samples of a contour C(p) based on its position relative to a reference contour RC(p) is shown.

[0268] On the left, the results of the initial iteration are shown: the grid represents a portion of the canvas, where white squares represent empty locations, gray squares represent 2D samples of the contour C(p), and black squares represent 2D samples of the reference contour RC(p). 2D samples marked with "E" are eroded, i.e., removed; 2D samples marked with "D" are expanded, i.e., adjacent empty locations are filled with new 2D samples.

[0269] In the middle, the results of the second iteration are shown, while the results of the last iteration are shown on the right.

[0270] According to a variant of this embodiment of sub-step 7031, only 2D samples of the contour C(p) outside the reference contour RC(p) are processed. That is, 2D samples can be removed but not added.

[0271] According to a variant of this embodiment of sub-step 7031, only 2D samples of the contour C(p) that are inside the reference contour RC(p) are processed. That is, 2D samples can be added but not deleted.

[0272] According to a variant of the embodiment of sub-step 7031, as Figure 10 As shown in Figure 2, the module determines whether a 2D sample of the contour C(p) is inside or outside the reference contour RC(p) as follows:

[0273] For each 2D sample “p” of the contour C(p):

[0274] - Calculate the normal of the 2D sample "p", where the normal points outside the current patch p;

[0275] - Calculate the tangent of the normal of the 2D sample "p" that divides the local window into two halves;

[0276] - Estimate the amount of 2D samples of the reference contour RC(p) in each half of the local window: C in (p), the amount of 2D samples corresponding to the reference profile RC(p) in the inner half of the local window, and C out (p), the amount of 2D samples corresponding to the reference profile RC(p) in the outer half of the local window;

[0277] -If C in (p) is greater than C ou t(p), then the 2D sample “p” is outside the reference contour RC(p). Otherwise, the 2D sample “p” is inside the reference contour.

[0278] In one embodiment, to calculate the normal of a 2D sample "p", the following method is used. Other methods, such as methods using floating point operations, may also be used. The following method may provide several advantages for bit-accurate systems.

[0279] In this method, a 2D normal is calculated in a 2D discrete space by analyzing the values ​​of the placeholder map of pixels surrounding the current pixel "p". The value of the normal can be one of the following eight possible orientations: 0 degrees, 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees, 315 degrees. The orientation (normal) of pixel "p" can be selected based on the distribution of occupied placeholder pixels in a 3×3 window around position "p", for example, Figure 13 To further simplify the calculation of the normal, the orientation can be obtained directly from a pre-computed lookup table. The index to the lookup table is based on a combination of the placeholder values ​​of the eight neighboring pixels of pixel "p". The index IDX can be calculated as follows:

[0280]

[0281] Where tl is the top left pixel, t is the top pixel, tr is the top right pixel, l is the left pixel, r is the right pixel, bl is the bottom left pixel, b is the bottom pixel, and br is the bottom right pixel. The index IDX is defined in [0; 255] and can be used directly to access the pre-calculated normal value LUT. The normal value LUT is defined as follows:

[0282]

[0283] In this case, the calculation of the discrete normal "n" can be performed directly for all 2D contour points as follows:

[0284]

[0285] To estimate the reference contour RC(p) inside a given half of the local window ( Figure 10 Several methods can be used to determine the amount of 2D samples (black squares in ).

[0286] According to an embodiment, the amount of 2D samples of the reference profile RC(p) is the number of 2D samples of the reference profile RC(p) inside a given half, regardless of their position.

[0287] According to an embodiment, the amount of 2D samples of the reference profile RC(p) is a weighted sum taking into account the position of each 2D sample of the reference profile RC(p) relative to the current 2D sample “p”:

[0288]

[0289] where “r” represents the 2D sample of the reference contour RC(p) in a given half window, W represents the number of 2D samples “r”, and d(.,.) represents the Euclidean distance between the current 2D sample “r” and the 2D sample “p”.

[0290] According to an embodiment of sub-step 7030, the module can obtain the parametric modeling curve of the profile C(p) and the parametric modeling curve of the reference profile RC(p), and can modify the profile C(p) by adjusting the parameters of the parametric modeling curve of the profile C(p) so that it fits the parametric modeling curve of the reference profile RC(p).

[0291] For example, the modeling curve may be a spline curve.

[0292] According to an embodiment of step 7030, in sub-step 7032, the contour C(p) may be modified by repositioning (moving) at least one 2D sample of the current patch "p" along the projection axis of the current patch "p".

[0293] The purpose of this movement is to avoid seams between 3D samples corresponding to 2D samples of the current patch p and 3D samples corresponding to neighboring 2D samples of the reference contour RC(p).

[0294] Figure 11 An example of an embodiment of sub-step 7032 is shown.

[0295] On the top side, the initial state of the contour C(p) is shown, the input to step 7032: a grid representing a portion of the canvas, where white squares represent empty positions, grey squares represent 2D samples of the current patch p, and black squares represent 2D samples of the reference contour RC(p). The depth at each position is also shown as a number. It can be seen that although the 2D samples of the current patch p (grey squares) are neighbors of the 2D samples of the reference contour RC(p) (black squares) in the grid, their corresponding 3D samples are not necessarily adjacent to them. For example, on the left column, the 2D sample of the current patch p at a depth of 8 from the canvas is at a distance of 3 (to 11) or 4 (to 12) from the two 3D samples corresponding to its two neighboring 2D samples of the reference contour C(p).

[0296] On the bottom side, the filtering result out of sub-step 7032 is shown: the grey squares with black numbers are the repositioned 2D samples of the current patch p, effectively closer (in depth) to their respective neighbouring 2D samples of the reference contour RC(p).

[0297] According to an embodiment, steps 7020, subsets 7031 and 7032 follow the following process:

[0298] enter:

[0299] ●Patch list

[0300] Nei(p) as the output of 7010

[0301] 3D boundary points of Nei(p)

[0302] OM and GEO images

[0303] Output:

[0304] Modified OM after removing potential points (at least the same one or the one after removing the points)

[0305] deal with:

[0306] In the following processing, the # line represents an implementation optimization that avoids computing the distance to the current depth of all points of Nei(p) because it projects the nearest point and stores the corresponding depth in the depth map.

[0307] Iteratively perform the following operations for all points on the boundary:

[0308] - Count the number of neighbors considering neighbor pixels (top right, bottom, lower neighbor pixels)

[0309] - Use a simple step to quickly remove points when no neighbor points are found in the local window (top left, bottom right)

[0310] - If the number of neighbors is less than 4, the method described in European patent application n°19305361.8 filed on March 22, 2019 is followed.

[0311] As a variant, in sub-step 7031 , the 2D sample of the contour C(p) may be eroded or dilated.

[0312]

[0313] According to an embodiment, the smoothing parameters can be obtained as follows:

[0314]

[0315]

[0316] in

[0317]

[0318] According to an embodiment of step 7030, in sub-step 7033, the contour C(p) may be modified by adding at least one additional 3D sample between the 3D samples corresponding to the 2D samples of the current patch p and the 3D samples corresponding to the 2D samples of the reference contour RC(p).

[0319] This embodiment allows filling the gaps between these 3D samples.

[0320] Adding at least one additional 3D sample involves adding at least one 2D sample in the current patch p, but the position of the at least one 2D sample is already occupied by another 2D sample in the current patch p.

[0321] According to an embodiment, the same 2D sample of the current patch p corresponds to a 3D sample and at least one additional 3D sample, the depth value of the at least one additional 3D sample being obtained by linearly interpolating the depth values ​​of at least two 3D samples.

[0322] According to another embodiment, the geometry-based filling method described in international application no. PCT / IB19 / 050492 is used for adding at least one additional 3D sample.

[0323] According to an embodiment, the contour orientation may be considered as adding filled 3D points to the outer and / or inner boundaries of the patch.

[0324] Figure 12 An example of an embodiment of sub-step 7033 is shown.

[0325] On the top side, it shows Figure 11 The left column of the grid on the left and the plot of the 2D samples on the canvas (2D coordinates) and along the projection axis (depth). As shown in the figure, when scanning along the projection axis, seams (gaps) may appear between the 3D samples corresponding to the 2D samples of the current patch p (grey squares) and the 3D samples corresponding to the 2D samples of the reference contour RC(p) (black squares).

[0326] In the middle, it is shown Figure 11 The left column of the grid on the right (the output of sub-step 7032) and the plot of the 2D samples on the canvas (2D coordinates) and along the projection axis (depth).

[0327] On the bottom side, the result of the filtering of the output of sub-step 7033 is shown, when the positions of two new additional 3D samples (dashed squares) have been linearly interpolated from the positions of a pair of existing 3D samples with a depth difference greater than T. The threshold value can be fixed or signaled in the bitstream.

[0328] For example, T=1.

[0329] According to this sub-step 7033, the additional 3D samples may be created directly in 3D and therefore have no corresponding location in the canvas to encode their color information (texture).

[0330] According to an embodiment of sub-step 7033 , the module obtains the color information of at least one additional 3D sample by interpolating the color information of at least two 3D samples.

[0331] According to a variant, at least two 3D samples have their color information encoded in the bitstream.

[0332] According to a variant, said at least two samples are two 3D samples used to obtain a depth value of at least one additional 3D sample.

[0333] According to an embodiment of sub-step 7033, the module encodes the color information of at least one additional 3D sample in an unoccupied block of the texture image using, for example, the same method as the EOM codeword.

[0334] According to an embodiment of sub-step 7033 , the module encodes the color information of at least one additional 3D sample as an EOM texture patch, as described in European patent application no. 19305341.0.

[0335] Sub-steps 7031, 7032 and 7033 complement each other, but they can also be used independently. Figure 7 Examples of embodiments of the method are shown, but the variants may exclude one or any combination of them.

[0336] Figure 7 The method utilizes several parameters to adjust its behavior. Such parameters can be signaled in the bitstream in a similar way to how the current V-PCC specification signals the current geometric smoothing filter parameters (see "Geometric Smoothing" parameters in Section 7.3.18 of V-PCC).

[0337] Furthermore, one or more aspects of the present embodiments relate to smoothing or filtering one or more attributes, such as color attributes of points of a 3D point cloud or 3D reconstruction, in 2D space, rather than in the more expensive 3D space (e.g., using a 3D mesh as described below).

[0338] When encoding point clouds, one approach used in the prior art to achieve good compression efficiency is to project multiple geometric and texture attribute information onto the same location (pixel) in the 2D image. That is, several layers of information are encoded for each input point cloud. Typically, two layers are considered. This means that several 2D geometric and / or 2D texture attribute images are generated for each input point cloud. In some embodiments, two depth (for geometry) and color (for texture) images are encoded for each input point cloud.

[0339] Attribute smoothing (3D domain)

[0340] Since the quantization stage of attributes in video coding has an impact (impairment) on visual quality, the attributes (e.g. color) of two nearby 3D points from two different patches may have significantly different colors. For this reason, a 3D color smoothing process has been introduced in V-PCC to smooth the attributes of 3D points detected as 2D patch boundaries. Basically, this process calculates the average color of some selected points around themselves.

[0341] One version of this 3D attribute smoothing process (called 3D color smoothing) uses a KD tree to calculate the average attribute value. The process then searches N neighboring 3D points around the current point to calculate the attribute value centroid (e.g., average color). A process (called 3D color grid smoothing) then uses the 3D grid to calculate the average attribute value for each 3D 8x8x8 cell and interpolates the average attribute value for each boundary point by computing a trilinear filter on the 3D grid.

[0342] Figure 14 An example of 3D color mesh geometric smoothing is shown in and will be described here. 3D color mesh geometric smoothing consists of three main stages:

[0343] 1) Generation of color attribute centroid grids;

[0344] 2) Identification of boundary points; and

[0345] 3) Smoothing of 3D grid color attributes.

[0346] One or more aspects of this embodiment provide improvements and advantages for these stages, particularly for the stage of smoothing 3D grid color attributes described herein.

[0347] Color attribute centroid grid generation stage

[0348] Examples of this stage are described in, for example, Clause 9.6.3.2 "Identification of Attribute-Centric Grids" of V-PCC:

[0349] The input to this process is:

[0350] – An array containing the reconstructed positions, RecPcGeom[i], where 0 ≤ i < PointCnt,

[0351] – aIdx, the attribute index,

[0352] – An array containing the reconstructed attributes, RecPcAttr[aIdx][i], where 0 ≤ i < PointCnt, and

[0353] – numComps, indicating the number of attribute components.

[0354] The output of this process is:

[0355] – An array containing the reconstructed centroid grid attribute values, attrCenterGrid[i][k], where 0 ≤ i < numCells and k = 0 to numComps - 1, inclusive,

[0356] – An array containing the reconstructed centroid grid average luminance values, meanLuma[i], where 0 ≤ i < numCells, and

[0357] – An array containing the reconstructed centroid grid luminance medians, attrMean[i][k], where 0 ≤ i < numCells and k = 0 to numComps - 1, inclusive.

[0358] For all x, y, and z within the range 0 to (numCells - 1) (inclusive), and m within the range 0 to numComps - 1 (inclusive), attrCenterGrid[x][y][z][m] and meanLuma[x[y][z] are initialized to 0.

[0359] For all x, y, and z in the range 0 to (numCells-1), inclusive, apply clause 9.6.3.1 to obtain the arrays cellDoSmoothing[x][y][z], cellCnt[x][y][z], and cellPatchIdx[x][y][z].

[0360] For i = 0; to (PointCnt-1) (inclusive), the following applies:

[0361] The variables xIdx, yIdx, and zIdx are derived as follows:

[0362] xIdx=(RecPcGeom[i][0] / GridSize)

[0363] yIdx=(RecPcGeom[i][1] / GridSize)

[0364] zIdx=(RecPcGeom[i][2] / GridSize)

[0365] If cellDoSmoothing[xIdx][yIdx][zIdx] is equal to 1, the following applies:

[0366] for(k=0;k <numComps;k++)

[0367] attrCenterGrid[xIdx][yIdx][zIdx][k]+=RecPcAttr[aIdx][i][k];

[0368] If cellCnt[x][y][z] is greater than 0 for all x, y, and z in the range 0 to (numCells-1) (inclusive), the mean and median luminance values ​​of the attribute with index aIdx belonging to the points in that cell are calculated and assigned to the arrays meanLuma[x][y][z] and medianLuma[x][y][z], respectively. Additionally, the luminance value of attrCenterGrid is assigned to centroidLuma[x][y][z].

[0369] In the case of a color attribute, determine the brightness value corresponding to the attribute value attrValue[k], 0≤k<3, as follows:

[0370] lumaValue=0.2126*RecPcAttr[aIdx][0]+0.7152*RecPcAttr[aIdx][1]+0.0722*RecPcAttr[aIdx][2];

[0371] meanLuma[xIdx][yIdx][zIdx]+=lumaValue;

[0372] Export properties center grid.

[0373] for(k=0;k <numComps;k++)

[0374] attrCenterGrid[xIdx][yIdx][zIdx][k]=attrCenterGrid[xIdx][yIdx][zI

[0375] dx][k] / cellCnt[xIdx][yIdx][zIdx]

[0376] meanLuma[xIdx][yIdx][zIdx]=meanLuma[xIdx][yIdx][zIdx] / cellCnt[x

[0377] Idx][yIdx][zIdx]

[0378] Identification of boundary points

[0379] An example of this phase is described in e.g. clause 9.6.4 "Identification of boundary points" of the V-PCC:

[0380] The inputs to this process are:

[0381] – Placeholder frame oFrame

[0382] – Patch index information from the active geometry_patch_parameter_set

[0383] The output of this processing is:

[0384] The container recPCBoudanyPointType is used to save a list of patch boundary types of the reconstructed point cloud.

[0385] The variable BoundaryPointType identifies whether the point is near the patch boundary. BoundaryPointType can take values ​​in the range 0 to 1 (inclusive). A BoundaryPointType value of 0 indicates that the point is not near the patch boundary. The two-dimensional (asps_frame_width) by (asps_frame_height) array BPTypes stores the BoundaryPointType value for each point in the projected point cloud frame.

[0386] The one-dimensional list recPCBoundaryPointType stores the BoundaryPointType of each point (x, y) in the projected point cloud frame for which the value of oFrame[y][x] is not equal to 0. The BoundaryPointType values ​​are assigned to BPTypes[y][x] and recPCBoundaryPointType as follows:

[0387]

[0388] In the above allocation process, if (x–2) or (x+2) is outside the range of 0 to asps_frame_width–1 (inclusive), or (y–2) or (y+2) is between 0 and asps_frame_height

[0389] If the value of BPTypes[y][x] is outside the range of –1 (inclusive), the value of recPCBoundaryPointType[pointIndex] is assumed to be equal to 1.

[0390] 3D mesh color attribute smoothing

[0391] Figure 15 A graphic representation of this stage is provided. Figure 15 As shown, the average attribute value of each 3D 8x8x8 cell is determined. The average attribute value of each boundary point is then interpolated by calculating the trilinear filter on the 3D grid.

[0392] Another example of this phase is described in e.g. clause 9.6.4 "Identification of boundary points" of the V-PCC:

[0393] This process is called when afp_attribute_smoothing_enabled_flag[aIdx] is equal to 1, where aIdx is the attribute index.

[0394] The inputs to this process are:

[0395] – Placeholder image corresponding to the current point cloud frame at nominal resolution, oFrame

[0396] – attribute index, aIdx,

[0397] – The number of components corresponding to the attribute index aIdx, numComps,

[0398] – Attribute smoothing control parameter set, afp_attribute_smoothing_grid_size[aIdx], afp_attribute_smoothing_threshold[aIdx],

[0399] afp_attribute_smoothing_threshold_local_entropy[aIdx],

[0400] afp_attribute_smoothing_threshold_variation[aIdx], and

[0401] afp_attribute_smoothing_threshold_difference[aIdx]

[0402] – Array of reconstructed attribute values containing the attribute index aIdx,

[0403] RecPcAttrSmIn[aIdx][i][j], 0 ≤ i < PointCnt, 0 ≤ j < numComps,

[0404] – Array of reconstructed (possibly smoothed) reconstruction positions,

[0405] RecPcGeomSmOut[i][j], 0 ≤ i < PointCnt, 0 ≤ j ≤ 2, and

[0406] – Patch index information corresponding to each point in RecPcGeomSmOut

[0407] The output of this process is:

[0408] – Array of reconstructed smoothed attribute values containing the attribute index aIdx,

[0409] RecPcAttrSmOut[aIdx][i][j], 0 ≤ i < PointCnt, 0 ≤ j < numComps.

[0410] The variable GridSize is set to AttributeSmoothingGridSize[aIdx].

[0411] As described in 9.6.3.2, the number of export cells numCells and the arrays of attrCenterGrid[i] and cellDoSmoothing[x][y][z] are derived, where i = 0 to numCells - 1 (inclusive):

[0412] For i = 0 to (PointCnt - 1) (inclusive), the following applies:

[0413] – otherClusterPtCnt is set equal to 0.

[0414] To determine the points on the patch boundary, Clause 9.6.4 is called to generate the array recPCBoundary[idx], 0 ≤ idx < PointCnt, which identifies whether each point from RecPcGeomSmOut is a boundary point.

[0415] For i = 0 to (PointCnt - 1) (inclusive), the following applies:

[0416] – If recPCBoundary[i] is equal to 1, then the following steps are performed:

[0417] – The variables pointGeom[j], where j ranges from 0 to 2 (inclusive), and pointAttr[k], where k ranges from 0 to (numComps - 1) (inclusive), are defined as follows:

[0418] for (j = 0; j < 3; j++)

[0419] pointGeom[j] = RecPcGeomSmOut[i][j]

[0420] for (k = 0; k < numComps; k++)

[0421] pointAttr[k] = RecPcAttrSmIn[aIdx][i][k]

[0422] Sub-clause 9.6.5 applies for the derivation of:

[0423] – The 2x2x2 grid neighborhood corresponding to the current position, pointGeom[i]

[0424] – The upper left corner of the 2x2x2 grid, s[i],

[0425] – The 2x2x2 grid position associated with the current position, t[i], i = 0 to 2 (inclusive)

[0426] – The boolean value otherClusterPtCn

[0427] If otherClusterPtCnt is equal to 1, then the following applies:

[0428] – The change in the attributes of the 2×2×2 cell neighborhood of the current cell is determined as follows:

[0429] The isOriginalCell array of a 2×2×2 cell neighborhood is determined as follows:

[0430]

[0431] – The change in attribute value within a cell (indicating activation) and the difference between the brightness centroid values ​​of the original cell and neighboring cells are determined as follows:

[0432]

[0433]

[0434] – Apply the 8-tap trilinear filter described in 9.6.6 to the attribute centroid, applying attrCentroid[x][y][z][k] in a 2×2×2 neighborhood where 0≤x,y,z<1,0≤k <numComps-1:

[0435] – If the following distortion criteria are met, the output of the trilinear filter will be added to recPCaSmOut.

[0436] – The variable distToCentroid2, defined as abs(Ycell-Ycur)*10 greater than afp_attribute_smoothing_threshold_smoothing[aIdx], where Ycell and Ycur are the centroid of the point in the current cell (attrCentroid) and the brightness value of the current point (pointAttr), respectively.

[0437] Therefore, one or more aspects of the present embodiments provide a novel process or apparatus for smoothing the attributes (e.g., color) of points for 3D reconstruction of a point cloud. Specifically, unlike the 3D color smoothing process described above, which requires interpolation using a 3D mesh, one or more aspects can provide color smoothing in a 2D plane to reduce complexity. One or more aspects calculate the color centroids of patch boundary points and smooth the patch boundary points based on these centroids in 2D space. The attribute frame is then updated prior to the attribute reconstruction process.

[0438] Figure 16 One or more aspects of the present color smoothing embodiments are shown, which provide a new approach to solving color problems at patch boundaries.

[0439] As described above, the patch boundary filtering process projects the boundary points of neighboring patches in the 2D patch domain to estimate the connectivity between patches and correct the geometry before the reconstruction process. One advantage is that it operates in the 2D domain and is less complex than in 3D.

[0440] Similarly, one or more aspects of the present embodiment correct the color of 3D points by calculating the centroid of the colors of the points surrounding the current patch and the neighboring patches for each 2D boundary point of the patch before the reconstruction process. In this case, the color image is updated before deprojection from 2D to 3D. The 3D reconstructed points will directly benefit from the reduced complexity of the color smoothing process in 2D. Therefore, one or more aspects avoid performing color smoothing in 3D.

[0441] like Figure 16 As shown, two main treatments are proposed:

[0442] 1. Color centroid map / frame generation: For each boundary point of the patch, calculate the color centroid frame, such as Figure 17A and 17B shown.

[0443] 2.2D color smoothing: For all previous color centroids, calculate the smoothed color centroid using all boundary points and update the attribute frame: update the attribute pixel value with the smoothed color centroid, such as Figure 17A and 17B shown.

[0444] Color centroid map / frame generation

[0445] For each boundary point of the patch, the frame of the color centroid is calculated, such as Figure 17A and 17B shown.

[0446] Another detailed example is shown below:

[0447] enter:

[0448] Attribute color image: AFrame[aIdx][mapIdx][cIdx][y][x], decoded attribute frame at nominal resolution, where mapIdx = 0..vpcc_map_count_minus1,

[0449] y=0..asps_frame_height-1, and x=0..asps_frame_width-1,

[0450] aIdx = 0..ai_attribute_count - 1, and

[0451] cIdx=0..ai_attribute_dimension_minus1[aIdx].

[0452] Placeholder: OFrame[y][x][0], the first component of the placeholder corresponding to the current point cloud frame at nominal resolution, where y = 0..asps_frame_height–1,

[0453] x=0..asps_frame_width–1.

[0454] The index of the patch p.

[0455] Output:

[0456] Color centroid image: CFrame[aIdx][mapIdx][cIdx][y][x], a frame containing the calculated centroid color attribute values, where mapIdx = 0..vpcc_map_count_minus1,

[0457] y=0..asps_frame_height–1, and x=0..asps_frame_width–1,

[0458] aIdx = 0..ai_attribute_count – 1, and

[0459] cIdx=0..ai_attribute_dimension_mins 1[aIdx].

[0460] *asps-Atlas sequence parameter set (defines the syntax structure of parameters).

[0461] deal with:

[0462] For all points (u, v) of the placeholder frame OFrame[u][v][0]

[0463] / / Boundary points identified by the module (i.e., if the 2D point is the boundary of the patch)

[0464] if(OFrame[u][v][0]&&

[0465] (!OFrame[u+1][v][0]||

[0466] !OFrame[u-1][v][0]||

[0467] !OFrame[u][v-1][0]||

[0468] ! OFrameu][v+1][0]))

[0469] / / Generate module attribute center (calculate the color centroid of the current point and store the value in the color centroid frame)

[0470] nbPoints = 0;

[0471] For all points (up, vp) in the neighborhood (*) of (u, v) of the same patch index p

[0472]

[0473] / / Module 3D boundary point generation

[0474] [1] Clause 9.4.6 “Reconstructing 3D Point Positions” of (SoTA)

[0475] (*) The neighborhood calculation can be done like this:

[0476] On a rectangular box of size [u-boxSizeU; u+boxSizeU][v-boxSizeV, v+boxSizeV], usually boxSizeU = 4 and boxSizeV = 4

[0477] On the circle around point (u, v): all points p(up, vp) with the condition sqrt((up-u)^2+(vp-v)^2) < distance, usually distance = 5

[0478] On a set of nearest points, usually N24-neighbors

[0479] Note that the output can be one image for all patches (same as geometry and texture images), or segmented over several images per patch.

[0480] 2D Color Smoothing

[0481] like Figure 18A and 18B As shown, for all boundary points, one or more aspects of the 2D color smoothing embodiment utilize all boundary points (using Figure 17A and 17B ) to calculate the smoothed color centroid and update the attribute value using the smoothed color centroid.

[0482] Figure 18B One or more aspects of the present 2D color smoothing embodiment are shown. Figure 18B Panel A of FIG. 5 shows points on the reference contour derived as described above (the reference contour is shown in light grey on the border). Figure 18B Panel B shows the centroid of a circle with neighboring points used to update the color centroid, with all identified patch boundaries (eg, BoundaryFrame). Figure 18B Panel C shows the smoothed color patch after 2D color smoothing has been applied.

[0483] Another example of smooth colors can be provided by:

[0484] enter:

[0485] Placeholder: OFrame[y][x][0], the first component of the placeholder corresponding to the current point cloud frame at nominal resolution, where y = 0..asps_frame_height–1,

[0486] x=0..asps_frame_width–1.

[0487] ●The index of the patch p.

[0488] Color centroid image: CFrame[aIdx][mapIdx][cIdx][y][x], a frame containing the calculated centroid color attribute values, where mapIdx = 0..vpcc_map_count_minus1,

[0489] y=0..asps_frame_height–1, and x=0..asps_frame_width–1,

[0490] aIdx = 0..ai_attribute_count – 1, and

[0491] cIdx=0..ai_attribute_dimension_mins1[aIdx].

[0492] RC(p), the reference contour of patch p by reprojecting the 2D samples of at least one neighboring patch to the projected position.

[0493] Output:

[0494] Attribute color image: Aframe[aIdx][mapIdx][cIdx][y][x], attribute frame updated at nominal resolution, where mapIdx = 0..vpcc_map_count_minus1,

[0495] y=0..asps_frame_height-1, and x=0..asps_frame_width-1,

[0496] aIdx = 0..ai_attribute_count - 1, and

[0497] cIdx=0..ai_attribute_dimension_minus1[aIdx].

[0498] deal with:

[0499] / / Module reference outline creation

[0500] For each patch

[0501] For each reference profile, RC(p):

[0502] Project the 3D boundary points into the current patch 2D domain

[0503] For each projected point, the associated color centroid is retrieved from CFrame so that we have a reference contour with an associated attribute value ARC(p)

[0504] / / Module smoothing color

[0505] The modules or processes for reference contour creation shown in the above embodiments have been extensively described. For the smooth color module or process, two additional variations are provided below and are respectively as follows: Figure 19 and Figure 20 shown.

[0506] Figure 19 Shown Figure 18A The first variant of the smooth color module or process shown. As shown in the following process, for each point (u, v) of the current patch in the boundary area to be filtered (i.e., a point close to the boundary patch), if the attribute value of the point is less than the threshold, the nearest point is found for each ARC(P). This value is added to the cumulative centroid value, otherwise the value is discarded. The average value is then retrieved by dividing the accumulated centroid value by the number of added attribute values, and this calculated value is assigned to the attribute frame (u, v). Equations 1 and 2 below determine the mean / average of the centroid.

[0507] Another detailed example is shown below:

[0508] For each 2D point (u, v) close to the boundary of the patch (*), p

[0509] nbPoints = 0;

[0510] Find the nearest 3D point in ARC(p),q

[0511] If distance (p, q) < threshold (usually threshold = 6)

[0512]

[0513] Figure 20 Shown Figure 18AAnother variation of the smooth color module or process shown. In this variation, the portion (highlighted in grey) that determines the mean / average of the centroids shown in the above variation has been further improved.

[0514] like Figure 20 As shown, the point (u, v) of patch p whose nearest centroid belongs to each identified boundary patch and whose position is close enough (according to a threshold) to the current deprojected (u, v) point (i.e., 3D point) is identified and called the centroid C

[0515] If C belongs to the current patch, activation is calculated according to process 1, otherwise activation is calculated according to another process 2. As an example, process 1 corresponds to the difference between the average values ​​of the attributes belonging to the centroid determined in the above attribute center generation step. As an example, process 2 corresponds to process 1, but is enhanced by verifying that the brightness difference between the current brightness point and the centroid is less than a distance threshold.

[0516] If the calculated activation is low (below a threshold), the centroid of the patch boundary is added to the calculation of the filter. Otherwise, the original properties of the point (u, v) are used in the filtering. The filtering can be an average weighted by the number of points in the neighborhood, or an average weighted by the distance between the current point (u, v) and the position of the identified centroid. This filtering is different from the trilinear filtering described above.

[0517] Another detailed example is shown below:

[0518] For each 2D point (u, v) close to the boundary of the patch (*), p

[0519] nbPoints = 0;

[0520] Find the nearest 3D point in ARC(p),q

[0521] If distance (p, q) < threshold (usually threshold = 6)

[0522]

[0523] Figure 21An example of a process 2100 according to one or more aspects of the present embodiment is shown. At 2110, one or more attributes of a point cloud are represented by a canvas of a patch of 2D samples and the values ​​of the 2D samples. At 2120, at least one 2D sample of a current patch is modified to fit a reference contour obtained by reprojecting at least one 2D sample of at least one neighboring patch of the current patch onto a projection plane onto which the 2D samples of the current patch have been projected. At 2130, one or more attributes of the point cloud are filtered based on the modified at least one 2D sample of the current patch. Additionally, in one embodiment, the one or more attributes of the point cloud may include color information. In yet another embodiment, the one or more attributes of the point cloud may include geometric information, such as depth information.

[0524] exist Figure 1-21 In the present invention, various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions can be modified or combined.

[0525] Some examples are described with respect to block diagrams and operational flow charts. Each block represents a circuit element, module, or code portion that includes one or more executable instructions for implementing a specified logical function. It should also be noted that in other implementations, the functions mentioned in the blocks may not occur in the order indicated. For example, depending on the functions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order.

[0526] The implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a computer program, a data stream, a bit stream, or a signal. Even if discussed only in the context of a single implementation form (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., an apparatus or a computer program).

[0527] These methods can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. A processor also includes a communication device.

[0528] In addition, these methods can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the implementation) can be stored on a computer-readable storage medium. A computer-readable storage medium can take the form of a computer-readable program product embodied in one or more computer-readable media and having computer-readable program code embodied thereon that can be executed by a computer. As used herein, a computer-readable storage medium can be considered a non-transitory storage medium that has an inherent ability to store information therein and an inherent ability to retrieve information therefrom. A computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Although more specific examples of computer-readable storage media to which the present embodiments can be applied are provided below, as will be readily understood by one of ordinary skill in the art, the following is merely illustrative and not an exhaustive list: portable computer disk; hard disk; read-only memory (ROM); erasable programmable read-only memory (EPROM or flash memory); portable compact disk read-only memory (CD-ROM); optical storage device; magnetic storage device; or any suitable combination of the foregoing.

[0529] The instructions may form an application program tangibly embodied on a processor-readable medium.

[0530] Instructions can be, for example, in hardware, firmware, software, or a combination thereof. Instructions can reside, for example, in an operating system, a separate application, or a combination of both. Thus, a processor can be characterized as, for example, a device configured to perform a process and a device that includes a processor-readable medium (such as a storage device) having instructions for performing the process. Furthermore, in addition to or in lieu of instructions, a processor-readable medium can store data values ​​generated by an implementation.

[0531] The device can be implemented with, for example, appropriate hardware, software, and firmware. Examples of such devices include personal computers, laptops, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, head-mounted display devices (HMDs, perspective glasses), projectors, "cave" (systems comprising multiple displays), servers, video encoders, video decoders, post-processors for processing outputs from video decoders, pre-processors for providing inputs to video encoders, network servers, set-top boxes, and any other device for processing point clouds, videos or images, or other communication devices. It should be clear that the device can be mobile and can even be installed in a mobile vehicle.

[0532] The computer software may be implemented by the processor 6010 or by hardware, or by a combination of hardware and software. As a non-limiting example, embodiments may also be implemented by one or more integrated circuits. The memory 6020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, for example, optical storage devices, magnetic storage devices, semiconductor-based storage devices, fixed memory, and removable memory. The processor 6010 may be of any type suitable for the technical environment and may, as a non-limiting example, include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0533] It will be apparent to one of ordinary skill in the art that implementations may generate various signals that are formatted to carry information that can be stored or transmitted, for example. The information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, signals may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0534] The terms used herein are merely for describing specific embodiments, rather than for limiting. As used herein, the singular "a," "an," and "the" may also be intended to include plural forms, unless the context clearly indicates otherwise. It will be further understood that, when used in this specification, the term "include / comprise" may specify the existence of stated, for example, features, integral bodies, steps, operations, elements, and / or components, but does not exclude the existence or addition of one or more other features, integral bodies, steps, operations, elements, components, and / or their groups. In addition, when an element is referred to as "responding" or "connected" to another element, it may directly respond to or be connected to another element, or there may be an intermediate element. On the contrary, when an element is referred to as "directly responding" or "directly connected" to other elements, there is no intermediate element.

[0535] It should be understood that the use of any symbol / term " / ", "and / or", and "at least one of", for example, in the case of "A / B", "A and / or B", and "at least one of A and B", may be intended to include selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As another example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). As will be apparent to one of ordinary skill in this and related arts, this can be extended to as many items as listed.

[0536] Various numerical values ​​may be used in this application, such as T. The specific values ​​may be used for example purposes, and the described aspects are not limited to these specific values.

[0537] It should be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the teachings of this application. There is no implicit order between the first element and the second element.

[0538] Reference to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof are often used to convey that a particular feature, structure, characteristic, etc. (described in connection with the embodiment / implementation) is included in at least one embodiment / implementation. Thus, the appearances of the phrases "in one embodiment," "in an embodiment," "in one implementation," or "in an implementation," and any other variations thereof, in various places throughout this application are not necessarily all referring to the same embodiment.

[0539] Similarly, the phrases "according to an embodiment / example / implementation" or "in an embodiment / example / implementation" and their variations are often used to convey that a particular feature, structure, or characteristic (described in conjunction with an embodiment / example / implementation) may be included in at least one embodiment / example / implementation. Therefore, the appearances of the phrases "according to an embodiment / example / implementation" or "in an embodiment / example / implementation" in various places in the specification do not necessarily refer to the same embodiment / example / implementation, nor are they necessarily separate or alternative embodiments / examples / implementations that are mutually exclusive of other embodiments / examples / implementations.

[0540] Reference numerals appearing in the claims are for illustrative purposes only and have no limiting effect on the scope of the claims.The present embodiments / examples and variations may be employed in any combination or subcombination even if not explicitly described.

[0541] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented in the form of a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0542] Although some of the figures include arrows on communication paths to illustrate a primary direction of communication, it is understood that communication can occur in the opposite direction of the depicted arrows.

[0543] Various implementations include decoding. As used herein, "decoding" may encompass, for example, all or part of the processing performed on a received point cloud frame (possibly including a received bitstream encoding one or more point cloud frames) to produce a final output suitable for display or further processing in the reconstructed point cloud domain. In various embodiments, such processing includes one or more of the processes typically performed by an image-based decoder.

[0544] As a further example, in one embodiment, "decoding" may refer only to entropy decoding, in another embodiment, "decoding" may refer only to differential decoding, and in another embodiment, "decoding" may refer to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" refers specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the particular implementation and is considered well understood by those skilled in the art.

[0545] Various implementations involve encoding. In a manner similar to the discussion above regarding "decoding," "encoding," as used in this application, can encompass, for example, all or part of the processing performed on an input point cloud frame to produce an encoded bitstream. In various embodiments, such processing includes one or more of the processes typically performed by an image-based decoder.

[0546] As a further example, in one embodiment, "encoding" may refer only to entropy encoding, in another embodiment, "encoding" may refer only to differential encoding, and in another embodiment, "encoding" may refer to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" refers specifically to a subset of operations or generally to a broader encoding process will be clear based on the context of the particular implementation and is considered well understood by those skilled in the art.

[0547] Note that the syntax elements used here are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0548] Various embodiments relate to rate-distortion optimization. In particular, in the coding process, given computational complexity constraints, a balance or trade-off between rate and distortion is often considered. Rate-distortion optimization can generally be formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all coding options, including all considered modes or coding parameter values, with a complete evaluation of their coding costs and associated distortion in the reconstructed signal after encoding and decoding. Faster approaches can also be used to save coding complexity, particularly by computing approximate distortion based on a prediction or prediction residual signal rather than the reconstructed signal. A hybrid of these two approaches can also be used, such as by using approximate distortion for only some possible coding options and full distortion for others. Other approaches only evaluate a subset of possible coding options. More generally, many approaches employ any of a variety of techniques to perform optimization, but optimization does not necessarily involve a complete evaluation of both coding cost and associated distortion.

[0549] Furthermore, this application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0550] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0551] Furthermore, the application may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from a memory device). Furthermore, during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often involved in one way or another.

[0552] In addition, as used herein, the word "signal" is particularly used to indicate something to the corresponding decoder. In this way, in an embodiment, the same parameters can be used on the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit savings are achieved in various embodiments. It should be understood that signaling can be accomplished in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signal" is mentioned above, the word "signal" can also be used as a noun here.

[0553] Many implementations have been described. However, it should be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. Furthermore, one of ordinary skill in the art will understand that other structures and processes may be substituted for those disclosed, and that the resulting implementations will perform at least substantially the same functions in at least substantially the same manner to achieve at least substantially the same results as the disclosed implementations. Therefore, this application contemplates these and other implementations.

[0554] Additional embodiments may include one or more of the following features, devices, or aspects, alone or in any combination, across the various claim categories and types:

[0555] Filtering or smoothing one or more attributes of a point cloud in 2D space rather than 3D space.

[0556] Using the reference contours of the patch to filter or smooth one or more attributes of the point cloud in 2D space.

[0557] Filtering or smoothing one or more attributes of the point cloud in 2D space based on one or more centroids on a reference contour of the patch.

[0558] • The reference contour according to one or more of the described embodiments is obtained by reprojecting at least one 2D sample of at least one neighboring patch of the current patch onto a projection plane onto which the 2D sample of the current patch has been projected.

[0559] The filtering or smoothing of one or more attributes of the point cloud according to one or more of the described embodiments is obtained by reprojecting the filtered one or more attributes in 2D into 3D based on the modified at least one 2D sample of the current patch.

[0560] • One or more attributes of the point cloud according to one or more of the described embodiments include color information.

[0561] • One or more attributes of a point cloud according to one or more of the described embodiments include geometric information, such as depth information.

[0562] • A bitstream or signal comprising one or more of the described syntax elements or variants thereof.

[0563] • A bitstream or signal including syntax conveying information generated according to any of the described embodiments.

[0564] Inserting an element in the signaling syntax that enables the decoder to adjust the residual in a way that corresponds to that used by the encoder.

[0565] • Creating and / or sending and / or receiving and / or decoding a bitstream or signal comprising one or more of said syntax elements or variants thereof.

[0566] • Creating and / or sending and / or receiving and / or decoding according to any of the described embodiments.

[0567] • A method, process, apparatus, medium storing instructions, medium storing data or signal according to any described embodiment.

[0568] • A television, set-top box, mobile phone, tablet or other electronic device that performs the adaptation of the filter parameters according to any of the described embodiments.

[0569] • A television, set-top box, mobile phone, tablet or other electronic device that performs the adaptation of the filter parameters according to any of the described embodiments and displays (e.g. using a monitor, screen or other type of display) the resulting image.

[0570] A television, set-top box, mobile phone, tablet or other electronic device that selects (e.g. using a tuner) a channel to receive a signal comprising an encoded image and performs the adaptation of the filter parameters according to any of the described embodiments.

[0571] A television, set-top box, mobile phone, tablet or other electronic device that receives over the air (eg using an antenna) a signal comprising the encoded image and performs the adaptation of the filter parameters according to any of the described embodiments.

Claims

1. A method comprising: - obtaining a patch representing the geometry of a 3D sample of the point cloud frame projected on a first projection plane, - obtaining a reference contour by reprojecting at least one 2D sample of at least one neighboring patch of said patch onto said first projection plane, - modifying at least one 2D sample of the patch to fit the reference contour.

2. The method according to claim 1, further comprising: The at least one neighboring patch is obtained from at least one candidate patch.

3. The method according to claim 2, wherein: When a 3D sample corresponding to a 2D sample of a candidate patch and a 3D sample corresponding to the patch are close to each other, the candidate patch is a neighboring patch of the patch.

4. The method according to claim 3, wherein: When the 3D bounding box of a candidate patch intersects the 3D bounding box of the patch, the candidate patch is a neighboring patch of the patch.

5. The method according to claim 1, wherein The 2D sample of the patch forms an outline of the patch.

6. The method according to claim 5, wherein: The contour of the patch is modified by adding and / or removing at least one 2D sample of the patch.

7. The method according to claim 6, wherein: If the 2D samples of the contour of the patch are inside or outside the reference contour, the 2D samples of the contour of the patch are modified.

8. The method according to claim 5, wherein The contour of the patch is modified by repositioning at least one 2D sample of the patch along a projection axis of the patch.

9. The method according to claim 5, wherein: The contour of the patch is modified by adding at least one additional 3D sample between at least one 3D sample corresponding to the 2D sample of the patch and at least one 3D sample corresponding to the 2D sample of the reference contour.

10. The method according to claim 1, further comprising: Filtering properties of the 2D samples of the patch.

11. The method according to claim 10, wherein: The attributes of the 2D samples of the patch include color information or depth information.

12. An apparatus comprising one or more processors configured to: - obtaining a patch representing the geometry of a 3D sample of the point cloud frame projected on a first projection plane, - obtaining a reference contour by reprojecting at least one 2D sample of at least one neighboring patch of said patch onto said first projection plane, - modifying at least one 2D sample of the patch to fit the reference contour.

13. A non-transitory computer-readable medium comprising instructions for causing one or more processors to perform the method of one of claims 1 to 11.

Images