Method and device for coding the geometric shape of point clouds

Abstract

To provide an encoding method and device for improving projection of a point cloud onto a two-dimensional image.SOLUTION: An encoding method includes: obtaining a first encoded depth image by encoding a first depth image in a bitstream, the first depth image representing depth values of nearer points of a point cloud; determining and encoding a depth coding mode per image region, the depth coding mode indicating whether depth values in an image region of a second depth image are also encoded in the bitstream, the second depth image representing depth values of farther points of the point cloud; and if at least one depth coding mode indicates that depth values in an image region of the second depth image are encoded in the bitstream, encoding at least partially the second depth image in the bitstream.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This principle generally relates to the coding and decoding of point clouds representing the outer surface of 3D objects. In particular, but not limited to, the technical field of this principle relates to the coding / decoding of depth images representing the geometric shape of such point clouds. [Background technology]

[0002] This section is intended to introduce to the reader various aspects of the technical field, which may relate to the various aspects of the principle described and / or claimed below. This discussion is intended to provide the reader with background information to facilitate a better understanding of the various aspects of the principle. Therefore, it should be understood that these descriptions should be read from this perspective and not as an endorsement of prior art.

[0003] A point cloud is a collection of data points in some coordinate system. In a three-dimensional coordinate system (3D space), these points are typically intended to represent the outer surface of a 3D object. Each point in a point cloud is often defined by its location (X, Y, and Z coordinates in 3D space) and, in some cases, by other relevant attributes such as color, transparency, reflectivity, and two-component normal vector, expressed in, for example, RGB or YUV color space.

[0004] Typically, a point cloud is represented as a set of six-component points (X, Y, Z, R, G, B) or equivalent (X, Y, Z, Y, U, V), where (X, Y, Z) defines the coordinates of a colored point in 3D space, and (R, G, B) or (Y, U, V) defines the color of this colored point.

[0005] A point cloud can be static or dynamic, depending on whether the cloud evolves over time. Note that in the case of a dynamic point cloud, the number of points is not constant, but rather usually evolves over time. Therefore, a dynamic point cloud is a time-ordered list of the set of points.

[0006] In practice, point clouds can be used for a variety of purposes, such as 3D scanning of objects like statues and buildings in cultural heritage / architecture, and sharing the spatial composition of these objects without the need for transmission or visits. They are also a way to ensure the preservation of knowledge about objects in case they are destroyed, for example, if a temple is destroyed by an earthquake. Such point clouds are typically static, colored, and massive.

[0007] Another use case is terrain and cartography, where 3D representation is used to ensure that maps are not limited to a flat plane and can include relief. Currently, Google Maps is a good example of 3D mapping, although it uses meshes rather than point clouds. Nevertheless, point clouds can be a suitable data format for 3D mapping, and such point clouds are typically static, colored, and massive.

[0008] The automotive industry and autonomous vehicles are also areas where point clouds can be useful. Autonomous vehicles need to be able to "investigate" their environment in order to make appropriate driving decisions based on the reality in their immediate vicinity. Conventional sensors like LiDAR generate dynamic point clouds that are used in the decision engine. These point clouds are not intended to be seen by humans, are usually small, not necessarily colored, and are dynamic and frequently captured. They may have other attributes, such as reflectivity provided by LiDAR, because this attribute can be good information about the material of the sensed object and can help in decision-making.

[0009] Virtual reality and immersive worlds have recently become a hot topic, with many predicting them to be the future of 2D flat imagery. The basic idea is to immerse the viewer in the entire environment around them, in contrast to standard television, where the viewer can only see the virtual world in front of them. Depending on the viewer's degree of freedom in the environment, there are several levels of immersion. Colored point clouds are a candidate format suitable for delivering virtual reality (or VR) worlds. They can be static or dynamic, typically average in size, and never exceed millions of points at once.

[0010] Point cloud compression is only successful in saving and transmitting 3D objects in immersive worlds if the bitstream size is small enough to actually allow saving / transmitting them to the end user.

[0011] It is crucial to be able to deliver dynamic point clouds to end users while maintaining an acceptable (or preferably very good) experience quality and consuming a reasonable bitrate. Efficiently compressing these dynamic point clouds is a key point for making the delivery chain of immersive worlds practical.

[0012] Image-based point cloud compression techniques are gaining increasing popularity due to their combination of compression efficiency and low complexity. These techniques proceed in two main steps: first, the point cloud, i.e., 3D points, are projected (orthographically projected) onto a 2D image. For example, at least one depth image represents the geometric shape of the point cloud, i.e., the spatial coordinates of the 3D points in 3D space, and at least one texture image represents the attributes associated with the 3D points of the point cloud, e.g., the texture / color information associated with these 3D points. Next, these techniques encode such depth and texture images using conventional video encoders.

[0013] Image-based point cloud compression techniques leverage the capabilities of 2D video encoders such as HEVC ("ITU-T H.265 Telecommunication standardization sector of ITU(10 / 2014), series H: audiovisual and multimedia systems, audiovisual service infrastructure, i.e., video coding, high-efficiency video coding, Recommendation ITU-T H.265") to achieve excellent compression performance while keeping complexity low by using a simple projection scheme.

[0014] One of the challenges of image-based point cloud compression techniques is that point clouds may not be suitable for projection onto images, especially when the point distribution follows a surface with many wrinkles (such as uneven areas in clothing) or when the point distribution does not follow a surface at all (such as fur or hair). In these situations, image-based point cloud compression techniques suffer from low compression efficiency (requiring many small projections, which reduces the efficiency of 2D video compression) or low quality (because it is difficult to project the point cloud onto the surface).

[0015] One approach used in modern technology to mitigate this problem involves projecting multiple geometric shapes and texture information onto the same spatial location (pixel) in an image. In other words, several depth and / or texture images may be generated for each 3D point in the point cloud.

[0016] This is, for example, the case of a so-called test model category 2 point cloud encoder (TMC2) as defined in ISO / IEC JTC1 / SC29 / WG11 / N17248, conducted in Macau, China in October 2017, where the point cloud is orthogonally projected onto a projection plane. Next, two depth values ​​are associated with each coordinate on the projection plane: one representing the depth value associated with the nearest point (minimum depth value) and the other representing the depth value of the farthest point (maximum depth value). Next, a first depth image is generated from the minimum depth value (D0), and a second depth image is generated from the difference between the maximum (D1) and minimum (D0) depth values ​​satisfying D1-D0 <= surface thickness, where the surface thickness is the maximum surface thickness.

[0017] Next, the depth image and associated metadata are encoded and decoded. Then, the geometric shape of the point cloud is reconstructed from the decoded depth image. Next, a color / texture is assigned to each point in the reconstructed point cloud, and two texture images are generated from the assigned colors / textures. Finally, the two texture images are encoded. In this way, the second depth image contains high-frequency features, such as significant contours, that are extremely difficult to code. [Overview of the Initiative]

[0018] The following provides a simplified overview of the principle in order to provide a basic understanding of some aspects of it. This overview is not a comprehensive summary of the principle. It is not intended to identify the main or important elements of the principle. The following overview merely presents some aspects of the principle in a simplified form as a prelude to the more detailed explanation provided below.

[0019] The principle of the present invention is a method for encoding the depth values ​​of points orthographically projected onto a point cloud projection plane, and aims to improve at least one of the shortcomings of the prior art. - Obtaining a first encoded depth image by encoding a first depth image in a bitstream, wherein the first depth image represents the depth values ​​of the nearest points in the point cloud. - In the bitstream, the depth coding mode for each image region is determined and encoded, and the depth coding mode indicates whether the depth values ​​within the image region of the second depth image are also encoded in the bitstream, and the second depth image represents the depth values ​​of the farther points in the point cloud, and this is determined and encoded. - If at least one depth coding mode indicates that depth values ​​within the image region of the second depth image are encoded in the bitstream, then the second depth image is at least partially encoded in the bitstream.

[0020] According to one embodiment, determining whether a depth value within an image region of a second depth image is encoded in a bitstream includes: - obtaining a decoded first depth image by decoding a first encoded depth image and a decoded second depth image by encoding and decoding the second depth image; - calculating a first rate-distortion cost by considering a first distance and a first bitrate, wherein the first distance is calculated between a depth value within a same-position image region of the decoded first depth image and a depth value within a same-position image region of the decoded second depth image, and the data rate is calculated with respect to encoding of the second depth image; - calculating a second rate-distortion cost by considering a second distance, wherein the data rate is considered to be zero here, and the second distance is calculated between a depth value of a same-position image region of the decoded first depth image and an interpolated depth value obtained by interpolating depth values in the decoded first depth image; and - if the second rate-distortion cost is lower than the first rate-distortion cost, indicating that the depth value within the same-position image region of the second depth image is not encoded in the bitstream; otherwise, indicating that the depth value within the same-position image region of the second depth image is encoded in the bitstream.

[0021] According to one embodiment, determining whether a depth value within an image region of a second depth image is encoded in a bitstream includes: - calculating an interpolated depth value for the image region of the second depth image by interpolating depth values in the first depth image; - calculating a distance between a depth value within the image region of the second depth image and an interpolated depth value obtained by interpolating depth values in a decoded first depth image obtained by decoding the first encoded depth image. - If the distance is below a threshold value, the depth coding mode of the image region indicates that the depth value within the image region of the second depth image is not encoded in the bitstream; otherwise, the depth coding mode of the image region indicates that the depth value within the image region of the second depth image is encoded in the bitstream.

[0022] According to one embodiment, the above first and second distances or the above distance are calculated between at least a part of the reconstructed point cloud and the corresponding part of the point cloud, and the at least a part of the point cloud is reconstructed from the decoded first depth image and the second depth image.

[0023] According to one embodiment, the at least a part of the point cloud is reconstructed from the depth value within the image region and from the depth value within at least one previously considered image region.

[0024] According to one embodiment, when the depth coding mode of the image region indicates that the depth value within the image region of the second depth image is not encoded in the bitstream, the depth value of the pixel within the image region of the second depth image is replaced with a constant value before at least partially encoding the second depth image.

[0025] According to one embodiment, the depth coding mode is encoded as metadata related to the reconstruction of the point cloud whose geometric shape is represented by the first and second depth images.

[0026] This principle aims to improve at least one of the drawbacks of the prior art by a method of decoding the depth value of a point orthogonally projected onto the projection plane of the original point cloud. - By decoding the bitstream, obtaining a decoded first depth image; - Obtaining, from the bitstream, the depth coding mode related to the image region of the decoded second depth image; -If the depth coding mode indicates that the depth values ​​within the image region of the decoded second depth image are encoded in the bitstream, then at least partially decode the second depth image from the bitstream, -Otherwise, the method includes calculating an interpolated depth value in the image region of a decoded second depth image by interpolating the depth value in the first decoded depth image.

[0027] According to one embodiment, if at least one depth coding mode indicates that depth values ​​within an image region of a second depth image are encoded / decoded in a bitstream, then the entire second depth image is encoded / decoded in / from a bitstream.

[0028] According to one embodiment, the size and shape of the image region of the second depth image are the size and shape of the second depth image.

[0029] In other embodiments of these, the principle relates to devices, computer program products, non-temporary computer-readable media, and video signals.

[0030] The specific properties of this principle, as well as other purposes, advantages, features, and uses of this principle, will become apparent from the following example description in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0031] The diagram illustrates an example of this principle. The diagram is as follows:

[0032] [Figure 1] A schematic diagram illustrating the steps of a method for encoding the geometric shape of a point cloud represented by first and second depth images, based on an example of this principle, is shown. [Figure 2] A schematic diagram of step 120 of the method shown in Figure 1 according to this embodiment of the principle is shown. [Figure 3] A schematic diagram of step 120 of the method shown in Figure 1 according to this embodiment of the principle is shown. [Figure 4] A schematic diagram illustrates the steps of a method for decoding the geometric shape of a point cloud from first and second depth images representing different depth values ​​of orthographically projected points of the original point cloud, using an example of this principle. [Figure 5] This schematic diagram illustrates the method for encoding the geometric shape and texture of point clouds as defined in the prior art (TMC2). [Figure 6] Figure 5 schematically illustrates examples of the use of methods 100 and 200 in the encoding method. [Figure 7] This schematic diagram illustrates the method for decoding the geometric shape and texture of a point cloud as defined in the prior art (TMC2). [Figure 8] A schematic example of using method 200 in the decoding method shown in Figure 7 is illustrated. [Figure 9] An example of a device architecture based on this principle is shown. [Figure 10] This example illustrates two remote devices communicating via a communication network based on this principle. [Figure 11] The syntax of a signal is shown using an example of this principle.

[0033] Similar or identical elements are referenced with the same reference number.

[0034] An example explanation of this principle. The principle is described below in detail with reference to the accompanying drawings, which illustrate examples of the principle. However, the principle may be embodied in many alternative forms and should not be construed as being limited to the examples described herein. Thus, the principle is susceptible to various modifications and alternative forms, some of which are illustrated in the drawings and described in detail herein. However, it should be understood that there is no intention to limit the principle to any particular form disclosed, but rather this disclosure covers all modifications, equivalents, and alternatives that fall within the spirit and scope of the principle, as defined by the claims.

[0035] The terms used herein are for illustrative purposes only and are not intended to limit the principles. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural form unless otherwise clearly indicated in the context. Where used herein, the terms “equip,” “equip,” “contain,” and / or “contain” specify the presence of the described feature, integer, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, where an element is referred to as “responding to” or “connecting to” another element, it may directly respond to or be able to connect to the other element, or there may be an intervening element. In contrast, where an element is referred to as “directly responding to” or “directly connected to” another element, there is no intervening element. Where used herein, the term “and / or” includes any combination of one or more of the related enumerated items and may be abbreviated as “ / .”

[0036] In this specification, various elements may be described using terms such as "first," "second," etc., but it will be understood that these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without deviating from the teachings of this principle, the first element can be called the second element, and similarly, the second element can be called the first element.

[0037] Some diagrams include arrows on the communication path to indicate the primary direction of communication, but please understand that communication may occur in the opposite direction to the depicted arrows.

[0038] Some examples are illustrated in relation to block diagrams and operation flowcharts, where each block represents a circuit element, module, or portion of code containing one or more executable instructions for performing a specified logical function. Note that in other embodiments, the functions described in a block may occur in a different order than those described. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or sometimes in reverse order depending on the related functions.

[0039] In this specification, any reference to “by example” or “in one example” means that a particular feature, structure, or property described in relation to an example may be included in at least one embodiment of the principle. The occurrence of the phrase “by example” or “in one example” in various places in this specification does not necessarily refer to the same example, and separate or alternative examples are not necessarily mutually exclusive.

[0040] The reference figures appearing in the claims are illustrative and do not limit the scope of the claims.

[0041] Although not explicitly described, these embodiments and variations may be used in any combination or partial combination.

[0042] This principle describes the encoding / decoding of the geometric shape of a point cloud from two depth images, but the geometric shape of a sequence of point clouds is encoded / decoded by / from a sequence (video) of two depth images, and therefore extends to the encoding / decoding of a sequence of point clouds (a temporarily dynamic point cloud), and the two depth images associated with a point cloud can be encoded independently of two depth images of another point cloud in the sequence.

[0043] As explained above, the point cloud is orthogonal projected onto a projection plane, and two depth images D0 and D1 are obtained from the depth values ​​associated with the projected 3D points. D0 is the first depth image representing the depth value of the nearest point in the point cloud, and D1 is the second depth image representing the depth value of the farthest point in the point cloud. The first depth image D0 is encoded, for example, using a conventional image / video encoder.

[0044] In the following, the term "image region" refers to a set of pixels in an image. These pixels may or may not be adjacent, but all pixels share at least one common property.

[0045] For example, the image itself may be considered an image region. An image can also be divided into multiple blocks, in which case each block becomes an image region.

[0046] Image regions may also have a non-rectangular shape. This is the case, for example, when pixels of an image having the same (or similar) extracted features are associated with each other to form an image region.

[0047] Examples of features extracted from an image may include color, texture, and normal vectors.

[0048] Figure 1 schematically illustrates the steps of method 100 for encoding the geometric shape of a point cloud represented by first (D0) and second (D1) depth images, using an example of the present principle.

[0049] In step 110, the first depth image D0 is encoded in bitstream B. In step 120, the module sets the depth coding mode (DCM) for each image region. iを The depth coding mode is determined to indicate whether the depth values ​​of pixels within the image region of the second depth image D1 are also encoded in bitstream B. This depth coding mode will be referred to as the "explicit" mode below.

[0050] In step 130, the module uses the depth coding mode DCM in bitstream B. i Encode it.

[0051] Step 140: At least one depth coding mode DCM i However, if it indicates that the depth values ​​of pixels within the image region of the second depth image D1 are encoded in bitstream B ("explicit" mode), the module encodes the second depth image D1 in bitstream B, at least partially.

[0052] Steps 130 and 140 are repeated until each of the I image regions has been considered.

[0053] According to this principle, an additional depth coding mode is encoded in the bitstream to indicate whether an image region of the second depth image D1 is explicitly (or implicitly) encoded in the bitstream. If the depth coding mode associated with an image region of the second depth image D1 indicates that the depth values ​​of the pixels in that image region are not encoded in the bitstream ("implicit" mode), the bitrate is reduced compared to when coded data representing the depth values ​​is effectively transmitted, as disclosed in the prior art. Therefore, transmitting such a depth coding mode for each image region improves the coding efficiency of depth images representing the geometric shape of the point cloud.

[0054] According to one embodiment, the size and shape of the image region of the second depth image are the size and shape of the second depth image, that is, the image region is the image itself.

[0055] Next, a single depth coding mode is transmitted, indicating whether (or not) the entire second depth image is encoded in the bitstream.

[0056] According to step 140, at least one depth coding mode DCM iIf it is set to "Explicit" mode, the entire second depth image D1 is encoded in bitstream B.

[0057] According to another embodiment, a depth coding mode is assigned to each image region of the second depth image.

[0058] The image region may have a rectangular shape, such as a block of image, or a non-rectangular shape such as a projection depth patch within the TMC2.

[0059] These embodiments improve coding efficiency by locally adapting the depth coding mode to the characteristics of the image content.

[0060] As shown in Figure 2, according to the embodiment of step 120, determining whether the depth values ​​of pixels within the image region of the second depth image D1 are encoded in the bitstream includes the following steps:

[0061] The module is the first encoded depth image

number

number

[0062] The decoded first depth image

number

number

number

number

number

[0063] Next, the first rate distortion compensation Cost0 is calculated by taking into account the first distance Dist0 and the first bitrate RA0.

[0064] The module then processes the decoded first depth image

number

number

number

[0065] The second quality metric, Dist1, is the depth value of the pixels within the current image area.

number

number

[0066] Next, by considering the second distance Dist1, a second rate distortion compensation Cost1 is calculated, and the data rate is considered null here because the second depth image is not encoded (transmitted).

[0067] If the second rate distortion compensation Cost1 is lower than the first rate distortion compensation Cost0, the depth coding mode DCMi for the current image region i is implicitly set to DCMi, meaning that the depth values ​​in the current image region of the second depth image D1 are not encoded in the bitstream. Otherwise, the depth coding mode DCM for the current image region i is set to DCMi. i This is set to "explicit," meaning that the depth values ​​within the current image region of the second depth image D1 are encoded in the bitstream.

[0068] The steps of this embodiment are repeated until each of the I image regions has been considered.

[0069] This embodiment of step 120 provides the best rate distortion trade-off for determining whether (or not) the depth values ​​of pixels in the image region of the second depth image are encoded in the bitstream.

[0070] As shown in Figure 3, according to an alternative embodiment of step 120, determining whether the depth values ​​of pixels within the image region of the second depth image D1 are encoded in the bitstream includes the following steps:

[0071] The module calculates the interpolated depth values ​​of pixels in the same image region of the second depth image D1 by interpolating the depth values ​​of pixels in the first depth image D0. The set of interpolated depth values ​​is:

number

[0072] Next, distance DIST is the depth value within the current image region i of the second depth image D1 shown.

number

number

[0073] If the distance DIST falls below the threshold TH, the depth coding mode of the current image region i is DCM. i This is set to "implicit," meaning that the depth value in the current image region of the second depth image D1 is not encoded in the bitstream. Otherwise, the depth coding mode of the current image region i is DCM. i This is set to "explicit," meaning that the depth value in the current image region of the second depth image D1 is encoded in the bitstream.

[0074] The steps of this embodiment are repeated until each of the I image regions has been considered.

[0075] This alternative embodiment of step 120 provides a second-best rate distortion trade-off because the metric is calculated without an encoding / decoding process, but reduces the complexity of the selection process compared to the complexity of the best embodiment above in Figure 2.

[0076] According to one embodiment, the distance DIST between two sets of ordered depth values ​​A and B is defined as follows:

number

number

number

[0077] The ordering of the set of numbers is based on the depth value.

number

number

[0078] Distance DIST is not limited to this embodiment and can be extended to any other well-known metric for calculating the distance between two sets of J values, such as the sum of absolute differences or the mean / maximum / minimum values ​​of the differences.

[0079] According to one embodiment, the distance DIST is calculated between at least a portion of the reconstructed point cloud and the corresponding portion of the original point cloud.

[0080] For example, the distance DIST is defined in ISO / IEC JTC1 / SC29 / WG1 MPEG2017 / N16763, Hobart, April 2017, Appendix B.

[0081] At least a portion of the point cloud is reconstructed from the decoded first and second depth images.

[0082] According to one embodiment, at least a portion of the point cloud is reconstructed from the depth values ​​of pixels within the image region.

[0083] According to one embodiment, at least a portion of the point cloud is reconstructed from the depth values ​​of pixels in the current image region and from the depth values ​​of pixels in at least one previously considered image region.

[0084] For example, according to this embodiment, a “temporary” second depth image is initialized with a constant value. The depth values ​​of the pixels in the temporary second depth image are then iteratively replaced by the depth values ​​of the encoded / decoded second depth image, or by padding with the depth values ​​of the nearest neighbor previously encoded according to the “explicit” mode, when the current image region is explicitly encoded (“explicit” mode).

[0085] Therefore, the reconstructed point cloud, which relies on encoding the depth values ​​of pixels within the previously considered image region, will be similar to the reconstructed point cloud.

[0086] It should be noted that in this embodiment, the “temporary” depth image is not encoded in the bitstream. The second depth image is still encoded according to the method in Figure 1.

[0087] According to the embodiment of step 140, the depth coding mode DCM associated with the image region i If set to "implicit", the depth values ​​of pixels within the image region of the second depth image are replaced with a constant value at least partially before encoding the second depth image D1.

[0088] According to one embodiment, the depth coding mode DCMi encodes the geometric shape as metadata related to the reconstruction of the point cloud represented by the first and second depth images.

[0089] The metadata may be associated, for example, with each image or each image region common to two images, and is used to reconstruct the geometric shape of the point cloud on both the encoding and decoding sides, as will be further explained in relation to Figures 5 and 6.

[0090] According to one embodiment, the depth coding mode DCMi is encoded, for example, as a syntactic element of an SEI message attached to a NAL unit associated with a first depth image D0.

[0091] Example of DCM in SEI messages in HEVC [Table 1] The `dcm_mode` parameter contains an identification number used to identify the depth coding mode. For example, `dcm_mode` 0 means "explicit" mode, and 1 means "implicit" mode.

[0092] In variations, depth coding modes can also be present in SPS or PPS messages.

[0093] According to another embodiment, the depth coding mode DCMi is encoded as a watermark embedded in the depth image.

[0094] As a variation, the depth coding mode DCMi is embedded as a visible watermark in the empty area of ​​the first depth image D0.

[0095] For example, a block of N×N pixels in a given corner of the first depth image D0, where all pixels in such a block are set to the same binary value, such as 0(1), indicates that the depth coding mode DCMi is set to "explicit" ("implicit").

[0096] The decoder then calculates the average value of the block. If this average is closer to 0 than the maximum value (where all pixel values ​​are equal to 1), the decoded block indicates that "explicit" mode was used; otherwise, it indicates that "implicit" mode was used.

[0097] According to another embodiment, the depth coding mode DCMi is added to the binary information of metadata related to the geometric shape of the point cloud represented by the first and second depth images, such as the occupancy map defined in TMC2.

[0098] This embodiment is better suited for specifying the DCMi depth coding mode at a finer resolution than per image.

[0099] We will now look in more detail at how this is implemented in TMC2. The top-level syntax of the current version of TMC2 is shown in Tables 1 and 2. Table 3 provides the syntax for encapsulating geometric shape (depth) and texture (color) streams. Tables 4 and 5 show the detailed syntax for occupancy map and block-to-patch index decoding. Tables 6 and 7 also show the syntax for arithmetic coding of base values. [Table 2] [Table 3] [Table 4] [Table 5] [Table 6]

[0100] The current syntax encodes block-by-block metadata in two steps: first, it encodes the block-to-patch index for all blocks in the patch image, and then it encodes the occupancy map of these blocks belonging to the patch.

[0101] The block-to-patch index defines the index of the patch associated with each block of the texture and depth image, where the blocks form a regular square grid. The size of the blocks is given by the "occupied resolution" parameter in the header of the frame group, and is typically set to 16 pixels.

[0102] An occupancy map indicating which pixels of the texture and depth image represent the point cloud to be reconstructed is also encoded block by block. In this case, the blocks form a grid within each "occupancy resolution" block, and the grid is of the size of "occupancy accuracy", usually set to 4 pixels.

[0103] Example of DCM mode encoded as metadata Example of DCM in the occupancy map (per image (frame)) - changes to Table 5 [Table 7]

[0104] According to one embodiment, a depth coding mode DCM related to an image region i is a binary value of a binary value sequence, and each binary value indicates the depth coding mode DCM of the image region i For example, "0" indicates the "implicit" mode and "1" indicates the "explicit" mode.

[0105] According to one embodiment, an entropy or run-length coding method can be used to encode the binary sequence.

[0106] FIG. 4 schematically shows a diagram of the steps of a method 200 for decoding the geometric shape of a point cloud from first (D0) and second (D1) depth images representing different depth values of the orthographically projected points of the original point cloud according to an example of the present principle.

[0107] In step 210, a decoded first depth image is obtained by decoding the bitstream B.

[0108] In step 220, the depth coding mode DCM related to the current image region i of the decoded second depth image i is decoded from the bitstream B.

[0109] In step 230, the depth coding mode DCMi However, if it indicates that the depth values ​​of pixels in the current image region of the decoded second depth image D1 are encoded in bitstream B ("explicit" mode), the module decodes the second depth image D1 from bitstream B, at least partially.

[0110] Otherwise, in step 240, the module decodes the first depth image

number

number

[0111] Steps 220-240 are repeated until each of the I image regions has been considered.

[0112] The geometric shape of the point cloud is then decoded as, for example, in TMC2, the first (

number

number

[0113] According to one embodiment of the method, the interpolated depth values ​​of pixels in an image region of a second depth image are calculated by interpolating the depth values ​​of pixels in a first depth image. - For each current pixel in the image region of the second depth image, determine the corresponding pixel in the first depth image. - Determining at least one adjacent pixel to the pixel at the same position in the first depth image, -Includes calculating an interpolated depth value for each current pixel, taking into account at least one adjacent pixel in the first depth image.

[0114] According to one embodiment, the spatial distance between a pixel at the same position in a first depth image and at least one adjacent pixel is below a given threshold.

[0115] According to one embodiment, the interpolated depth value of the current pixel in the image region of the second depth image is the depth value of the nearest adjacent pixel among the at least one adjacent pixels in the first depth image. According to one embodiment, the interpolated depth value of the current pixel in the image region of the second depth image is the maximum depth value of the at least one adjacent pixel in the first depth image.

[0116] According to one embodiment, the interpolated depth value of the current pixel in the image region of the second depth image is the minimum depth value of at least one adjacent pixel in the first depth image.

[0117] According to one embodiment, the interpolated depth value of the current pixel in the image region of the second depth image is the average of the depth values ​​of at least one adjacent pixel in the first depth image.

[0118] Figure 5 schematically illustrates how the geometric shape and texture of the point cloud defined in TMC2 are encoded.

[0119] Essentially, the encoder captures the geometric shape information of the original point cloud PC in the first (D0) and second (D1) depth images.

[0120] As an example, the first and second depth images are acquired in TMC2 as follows:

[0121] Depth patches (a collection of 3D points in the point cloud PC) are obtained by clustering the points in the point cloud PC according to the normal vectors at these points. All extracted depth patches are then projected onto a 2D grid and packed while minimizing unused space, ensuring that every TxT (e.g., 16x16) block of the grid is associated with a unique patch, where T is a user-defined parameter signaled to the bitstream.

[0122] Next, a depth image is generated using the 3D-to-2D mapping calculated during the packing process, more specifically, the packing position and size of the projection area of ​​each patch. More precisely, let H(u,v) be the set of points of the current patch projected onto the same pixel (u,v). The first layer, also called the nearest layer or first depth image D0, stores the points of H(u,v) with the smallest depth value. The second layer, called the furthest layer or second depth image D1, captures the points of H(u,v) with the highest depth value within the interval [D, D+Δ], where D is the depth value of the pixel in the first depth image D0 and Δ is a user-defined parameter representing the surface thickness.

[0123] Next, the first depth image D0 outputs the packing process. A padding process is also used to fill in the empty spaces between patches in order to generate a piecewise smooth first depth image suitable for video compression.

[0124] The generated depth images / layers D0 and D1 are then stored as video frames and compressed using any conventional video codec such as HEVC.

[0125] The encoder encodes / decodes the first and second depth images, and the decoded first and second depth images

number

[0126] According to one embodiment, for each reconstructed point, the color of the nearest point in the original point cloud is assigned as the color to be encoded.

[0127] Then, by storing the encoded color information of each reconstructed point at the same location as the depth image, i.e., (i,u,v), the first and second texture images T0 and T1 are generated.

[0128] Figure 6 schematically illustrates examples of the use of methods 100 and 200 in the encoding method of Figure 5.

[0129] In this example, the first depth image

number

number

number

[0130] Figure 7 schematically illustrates a method for decoding the geometric shape and texture of a point cloud, as defined in the prior art (TMC2).

[0131] Decoded first depth image

number

number

number

[0132] Therefore, the geometric shape of the point cloud is reconstructed by back-projecting the decoded first and second depth images, and optionally the metadata.

[0133] Figure 8 schematically shows an example of using method 200 in the decoding method of Figure 7.

[0134] In this example, the decoding of the first and second depth images in Figure 7 is replaced by the decoding method in Figure 4.

[0135] In Figures 1 to 8, modules are functional units that may or may not be associated with distinct physical units. For example, these modules, or parts thereof, may be combined into a unique component or circuit, or contribute to software functionality. Conversely, some modules may consist of potentially separate physical entities. Devices compatible with this principle may be implemented using pure hardware, for example, using dedicated hardware such as ASICs, FPGAs, or VLSIs, which are application-specific integrated circuits, field-programmable gate arrays, and very large-scale integrations, respectively, or from several integrated electronic components embedded in the device, or from a mixture of hardware and software components.

[0136] Figure 9 shows an exemplary architecture of device 90 that may be configured to carry out the methods described in relation to Figures 1 to 8.

[0137] Device 90 consists of the following elements, which are linked to each other by the data and address bus 91: - For example, a microprocessor 92 (or CPU) which is a DSP (or digital signal processor), -ROM (or read-only memory) 93, -RAM (or random access memory) 94, - I / O interface 95 that receives data sent from the application, and - Includes battery 96.

[0138] For example, the battery 96 is located outside the device. In each of the memories described herein, the word "register" as used herein can correspond to a small area (a few bits) or a very large area (e.g., an entire program or a large amount of received or decoded data). ROM 93 contains at least the program and parameters. ROM 93 may store algorithms and instructions for performing the technique according to this principle. When power is applied, the CPU 92 uploads the program to RAM and executes the corresponding instructions.

[0139] RAM94 contains, within its registers, a program executed by CPU92 and uploaded after power-up of device90, input data within the registers, intermediate data for different states of methods within the registers, and other variables within the registers used for executing methods.

[0140] The embodiments described herein may be implemented, for example, in the form of methods or processes, apparatus, software programs, data streams, or signals. Even if considered only in the context of a single embodiment (for example, considered only as a method or device), the implementation of the considered function may also be implemented in other forms (for example, programs). Apparatus may be implemented, for example, in appropriate hardware, software, and firmware. These methods may be implemented, for example, in apparatus, and can be implemented in a processor, for example, in a broader sense, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. A processor also includes communication devices, for example, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the communication of information between end users.

[0141] According to the example of encoding or encoding, the p-point cloud PC is obtained from a source. For example, the source is: - Local memory (93 or 94), e.g., video memory or RAM (or random access memory), flash memory, ROM (or read-only memory), hard disk, - Storage device interface (95), for example, interface with mass storage device, RAM, flash memory, ROM, optical disk, or magnetic support, -Communication interfaces (95), for example, wired interfaces (e.g., bus interfaces, wide area network interfaces, local area network interfaces) or wireless interfaces (such as IEEE 802.11 interfaces or Bluetooth® interfaces), and - Belongs to a set that includes image capture circuits (for example, sensors such as CCD (or charge-coupled devices) or CMOS (or complementary metal-oxide-semiconductor devices)).

[0142] According to the example of decoding or decoding, the decoded first and / or second depth images, or reconstructed point cloud, are sent to the destination, specifically, the destination is - Local memory (93 or 94), e.g., video memory or RAM, flash memory, hard disk, - Storage device interface (95), for example, interface with mass storage device, RAM, flash memory, ROM, optical disk, or magnetic support, -Communication interfaces (95), for example, wired interfaces (for example, bus interfaces (for example, USB (or Universal Serial Bus)), wide area network interfaces, local area network interfaces, HDMI (High Definition Multimedia Interface) interfaces) or wireless interfaces (such as IEEE 802.11 interfaces, WiFi®, or Bluetooth® interfaces), and - Belongs to a set that includes displays.

[0143] According to the example of encoding or encoding, bitstream B is sent to a destination. For example, bitstream B is stored in local or remote memory, e.g., video memory (94) or RAM (94), or hard disk (93). In a variation, one or both bitstreams are sent to a storage interface (95), e.g., mass storage, flash memory, ROM, optical disk, or magnetic support, and / or transmitted via a communication interface (95), e.g., a point-to-point link, communication bus, point-to-multipoint link, or broadcast network.

[0144] In the example of decoding or decoding, bitstream B is obtained from a source. Exemplary, the bitstream is read from local memory, e.g., video memory (94), RAM (94), ROM (93), flash memory (93), or hard disk (93). In a modified example, the bitstream is received from a storage device interface (95), e.g., an interface with mass storage, RAM, ROM, flash memory, optical disk, or magnetic support, and / or from a communication interface (95), e.g., a point-to-point link, bus, point-to-multipoint link, or broadcast network.

[0145] For example, a device 90 configured to perform the encoding method described in relation to Figures 1 to 3, or Figures 5 and 6, - Mobile devices, -Communication devices, - Game devices, - Tablet (or tablet computer), -Laptop, -Still camera, -Video camera, - Encoding chip, - Still image server, and - Belongs to a set that includes video servers (e.g., broadcast servers, video-on-demand servers, or web servers).

[0146] For example, a device 90 configured to perform the decoding method described in relation to Figure 4, or Figures 7 and 8, - Mobile devices, -Communication devices, - Game devices, - Set-top box, - Television receiver - Tablet (or tablet computer), -Laptop, - Display, and - Belongs to a set that includes a decryption chip.

[0147] According to an example of this principle shown in Figure 10, in a transmission context between two remote devices A and B over a communication network NET, device A includes a processor associated with memory RAM and ROM configured to implement a method for encoding the geometric shape of a point cloud as described in relation to Figures 1 to 3, or Figures 5 and 6, and device B includes a processor associated with memory RAM and ROM configured to implement a method for decoding a point cloud as described in relation to Figure 4, or Figures 7 and 8.

[0148] For example, the network is a broadcast network adapted to transmit still or video images from device A to decoding devices including device B.

[0149] The signal intended to be transmitted by device A carries bitstream B. Bitstream B includes an encoded first depth image and, optionally, at least a portion of an encoded second depth image, as described in relation to Figure 1. This signal is transmitted via at least one depth coding mode DCM. iを It further includes the information data to be represented. Each depth coding mode indicates whether the depth values ​​of the pixels in image region i of the second depth image are encoded in bitstream B ("explicit" mode) or not ("implicit" mode).

[0150] Figure 11 shows an example of the syntax of such a signal when data is transmitted via a packet-based transmission protocol. Each transmitted packet P includes a header H and a payload PAYLOAD. Bits in the header H, for example, depth coding mode DCM i This is a dedicated ID to represent. Therefore, at least one bit of header H represents at least one depth coding mode DCM i It is used to represent [something].

[0151] The implementation of the various processes and features described herein can be embodied in a variety of different devices or applications. Examples of such devices include encoders, decoders, post-processors that process the output from decoders, pre-processors that provide input to encoders, video coders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, mobile phones, PDAs, and any other devices that process images or video, or other communication devices. As is evident, the devices can be portable and can even be mounted on mobile vehicles.

[0152] Furthermore, the method may be carried out by instructions executed by a processor, and such instructions (and / or data values ​​generated by their execution) may be stored in a computer-readable storage medium. The computer-readable storage medium may be embodied in one or more computer-readable media and may take the form of a computer-readable program product in which computer-readable program code executable by a computer is embodied. As used herein, the computer-readable storage medium is considered a non-temporary storage medium given the inherent ability to store information therein and the inherent ability to provide information retrieval therefrom. The computer-readable storage medium may be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of any of the foregoing. The following provides more specific examples of computer-readable storage media to which this principle can be applied, but it should be understood that, as will be easily understood by those skilled in the art, portable computer diskettes, hard disks, read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing are merely illustrative and not an exhaustive list.

[0153] Instructions can form application programs that are tangibly implemented in a processor-readable medium.

[0154] Instructions may reside, for example, in hardware, firmware, software, or a combination thereof. Instructions can also be found, for example, in an operating system, a separate application, or a combination of these two. Therefore, a processor can be characterized as both, for example, a device configured to execute a process, and a device containing a processor-readable medium (such as a storage device) that has instructions for executing the process. Furthermore, the processor-readable medium may store data values ​​generated by the embodiment, in addition to or instead of instructions.

[0155] As will be apparent to those skilled in the art, embodiments can generate a wide variety of signals formatted to transmit information, which can, for example, be stored or transmitted. The information may include, for example, instructions for performing a method, or data generated by one of the embodiments described. For example, a signal may be formatted to transmit, as data, rules for writing or reading the syntax of the described example of the Principle, or to transmit, as data, actual syntax values ​​described by the described example of the Principle. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information transmitted by the signal may be, for example, analog or digital information. The signal may be transmitted over a wide variety of different wired or wireless links, as is known. The signal may be stored in a processor-readable medium.

[0156] Numerous embodiments have been described. Nevertheless, it should be understood that various modifications can be made. For example, elements of different embodiments can be combined, supplemented, modified, or deleted to produce other embodiments. Furthermore, those skilled in the art will understand that other structures and processes can be used in place of the disclosed structures and processes, and that the resulting embodiments will perform at least substantially the same functions, in at least substantially the same manner, and achieve at least substantially the same results, as the disclosed embodiments. Accordingly, these embodiments and other embodiments are conceived in this application.

Claims

1. A method for reconstructing the geometric shape of a point cloud, Decoding a first depth image of one or more depth images from a bitstream, wherein the one or more depth images represent different depth values ​​of points orthographically projected onto the projection plane of the point cloud. Decoding a depth coding mode associated with a second depth image of one or more depth images from an SPS or PPS message in the bitstream, wherein the depth coding mode indicates whether or not the second depth image is encoded in the bitstream. Obtaining a first depth value for at least one first pixel of the first depth image, In response to the depth coding mode, a second depth value is determined for at least one second pixel of the second depth image, wherein the at least one second pixel is in the same position as the at least one first pixel, and the second depth value is determined from the first depth value of the at least one first pixel and the first depth value of at least one adjacent pixel of the at least one first pixel in the first depth image. Includes, A method wherein the at least one adjacent pixel is acquired in response to a determination that the spatial distance between the at least one first pixel and the at least one adjacent pixel in the first depth image is below a given threshold.

2. The method according to claim 1, wherein the second depth value is determined as the average of the first depth values ​​of at least one adjacent pixel.

3. The method according to claim 1, wherein the second depth value is determined as the maximum first depth value of at least one adjacent pixel.

4. The method according to claim 1, wherein the second depth value is determined as the minimum first depth value of at least one adjacent pixel.

5. The method according to claim 1, further comprising back-projecting the first depth image and the second depth image.

6. The method according to claim 1, wherein the depth coding mode is encoded as metadata associated with the reconstruction of the point cloud, and the geometric shape of the point cloud is represented by one or more depth images.

7. The method according to claim 1, wherein, if the depth coding mode indicates that the second depth image is encoded in the bitstream, the method further comprises decoding the second depth image from the bitstream.

8. An apparatus for reconstructing the geometric shape of a point cloud, Decoding a first depth image of one or more depth images from a bitstream, wherein the one or more depth images represent different depth values ​​of points orthographically projected onto the projection plane of the point cloud. Decoding a depth coding mode associated with a second depth image of one or more depth images from an SPS or PPS message in the bitstream, wherein the depth coding mode indicates whether or not the second depth image is encoded in the bitstream. Obtaining a first depth value for at least one first pixel of the first depth image, In response to the depth coding mode, a second depth value is determined for at least one second pixel of the second depth image, wherein the at least one second pixel is in the same position as the at least one first pixel, and the second depth value is determined from the first depth value of the at least one first pixel and the first depth value of at least one adjacent pixel of the at least one first pixel in the first depth image. Includes at least one processor configured to perform the following: The apparatus is characterized in that the at least one adjacent pixel is acquired in response to a determination that the spatial distance between the at least one first pixel and the at least one adjacent pixel in the first depth image is below a given threshold.

9. The apparatus according to claim 8, wherein the second depth value is determined as the average of the first depth values ​​of at least one adjacent pixel.

10. The apparatus according to claim 8, wherein the second depth value is determined as the maximum first depth value of at least one adjacent pixel.

11. The apparatus according to claim 8, wherein the second depth value is determined as the minimum first depth value of at least one adjacent pixel.

12. The apparatus according to claim 8, wherein at least one processor is further configured to back-project the first depth image and the second depth image.

13. The apparatus according to claim 8, wherein the depth coding mode is encoded as metadata associated with the reconstruction of the point cloud, and the geometric shape of the point cloud is represented by one or more depth images.

14. A non-temporary computer-readable medium comprising one or more processors, A method for reconstructing the geometric shape of a point cloud, Decoding a first depth image of one or more depth images from a bitstream, wherein the one or more depth images represent different depth values ​​of points orthographically projected onto the projection plane of the point cloud. Decoding a depth coding mode associated with a second depth image of one or more depth images from an SPS or PPS message in the bitstream, wherein the depth coding mode indicates whether or not the second depth image is encoded in the bitstream. Obtaining a first depth value for at least one first pixel of the first depth image, In response to the depth coding mode, a second depth value is determined for at least one second pixel of the second depth image, wherein the at least one second pixel is in the same position as the at least one first pixel, and the second depth value is determined from the first depth value of the at least one first pixel and the first depth value of at least one adjacent pixel of the at least one first pixel in the first depth image. Includes instructions for performing a method that includes, The at least one adjacent pixel is acquired in response to a determination that the spatial distance between the at least one first pixel and the at least one adjacent pixel in the first depth image is below a given threshold, in a non-temporary computer-readable medium.

Images