A method and device for encoding / reconstructing 3D points

Abstract

The present embodiment relates to a method for encoding a 3D point, the geometry of which is represented by a geometric image and the attributes of which are represented by an attribute image. The method checks whether the depth value of a pixel in a first geometric image in the geometric image and the depth value of a co-located pixel in a second geometric image in the geometric image are different (different values). When the depth value of a pixel in the first geometric image and the depth value of a co-located pixel in the second geometric image are different, the method assigns (encodes) the attributes of the 3D point defined according to the 2D spatial coordinates of the pixel in the first geometric image and the depth value of the co-located pixel in the second geometric image.

Description

Technical Field

[0001] The present embodiment mainly relates to the encoding and reconstruction of 3D points. In particular, but not exclusively, the technical field of the present embodiment relates to the encoding / reconstruction of a point cloud representing the outer surface of a 3D object. Background Art

[0002] This section is intended to introduce the reader to various aspects of the technology that may be related to various aspects of the embodiments of the present invention described and / or claimed below. It is believed that this discussion is helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present embodiments. Therefore, it should be understood that these statements should be read in this context and not as admissions of prior art.

[0003] A point cloud is a set of data points in a certain coordinate system. In a three-dimensional coordinate system (3D space), these points are usually intended to represent the outer surface of a 3D object. Each point of a point cloud is usually defined by its position (X, Y and Z coordinates in 3D space) and possibly by other related properties, such as color (in, for example, RGB or YUV color space), transparency, reflectivity, a two-component normal vector, etc.

[0004] A point cloud is usually represented as a set of 6-component points (X,Y,Z,R,G,B) or equivalently (X,Y,Z,Y,U,V), where (X,Y,Z) define the coordinates of a color point in 3D space and (R,G,B) or (Y,U,V) define the color of that color point.

[0005] A point cloud can be static or dynamic, depending on whether the cloud evolves with respect to time. It should be noted that in the case of a dynamic point cloud, the number of points is not constant, but on the contrary, usually evolves with time. Therefore, a dynamic point cloud is a time-ordered list of a set of points.

[0006] In fact, point clouds can be used for various purposes, such as cultural heritage / architecture, where objects like statues or buildings are scanned in 3D in order to share the spatial configuration of the object without sending or accessing it. Moreover, this is a way to ensure that knowledge about the object is preserved in the event that the object may be destroyed (such as a temple destroyed by an earthquake). Such point clouds are usually static, colored and huge.

[0007] Another use case is in topographic maps and cartography using 3D representations, where the maps are not limited to flat surfaces and can contain undulations. Google Maps is now a good example of a 3D map, but uses a grid instead of a point cloud. However, a point cloud can be a suitable data format for a 3D map, and such a point cloud is usually static, colored, and large.

[0008] The automotive industry and autonomous vehicles are also areas where point clouds can be used. Autonomous vehicles should be able to "probe" their environment to make good driving decisions based on the reality in their immediate vicinity. Typical sensors like LIDAR produce dynamic point clouds that are used by decision engines. These point clouds are not intended to be viewed by humans, and they are usually small, not necessarily colored, and dynamic, with a high capture frequency. They can have other properties such as reflectivity provided by LIDAR, as this property is good information about the material of the sensed object and can help make said decisions.

[0009] Virtual reality and immersive worlds have become a hot topic lately and are foreseen by many as the future of 2D flat video. The basic idea is to immerse the viewer in the environment around him, as opposed to standard TV where the viewer can only see the virtual world in front of him. There are several levels in the immersion, depending on the degree of freedom the viewer has in the environment. Colored point clouds are a good format candidate for distributing virtual reality (or VR) worlds. They can be static or dynamic and are usually of average size, such as no more than millions of points at a time.

[0010] Point cloud compression will successfully store / transmit 3D objects for immersive worlds only if the size of the bitstream is low enough to allow practical storage / transmission to the end user.

[0011] It is crucial to be able to distribute dynamic point clouds to end users with reasonable consumption of bit rate while maintaining acceptable (or preferably very good) quality of experience. In order to make the distribution chain of immersive worlds feasible, efficient compression of these dynamic point clouds is a key point.

[0012] Image-based point cloud compression techniques are becoming increasingly popular due to their combination of compression efficiency and low complexity. They proceed in two main steps: First, they project (orthogonally project) a point cloud (i.e., 3D points) onto at least one 2D image plane. For example, at least one 2D geometry (also denoted as depth) image is thus obtained to represent the geometry of the point cloud, i.e., the spatial coordinates of the 3D points in 3D space, and at least one 2D attribute (also denoted as texture) image is also obtained to represent the attributes associated with the 3D points of the point cloud, such as texture / color information associated with those 3D points. Next, these techniques encode such geometry and attribute images into at least one geometry and attribute layer using a conventional video encoder.

[0013] Image-based point cloud compression techniques achieve good compression performance by leveraging the capabilities of 2D video encoders, such as HEVC ("ITU-T H.265 Telecommunication standardization sector of ITU (10 / 2014), series H: audiovisual and multimedia systems, infrastructure of audiovisual services-coding of moving video, High efficiency video coding, Recommendation ITU-T H.265"), while they keep the complexity low by using a simple projection scheme.

[0014] One of the challenges of image-based point cloud compression techniques is that the point cloud may not be suitable for projection onto an image, especially when the point distribution follows a surface with many wrinkles (concave / convex areas, such as in clothing) or when the point distribution does not follow the surface at all (such as in fur or hair). In these cases, image-based point cloud compression techniques suffer from low compression efficiency (requiring many small projections, thereby reducing the efficiency of 2D video compression) or poor quality (due to the difficulty of projecting the point cloud onto the surface).

[0015] One of the methods used in the prior art to alleviate this problem is to project multiple geometric shapes and attribute information onto the same spatial position of the image. This means that several geometric and / or attribute images with the same projection coordinates (the same 2D spatial coordinates of the pixel) can be generated for each 3D point of the point cloud.

[0016] This is the case, for example, in the so-called Test Model Type 2 Point Cloud Coder (TMC2) defined in ISO / IEC JTC1 / SC29 / WG11 MPEG2018 / N17767, Ljubljana, July 2018 (Appendix A), in which the point cloud is orthogonally projected onto a projection plane. For each coordinate of the projection plane, two geometric images are generated: one representing the depth value associated with the nearest point (the minimum depth value) and the other representing the depth value of the farthest point (the maximum depth value). Then, a first geometric image is generated from the minimum depth value (D0) and a second geometric image is generated from the absolute value of the maximum depth value (D1), where D1-D0 is less than or equal to the maximum surface thickness. A first and a second attribute image associated with the first (D0) and second (D1) geometric images are also generated. The attribute image and the geometric image are then encoded and decoded using any conventional video codec such as HEVC. Thus, the geometry of the point cloud is reconstructed by deprojecting the information comprised in the decoded first and second geometry images, and attributes are associated with the 3D reconstructed from the information comprised in the decoded attribute images.

[0017] The disadvantage of capturing two geometry (and two attribute) values ​​is that two 3D points are systematically reconstructed from two geometry images, creating such duplicate reconstructed 3D points when the depth value of a pixel in the first geometry image is equal to the depth value of a co-located pixel in the second geometry image. Next, encoding unnecessary duplicate points increases the bit rate for sending the encoded set of 3D points. In addition, computational and storage resources for handling such pseudo-duplicate 3D points are also wasted on the encoding and decoding side. Summary of the invention

[0018] A simplified overview of an embodiment of the present invention is provided below to provide a basic understanding of some aspects of an embodiment of the present invention. This overview is not an extensive review of the present embodiment. It is not intended to identify the key or important elements of the present embodiment. The following overview only presents some aspects of the present embodiment in a simplified form as a preface to the more detailed description provided below.

[0019] Embodiments of the present invention are intended to remedy at least one shortcoming of the prior art by using a method and apparatus for encoding a 3D point whose geometry is represented by a geometric image and whose attributes are represented by an attribute image. When the depth value of a pixel in a first geometric image of the geometric images and the depth value of a co-located pixel in a second geometric image of the geometric images are not the same, the method assigns an attribute of a 3D point to a co-located pixel of the attribute image, the geometry of the 3D point being defined according to the 2D spatial coordinates of the co-located pixel in the first geometric image and the depth value of the co-located pixel in the second geometric image.

[0020] According to an embodiment, the method further comprises: assigning a dummy attribute to a pixel of the attribute image, which occurs when a depth value of a co-located pixel in the first geometric image and a depth value of a co-located pixel in the second geometric image are the same.

[0021] According to one embodiment, the pseudo attribute is an attribute of the co-located pixel of another attribute image.

[0022] According to an embodiment, the pseudo attribute is an average value of attributes associated with neighboring pixels located around the pixel.

[0023] According to an embodiment, the method further comprises: sending information data indicating whether depth values ​​of pixels in the first geometric image and depth values ​​of co-located pixels in the second geometric image are compared before reconstructing 3D points from the geometric images.

[0024] According to another aspect of the present invention, the present embodiment relates to a bitstream carrying encoded attributes of 3D points, wherein the bitstream is structured into multiple blocks, patches of blocks and frames of patches, wherein the information data is valid at a frame group level, a frame level, a patch level or a block level.

[0025] According to another aspect of the present invention, the present embodiment relates to a method for reconstructing a 3D point from a geometric image representing a geometric shape of the 3D point, wherein the method comprises: reconstructing the 3D point based on the 2D spatial coordinates of a pixel in a first geometric image of the geometric image and the depth value of a co-located pixel in a second geometric image of the geometric image, which occurs when the depth value of the pixel in the first geometric image and the depth value of the co-located pixel in the second depth image are not the same.

[0026] According to an embodiment, the method further comprises: receiving information data indicating whether depth values ​​of pixels in the first geometric image and depth values ​​of co-located pixels in the second geometric image are compared before reconstructing 3D points from the geometric images.

[0027] According to one embodiment, the bitstream carrying the coded attributes of 3D points is structured into multiple blocks, patches of blocks and frames of patches, and the information data is valid at the frame group level, at the frame level, at the patch level or at the block level.

[0028] According to an embodiment, the attribute of the 3D point is a color value or a texture value.

[0029] One or more of at least one embodiment also provides an apparatus, a computer program product, a non-transitory computer-readable medium, and a bitstream.

[0030] The specific nature of embodiments of the present invention as well as other objects, advantages, features and uses of embodiments of the present invention will become apparent from the following example description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In the accompanying drawings, an example of this embodiment is shown. It shows:

[0032] – Figure 1 A diagram schematically shows steps of a method 100 for encoding attributes associated with a 3D point according to an example of the present embodiment;

[0033] – Figure 2 A diagram schematically shows steps of a method 200 for reconstructing a 3D point from a geometric image representing the geometric shape of the 3D point according to an example of the present embodiment.

[0034] – Figure 3 Schematically illustrates the method defined in TMC2 for encoding the geometry and attributes of a point cloud;

[0035] – Figure 4 Schematically illustrates the methods defined in TMC2 for decoding the geometry and attributes of a point cloud;

[0036] – Figure 5 A block diagram showing an example of a system in which various aspects and embodiments are implemented;

[0037] – Figure 6 The syntax elements of TCM2 modified according to this embodiment are shown.

[0038] Example of "group_of_frames_header()";

[0039] – Figure 7 , Figure 7a and Figure 7b The expression of TCM2 modified according to an embodiment of the present invention is shown as

[0040] An example of a syntax element of "frame_auxiliary_information(frame_index)"; and

[0041] – Figure 8 , Figure 8a and Figure 8b The expression of TCM2 modified according to an embodiment of the present invention is shown as

[0042] Another example of the syntax element of "frame_auxiliary_information(frame_index)".

[0043] Similar or identical elements are denoted by the same reference numerals.

[0044] Description of Embodiments of the Invention

[0045] The present invention will be described more fully below with reference to the accompanying drawings, in which examples of the present invention are shown. However, embodiments of the present invention may be implemented in many alternative forms and should not be construed as being limited to the examples set forth herein. Therefore, although embodiments of the present invention are susceptible to various modifications and alternative forms, specific examples thereof are shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that it is not desirable to limit embodiments of the present invention to the specific forms disclosed, but rather, this specification will cover all modifications, equivalents, and alternatives that fall within the spirit and scope of embodiments of the present invention as defined by the claims.

[0046] The terms used herein are only for the purpose of describing specific examples, rather than to limit embodiments of the present invention. As used herein, the singular forms "a", "an" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the terms "include" and / or "comprise" when used in this specification specify the existence of stated features, integers, steps, operations, elements and / or components, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. In addition, when an element is referred to as "responding" or "connected" to another element, it can directly respond or be connected to another element, or there can be an intermediate element. On the contrary, when an element is referred to as "directly responding" or "directly connected" to other elements, there is no intermediate element. As used herein, the term "and / or" includes any and all combinations of one or more related listed items, and can be abbreviated as " / ".

[0047] It should be understood that although the terms first, second, etc. can be used to describe various elements here, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the teaching of the present embodiment, the first element can be referred to as the second element, and similarly, the second element can be referred to as the first element.

[0048] Although some diagrams include arrows on communication paths to illustrate a primary direction of communication, it is to be understood that communication can occur in the opposite direction to the depicted arrows.

[0049] Some examples are described with respect to block diagrams and operational flow charts, wherein each block represents a portion of a circuit element, module, or code, which includes one or more executable instructions for implementing a specified logical function (s). It should also be noted that in other implementations, the (one or more) functions mentioned in the blocks may not occur in the order mentioned. For example, depending on the functions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order.

[0050] References herein to "according to an example" or "in an example" mean that a particular feature, structure, or characteristic described in conjunction with the example may be included in at least one implementation of the present embodiment. The phrases "according to an example" or "in an example" that appear in various places in the specification do not necessarily all refer to the same example, nor are separate or alternative examples necessarily mutually exclusive with other examples.

[0051] Reference signs appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims.

[0052] Although not explicitly described, the present examples and variations may be employed in any combination or sub-combination.

[0053] This embodiment is described for encoding / reconstructing two geometric images and two attribute images representing a point cloud, but can be extended to encoding / reconstructing geometric images of two sequences (videos) and attribute images of two sequences (videos) representing a point cloud sequence (temporal dynamic point cloud), because the geometry (two geometric images) and attributes (texture / color) of the point cloud of the point cloud sequence are then encoded / reconstructed independently of the geometry (two geometric images) and attributes (texture / color) of another point cloud of the point cloud sequence.

[0054] In general, the present embodiment relates to a method for encoding the properties of 3D points whose geometry is represented by a geometric image. The 3D points may form a point cloud representing, for example, the outer surface of a 3D object. However, the method is not limited to the encoding of point clouds and can be extended to any other collection of 3D points. The geometry (3D coordinates) of the 3D points is represented as a geometric image.

[0055] The method checks whether the depth value of a pixel in a first geometric image in the geometric image and the depth value of a co-located pixel in a second geometric image in the geometric image are different (different values). When the depth value of a pixel in the first geometric image and the depth value of a co-located pixel in the second geometric image are different, the method assigns (encodes) the attributes of a 3D point defined according to the 2D spatial coordinates of the pixel in the first geometric image and the depth value of the co-located pixel in the second geometric image. Otherwise, the method assigns a pseudo value as the attribute of the 3D point.

[0056] Therefore, the method modifies the usual geometry and attribute encoding of a set of 3D points by avoiding the encoding of spurious duplicate 3D points as occurs, for example, in TMC2. This avoids wasting computational and storage resources on the encoding side and limits the bit rate at which the encoded 3D points are transmitted.

[0057] The present embodiment also relates to a method for reconstructing a 3D point from a geometric image representing the geometric shape of the 3D point.

[0058] The method checks whether the depth value of a pixel in a first geometric image in the geometric image and the depth value of a co-located pixel in a second geometric image in the geometric image are different. Then, when the depth value of the pixel in the first geometric image and the depth value of the co-located pixel in the second depth image are different, the method reconstructs a 3D point from the 2D spatial coordinates of the pixel in the first geometric image and the depth value of the co-located pixel in the second geometric image. Otherwise, the 3D point is not reconstructed.

[0059] Therefore, the method modifies the usual geometry and property reconstruction of 3D points by avoiding the creation of spurious duplicate 3D points as occurs, for example, in TMC2. This avoids wasting computational and storage resources on the decoding side.

[0060] Examples of attributes extracted from an image could be color, texture, normal vector, etc. .

[0061] Figure 1 The diagram schematically shows steps of a method 100 for encoding attributes of a 3D point according to an example of the present embodiment.

[0062] For example, the 3D points may form a point cloud, but the method is not limited to point clouds and may be applied to any set of 3D points.

[0063] In step 110 , module M1 may obtain a geometric image representing the geometry of a 3D point: two of the three coordinates of the 3D point are represented by 2D coordinates of a pixel in the geometric image, and the pixel value represents the third coordinate (depth value) of the 3D point.

[0064] For example, in TMC2, the 3D points can be orthogonally projected onto a projection plane, and two geometric images D0 and D1 can be obtained from the depth values ​​associated with the projected 3D points. D0 is a first geometric image that represents the depth value of the closest 3D point to the projection plane, and D1 is a second geometric image that represents the depth value of the farthest 3D point. The geometric images can be encoded using, for example, a conventional image / video encoder such as HEVC.

[0065] In step 120 , the module M2 may obtain a first attribute image, for example T0 , which represents attributes of a 3D point RP defined by 2D spatial coordinates and depth values ​​of pixels in a first geometric image D0 of the obtained geometric images.

[0066] For example, in TMC2, the attributes of the 3D point RP are obtained from the original 3D point (see Section 2.5 of Appendix A for more details). The first attribute image T0 is encoded using a conventional image / video encoder such as HEVC ( Figure 1 not shown).

[0067] In step 130, the module may obtain a second attribute image, for example, T1. The second attribute image T1 represents the attributes of the supplementary 3D points SP defined according to the 2D space coordinates and depth values ​​of the pixels in the first geometric image.

[0068] First, in step 130, the module compares the depth value of the pixel P in the first geometric image (e.g., D0) with the depth value of the co-located pixel CP in the second geometric image (e.g., D1). Next, when the depth value of the pixel P in the first geometric image D0 and the depth value of the co-located pixel CP in the second geometric image D1 are not the same, the module M3 can assign the attribute of a 3D point to the co-located pixel in the second attribute image T1, the geometry of which is defined according to the 2D spatial coordinates of the pixel P in the first geometric image and the depth value of the co-located pixel CP in the second geometric image. Otherwise, the module M4 can assign a pseudo attribute DUM to the co-located pixel in the second attribute image.

[0069] The second attribute image T1 ( Figure 1 not shown).

[0070] According to an embodiment, the pseudo attribute is an attribute of the co-located pixels of the first attribute image T0.

[0071] According to an embodiment, the pseudo attribute is an average value of attributes associated with neighboring pixels located around the pixel P.

[0072] Figure 2 A diagram schematically shows steps of a method 200 for reconstructing a 3D point from a geometric image representing the geometric shape of the 3D point according to an example of the present embodiment.

[0073] In step 210 , the module may compare a depth value of a pixel P in a first geometric image (eg, D0 ) of the geometric images with a depth value of a co-located pixel CP in a second geometric image (eg, D1 ) of the geometric images.

[0074] In step 220 , module M5 may define a 3D point RP from the 2D spatial coordinates and the depth value of the pixel P in the first geometric image D0 .

[0075] In step 230, when the depth value of the pixel P in the first geometric image D0 is different from the depth value of the co-located pixel CP in the second geometric image D0, the module M6 defines a supplementary 3D point SP according to the 2D spatial coordinates of the pixel P in the first geometric image (for example, D0) and the depth value of the co-located pixel CP in the second geometric image (for example, D1).

[0076] The attribute of the 3D point RP defined according to the 2D spatial coordinates and depth value of the pixel in the first geometric image D0 is the value of the co-located pixel in the first attribute image T0. The attribute of the 3D point SP defined according to the 2D spatial coordinates of the pixel in the first geometric image D0 and the depth value of the co-located pixel in the second geometric image D1 is the value of the co-located pixel in the second attribute image T1 (not equal to the value of the pseudo value DUM).

[0077] The method 100 encodes the attributes of the 3D points and, in particular, implements a first function of assigning a pseudo value to a pixel value in a second attribute image T1 when the depth value of a co-located pixel in a first geometric image D0 and the depth value of a co-located pixel in a second geometric image D1 are the same (step 130).

[0078] This first function therefore limits the bit rate required to transmit the properties of the 3D points and reduces computational and storage resources.

[0079] The method 200 reconstructs a 3D point from a geometric image representing a geometric shape of the 3D point, and in particular, when the depth value of the pixel of the first geometric image and the depth value of the co-located pixel of the second geometric image are not the same, implements a second function that defines the 3D point based on the 2D spatial coordinates of the pixel of the first geometric image in the geometric image and the depth value of the co-located pixel of the second geometric image in the geometric image (step 230).

[0080] According to a variant, when the information data ID indicates a first value, the first and second functions are enabled, and when the information data ID indicates a second value, the first and second functions are disabled. Thus, the information data ID indicates whether the method 100 or 200 checks whether the depth value of the pixel of the first geometric image and the depth value of the co-located pixel of the second geometric image are different before encoding the attribute of the 3D point (method 100) or before reconstructing the 3D point (method 200).

[0081] According to one embodiment, the first and second functions are enabled / disabled on a frame group level.

[0082] The information data ID is then associated with a syntax element regarding a group of frames.

[0083] Figure 6 The syntax element "group_of_frames_header()" of TCM2 is shown, which includes a field of "remove_duplicate_coding_group_of_frames" indicating an information data ID associated with a group of frames.

[0084] According to this embodiment, Figure 6 This syntax element can be used to signal the information data ID.

[0085] According to an embodiment, the first and second functions are enabled / disabled at a frame group level and a frame level.

[0086] The information data ID is then associated with a syntax element relative to the frame group, such as the syntax element "group_of_frames_header()" of TCM2 ( Figure 6 ), and the information data is also associated with the syntax elements relative to the frame, such as Figure 7 , Figure 7a and Figure 7b shown.

[0087] Figure 7 The representation of TCM2 is shown as "frame_auxiliary_information(frame_index)" and as Figure 7 , Figure 7a and Figure 7b An example of a modified syntax element (grey shaded area) is shown in FIG.

[0088] According to this embodiment, Figure 6 The grammatical elements and Figure 7 , Figure 7a and Figure 7b This syntax element can be used to signal the information data ID.

[0089] According to an embodiment, the first and second functions are enabled / disabled at a frame group level, a frame level and a patch level.A patch may be defined as a portion of an image.

[0090] The information data ID is then associated with a syntax element relative to the frame group, such as the syntax element "group_of_frames_header()" of TCM2 ( Figure 6 ), the information data is also associated with syntax elements relative to the frame, such as Figure 8 , Figure 8a and Figure 8b shown.

[0091] Figure 8 The representation of TCM2 is shown as "frame_auxiliary_information(frame_index)" and as Figure 8 , Figure 8a and Figure 8b An example of a modified syntax element (grey shaded area) is shown. According to this embodiment, Figure 6 The grammatical elements and Figure 8 , Figure 8a and Figure 8b This syntax element can be used to signal the information data ID.

[0092] According to a variant of the last embodiment, the first and second functions are enabled / disabled at frame block level.

[0093] For example, when the patch overlaps with at least one image block, the information data ID may be signaled to indicate whether the first and second functions are enabled (or not enabled) for the image block.

[0094] This produces dense 3D points in certain parts of the patch.

[0095] Figure 3 The method defined in TMC2 (Appendix A) for encoding the geometry and attributes of a point cloud is schematically shown.

[0096] Basically, the encoder captures the geometric information of the point cloud PC in a first (D0) and a second (D1) geometric image.

[0097] As an example, the first and second geometric images are obtained in TMC2 as follows.

[0098] A geometric patch (a set of 3D points of a point cloud PC) is obtained by clustering the points of the point cloud PC according to their normal vectors at the points. All extracted geometric patches are then projected onto a 2D grid and packed, while trying to minimize unused space and ensuring that each TxT (e.g., 16×16) block of the grid is associated with a unique patch, where T is a user-defined parameter signaled into the bitstream.

[0099] A geometric image is then generated by utilizing the 3D to 2D mapping calculated during the packing process, more specifically, the packing position and size of the projected area of ​​each patch. More precisely, let H(u,v) be the set of points of the current patch that are projected to the same pixel (u,v). The first layer (also called the nearest layer or the first geometric image D0) stores the points with the smallest geometric value of H(u,v). The second layer (called the farthest layer or the second geometric image D1) captures the points with the highest geometric value within the interval [D,D+Δ] of H(u,v), where D is the geometric value of the pixel in the first geometric image D0 and is a user-defined parameter describing the thickness of the surface.

[0100] The first geometric image D0 is then outputted by the packing process. A padding process is also used to fill in the empty spaces between patches in order to generate a piecewise smooth first geometric image suitable for video compression.

[0101] The generated geometric images / layers D0 and D1 are then stored as video frames and compressed using any conventional video codec such as HEVC.

[0102] The encoder also captures the attribute information of the original point cloud PC in two texture (attribute) images by encoding / decoding the first and second geometric images and decoding the decoded first and second geometric images. Deprojection is performed to reconstruct the geometry of the point cloud. Once reconstructed, a color is assigned (color transferred) to each point of the reconstructed point cloud according to the color information of the original point cloud PC in a way that minimizes the color information encoding error.

[0103] According to one embodiment, for each reconstructed point, the color of its nearest point in the original point cloud is assigned as the color it is to be encoded.

[0104] Then, the first and second attribute images T0, T1 are generated by storing the color information to be encoded of each reconstructed point at the same position (ie, (i, u, v)) in the geometric image.

[0105] For example, when a point cloud needs to be reconstructed, that is, when the geometry and possible attributes of the point cloud are needed, methods 100 and 200 can be used in TMC2 (Appendix A). Figure 1 ) on the encoding side. This is the case, for example, for generating attribute images and for reconstructing geometric images.

[0106] Figure 4 The method for decoding the geometry and attributes of a point cloud as defined by TMC2 is schematically shown.

[0107] By decoding the bit stream BT, a decoded first geometric image is obtained and the decoded second geometric image Possibly, metadata is also decoded to reconstruct the point cloud geometric shapes.

[0108] Thus, the geometry of the point cloud is reconstructed by deprojecting the decoded first and second geometric images and possibly the metadata.

[0109] When a reconstructed point cloud is required, that is, when the geometry of the point cloud is required, the method 200 can also be used in TMC2 (Appendix A of Figure 2 ). This is the case, for example, for reconstructing the geometry of the point cloud.

[0110] exist Figure 1-8b In the embodiment, the modules are functional units that may or may not be associated with distinguishable physical units. For example, these modules or some of them may be introduced together into a unique component or circuit, or contribute to the functionality of the software. In contrast, some modules may consist of separate physical entities. Devices compatible with this embodiment are implemented using pure hardware, such as dedicated hardware, such as ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array) or VLSI (Very Large Scale Integration), or by several integrated electronic components embedded in a device or by a blend of hardware and software components.

[0111] Figure 5 A block diagram of an example of a system in which various aspects and embodiments are implemented is shown. System 5000 can be implemented as a device including various components described below, and is configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 5000 can be implemented in a single integrated circuit, multiple ICs and / or discrete components individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 5000 are distributed on multiple ICs and / or discrete components. In various embodiments, the system 5000 is communicatively coupled to other similar systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports. In various embodiments, the system 5000 is configured to implement one or more aspects described herein.

[0112] The system 5000 includes at least one processor 5010, which is configured to execute instructions loaded therein, for implementing various aspects described herein, for example. The processor 5010 may include embedded memory, input-output interface, and various other circuits known in the art. The system 5000 includes at least one memory 5020 (e.g., volatile memory device and / or non-volatile memory device). The system 5000 includes a storage device 5040, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drive, and / or optical disk drive. As a non-limiting example, the storage device 5040 may include an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0113] The system 5000 includes an encoder / decoder module 5030, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 5030 may include its own processor and memory. The encoder / decoder module 5030 represents a module (one or more) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding module and the decoding module. In addition, the encoder / decoder module 5030 may be implemented as a separate element of the system 5000 or may be incorporated into the processor 5010 as a combination of hardware and software as known to those skilled in the art.

[0114] Program code to be loaded onto the processor 5010 or encoder / decoder 5030 to perform various aspects described in this document may be stored in the storage device 5040 and subsequently loaded onto the memory 5020 for execution by the processor 5010. According to various embodiments, one or more of the processor 5010, memory 5020, storage device 5040, and encoder / decoder module 5030 may store one or more of various items during execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or portions of the decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0115] In several embodiments, memory within the processor 5010 and / or the encoder / decoder module 5030 is used to store instructions and provide working memory for processing required during encoding or decoding.

[0116] However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 5010 or the encoder / decoder module 5030) is used for one or more of these functions. The external memory may be a memory 5020 and / or a storage device 5040, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as working memory for MPEG-2, HEVC, VVC (Universal Video Coding), or TMC2.

[0117] As shown in block 5130, input to the elements of system 5000 may be provided through various input devices. Such input devices include, but are not limited to: (i) an RF portion that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0118] In various embodiments, the input device of block 5130 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements necessary for: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band limiting the frequency band to a narrower frequency band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband.

[0119] In one set-top box embodiment, the RF section and its associated input processing elements receive RF signals transmitted over a wired (eg, cable) medium and perform frequency selection to a desired frequency band by filtering, down-converting, and filtering again.

[0120] Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions.

[0121] Adding components may include inserting components between existing components, such as inserting an amplifier and an analog-to-digital converter.In various embodiments, the RF portion includes an antenna.

[0122] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 5000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Solomon error correction) may be implemented as needed, for example, in a separate input processing IC or processor 5010. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed in a separate interface IC or in processor 5010. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 5010 and encoder / decoder 5030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on output devices.

[0123] The various components of system 5000 may be arranged in an integrated housing. In the integrated housing, the various components may be interconnected and transmit data between them using a suitable connection arrangement (eg, an internal bus known in the art, including an I2C bus, wiring, and a printed circuit board).

[0124] The system 5000 includes a communication interface 5050 that enables communication with other devices via a communication channel 5060. The communication interface 5050 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 5060. The communication interface 5050 may include, but is not limited to, a modem or a network card, and the communication channel 5060 may be implemented, for example, within a wired and / or wireless medium.

[0125] In various embodiments, a Wi-Fi network (e.g., IEEE 802.11) is used to stream data to the system 5000. The Wi-Fi signals of these embodiments are received through a communication channel 5060 suitable for Wi-Fi communications and a communication interface 5050. The communication channel 5060 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet to allow streaming applications and other cloud communications.

[0126] Other embodiments provide streaming data to the system 5000 using a set top box that delivers the data through the HDMI connection of the input box 5130.

[0127] Still other embodiments use the RF connection of input box 5130 to provide streaming data to system 5000 .

[0128] The streaming data may be used as a means of signaling information used by the system 5000. The signaling information may include the information data ID as described above.

[0129] It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder.

[0130] The system 5000 can provide output signals to various output devices, including a display 5100, speakers 5110, and other peripheral devices 5120. In various examples of various embodiments, the other peripheral devices 5120 include one or more of the following: a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide functions based on the output of the system 5000.

[0131] In various embodiments, control signals are transmitted between the system 5000 and the display 5100, speakers 5110, or other peripheral devices 5120 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention.

[0132] The output devices may be communicatively coupled to the system 5000 via dedicated connections through respective interfaces 5070 , 5080 , and 5090 .

[0133] Alternatively, the output device may be connected to the system 5000 via the communication interface 5050 using the communication channel 5060. The display 5100 and the speaker 5110 may be integrated with the other components of the system 5000 into a single unit in an electronic device (eg, a television).

[0134] In various embodiments, the display interface 5070 includes a display driver, such as a timing controller (T Con) chip.

[0135] For example, if the RF portion of the input box 5130 is part of a separate set-top box, the display 5100 and speaker 5110 may alternatively be separate from one or more of the other components. In various embodiments where the display 5100 and speaker 5110 are external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0136] The embodiments of various processes and features described herein can be embodied in various devices or applications. Examples of such devices include encoders, decoders, postprocessors for processing the output from decoders, preprocessors for providing input to encoders, video encoders, video decoders, video codecs, network servers, set-top boxes, laptop computers, personal computers, cellular phones, PDAs, and any other devices or other communication devices for processing pictures or videos. It should be clear that the device can be mobile and can even be installed in a mobile vehicle.

[0137] In addition, the method may be implemented by instructions being executed by a processor, and such instructions (and / or data values ​​generated by the implementation scheme) may be stored on a computer-readable storage medium. A computer-readable storage medium may take the form of a computer-readable program product that is embedded in one or more computer-readable media and has a computer-executable computer-readable program code embedded thereon. As used herein, a computer-readable storage medium is considered to be a non-transitory storage medium that is endowed with an inherent ability to store information therein and to provide an inherent ability to retrieve information therefrom. A computer-readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or apparatus, or any suitable combination of the foregoing. It should be understood that, although more specific examples of computer-readable storage media to which the present embodiment may be applied are provided, as will be readily understood by those of ordinary skill in the art, the following are merely illustrative and non-exhaustive enumerations: portable computer disks; hard disks; read-only memories (ROMs); erasable programmable read-only memories (EPROMs or flash memories); portable compact disk read-only memories (CD-ROMs); optical storage devices; magnetic storage devices; or any suitable combination of the foregoing.

[0138] The instructions may form an application program tangibly embodied on a processor-readable medium.

[0139] Instructions may be, for example, hardware, firmware, software, or a combination thereof. Instructions may be found, for example, in an operating system, a separate application, or a combination of both. Thus, a processor may be characterized as both, for example, a device configured to perform a process and a device including a processor-readable medium (such as a storage device) having instructions for performing the process. In addition, the processor-readable medium may store data values ​​generated by the implementation in addition to or in lieu of the instructions.

[0140] As will be apparent to those skilled in the art, implementations may generate various signals formatted to carry information that may be stored or sent, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry as data the rules for writing or reading the grammar of the example of the present embodiment, or to carry as data the actual grammar values ​​written by the example of the present embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0141] Many implementations have been described. However, it should be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. In addition, it will be appreciated by those skilled in the art that other structures and processes may replace those disclosed, and the resulting implementations will perform at least substantially the same function (one or more) in at least substantially the same manner (one or more) to achieve at least substantially the same result (one or more) as the disclosed implementations. Therefore, the present application contemplates these and other implementations.

Claims

1. A method comprising encoding a 3D point, wherein the geometry of the 3D point is represented by a first geometric image and a second geometric image, and the attributes are represented by a first attribute image and a second attribute image, wherein encoding the 3D point comprises: Encodes the syntax elements associated with a set of point cloud frames, A first value of the syntax element indicates that a depth value of a pixel in a first geometric image and a depth value of a collocated pixel in a second geometric image are to be compared before reconstructing a 3D point using the depth value of the collocated pixel in the second geometric image, The second value of the syntax element indicates that the depth value of the pixel in the first geometric image and the depth value of the co-located pixel in the second geometric image will not be compared, encoding the first geometric image and the second geometric image, wherein the first geometric image and the second geometric image store the depth values ​​of the 3D points of the point cloud frame projected onto the projection plane. 2 . The method according to claim 1 , wherein the attribute of the 3D point is a color value or a texture value.

3. The method according to claim 1, wherein: The first geometric image, the second geometric image, and the syntax element are encoded in a bitstream structured into a plurality of blocks, patches of blocks, and frames of patches, and wherein, in response to determining that the syntax element is equal to the first value, another syntax element is encoded in the bitstream at a frame level, at a patch level, or at a block level, the another syntax element indicating the same content as the syntax element.

4. The method according to claim 1, further comprising: encoding a first attribute image, wherein the first attribute image stores attribute values ​​of the 3D points projected on the projection plane, and depth values ​​of the 3D points are stored in the first geometric image, wherein first attribute values ​​of the first 3D points are stored in co-located pixels in the first attribute image, A second attribute image is encoded, wherein the second attribute image stores attribute values ​​of the 3D points projected on the projection plane, wherein in response to determining that a first depth value of a pixel in the first geometric image and a second depth value of a co-located pixel in the second geometric image are the same, a pseudo attribute value is assigned to the co-located pixel in the second attribute image.

5. The method according to claim 4, wherein: The pseudo attribute is an attribute of the co-located pixel of another attribute image, or the pseudo attribute is an average value of attributes associated with neighboring pixels located around the pixel.

6. A method comprising: Decodes syntax elements associated with a set of point cloud frames, A first value of the syntax element indicates that a depth value of a pixel in a first geometric image and a depth value of a collocated pixel in a second geometric image are to be compared before reconstructing a 3D point using the depth value of the collocated pixel in the second geometric image, The second value of the syntax element indicates that the depth value of the pixel in the first geometric image and the depth value of the co-located pixel in the second geometric image are not to be compared, decoding the first geometric image and the second geometric image, the first geometric image and the second geometric image storing depth values ​​of 3D points of a point cloud frame projected onto a projection plane, reconstructing a first 3D point according to a first 2D coordinate in the first geometric image and a first depth value of a first pixel of the first geometric image at the first 2D coordinate, In response to determining that the decoded syntax element is equal to the first value, determining whether the first depth value of the first pixel is the same as a second depth value of a collocated pixel in the second geometric image, In response to determining that the first depth value and the second depth value are not the same, a second 3D point is reconstructed from the first 2D coordinate and the second depth value.

7. The method of claim 6, wherein a bitstream carrying the encoded attributes of a 3D point is structured into a plurality of blocks, patches of blocks, and frames of patches, wherein in response to determining that the decoded syntax element is equal to the first value, another syntax element is signaled at a frame level, at a patch level, or at a block level, the other syntax element indicating the same content as the syntax element. 8 . The method of claim 6 , wherein in response to determining that the first depth value and the second depth value are the same, reconstructing the second 3D point from the 2D spatial coordinates of the collocated pixel in the second geometric image and the second depth value is skipped.

9. An apparatus for encoding a 3D point, the geometry of the 3D point being represented by a first geometric image and a second geometric image and the attributes of the 3D point being represented by a first attribute image and a second attribute image, wherein the apparatus comprises a processor configured to: Encodes the syntax elements associated with a set of point cloud frames, A first value of the syntax element indicates that a depth value of a pixel in a first geometric image and a depth value of a collocated pixel in a second geometric image are to be compared before reconstructing a 3D point using the depth value of the collocated pixel in the second geometric image, The second value of the syntax element indicates that the depth value of the pixel in the first geometric image and the depth value of the co-located pixel in the second geometric image will not be compared, encoding the first geometric image and the second geometric image, wherein the first geometric image and the second geometric image store the depth values ​​of the 3D points of the point cloud frame projected onto the projection plane. 10 . The apparatus according to claim 9 , wherein the attribute of the 3D point is a color value or a texture value.

11. The device according to claim 9, wherein: The first geometric image, the second geometric image, and the syntax element are encoded in a bitstream structured into a plurality of blocks, patches of blocks, and frames of patches, and wherein, in response to determining that the syntax element is equal to the first value, another syntax element is encoded in the bitstream at a frame level, at a patch level, or at a block level, the another syntax element indicating the same content as the syntax element.

12. The apparatus of claim 9, wherein the processor is further configured to: A first attribute image is encoded, wherein the first attribute image stores attribute values ​​of the 3D points projected on the projection plane, and depth values ​​of the 3D points are stored in the first geometric image, wherein: A first attribute value of the first 3D point is stored in a co-located pixel in the first attribute image, A second attribute image is encoded, wherein the second attribute image stores attribute values ​​of the 3D points projected on the projection plane, wherein in response to determining that a first depth value of a pixel in the first geometric image and a second depth value of a co-located pixel in the second geometric image are the same, a pseudo attribute value is assigned to the co-located pixel in the second attribute image. 13 . The apparatus according to claim 12 , wherein the pseudo attribute is an attribute of the co-located pixel of another attribute image, or the pseudo attribute is an average value of attributes associated with neighboring pixels located around the pixel.

14. An apparatus for reconstructing a 3D point from a geometric image representing a geometric shape of the 3D point, wherein the apparatus comprises a processor configured to: Decodes syntax elements associated with a set of point cloud frames, A first value of the syntax element indicates that a depth value of a pixel in a first geometric image and a depth value of a collocated pixel in a second geometric image are to be compared before reconstructing a 3D point using the depth value of the collocated pixel in the second geometric image, The second value of the syntax element indicates that the depth value of the pixel in the first geometric image and the depth value of the co-located pixel in the second geometric image are not to be compared, decoding the first geometric image and the second geometric image, the first geometric image and the second geometric image storing depth values ​​of 3D points of a point cloud frame projected onto a projection plane, reconstructing a first 3D point according to a first 2D coordinate in the first geometric image and a first depth value of a first pixel of the first geometric image at the first 2D coordinate, In response to determining that the decoded syntax element is equal to the first value, determining whether the first depth value of the first pixel is the same as a second depth value of a collocated pixel in the second geometric image, In response to determining that the first depth value and the second depth value are not the same, a second 3D point is reconstructed from the first 2D coordinate and the second depth value.

15. The apparatus of claim 14, wherein a bitstream carrying the encoded attributes of 3D points is structured into a plurality of blocks, patches of blocks, and frames of patches, wherein in response to determining that the decoded syntax element is equal to the first value, another syntax element is signaled at a frame level, at a patch level, or at a block level, the another syntax element indicating the same content as the syntax element. 16 . The apparatus of claim 14 , wherein in response to determining that the first depth value and the second depth value are the same, reconstructing the second 3D point from the 2D spatial coordinates of the collocated pixel in the second geometric image and the second depth value is skipped.

17. A non-transitory storage medium carrying instructions of a program code for performing the steps of the method according to claim 1 when the program is executed on a computing device.

18. A non-transitory storage medium carrying instructions of a program code for performing the steps of the method according to claim 6 when the program is executed on a computing device.

Images