Method and device for encoding / reconstructing three-dimensional points

Abstract

To provide a method for encoding three-dimensional points whose geometry is represented by a geometry image and whose attributes are represented by attribute images.SOLUTION: There is provided a method for checking whether a depth value of a pixel in a first geometry image D0 and a depth value of a pixel located at the same location in a second geometry image D1 are not the same (not identical values). A module M3 assigns (encodes) an attribute of a three-dimensional point defined from the two-dimensional spatial coordinates of the pixel in the first geometry image and the depth value of the pixel located at the same location in the second geometry image when the depth value of the pixel in the first geometry image and the depth value of the pixel located at the same location in the second geometry image are not the same.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present embodiments relate generally to the coding and reconstruction of 3D points, and in particular, but not exclusively, to the coding / reconstruction of point clouds representing the exterior surfaces of 3D objects. [Background technology]

[0002] This section is intended to introduce the reader to various aspects of the art, which may be related to various aspects of the present embodiments described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present embodiments. As such, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0003] A point cloud is a set of data points in some coordinate system. In a three-dimensional coordinate system (3D space), these points are usually intended to represent the exterior surface of a 3D object. Each point in a point cloud is often defined by its position (X, Y, Z coordinates in 3D space) and possibly by other associated attributes such as color, expressed for example in RGB or YUV color space, transparency, reflectance, two-component normal vector, etc.

[0004] We usually represent a point cloud as a set of six-component points (X,Y,Z,R,G,B) or equivalently (X,Y,Z,Y,U,V), where (X,Y,Z) define the coordinates of a colored point in 3D space, and (R,G,B) or (Y,U,V) define the color of this colored point.

[0005] A point cloud can be static or dynamic, depending on whether the cloud evolves over time. In a dynamic point cloud, the number of points is not constant, but rather typically evolves over time. Thus, a dynamic point cloud is a time-ordered list of a set of points.

[0006] In practice, point clouds can be used for various purposes, such as cultural heritage / architecture, to scan objects such as statues or buildings in 3D to share the spatial configuration of the object without having to transmit or visit it, or as a way to ensure that knowledge of the object is preserved in case it is destroyed, for example a temple may be destroyed by an earthquake. Such point clouds are usually static, colored and huge.

[0007] Another use case is in topography and cartography, where using a 3D representation the map is not limited to a flat surface but can include relief. Currently, Google Maps is a good example of a 3D map, but it uses a mesh rather than a point cloud. Nevertheless, point clouds can be the preferred data format for 3D maps, and such point clouds are usually static, colored, and large.

[0008] The automotive industry and autonomous vehicles are also areas where point clouds can be used. An autonomous vehicle should be able to "spy" on its environment to make good driving decisions based on the reality in its immediate vicinity. Typical sensors like LIDAR generate dynamic point clouds that are used in decision engines. These point clouds are not intended for humans to see, are typically small, not necessarily colored, and are dynamic since they are captured at a high frequency. They may have other attributes like reflectivity provided by Lidar, which may be useful for decision making since this is good information about the material of the sensed object.

[0009] Virtual reality and immersive worlds have been a hot topic lately, with many predicting them as the future of 2D flat video. The basic idea is to immerse the viewer in the environment with everything around him, as opposed to standard television, where the viewer can only see the virtual world in front of him. There are several stages of immersion, depending on the viewer's degree of freedom in the environment. Colored point clouds are good format candidates to deliver virtual reality (or VR) worlds. They can be static or dynamic, and are usually of average size, no more than a few million points at a time.

[0010] Point cloud compression is successful for storing / transmitting 3D objects for immersive worlds only if the size of the bitstream is small enough to allow practical storage / transmission to the end user.

[0011] It is important to be able to deliver dynamic point clouds to end users with a reasonable consumption of bitrate while maintaining an acceptable (or preferably very good) quality of experience. Efficient compression of these dynamic point clouds is a key aspect to making an immersive world delivery chain practical.

[0012] Image-based point cloud compression techniques are becoming increasingly popular due to their combination of compression efficiency and low complexity. They involve two main steps: first, projecting (orthogonally projecting) the point cloud, i.e., the 3D points, onto at least one 2D image plan. For example, at least one 2D geometry (also called depth) image represents the geometry of the point cloud, i.e., the spatial coordinates of the 3D points in the 3D space, and at least one 2D attribute (also called texture) image represents the attributes associated with the 3D points of the point cloud, e.g., the texture / color information associated with those 3D points. Then, these techniques use legacy video encoders to encode such geometry and attribute images into at least one geometry and attribute layer.

[0013] Image-based point cloud compression techniques achieve good compression performance by leveraging the capabilities of 2D video coders such as HEVC ("ITU-T H.265 Telecommunication standardization sector of ITU (10 / 2014), series H: audiovisual and multimedia systems, infrastructure of audiovisual services-coding of moving video, High efficiency video coding, Recommendation ITU-T H.265"), while keeping complexity low by using a simple projection scheme.

[0014] One challenge of image-based point cloud compression techniques is that the point cloud may not be suitable for projection onto an image, especially when the point distribution follows a surface with many folds (concave / convex regions as in clothing) or does not follow a surface at all (like fur or hair). In these situations, image-based point cloud compression techniques suffer from low compression efficiency (many small projections are required, making 2D video compression less efficient) or low quality (due to the difficulty of projecting the point cloud onto a surface).

[0015] One of the approaches used in the state of the art to mitigate this problem involves projecting multiple geometry and attribute information to the same spatial location in the image. This means that for each 3D point of the point cloud that has the same projected coordinates (the same 2D spatial coordinates of a pixel), several geometry and / or attribute images can be generated.

[0016] This is the case for example for the so-called Test Model Category 2 Point Cloud Coder (TMC2), as defined in ISO / IEC JTC1 / SC29 / WG11 MPEG2018 / N17767, Ljubljana, July 2018 (Annex A), where the point cloud is orthogonally projected onto a projection plane. Then, for each coordinate of said projection plane, two geometry images are generated, one representing the depth value associated with the nearest point (minimum depth value) and the other representing the depth value of the furthest point (maximum depth value). Then, a first geometry image is generated from the minimum depth value (D0) and a second geometry image is generated from the absolute value of the maximum depth value (D1), where D1-D0 is less than or equal to the maximum surface thickness. A first and a second attribute image are also generated in association with the first (D0) and second (D1) geometry images. Then, both the attribute images and the geometry images are encoded and decoded using a legacy video codec, such as HEVC. Thus, the geometry of the point cloud is reconstructed by backprojection of the information contained in the decoded first and second geometry images and the attributes are related to the reconstructed 3D from the information contained in the decoded attribute image.

[0017] The drawback of capturing two geometry (and two attribute) values ​​is that when two 3D points are systematically reconstructed from two geometry images, and the depth value of a pixel in the first geometry image is equal to the depth value of a pixel located at the same location in the second geometry image, a duplicate reconstructed 3D point is created. Then, encoding the unnecessary duplicate points increases the bit rate for transmitting the set of encoded 3D points. Moreover, computing and storage resources are wasted on both the encoding and decoding sides to deal with such false duplicate 3D points. Summary of the Invention

[0018] The following presents a simplified summary of the embodiments in order to provide a basic understanding of some aspects of the embodiments. This summary is not an extensive overview of the embodiments. It is not intended to identify key or critical elements of the embodiments. The following summary merely presents some aspects of the embodiments in a simplified form as a prelude to the more detailed description provided below.

[0019] The present embodiment sets out to remedy at least one of the shortcomings of the prior art with a method and apparatus for encoding a 3D point whose geometry is represented by a geometry image and whose attributes are represented by an attribute image, where if a depth value of a pixel in a first one of the geometry images and a depth value of a co-located pixel in a second one of the geometry images are not the same, the method assigns attributes of a 3D point whose geometry is defined from the 2D spatial coordinates of the co-located pixel in the first geometry image and the depth value of the co-located pixel in the second geometry image to the co-located pixel in the attribute image.

[0020] According to one embodiment, the method further comprises assigning a dummy attribute to a pixel in the attribute image if a depth value of a pixel located at the same location in the first geometry image and a depth value of a pixel located at the same location in the second geometry image are the same.

[0021] According to one embodiment, the dummy attribute is an attribute of a co-located pixel in another attribute image.

[0022] According to one embodiment, the dummy attribute is the average of the attributes associated with the neighboring pixels located around said pixel.

[0023] According to one embodiment, the method further comprises transmitting information data indicating whether depth values ​​of pixels in the first geometry image and depth values ​​of co-located pixels in the second geometry image are compared prior to reconstructing 3D points from the geometry image.

[0024] According to another of their aspects, the present embodiments relate to a bitstream carrying coded attributes of 3D points being structured as a number of blocks, patches of blocks and frames of patches, said information data being enabled at a group of frame level, at a frame level, at a patch level or at a block level.

[0025] According to another of their aspects, the present embodiments relate to a method for reconstructing a 3D point from geometry images representing the geometry of the 3D point, the method comprising: reconstructing a 3D point from 2D spatial coordinates of a pixel in a first one of the geometry images and a depth value of a pixel located at the same location in a second one of the geometry images if a depth value of the pixel in the first geometry image and a depth value of the pixel located at the same location in the second depth image are not identical.

[0026] According to one embodiment, the method further comprises receiving information data indicating whether depth values ​​of pixels in the first geometry image and depth values ​​of co-located pixels in the second geometry image are compared prior to reconstructing 3D points from the geometry images.

[0027] According to one embodiment, the bitstream carrying the coded attributes of the 3D points is structured as a number of blocks, patches of blocks and frames of patches, said information data being enabled at a group of frames level, at a frame level, at a patch level or at a block level.

[0028] According to one embodiment, the attribute of the 3D point is a color value or a texture value.

[0029] One or more of at least one of the embodiments also provide an apparatus, a computer program product, a non-transitory computer readable medium, and a bitstream.

[0030] The particular nature of the present embodiment, as well as other objects, advantages, features and uses of the present embodiment, will become apparent from the following description of examples taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0031] An example of this embodiment is shown in the drawings. [Figure 1] 1 shows a schematic diagram of the steps of a method 100 for encoding attributes associated with 3D points according to an example of this embodiment. [Diagram 2] 2 shows a schematic diagram of steps of a method 200 for reconstructing 3D points from a geometry image representative of the geometry of said 3D points according to an example of this embodiment. [Diagram 3] 1 illustrates a schematic of a method for encoding geometry and attributes of a point cloud defined in TMC2. [Figure 4] 1 illustrates a schematic of a method for decoding geometry and attributes of a point cloud defined in TMC2. [Diagram 5] 1 illustrates a block diagram of an example system in which various aspects and embodiments may be implemented. [Figure 6] 13 shows an example of a syntax element "group_of_frames_header()" of TCM2 modified according to the present embodiment. [Figure 7] 13 shows an example of a syntax element indicated by "frame_auxiliary_information(frame_index)" in TCM2 and modified according to this embodiment. [Figure 8] 13 shows another example of a syntax element indicated by "frame_auxiliary_information(frame_index)" in TCM2 and modified according to this embodiment.

[0032] Similar or identical elements are referred to with the same reference numbers.

[0033] Description of the present embodiment The present embodiment will be described in detail below with reference to the attached figures, in which examples of the present embodiment are shown. However, the present embodiment may be embodied in many alternative forms and should not be construed as being limited to the examples described herein. Thus, while the present embodiment is susceptible to various modifications and alternative forms, specific examples thereof are shown by way of example in the drawings and are described in detail herein. However, it is understood that there is no intention to limit the present embodiment to the particular forms disclosed, but rather the specification covers all modifications, equivalents, and alternatives included within the spirit and scope of the present embodiment as defined by the claims.

[0034] The terms used herein are intended to describe particular examples only and are not intended to limit the present embodiment. As used herein, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly dictates otherwise. It will be further understood that as used herein, the terms "comprises," "comprising," "including," and / or "comprising" specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as "responsive to" or "connected" to another element, it may be directly responsive or connected to the other element, or intervening elements may be present. In contrast, when an element is referred to as "directly responsive to" or "directly connected" to another element, there are no intervening elements. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ."

[0035] In this specification, the terms "first", "second" and the like may be used to describe various elements, but it is understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the teachings of the present embodiment.

[0036] Some diagrams include arrows on communication paths to indicate a primary direction of communication, however, it should be understood that communication may occur in a direction opposite to that depicted arrow.

[0037] Some examples are described with reference to block diagrams and operational flow charts, with each block representing a circuit element, module, or portion of code that includes one or more executable instructions for implementing a specified logical function(s). It should also be noted that in other implementations, the function(s) noted in the blocks may occur in an order different from the order noted. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved.

[0038] References herein to "by one example" or "in one example" mean that a particular feature, structure, or characteristic described with respect to the example may be included in at least one implementation of the present embodiment. The appearances of the phrases "by one example" or "in one example" in various places in the specification do not necessarily all refer to the same example, nor do separate or alternative examples necessarily mutually exclude other examples.

[0039] Reference numerals appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims.

[0040] Although not explicitly described, the present examples and variations may be used in any combination or subcombination.

[0041] The present embodiment describes encoding / reconstructing two geometry images and two attribute images representing a point cloud, but extends to encoding / reconstructing two sequences (videos) of geometry images and two sequences (videos) of attribute images representing a sequence of point clouds (temporally dynamic point clouds), since the geometry (two geometry images) and attributes (texture / color) of a point cloud of one sequence of point clouds are encoded / reconstructed independently of the geometry (two geometry images) and attributes (texture / color) of another point cloud of the sequence of point clouds.

[0042] Generally speaking, the present embodiment relates to a method for encoding attributes of 3D points whose geometry is represented by a geometry image. The 3D points may for example form a point cloud representing an external surface of a 3D object. However, the method is not limited to encoding point clouds, but is extended to other sets of 3D points. The geometry (3D coordinates) of the 3D points are represented as a geometry image.

[0043] The method checks whether a depth value of a pixel in a first one of the geometry images and a depth value of a pixel located at the same location in a second one of the geometry images are not the same (not the same value). If a depth value of a pixel in the first geometry image and a depth value of a pixel located at the same location in the second geometry image are not the same, the method assigns (encodes) an attribute of a 3D point defined from the 2D spatial coordinates of the pixel in the first geometry image and the depth value of the pixel located at the same location in the second geometry image. Otherwise, the method assigns a dummy value as the attribute of the 3D point.

[0044] The method thus modifies the usual encoding of geometry and attributes of a set of 3D points by avoiding the encoding of false double 3D points, as occurs for example in TMC2, which avoids the waste of computing and storage resources on the encoding side and limits the bit rate for transmitting the encoded 3D points.

[0045] The present embodiment also relates to a method for reconstructing 3D points from a geometry image representative of the geometry of said 3D points.

[0046] The method checks whether a depth value of a pixel in a first one of the geometry images and a depth value of a co-located pixel in a second one of the geometry images are not the same. Then, if the depth value of the pixel in the first geometry image and the depth value of the co-located pixel in the second depth image are not the same, the method reconstructs a 3D point from the 2D spatial coordinates of the pixel in the first geometry image and the depth value of the co-located pixel in the second geometry image. Otherwise, the 3D point is not reconstructed.

[0047] Thus, the method modifies the usual reconstruction of geometry and attributes of 3D points by avoiding the creation of false double 3D points, as happens for example in TMC2, thus avoiding the waste of computing and storage resources on the decoding side.

[0048] Examples of attributes extracted from an image may be color, texture, normal vectors, etc.

[0049] FIG. 1 shows a schematic diagram of the steps of a method 100 for encoding attributes of 3D points according to an example of this embodiment.

[0050] For example, the 3D points may form a point cloud, but the method is not limited to point clouds and may be applied to any set of 3D points.

[0051] In step 110, module M1 is able to obtain a geometry image representing the geometry of a 3D point: two of the three coordinates of said 3D point are represented by the 2D coordinate of a pixel in the geometry image and a pixel value represents the third coordinate (depth value) of said 3D point.

[0052] For example, in TMC2, 3D points may be orthogonally projected onto a projection plane, and two geometry images D0 and D1 may be obtained from the depth values ​​associated with the projected 3D points. D0 is a first geometry image representing the depth value of the closest 3D point of the projection plane, and D1 is a second geometry image representing the depth value of the furthest 3D point. The geometry images may be encoded using a legacy image / video encoder, such as HEVC.

[0053] In step 120, module M2 may obtain a first attribute image, e.g., T0, representing attributes of a 3D point RP defined from the 2D spatial coordinates and depth values ​​of pixels in a first geometry image D0 of the acquired geometry images.

[0054] For example, in TMC2, the attributes of the 3D point RP are obtained from the original 3D point (see Section 2.5 of Appendix A for details). The first attribute image T0 is encoded using, for example, a legacy image / video encoder such as HEVC (not shown in Figure 1).

[0055] In step 130, the module may obtain a second attribute image, e.g., T1, which represents attributes of supplementary 3D points SP defined from 2D spatial coordinates and depth values ​​in the first geometry image.

[0056] First, in step 130, the module compares the depth value of a pixel P in a first geometry image, for example D0, with the depth value of a pixel CP located at the same location in a second geometry image, for example D1. Then, if the depth value of the pixel P in the first geometry image D0 and the depth value of the pixel CP located at the same location in the second geometry image D1 are not the same, the module M3 may assign the attribute of a 3D point whose geometry is defined from the 2D spatial coordinates of the pixel P in the first geometry image and the depth value of the pixel CP located at the same location in the second geometry image T1 to the pixel located at the same location in the second attribute image T1. Otherwise, the module M4 may assign a dummy attribute DUM to the pixel located at the same location in the second attribute image.

[0057] The second attribute image T1 is encoded using a legacy image / video encoder such as, for example, HEVC (not shown in FIG. 1).

[0058] According to one embodiment, the dummy attributes are attributes of the co-located pixel of the first attribute image T0.

[0059] According to one embodiment, the dummy attribute is the average of the attributes associated with the neighbouring pixels located around said pixel P.

[0060] FIG. 2 shows a schematic diagram of the steps of a method 200 for reconstructing 3D points from a geometry image representative of the geometry of said 3D points according to an example of this embodiment.

[0061] In step 210, the module may compare a depth value of a pixel P in a first geometry image of the geometry images, e.g., D0, with a depth value of a pixel CP located in the same location in a second geometry image of the geometry images, e.g., D1.

[0062] In step 220, module M5 may define a 3D point RP from the 2D spatial coordinates and the depth value of a pixel P in the first geometry image D0.

[0063] In step 230, module M6 defines a complementary 3D point SP from the 2D spatial coordinates of pixel P in the first geometry image, e.g. D0, and the depth value of pixel CP located at the same location in the second geometry image, e.g. D1, if the depth value of pixel P in the first geometry image D0 and the depth value of pixel CP located at the same location in the second geometry image D0 are not the same.

[0064] The attribute of a 3D point RP defined from the 2D spatial coordinates and depth value of a pixel in the first geometry image D0 is the value of the co-located pixel in the first attribute image T0. The attribute of a 3D point SP defined from the 2D spatial coordinates of a pixel in the first geometry image D0 and the depth value of the co-located pixel in the second geometry image D1 is the value of the co-located pixel in the second attribute image T1 (a value not equal to the dummy value DUM).

[0065] The method 100 for encoding an attribute of a 3D point implements, in particular, a first function of assigning a dummy value to a pixel value in a second attribute image T1 if the depth value of a pixel located at the same location in the first geometry image D0 and the depth value of a pixel located at the same location in the second geometry image D1 are the same (step 130).

[0066] This first feature therefore limits the bit rate required to transmit the attributes of a 3D point, reducing computing and storage resources.

[0067] The method 200 implements a second function (step 230) of reconstructing a 3D point from geometry images representing the geometry of the 3D point, and in particular, defining the 3D point from the 2D spatial coordinates of a pixel in a first one of the geometry images and a depth value of a pixel located at the same location in a second one of the geometry images if the depth value of the pixel in the first geometry image and the depth value of the pixel located at the same location in the second geometry image are not the same.

[0068] According to a variant, said first and second functions are enabled when the information data ID represents a first value and disabled when said information data ID represents a second value. The information data ID thus indicates whether the method 100 or 200 checks, before encoding the attributes of the 3D point (method 100) or before reconstructing the 3D point (method 200), whether the depth value of a pixel in the first geometry image and the depth value of said co-located pixel in the second geometry image are not the same.

[0069] According to one embodiment, the first and second features are enabled / disabled in frame-level groups.

[0070] The information data ID is then associated with a syntax element that is relevant to the group of frames.

[0071] FIG. 6 shows an example of a TCM2 syntax element "group_of_frames_header()," which includes a field indicated as "remove_duplicate_coding_group_of_frames," which represents the information data ID associated with a group of frames.

[0072] This syntax element in FIG. 6 may be used to signal the information data ID according to this embodiment.

[0073] According to one embodiment, the first and second features are enabled / disabled at a group and frame level.

[0074] The information data ID is then associated with syntax elements related to groups of frames, for example the syntax element "group_of_frames_header()" of TCM2 (Figure 6), and the information data is also associated with syntax elements related to frames as shown in Figures 7-7b.

[0075] Figure 7 shows examples of syntax elements indicated by “frame_auxiliary_information(frame_index)” in TCM2 and modified as shown in Figures 7, 7a, and 7b (grey shaded areas).

[0076] The syntax element of FIG. 6 and this syntax element of FIGS. 7-7b may be used to signal the information data ID according to this embodiment.

[0077] According to one embodiment, the first and second features are enabled / disabled at a group, frame level and at a patch level, a patch being defined as a portion of an image.

[0078] The information data ID is then associated with syntax elements related to a group of frames, for example the syntax element "group_of_frames_header()" in TCM2 (Figure 6), and the information data is also associated with syntax elements related to frames as shown in Figures 8-8b.

[0079] Figure 8 shows an example of a syntax element indicated by TCM2 "frame_auxiliary_information(frame_index)" and modified as shown in Figures 8, 8a and 8b (grey shaded area). This syntax element of Figure 6 and Figures 8-7b can be used to signal information data ID according to this embodiment.

[0080] According to a variant of said last embodiment, said first and second functions are enabled / disabled at frame block level.

[0081] For example, if the patch overlaps at least one image block, the information data ID may be signaled to indicate whether the first and second features are enabled (or not enabled) for the image block.

[0082] This creates dense 3D points in parts of the patch.

[0083] FIG. 3 illustrates diagrammatically a method for encoding the geometry and attributes of a point cloud defined in TMC2.

[0084] Essentially, the encoder captures the geometric information of the point cloud PC in a first (D0) and a second (D1) geometric image.

[0085] As an example, the first and second geometry images are acquired at TMC2 as follows.

[0086] A geometry patch (a set of 3D points of the point cloud PC) is obtained by clustering the points of the point cloud PC according to the normal vectors at these points. Then, all extracted geometry patches are projected and packed into a 2D grid, minimizing unused space and ensuring that every TxT (e.g., 16x16) block of the grid is associated with a unique patch, where T is a user-defined parameter signaled in the bitstream.

[0087] A geometry image is then generated utilizing the 3D to 2D mapping calculated during the packing process, more specifically, the packing position and size of the projected region of each patch. More precisely, let H(u,v) be the set of points of the current patch that are projected to the same pixel (u,v). The first layer, also called the closest layer or first geometry image D0, stores the points of H(u,v) that have the smallest geometry value. The second layer, called the farthest layer or second geometry image D1, captures the points of H(u,v) that have the highest geometry value within the interval [D,D+Δ], where D is the geometry value of the pixel in the first geometry image D0 and Δ is a user-defined parameter that represents the thickness of the surface.

[0088] The first geometry image D0 then outputs the packing process. The padding process is also used to fill empty spaces between the patches to generate a piecewise smooth first geometry image suitable for video compression.

[0089] The generated geometry images / layers D0 and D1 are then stored as video frames and compressed using any legacy video codec, such as HEVC.

[0090] The encoder encodes / decodes the first and second geometry images, and the decoded first and second geometry images

number

[0091] According to one embodiment, for each reconstructed point, the color of its closest point in the original point cloud is assigned as its color to be coded.

[0092] The first and second attribute images T0, T1 are generated by storing the color information to be coded for each reconstructed point in the same location as the geometry image, ie (i, u, v).

[0093] For example, methods 100 and 200 can be used on the encoding side of TMC2 (Figure 1 in Appendix A) when a reconstructed point cloud is needed, i.e., when the geometry and possibly attributes of the point cloud are needed. This is the case, for example, when generating attribute images and when reconstructing geometry images.

[0094] FIG. 4 shows a schematic of a method for decoding the geometry and attributes of a point cloud, as defined in TMC2.

[0095] Decoded first geometry image

number

number

number

[0096] Thus, the geometry of the point cloud is reconstructed by backprojecting said decoded first and second geometry images and possibly said metadata.

[0097] Method 200 can also be used on the decoding side of TMC2 (Figure 2 in Appendix A) when a reconstructed point cloud is required, i.e. when the geometry of the point cloud is required. This is the case, for example, when reconstructing the geometry of a point cloud.

[0098] In Figs. 1-8b, modules are functional units that may or may not be related to distinct physical units. For example, these modules or some of them may be organized into unique components or circuits or contribute to the functionality of software. Conversely, some modules may potentially be composed of separate physical entities. Apparatus compatible with the present embodiment may be implemented using pure hardware, for example using dedicated hardware such as ASICs, FPGAs, VLSIs, which are "Application Specific Integrated Circuits", "Field Programmable Gate Arrays", and "Very Large Scale Integrated Circuits", respectively, or from several integrated electronic components embedded in the device, or from a blend of hardware and software components.

[0099] FIG. 5 illustrates a block diagram of an example system in which various aspects and embodiments are implemented. System 5000 may be embodied as a device including various components described below and configured to implement one or more of the aspects described in the present application. Examples of such devices include various electronic devices, such as, but not limited to, personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television sets, personal video recording systems, connected home appliances, and servers. The elements of system 5000, either alone or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 5000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 5000 is communicatively coupled to other similar systems or to other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 5000 is configured to implement one or more of the aspects described in this document.

[0100] The system 5000 includes at least one processor 5010 configured to execute loaded instructions, for example, in implementing various aspects described herein. The processor 5010 may include embedded memory, input / output interfaces, and various other circuits, as is known in the art. The system 5000 includes at least one memory 5020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 5000 includes a storage device 5040, which may include non-volatile and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. The storage device 5040 may include, by way of non-limiting example, an internal storage device, an attached storage device, and / or a network-accessible storage device. The system 5000 includes an encoder / decoder module 5030 configured to process data to provide, for example, encoded or decoded video, which may include its own processor and memory. The encoder / decoder module 5030 represents a module or modules that may be included in a device that performs encoding and / or decoding functions. As is known, a device may include one or both of an encoding and decoding module. Furthermore, the encoder / decoder module 5030 may be implemented as a separate element of the system 5000 or may be incorporated within the processor 5010 as a combination of hardware and software, as is known to those skilled in the art.

[0101] Program code to be loaded into the processor 5010 or the encoder / decoder 5030 to implement various aspects described herein may be stored in the storage device 5040 and subsequently loaded into the memory 5020 for execution by the processor 5010. According to various embodiments, one or more of the processor 5010, memory 5020, storage device 5040, and encoder / decoder module 5030 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video, point clouds, reconstructed point clouds or portions of the decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and computational logic.

[0102] In some embodiments, memory within the processor 5010 and / or the encoder / decoder module 5030 is used to store instructions and provide working memory for processing required during encoding or decoding.

[0103] However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 5010 or the encoder / decoder module 5030) is used for one or more of these functions. The external memory may be the memory 5020 and / or the storage device 5040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations, such as MPEG-2, HEVC, VVC (Versatile Video Coding) or TMC2.

[0104] Input to the elements of system 5000 may be provided through various input devices as shown in block 5130. Such input devices include, but are not limited to, (i) an RF section that receives an RF signal transmitted over the air by, for example, a broadcaster, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0105] In various embodiments, the input devices of block 5130 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements necessary to (i) select a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-convert the selected signal, (iii) band-limit again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulate the down-converted band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired data packet stream. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include, for example, a wavelength tuner that performs a variety of these functions, including down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to baseband) or to baseband.

[0106] In one set-top box embodiment, the RF section and its associated input processing elements perform frequency selection by receiving, filtering, downconverting, and refiltering RF signals transmitted over a wired (e.g., cable) medium to the desired frequency band.

[0107] Various embodiments rearrange the order of the above (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functionality.

[0108] Adding elements may include inserting elements between existing elements, for example, inserting amplifiers, analog-to-digital converters, etc. In various embodiments, the RF section includes an antenna.

[0109] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 5000 to other electronic devices over USB and / or HDMI connections. It should be appreciated that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or within the processor 5010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented in a separate interface IC or within the processor 5010, as desired. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including the processor 5010, which operates in combination with memory and storage elements, and the encoder / decoder 5030, to process the data stream, as desired, for example, for display on an output device.

[0110] The various elements of system 5000 may be provided within a unitary housing in which the various elements may be interconnected and data may be transmitted between them using suitable connection arrangements, e.g., internal buses known in the art, including an I2C bus, hardwired, and printed circuit boards.

[0111] The system 5000 includes a communication interface 5050 that enables communication with other devices over a communication channel 5060. The communication interface 5050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 5060. The communication interface 5050 may include, but is not limited to, a modem or a network card, and the communication channel 5060 may be implemented in a wired and / or wireless medium, for example.

[0112] In various embodiments, data is streamed to the system 5000 using a Wi-Fi network, such as IEEE 802.11. The Wi-Fi signal in these embodiments is received via a communication channel 5060 and communication interface 5050 adapted for Wi-Fi communication. The communication channel 5060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, enabling streaming applications and other over-the-top communications.

[0113] Another embodiment provides streamed data to the system 5000 using a set-top box that delivers data via an HDMI connection in input block 5130.

[0114] Yet another embodiment uses the RF connection of input block 5130 to provide streamed data to system 5000.

[0115] The streamed data may be used as a method to signal information used by the system 5000. The signaling information may include the information data ID, as described above.

[0116] It should be appreciated that signaling can be accomplished in various ways, for example, one or more syntax elements, flags, etc. are used in various embodiments to signal information to a corresponding decoder.

[0117] The system 5000 can provide output signals to various output devices, including a display 5100, speakers 5110, and other peripheral devices 5120, which in various example embodiments may include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of the system 5000.

[0118] In various embodiments, control signals are communicated between the system 5000 and a display 5100, a speaker 5110, or other peripheral devices. 5120 using signaling such as AV.Link, CEC, or other communications protocols that allow control between devices with or without user intervention.

[0119] Output devices may be communicatively coupled to the system 5000 via dedicated connections through respective interfaces 5070, 5080, and 5090.

[0120] Alternatively, output devices may be connected to the system 5000 via the communications interface 5050 using communications channel 5060. The display 5100 and speakers 5110 may be integrated in a single unit with other components of the system 5000 in an electronic device such as a television.

[0121] In various embodiments, the display interface 5070 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0122] Alternatively, the display 5100 and speakers 5110 may be separate from one or more of the other components, for example if the RF section of the input 5130 is part of a separate set-top box. In various embodiments in which the display 5100 and speakers 5110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0123] Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications. Examples of such equipment include encoders, decoders, post-processors that process the output from the decoder, pre-processors that provide input to the encoder, video encoders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, cell phones, PDAs, and any other device for processing pictures or videos, or other communication devices. As should be clear, the equipment may be portable, even mounted on a moving vehicle.

[0124] In addition, the method may be implemented by instructions executed by a processor, and such instructions (and / or data values ​​produced by the implementation) may be stored in a computer-readable storage medium. The computer-readable storage medium may take the form of a computer-readable program product embodied in one or more computer-readable medium(s) and having computer-executable computer-readable program code embodied thereon. As used herein, a computer-readable storage medium is considered to be a non-transitory storage medium given the inherent capability of storing information thereon, as well as the inherent capability of providing retrieval of information therefrom. The computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. The following provides more specific examples of computer-readable storage media to which the present embodiment may be applied, but it should be understood that a portable computer diskette, a hard disk, a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or a suitable combination of any of the foregoing, is merely an illustrative and not an exhaustive list, as would be readily understood by one of ordinary skill in the art.

[0125] The instructions may form an application program tangibly embodied on a processor-readable medium.

[0126] The instructions may reside, for example, in hardware, firmware, software, or a combination. The instructions may be found, for example, in an operating system, a separate application, or a combination of the two. Thus, a processor may be characterized, for example, as both a device configured to perform a process and a device that includes a processor-readable medium (such as a storage device) having instructions for performing a process. Furthermore, a processor-readable medium may store data values ​​generated by an implementation in addition to or in lieu of instructions.

[0127] As will be apparent to one skilled in the art, implementations can generate a wide variety of signals formatted to carry information, which can be stored or transmitted, for example. Information can include, for example, instructions to perform a method or data generated by one of the described implementations. For example, a signal can be formatted to carry as data the rules for writing or reading the syntax of the described embodiment example, or to carry as data the actual syntax value written by the described embodiment example. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a wide variety of different wired or wireless links, as is known. The signal can be stored in a processor-readable medium.

[0128] A number of implementations have been described. Nevertheless, it will be understood that various modifications can be made. For example, elements of different implementations can be combined, supplemented, modified, or eliminated to produce other implementations. Also, those skilled in the art will understand that other structures and processes can be substituted for those disclosed, with the resulting implementations performing at least substantially the same function(s) and in at least substantially the same manner(s) to achieve at least substantially the same result(s) as the disclosed implementations. Accordingly, these and other implementations are contemplated by the present application.

Claims

1. 1. A method for encoding 3D points whose geometry is represented by geometry images and whose attributes are represented by attribute images, wherein if a depth value of a pixel in a first of the geometry images and a depth value of a co-located pixel in a second of the geometry images are not the same, the method assigns attributes of a 3D point whose geometry is defined from 2D spatial coordinates of the co-located pixel in the first geometry image and the depth value of the co-located pixel in the second geometry image to the co-located pixel in the attribute image.

2. 2. The method of claim 1, further comprising: assigning a dummy attribute to a pixel of the attribute image if a depth value of a co-located pixel in the first geometry image and a depth value of a co-located pixel in the second geometry image are the same.

3. The method of claim 1 or 2, wherein the dummy attribute is an attribute of the co-located pixel of another attribute image.

4. The method according to claim 2 or 3, wherein the dummy attribute is an average of attributes associated with neighbouring pixels located around the pixel.

5. 5. The method according to claim 1, further comprising transmitting information data indicating whether depth values ​​of pixels in the first geometry image and depth values ​​of co-located pixels in the second geometry image are compared before reconstructing 3D points from the geometry images.

6. 6. The method of claim 5, wherein the bitstream carrying the coded attributes of the 3D points is structured as a number of blocks, patches of blocks, and frames of patches, and the information data is enabled at a group of frame level, at a frame level, at a patch level, or at a block level.

7. 1. A method for reconstructing a 3D point from geometry images representing the geometry of the 3D point, the method comprising: if a depth value of a pixel in a first geometry image and a depth value of a co-located pixel in a second depth image are not the same, reconstructing the 3D point from 2D spatial coordinates of the pixel in the first geometry image and a depth value of the co-located pixel in the second depth image.

8. 8. The method of claim 7, further comprising receiving information data indicating whether depth values ​​of pixels in the first geometry image and depth values ​​of co-located pixels in the second geometry image are compared prior to reconstructing 3D points from the geometry images.

9. 9. The method of claim 8, wherein the bitstream carrying the coded attributes of the 3D points is structured as a number of blocks, patches of blocks, and frames of patches, and the information data is enabled at a group of frame level, at a frame level, at a patch level, or at a block level.

10. The method according to any one of claims 1 to 9, wherein the attributes of the 3D points are colour or texture values.

11. 1. An apparatus for encoding 3D points whose geometry is represented by a geometry image and whose attributes are represented by an attribute image, the apparatus comprising: a processor configured to assign attributes of a 3D point whose geometry is defined from 2D spatial coordinates of a co-located pixel in the first geometry image and a depth value of the co-located pixel in the second geometry image to a co-located pixel in the attribute image if a depth value of a pixel in a first of the geometry images and a depth value of the co-located pixel in a second of the geometry images are not the same.

12. 12. The apparatus of claim 11, wherein the method further comprises assigning a dummy attribute to a pixel of the attribute image if a depth value of a pixel located at the same location in the first geometry image and a depth value of a pixel located at the same location in the second geometry image are the same.

13. The apparatus according to claim 11 or 12, wherein the dummy attribute is an attribute of the co-located pixel of another attribute image.

14. 14. Apparatus according to claim 12 or 13, wherein the dummy attribute is an average of attributes associated with neighbouring pixels located around the pixel.

15. 15. The apparatus of claim 11, wherein the processor is further configured to transmit information data indicating whether depth values ​​of pixels in the first geometry image and depth values ​​of co-located pixels in the second geometry image are compared before reconstructing 3D points from the geometry images.

16. 16. The apparatus of claim 15, wherein the bitstream carrying the coded attributes of the 3D points is structured as a number of blocks, patches of blocks, and frames of patches, and the information data is enabled at a group of frame level, at a frame level, at a patch level, or at a block level.

17. 1. An apparatus for reconstructing a 3D point from geometry images representing a geometry of the 3D point, the apparatus comprising: a processor configured to reconstruct a 3D point from 2D spatial coordinates of a pixel in a first geometry image and a depth value of a co-located pixel in a second depth image when a depth value of the pixel in the first geometry image and a depth value of the co-located pixel in the second depth image are not identical.

18. 20. The apparatus of claim 17, wherein the processor is further configured to receive information data indicating whether depth values ​​of pixels in the first geometry image and depth values ​​of co-located pixels in the second geometry image are compared prior to reconstructing 3D points from the geometry images.

19. 20. The apparatus of claim 18, wherein the bitstream carrying the coded attributes of the 3D points is structured as a number of blocks, patches of blocks, and frames of patches, and the information data is enabled at a group of frame level, at a frame level, at a patch level, or at a block level.

20. The device according to any one of claims 11 to 19, wherein the attributes of the 3D points are colour values ​​or texture values.

21. A computer program product comprising program code instructions for carrying out the steps of the method according to claims 1 to 10, when said program code instructions are run on a computer.

22. A non-transitory storage medium carrying program code instructions for performing the steps of the method according to any one of claims 1 to 10, when the program is executed on a computing device.

23. 1. A bitstream carrying a geometry and attribute image representing geometry and attributes of a 3D point, the bitstream further carrying information data indicating whether reconstructing a 3D point from the geometry and attribute image requires checking whether a depth value of a pixel of a first geometry image and a depth value of a co-located pixel of a second geometry image are not the same before reconstructing the 3D point.

Images