Method, program product, computer readable medium and apparatus for processing point cloud
By receiving and filtering video streams interwoven with depth images, and utilizing a two-layer point cloud coding structure and video codec compression technology, the problem of efficient compression of dynamic point cloud data on terminal devices is solved, achieving high-quality point cloud reconstruction and low bit rate consumption.
Patent Information
- Application Number
- CN202410905172.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-03-15
- Filing Date
- 2019-01-21
- Publication Date
- 2024-11-01
AI Technical Summary
Existing technologies struggle to effectively compress dynamic point cloud data to deliver a high-quality, immersive experience on end-user devices while maintaining reasonable bit rate consumption.
By receiving and filtering a video stream interleaved with depth images, the depth values of the point cloud frames are encoded into two-layer depth images using a two-layer point cloud coding structure. The image data is then compressed using a video codec to generate bitstreams for the base layer and enhancement layer.
It achieves efficient compression of dynamic point cloud data, ensuring high-quality point cloud reconstruction on terminal devices, reducing bit rate consumption, and maintaining the quality of the immersive experience.
Smart Images

Figure CN118890482A_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application with application date of January 21, 2019, application number 201980018118.8, and invention name “Processing Point Cloud”. Technical Field
[0002] At least one of the present embodiments generally relates to processing of point clouds. Background Art
[0003] This section is intended to introduce various aspects of the art to the reader, which may be related to various aspects of at least one of the embodiments described and / or claimed below. It is believed that this discussion is helpful to provide background information to the reader to facilitate a better understanding of various aspects of at least one embodiment.
[0004] Point clouds can be used for various purposes such as cultural heritage / architecture, where objects such as statues or buildings are scanned in 3D to share the spatial configuration of the object without sending or accessing the object. Also, this is a way to ensure that knowledge of the object is preserved in case the object may be destroyed; for example, a temple was destroyed by an earthquake. Such point clouds are usually static, colored, and huge.
[0005] Another use case is in topography and mapping, where using a 3D representation allows maps to be more than just flat and can include relief. Google Maps is now a good example of a 3D map, but uses a mesh instead of a point cloud. However, a point cloud can be a suitable data format for a 3D map, and such a point cloud is usually static, colored, and large.
[0006] The automotive industry and self-driving cars are also areas where point clouds can be used. Self-driving cars should be able to "probe" their environment to make good driving decisions based on the actual situation of their neighbors. Typical sensors such as LIDAR (Light Detection and Ranging) produce dynamic point clouds used by decision engines. These point clouds are not intended for human viewing, and they are usually small, not necessarily colored, and dynamic with a high capture frequency. These point clouds can have other properties, such as reflectivity provided by LIDAR, because this property provides good information about the material of the sensed object and can help make decisions.
[0007] Virtual reality and immersive worlds have become a hot topic recently and are foreseen by many as the future of 2D flat video. The basic idea is to immerse the viewer in an environment that surrounds the viewer, as opposed to standard TV where the viewer can only see a virtual world in front of the viewer. There are several levels of immersion, depending on the degree of freedom of the viewer in the environment. Point clouds are a good format candidate for distributing virtual reality (VR) worlds.
[0008] In many applications, it is important to be able to distribute dynamic point clouds to end users (or store them in a server) by consuming only a reasonable amount of bitrate (or storage space for storing the application) while maintaining an acceptable (or preferably very good) quality of experience. In order to make the distribution chain of many immersive worlds feasible, efficient compression of these dynamic point clouds is a key point.
[0009] At least one embodiment has been designed with the foregoing in mind. Summary of the invention
[0010] A simplified summary of at least one present embodiment is provided below to provide a basic understanding of some aspects of the present disclosure. Summary of the invention is not a broad overview of the embodiments. It is not intended to identify the key or important elements of the embodiments. The following summary of the invention only provides some aspects of at least one present embodiment in a simplified form as a preface to the more detailed description provided elsewhere herein.
[0011] According to the general aspect of at least one embodiment, a method is provided, the method comprising: receiving a video stream including a first depth image interlaced with a second depth image, wherein the first depth image represents a first set of depth values of a first point associated with a point cloud frame, the first point being a point of the point cloud that is closer to a projection plane, and wherein the second depth image represents a second set of depth values of a second point associated with the point cloud frame, the second point being a point of the point cloud that is farther from the projection plane; and filtering pixel values of the second depth image by using pixel values of the first depth image.
[0012] A computer program product is also provided, comprising instructions stored on a non-transitory computer-readable medium, which instructions, when the program is executed by one or more processors, cause the one or more processors to perform the method.
[0013] Also provided is a non-transitory computer-readable medium comprising instructions for causing one or more processors to perform the method.
[0014] Also provided is a device comprising: a processor configured to: receive a video stream comprising a first depth image interlaced with a second depth image, wherein the first depth image represents a first set of depth values of a first point associated with a point cloud frame, the first point being a point of the point cloud that is closer to a projection plane, and wherein the second depth image represents a second set of depth values of a second point associated with the point cloud frame, the second point being a point of the point cloud that is farther from the projection plane; and filter pixel values of the second depth image by using pixel values of the first depth image.
[0015] According to a general aspect of at least one embodiment, a method is provided, the method comprising: populating a first depth image representing depth values of closer points of a point cloud frame and a second depth image representing depth values of farther points of the point cloud frame. The method also includes encoding a video stream, the video stream comprising a temporal interleaving of the encoded first and second images.
[0016] A method is also provided, the method comprising: decoding a video stream to provide a first depth image representing depth values of nearer points of a point cloud frame and a second depth image representing depth values of farther points of the point cloud frame; and
[0017] - filtering the pixel values of the second depth image by using the pixel values of the first depth image.
[0018] At least one or more of the present embodiments also provides an apparatus for implementing the above method, a computer program product, a non-transitory computer-readable medium, and a signal carrying data representing depth values of points in a point cloud frame and data representing surface thickness, wherein the surface thickness defines the maximum difference between the depth values of two points in the point cloud projected onto a projection plane along the same projection line.
[0019] Specific properties of at least one of the present embodiments as well as other objects, advantages, features and uses of at least one of the present embodiments will become apparent from the following description of examples taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In the accompanying drawings, examples of several embodiments are illustrated. The drawings show:
[0021] - Figure 1 A schematic block diagram of an example of a two-layer point cloud coding structure according to at least one of the present embodiments is illustrated;
[0022] - Figure 2 A schematic block diagram of an example of a two-layer based point cloud decoding structure according to at least one of the present embodiments is illustrated;
[0023] - Figure 3 illustrates a schematic block diagram of an example of an image-based point cloud encoder according to at least one of the present embodiments;
[0024] - Figure 4 illustrates a schematic block diagram of an example of an image-based point cloud decoder according to at least one of the present embodiments;
[0025] - Figure 5 schematically illustrates an example of a syntax representing a bitstream of a base layer BL according to at least one of the present embodiments;
[0026] - Figure 6illustrates a schematic block diagram of an example of a system in which various aspects and embodiments are implemented;
[0027] - Figure 7 A schematic block diagram illustrating sub-steps of steps 3200, 3300 and 3500 according to at least one of the present embodiments; and
[0028] - Figure 8 A schematic block diagram of sub-steps of steps 4200 and 4500 according to at least one of the present embodiments is illustrated. DETAILED DESCRIPTION
[0029] At least one of the present embodiments will be described more fully below with reference to the accompanying drawings, in which examples of at least one of the present embodiments are shown. However, the embodiments may be implemented in many alternative forms and should not be construed as being limited to the examples set forth herein. Therefore, it should be understood that there is no intention to limit the embodiments to the specific forms disclosed. On the contrary, the present disclosure is intended to cover all modifications, equivalents, and alternative forms falling within the spirit and scope of the present application.
[0030] When the accompanying drawings are presented as flow charts, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when the accompanying drawings are presented as block diagrams, it should be understood that it also provides a flow chart of the corresponding method / process. Similar or identical elements are referred to by the same reference numerals.
[0031] The aspects described and contemplated below can be implemented in many different forms. Figure 1-Figure 8 Some embodiments are provided, but other embodiments are contemplated and are not intended to be construed herein. Figure 1-Figure 8 The discussion does not limit the breadth of the embodiments.
[0032] At least one aspect generally relates to point cloud encoding and decoding, and at least another aspect generally relates to transmitting the generated or encoded bitstream.
[0033] More specifically, various methods and other aspects described herein below can be used to modify a module, e.g. Figure 3 The patch packing module PPM, the geometric image generator GIG, the filling process, Figure 4 The video decoder VDEC and the geometry generation module GGM are shown.
[0034] Furthermore, the present aspects are not limited to MPEG standards such as MPEG-I Part 5 related to point cloud compression, and may be applied to, for example, other standards and recommendations existing or developed in the future, as well as extensions of any such standards and recommendations (including MPEG-I Part 5). Unless otherwise indicated or technically excluded, the aspects described in this application may be used alone or in combination.
[0035] In the following, image data refers to data of one or more arrays of 2D samples of, for example, a specific image / video format. A specific image / video format may specify information about pixel values of an image (or video). For example, a specific image / video format may also specify information that can be used by a display and / or any other device to visualize and / or decode an image (or video). An image typically includes a first component in the shape of a first 2D sample array, typically representing the luminance (or brightness) of the image. An image may also include a second component and a third component in the shape of other 2D sample arrays, typically representing the chromaticity (or chromaticity) of the image. Some embodiments use a collection of 2D color sample arrays to represent the same information, such as a traditional three-color RGB representation.
[0036] In one or more embodiments, a pixel value is represented by a vector of C values, where C is the number of components. Each value of the vector is typically represented by a number of bits, which may define a dynamic range of the pixel value.
[0037] An image block refers to a collection of pixels belonging to an image. The pixel values of an image block (or image block data) refer to the values of the pixels belonging to the image block. Although rectangles are common, image blocks can have any shape.
[0038] A point cloud may be represented by a data set of 3D samples within a 3D volumetric space having unique coordinates and may also have one or more attributes.
[0039] The 3D samples of the data set can be defined by their spatial position (X, Y, and Z coordinates in 3D space) and possibly by one or more associated attributes (such as, for example, color expressed in RGB or YUV color space, transparency, reflectivity, a two-component normal vector, or any feature that represents the characteristics of the sample). For example, a 3D sample can be defined by 6 components (X, Y, Z, R, G, B) or equivalently by (X, Y, Z, y, U, V), where (X, Y, Z) defines the coordinates of a point in 3D space and (R, G, B) or (y, U, V) defines the color of the 3D sample. The same type of attributes can appear multiple times. For example, multiple color attributes can provide color information from different viewpoints.
[0040] A point cloud can be static or dynamic, depending on whether the cloud changes over time. A static point cloud or a dynamic point cloud instance is usually represented as a point cloud frame. It should be noted that in the case of a dynamic point cloud, the number of points is usually not constant, but instead usually changes over time. More generally, a point cloud can be considered dynamic if anything (such as, for example, the number of points, the position of one or more points, or any attribute of any point) changes over time.
[0041] As an example, a 2D sample can be defined by 6 components (u,v,Z,R,G,B) or equivalently by (u,v,Z,y,U,V). (u,v) defines the coordinates of the 2D sample in the 2D space of the projection plane. Z is the depth value of the 3D sample projected onto the projection plane. (R,G,B) or (y,U,V) defines the color of the 3D sample.
[0042] Figure 1 A schematic block diagram of an example of a two-layer based point cloud coding structure 1000 according to at least one of the present embodiments is illustrated.
[0043] The two-layer based point cloud coding structure 1000 can provide a bit stream B representing an input point cloud frame IPCF. Possibly, the input point cloud frame IPCF represents a frame of a dynamic point cloud. Then, the frame of the dynamic point cloud can be encoded by the two-layer based point cloud coding structure 1000 independently of another frame.
[0044] Basically, the two-layer based point cloud coding structure 1000 can provide the ability to construct a bitstream B into a base layer BL and an enhancement layer EL. The base layer BL can provide a lossy representation of the input point cloud frame IPCF, and the enhancement layer EL can provide a lossless representation by encoding isolated points not represented by the base layer BL. As a possible alternative, the enhancement layer EL can provide a higher quality (but lossy) representation of the input point cloud frame IPCF by encoding additional points not represented by the base layer BL.
[0045] The base layer BL can be composed of Figure 3 The image-based encoder 3000 shown in FIG. 1 can provide a geometry / texture image representing the geometry / attributes of the 3D samples of the input point cloud frame IPCF and can allow isolated 3D samples to be discarded. The base layer BL can be composed of Figure 4 The image-based decoder 4000 shown decodes and can provide an intermediate reconstructed point cloud frame IRPCF.
[0046] Then, return to Figure 1 In the two-layer based point cloud coding 1000, the comparator COMP can compare the 3D samples of the input point cloud frame IPCF with the 3D samples of the intermediate reconstructed point cloud frame IRPCF to detect / locate the lost / isolated 3D samples. Next, the encoder ENC can encode the lost 3D samples and can provide the enhancement layer EL. Finally, the base layer BL and the enhancement layer EL can be multiplexed together by the multiplexer MUX to generate a bitstream B.
[0047] According to an embodiment, the encoder ENC may comprise a detector which may detect 3D reference samples R of the intermediate reconstructed point cloud frame IRPCF and associate them to the missing 3D samples M. The encoder ENC may also be an image based encoder.
[0048] For example, the 3D reference sample R associated with a lost 3D sample M may be its nearest neighbor according to a given metric.
[0049] According to an embodiment, the encoder ENC may then encode the spatial position of the lost 3D sample M and its properties as a difference determined from the spatial position and properties of the 3D reference sample R.
[0050] In a variant, those differences may be coded separately.
[0051] For example, for a missing 3D sample M, using the spatial coordinates x(M), y(M), and z(M), the x-coordinate difference Dx(M), y-coordinate position difference Dy(M), z-difference Dz(M), R attribute component difference Dr(M), G attribute component difference Dg(M), and B attribute component difference Db(M) can be calculated as follows:
[0052] Dx(M)=x(M)-x(R),
[0053] Among them, x(M) is Figure 3 The x-coordinates of the 3D samples M and R in the provided geometry image,
[0054] Dy(M)=y(M)-y(R)
[0055] Among them, y(M) is Figure 3 The y coordinates of the 3D sample M and corresponding R in the provided geometry image,
[0056] Dz(M)=z(M)-z(R)
[0057] Among them, z(M) is Figure 3 The z coordinates of the 3D samples M and R in the provided geometry image,
[0058] Dr(M)=R(M)-R(R).
[0059] Where R(M), respectively R(R) is the r color component of the color attribute of the 3D sample M, respectively R,
[0060] Dg(M)=G(M)-G(R).
[0061] Where G(M), respectively G(R) is the g color component of the color attribute of the 3D sample M, respectively R,
[0062] Db(M)=B(M)-B(R).
[0063] Among them, B(M) and B(R) are the b color components of the color attributes of the 3D samples M and R respectively.
[0064] Figure 2 A schematic block diagram of an example of a two-layer based point cloud decoding structure 2000 according to at least one of the present embodiments is illustrated.
[0065] The behavior of the two-layer based point cloud decoding structure 2000 depends on its capabilities.
[0066] The two-layer based point cloud decoding structure 2000 with limited capabilities can access only the base layer BL from the bitstream B using the demultiplexer DMUX, and then can provide a true (but lossy) version RPCF of the input point cloud frame IPCF by decoding the base layer BL by the point cloud decoder 4000, as shown in Figure 4 As shown in the picture.
[0067] The two-layer based point cloud decoding structure 2000 with full capabilities can access both the base layer BL and the enhancement layer EL from the bitstream B by using the demultiplexer DMUX. Figure 4 As shown, the point cloud decoder 4000 can determine the reconstructed point cloud frame RPCF from the base layer BL. The decoder DEC can determine the supplementary point cloud frame CPCF from the enhancement layer EL. Then, the combiner COM can combine the reconstructed point cloud frame RPCF and the supplementary point cloud frame CPCF together to provide a lossless (or higher quality) representation (reconstruction) CRPCF of the input point cloud frame IPCF.
[0068] Figure 3 A schematic block diagram of an example of an image-based point cloud encoder 3000 according to at least one of the present embodiments is illustrated.
[0069] The image-based point cloud encoder 3000 utilizes existing video codecs to compress the geometry and texture (attribute) information of dynamic point clouds. This is achieved by essentially converting the point cloud data into a collection of different video sequences.
[0070] In certain embodiments, the two videos (one for capturing the geometry information of the point cloud data and the other for capturing the texture information) may be generated and compressed using an existing video codec. An example of an existing video codec is the HEVC Main Profile encoder / decoder (ITU-T H.265 ITU Telecommunication Standardization Sector (02 / 2018), Series H: Audiovisual and Multimedia Systems, Infrastructure of Audiovisual Services - Motion Video Coding, High Efficiency Video Coding, ITU-T H.265 Recommendation).
[0071] Additional metadata for interpreting the two videos is usually generated and compressed separately. Such additional metadata includes, for example, an occupancy map OM and / or auxiliary patch information PI.
[0072] The generated video bitstream and metadata may then be multiplexed together to generate a combined bitstream.
[0073] It should be noted that metadata usually represents a small amount of overall information. Most of the information is in the video bitstream.
[0074] An example of such a point cloud encoding / decoding process is given by the Test Model Category 2 algorithm (also known as V-PCC) defined in ISO / IEC JTC1 / SC29 / WG11 MPEG2018 / N 18030 and N 17996 (Macau, October 2018).
[0075] In step 3100 , the patch generation module PGM may generate at least one patch by decomposing 3D samples of a data set representing an input point cloud frame IPCF into 2D samples on a given projection plane using a strategy that provides optimal compression.
[0076] A patch can be defined as a collection of 2D samples.
[0077] For example, in V-PCC, the normal of each 3D sample is first estimated, for example, as described by Hoppe et al. (Hugues Hoppe, Tony DeRose, Tom Duchamp, John McDonald, Werner Stuetzle. Surface reconstruction from unorganized points. ACM SIGGRAPH 1992 Proceedings, 71-78). Next, the initial clustering of the input point cloud frame IPCF is obtained by associating each 3D sample with one of the six oriented planes of the 3D bounding box of the 3D sample containing the input point cloud frame IPCF. More precisely, each 3D sample is clustered and associated with the oriented plane with the closest normal (i.e., maximizing the dot product of the point normal and the plane normal). Then, the 3D sample is projected onto its associated plane. The collection of 3D samples that form a connected area in their plane is called a connected component. A connected component is a collection of at least one 3D sample with similar normals and the same associated oriented plane. The initial clustering is then improved by iteratively updating the cluster associated with each 3D sample based on the clustering of its normal and its nearest neighbors. The final step consists of generating a patch from each connected component by projecting the 3D samples of each connected component onto the oriented plane associated with the connected component.
[0078] The patches are associated with auxiliary patch information PI, which represents auxiliary patch information defined for each patch to explain the projected 2D samples corresponding to geometric shape and / or attribute information.
[0079] For example, in V-PCC, the auxiliary patch information PI includes: 1) information indicating one of the six oriented planes of the 3D bounding box of the 3D sample containing the connected component; 2) information related to the plane normal; 3) information determining the 3D position of the connected component relative to the patch, expressed in depth, tangential displacement, and bi-tangential displacement; 4) information such as coordinates (u0, v0, u1, v1) in the projection plane that defines the 2D bounding box containing the patch.
[0080] In step 3200, the patch packing module PPM may map (place) at least one generated patch onto a 2D grid (also referred to as a canvas) in a manner that minimizes unused space and without any overlap, and may ensure that each T×T (e.g., 16×16) block of the 2D grid is associated with a unique patch. A given minimum block size T×T of the 2D grid may specify a minimum distance between different patches placed on the 2D grid. The 2D grid resolution may depend on the size of the input point cloud, and its width W and height H as well as the block size T may be sent to the decoder as metadata.
[0081] The auxiliary patch information PI may also include information about the association between blocks and patches of the 2D mesh.
[0082] In V-PCC, the auxiliary information PI includes block-to-patch index information for determining association between blocks of a 2D grid and patch indices.
[0083] A T×T block containing 2D samples belonging to a patch may be considered as an occupied block in the corresponding occupancy map OM. A block of the occupancy map OM may then indicate whether the block is occupied, ie contains 2D samples belonging to a patch.
[0084] The image generation process (steps 3300 and 3400) utilizes the mapping of at least one generated patch onto the 2D mesh calculated during step 3200 to store the geometry and texture of the input point cloud frame IPCF as an image.
[0085] In step 3300, the geometry image generator GIG may generate at least one geometry image G1 from the input point cloud frame IPCF, the occupancy map OM and the auxiliary patch information PI. The geometry image generator GIG may utilize the occupancy map information to detect (locate) occupied blocks and thereby detect (locate) non-empty pixels in the geometry image G1.
[0086] The geometry image G1 may represent the geometry of the input point cloud frame IPCF and may be, for example, a monochrome image of W×H pixels represented in a YUV420-8 bit format.
[0087] To better handle the case where multiple 3D samples are projected (mapped) to the same 2D sample of the projection plane (along the same projection direction (line)), multiple images called layers can be generated. Thus, the 2D samples of a patch can have different depth values D1, ..., Dn associated with them, and multiple geometry images are generated.
[0088] In V-PCC, the 2D samples of a patch are projected onto two layers. The first layer, also called the near layer, may store, for example, the depth value D0 associated with the 2D sample with the lowest depth. The second layer, called the far layer, may store, for example, the difference between the depth values D1 and D0 associated with the 2D sample with the highest depth. Thus, the second depth image stores information in the interval [0, Δ] corresponding to depth values in the range [D0, D0+Δ], where Δ is a user-defined parameter describing the thickness of the surface.
[0089] In this way, the second layer may contain obvious contour-like high-frequency features. Therefore, this clearly shows that it is difficult to encode the second depth image by using a conventional video encoder, and therefore, the depth value is poorly reconstructed from the decoded second depth image, which results in poor quality of the geometry of the reconstructed point cloud frame.
[0090] According to an embodiment, the geometry image generation module GIG may encode (derive) depth values associated with 2D samples of the first layer and the second layer by using the auxiliary patch information PI.
[0091] In V-PCC, the position of a 3D sample in a patch with corresponding connected components can be represented by depth δ(u,v), tangential displacement s(u,v), and bi-tangential displacement r(u,v) as follows:
[0092] δ(u,v)=δ0+g(u,v)
[0093] s(u,v)=s0–u0+u
[0094] r(u,v)=r0–v0+v
[0095] where g(u,v) is the brightness component of the geometry image, (u,v) is the pixel associated with the 3D sample on the projection plane, (δ0,s0,r0) is the 3D position of the corresponding patch of the connected component to which the 3D sample belongs, and (u0,v0,u1,v1) are the coordinates in the projection plane that define a 2D bounding box containing the projection of the patch associated with the connected component.
[0096] Thus, the geometry image generation module GIG may encode (derive) the depth value associated with the 2D sample of the layer (first or second or both) as a luminance component g(u,v) given by: g(u,v) = δ(u,v) - δ0. Note that this relationship may be employed to reconstruct the 3D sample position (δ0, s0, r0) from the reconstructed geometry image g(u,v) and the accompanying auxiliary patch information PI.
[0097] According to an embodiment, the projection mode may be used to indicate whether the first geometry image GI0 may store depth values of 2D samples of the first or second layer and the second geometry image GI1 may store depth values associated with the 2D samples of the second or first layer.
[0098] For example, when the projection mode is equal to 0, the first geometric image GI0 may store depth values of the 2D samples of the first layer, and the second geometric image GI1 may store depth values associated with the 2D samples of the second layer. Correspondingly, when the projection mode is equal to 1, the first geometric image GI0 may store depth values of the 2D samples of the second layer, and the second geometric image GI1 may store depth values associated with the 2D samples of the first layer.
[0099] According to an embodiment, the frame projection mode may be used to indicate whether a fixed projection mode is used for all patches, or a variable projection mode in which each patch may use a different projection mode.
[0100] The projection mode and / or the frame projection mode may be transmitted as metadata.
[0101] For example, the frame projection mode decision algorithm may be provided in Section 2.2.1.3.1 of V-PCC.
[0102] According to an embodiment, when the frame projection indicates that a variable projection mode may be used, the patch projection mode may be used to indicate an appropriate mode for (de)projecting the patch.
[0103] The patch projection mode may be transmitted as metadata and, possibly, may be information contained in the auxiliary patch information PI.
[0104] As an example, a patch projection mode decision algorithm is provided in Section 2.2.1.3.2 of V-PCC.
[0105] According to an embodiment of step 3300, a pixel value in a first geometric shape image (e.g., GI1) corresponding to a 2D sample (u, v) of a patch may represent a depth value associated with at least one intervening 3D sample defined along a projection line corresponding to the 2D sample (u, v). The intervening 3D sample resides along the projection line and shares the same coordinates of the 2D sample (u, v) whose depth value D0 is encoded in the second geometric shape image (e.g., GI0). In addition, the intervening 3D sample may have a depth value between the depth value D0 and the depth value D0+ST (surface thickness value). A designated bit may be associated with each of the intervening 3D samples, the designated bit being set to 1 if there is an intervening 3D sample, otherwise being set to 0.
[0106] Then, all the designated bits along the projection lines may be concatenated to form a codeword, hereinafter referred to as an Enhanced Delta Depth (EDD) code.Finally, all EDD codes may be packed in an image, for example in a first geometry image GI1 or an occupancy map OM.
[0107] In step 3400, the texture image generator TIG may generate a texture image from an input point cloud frame IPCF, an occupancy map OM, auxiliary patch information PI, and from at least one decoded geometry image DGI (output of a video decoder VDEC) Figure 4 At least one texture image TI is generated from the geometry of the reconstructed point cloud frame derived from step 4200).
[0108] The texture image TI may represent the texture of the input point cloud frame IPCF and may be, for example, an image of W×H pixels represented in a YUV420-8 bit format.
[0109] The texture image generator TG may utilize the occupancy map information to detect (locate) occupied blocks and thus detect (locate) non-empty pixels in the texture image.
[0110] The texture image generator TIG may be adapted to generate a texture image TI and associate it with each geometry image / layer DGI.
[0111] According to an embodiment, the texture image generator TIG may encode (store) the texture (attribute) value T0 associated with the 2D sample of the first layer as a pixel value of the first texture image TI0, and encode (store) the texture value T1 associated with the 2D sample of the second layer as a pixel value of the second texture image TI1.
[0112] Alternatively, the texture image generation module TIG may encode (store) texture values T1 associated with 2D samples of the second layer as pixel values of the first texture image TI0, and encode (store) texture values D0 associated with 2D samples of the first layer as pixel values of the second geometric shape image GI1.
[0113] For example, the color of the 3D sample may be obtained as described in Section 2.2.3, Section 2.2.4, Section 2.2.5, Section 2.2.8, or Section 2.5 of V-PCC.
[0114] According to an embodiment, a padding process may be applied to the geometry and / or texture image. The purpose is to fill the empty spaces between patches to generate a piecewise smooth image suitable for video compression.
[0115] Image padding examples are provided in Sections 2.2.6 and 2.2.7 of V-PCC.
[0116] In step 3500, the video encoder VENC may encode the generated images / layers TI and GI.
[0117] In step 3600, the encoder OMENC may encode the occupancy map into an image, such as described in detail in Section 2.2.2 of V-PCC. Lossy or lossless coding may be used.
[0118] According to an embodiment, the video encoder ENC and / or OMENC may be an HEVC-based encoder.
[0119] In step 3700, the encoder PI ENC may encode the auxiliary patch information PI and possibly additional metadata such as the block size T, width W and height H of the geometry / texture image.
[0120] According to an embodiment, the auxiliary patch information may be differentially encoded (eg, as defined in Section 2.4.1 of V-PCC).
[0121] In step 3800, the generated video bitstreams as outputs of steps 3500, 3600 and 3700 may be multiplexed together to generate a bitstream representing the base layer BL. It should be noted that the metadata information represents a small portion of the entire bitstream. Most of the information is compressed using a video codec.
[0122] Figure 4 A schematic block diagram of an example of an image-based point cloud decoder 4000 according to at least one of the present embodiments is illustrated.
[0123] In step 4100, the demultiplexer DMUX may access coding information representing a bitstream of a base layer BL.
[0124] In step 4200, the video decoder VDEC may decode the encoded information to derive at least one decoded geometry image DGI and at least one decoded texture image DTI.
[0125] In step 4300 , the decoder OMDEC may decode the encoded information to derive a decoded occupancy map DOM.
[0126] According to an embodiment, the video decoder VDEC and / or OMDEC may be a HEVC-based decoder.
[0127] In step 4400 , the decoder PIDEC may decode the encoded information to derive auxiliary patch information DPI.
[0128] Possibly, metadata can also be derived from the bitstream BL.
[0129] In step 4500, the geometry generation module GGM may derive the geometry RG of the reconstructed point cloud frame RPCF (or IRPCF) from at least one decoded geometry image DGI, a decoded occupancy map DOM, decoded auxiliary patch information DPI and possible additional metadata.
[0130] The geometry generation module GGM may utilize the decoded occupancy map information DOM to locate non-empty pixels in at least one decoded geometry image DGI. The 3D coordinates of the reconstructed 3D samples associated with the non-empty pixels may then be derived from the coordinates of the non-empty pixels and the values of the reconstructed 2D samples.
[0131] According to an embodiment, the geometry generation module GGM may derive the 3D coordinates of the reconstructed 3D sample from the coordinates of the non-empty pixels.
[0132] According to an embodiment, the geometry generation module GGM may derive the 3D coordinates of the reconstructed 3D sample from the coordinates of non-empty pixels, the values of said non-empty pixels of one of at least one decoded geometry images DGI, the decoded auxiliary patch information and possibly from additional metadata.
[0133] The use of non-empty pixels is based on the 2D pixel relationship with the 3D sample. For example, using the projection in V-PCC, the 3D coordinates of the reconstructed 3D sample can be expressed as follows using depth δ(u,v), tangential displacement s(u,v) and double tangential displacement r(u,v):
[0134] δ(u,v)=δ0+g(u,v)
[0135] s(u,v)=s0–u0+u
[0136] r(u,v)=r0–v0+v
[0137] where g(u,v) is the brightness component of the decoded geometry image DGI, (u,v) is the pixel associated with the reconstructed 3D sample, (δ0,s0,r0) is the 3D position of the connected component to which the reconstructed 3D sample belongs, and (u0,v0,u1,v1) are the coordinates in the projection plane that define the projected 2D bounding box containing the patch associated with the connected component.
[0138] In step 4600, the texture generation module TGM may derive the texture of the reconstructed point cloud frame RPCF (or IRPCF) from the geometric shape RG and at least one decoded texture image DTI.
[0139] Figure 5 An example syntax representing a bitstream of a base layer BL according to at least one of the present embodiments is schematically illustrated.
[0140] The bit stream includes a bit stream header BSH and at least one frame stream group GOFS (Group Of Frame Stream).
[0141] The frame stream group GOFS includes a header HS, at least one syntax element OMS representing an occupancy map OM, at least one syntax element GVS representing at least one geometry image (or video), at least one syntax element TVS representing at least one texture image (or video), and at least one syntax element PIS representing auxiliary patch information and other additional metadata.
[0142] In a variant, the group of frame flows GOFS comprises at least one frame flow.
[0143] Figure 6 A schematic block diagram illustrating an example of a system in which various aspects and embodiments are implemented is shown.
[0144] System 6000 can be embodied as one or more devices including various components described below, and is configured to perform one or more aspects described in this document. Examples of devices that can form all or part of system 6000 include personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, networked home appliances, head-mounted display devices (HMD, perspective glasses), projectors (projectors), "cave" (systems including multiple displays), servers, video encoders, video decoders, post-processors for processing outputs from video decoders, pre-processors for providing inputs to video encoders, network servers, set-top boxes, and any other devices or other communication devices for processing point clouds, videos or images. The elements of system 6000 can be embodied in a single integrated circuit, multiple ICs and / or discrete components, either individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 6000 can be distributed over multiple ICs and / or discrete components. In various embodiments, system 6000 can be coupled to other similar systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports. In various embodiments, system 6000 can be configured to implement one or more aspects described in this document.
[0145] The system 6000 may include at least one processor 6010 configured to execute instructions loaded therein for implementing, for example, various aspects described in this document. The processor 6010 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 6000 may include at least one memory 6020 (e.g., a volatile storage device and / or a non-volatile storage device). The system 6000 may include a storage device 6040, which may include a non-volatile memory and / or a volatile memory, including, but not limited to, an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 6040 may include an internal storage device, an attached storage device, and / or a network-accessible storage device.
[0146] The system 6000 may include a coder / decoder module 6030 configured to, for example, process data to provide encoded data or decoded data, and the coder / decoder module 6030 may include its own processor and memory. The coder / decoder module 6030 may represent a module that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of the encoding and decoding modules. In addition, the coder / decoder module 6030 may be implemented as a separate element of the system 6000, or may be incorporated into the processor 6010 as a combination of hardware and software known to those skilled in the art.
[0147] Program code to be loaded onto the processor 6010 or the encoder / decoder 6030 to perform various aspects described in this document may be stored in the storage device 6040 and subsequently loaded onto the memory 6020 for execution by the processor 6010. According to various embodiments, one or more of the processor 6010, the memory 6020, the storage device 6040, and the encoder / decoder module 6030 may store one or more of various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, point cloud frames, encoded / decoded geometry / texture video / images, or portions of encoded / decoded geometry / texture video / images, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operation logic.
[0148] In several embodiments, memory internal to the processor 6010 and / or encoder / decoder module 6030 may be used to store instructions and provide working memory for processes that may be performed during encoding or decoding.
[0149] However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 6010 or the encoder / decoder module 6030) may be used for one or more of these functions. The external memory may be a memory 6020 and / or a storage device 6040, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory may be used to store an operating system for the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM may be used as working memory for video encoding and decoding operations such as for MPEG-2 Part 2 (also known as ITU-T Recommendation H.262 and ISO / IEC 13818-2, also known as MPEG-2 Video), HEVC (High Efficiency Video Coding), or VVC (Versatile Video Coding).
[0150] Input to the elements of system 6000 may be provided through various input devices as shown in block 6130. Such input devices include, but are not limited to, (i) an RF section that may receive an RF signal transmitted over the air, for example, by a broadcaster; (ii) a composite input terminal; (iii) a USB input terminal; and / or (iv) an HDMI input terminal.
[0151] In various embodiments, the input device of block 6130 may have associated various input processing elements as known in the art. For example, the RF portion may be associated with the following required elements: (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band); (ii) down-converting the selected signal; (iii) band-limiting again to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments; (iv) demodulating the down-converted and band-limited signal; (v) performing error correction; and (vi) demultiplexing to select the desired data packet stream. The RF portion of various embodiments may include one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to baseband) or baseband.
[0152] In one set-top box embodiment, the RF section and its associated input processing elements can receive RF signals transmitted via a wired (e.g., cable) medium. The RF section can then perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band.
[0153] Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions.
[0154] Added elements may include inserted elements between existing elements, such as, for example, inserting amplifiers and analog-to-digital converters.In various embodiments, the RF portion may include an antenna.
[0155] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 6000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, in a separate input processing IC or in the processor 6010 as needed. Similarly, aspects of USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 6010 as needed. The demodulated, error-corrected, and demultiplexed streams may be provided to various processing elements, including, for example, the processor 6010, and an encoder / decoder 6030 operating in conjunction with memory and storage elements, to process the data stream as needed for presentation on an output device.
[0156] The various components of the system 6000 may be arranged in an integrated housing. In the integrated housing, the various components may be interconnected using suitable connection means 6140 and transmit data between them, such as an internal bus known in the art, including an I2C bus, wiring, and a printed circuit board.
[0157] The system 6000 may include a communication interface 6050 that enables communication with other devices via a communication channel 6060. The communication interface 6050 may include, but is not limited to, a transceiver configured to send and receive data via the communication channel 3060. The communication interface 6050 may include, but is not limited to, a modem or a network card, and the communication channel 6060 may be implemented, for example, within a wired and / or wireless medium.
[0158] In various embodiments, a Wi-Fi network such as IEEE 802.11 may be used to stream data to the system 6000. The Wi-Fi signals of these embodiments may be received via a communication channel 6060 suitable for Wi-Fi communications and a communication interface 6050. The communication channel 6060 of these embodiments may typically connect to an access point or router that provides access to external networks including the Internet to allow streaming applications and other over-the-top communications.
[0159] Other embodiments may provide streaming data to the system 6000 using a set top box that transmits the data via the HDMI connection of the input block 6130.
[0160] Yet another embodiment may use the RF connection of input block 6130 to provide streaming data to system 6000 .
[0161] It should be understood that signaling can be accomplished in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. can be used to signal information to the corresponding decoder.
[0162] The system 6000 may provide output signals to various output devices including a display 6100, speakers 6110, and other peripheral devices 6120. In various examples of embodiments, the other peripheral devices 6120 may include one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide functionality based on the output of the system 3000.
[0163] In various embodiments, control signals may be communicated between the system 3000 and the display 6100, speakers 6110, or other peripheral devices 6120 using signaling such as AV.Link (Audio / Video Link), CEC (Consumer Electronics Control), or other communication protocols that enable device-to-device control with or without user intervention.
[0164] Output devices may be communicatively coupled to the system 6000 through respective interfaces 6070 , 6080 , and 6090 via dedicated connections.
[0165] Alternatively, the output device may be connected to the system 3000 using the communication channel 6060 via the communication interface 6050. In an electronic device such as, for example, a television, the display 6100 and the speaker 6110 may be integrated with the other components of the system 6000 in a single unit.
[0166] In various embodiments, the display interface 6070 may include a display driver, such as, for example, a timing controller (TCon) chip.
[0167] For example, if the RF portion of input 6130 is part of a separate set-top box, the display 6100 and speaker 6110 may alternatively be separate from one or more other components. In various embodiments where the display 6100 and speaker 6110 may be external components, the output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.
[0168] Figure 7 A schematic block diagram of sub-steps of steps 3200 , 3300 , and 3500 according to at least one of the present embodiments is illustrated.
[0169] As mentioned above Figure 3 As explained, there is usually more than one point along the same projection direction (line).In order to minimize the reconstruction error, a first and a second depth image are generated.
[0170] According to an embodiment of step 3300, in sub-step 3310, the geometric shape image generator GI can be suitable for generating a first depth image GI0 storing a lower depth value D0 and a second depth image GI1 storing a deeper depth value D1, wherein the depth value D0 is a depth value associated with a 2D sample with a lower depth (a point in the point cloud that is closer to the projection surface), and the depth value D1 is a depth value associated with a 2D sample with a deeper depth (a point in the point cloud that is farther from the projection surface).
[0171] According to an embodiment of step 3300, in sub-step 3310, the geometric shape image generator GI can be suitable for generating a first depth image GI0 storing a minimum depth value D0 and a second depth image GI1 storing a maximum depth value D1, wherein the minimum depth value D0 is the depth value associated with the 2D sample having the lowest depth, and the maximum depth value D1 is the depth value associated with the 2D sample having the deepest depth.
[0172] According to a variation of sub-step 3310 , the first depth image GI0 may store the maximum depth value D1 , ie the depth value associated with the 2D sample with the deepest depth, and the second depth image GI1 may store the minimum depth value D0 , ie the depth value associated with the 2D sample with the lowest depth.
[0173] If the depth image does not store the difference of the depth values but the absolute depth values, the storage of obvious contour-like high-frequency features as in the prior art is avoided. Therefore, the first and second depth images are better encoded by the conventional video encoder, and therefore, the depth values are well reconstructed from the decoded second depth image, which makes the quality of the geometry of the reconstructed point cloud frame high.
[0174] According to an embodiment of step 3300, in sub-step 3320, the geometric shape image generator GIG may be adapted to apply a filling process to the first and second depth images to fill in the blank space between at least one generated patch, thereby generating smooth first and second depth images.
[0175] For example, the image padding examples provided in Sections 2.2.6 and 2.2.7 of V-PCC may be used.
[0176] According to the embodiment of step 3500, in sub-step 3510, when the first and second filled depth images GI01 and GI11 can be generated from the first input point cloud frame IPCF1, and the first and second filled depth images GI02 and GI12 can be generated from the second input point cloud frame IPCF2 ( Figure 7 When the output of step 3300 is obtained, the encoder VENC can be adapted to time-interleave the padded first depth image and the second depth image into a video stream.
[0177] For example, the time interleaving may be adapted to generate a video stream by first considering the first and second filled depth images generated from the first input point cloud frame IPCF1, and then considering the first and second filled images: GI01, GI11, GI02 and GI12 generated from the second input point cloud frame IPFC2.
[0178] This example of time interleaving is not limiting, and the reader will understand that such time interleaving may be applied to more than two input point cloud frames.
[0179] In a variation, instead of time-interleaving the first and second padded depth images in the same bitstream, the first padded depth image may be output in a substream and the second padded depth image may be output in another substream.
[0180] According to an embodiment of step 3500, in sub-step 3520, the video encoder VENC may be adapted to encode the video stream by using a prediction of a first (respectively second) filled depth image generated from a current input point cloud frame by a first (respectively second) filled depth image generated from a previously encoded input point cloud frame.
[0181] According to an embodiment of step 3500 , in sub-step 3520 , the second filled depth image generated from the input point cloud frame may be predicted by the first filled depth image generated from the previously encoded input point cloud frame.
[0182] According to an embodiment of step 3500, in sub-step 3520, the first filled depth image generated from the current input point cloud frame can be predicted by the first filled depth image generated from the previously encoded input point cloud frame, and the second filled depth image generated from the current input point cloud frame can be predicted by the first filled depth image generated from the current input point cloud frame.
[0183] Figure 8 A schematic block diagram of sub-steps of steps 4200 and 4500 according to at least one of the present embodiments is illustrated.
[0184] In step 4200, the video decoder VDEC decodes the video stream to provide a decoded video stream, the output of the demultiplexer DMUX (step 4100), and derives at least one decoded geometry image DGI from said decoded video stream.
[0185] According to an embodiment of step 4200, in sub-step 4210, decoding the video stream may use a first (respectively second) depth image relative to a current input point cloud frame by predicting a first (respectively second) depth image relative to a previously decoded input point cloud frame.
[0186] According to an embodiment of step 4200 , in sub-step 4210 , a second depth image relative to the input point cloud frame may be predicted by a first depth image relative to a previously decoded input point cloud frame.
[0187] According to an embodiment of step 4200, in sub-step 4210, a first depth image relative to a current input point cloud frame may be predicted by a first depth image relative to a previously decoded input point cloud frame, and a second depth image relative to the current input point cloud frame may be predicted by a first depth image relative to the current input point cloud frame.
[0188] According to an embodiment of step 4200, in sub-step 4220, when the first and second decoded depth images DGI01 and DGI11 can be derived from the decoded video stream and the first and second decoded depth images DGI02 and DGI12 can be derived from the decoded video stream, the decoder VDEC can be adapted to access the first and second decoded depth images by time deinterleaving.
[0189] According to an embodiment of step 4200, in sub-step 4220, the time deinterlacing may be suitable for accessing a decoded video stream representing the first and second decoded depth images DGI01 and DGI11 and the first and second decoded depth images DGI02 and DGI12, wherein the first and second decoded depth images DGI01 and DGI11 represent depth values of 3D samples representing the first input point cloud frame IPCF1, and the first and second decoded depth images DGI02 and DGI12 represent depth values of 3D samples representing the second input point cloud frame IPCF2.
[0190] For example, the time deinterlacing may be adapted to first access information of a decoded video stream representing the first and second decoded depth images (representing depth values of 3D samples representing the first input point cloud frame IPCF1), and next, access information of a decoded video stream representing the first and second decoded images generated from the second input point cloud frame IPFC2: DGI01, DGI11, DGI02, and DGI12.
[0191] This example of temporal deinterlacing is not limiting, and the reader will understand that such temporal deinterlacing can be applied to more than two input point cloud frames.
[0192] According to an embodiment of step 4200, the first decoded depth image DGI0 may store information representing lower depth values, i.e., information representing depth values associated with 2D samples having lower depths, and the second decoded depth image DGI1 may store information representing deeper depth values, i.e., depth values associated with 2D samples having deeper depths.
[0193] According to an embodiment of step 4200, the first decoded depth image DGI0 may store information representing a minimum depth value, i.e., information representing a depth value associated with a 2D sample having the lowest depth, and the second decoded depth image DGI1 may store information representing a maximum depth value, i.e., a depth value associated with a 2D sample having the deepest depth.
[0194] According to a variant, the first decoded depth image DGI0 may store information representing deeper depth values, i.e., information representing depth values associated with 2D samples having deeper depths, and the second decoded depth image DGI1 may store information representing lower depth values, i.e., depth values associated with 2D samples having lower depths.
[0195] According to a variant, the first decoded depth image DGI0 may store information representing a maximum depth value, i.e., information representing a depth value associated with a 2D sample having the deepest depth, and the second decoded depth image DGI1 may store information representing a minimum depth value, i.e., a depth value associated with a 2D sample having the lowest depth.
[0196] According to an embodiment of sub-step 4510, the generation module GGM may be adapted to filter one of said at least one decoded depth image DGI.
[0197] Filtering the at least one decoded depth image may improve the reconstruction quality of the 3D coordinates of the reconstructed 3D samples.
[0198] According to an embodiment of sub-step 4510, filtering the decoded depth image uses pixel values of another depth image.
[0199] According to an embodiment of step 4500, in sub-step 4510, the first decoded depth image DGI0 stores a minimum depth value and the second decoded depth image DGI1 stores a maximum depth value, and the generation module GGM may be adapted to filter the second depth image DGI1. Next, in sub-step 4520, the generation module GGM may derive the 3D coordinates of the reconstructed 3D sample from the coordinates of the non-empty pixels, the values of the non-empty pixels of the first decoded depth image DGI0 and the second filtered decoded depth image DG11, the decoded auxiliary patch information, and possibly from additional metadata.
[0200] Filtering the pixel values of the decoded second depth image ensures that the difference between the pixel values of the first decoded depth image DGI0 and the pixel values of the second filtered decoded depth image DGI1 is within a given range.
[0201] According to an embodiment, the given range extends from 0 to a value representing the surface thickness ST.
[0202] The surface thickness is the maximum value of the difference between the pixel value of the depth image DI0 and the surface thickness allowed on the encoder side, which is an assumption used to calculate the depths D0 and D1 of the same pixel (u, v) on the encoder side.
[0203] According to an embodiment of sub-step 4510, the pixel value of the pixel at the coordinate (u, v) of the second filtered decoded depth image Given by:
[0204]
[0205] in, Accordingly are pixel values of the first decoded depth image DGI0, respectively the second decoded depth image DGI1. Note that if condition a) is not satisfied, condition b) is checked, and if condition b) is not satisfied, condition c) is checked.
[0206] According to the described embodiment, filtering the pixel values comprises clipping the difference between the pixel values of the first and second decoded depth images according to a given range extending from 0 to the surface thickness ST.
[0207] Such filtering is of low complexity.
[0208] According to an embodiment of sub-step 4510, the pixel value of the pixel at the coordinate (u, v) of the second filtered decoded depth image Given by:
[0209]
[0210] in, and
[0211]
[0212] According to the described embodiment, filtering the pixel values is based on linearly mapping the difference of the pixel values of the first and second decoded depth images to a given range extending from 0 to the surface thickness ST.
[0213] This filtering improves the quality of the 3D reconstructed 3D samples compared to previous embodiments, but increases complexity.
[0214] According to an embodiment, at least one boundary of a given range is transmitted.
[0215] Preferably, only the upper boundary of a given range is transmitted.
[0216] There are several ways to transmit (signal) the upper boundary (ST) into the bitstream.
[0217] According to one embodiment, the upper boundary (ST) is signaled by a given number of bits (eg, 3 bits) indicating the value of the upper boundary (ST).
[0218] According to another embodiment, the upper boundary (ST) is signaled by using:
[0219] 1 bit, which indicates whether filtering is used (sub-step 4510). For example, 1 means filtering is used, while 0 means otherwise.
[0220] If the first bit is equal to 1, it is followed by a given number of bits (eg 3 bits) to indicate the value of the upper boundary ST.
[0221] According to an embodiment, it is possible, for example, to Figure 5 The upper boundary ST is signaled once for the entire bitstream in the header BSH of the frame stream group, or multiple times when geometry or texture images are intra-coded, for example in the header HS of the frame stream group GOFS, or in the syntax elements GVS or TVS (for example, as metadata).
[0222] Signaling the upper boundary ST multiple times allows selecting different surface thickness values for each frame or each group of frames.
[0223] exist Figure 1-Figure 8 In the present invention, various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for the correct operation of the method, the order and / or use of specific steps and / or actions can be modified or combined.
[0224] Some examples are described with reference to block diagrams and operational flow charts. Each box represents a circuit element, a module, or a portion of code including one or more executable instructions for implementing a specified logical function. It should also be noted that in other implementations, the functions marked in the boxes may not occur in the order shown. For example, depending on the functions involved, two boxes shown in succession may actually be executed substantially simultaneously, or the boxes may sometimes be executed in reverse order.
[0225] The implementations and aspects described herein may be implemented, for example, in a method or process, an apparatus, a computer program, a data stream, a bit stream, or a signal. Even if only discussed above in a single form of implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a computer program).
[0226] The method may be implemented in, for example, a processor, which refers to a processing device generally including, for example, a computer, a microprocessor, an integrated circuit or a programmable logic device. The processor also includes a communication device.
[0227] In addition, the method can be implemented by instructions executed by a processor, and such instructions (and / or data values generated by the implementation) can be stored on a computer-readable storage medium. The computer-readable storage medium can take the form of a computer-readable program product embodied in one or more computer-readable media and having a computer-readable program code embodied thereon and executable by a computer. In view of the inherent ability to store information therein and to provide the inherent ability to retrieve information therefrom, the computer-readable storage medium used herein can be considered to be a non-transitory storage medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or apparatus, or any suitable combination of the foregoing. It should be understood that although more specific examples of computer-readable storage media to which the present embodiment can be applied are provided below, they are only exemplary and non-exhaustive lists that are easily understood by those of ordinary skill in the art: portable computer floppy disk; hard disk; read-only memory (ROM); erasable programmable read-only memory (EPROM or flash memory); portable compact disk read-only memory (CD-ROM); optical storage device; magnetic storage device; or any suitable combination of the foregoing.
[0228] The instructions may form an application program tangibly embodied on a processor-readable medium.
[0229] Instructions may be in, for example, hardware, firmware, software, or a combination. Instructions may be found in, for example, an operating system, a separate application, or a combination of both. Thus, a processor may be characterized as, for example, a device configured to perform a process and a device including a processor-readable medium (such as a storage device) having instructions for implementing the process. Furthermore, in addition to or in lieu of instructions, a processor-readable medium may store data values generated by an implementation.
[0230] The device can be implemented, for example, with appropriate hardware, software, and firmware. Examples of such equipment include personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, networked home appliances, head-mounted display devices (HMD, perspective glasses), projectors (projectors), "cave" (systems including multiple displays), servers, video encoders, video decoders, post-processors for processing outputs from video decoders, pre-processors for providing inputs to video encoders, network servers, set-top boxes, and any other devices or other communication devices for processing point clouds, videos, or images. It should be clear that the equipment can be mobile and can even be installed in a mobile vehicle.
[0231] The computer software may be implemented by the processor 6010 or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may also be implemented by one or more integrated circuits. The memory 6020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as a non-limiting example, optical storage devices, magnetic storage devices, semiconductor-based storage devices, fixed memory, and removable memory. The processor 6010 may be of any type suitable for the technical environment and may include, as a non-limiting example, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.
[0232] It will be apparent to one of ordinary skill in the art that implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing the method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, a signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.
[0233] The terms used herein are only for the purpose of describing specific embodiments and are not intended to be limiting. As used herein, the singular forms of "a", "an" and "the" may also be intended to include plural forms unless the context clearly indicates otherwise. It will also be understood that when used in this specification, the terms "includes / comprises" and / or "including / comprising" may specify the presence of stated, for example, features, integers, steps, operations, elements and / or components, but do not exclude the presence or increase of one or more other features, integers, steps, operations, elements, components and / or groups thereof. In addition, when an element is referred to as "responsive" or "connected" to another element, it may directly respond to or be connected to another element, or there may be an intermediate element. On the contrary, when an element is referred to as "directly responding" or "directly connected" to other elements, there is no intermediate element.
[0234] It should be understood that the use of any symbol / term " / ", "and / or", and "at least one of", for example, in the context of "A / B", "A and / or B", and "at least one of A and B", may be intended to include selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the context of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A and B and C). As will be apparent to one of ordinary skill in this and related arts, this may be extended to as many items as listed.
[0235] Various numerical values may be used in the present application. Specific values may be for example purposes, and the described aspects are not limited to these specific values.
[0236] It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements are not limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the teachings of the present application, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. There is no implied ordering between the first element and the second element.
[0237] References to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof are often used to convey that a particular feature, structure, characteristic, etc. (described in connection with the embodiment / implementation) is included in at least one embodiment / implementation. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations thereof in various places throughout this application are not necessarily all referring to the same embodiment.
[0238] Similarly, references to "according to an embodiment / example / implementation" or "in an embodiment / example / implementation" and other variations thereof are often used herein to convey that a particular feature, structure, or characteristic (described in conjunction with an embodiment / example / implementation) may be included in at least one embodiment / example / implementation. Therefore, the appearance of the expression "according to an embodiment / example / implementation" or "in an embodiment / example / implementation" in various places in the specification does not necessarily all refer to the same embodiment / example / implementation, nor do separate or alternative embodiments / examples / implementations necessarily exclude other embodiments / examples / implementations.
[0239] Reference numerals appearing in the claims are illustrative only and have no limiting effect on the scope of the claims.The present embodiments / examples and variations may be employed in any combination or sub-combination although not explicitly described.
[0240] When the drawings are presented as flow charts, it should be understood that it also provides a block diagram of the corresponding device. Similarly, when the drawings are presented as block diagrams, it should be understood that it also provides a flow chart of the corresponding method / process.
[0241] Although some figures include arrows on communication paths to illustrate a primary direction of communication, it should be understood that communication can occur in the opposite direction to the drawn arrows.
[0242] Various implementations involve decoding. "Decoding" as used in this application can encompass all or part of the processing performed on a received point cloud frame, for example, to produce a final output suitable for display. In various embodiments, such processing includes one or more processes typically performed by an image-based decoder. In various embodiments, such processing also includes or alternatively includes processing performed by a decoder of the various implementations described in this application.
[0243] As another example, in one embodiment, "decoding" may refer only to entropy decoding; in another embodiment, "decoding" may refer only to differential decoding; and in another embodiment, "decoding" may refer to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" may be intended to refer specifically to a subset of operations or broadly to a broader decoding process will be apparent and believed to be well understood by those skilled in the art based on the context of the particular description.
[0244] Various implementations involve encoding. In a manner similar to the above discussion about "decoding", "encoding" as used in this application may include, for example, all or part of the processing performed on an input point cloud frame to produce an encoded bitstream. In various embodiments, such processing includes one or more processes typically performed by an image-based decoder. In various embodiments, such processing also includes or alternatively includes processing performed by an encoder of the various implementations described in this application.
[0245] As another example, in one embodiment, "encoding" may refer only to entropy encoding; in another embodiment, "encoding" may refer only to differential encoding; and in another embodiment, "encoding" may refer to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" may be intended to refer specifically to a subset of operations or broadly to a broader encoding process will be apparent, and believed to be well understood by those skilled in the art, based on the context of the particular description.
[0246] Note that the syntax elements used herein, such as BSH, GVS or TVS are descriptive terms. Therefore, they do not exclude the use of other syntax element names.
[0247] Various embodiments relate to rate-distortion optimization. Specifically, during the encoding process, a balance or trade-off between rate and distortion is generally considered, usually in view of constraints on computational complexity. Rate-distortion optimization can generally be formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different methods for solving the rate-distortion optimization problem. For example, the method can be based on extensive testing of all coding options, including all considered modes or coding parameter values, and a complete evaluation of its coding cost and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to save coding complexity, especially calculating approximate distortion based on prediction or prediction residual signals rather than reconstructed signals. A mixture of these two methods can also be used, such as using approximate distortion only for some of the possible coding options and using complete distortion for other coding options. Other methods only evaluate a subset of possible coding options. More generally, many methods use any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of both coding cost and associated distortion.
[0248] Additionally, the present application may refer to “determining” various information. Determining information may include, for example, one or more of the following: estimating information, calculating information, predicting information, or retrieving information from a memory.
[0249] Furthermore, the present application may refer to "accessing" various information. Accessing information may include, for example, one or more of: receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.
[0250] Additionally, the present application may involve "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of: accessing information, or retrieving information (e.g., from a memory). Additionally, for example, during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved in one way or another.
[0251] In addition, as used herein, the word "signaling" refers to indicating something to a corresponding decoder, etc. For example, in some embodiments, the encoder signals specific data representing surface thickness, which defines the maximum difference between the depth values of two points in the point cloud projected on the projection plane along the same projection line. In this way, in an embodiment, the same parameters can be used on the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit savings are achieved in various embodiments. It should be understood that signaling can be done in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signal" is mentioned above, the word "signal" can also be used as a noun in this article.
[0252] A plurality of implementations have been described. However, it will be appreciated that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. In addition, it will be appreciated by those of ordinary skill in the art that other structures and processes may replace the disclosed structures and processes, and that the resulting implementations will perform at least substantially the same functions as the disclosed implementations in at least substantially the same manner to achieve at least substantially the same results. Therefore, these and other implementations are contemplated by the present application.
Claims
1. A method comprising: Receiving a video stream comprising a first depth image interleaved with a second depth image, wherein the first depth image represents a first set of depth values for a first point associated with a point cloud frame, the first point being a point of the point cloud that is closer to a projection plane, and wherein the second depth image represents a second set of depth values for a second point associated with the point cloud frame, the second point being a point of the point cloud that is farther from the projection plane; and The pixel values of the second depth image are filtered by using the pixel values of the first depth image.
2. The method according to claim 1, wherein: Filtering the pixel values of the second depth image is based on a difference between the pixel values of the first depth image and the pixel values of the second depth image.
3. The method according to claim 2, wherein: Filtering the pixel values comprises clipping of the differences.
4. The method according to claim 3, wherein: The clipping is based on a given range.
5. The method according to claim 4, wherein: The given range extends from 0 to the surface thickness.
6. The method according to claim 4, wherein: At least one boundary of the given range is transmitted.
7. The method according to claim 1, further comprising: The first set of depth values of the first depth image and the second set of depth values of the second depth image associated with the point cloud frame are de-interlaced.
8. A computer program product comprising instructions stored on a non-transitory computer readable medium, which, when the program is executed by one or more processors, cause the one or more processors to perform the method according to claim 1.
9. A non-transitory computer-readable medium comprising instructions for causing one or more processors to perform the method according to claim 1.
10. An apparatus comprising: The processor is configured as: Receiving a video stream comprising a first depth image interleaved with a second depth image, wherein the first depth image represents a first set of depth values for a first point associated with a point cloud frame, the first point being a point of the point cloud that is closer to a projection plane, and wherein the second depth image represents a second set of depth values for a second point associated with the point cloud frame, the second point being a point of the point cloud that is farther from the projection plane; and The pixel values of the second depth image are filtered by using the pixel values of the first depth image.
11. The device according to claim 10, wherein: Filtering the pixel values of the second depth image is based on a difference between the pixel values of the first depth image and the pixel values of the second depth image.
12. The device according to claim 11, wherein Filtering the pixel values comprises clipping of the differences.
13. The device according to claim 12, wherein: The clipping is based on a given range.
14. The device according to claim 13, wherein: The given range extends from 0 to the surface thickness.
15. The device according to claim 13, wherein: At least one boundary of the given range is transmitted.
16. The device according to claim 10, wherein: The processor is also configured to: The first set of depth values of the first depth image and the second set of depth values of the second depth image associated with the point cloud frame are de-interlaced.
