Scaling parameters for V-PCC

Abstract

A video decoding mechanism is disclosed. The mechanism includes receiving a codestream comprising a plurality of two-dimensional (2D) slices and three-dimensional (3D) bounding box ratios in an atlas frame, decoding the 2D slices, reconstructing a point cloud by converting the 2D slices into a 3D slice coordinate system defined by each projection plane of the 3D bounding box, and applying the 3D bounding box ratios to the 3D bounding box.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 969,411, filed on February 3, 2020, by Vladyslav Zakharchenko et al., entitled “Signaling Physical Dimensions of the Point Cloud Sample,” and U.S. Provisional Patent Application No. 62 / 961,028, filed on January 14, 2020, by Vladyslav Zakharchenko et al., entitled “Signaling Physical Dimensions of the Point Cloud Sample,” each of which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to video coding, and more particularly to improvements that support scaling of coded point clouds based on associated physical size in video point cloud coding (V-PCC). Background Art

[0004] Even relatively short videos require a large amount of video data to describe them, which can create difficulties when the data is to be streamed or otherwise transmitted across communications networks with limited bandwidth capacity. Therefore, video data is often compressed before being transmitted over modern telecommunications networks. Since memory resources may be limited, the size of the video can also be an issue when storing the video on a storage device. Video compression devices typically use software and / or hardware to encode the video data at the source side before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination side by a video decompression device that decodes the video data. With limited network resources and the growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase the compression ratio with little impact on image quality. Summary of the Invention

[0005] In one embodiment, the present invention includes a method implemented by a decoder, the method comprising: a receiver of the decoder receiving a code stream including a plurality of two-dimensional (2D) slices and three-dimensional (3D) bounding box ratios in an atlas frame; a processor of the decoder decoding the 2D slices; reconstructing a point cloud by converting the 2D slices into a 3D slice coordinate system defined by each projection plane of a 3D bounding box by the processor; and applying the 3D bounding box ratios to the 3D bounding box by the processor.

[0006] Point cloud coding (PCC) systems encode point clouds by projecting points onto the faces of a 3D bounding box. This allows data to be converted between an encodable 2D format and a displayable 3D format, and vice versa. For example, encoding a point cloud can include projecting a 2D representation onto each face of a 3D bounding box to obtain a set of encodable 2D slices. Furthermore, reconstructing the point cloud can include projecting 2D slices from the 3D bounding box faces onto points (based on geometric data) to reconstruct a 3D point cloud. 3D bounding boxes use a local coordinate system and a predetermined unit size. Therefore, different point clouds can be encoded to a common standard size. However, in some cases, point clouds should be scaled to real-world size. For example, a point cloud of an object can be encoded based on a 3D bounding box of a predetermined size. The point cloud can then be used in an augmented reality (AR) context. In AR, pre-recorded and / or computer-generated content is overlaid on live video captured by an end user. Due to the mismatch between the local coordinate system and the size of the object captured by the user, some video PCC systems may not be able to correctly integrate the point cloud into the video.

[0007] This example includes a 3D bounding box scale parameter, which can be encoded into the PCC bitstream. The 3D bounding box scale is a parameter used to transform the encoded sample positions of a point cloud from a local coordinate system to a final real-world coordinate system. This enables the reconstructed point cloud to be properly resized so that it can be overlaid onto the real-world video. The 3D bounding box scale is a factor based on the sample density within the 3D bounding box and the size of the 3D bounding box. The 3D bounding box scale can be expressed in meters. Furthermore, a flag can be encoded into the PCC bitstream to indicate whether the 3D bounding box scale is present. Upon receiving this flag, the decoder can read it and obtain the 3D bounding box scale based on it. The decoder can then scale the 3D bounding box, decode the slice, reconstruct the point cloud based on the 3D bounding box, and overlay the reconstructed point cloud onto the real-world video. Therefore, this example increases the functionality of the encoder and decoder by creating AR support. Furthermore, the present invention supports mechanisms to improve decoding efficiency, thereby reducing processor, memory, and / or network resource usage at the encoder and / or decoder.

[0008] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the code stream further includes a flag indicating whether 3D bounding box information exists.

[0009] Optionally, according to any of the above aspects, in another implementation of the aspect, the 3D bounding box scale is applied based on the value of the flag.

[0010] Optionally, according to any of the above aspects, in another implementation of the aspect, the 3D slice coordinate system is transformed into a target 3D coordinate system in meters.

[0011] Optionally, according to any of the above aspects, in another implementation of the aspect, the code stream further includes an occupancy component, a geometric component and an attribute component, and the point cloud is reconstructed based on the occupancy component, the geometric component and the attribute component.

[0012] Optionally, according to any of the above aspects, in another implementation of the aspect, the occupancy component informs the decoder which samples in the 2D component are associated with video data, the geometry component includes information representing the position of the samples in 3D space, and the attribute component provides characteristics of the samples.

[0013] Optionally, according to any of the above aspects, in another implementation of the aspect, the processor is further included in forwarding the reconstructed atlas frame for display.

[0014] In one embodiment, the present invention includes a method implemented by an encoder, the method comprising: a processor of the encoder using orthogonal projection to a 3D bounding box to generate a plurality of 2D slices representing a point cloud; the processor encoding the 2D slices into atlas frames in a bitstream; the processor determining a 3D bounding box scale to be applied to the 3D bounding box; the processor encoding the 3D bounding box scale into the bitstream; and a memory of the encoder storing the bitstream for transmission to a decoder.

[0015] PCC systems encode point clouds by projecting points onto the faces of a 3D bounding box. This allows data to be converted between an encodable 2D format and a displayable 3D format, and vice versa. For example, encoding a point cloud may include projecting a 2D representation onto each face of a 3D bounding box to obtain a set of encodable 2D slices. Furthermore, reconstructing the point cloud may include projecting 2D slices from the 3D bounding box faces onto points (based on geometric data) to reconstruct a 3D point cloud. 3D bounding boxes use a local coordinate system and a predetermined unit size. Therefore, different point clouds can be encoded to a common standard size. However, in some cases, point clouds should be scaled to real-world size. For example, a point cloud of an object can be encoded based on a 3D bounding box of a predetermined size. The point cloud can then be used in an AR context. In AR, pre-recorded and / or computer-generated content is overlaid on a live video captured by an end user. Due to the mismatch between the local coordinate system and the size of the object captured by the user, some video PCC systems may not be able to correctly integrate the point cloud into the video.

[0016] This example includes a 3D bounding box scale parameter, which can be encoded into the PCC bitstream. The 3D bounding box scale is a parameter used to transform the encoded sample positions of a point cloud from a local coordinate system to a final real-world coordinate system. This enables the reconstructed point cloud to be properly resized so that it can be overlaid onto the real-world video. The 3D bounding box scale is a factor based on the sample density within the 3D bounding box and the size of the 3D bounding box. The 3D bounding box scale can be expressed in meters. Furthermore, a flag can be encoded into the PCC bitstream to indicate whether the 3D bounding box scale is present. Upon receiving this flag, the decoder can read it and obtain the 3D bounding box scale based on it. The decoder can then scale the 3D bounding box, decode the slice, reconstruct the point cloud based on the 3D bounding box, and overlay the reconstructed point cloud onto the real-world video. Therefore, this example increases the functionality of the encoder and decoder by creating AR support. Furthermore, the present invention supports mechanisms to improve decoding efficiency, thereby reducing processor, memory, and / or network resource usage at the encoder and / or decoder.

[0017] Optionally, according to any of the above aspects, in another implementation of the aspect, the 3D bounding box scale includes parameters for rescaling samples in the 3D bounding box from a local coordinate system to a real-world size.

[0018] Optionally, according to any of the above aspects, in another implementation of the aspect, the processor is further included encoding a flag indicating whether 3D bounding box information exists into the bitstream.

[0019] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the value of the flag indicates whether the 3D bounding box ratio is applied to the 3D bounding box.

[0020] Optionally, according to any of the above aspects, in another implementation of the aspect, the 3D bounding box is scaled in meters.

[0021] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the processor is further included encoding the occupancy component, the geometry component and the attribute component into the code stream.

[0022] Optionally, according to any of the above aspects, in another implementation of the aspect, the occupancy component informs the decoder which samples in the atlas frame are associated with video data, the geometry component includes information representing the position of the samples in 3D space, and the attribute component provides characteristics of the samples.

[0023] In one embodiment, the present invention includes a video decoding device, which includes: a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, the receiver, the memory, and the transmitter are used to execute the method according to any one of the above aspects.

[0024] In one embodiment, the present invention includes a non-transitory computer-readable medium including a computer program product for use in a video decoding device, the computer program product including computer-executable instructions stored in the non-transitory computer-readable medium. When a processor executes the computer-executable instructions, the video decoding device performs the method according to any one of the above aspects.

[0025] In one embodiment, the present invention includes a decoder comprising: a receiving module for receiving a codestream comprising a plurality of 2D slices and 3D bounding box ratios in an atlas frame; a decoding module for decoding the slices; a reconstruction module for reconstructing a point cloud by converting the slices into a 3D slice coordinate system defined by each projection plane of the 3D bounding box; and an application module for applying the 3D bounding box ratios to the 3D bounding box.

[0026] PCC systems encode point clouds by projecting points onto the faces of a 3D bounding box. This allows data to be converted between an encodable 2D format and a displayable 3D format, and vice versa. For example, encoding a point cloud may include projecting a 2D representation onto each face of a 3D bounding box to obtain a set of encodable 2D slices. Furthermore, reconstructing the point cloud may include projecting 2D slices from the 3D bounding box faces onto points (based on geometric data) to reconstruct a 3D point cloud. 3D bounding boxes use a local coordinate system and a predetermined unit size. Therefore, different point clouds can be encoded to a common standard size. However, in some cases, point clouds should be scaled to real-world size. For example, a point cloud of an object can be encoded based on a 3D bounding box of a predetermined size. The point cloud can then be used in an augmented reality (AR) context. In AR, pre-recorded and / or computer-generated content is overlaid on a live video captured by an end user. Because the local coordinate system does not match the size of the object captured by the user, some video PCC systems may not be able to correctly integrate the point cloud into the video.

[0027] This example includes a 3D bounding box scale parameter, which can be encoded into the PCC bitstream. The 3D bounding box scale is a parameter used to transform the encoded sample positions of a point cloud from a local coordinate system to a final real-world coordinate system. This enables the reconstructed point cloud to be properly resized so that it can be overlaid onto the real-world video. The 3D bounding box scale is a factor based on the sample density within the 3D bounding box and the size of the 3D bounding box. The 3D bounding box scale can be expressed in meters. Furthermore, a flag can be encoded into the PCC bitstream to indicate whether the 3D bounding box scale is present. Upon receiving this flag, the decoder can read it and obtain the 3D bounding box scale based on it. The decoder can then scale the 3D bounding box, decode the slice, reconstruct the point cloud based on the 3D bounding box, and overlay the reconstructed point cloud onto the real-world video. Therefore, this example increases the functionality of the encoder and decoder by creating AR support. Furthermore, the present invention supports mechanisms to improve decoding efficiency, thereby reducing processor, memory, and / or network resource usage at the encoder and / or decoder.

[0028] Optionally, according to any of the above aspects, in another implementation of the aspect, the decoder is further used to execute the method according to any of the above aspects.

[0029] In one embodiment, the present invention includes an encoder comprising: a using module for applying orthogonal projection to a 3D bounding box to generate a plurality of 2D slices representing a point cloud; a determining module for determining a 3D bounding box scale to be applied to the 3D bounding box; an encoding module for encoding the 2D slices into atlas frames in a bitstream; encoding the 3D bounding box scale into the bitstream; and a storage module for storing the bitstream for transmission to a decoder.

[0030] PCC systems encode point clouds by projecting points onto the faces of a 3D bounding box. This allows data to be converted between an encodable 2D format and a displayable 3D format, and vice versa. For example, encoding a point cloud may include projecting a 2D representation onto each face of a 3D bounding box to obtain a set of encodable 2D slices. Furthermore, reconstructing the point cloud may include projecting 2D slices from the 3D bounding box faces onto points (based on geometric data) to reconstruct a 3D point cloud. 3D bounding boxes use a local coordinate system and a predetermined unit size. Therefore, different point clouds can be encoded to a common standard size. However, in some cases, point clouds should be scaled to real-world size. For example, a point cloud of an object can be encoded based on a 3D bounding box of a predetermined size. The point cloud can then be used in an AR context. In AR, pre-recorded and / or computer-generated content is overlaid on a live video captured by an end user. Due to the mismatch between the local coordinate system and the size of the object captured by the user, some video PCC systems may not be able to correctly integrate the point cloud into the video.

[0031] This example includes a 3D bounding box scale parameter, which can be encoded into the PCC bitstream. The 3D bounding box scale is a parameter used to transform the encoded sample positions of a point cloud from a local coordinate system to a final real-world coordinate system. This enables the reconstructed point cloud to be properly resized so that it can be overlaid onto the real-world video. The 3D bounding box scale is a factor based on the sample density within the 3D bounding box and the size of the 3D bounding box. The 3D bounding box scale can be expressed in meters. Furthermore, a flag can be encoded into the PCC bitstream to indicate whether the 3D bounding box scale is present. Upon receiving this flag, the decoder can read it and obtain the 3D bounding box scale based on it. The decoder can then scale the 3D bounding box, decode the slice, reconstruct the point cloud based on the 3D bounding box, and overlay the reconstructed point cloud onto the real-world video. Therefore, this example increases the functionality of the encoder and decoder by creating AR support. Furthermore, the present invention supports mechanisms to improve decoding efficiency, thereby reducing processor, memory, and / or network resource usage at the encoder and / or decoder.

[0032] Optionally, according to any of the above aspects, in another implementation of the aspect, the encoder is further used to execute the method according to any of the above aspects.

[0033] In one embodiment, the present invention includes a method implemented by user equipment (UE), comprising: a central processing unit (CPU) decoding slices in atlas data and vertices in geometric data from a video point cloud coding (V-PCC) code stream; storing the slices and the vertices in one or more buffers; a GPU binding the slices and the vertices to one or more GPU shaders as textures; and the GPU shader rendering the slices onto the vertices through a transformation matrix to generate a 2D image of a 3D reconstructed point cloud.

[0034] There are a variety of rendering techniques for rendering video data. However, simply applying rendering techniques to PCC video data may result in an inefficient rendering process. For example, many types of video data include a single type of data that can be decoded, transferred from the CPU to the graphics processing unit (GPU), and rendered for display. However, PCC uses multiple types of video data that interact with each other. Therefore, some rendering systems can decode atlas video data, geometry video data, occupancy video data, and / or attribute video data and transfer the atlas video data, geometry video data, occupancy video data, and / or attribute video data from the CPU to the GPU for certain rendering tasks. The data is then transferred back to the CPU for combination. The result is then sent back to the GPU to complete the rendering before display. This movement between the CPU and GPU causes each rendered frame to be transferred multiple times between the CPU buffer and the GPU buffer, which significantly increases latency and resource usage during the rendering process.

[0035] This example includes a V-PCC renderer that is used to prevent repeated memory movement during the rendering process. The CPU can be used to decode atlas data (e.g., slices), occupancy data, geometry data (e.g., vertices), and / or attribute data. This data is stored in one or more buffers, such as vertex buffers and / or uniform buffers. GPU shaders can then bind to the data in the buffers to reconstruct a 3D point cloud and render a 2D image from the point cloud. For example, a GPU shader can apply occupied pixels as textures to remove slice data, geometry data, and / or attribute data that are artifacts from the compression process. Another GPU shader can initialize a model matrix and generate vertices in a 3D model from the geometry data. Yet another shader can project slice and / or attribute pixels onto vertices as textures. This can be performed without copying such data from the CPU buffers to the GPU buffers. Transformations can also be used to generate 2D images for display from the 3D point cloud. Textures (e.g., sub-blocks of slices) can be assigned to multiple shaders, which increases the parallel processing capabilities of the GPU. In some examples, an entire frame of a V-PCC image can be rendered with a single draw call to the GPU. This rendering can be performed quickly enough to enable most user devices to use V-PCC content in AR applications. Thus, the disclosed examples increase the capabilities of the decoder when decoding V-PCC data. Furthermore, the examples reduce the use of processor, memory, and / or GPU resources, thereby increasing the functionality of the decoder.

[0036] Optionally, according to any of the above aspects, in another implementation of the aspect, the GPU shader renders the slice on the vertex without copying the slice or vertex to GPU memory.

[0037] Optionally, according to any of the above aspects, in another implementation of the aspect, rendering the slices onto the vertices includes rendering all slices associated with a single V-PCC frame onto the vertices in parallel in response to a single draw call from the CPU.

[0038] Optionally, according to any of the above aspects, in another implementation of the aspect, rendering the slices in parallel includes obtaining a slice index of the slice from the CPU buffer, and determining a slice orientation of each slice in the slice based on the slice index.

[0039] Optionally, according to any of the above aspects, in another implementation of the aspect, it also includes: the CPU decoding occupied pixels from the V-PCC code stream; storing the occupied pixels in the buffer; the GPU binding the occupied pixels to the GPU shader as a texture; before rendering the slice onto the vertex, the GPU shader applies the occupied pixels to reject one or more points associated with the vertex.

[0040] Optionally, according to any of the above aspects, in another implementation of the aspect, the occupied pixels are applied to the slices and the vertices.

[0041] Optionally, according to any of the above aspects, in another implementation of the aspect, it also includes initializing a model matrix before rendering the slice onto the vertex to fit the vertex to a normalized unit square around the origin.

[0042] Optionally, according to any of the above aspects, in another implementation of the aspect, the GPU is further included in subdividing the slice into sub-blocks, wherein each sub-block is rendered by a different GPU shader.

[0043] Optionally, according to any of the above aspects, in another implementation of the aspect, the buffer includes a unified buffer that stores the slice in a cache memory.

[0044] Optionally, according to any of the above aspects, in another implementation of the aspect, the buffer includes a vertex buffer for storing the vertices.

[0045] Optionally, according to any of the above aspects, in another implementation of the aspect, the GPU shader includes a geometry shader, a compute shader, or a combination thereof.

[0046] Optionally, according to any of the above aspects, in another implementation of the aspect, the maximum number of vertices emitted by the geometry shader is equal to the occupied resolution multiplied by the number of layers.

[0047] Optionally, according to any of the above aspects, in another implementation of the aspect, rendering the slice onto the vertex through the transformation matrix includes rendering the slice and the vertex onto a recorded image to create augmented reality (AR) content.

[0048] Optionally, according to any of the above aspects, in another implementation of the aspect, the transformation matrix includes a canvas to piece matrix for transforming from atlas space to piece space, a model matrix for transforming from model space to world space, a camera view matrix for transforming from the world space to camera space, and a projection matrix for transforming from the camera space to screen space.

[0049] Optionally, according to any of the above aspects, in another implementation manner of the aspect, rendering the slice onto the vertex includes projecting the slice onto the point of the vertex.

[0050] Optionally, according to any of the above aspects, in another implementation of the aspect, it further includes forwarding the 2D image of the 3D point cloud to a screen.

[0051] For the sake of clarity, any of the embodiments described above may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the present invention.

[0052] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] For a more complete understanding of the present invention, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0054] Figure 1 is a flow chart of an exemplary method of decoding a video signal.

[0055] Figure 2 is a schematic diagram of an exemplary encoding and decoding (codec) system for video coding.

[0056] Figure 3 is a schematic diagram of an exemplary video encoder.

[0057] Figure 4 is a schematic diagram of an exemplary video decoder.

[0058] Figure 5 is an example of point cloud media that can be decoded according to the point cloud compression (PCC) mechanism.

[0059] Figure 6 is an example of a patch created from a point cloud.

[0060] Figure 7A An exemplary occupancy frame associated with a set of tiles is shown.

[0061] Figure 7B An exemplary geometric frame associated with a set of patches is shown.

[0062] Figure 7C An exemplary atlas frame associated with a set of slices is shown.

[0063] Figure 8 is an exemplary mechanism for scaling the reconstructed point cloud by scaling the corresponding 3D bounding box.

[0064] Figure 9 is a schematic diagram of an exemplary video PCC (V-PCC) code stream for scaling a point cloud.

[0065] Figure 10 is a schematic diagram of an exemplary video decoding apparatus.

[0066] Figure 11 is a flow chart of an exemplary method for encoding a PCC codestream to support scaling of a point cloud during reconstruction.

[0067] Figure 12 is a flowchart of an exemplary method for decoding a PCC code stream to reconstruct a scaled point cloud.

[0068] Figure 13 is a schematic diagram of an exemplary system for decoding a PCC code stream to support scaling of a point cloud during reconstruction.

[0069] Figure 14 is a schematic diagram of an exemplary data unit set in a V-PCC code stream.

[0070] Figure 15 is a schematic diagram of an exemplary V-PCC renderer.

[0071] Figure 16 is a flowchart of an exemplary method for rendering a V-PCC code stream. DETAILED DESCRIPTION

[0072] It should be understood at the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The present invention should in no way be limited to the illustrative embodiments, drawings, and techniques described below, including the exemplary designs and embodiments illustrated and described herein, but may be modified within the scope of the appended claims and their full scope of equivalents.

[0073] The following terms are defined as follows, unless used in a contrary context herein. Specifically, the following definitions are intended to more clearly describe the present invention. However, terms may be defined differently in different contexts. Therefore, the following definitions should be considered supplementary information and should not be construed as limiting any other definitions provided herein for these terms.

[0074] An encoder is a device used to compress video data into a bitstream through an encoding process. A decoder is a device used to reconstruct video data from the bitstream for display through a decoding process. A point cloud / point cloud representation is a set of points (e.g., samples) in a three-dimensional (3D) space, where each point can include a position, color, and / or one or more attributes. A 3D point cloud is a collection of data points. A bitstream is a series of bits that include video data that is compressed for transmission between an encoder and a decoder. In the context of point cloud coding (PCC), a bitstream includes a series of bits that encode a video point cloud compression (V-PCC) component and associated parameters.

[0075] A V-PCC component (or more generally a PCC component) can be a specific type of atlas data, occupancy data, geometry data, or attribute data associated with a V-PCC point cloud. An atlas / atlas frame can be a collection of two-dimensional (2D) bounding boxes (also called patches) projected into a rectangular frame, which correspond to volumes in 3D space (called 3D bounding boxes) on which the volume data is rendered. Each 2D bounding box / patch represents a subset of the point cloud. A patch is a 2D rectangular area in the atlas associated with volume information. A 3D bounding box is a volume defined as a cuboid entity with six rectangular faces placed at right angles. The 3D bounding box scale is a parameter that represents the scale to be applied to the 3D bounding box parameters that can be specified for an object. A 3D patch coordinate system is a mechanism that uses numbers to uniquely identify the locations to which samples of a patch correspond in 3D space. An occupied pixel is a value that indicates whether an atlas sample corresponds to an associated sample in 3D space. An occupancy frame can be a collection of occupancy values that constitute a 2D array and represent the entire occupancy information of a single atlas frame. Thus, the occupancy frame can indicate whether each sample position in the atlas corresponds to a valid 3D point in the point cloud representation. A geometry pixel is a set of Cartesian coordinates associated with a volume frame. A geometry map is a frame that includes information about a geometric piece projected at a specific depth. Attributes can be scalar or vector properties that are optionally associated with each point in the volume frame of a point cloud, such as color, reflectivity, surface normal, timestamp, material identifier (ID), etc. A volume frame is a set of 3D points specified by Cartesian coordinates and 0 or more corresponding attribute sets at a specific moment. The complete set of atlas data, occupancy pixels, geometry maps, or attribute pixels associated with a specific moment / volume frame can be referred to as an atlas frame, occupancy frame, geometry frame, and attribute frame, respectively. Atlas data, occupancy data, geometry data, or attribute data can be components of a point cloud and can therefore be referred to as an atlas component, occupancy component, geometry component, and attribute frame component, respectively. A flag is a value that indicates the data of an associated function or process.

[0076] Video point cloud coding (PCC) is a mechanism for efficiently encoding three-dimensional (3D) objects represented by 3D point clouds of different colors, lighting and / or other attributes. The point cloud and associated motion / changes over time are captured and included in PCC components. The PCC components are then encoded. The position of each valid point in the cloud at a certain moment is stored as a geometry map in a geometry frame. The color and brightness are stored as slices. Specifically, the slices at a certain moment are packed into atlas frames. Slices typically do not cover the entire atlas frame. Therefore, an occupancy frame is also generated, which indicates which parts of the atlas frame include slice data. Optionally, attributes of the points (such as transparency) can be included in the attribute frame. Therefore, each PCC frame can be encoded as multiple frames including different components describing the point cloud at the corresponding moment.

[0077] PCC systems encode point clouds by projecting points onto the faces of a 3D bounding box. This allows data to be converted between an encodable 2D format and a displayable 3D format, and vice versa. For example, encoding a point cloud may include projecting a 2D representation onto each face of a 3D bounding box to obtain a set of encodable 2D slices. Furthermore, reconstructing the point cloud may include projecting 2D slices from the 3D bounding box faces onto points (based on geometric data) to reconstruct a 3D point cloud. 3D bounding boxes use a local coordinate system and a predetermined unit size. Therefore, different point clouds can be encoded to a common standard size. However, in some cases, point clouds should be scaled to real-world size. For example, a point cloud of an object can be encoded based on a 3D bounding box of a predetermined size. The point cloud can then be used in an augmented reality (AR) context. In AR, pre-recorded and / or computer-generated content is overlaid on a live video captured by an end user. Because the local coordinate system does not match the size of the object captured by the user, some video PCC systems may not be able to correctly integrate the point cloud into the video.

[0078] This document discloses a 3D bounding box scale parameter that can be encoded into the PCC bitstream. The 3D bounding box scale is a parameter used to transform the encoded sample positions of a point cloud from a local coordinate system to a final real-world coordinate system. This enables the reconstructed point cloud to be properly resized so that it can be overlaid onto real-world video. The 3D bounding box scale is a factor based on the sample density within the 3D bounding box and the size of the 3D bounding box. The 3D bounding box scale can be expressed in meters. Furthermore, a flag can be encoded into the PCC bitstream to indicate whether the 3D bounding box scale is present. Upon receiving this flag, the decoder can read it and, based on it, obtain the 3D bounding box scale. The decoder can then scale the 3D bounding box, decode the slice, reconstruct the point cloud based on the 3D bounding box, and overlay the reconstructed point cloud onto the real-world video. Thus, this example increases the functionality of the encoder and decoder by creating AR support. Furthermore, the present invention supports mechanisms to improve decoding efficiency, thereby reducing processor, memory, and / or network resource usage at the encoder and / or decoder.

[0079] Figures 1 to 4 Describes various coding mechanisms for encoding and decoding video. Figures 5 to 8 Describes the method used to convert point clouds into Figures 1 to 4 The mechanisms for encoding and decoding the format are described in the .

[0080] Figure 1 The flowchart of an exemplary method 100 for decoding a video signal is shown. Specifically, a video signal is encoded at an encoder. The encoding process compresses the video signal using various mechanisms to reduce the video file size. The smaller file size allows the compressed video file to be sent to the user while reducing associated bandwidth overhead. The decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process typically corresponds to the encoding process, allowing the decoder to consistently reconstruct the video signal.

[0081] In step 101, a video signal is input into an encoder. For example, the video signal can be an uncompressed video file stored in a memory. For another example, the video file can be captured by a video capture device (e.g., a camera) and encoded to support live streaming of the video. The video file can include an audio component and a video component. The video component includes a series of image frames. When these image frames are viewed in sequence, they give the visual effect of motion. These frames include pixels represented by light, referred to herein as luminance components (or luminance samples), and pixels represented by color, referred to as chrominance components (or chrominance samples). In some examples, these frames can also include depth values to support three-dimensional viewing.

[0082] In step 103, the video is segmented into blocks. Segmentation involves subdividing the pixels in each frame into square blocks and / or rectangular blocks for compression. For example, in high efficiency video coding (HEVC) (also known as H.265 and MPEG-H Part 2), frames can be first divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64×64 pixels). These CTUs include luma samples and chroma samples. A coding tree can be used to divide the CTU into blocks, and then repeatedly subdivide these blocks until a configuration that supports further encoding is obtained. For example, the luma component of a frame can be subdivided until each block includes relatively uniform luma values. Additionally, the chroma component of a frame can be subdivided until each block includes relatively uniform color values. Thus, the segmentation mechanism varies depending on the content of the video frame.

[0083] In step 105, the image blocks segmented in step 103 are compressed using various compression mechanisms. For example, inter-frame prediction and / or intra-frame prediction can be used. Inter-frame prediction is designed to take advantage of the fact that objects in a typical scene often appear in consecutive frames. Therefore, blocks describing objects in a reference frame do not need to be repeated in adjacent frames. Specifically, an object (e.g., a table) can remain in a fixed position across multiple frames. Therefore, the table is described once, and adjacent frames can re-reference the reference frame. Pattern matching mechanisms can be used to match objects across multiple frames. In addition, moving objects can be represented across multiple frames due to, for example, object motion or camera movement. In one specific example, a video may show a car moving across the screen across multiple frames. Motion vectors can be used to describe this motion. A motion vector is a two-dimensional vector that provides the offset between the coordinates of an object in one frame and the coordinates of the object in a reference frame. Therefore, inter-frame prediction can encode image blocks in the current frame as a set of motion vectors, representing the offset between the image blocks in the current frame and the corresponding blocks in the reference frame.

[0084] Intra-frame prediction encodes blocks in a common frame. Intra-frame prediction exploits the fact that luma and chroma components tend to cluster within a frame. For example, a patch of green in one part of a tree is often adjacent to similar patches of green. Intra-frame prediction uses a variety of directional prediction modes (e.g., 33 in HEVC), planar mode, and direct current (DC) mode. These directional modes indicate that samples in the current block are similar or identical to samples in neighboring blocks in the corresponding direction. Planar mode indicates that a series of blocks in a row or column (e.g., a plane) can be interpolated based on neighboring blocks along the edge of that row. Planar mode effectively represents a smooth transition of light or color across rows or columns by using a relatively constant slope of varying values. DC mode is used for boundary smoothing and indicates that a block is similar or identical to the average of samples from all neighboring blocks relative to the angular direction of the directional prediction mode. Therefore, intra-frame prediction blocks can represent image blocks as various relative prediction mode values rather than actual values. Furthermore, inter-frame prediction blocks can represent image blocks as motion vector values rather than actual values. In either case, the prediction block may not accurately represent the image block in some cases. All differences are stored in residual blocks. Transformations can be applied to residual blocks to further compress the file.

[0085] In step 107, various filtering techniques can be applied. In HEVC, filters are applied according to an in-loop filtering scheme. The block-based prediction described above may produce blocky images at the decoder. In addition, a block-based prediction scheme can encode a block and then reconstruct the encoded block for subsequent use as a reference block. The in-loop filtering scheme iteratively applies a noise suppression filter, a deblocking filter, an adaptive loop filter, and a sample adaptive offset (SAO) filter to the block / frame. These filters reduce blocking artifacts so that the encoded file can be accurately reconstructed. In addition, these filters reduce artifacts in the reconstructed reference block, making it less likely that the artifacts will produce other artifacts in subsequent blocks encoded based on the reconstructed reference block.

[0086] Once the video signal has been segmented, compressed, and filtered, the resulting data is encoded into a bitstream in step 109. The bitstream includes the data described above as well as any indicative data necessary to support proper reconstruction of the video signal at the decoder. For example, this data may include segmentation data, prediction data, residual blocks, and various flags that provide decoding instructions to the decoder. The bitstream can be stored in memory so that it can be sent to the decoder upon request. The bitstream can also be broadcast and / or multicast to multiple decoders. Creating the bitstream is an iterative process. Therefore, steps 101, 103, 105, 107, and 109 can be performed continuously and / or simultaneously across multiple frames and blocks. Figure 1The shown order is presented for clarity and ease of description and is not intended to limit the video coding process to a specific order.

[0087] In step 111, the decoder receives the bitstream and begins the decoding process. Specifically, the decoder uses an entropy decoding scheme to convert the bitstream into corresponding syntax data and video data. In step 111, the decoder uses the syntax data in the bitstream to determine the frame segmentation. This segmentation should match the results of the block segmentation in step 103. The entropy encoding / decoding used in step 111 is described below. The encoder makes many choices during the compression process, for example, selecting a block segmentation scheme from several possible options based on the spatial positioning of values in one or more input images. Indicating the exact choice may use a large number of bits (bins). As used herein, a "bit" is a variable binary value (e.g., a bit value that may vary depending on the content). Entropy encoding causes the encoder to discard any options that are clearly unsuitable for a particular situation, leaving a set of available options. A codeword is then assigned to each available option. The length of the codeword depends on the number of available options (e.g., one binary symbol for two options, two binary symbols for three to four options, and so on). The encoder then encodes the codeword for the selected option. This approach reduces the codeword size because the codeword is as large as expected, uniquely indicating a selection from a small subset of permissible options rather than a selection from a potentially large set of all possible options. The decoder then decodes the selection by determining the set of permissible options in a similar manner to the encoder. By determining the set of permissible options, the decoder can read the codeword and determine the selection made by the encoder.

[0088] In step 113, the decoder performs block decoding. Specifically, the decoder uses an inverse transform to generate a residual block. The decoder then uses the residual block and the corresponding prediction block to reconstruct the image block based on the segmentation. The prediction block may include the intra-frame prediction block and the inter-frame prediction block generated by the encoder in step 105. Next, the reconstructed image block is placed in the frame of the reconstructed video signal based on the segmentation data determined in step 111. The syntax used for step 113 can also be indicated in the bitstream through the entropy coding described above.

[0089] In step 115, filtering is performed on the frame of the reconstructed video signal in a manner similar to step 107 at the encoder. For example, a noise suppression filter, a deblocking filter, an adaptive loop filter, and a SAO filter can be applied to the frame to remove blocking artifacts. Once the frame has been filtered, in step 117, the video signal can be output to a display for viewing by an end user.

[0090] Figure 21 is a schematic diagram of an exemplary encoding and decoding (CODEC) system 200 for video decoding. Specifically, CODEC system 200 provides functionality that supports implementation of method 100. CODEC system 200 is broadly used to describe components used at both an encoder and a decoder. CODEC system 200 receives a video signal and segments the video signal, as described in conjunction with steps 101 and 103 of method 100, to obtain segmented video signals 201. Then, when acting as an encoder, CODEC system 200 compresses segmented video signal 201 into an encoded bitstream, as described in conjunction with steps 105, 107, and 109 of method 100. When acting as a decoder, CODEC system 200 generates an output video signal from the bitstream, as described in conjunction with steps 111, 113, 115, and 117 of method 100. The codec system 200 includes a general decoder control component 211, a transform scaling and quantization component 213, an intra-frame estimation component 215, an intra-frame prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded picture buffer component 223, and a header format and context adaptive binary arithmetic coding (CABAC) component 231. These components are coupled as shown. Figure 2 In FIG, black lines represent the movement of data to be encoded / decoded, while dashed lines represent the movement of control data that controls the operation of other components. Components in codec system 200 may all reside in an encoder. A decoder may include a subset of the components in codec system 200. For example, a decoder may include an intra-frame prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded image buffer component 223. These components are described below.

[0091] The segmented video signal 201 is a captured video sequence that has been segmented into pixel blocks by a coding tree. The coding tree uses various partitioning modes to subdivide the pixel blocks into smaller pixel blocks. These blocks can then be further subdivided into smaller blocks. These blocks can be called nodes on the coding tree. A larger parent node is divided into smaller child nodes. The number of times a node is subdivided is called the depth of the node / coding tree. In some cases, the divided blocks may be included in a coding unit (CU). For example, a CU may be a sub-part of a CTU, including a luminance block, one or more red difference chrominance (Cr) blocks and one or more blue difference chrominance (Cb) blocks and the corresponding syntax instructions of the CU. The partitioning mode may include a binary tree (BT), a triple tree (TT) and a quad tree (QT), which are used to partition the node into 2, 3 or 4 child nodes of different shapes, respectively, according to the partitioning mode used. The segmented video signal 201 is forwarded to the general decoder control component 211, the transform scaling and quantization component 213, the intra-frame estimation component 215, the filter control analysis component 227 and the motion estimation component 221 for compression.

[0092] The universal decoder control component 211 is used to make decisions related to encoding images in a video sequence into a bitstream based on application constraints. For example, the universal decoder control component 211 manages the optimization of bitrate / bitstream size relative to reconstruction quality. These decisions can be made based on storage space / bandwidth availability and image resolution requirements. The universal decoder control component 211 also manages buffer utilization based on transmission speed to mitigate buffer underruns and overflows. To address these issues, the universal decoder control component 211 manages segmentation, prediction, and filtering performed by other components. For example, the universal decoder control component 211 can dynamically increase compression complexity to improve resolution and increase bandwidth utilization, or decrease compression complexity to reduce resolution and bandwidth utilization. Thus, the universal decoder control component 211 controls other components in the codec system 200 to balance video signal reconstruction quality with bitrate. The universal decoder control component 211 generates control data used to control the operation of other components. The control data is also forwarded to the header format and CABAC component 231 for encoding into the codestream, thereby instructing the decoder on the parameters to be used for decoding.

[0093] The segmented video signal 201 is also sent to the motion estimation component 221 and the motion compensation component 219 for inter-frame prediction. The frames or slices of the segmented video signal 201 may be divided into a plurality of video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter-frame prediction decoding on the received video blocks relative to one or more blocks in one or more reference frames to provide temporal prediction. The codec system 200 may perform multiple decoding passes to select an appropriate decoding mode for each block of video data, etc.

[0094] The motion estimation component 221 and the motion compensation component 219 can be highly integrated, but for conceptual purposes, they are described separately. Motion estimation, performed by the motion estimation component 221, is the process of generating motion vectors, which are used to estimate the motion of video blocks. For example, a motion vector can represent the displacement of a coded object relative to a prediction block. A prediction block is a block that is found to closely match the block to be coded in terms of pixel difference. A prediction block can also be referred to as a reference block. This pixel difference can be determined by the sum of absolute differences (SAD), the sum of square differences (SSD), or other difference metrics. HEVC uses several coding objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU can be divided into CTBs, which are then divided into CBs, which are included in a CU. A CU can be encoded as a prediction unit (PU) that includes prediction data and / or a transform unit (TU) that includes the transform residual data of the CU. The motion estimation component 221 uses rate-distortion analysis as part of the rate-distortion optimization process to generate motion vectors, PUs, and TUs. For example, the motion estimation component 221 may determine multiple reference blocks, multiple motion vectors, etc. for the current block / frame, and may select a reference block, motion vector, etc. having an optimal rate-distortion characteristic. The optimal rate-distortion characteristic balances the quality of video reconstruction (e.g., the amount of data lost due to compression) and decoding efficiency (e.g., the size of the final encoded image).

[0095] In some examples, the codec system 200 can calculate values for sub-integer pixel positions of a reference image stored in the decoded picture buffer component 223. For example, the video codec system 200 can interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference image. Accordingly, the motion estimation component 221 can perform motion search relative to integer-pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision. The motion estimation component 221 calculates motion vectors for PUs of video blocks in inter-coded slices by comparing the positions of the PUs with the positions of prediction blocks of the reference image. The motion estimation component 221 outputs the calculated motion vectors as motion data to the header format and CABAC component 231 for encoding and as motion data to the motion compensation component 219.

[0096] The motion compensation performed by the motion compensation component 219 may involve obtaining or generating a prediction block based on the motion vector determined by the motion estimation component 221. Likewise, in some examples, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated. Upon receiving the motion vector for the PU of the current video block, the motion compensation component 219 may locate the prediction block to which the motion vector points. The pixel values of the prediction block are then subtracted from the pixel values of the current video block being decoded to obtain the pixel differences, thereby forming a residual video block. Generally speaking, the motion estimation component 221 performs motion estimation with respect to the luma component, and the motion compensation component 219 applies the motion vector calculated based on the luma component to the chroma and luma components. The prediction block and the residual block are forwarded to the transform, scaling, and quantization component 213.

[0097] The segmented video signal 201 is also sent to an intra-frame estimation component 215 and an intra-frame prediction component 217. Like the motion estimation component 221 and the motion compensation component 219, the intra-frame estimation component 215 and the intra-frame prediction component 217 can be highly integrated, but for conceptual purposes, they are described separately. The intra-frame estimation component 215 and the intra-frame prediction component 217 perform intra-frame prediction on the current block relative to blocks in the current frame, replacing the inter-frame prediction performed between frames by the motion estimation component 221 and the motion compensation component 219 as described above. Specifically, the intra-frame estimation component 215 determines an intra-frame prediction mode to encode the current block. In some examples, the intra-frame estimation component 215 selects an appropriate intra-frame prediction mode from a plurality of tested intra-frame prediction modes to encode the current block. The selected intra-frame prediction mode is then forwarded to the header format and CABAC component 231 for encoding.

[0098] For example, the intra-frame estimation component 215 performs a rate-distortion analysis on various tested intra-frame prediction modes to calculate rate-distortion values and selects the intra-frame prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between the encoded block and the original unencoded block that is encoded to generate the encoded block, as well as determines the code rate (e.g., number of bits) used to generate the encoded block. The intra-frame estimation component 215 calculates a ratio based on the distortion and rate of the various encoded blocks to determine the intra-frame prediction mode that exhibits the best rate-distortion value for the block. In addition, the intra-frame estimation component 215 can be used to decode the depth block of the depth image using a depth modeling mode (DMM) according to rate-distortion optimization (RDO).

[0099] The intra prediction component 217, when implemented on an encoder, can generate a residual block from the prediction block based on the selected intra prediction mode determined by the intra estimation component 215, or, when implemented on a decoder, can read the residual block from the bitstream. The residual block contains the difference between the predicted block and the original block, represented as a matrix. The residual block is then forwarded to the transform, scaling, and quantization component 213. The intra estimation component 215 and the intra prediction component 217 can operate on both luma and chroma components.

[0100] The transform scaling and quantization component 213 is used to further compress the residual block. The transform scaling and quantization component 213 applies a transform such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform to the residual block, thereby generating a video block comprising residual transform coefficient values. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used. The transform converts the residual information from the pixel value domain to a transform domain, such as the frequency domain. The transform scaling and quantization component 213 is also used to scale the transformed residual information based on frequency, for example. This scaling involves applying a scaling factor to the residual information to quantize different frequency information at different granularities, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also used to quantize the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all coefficients. The degree of quantization can be modified by adjusting the quantization parameter. In some examples, the transform scaling and quantization component 213 may then perform a scan on the matrix comprising the quantized transform coefficients. The quantized transform coefficients are forwarded to the header format and CABAC component 231 for encoding into the codestream.

[0101] The scaling and inverse transform component 229 performs the opposite operations to the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 applies inverse scaling, inverse transform, and / or inverse quantization to reconstruct a residual block in the pixel domain, for example, to be subsequently used as a reference block. This reference block can become a prediction block for another current block. The motion estimation component 221 and / or the motion compensation component 219 can calculate a reference block by adding the residual block back to the corresponding prediction block to be used for motion estimation of subsequent blocks / frames. Filters are applied to the reconstructed reference block to reduce artifacts generated during scaling, quantization, and transforming. These artifacts can make the prediction inaccurate (and generate additional artifacts) when predicting subsequent blocks.

[0102] The filter control analysis component 227 and the in-loop filter component 225 apply filters to the residual block and / or the reconstructed image block. For example, the transformed residual block from the scaling and inverse transform component 229 can be combined with the corresponding prediction block from the intra-frame prediction component 217 and / or the motion compensation component 219 to reconstruct the original image block. The filter can then be applied to the reconstructed image block. In some examples, the filter can be applied to the residual block in turn. Figure 2 The other components in the, filter control analysis component 227 and the in-loop filter component 225 are highly integrated and can be implemented together, but are described separately for conceptual purposes. The filters applied to the reconstructed reference block are applied to specific spatial regions, and these filters include multiple parameters to adjust the way in which these filters are used. The filter control analysis component 227 analyzes the reconstructed reference block to determine the locations where these filters need to be used and sets the corresponding parameters. This data is forwarded as filter control data to the header format and CABAC component 231 for encoding. The in-loop filter component 225 applies these filters based on the filter control data. These filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. These filters can be applied in the spatial / pixel domain (for example, for reconstructed pixel blocks) or the frequency domain according to the examples.

[0103] When operating as an encoder, the filtered reconstructed image blocks, residual blocks, and / or prediction blocks are stored in the decoded image buffer component 223 for subsequent use in motion estimation, as described above. When operating as a decoder, the decoded image buffer component 223 stores the reconstructed and filtered blocks and forwards them to the display as part of the output video signal. The decoded image buffer component 223 can be any storage device capable of storing prediction blocks, residual blocks, and / or reconstructed image blocks.

[0104] The header format and CABAC component 231 receives data from various components of the codec system 200 and encodes this data into an encoded bitstream to be sent to the decoder. Specifically, the header format and CABAC component 231 generates various headers to encode control data (e.g., general control data and filter control data). In addition, prediction data (including intra-frame prediction data and motion data) and residual data in the form of quantized transform coefficient data are encoded into the bitstream. The final bitstream includes all the information required by the decoder to reconstruct the original segmented video signal 201. This information may also include an intra-frame prediction mode index table (also called a codeword mapping table), definitions of the coding contexts of various blocks, representation of the most likely intra-frame prediction mode, representation of segmentation information, etc. This data can be encoded using entropy coding. For example, this information can be encoded using context adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique. After entropy coding, the encoded bitstream can be sent to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0105] Figure 3 is a block diagram of an exemplary video encoder 300. Video encoder 300 can be used to implement the encoding functionality of codec system 200 and / or implement steps 101, 103, 105, 107, and / or 109 of method 100. Encoder 300 segments an input video signal to obtain segmented video signals 301, which are substantially similar to segmented video signal 201. Segmented video signals 301 are then compressed by components within encoder 300 and encoded into a bitstream.

[0106] Specifically, the segmented video signal 301 is forwarded to an intra-frame prediction component 317 for intra-frame prediction. Intra-frame prediction component 317 can be substantially similar to intra-frame estimation component 215 and intra-frame prediction component 217. The segmented video signal 301 is also forwarded to a motion compensation component 321 for inter-frame prediction based on reference blocks in a decoded picture buffer component 323. Motion compensation component 321 can be substantially similar to motion estimation component 221 and motion compensation component 219. The prediction blocks and residual blocks from intra-frame prediction component 317 and motion compensation component 321 are forwarded to a transform and quantization component 313 for transform and quantization of the residual blocks. Transform and quantization component 313 can be substantially similar to transform, scaling, and quantization component 213. The transformed and quantized residual blocks and corresponding prediction blocks (along with associated control data) are forwarded to an entropy coding component 331 for encoding in the bitstream. Entropy coding component 331 can be substantially similar to header format and CABAC component 231.

[0107] The transformed and quantized residual block and / or the corresponding prediction block are also forwarded from the transform and quantization component 313 to the inverse transform and dequantization component 329 for reconstruction as a reference block for use by the motion compensation component 321. The inverse transform and dequantization component 329 can be substantially similar to the scaling and inverse transform component 229. According to an example, an in-loop filter in the in-loop filter component 325 is also applied to the residual block and / or the reconstructed reference block. The in-loop filter component 325 can be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 can include multiple filters, as described in conjunction with the in-loop filter component 225. The filtered block is then stored in the decoded image buffer component 323 for use as a reference block by the motion compensation component 321. The decoded image buffer component 323 can be substantially similar to the decoded image buffer component 223.

[0108] Figure 4 is a block diagram of an exemplary video decoder 400. Video decoder 400 can be used to implement the decoding function of codec system 200 and / or implement steps 111, 113, 115, and / or 117 of operating method 100. For example, decoder 400 receives a bitstream from encoder 300 and generates a reconstructed output video signal based on the bitstream for display to an end user.

[0109] The bitstream is received by entropy decoding component 433. Entropy decoding component 433 is used to perform an entropy decoding scheme, such as CAVLC, CABAC, SBAC, PIPE decoding, or other entropy decoding techniques. For example, entropy decoding component 433 can use header information to provide context for parsing additional data encoded as codewords in the bitstream. The decoded information includes any information required to decode the video signal, such as general control data, filter control data, segmentation information, motion data, prediction data, and quantized transform coefficients in the residual block. The quantized transform coefficients are forwarded to inverse transform and dequantization component 429 for reconstruction into the residual block. Inverse transform and dequantization component 429 can be similar to inverse transform and dequantization component 329.

[0110] The reconstructed residual block and / or prediction block is forwarded to the intra-frame prediction component 417 for reconstruction into an image block based on the intra-frame prediction operation. The intra-frame prediction component 417 can be similar to the intra-frame estimation component 215 and the intra-frame prediction component 217. Specifically, the intra-frame prediction component 417 uses the prediction mode to locate the reference block in the frame and adds the residual block to the result to reconstruct the intra-frame predicted image block. The reconstructed intra-frame predicted image block and / or residual block and the corresponding inter-frame prediction data are forwarded to the decoded image buffer component 423 via the in-loop filter component 425. The decoded image buffer component 423 and the in-loop filter component 425 can be substantially similar to the decoded image buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, residual block, and / or prediction block. This information is stored in the decoded image buffer component 423. The reconstructed image block from the decoded image buffer component 423 is forwarded to the motion compensation component 421 for inter-frame prediction. Motion compensation component 421 can be substantially similar to motion estimation component 221 and / or motion compensation component 219. Specifically, motion compensation component 421 uses the motion vector of the reference block to generate a prediction block and applies the residual block to the result to reconstruct the image block. The resulting reconstructed block can also be forwarded to decoded image buffer component 423 via in-loop filter component 425. Decoded image buffer component 423 continues to store other reconstructed image blocks. These reconstructed image blocks can be reconstructed into frames using segmentation information. These frames can also be arranged in a sequence. This sequence is output to a display as a reconstructed output video signal.

[0111] The above mechanisms are exemplary machines and / or processes that can be used to encode and decode 2D videos presented in square / rectangular frames. However, PCC video data uses points to represent irregular 3D objects. The following mechanisms are PCC-specific video decoding processes. These mechanisms can be used to represent 3D point clouds in 2D square / rectangular frames so that 2D frames can be encoded / decoded by the above mechanisms. As a general example, a 3D point cloud at a specific moment can be described by a geometry frame that represents the position of each point. The color value and light value of a point can be represented as a 2D slice projected onto the point, as shown in the geometry frame. The slices at a moment are packed into a square / rectangular atlas frame, which can be encoded / decoded by the above mechanisms. Since a slice may not completely cover the atlas frame, a corresponding occupancy frame can be used to indicate which areas of the atlas frame include slice data and which areas have no available video data. Any additional attributes of the point can be decoded in the corresponding attribute frame. These mechanisms will be described in more detail below.

[0112] Figure 5 is an example of point cloud media 500 that can be decoded according to the PCC mechanism. Therefore, when executing method 100, point cloud media 500 can be encoded by an encoder (e.g., codec system 200 and / or encoder 300) and reconstructed by a decoder (e.g., codec system 200 and / or decoder 400).

[0113] Figures 1 to 4 The mechanisms described in generally assume that 2D frames are being encoded. However, point cloud media 500 is a cloud of points that changes over time. Specifically, point cloud media 500 (which may also be referred to as a point cloud and / or point cloud representation) is a collection of points in 3D space. These points may also be referred to as samples. Each point can be associated with various types of data. For example, each point can be described by a position. A position is a location in 3D space and can be described as a set of Cartesian coordinates. Additionally, each point can include a color. Color can be described by brightness (e.g., light) and chromaticity (e.g., color). Color can be described by red (R), green (G), and blue (B) values (expressed as (R, G, B)) or brightness (Y), blue projection (U), and red projection (V) (expressed as (Y, U, V)). These points may also include other attributes. Attributes are optional scalar or vector properties that can be associated with each point in the point cloud. Attributes may include reflectivity, transparency, surface normal, timestamp, material identifier (ID), and the like.

[0114] Since each point in the point cloud media 500 can be associated with multiple types of data, Figures 1 to 4Several supporting mechanisms are used to prepare the point cloud media 500 for compression. For example, the point cloud media 500 can be categorized into frames, where each frame includes all the data related to the point cloud for a specific state or moment in time. Figure 5 A single frame of point cloud media 500 is described. The point cloud media 500 is then decoded frame by frame. The point cloud media 500 can be surrounded by a 3D bounding box 501. The 3D bounding box 501 is a 3D rectangular prism that is sized to enclose all points of the point cloud media 500 for the corresponding frame. The 3D bounding box 501 can be formally defined as the volume of a rectangular solid defined as six rectangular faces placed at right angles. It should be noted that in the case where the point cloud media 500 includes disjoint sets, multiple 3D bounding boxes 501 can be used. For example, the point cloud media 500 can describe two unconnected figures, in which case a 3D bounding box 501 will be placed around each figure. The points in the 3D bounding box 501 are processed as described below.

[0115] Figure 6 6 is an example of a slice 603 created from a point cloud 600. Point cloud 600 is a single frame of point cloud media 500. Furthermore, point cloud 600 is surrounded by a 3D bounding box 601 that is substantially similar to 3D bounding box 501. Thus, when performing method 100, point cloud 600 can be encoded by an encoder (e.g., codec system 200 and / or encoder 300) and reconstructed by a decoder (e.g., codec system 200 and / or decoder 400).

[0116] 3D bounding box 601 includes six faces and, therefore, six 2D rectangular frames 602, each positioned on a face of 3D bounding box 601 (e.g., top, bottom, left, right, front, and back). Point cloud 600 can be converted from 3D data to 2D data by projecting it onto the corresponding 2D rectangular frame 602. This creates a patch 603. It should be noted that 2D rectangular frame 602 can also be referred to as a 2D bounding box (e.g., of 3D bounding box 601). Patch 603 is a 2D representation of the 3D point cloud, wherein patch 603 includes a representation of point cloud 600 visible from the corresponding 2D rectangular frame 602. A patch 603 can be formally defined as a 2D rectangular region within an atlas associated with volumetric information. It should be noted that the representation of point cloud 600 visible from 2D rectangular frame 602 can include multiple non-intersecting components. Therefore, 2D rectangular frame 602 can include multiple patches 603. Therefore, the point cloud 600 can be represented by more than six slices 603. Slices 603 can also be referred to as atlases, atlas data, atlas information, and / or atlas components. By converting the 3D data into a 2D format, the slices 603 of the point cloud 600 can be decoded according to video decoding mechanisms, such as inter-frame prediction and / or intra-frame prediction.

[0117] 7A to 7C Shown is a Figure 6 The mechanism for encoding a 3D point cloud that has been converted into 2D information is described in [1]. Specifically, Figure 7A An exemplary occupancy frame 710 associated with a set of patches (e.g., patch 603) is shown. Occupancy frame 710 includes occupied pixels 713 and 715, with occupancy values encoded in binary form. Occupied pixels 713 and 715 are values that indicate whether an atlas sample corresponds to an associated sample in 3D space. For example, an occupancy value of 0 (depicted as occupied pixel 715) indicates that a portion of bounding box 601 is not occupied by one of patches 603. Those portions of bounding box 601 represented by 0 do not participate in the reconstruction of the volumetric representation (e.g., point cloud 600). In contrast, an occupancy value of 1 (depicted as occupied pixel 713) indicates that a portion of bounding box 601 is occupied by one of patches 603. Those portions of bounding box 601 represented by 1 do participate in the reconstruction of the volumetric representation (e.g., point cloud 600). Occupancy frame 710 can be formally defined as a collection of occupancy values that form a 2D array and represent the entire occupancy information for a single atlas frame. Therefore, occupancy frame 710 can indicate whether each sample location in the atlas corresponds to a valid 3D point in the point cloud representation.

[0118] The various slices created by projecting 3D information onto a 2D plane can be packed into rectangular (or square) video frames. This approach can be advantageous because various video codecs (such as AVC, HEVC, and VVC) are pre-configured to decode such video frames. Therefore, the PCC codec can use other video codecs to decode the blocks. Figure 7A As shown, slices can be packed into frames. Slices can be packed using any algorithm. For example, slices can be packed into frames based on size. In one specific example, slices are packed from largest to smallest. The largest slices can be placed first in any open space, and once a size threshold is exceeded, smaller slices fill the gaps. Figure 7A As shown, this packing scheme will produce empty spaces that do not include patch data. In order to avoid encoding empty spaces, an occupancy frame 710 is used. The occupancy frame 710 includes all occupancy data of the point cloud at a specific moment. Specifically, the occupancy frame 710 includes one or more occupancy values (also called occupancy data, occupancy information and / or occupancy components). The occupied pixels 713 and / or 715 are implemented as a 2D array corresponding to the atlas pixels 733 (a set of patches), and the occupancy value indicates whether each sample position in the atlas pixel 733 corresponds to a valid 3D point in the point cloud representation. As shown in FIG. Figure 7AAs shown, occupied frame 710 includes a valid data region depicted as occupied pixels 713. The valid data region indicates that atlas pixel 733 / slice data is present in the corresponding location in occupied frame 710. Occupied frame 710 also includes an invalid data region depicted as occupied pixels 715. The invalid data region indicates that atlas pixel 733 / slice data is not present in the corresponding location in occupied frame 710.

[0119] Figure 7B An exemplary geometry frame 720 associated with a set of patches (e.g., patch 603) is shown. The geometry frame 720 (also referred to as a geometry map) provides or describes the outline or topography of each patch 603. Specifically, the geometry frame 720 represents the distance of each point in the patch 603 from the planar surface (e.g., 2D rectangular frame 602) of the bounding box 601. The geometry frame 720 includes geometry pixels 723. The geometry pixels 723 can be defined as a set of Cartesian coordinates associated with a volume frame. A volume frame is a set of 3D points specified by Cartesian coordinates and zero or more corresponding attribute sets at a particular moment. Therefore, a volume frame is a combination of an occupancy frame 710, a geometry frame 720, an atlas frame 730, and (optionally) one or more attribute frames that describe the point cloud at a specified moment. For example, the geometry pixels 723 can describe the position of each point in the 3D bounding box at the corresponding moment.

[0120] As described above, the geometry frame 720 includes geometry pixels 723 (also referred to as geometry maps, geometry data, geometry information, and / or geometry components) of the point cloud at a particular moment in time. The geometry pixels 723 are a 2D array created by aggregating the geometry information associated with each patch, where the geometry information / data is a set of Cartesian coordinates associated with the point cloud frame. Specifically, each patch is projected from a point in 3D space. This projection has the effect of removing 3D information from the patch. The geometry pixels 723 retain the 3D information removed from the patch. For example, each sample in the patch is obtained from a point in 3D space. Therefore, the geometry frame 720 can include 3D coordinates associated with each sample in each patch. Therefore, the geometry frame 720 / geometry pixels 723 can be used by the decoder to map / convert the 2D patches back into 3D space to reconstruct the 3D point cloud. Specifically, the decoder can map each patch sample to the appropriate 3D coordinate to reconstruct the point cloud.

[0121] Figure 7CAn exemplary atlas frame 730 associated with a set of slices (e.g., slice 603) is shown. Atlas frame 730 provides or describes samples of slice 603 within bounding box 601. Atlas frame 730 can be formally defined as a collection of two-dimensional (2D) bounding boxes (also referred to as slices and / or atlas pixels 733) projected into a rectangular frame. These 2D bounding boxes correspond to volumes in 3D space (referred to as 3D bounding boxes) on which volumetric data is rendered. Atlas frame 730 may include, for example, color components and / or brightness components for points in slice 603. The color components may be based on an RGB color model, a YUV color model, or another known color model. Occupancy frame 710, geometry frame 720, and atlas frame 730 may be used to decode point cloud 600 and / or point cloud media 500. Thus, when method 100 is performed, occupancy frame 710 , geometry frame 720 , and atlas frame 730 may be encoded by an encoder (eg, codec system 200 and / or encoder 300 ) and reconstructed by a decoder (eg, codec system 200 and / or decoder 400 ).

[0122] As described above, the atlas frame 730 includes one or more atlas pixels 733 (also referred to as atlas data, atlas information, atlas components and / or slices) of the point cloud at a particular moment in time. An atlas pixel 733 is a collection of 2D bounding boxes projected into a rectangular frame corresponding to a 3D bounding box in 3D space, where each 2D bounding box / slice represents a subset of the point cloud. Specifically, an atlas pixel 733 includes Figure 6 The slices created when the 3D point cloud is projected into 2D space. Thus, an atlas pixel 733 / slice includes image data (e.g., color values and light values) associated with the point cloud at the corresponding moment in time. An atlas pixel 733 corresponds to Figure 7A The occupied frame 710 and Figure 7B Specifically, atlas pixel 733 includes data in the valid data region according to occupied pixel 713 and does not include data in the invalid data region according to occupied pixel 715. In addition, geometry pixel 723 includes 3D information of the sample in atlas pixel 733.

[0123] It should also be noted that the point cloud can include attribute pixels (also known as attribute data, attribute information and / or attribute components). These attribute pixels can be included in the atlas frame. The attribute pixels can include all data about the corresponding attributes of the point cloud at a specific moment. Examples of attribute frames are not shown because attribute pixels can include a wide range of different data. Specifically, attributes can be any scalar or vector property associated with each point in the point cloud, such as reflectivity, surface normal, timestamp, material ID, etc. In addition, attributes are optional (e.g., user defined) and can vary depending on the application. However, when used, point cloud attributes can be included in the attribute frame in a manner similar to atlas pixels 733, geometry pixels 723, and occupied pixels.

[0124] Thus, the encoder can compress the point cloud frame into an atlas frame 730 of atlas pixels 733, a geometry frame 720 of geometry pixels 723, an occupancy frame 710 of occupancy pixel 713 and 715 values, and (optionally) an attribute frame (commonly referred to as a volume frame) of attribute pixels. The atlas frame 730, geometry frame 720, occupancy frame 710 and / or attribute frame can be further compressed, for example, by different encoders for transmission to the decoder. The decoder can decompress the atlas frame 730, geometry frame 720, occupancy frame 710 and / or attribute frame. The decoder can then use the atlas frame 730, geometry frame 720, occupancy frame 710 and / or attribute frame to reconstruct the point cloud frame to determine the reconstructed point cloud at the corresponding moment. The reconstructed point cloud frame can then be included in sequence to reconstruct the original point cloud sequence (e.g., for display and / or for data analysis). In a specific example, the original point cloud sequence can be reconstructed by using a combination of Figures 1 to 4 Techniques are described for encoding and decoding atlas frames 730 and / or atlas pixels 733 (eg, by using VVC, HEVC, and / or AVC codecs).

[0125] Figure 8800 is an exemplary mechanism for scaling a reconstructed point cloud by scaling a corresponding 3D bounding box 801. For example, a point cloud (e.g., point cloud media 500) can be reconstructed in a 3D bounding box 801, which can be substantially similar to 3D bounding boxes 501 and / or 601. In one specific example, a set of slices 603 can be decoded from an occupancy frame 710, a geometry frame 720, an atlas frame 730, and / or an attribute frame. The slices can be projected onto points in 3D bounding box 801 to reconstruct the point cloud. 3D bounding box 801 can then be scaled before display. This causes the reconstructed point cloud included in 3D bounding box 801 to also be scaled. Thus, mechanism 800 can be used when implementing methods 100, 1100, and / or 1200. Furthermore, codec system 200, encoder 300, decoder 400, and / or video decoding device 1000 can use mechanism 800, for example, when reconstructing a point cloud for error testing at an encoder or display at a decoder. In addition, the mechanism 800 may be indicated by a PCC code stream (eg, V-PCC code stream 900 ).

[0126] As described above, the point cloud can be reconstructed into a reconstructed point cloud 803 in a 3D bounding box 801. Specifically, geometric pixels can be used to locate points in the 3D bounding box 801. Occupancy pixels can be used to locate atlases / slices in the atlas frame. The atlas can be decoded and projected onto the points in the 3D bounding box 801. Any attribute pixels in the attribute frame can also be applied. However, one problem with many PCC systems is that point clouds can be encoded and decoded to arbitrary sizes. For example, some PCC systems may encode all point clouds to the same size. This approach forces the user to appropriately adjust the size of the cloud when displayed. This approach may not be suitable for augmented reality (AR) purposes.

[0127] For example, AR typically enables users to record real-time video using a video recording device. The AR system then overlays predetermined video content on the real-time video and displays the result to the user. This creates the impression of fictional, technological, or other generated visual content that is overlaid onto the video in the user's environment. Example AR devices may include smartphones, tablets, laptops, or other computing devices with cameras and displays. Point clouds of arbitrary size may not be suitable for this application because the AR system may not be able to appropriately resize the point cloud to accommodate the real-time video. In the example shown, the reconstructed cloud 803 depicts a human figure. For example, the human figure can be displayed as a doll that occupies relatively little space in the scene, or as a human-sized figure that occupies a large portion of the scene. Requiring the user to appropriately resize the reconstructed cloud 803 may make the application cumbersome and / or unusable.

[0128] The present invention includes a 3D bounding box scale 805 and a 3D bounding box scale 806. 3D bounding box scale 805 / 806 is a parameter representing a scale to be applied to the 3D bounding box 801 parameters that may be specified for an object (e.g., reconstructed cloud 803). For example, 3D bounding box scale 805 may be a value that scales 3D bounding box 801, and thus the reconstructed point cloud 803, to a smaller size (e.g., the scale of a doll). Furthermore, 3D bounding box scale 806 may be a value that scales 3D bounding box 801, and thus the reconstructed point cloud 803, to a larger size (e.g., the scale of a person). This enables the reconstructed point cloud 803 to be correctly sized so that it can be overlaid onto a real-world video. 3D bounding box scale 805 / 806 may be a factor based on the sample density of the reconstructed point cloud 803 within 3D bounding box 801 and the size of 3D bounding box 801. In some examples, 3D bounding box scale 805 / 806 may be expressed in meters. For example, the 3D bounding box scale 805 / 806 can be expressed as a value representing the sample density value of the reconstructed point cloud 803 divided by the size of the 3D bounding box 801, or vice versa. The appropriate 3D bounding box scale 805 / 806 can be determined by the encoder based on the expected size of the reconstructed point cloud 803. The 3D bounding box scale 805 / 806 selected by the encoder can be encoded into the V-PCC code stream as a parameter. The decoder can obtain the indicated 3D bounding box scale 805 / 806 from the V-PCC code stream and scale the 3D bounding box 801 and the reconstructed point cloud 803 before rendering the reconstructed point cloud 803 for overlaying and displaying real-time video in an AR context. In other examples, the 3D bounding box scale 805 / 806 can also be used to scale any reconstructed point cloud 803 for other applications, such as for displaying the reconstructed point cloud 803 relative to a scene with a relative size that varies in size.

[0129] Figure 9 is a schematic diagram of an exemplary V-PCC codestream 900 for scaling a point cloud. For example, V-PCC codestream 900 can be generated by codec system 200 and / or encoder 300 for decoding by codec system 200 and / or decoder 400 according to method 100. Furthermore, V-PCC codestream 900 can be used to encode a set of slices 603 from point cloud media 500 into occupancy frames 710, geometry frames 720, atlas frames 730, and / or attribute frames. Furthermore, V-PCC codestream 900 can encode 3D bounding box scales 805 / 806 to support scaling at the decoder.

[0130] The V-PCC code stream 900 includes a series of V-PCC units 910 and supports supplemental enhancement information (SEI) messages 920. The V-PCC unit 910 is a data container sized to be placed in a data packet for transmission to a decoder. Specifically, the V-PCC unit 910 includes a V-PCC component associated with one or more volume frames representing a point cloud at a corresponding moment in a V-PCC video sequence. In a specific example, the V-PCC unit 910 can be configured as a network abstraction layer (NAL) unit corresponding to the universal video coding (VVC) standard. The NAL unit is a data container of data packet size. For example, the size of a single NAL unit is typically designed to enable network transmission. The NAL unit may include a header indicating the NAL unit type and a payload including associated data. The V-PCC unit 910 includes a V-PCC unit payload 914 and a V-PCC unit header 912. The V-PCC unit payload 914 includes video data of a corresponding type associated with the volumetric frame. The V-PCC unit header 912 is a header indicating the type of data included in the V-PCC unit payload 914.

[0131] According to an example, the V-PCC unit payload 914 may include a V-PCC parameter set 911, an atlas frame 913, an occupancy frame 915, a geometry frame 917, and / or an attribute frame 919. The V-PCC parameter set 911 is a set of parameters that describe the settings for decoding one or more corresponding volume frames. For example, the V-PCC parameter set 911 may include an atlas sequence parameter set (SPS) 921. The atlas SPS 921 is a syntax structure that includes syntax elements that apply to one or more complete coded atlas sequences. The V-PCC parameter set 911 may also include other parameter sets, such as an atlas frame parameter set, a video-based visual volumetric coding (V3C) parameter set, and the like. The main difference between the parameter sets is whether the included parameters are related to V-PCC at the video level (e.g., V3C parameter set), sequence level (e.g., atlas SPS), and / or frame level (e.g., atlas frame parameter set). Other parameter sets may be used for specific purposes and the above list is not exhaustive.

[0132] Atlas frame 913 is a collection of 2D bounding boxes / patches / atlases projected into a rectangular frame. These 2D bounding boxes / patches / atlases correspond to volumes in 3D space (referred to as 3D bounding boxes), and volumetric data is rendered on atlas frame 913. Atlas frame 913 can be substantially similar to atlas frame 730. Occupancy frame 915 is a collection of occupancy values that form a 2D array and represent the entire occupancy information for a single atlas frame. An occupancy pixel is a value indicating whether an atlas sample corresponds to an associated sample in 3D space. Occupancy frame 915 can be substantially similar to occupancy frame 710. Geometry frame 917 (also referred to as a geometry map) is a frame that includes geometry patch information projected at a specific depth. A geometry pixel is a set of Cartesian coordinates associated with a volumetric frame. Geometry frame 917 can be substantially similar to geometry frame 720. Attribute frame 919 is a frame that includes scalar or vector properties, such as color, reflectivity, surface normal, timestamp, material identifier (ID), and the like, optionally associated with each point in the volumetric frame of the point cloud. A volumetric frame is a collection of 3D points specified by Cartesian coordinates and zero or more corresponding attribute sets at a specific moment in time. A point cloud video consists of a series of volumetric frames. Therefore, the V-PCC parameter set 911 includes parameters indicating how to decode one or more volumetric frames, and the atlas frame 913, occupancy frame 915, geometry frame 917, and attribute frame 919 contain the video data used to reconstruct the volumetric frames.

[0133] The V-PCC codestream 900 also includes various SEI messages 920. An SEI message is a syntactic structure with specified semantics that conveys information not required by the decoding process in order to determine the values of samples in the decoded image. Therefore, SEI messages can be used to transmit data that is not directly relevant to decoding the V-PCC unit 910. In the example shown, the V-PCC codestream 900 includes a scene object information (SOI) SEI message 923. Other SEI messages may also be used for specific purposes.

[0134] The SOI SEI message 923 is an SEI message that defines a set of objects that can exist in a volumetric scene and optionally assigns different properties to these objects. These objects may then be associated with different types of information, including slices and 2D volume rectangles, which can be defined using the Slice Information and Volume Rectangle Information SEI messages.

[0135] As described above, some PCC systems do not include data sufficient to adjust the size of the reconstructed point cloud based on real-world measurements. The V-PCC codestream 900 includes parameters sufficient to overcome these limitations. For example, the atlas SPS 921 may include volumetric usability information (VUI) parameters. Specifically, the atlas SPS 921 may include a VUI unit flag (vui_unit_in_metres_flag) 933 expressed in meters. A flag is a value indicating data of an associated function or process. The vui_unit_in_metres_flag 933 is a flag indicating whether the coordinate data of the point cloud is expressed in meters. In a specific example, the vui_unit_in_metres_flag 933, when equal to 1, indicates that the real-world coordinate information is expressed in meters. In addition, the vui_unit_in_metres_flag 933, when equal to 0, indicates that the world coordinates are unitless.

[0136] In addition, the SOI SEI message 923 may include a 3D bounding box scale, which may be represented as SOI 3D bounding box scale log2 (soi_3d_bounding_box_scale_log2) 931. soi_3d_bounding_box_scale_log2 931 is a parameter that represents the scale to be applied to the 3D bounding box parameters that may be specified for the object. For example, a volumetric frame of a point cloud may be reconstructed into a 3D bounding box using data from the atlas frame 913, the occupancy frame 915, the geometry frame 917, and / or the attribute frame 919. This produces one or more point clouds of arbitrary unitless size described by a local coordinate system (e.g., a Cartesian coordinate system). The scale in soi_3d_bounding_box_scale_log2 931 may then be applied to perform a transformation. For example, soi_3d_bounding_box_scale_log2 931 may be applied to transform the local coordinate system into a final target 3D coordinate system that describes the object at its real-world size. This allows the reconstructed point cloud to be correctly sized so that the reconstructed point cloud can be overlaid onto the real-world video (e.g., without requiring a significant amount of additional processing by a corresponding application at the decoder and / or without requiring user intervention at the decoder). soi_3d_bounding_box_scale_log2 931 can be a factor based on the sample density in the 3D bounding box and the size of the 3D bounding box. For example, soi_3d_bounding_box_scale_log2 931 can relate the number of spatial units in the point cloud frame to a step value defined based on the spatial frequency in the 3D bounding box. In one specific example, soi_3d_bounding_box_scale_log2 931 can represent the number of steps and the step size (e.g., expressed in meters) that is equal to the quotient of the desired spatial frequency divided by the number of spatial units in the point cloud frame. As another example, when the spatial rate of the point cloud frame signal is 1 m -1 When , the number of spatial units can be equal to 1000 and the spatial frequency can be equal to 1 million, which produces a step size of 1 mm.

[0137] The SOI SEI message 923 may also include an SOI 3D bounding box present flag (soi_3d_bounding_box_present_flag) 932. The soi_3d_bounding_box_present_flag 932 may be set to indicate whether the soi_3d_bounding_box_scale_log2 931 is included in the SOI SEI message 923. For example, the soi_3d_bounding_box_present_flag 932 may be set to 1 to indicate that 3D bounding box information is present in the current scene object information SEI message. Alternatively, the soi_3d_bounding_box_present_flag 932 may be set to 0 to indicate that 3D bounding box information is not present. Thus, the decoder can reconstruct the point cloud in the 3D bounding box, check soi_3d_bounding_box_present_flag 932 to determine whether soi_3d_bounding_box_scale_log2 931 exists, obtain parameters from soi_3d_bounding_box_scale_log2 931, and scale the 3D bounding box and reconstructed point cloud to real-world size, for example in meters. The renderer at the decoder can then render the scaled point cloud for display on the screen by converting the 3D cloud into a 2D perspective, overlaying the rendered point cloud onto the recorded / live / real-world video, and then displaying the combined video and point cloud representation as an AR video. Thus, this example increases the functionality of the encoder and decoder by creating AR support. In addition, the present invention supports mechanisms for improving decoding efficiency and, therefore, supports reducing processor, memory, and / or network resource usage at the encoder and / or decoder.

[0138] The specific implementation of the above information is described in more detail below. Point clouds can be an integral part of various applications including the entertainment industry, smart car navigation, geospatial detection, 3D modeling of real-world objects, visualization, etc. Given the non-uniformly sampled geometric pixels of point clouds, it may be beneficial to create a compact representation for storing and transmitting such data. Compared to other 3D presentations, irregular point clouds may be more versatile and applicable to a wider range of sensors and data acquisition strategies. For example, for 3D presentation in a virtual real world or remote rendering in a telepresence environment, the rendering of virtual characters and real-time instructions can be processed as a dense point cloud dataset. Such content can be used as immersive six-degree-of-freedom, dynamic augmented / virtual reality objects, cultural heritage, graphic information system mapping, computer-aided design and drawing, autonomous navigation, etc.

[0139] A dynamic point cloud sequence represents a sequence of point cloud frames. The V-PCC encoding and decoding scheme is based on segmenting the 3D point cloud data into a set of 3D patches, which are represented by 3D bounding boxes (Patch3dPosX, Patch3dPosY, Patch3dPosMinZ and the normal axis representation - PatchAxisZ), and then orthogonally projected onto a plane to obtain a set of 2D projection patches with 2D bounding boxes (Patch2dPosX, Patch2dPosY, Patch2dSizeX, Patch2dSizeY). Therefore, V-PCC uses a combination of 3D bounding box information and 2D bounding box information.

[0140] Therefore, a set of 3D points in a point cloud frame is iteratively segmented into 3D slices based on the definition of a smooth continuous surface criterion and projected onto the sides of the bounding box to form 2D slices. A collection of slices forms a slice tile group, where the slice tile groups are combined in the atlas data of a given point cloud frame. Each element of the atlas data can be called a slice, has a specific and unique index, and corresponds to a unique 3D bounding box within the 3D point cloud frame. In addition, if a slice in a point cloud frame has a corresponding reference slice in a reference point cloud frame, the index of the reference slice in the reference slice tile group should be transmitted in the codestream.

[0141] The representation of the V-PCC unit payload describing the time series of point clouds may include occupancy video data frames (2D frames), attribute video data frames (2x 2D frames: near layer and far layer), geometry video data frames (2x 2D frames: near layer and far layer) and / or atlas data. The atlas data may be 1x groups of slices T(i, 0) to T(i, m), which may include information for reconstructing a point cloud frame from the occupancy component, attribute component, and geometry component of the V-PCC unit payload, where i is the atlas data index corresponding to the 3D PCC frame, where im+1 is the number of 3D slices generated for 3D point cloud frame i, and where T(i, j) is referred to as a slice.

[0142] The patch can be predicted by referring to the reference frame / data unit of the current data unit. Prediction between near and far layers can only be performed within the same V-PCC frame. The process of obtaining a single 2D patch from the corresponding 3D bounding box in the point cloud frame is as follows. The 3D point cloud frame is divided into independent 3D patches, which represent 3D information defined by the bounding box in 3D space. A projection plane is defined for each 3D patch as one of the sides of the 3D bounding box of the point cloud. For example, the criterion for selecting the projection plane can be that the area of the projected 3D patch is the largest among all candidates. The normal axis of the patch (PatchAxisZ) can be defined to be orthogonal to the projection plane. Tangent axes and bitangent axes can be defined to create a right-handed 3D coordinate system. The corresponding values of the patch data frame elements with index patchIdx as Patch3dPosX, Patch3dPosY, Patch3dPosMinZ (describing the 3D position of the 3D patch) and Patch2dPosX, Patch2dPosY, Patch2dSizeX, Patch2dSizeY (describing the 2D coordinates of the 2D projection of the patch) can be stored in the corresponding elements of the atlas data V-PCC unit.

[0143] UV mapping is a 3D modeling process that converts a 2D projection onto a 3D patch surface for texture mapping. The letters U and V represent the axes of the 2D projection. The letters X, Y, and Z represent the 3D coordinates of the 3D patch's bounding box.

[0144] Volumetric usability information (VUI) can be used to define aspects related to point cloud rendering in the real world. For example, this information can provide the temporal sampling rate that determines the frame duration. A common use case is in immersive applications where point clouds represent real physical objects and should be rendered in the real world. However, some V-PCC systems do not use information representing the physical size of the samples (3D pixels) in the real world in the V-PCC bitstream.

[0145] In order to provide scaling information in the V-PCC codestream, the VUI parameters representing the scaling of the point cloud frame samples are introduced in the following format. This block defines the physical representation of a single occupancy sample of a 3D grid representing a discrete point cloud frame. The physical size of the block is expressed as a step size:

[0146] SizeStep=vui_num_units_in_block÷vui_block_scale

[0147] The parameters defined in this equation can be carried in the VUI syntax structure and modified accordingly, as shown below.

[0148]

[0149]

[0150] For example, the semantics of these elements may be as follows. vui_scaling_info_present_flag, when equal to 1, indicates that vui_num_units_in_block and vui_block_scale are present in the vui_parameters() syntax structure. vui_scaling_info_present_flag, when equal to 0, indicates that vui_num_units_in_block and vui_block_scale are not present in the vui_parameters() syntax structure. vui_scaling_info_present_flag, when not present in the codestream, may be inferred to be 0. vui_num_units_in_block is the spatial frequency vui_time_scale m -1 The number of spatial units that are run with a size corresponding to one increment of the step counter (called step size). vui_num_units_in_block should be greater than 0. The step size (in m) is equal to the quotient of vui_num_units_in_block divided by vui_block_scale. For example, when the spatial rate of the point cloud frame signal is 1m -1 When , vui_time_scale can be equal to 1,000, vui_num_units_in_block can be equal to 1,000,000, so the step size can be equal to 0.001m or 1mm. vui_block_scale is the number of spatial units passed in one point cloud frame. For example, using 1m -1 The vui_block_scale of the spatial coordinate system in which the spatial frequency distance is measured is 1. The value of vui_block_scale should be greater than 0.

[0151] A VUI parameter that carries the physical size of the sample in 3D space is introduced in the bitstream to enable almost instant and precise placement of augmented objects in real scenes. Existing solutions do not provide this information. Otherwise, scaling information can be derived based on the content type and application, but the scaling result is inaccurate. Therefore, a VUI parameter is added that represents the scaling information of the point cloud sample in the real world. The block size is expressed in the form of units per block (num_units_in_block) and a scaling factor (vui_block_scale), so that the scaling factor expressed in integer form has a flexible range. In addition, an additional VUI parameter flag is added to enable or disable the scaling parameter of the point cloud block. If the scaling parameter does not exist in the point cloud, the block size is inferred to be 1 millimeter (millimeter, mm).

[0152] Figure 10 is a schematic diagram of an exemplary video decoding device 1000. Video decoding device 1000 is suitable for implementing the disclosed examples / embodiments described herein. Video decoding device 1000 includes a downstream port 1020, an upstream port 1050, and / or a transceiver unit (Tx / Rx) 1010, wherein transceiver unit (Tx / Rx) 1010 includes a transmitter and / or a receiver for transmitting data upstream and / or downstream via a network. Video decoding device 1000 also includes a processor 1030, including a logic unit and / or a central processing unit (CPU) for processing data; and a memory 1032 for storing data. Video decoding device 1000 may also include electrical components, optical-to-electrical (OE) components, electrical-to-optical (EO) components, and / or wireless communication components coupled to upstream port 1050 and / or downstream port 1020 for transmitting data via an electrical, optical, or wireless communication network. The video decoding device 1000 may also include an input and / or output (I / O) device 1060 for communicating data with a user. The I / O device 1060 may include an output device, such as a display for displaying video data, a speaker for outputting audio data, etc. The I / O device 1060 may also include an input device, such as a keyboard, a mouse, a trackball, etc., and / or corresponding interfaces for interacting with such output devices. The video decoding device 1000 may also include a graphics processing unit (GPU) 1033, which is a dedicated processor for processing video data. The GPU 1033 may include internal memory and / or may directly process data from the CPU memory.

[0153] Processor 1030 is implemented using hardware and software. Processor 1030 can be implemented as one or more CPU chips, cores (e.g., a multi-core processor), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). Processor 1030 communicates with downlink port 1020, Tx / Rx 1010, uplink port 1050, and memory 1032. Processor 1030 includes a decoding module 1014. Decoding module 1014 implements the disclosed embodiments described herein (e.g., methods 100, 1100, and 1200), which can use point cloud media 500 divided into a set of slices 603 and encoded into occupancy frames 710, geometry frames 720, and atlas frames 730 in a V-PCC codestream 900. Decoding module 1014 can also implement mechanism 800 and any other methods / mechanisms described herein. Furthermore, the decoding module 1014 can implement the codec system 200, the encoder 300, and / or the decoder 400. For example, the decoding module 1014 at the encoder can encode the 3D bounding box ratio and / or a flag indicating the presence of the 3D bounding box ratio into the V-PCC codestream. Furthermore, the decoding module 1014 at the decoder can then use the 3D bounding box ratio to scale the 3D bounding box and the point cloud included in the 3D bounding box. This enables the encoder to specify the real-world size of the point cloud (e.g., expressed in meters). The decoder can then scale the point cloud to an appropriate real-world size, such as to overlay it on other video data. Thus, the decoding module 1014 enables the video decoding device 1000 to provide additional functionality and / or decoding efficiency when decoding video data. Thus, the decoding module 1014 improves the functionality of the video decoding device 1000 and addresses issues unique to the field of video decoding. Furthermore, the decoding module 1014 can transition the video decoding device 1000 to different states. Alternatively, the decoding module 1014 may be implemented as instructions stored in the memory 1032 and executed by the processor 1030 (eg, implemented as a computer program product stored on a non-transitory medium).

[0154] The memory 1032 includes one or more memory types, such as a magnetic disk, a tape drive, a solid-state drive, a read-only memory (ROM), a random access memory (RAM), a flash memory, a ternary content-addressable memory (TCAM), a static random-access memory (SRAM), etc. The memory 1032 can be used as an overflow data storage device to store programs when they are selected for execution and to store instructions and data read during the execution of the programs.

[0155] Figure 11 1 is a flow chart of an exemplary method 1100 for encoding a PCC codestream (e.g., V-PCC codestream 900) to support scaling of a point cloud during reconstruction, for example, according to mechanism 800. An encoder (e.g., codec system 200, encoder 300, and / or video decoding device 1000) may use method 1100 when executing method 100. Thus, method 1100 may be used for point cloud media 500 that is partitioned into a set of slices 603 and encoded into occupancy frames 710, geometry frames 720, atlas frames 730, and / or attribute frames.

[0156] Method 1100 may begin when an encoder receives a video sequence comprising a point cloud. Specifically, the point cloud is represented as a sequence of volumetric frames that depict the point cloud and any changes to the point cloud over time. The encoder then determines, based on user input, to encode the video sequence into a V-PCC bitstream. In step 1101, the encoder projects the point cloud of the volumetric frames onto a 2D surface of a 3D bounding box using orthogonal projection. This 2D surface may be referred to as a 2D bounding box. The projection generates multiple 2D slices representing the point cloud from different angles.

[0157] In step 1103, the encoder encodes the 2D slices into atlas frames and encodes the atlas frames into the V-PCC codestream. These slices may also be referred to as atlases, atlas information, atlas video data, atlas components, etc. An atlas / slice comprises a 2D projection of the luminance and / or chrominance components of a point cloud. In a specific example, slices may be packed into atlas frames. This positions the slices into a square / rectangular frame that can be easily encoded by various standardized encoders (e.g., VVC, AVC, HEVC, etc.). However, due to the irregular nature of the slices, slices typically leave gaps in the atlas frame. Therefore, the atlas frame includes portions without video data.

[0158] Therefore, in step 1105, the encoder generates an occupancy frame of occupied pixels and encodes the occupancy frame into the V-PCC codestream. Occupancy pixels may also be referred to as occupancy information, occupancy data, occupancy components, etc. An occupancy component informs the decoder which samples in the atlas frame are associated with the video data. As used herein, a sample is any portion of the video data that describes one or more points in a point cloud. In step 1105, the encoder also determines a geometry frame of geometry pixels and encodes the geometry frame into the V-PCC codestream. Geometry pixels may also be referred to as geometry information, geometry data, geometry components, etc. A geometry component includes information representing the location of a sample / point in 3D space. Furthermore, points in a point cloud may be associated with one or more attributes. Attributes may be scalar or vector properties that are optionally associated with each point in the volumetric frame of the point cloud, such as color, reflectivity, surface normal, timestamp, material ID, etc. In step 1105, the encoder may also optionally determine an attribute frame of attribute pixels and encode the attribute frame into the V-PCC codestream. Attribute pixels may also be referred to as attribute information, attribute data, attribute components, etc. The attribute component provides characteristics of the sample.

[0159] In step 1107, the encoder may determine a scale to be applied to the point cloud (or portion thereof). For example, the encoder may determine a 3D bounding box scale to be applied to a 3D bounding box. For example, the 3D bounding box scale may include parameters for rescaling samples within the 3D bounding box from a local coordinate system to real-world size. The 3D bounding box scale may represent a scale to be applied to 3D bounding box parameters that may be specified for an object. For example, a volumetric frame of a point cloud may be reconstructed into a 3D bounding box using data from an atlas frame, an occupancy frame, a geometry frame, and / or an attribute frame. This produces one or more point clouds encoded at an arbitrary unitless size according to a local coordinate system (e.g., a Cartesian coordinate system). The 3D bounding box scale may be implemented as a transform that may be applied to scale the entire point cloud within the bounding box by transforming the local coordinate system to a final target 3D coordinate system that describes the real-world size of the object. This allows the point cloud reconstructed at the decoder to be correctly sized so that the reconstructed point cloud can be overlaid onto real-world video (e.g., without requiring significant additional processing by a corresponding application at the decoder and / or without requiring user intervention at the decoder). The 3D bounding box scale can be a factor based on the sample density in the 3D bounding box and the size of the 3D bounding box. For example, the 3D bounding box scale can relate the number of spatial units in the point cloud frame to a step value defined based on the spatial frequency in the 3D bounding box. In a specific example, the 3D bounding box scale can represent the number of steps and the step size (e.g., expressed in meters) equal to the quotient of the desired spatial frequency divided by the number of spatial units in the point cloud frame.

[0160] In step 1109, the encoder may encode the 3D bounding box scale into the V-PCC code stream. In some cases, the 3D bounding box scale may be encoded into the V-PCC code stream as the soi_3d_bounding_box_scale_log2 parameter in the SOI SEI message. In one example, the 3D bounding box may be scaled in meters. Therefore, the encoder may also encode a flag to indicate whether the size of the 3D bounding box is expressed in meters. In a specific example, the flag may be the vui_unit_in_metres_flag in the atlas SPS encoded into the V-PCC code stream. The vui_unit_in_metres_flag may be set to 1 to indicate that the real-world coordinate information is expressed in meters, or set to 0 to indicate that the world coordinates are unitless.

[0161] In step 1111, the encoder may encode a flag into the V-PCC codestream to indicate whether the 3D bounding box information is present in the V-PCC codestream. For example, the value of the flag may indicate whether the 3D bounding box scale is applied to the 3D bounding box. In a specific example, the flag may be encoded as soi_3d_bounding_box_present_flag. soi_3d_bounding_box_present_flag may indicate whether soi_3d_bounding_box_scale_log2 is encoded in the SOI SEI message of the V-PCC codestream. In a specific example, soi_3d_bounding_box_present_flag may be set to 1 to indicate that the 3D bounding box information is present in the current SOI SEI message, or may be set to 0 to indicate that the 3D bounding box information is not present in the SOI SEI message.

[0162] In step 1113, the encoder can store the codestream for transmission to the decoder. The encoder can also send the codestream to the decoder upon request and / or to an intermediate content server for storage as needed. Thus, method 1100 encodes sufficient information into the V-PCC codestream to indicate to the decoder whether to scale the point cloud and how to perform such scaling, for example, for use with real-world-sized video (e.g., in AR).

[0163] Figure 121 is a flow chart of an exemplary method 1200 for decoding a PCC codestream (e.g., V-PCC codestream 900) to reconstruct a scaled point cloud, for example, according to mechanism 800. A decoder (e.g., codec system 200, decoder 400, and / or video decoding device 1000) may use method 1200 when executing method 100. Thus, method 1200 may be used for point cloud media 500 that is partitioned into a set of slices 603 and encoded into occupancy frames 710, geometry frames 720, atlas frames 730, and / or attribute frames.

[0164] Method 1200 may begin when a decoder begins receiving a V-PCC codestream representing decoded data of a point cloud video sequence, for example, as a result of method 1100. In step 1201, the decoder may receive the V-PCC codestream. The codestream may include multiple 2D slices / 2D components / atlases in atlas frames. The codestream may also include occupied pixels, geometry pixels, and / or attribute pixels in occupancy frames, geometry frames, and attribute frames, respectively. Occupied pixels may also be referred to as occupancy information, occupancy data, occupancy components, etc. Occupancy components inform the decoder which samples in the atlas frames are associated with the video data. As used herein, a sample is any portion of the video data that describes one or more points in a point cloud. Geometry pixels may also be referred to as geometry information, geometry data, geometry components, etc. A geometry component includes information representing the position of a sample / point in 3D space. Furthermore, points in a point cloud may be associated with one or more attributes. Attributes may be scalar or vector properties that are optionally associated with each point in a volumetric frame of a point cloud, such as color, reflectivity, surface normal, timestamp, material ID, etc. Attributed pixels may also be referred to as attribute information, attribute data, attribute components, etc. Attribute components (when present) provide attributes of the sample. The V-PCC codestream may also include a 3D bounding box scale, a flag indicating whether 3D bounding box information is present, and / or a flag indicating that the point cloud is expressed in meters.

[0165] In step 1203, the slice is decoded. For example, the decoder may decode the slice / atlas by using inter-frame prediction based on a reference frame and / or intra-frame prediction based on blocks in the same frame.

[0166] In step 1205, the point cloud is reconstructed by converting the slice into a 3D slice coordinate system by projecting it into a 3D bounding box. This can be achieved by converting the slice from a local coordinate system into a 3D slice coordinate system defined by each projection plane of the 3D bounding box (e.g., each of the six sides). In some examples, the 3D slice coordinate system can be transformed into a target 3D coordinate system that operates with real-world space using meters as units. This decoding and reconstruction can be performed using occupancy components, geometry components, and attribute components. For example, a decoder can use occupied pixels to determine which parts of an atlas frame include video data. Those parts of the atlas frame can be decoded to obtain atlases / slices. The geometry pixels in the geometry frame can be used to locate points within the 3D bounding box. The slice / atlas pixels can then be projected onto points in the 3D bounding box. Attribute pixels can also be applied to these points when they exist in the attribute frame.

[0167] In step 1207, a 3D bounding box scale is applied to the 3D bounding box and all included points, for example, based on the value of a flag. In a particular implementation, the 3D bounding box scale may include parameters for rescaling samples in the 3D bounding box from the local coordinate system used during reconstruction to real-world size. The 3D bounding box scale may represent a scale to be applied to 3D bounding box parameters that may be specified for an object.

[0168] For example, in step 1205, the volumetric frame of the point cloud reconstructed in the 3D bounding box can use an arbitrary unitless size based on a local coordinate system (e.g., a Cartesian coordinate system). The 3D bounding box scale can be implemented as a transformation that can be applied to scale the entire point cloud in the bounding box by transforming the local coordinate system to a final target 3D coordinate system that describes objects of real-world size. This allows the point cloud to be correctly sized so that the reconstructed point cloud can be overlaid on the real-world video (e.g., without requiring a large amount of additional processing by the corresponding application at the decoder and / or without requiring user intervention at the decoder). The 3D bounding box scale can be a factor based on the sample density in the 3D bounding box and the size of the 3D bounding box. For example, the 3D bounding box scale can relate the number of spatial units in the point cloud frame to a step value defined based on the spatial frequency in the 3D bounding box. In a specific example, the 3D bounding box scale can represent a number of steps and a step size (e.g., expressed in meters) that is equal to the quotient of the desired spatial frequency divided by the number of spatial units in the point cloud frame. In some cases, the 3D bounding box scale can be obtained from the soi_3d_bounding_box_scale_log2 parameter in the SOI SEI message in the V-PCC codestream.

[0169] In addition, this flag may indicate whether 3D bounding box information is present in the V-PCC codestream. For example, the value of the flag may indicate whether a 3D bounding box scale is applied to the 3D bounding box. In a specific example, the flag may be encoded as soi_3d_bounding_box_present_flag. soi_3d_bounding_box_present_flag may indicate whether soi_3d_bounding_box_scale_log2 is encoded in the SOI SEI message of the V-PCC codestream. In a specific example, soi_3d_bounding_box_present_flag may be set to 1 to indicate that 3D bounding box information is present in the current SOI SEI message, or may be set to 0 to indicate that 3D bounding box information is not present in the SOI SEI message.

[0170] In addition, the 3D bounding box can be scaled in meters. Therefore, the V-PCC codestream can include a flag indicating whether the 3D bounding box size is expressed in meters. In a specific example, this flag can be the vui_unit_in_metres_flag encoded in the atlas SPS in the V-PCC codestream. The vui_unit_in_metres_flag can be set to 1 to indicate that the real-world coordinate information is expressed in meters, or set to 0 to indicate that the world coordinates are unitless.

[0171] After rendering, the point cloud, including the reconstructed atlas frame projected onto the geometric pixels, can also be forwarded for display. Rendering can include positioning the viewport relative to the 3D bounding box. This converts the 3D image into a 2D image that can be viewed on the user's screen. For example, the rendered content can be rendered and combined with real-time video data to create AR content, which is then displayed to the user on a user device such as a smartphone, tablet, or laptop. By using the 3D bounding box scale, the rendered reconstructed point cloud can be directly aligned with the coordinates used by applications that use real-world dimensions and sizes. Therefore, the rendered point cloud can be easily overlaid on the video data of an AR application. As another example, the rendered point cloud can be easily localized in any digital space configured based on real-world dimensions. Therefore, the 3D bounding box scale and associated flags provide additional functionality for encoders and decoders and support improved coding efficiency.

[0172] Figure 131 is a schematic diagram of an example system 1300 for decoding a PCC codestream (e.g., V-PCC codestream 900) to support scaling of a point cloud during reconstruction, for example, according to mechanism 800. System 1300 can be implemented by an encoder and a decoder, such as codec system 200, encoder 300, decoder 400, and / or video decoding device 1000. Thus, system 1300 can be used for point cloud media 500 that is partitioned into a set of slices 603 and encoded into occupancy frames 710, geometry frames 720, atlas frames 730, and / or attribute frames. Furthermore, system 1400 can be used when implementing methods 100, 1100, and / or 1200.

[0173] System 1300 includes a video encoder 1302. Video encoder 1302 includes a using module 1303 for applying orthogonal projection to a 3D bounding box to generate a plurality of 2D slices representing a point cloud. Video encoder 1302 also includes a determining module 1304 for determining a 3D bounding box scale to be applied to the 3D bounding box. Video encoder 1302 also includes an encoding module 1305 for encoding the 2D slices into atlas frames in a bitstream. Encoding module 1305 is further configured to encode the 3D bounding box scale into the bitstream. Video encoder 1302 also includes a storage module 1306 for storing the bitstream for transmission to a decoder. Video encoder 1302 also includes a transmitting module 1307 for transmitting the bitstream to a video decoder 1310. Video encoder 1302 may also be configured to perform any of the steps of method 1100.

[0174] System 1300 also includes a video decoder 1310. Video decoder 1310 includes a receiving module 1311 for receiving a codestream including a plurality of 2D slices and 3D bounding box scales in an atlas frame. Video decoder 1310 also includes a decoding module 1313 for decoding the slices. Video decoder 1310 also includes a reconstruction module 1315 for reconstructing a point cloud by converting the slices into a 3D slice coordinate system defined by each projection plane of the 3D bounding box. Video decoder 1310 also includes an application module 1317 for applying the 3D bounding box scales to the 3D bounding box. Video decoder 1310 also includes a forwarding module 1319 for forwarding the reconstructed point cloud for display. Video decoder 1310 may also be configured to perform any of the steps of method 1200.

[0175] There are a variety of rendering techniques for rendering video data. However, simply applying rendering techniques to PCC video data may result in an inefficient rendering process. For example, many types of video data include a single type of data that can be decoded, transferred from the CPU to the graphics processing unit (GPU), and rendered for display. However, PCC uses multiple types of video data that interact with each other. Therefore, some rendering systems can decode atlas video data, geometry video data, occupancy video data, and / or attribute video data and transfer the atlas video data, geometry video data, occupancy video data, and / or attribute video data from the CPU to the GPU for certain rendering tasks. The data is then transferred back to the CPU for combination. The result is then sent back to the GPU to complete the rendering before display. This movement between the CPU and GPU causes each rendered frame to be transferred multiple times between the CPU buffer and the GPU buffer, which significantly increases latency and resource usage during the rendering process.

[0176] This document also discloses a V-PCC renderer that prevents repeated memory movement during the rendering process. The CPU can be used to decode atlas data (e.g., slices), occupancy data, geometry data (e.g., vertices), and / or attribute data. This data is stored in one or more buffers, such as vertex buffers and / or uniform buffers. GPU shaders can then bind to the data in the buffers to reconstruct a 3D point cloud and render a 2D image from the point cloud. For example, a GPU shader can apply occupied pixels as textures to remove slice data, geometry data, and / or attribute data that are artifacts from the compression process. Another GPU shader can initialize a model matrix and generate vertices in a 3D model from the geometry data. Yet another shader can project slice and / or attribute pixels onto vertices as textures. This can be performed without copying such data from the CPU buffer to the GPU buffer. Transformations can also be used to generate 2D images for display from the 3D point cloud. Textures (e.g., sub-blocks of slices) can be assigned to multiple shaders, which improves the parallel processing capabilities of the GPU. In some examples, an entire frame of a V-PCC image can be rendered with a single draw call to the GPU. This rendering can be performed quickly enough to enable most user devices to use V-PCC content in augmented reality (AR) applications. Thus, the disclosed examples increase the capabilities of the decoder when decoding V-PCC data. Furthermore, the examples reduce the use of processor, memory, and / or GPU resources, thereby increasing the functionality of the decoder.

[0177] Figure 14FIG1 is a diagram illustrating an exemplary set of data units 1400 in a V-PCC codestream. For example, data unit 1400 may be included in V-PCC codestream 900. Thus, data unit 1400 may be used in conjunction with methods 100, 1100, and / or 1200, codec system 200, encoder 300, decoder 400, point cloud media 500, slice 603, occupancy frame 710, geometry frame 720, atlas frame 730, mechanism 800, video decoding device 1000, and / or system 1300.

[0178] Data unit 1400 includes multiple V-PCC units 1410, which are data units that include V-PCC data. V-PCC units 1410 may include a payload and an indicator of the data type included therein. V-PCC units 1410 may be organized into a substream of V-PCC parameter sets 1420. A V-PCC parameter set 1420 is any V-PCC unit 1410 that includes parameters for indicating how to decode the associated video data. V-PCC parameter sets 1420 may include occupancy parameters 1421, geometry parameters 1423, and / or attribute parameters 1425. Occupancy parameters 1421 include data describing how to interpret decoded occupancy video data 1441. Geometry parameters 1423 include data describing how to interpret geometry video data 1443. Attribute parameters 1425 include data describing how to interpret attribute video data 1445. V-PCC parameter sets 1420 may be substantially similar to V-PCC parameter set 911.

[0179] The V-PCC unit 1410 may be further organized into substreams including occupancy video data 1441, geometry video data 1443, and / or attribute video data 1445, which may include data substantially similar to occupancy frames 915, geometry frames 917, and attribute frames 919, respectively. The V-PCC unit 1410 may also include an atlas substream 1430, which may include one or more atlas frames 913. The atlas substream 1430 may include an atlas NAL unit 1431, which is a data unit including atlas data and is sized for transmission over a network. The atlas NAL unit 1431 may include an atlas sequence parameter set (SPS) 1433, an atlas frame parameter set (FPS), an atlas partition group layer 1437, and / or a supplemental enhancement information (SEI) message 1439. The atlas SPS 1433 includes data describing the decoding mechanism settings associated with the entire atlas data sequence. The atlas FPS 1435 includes data describing the decoding mechanism settings associated with one or more corresponding V-PCC frames. The atlas tile group layer 1437 includes data describing the decoding mechanism settings associated with one or more corresponding atlas tile groups, which may include slices or sub-blocks thereof. The SEI message 1439 may include data that is not used to decode the video data but may be used to check whether the codestream conforms to the standard and / or to describe video playback settings.

[0180] Figure 15 15 is a schematic diagram of an exemplary V-PCC renderer 1510. The V-PCC renderer 1510 may be used in the codec system 200, the decoder 400, and / or the video decoding device 1000. Therefore, the V-PCC renderer 1510 may also be used in conjunction with the methods 100, 1100, and / or 1200, the encoder 300, the point cloud media 500, the slice 603, the occupancy frame 710, the geometry frame 720, the atlas frame 730, the mechanism 800, the V-PCC codestream 900, the system 1300, and / or the data unit 1400.

[0181] The V-PCC renderer 1510 is implemented using software and hardware. Therefore, the V-PCC renderer 1510 is a set of components that uses hardware resources to reconstruct a coded point cloud for display to a user, for example, via a display. The V-PCC renderer 1510 may include a camera 1513, which is a component for capturing images and / or videos using a hardware camera (e.g., a digital camera). The V-PCC renderer 1510 may also include an augmented reality (AR) component 1511. The AR component 1513 is used to overlay pre-recorded and / or generated graphics, for example, from the graphics component 1520 onto an image captured by the camera 1513 for display to the user. This enables the AR component 1511 to enhance the user's view of the displayed reality using pre-recorded and / or generated graphics (e.g., V-PCC video).

[0182] The V-PCC renderer 1510 also includes a graphics component 1520, which is a component for decoding and reconstructing the point cloud for display to the user. The graphics component 1520 may include a texture component 1521, a vertex buffer 1523, a uniform buffer 1525, and an application 1527. The texture component 1521 is used to reconstruct various textures from the decoded video. The texture component 1521 may include a video decoder 1529, which is a component for decoding atlas pixels, geometry pixels, occupancy pixels, attribute pixels, and / or associated parameters from the V-PCC codestream. The texture component 1521 can then treat each of the decoded atlas pixels, geometry pixels, occupancy pixels, and / or attribute pixels as a texture when reconstructing a V-PCC frame for display and / or storing such data in memory. The vertex buffer 1523 is a memory component (e.g., in cache memory) for storing vertices from the geometry data. Uniform buffer 1525 is a memory component (e.g., in cache memory) for storing atlas data, occupied pixels, and / or attribute pixels applied to vertices in vertex buffer 1523. Application 1527 is a component for applying atlas data, occupied pixels, and / or attribute pixels to vertices. For example, application 1527 can initialize a model matrix, such as by applying a corresponding transformation, and project the atlas data, occupied pixels, and / or attribute pixels onto the vertices. Thus, graphics component 1520 can decode and reconstruct V-PCC frames for use by AR component 1511, and / or output V-PCC frames from camera component 1513 for rendering by V-PCC renderer 1510.

[0183] The graphics component 1520 can use various application programming interfaces (APIs) to interact with the hardware. For example, the graphics component 1520 can communicate with the hardware through the Open Graphics Library (OPENGL) 1531 and / or VULKAN 1533 (both of which are exemplary graphics APIs).

[0184] In an exemplary implementation, the rendering process can be described as follows. The CPU and / or GPU can begin by initializing global variables, which can include initializing a model matrix. The model matrix can define a transformation from the object's model space to world space. The model matrix can be initialized using the following transformation so that the model fits into a normalized unit square in world space coordinates from (-0.5, -0.5, -0.5) to (0.5, 0.5, 0.5), where b is the bounding box of the model.

[0185] vec3 s = b[1] - b[0];

[0186] mModel=yawPitchRoll(radians(ctx.rot[0]), radians(ctx.rot[1]), radians(ctx.rot[2]))*scale(vec3(1.0f / max(max(sx, sy), sz)))*translate(-0.5f*(b[1]+b[0]));

[0187] This transform translates the model so that it is centered at the origin. The model is then uniformly scaled by 1 / maximum dimension of its bounding box. The transform then rotates the model so that it aligns with the global x, y, and z axes. The camera view matrix can also be initialized. The camera view matrix can be used to create a transformation from world space to camera space. A projection matrix can also be used, which transforms the model from camera space to screen space.

[0188] The transformation matrix can be updated by the CPU, GPU, or a combination thereof as described above. The GPU can then render the point cloud, for example, based on the geometry pixels, atlas pixels, occupancy pixels, and / or attribute pixels decoded by a decoder (e.g., running on the CPU) and stored in a GPU buffer. For each frame, the point cloud can be rendered using the following process. A block-to-slice mapping can be generated. The block-to-slice mapping defines a mapping of blocks in the atlas space to corresponding slices.

[0189] GPU shader programs can be bound to data in GPU buffers. For example, a geometry shader program can be bound to emit points directly onto a model matrix via the GPU. This reduces CPU overhead and improves overall system performance. This concept can also be applied using compute shader methods. GPU textures can also be updated. For example, decoded images can be bound directly from the occupation video stream, geometry video stream, and attribute video stream as GPU textures in the luma-chroma (YUV) color space, without any copies between the CPU and GPU buffers.

[0190] The GPU buffers can then be updated. For example, the uniform buffer (ubo) can be updated with the following data: frame width, frame height, occupied resolution, and model view projection (modelViewProj) matrix. The uniform buffer can be stored in cache memory and can include data that is constant for the corresponding frame. The GPU can then render the slices. In some examples, the GPU can render all slices associated with the frame sequentially using multiple (e.g., six) draw calls. In another example, the GPU can render all slices in parallel using a single draw call. The parallel option can further reduce driver overhead and improve performance.

[0191] An example including rendering slices using separate draw calls can be implemented as follows. For each slice, a vertex buffer can be updated. The vertex buffer defines the blocks mapped to the atlas space of the corresponding slice. A uniform buffer can also be updated for each slice. The uniform buffer can include a patch data ubo (patch_data_ubo) section that includes slice-specific information including the slice 2D offset and size in the atlas space for sampling the texture, the slice 3D offset in the world space, the slice axis, the slice projection mode, and the canvas to slice matrix that defines the mapping from the atlas space to the slice space. This matrix can be used as the inverse matrix of the slice to canvas matrix. Points can also be rendered using draw calls. The primitive count can be set to the number of blocks corresponding to the slice. Primitives (points) are emitted directly on the GPU using a geometry shader or a compute shader.

[0192] An example including rendering a tile using a single draw call can be implemented as follows. A uniform buffer can be updated. The uniform buffer can include the tile data as a global array that includes the canvas / atlas to tile matrix. The vertex buffer can also be updated. In the vertex buffer, each entry can define a tile in atlas space and a corresponding tile index. Points for each tile in a frame (e.g., six tiles per frame) can be rendered onto vertices using a single draw call. The primitive count can be set to the number of tiles. Primitives (points) are emitted directly on the GPU using a geometry shader or a compute shader.

[0193] The GPU rendering process can be implemented as follows. The vertex shader can copy each vertex data to the geometry shader. The geometry shader enables instancing, where multiple vertices are emitted per instance. This reduces CPU overhead. The maximum number of vertices emitted per geometry shader call can be equal to the occupied resolution multiplied by the number of layers. Each call can process a sub-tile in atlas space. Rendering can be performed using the following algorithm.

[0194] For each x,y in the subplot:

[0195] If occupancy_value[x,y]=0, reject;

[0196] Convert atlas coordinates x, y to local patch coordinates u, v;

[0197] For each layer:

[0198] Get geometry value [x, y, layer_index];

[0199] Get attribute value [x, y, layer_index];

[0200] Set point size;

[0201] Launch vertex;

[0202] A rendering mechanism for V-PCC data has been described. This approach is based on using geometry shaders or compute shaders on the GPU and mapping video textures directly to the output. The fact that most of the data is processed on the GPU, rather than decompressed, can increase rendering speed. Rendering can be performed based on the use of geometry shaders or compute shaders. The output of the video decoder is used directly in GPU memory to reduce the processing latency associated with transferring data from the CPU to GPU memory space. This avoids memory copies. Slices can be subdivided into several independent blocks that are sized to be processed by a single geometry shader. This is referred to as instancing in this article. Slice index data is associated with each vertex. This enables essentially instantaneous 3D reconstruction from the corresponding output of the hardware video decoder and the associated decompressed metadata. The early rejection process for rendering is based on the occupancy map and the block-to-slice information carried in the atlas frame.

[0203] Figure 161 is a flow chart of an exemplary method 1600 for rendering a V-PCC codestream (e.g., the V-PCC codestream 900 including the data unit 1400). The method 1600 can be implemented in the V-PCC renderer 1510 running on the codec system 200, the decoder 400, and / or the video decoding device 1000. Therefore, the method 1600 can also be used in conjunction with the methods 100, 1100, and / or 1200, the encoder 300, the point cloud media 500, the slice 603, the occupancy frame 710, the geometry frame 720, the atlas frame 730, the mechanism 800, and / or the system 1300.

[0204] Method 1600 may be implemented on a user equipment (UE). A UE is any device operable by a user to decode and display video content. For example, a UE may include a V-PCC renderer 1510 implemented by a camera, a web browser, a CPU, a CPU buffer, a GPU, a GPU buffer, a receiver, a transmitter, and / or a connected display. The display may be part of the UE or a separate device connected via a wired or wireless connection. Method 1600 may begin by instructing the UE to decode and display a V-PCC image, for example, as part of an augmented reality (AR).

[0205] In step 1601, a CPU running on a decoder may decode a V-PCC codestream. This may include decoding atlas data from the V-PCC codestream. The atlas data may include slices, the geometry data may include vertices, the occupancy data may include occupied pixels, and / or the attribute data may include attribute pixels. After decoding these data, in step 1603, the slice / atlas pixels, vertex / geometry pixels, occupied pixels, and / or attribute pixels are stored in one or more graphics processing unit (GPU) buffers. For example, the CPU may decode the V-PCC codestream and store the data in a CPU buffer. The CPU may issue one or more draw calls to initiate the GPU. As part of the one or more draw calls, slice / atlas pixels, vertex / geometry pixels, occupied pixels, and / or attribute pixels may be sent from one or more CPU buffers to one or more GPU buffers. For example, the GPU buffer may include a uniform buffer for storing slices and a vertex buffer for storing vertices.

[0206] The GPU can then reconstruct a 3D V-PPC point cloud from the data in the GPU buffer and render a 2D image from the point cloud for display to the user without moving the V-PCC data back to the CPU buffer. Keeping the V-PCC data in the GPU buffer during rendering prevents copying data between buffers, reducing decoder / renderer latency and resource usage. As a result, the GPU shader renders slices at the vertex level without copying the slices or vertices to CPU memory. The GPU can render each frame based on a single draw call, or it can render each slice of a frame based on a single draw call.

[0207] In step 1605, the GPU may optionally subdivide one or more slices into sub-blocks to support increased parallel processing. The GPU may then treat each set of V-PCC data as a texture. The GPU may bind a GPU shader to each texture, which may enable each type of V-PCC data to be rendered in parallel and combined as needed without sending the data back to the CPU for further processing. Thus, the GPU binds the slice and / or its sub-blocks, vertices, occupied pixels, and / or attribute data (if any) to corresponding GPU shaders. GPU shaders may include geometry shaders, compute shaders, or a combination thereof.

[0208] In step 1607, the GPU shader applies the occupied pixels to the slices, geometry pixels, and / or attribute pixels in the GPU buffer. In some cases, redundant data is included as atlas data, geometry data, and / or attribute data to improve compression. The occupied pixels can be applied to remove such redundant data that does not include actual video data. Thus, the occupied pixels can be applied to slices, vertices, and / or attribute pixels. Thus, before rendering a slice onto a vertex, the occupied pixels can be applied to reject one or more points associated with a vertex, slice, and / or attribute pixel.

[0209] In step 1609, the GPU and / or corresponding shader can initialize a model matrix to fit the vertices to a normalized unit square around the origin before rendering the slice onto the vertices. The model matrix creates a space to receive the vertices of the point cloud. One or more GPU shaders can then create geometric pixels of the reconstructed point cloud by including the vertices in the model matrix. In some examples, the maximum number of vertices emitted by the geometry shader can be equal to the occupied resolution multiplied by the number of layers of vertices in the geometry data.

[0210] In step 1611, the GPU shader may render the slice onto the vertex. For example, rendering the slice onto the vertex includes projecting the slice onto the point of the vertex. In addition, the GPU shader may render attribute pixels (if any) onto the vertex. This creates a 3D point cloud comprising 3D vertices that include slice textures and / or attribute data. Various transformation matrices may then be applied to the model matrix to generate a 2D image of the appropriate size on the 3D reconstructed point cloud. For example, the transformation matrices may include a canvas to slice matrix for transforming from atlas space to slice space and / or model space, a model matrix for transforming from model space to world space (e.g., with real-world coordinates), a camera view matrix for transforming world space to camera space (e.g., creating a 2D view of the 3D point cloud), and a projection matrix for transforming from camera space to screen space (e.g., using coordinates available to the display).

[0211] In some cases, rendering the slices onto the vertices includes rendering all slices associated with a single V-PCC frame onto the vertices in parallel in response to a single draw call from the CPU. In some cases, rendering the slices onto the vertices includes rendering each slice and / or sub-block associated with the single V-PCC frame onto the vertices in response to a separate draw call from the CPU. Rendering the slices in parallel may include obtaining a slice index for the slice from the GPU buffer and determining a slice orientation for each slice based on the slice index. This enables the GPU to correctly determine the slice orientation and render the slice without querying the CPU. Each slice can be rendered by a different GPU shader. When sub-blocks are used, each sub-block can be rendered by a different GPU shader.

[0212] When using AR, rendering the slices onto the vertices via the transformation matrix may include rendering the slices and vertices onto a recorded image to create AR content. For example, the GPU may receive a recorded image from a connected camera via a draw call. The GPU may then render the V-PCC content onto the recorded image data to create a composite image that includes a substantially real-time recorded image and an overlay rendering of a point cloud pre-recorded and / or pre-generated by an encoder.

[0213] In step 1613 , the GPU may forward the 2D image of the 3D point cloud from one or more GPU buffers to the screen for display.

[0214] A first component is directly coupled to a second component when there are no intervening components between them other than a wire, trace, or other medium. A first component is indirectly coupled to a second component when there are intervening components between them other than a wire, trace, or other medium. The term "coupled" and variations thereof encompass both direct and indirect couplings. Unless otherwise specified, the use of the term "about" is intended to include a range of ±10% of the following figure.

[0215] It should also be understood that the steps of the exemplary methods set forth herein do not necessarily need to be performed in the order described, and the order of the steps of these methods should be understood to be merely exemplary. Similarly, in methods consistent with various embodiments of the present invention, these methods may include other steps, and certain steps may be omitted or combined.

[0216] Although the present invention provides several embodiments, it should be understood that the disclosed systems and methods may be embodied in a variety of other specific forms without departing from the spirit or scope of the present invention. The examples of the present invention are to be considered illustrative rather than restrictive, and the present invention is not limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0217] In addition, the techniques, systems, subsystems, and methods described and illustrated as separate or independent in various embodiments may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present invention. Other changes, substitutions, and modifications can be determined by those skilled in the art and may be made without departing from the spirit and scope of the present invention.

Claims

1. A method implemented by a decoder, characterized in that The method comprises: The receiver of the decoder receives a code stream including a plurality of two-dimensional (2D) slices and three-dimensional (3D) bounding box ratios in an atlas frame; The processor of the decoder decodes the 2D slice; reconstructing a point cloud by converting, by the processor, the 2D slice into a 3D slice coordinate system defined by each projection plane of the 3D bounding box; The processor applies the 3D bounding box scale to the 3D bounding box.

2. The method according to claim 1, characterized in that The code stream further includes a flag indicating whether 3D bounding box information exists.

3. The method according to claim 2, characterized in that The 3D bounding box scale is applied based on the value of the flag.

4. The method according to claim 1, wherein The 3D slice coordinate system is transformed into a target 3D coordinate system in meters.

5. The method according to any one of claims 1 to 4, characterized in that The code stream further includes an occupancy component, a geometric component, and an attribute component, and the point cloud is reconstructed based on the occupancy component, the geometric component, and the attribute component.

6. The method according to claim 5, characterized in that The occupancy component informs the decoder which samples in the 2D component are associated with video data, the geometry component includes information representing the location of the samples in 3D space, and the attribute component provides characteristics of the samples.

7. The method according to claim 1, characterized in that Also included is the processor forwarding the reconstructed atlas frame for display.

8. A method implemented by an encoder, characterized in that The method comprises: The processor of the encoder uses orthogonal projection onto a three-dimensional (3D) bounding box to generate a plurality of two-dimensional (2D) slices representing the point cloud; The processor encodes the 2D slice into an atlas frame in a bitstream; The processor determines a 3D bounding box scale to apply to the 3D bounding box; The processor encodes the 3D bounding box ratio into the bitstream; The memory of the encoder stores the code stream for sending to the decoder.

9. The method according to claim 8, characterized in that The 3D bounding box scale includes parameters for rescaling samples in the 3D bounding box from a local coordinate system to a real-world size.

10. The method according to claim 8 or 9, characterized in that The processor is further included in encoding a flag indicating whether 3D bounding box information exists into the bitstream.

11. The method according to claim 10, characterized in that The value of the flag indicates whether to apply the 3D bounding box scale to the 3D bounding box.

12. The method according to claim 8 or 9, characterized in that The 3D bounding box is scaled in meters.

13. The method according to claim 8 or 9, characterized in that The processor is also included in encoding an occupancy component, a geometry component, and an attribute component into the code stream.

14. The method according to claim 13, wherein: The occupancy component informs the decoder which samples in the atlas frame are associated with video data, the geometry component includes information representing the location of the samples in 3D space, and the attribute component provides characteristics of the samples.

15. A video decoding device, characterized in that: include: A processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, the receiver, the memory, and the transmitter are configured to perform the method according to any one of claims 1 to 14.

16. A non-transitory computer-readable medium, characterized in that A computer program product for use with a video decoding device, wherein the computer program product comprises computer-executable instructions stored in the non-transitory computer-readable medium, and when the computer-executable instructions are executed by a processor, the video decoding device performs the method according to any one of claims 1 to 14.

17. A decoder, characterized in that: include: a receiving module configured to receive a code stream including a plurality of two-dimensional (2D) slices and a three-dimensional (3D) bounding box ratio in an atlas frame; A decoding module, configured to decode the 2D slice; a reconstruction module for reconstructing a point cloud by transforming the 2D slice into a 3D slice coordinate system defined by each projection plane of the 3D bounding box; An application module is configured to apply the 3D bounding box ratio to the 3D bounding box.

18. The decoder according to claim 17, characterized in that The decoder is further configured to perform the method according to any one of claims 1 to 7.

19. An encoder, characterized in that include: Using a module for performing orthogonal projection onto a three-dimensional (3D) bounding box to generate a plurality of two-dimensional (2D) slices representing a point cloud; a determination module for determining a 3D bounding box ratio applied to the 3D bounding box; Encoding module for: Encoding the 2D slice into an atlas frame in a bitstream; encoding the 3D bounding box ratio into the bitstream; The storage module is used to store the code stream for sending to the decoder.

20. The encoder according to claim 19, wherein The encoder is further configured to perform the method according to any one of claims 8 to 14.

21. A computer-readable storage medium, characterized in that The method comprises an encoded code stream obtained according to any one of claims 8 to 14.

Images