Indicating camera parameters in point cloud coding

Abstract

A video coding mechanism is disclosed. The mechanism includes receiving a bitstream including a plurality of two dimensional (2D) patches in a sprite frame and a camera offset of a camera. The patches are decoded and converted to a three dimensional (3D) patch coordinate system to obtain a point cloud frame. An offset matrix is determined according to the camera offset. The offset matrix is then applied to the point cloud frame.

Description

[0001] Related applications cross-application

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 957,674, filed January 6, 2020, entitled “Signaling Camera Parameters In Point Cloud Coding,” which is hereby incorporated by reference in its entirety. Technical Field

[0003] This invention generally relates to video decoding, and more specifically to improvements that support the default viewing position and angle of video point cloud coding (V-PCC). Background Technology

[0004] Even short videos require a significant amount of video data to depict, which can be challenging when streaming or otherwise transmitting data over bandwidth-constrained communication networks. Therefore, video data is typically compressed before transmission over modern telecommunications networks. Due to limited memory resources, the size of the video must be considered when storing it on storage devices. Video compression devices typically encode video data using software and / or hardware at the source side before transmitting or storing it, reducing the amount of data required to represent the digital video image. Video decompression devices then decode the video data and receive the compressed data at the destination side. Given limited network resources and the growing demand for higher video quality, there is a need to improve compression and decompression techniques to increase compression ratios with minimal impact on image quality. Summary of the Invention

[0005] In one embodiment, the present invention includes a method implemented by a decoder, the method comprising: a receiver of the decoder receiving a bitstream, the bitstream including multiple slices in an atlas frame and a camera offset of a camera; a processor of the decoder decoding the slices in the atlas frame; the processor converting the slices into a three-dimensional (3D) slice coordinate system to obtain point cloud content; the processor determining an offset matrix based on the camera offset; and the processor applying the offset matrix to the point cloud content.

[0006] Point cloud coding (PCC) systems decode PCC components to reconstruct 3D point clouds. Then, some PCC systems position a camera at a default location predefined by the decoder. The camera is used to render the 3D point cloud as a two dimensional (2D) image that can be displayed on a flat screen. The camera position defines the portion of the 3D point cloud that is depicted to the user at a corresponding time instant. These systems can require the user to control the camera and move the camera to a preferred position to render the desired content from a preferred angle. This can be problematic for certain applications because the user can not know the best position to view the desired content. For example, a user can watch a PCC video of a football game. The default camera position can be close to the center of the football field. The user can want to watch the PCC video from a position close to a quarterback or a forward, but can not know the position of the quarterback or forward on the field. This results in a time-consuming search, which leads to a poor user experience.

[0007] The present example includes mechanisms for indicating preferred viewing positions and angles. For example, a content creator can select one or more viewing positions and / or viewing angles at the time of encoding. Then, a user can select a preferred viewing position when watching the PCC video. To achieve this, the encoder can encode one or more camera offsets for determining a camera position and corresponding camera rotations for determining a camera viewing angle. The camera offset (e.g., camera position) includes X, Y, Z offsets relative to a predefined point of a bounding box that includes the PCC content. The camera rotation includes a viewing vector that describes a direction (e.g., viewing angle) that the camera points from the position described by the camera offset. The decoder can read the camera offset and camera rotation and then render the PCC video accordingly. For example, the PCC video can be in a model matrix during decoding. The camera information can be applied as a transformation to the model matrix in order to render a 2D image from the 3D point cloud. For example, the camera offset can be used to create an offset matrix and the camera rotation can be used to create a rotation matrix. Then, the offset matrix and the rotation matrix can be applied to the model matrix to transform the PCC content into a 2D image that displays the point cloud from a predetermined position and angle. Thus, the present example increases the functionality of the encoder and decoder by creating predetermined and selectable viewing positions and angles for PCC video content. Furthermore, the present invention supports mechanisms that improve coding efficiency, thus supporting reduced processor, memory, and / or network resource usage in the encoder and / or decoder.

[0008] Optionally, according to any of the above aspects, in another implementation form of this aspect, the camera offset represents a first offset corresponding to an X axis, a second offset corresponding to a Y axis, and a third offset corresponding to a Z axis.

[0009] Optionally, in any of the preceding aspects, in another implementation form of the aspect, the offset matrix comprises four rows and four columns, the first offset is located at the first row and the fourth column, the second offset is located at the second row and the fourth column, and the third offset is located at the third row and the fourth column.

[0010] Optionally, in any of the preceding aspects, in another implementation form of the aspect, the bitstream further comprises a camera rotation of the camera, and the method further comprises: determining, by the processor, a rotation matrix based on the camera rotation; and applying, by the processor, the rotation matrix to the point cloud content.

[0011] Optionally, in any of the preceding aspects, in another implementation form of the aspect, the camera rotation comprises an X component of the rotation of the camera represented in quaternion representation, a Y component of the rotation of the camera represented in quaternion representation, and a Z component of the rotation of the camera represented in quaternion representation.

[0012] Optionally, in any of the preceding aspects, in another implementation form of the aspect, the rotation matrix comprises an X component, a Y component, a Z component, and a W component of the rotation of the camera represented in quaternion representation.

[0013] Optionally, in any of the preceding aspects, in another implementation form of the aspect, the W component is calculated according to the following formula: qW = Sqrt(1-(qX2+qY2+qZ2)), wherein qW, qX, qY, and qZ represent the W component, the X component, the Y component, and the Z component, respectively, and Sqrt is a square root function.

[0014] Optionally, in any of the preceding aspects, in another implementation form of the aspect, the patches in the atlas frame are 2D patches.

[0015] In one embodiment, the present application comprises a method implemented by an encoder, the method comprising: encoding, by the processor, a plurality of 2D patches and a geometry describing a point cloud frame into a point cloud coding (PCC) bitstream; determining, by the processor, a camera offset describing a position of a camera relative to the point cloud frame to support generating an offset matrix for rendering a camera view of the point cloud frame; encoding, by the processor, the camera offset into the PCC bitstream; and storing, in a memory of the encoder, the PCC bitstream, the PCC bitstream being configured to be transmitted to a decoder.

[0016] PCC systems decode the PCC components to reconstruct a 3D point cloud. Then, some PCC systems position a camera at a default location predefined by the decoder. The camera is used to render the 3D point cloud into a 2D image that can be displayed on a flat screen. The camera position defines the portion of the 3D point cloud that is depicted to the user at a corresponding time instant. These systems can require the user to control the camera and move the camera to a preferred position to render the desired content from a preferred angle. This can be problematic for certain applications because the user can not know the best position to view the desired content. For example, a user can be watching a PCC video of a football game. The default camera position can be close to the center of the football field. The user can want to watch the PCC video from a position close to a quarterback or a forward, but can not know the position of the quarterback or forward on the field. This results in a time-consuming search, which leads to a poor user experience.

[0017] The present example includes a mechanism for indicating a preferred viewing position and angle. For example, a content creator can select one or more viewing positions and / or viewing angles at the time of encoding. Then, a user can select a preferred viewing position when watching the PCC video. To achieve this, the encoder can encode one or more camera offsets for determining a camera position and a corresponding camera rotation for determining a camera viewing angle. The camera offset (e.g., camera position) includes an X, Y, Z offset relative to a predefined point of a bounding box that includes the PCC content. The camera rotation includes a viewing vector that describes a direction (e.g., viewing angle) that the camera is pointing from the position described by the camera offset. The decoder can read the camera offset and camera rotation and then render the PCC video accordingly. For example, the PCC video can be in a model matrix during decoding. The camera information can be applied as a transformation to the model matrix in order to render a 2D image from the 3D point cloud. For example, the camera offset can be used to create an offset matrix and the camera rotation can be used to create a rotation matrix. Then, the offset matrix and the rotation matrix can be applied to the model matrix to transform the PCC content into a 2D image that displays the point cloud from a predetermined position and angle. Thus, the present example increases the functionality of the encoder and decoder by creating predetermined and selectable viewing positions and angles for PCC video content. Furthermore, the present invention supports mechanisms that improve coding efficiency, thus supporting reduced processor, memory, and / or network resource usage in the encoder and / or decoder.

[0018] Optionally, according to any of the above aspects, in another implementation form of this aspect, the camera offset represents a first offset corresponding to an X axis, a second offset corresponding to a Y axis, and a third offset corresponding to a Z axis.

[0019] Optionally, in any of the preceding aspects, in another implementation form of the aspect, the offset matrix comprises four rows and four columns, the first offset is located at the first row and the fourth column, the second offset is located at the second row and the fourth column, and the third offset is located at the third row and the fourth column.

[0020] Optionally, in any of the preceding aspects, in another implementation form of the aspect, the method further comprises: determining, by the processor, a camera rotation describing a viewing vector of the camera with respect to the point cloud frame to support generating a rotation matrix for rendering a camera view of the point cloud frame; and encoding, by the processor, the camera rotation into the PCC bitstream.

[0021] Optionally, in any of the preceding aspects, in another implementation form of the aspect, the camera rotation comprises an X component of the rotation of the camera represented in quaternion representation, a Y component of the rotation of the camera represented in quaternion representation, and a Z component of the rotation of the camera represented in quaternion representation.

[0022] Optionally, in any of the preceding aspects, in another implementation form of the aspect, the rotation matrix comprises an X component of the rotation of the camera represented in quaternion representation, a Y component of the rotation of the camera represented in quaternion representation, a Z component of the rotation of the camera represented in quaternion representation, and a W component of the rotation of the camera represented in quaternion representation.

[0023] Optionally, in any of the preceding aspects, in another implementation form of the aspect, the W component is calculated according to the following formula: qW = Sqrt(1-(qX2+qY2+qZ2)), wherein qW, qX, qY and qZ represent the W component, the X component, the Y component and the Z component, respectively, and Sqrt is a square root function.

[0024] In an embodiment, the present application comprises a video coding device, the video coding device comprising: a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, the receiver, the memory and the transmitter are configured to perform the method of any of the preceding aspects.

[0025] In an embodiment, the present application comprises a non-transitory computer readable medium comprising a computer program product for use by a video coding device, wherein the computer program product comprises computer executable instructions stored in the non-transitory computer readable medium such that when executed by a processor cause the video coding device to perform the method according to any of the preceding aspects.

[0026] In an embodiment, the present application comprises a decoder, the decoder comprising: a receiving module configured to receive a bitstream comprising a plurality of patches in an atlas frame and camera offsets of a camera; a decoding module configured to decode the patches; a converting module configured to convert the patches to a 3D patch coordinate system to obtain a point cloud frame; a determining module configured to determine an offset matrix according to the camera offsets; and an applying module configured to apply the offset matrix to the point cloud frame.

[0027] PCC systems decode the PCC components to reconstruct a 3D point cloud. Then, some PCC systems position a camera at a default location predefined by the decoder. The camera is used to render the 3D point cloud into a 2D image that can be displayed on a flat screen. The camera position defines the portion of the 3D point cloud that is depicted to the user at a corresponding time instant. These systems can require the user to control the camera and move the camera to a preferred position to render the desired content from a preferred angle. This can be problematic for some applications because the user can not know the best position to view the desired content. For example, a user can watch a PCC video of a football game. The default camera position can be close to the center of the football field. The user can want to watch the PCC video from a position close to a quarterback or a forward, but can not know the position of the quarterback or the forward on the field. This results in a time-consuming search, which leads to a poor user experience.

[0028] The present example includes mechanisms for indicating preferred viewing positions and angles. For example, a content creator can select one or more viewing positions and / or viewing angles at the time of encoding. Then, a user can select a preferred viewing position when watching the PCC video. To achieve this, the encoder can encode one or more camera offsets for determining a camera position and corresponding camera rotations for determining a camera viewing angle. The camera offset (e.g., camera position) includes X, Y, Z offsets relative to predefined points of a bounding box that includes the PCC content. The camera rotation includes a viewing vector that describes a direction (e.g., viewing angle) that the camera points from the position described by the camera offset. The decoder can read the camera offset and the camera rotation and then render the PCC video accordingly. For example, the PCC video can be in a model matrix during decoding. The camera information can be applied as a transformation to the model matrix in order to render a 2D image from the 3D point cloud. For example, the camera offset can be used to create an offset matrix and the camera rotation can be used to create a rotation matrix. Then, the offset matrix and the rotation matrix can be applied to the model matrix to transform the PCC content into a 2D image that displays the point cloud from a predetermined position and angle. Thus, the present example increases the functionality of the encoder and the decoder by creating predetermined and selectable viewing positions and angles for PCC video content. Furthermore, the present invention supports mechanisms that improve coding efficiency, thus supporting reduced processor, memory, and / or network resource usage in the encoder and / or the decoder.

[0029] Optionally, according to any of the above aspects, in another implementation form of the aspect, the decoder is further configured to perform the method according to any of the above aspects.

[0030] In one embodiment, the present disclosure includes an encoder comprising: a determining module for determining a camera offset describing a position of a camera relative to a point cloud frame to support generating an offset matrix for rendering a camera view of the point cloud frame; an encoding module for: encoding a plurality of 2D patches and geometry describing the point cloud frame into a PCC bitstream; encoding the camera offset into the PCC bitstream; a storing module for storing the bitstream for transmission to a decoder.

[0031] PCC systems decode PCC components to reconstruct a 3D point cloud. Then, some PCC systems position a camera at a default position predefined by the decoder. The camera is used to render the 3D point cloud into a 2D image that can be displayed on a flat screen. The camera position defines the portion of the 3D point cloud that is depicted to the user at a corresponding time instant. These systems can require the user to control the camera and move the camera to a preferred position to render the desired content from a preferred angle. This can be problematic for certain applications because the user can not know the best position to view the desired content. For example, a user can watch a PCC video of a football game. The default camera position can be close to the center of the football field. The user can want to watch the PCC video from a position close to a quarterback or a forward, but can not know the position of the quarterback or forward on the field. This results in a time-consuming search, thereby degrading the user experience.

[0032] The present example includes a mechanism for indicating a preferred viewing position and angle. For example, a content creator can select one or more viewing positions and / or viewing angles at the time of encoding. Then, a user can select a preferred viewing position when watching the PCC video. To achieve this, an encoder can encode one or more camera offsets for determining a camera position and corresponding camera rotations for determining a camera viewing angle. The camera offset (e.g., camera position) includes an X, Y, Z offset relative to a predefined point of a bounding box that includes the PCC content. The camera rotation includes a viewing vector that describes a direction (e.g., viewing angle) that the camera is pointing from the position described by the camera offset. A decoder can read the camera offset and camera rotation and then render the PCC video accordingly. For example, the PCC video can be in a model matrix during decoding. The camera information can be applied as a transformation to the model matrix in order to render a 2D image from the 3D point cloud. For example, the camera offset can be used to create an offset matrix and the camera rotation can be used to create a rotation matrix. Then, the offset matrix and the rotation matrix can be applied to the model matrix to transform the PCC content into a 2D image that displays the point cloud from a predetermined position and angle. Thus, the present example increases the functionality of the encoder and decoder by creating predetermined and selectable viewing positions and angles for PCC video content. Furthermore, the present disclosure supports mechanisms that improve coding efficiency, thereby supporting reduced processor, memory, and / or network resource usage in the encoder and / or decoder.

[0033] Optionally, in any of the preceding aspects, in another implementation form of the aspect, the encoder is further configured to perform the method according to any of the preceding aspects.

[0034] For clarity, any of the above embodiments can be combined with any one or more of the above embodiments or previous embodiments to create new embodiments within the scope of the present application.

[0035] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF DRAWINGS

[0036] For a more complete understanding of the present application, reference is now made to the following brief description of the drawings and the detailed description, taken in conjunction with the accompanying drawings and in which like reference numerals represent like parts.

[0037] Figure 1 Flowchart of an example method of coding a video signal.

[0038] Figure 2 Diagram of an example encoding and decoding (codec) system for video coding.

[0039] Figure 3 Diagram of an example video encoder.

[0040] Figure 4 Diagram of an example video decoder.

[0041] Figure 5 Example of a point cloud media that can be coded according to a point cloud compression (PCC) mechanism.

[0042] Figure 6 Example of a patch created from a point cloud.

[0043] Figure 7A Example occupancy frame associated with a set of patches is shown.

[0044] Figure 7B Example geometry frame associated with a set of patches is shown.

[0045] Figure 7C Example atlas frame associated with a set of patches is shown.

[0046] Figure 8 Example mechanism to define camera positions and view vectors to render V-PCC frames.

[0047] Figure 9 Diagram of an example V-PCC bitstream to indicate camera parameters for rendering a point cloud.

[0048] Figure 10 Schematic diagram of an example video coding device.

[0049] Figure 11 Flowchart of an example method of encoding a V-PCC bitstream to indicate camera parameters for rendering a point cloud.

[0050] Figure 12 Flowchart of an example method of decoding a V-PCC bitstream to render a point cloud according to indicated camera parameters.

[0051] Figure 13 Schematic diagram of an example system for indicating camera parameters for rendering a point cloud. DETAILED DESCRIPTION

[0052] It should be understood at the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or in existence. The disclosure should not be limited to the illustrative implementations set forth below, but should be given the broadest possible interpretation accessible under the statutes and regulations.

[0053] The following terms are defined as follows, unless used in a contrary context herein. Specifically, the following definitions are intended to provide more clarity to the present disclosure. However, the terms can be described differently in different contexts. Accordingly, the following definitions should be considered as supplemental information and should not be considered as limiting any other definitions provided herein for these terms.

[0054] An encoder is a device that employs an encoding process to compress video data into a bitstream. A decoder is a device that employs a decoding process to reconstruct video data from a bitstream for display. A point cloud / point cloud representation is a set of points (e.g., samples) in a three-dimensional (3D) space, where each point can include a position, a color, and / or an attribute. A bitstream is a sequence of bits that includes video data compressed for transmission between an encoder and a decoder. In the PCC context, a bitstream includes a sequence of bits of coded V-PCC components and associated parameters.

[0055] A V-PCC component (or more generally, a PCC component) can be a particular type of atlas data, occupancy data, geometry data, or attribute data associated with a V-PCC point cloud. An atlas / atlas frame can be a collection of two-dimensional (2D) bounding boxes (also referred to as tiles) projected to a rectangular frame, which corresponds to a volume (referred to as a 3D bounding box) in 3D space on which the volume data is rendered. Each 2D bounding box / tile represents a subset of the point cloud. A tile is a 2D rectangular region in an atlas associated with volume information. Projection is the rendering of an image (e.g., a tile) on a surface. A 3D bounding box is a volume defined as a cuboid entity with six rectangular faces placed at right angles. Occupancy is a set of values representing whether an atlas sample corresponds to an associated sample in 3D space. An occupancy frame can be a collection of occupancy values constituting a 2D array and representing the entire occupancy information of a single atlas frame. Thus, an occupancy frame can represent whether each sample location in an atlas corresponds to a valid 3D point in a point cloud representation. Geometry is a set of Cartesian coordinates associated with a volume frame. A geometry map is a frame that includes geometry tile information projected at a particular depth. Attributes can be scalar or vector attributes, such as color, reflectance, surface normal, timestamp, material identifier (ID), etc., that are optionally associated with each point in a volume frame of a point cloud. A volume frame, also referred to as a point cloud frame, is a set of 3D points represented by Cartesian coordinates and zero or more corresponding sets of attributes at a particular time instant. Point cloud content is any data included in a point cloud frame. The complete set of atlas data, occupancy data, geometry map, or attributes associated with a particular time instant / volume frame can be referred to as an atlas frame, an occupancy frame, a geometry frame, and an attribute frame, respectively. Atlas data, occupancy data, geometry data, or attribute data can be components of a point cloud, and thus can be referred to as an atlas component, an occupancy component, a geometry component, and an attribute frame component, respectively.

[0056] A camera is a viewport that provides a 2D view of a 3D object. A camera object can be used to render a 2D representation of a 3D point cloud at a corresponding instant. A camera and / or a camera object can be implemented as a camera model, which is one or more matrices that include a camera position and / or a camera rotation. A coordinate system is a mechanism that uses a number of numerical names to uniquely determine the location of a point in space. A 3D coordinate system is a coordinate system that describes a location in 3D space. A camera offset is a parameter that represents a distance in 3D space between a camera position and a predefined point of a 3D bounding box that includes a point cloud. An offset matrix is a matrix representation of a camera offset that can be used to transform a 3D model of a point in a model matrix into a 2D representation. A matrix row is a set of numbers that extends horizontally from the left side of a matrix to the right side of the matrix. A camera rotation is a parameter that represents a viewing vector from a camera that points to a 3D bounding box that includes a point cloud. A rotation matrix is a matrix representation of a camera rotation that can be used to transform a 3D model of a point in a model matrix into a 2D representation. Quaternion notation is a vector representation in the complex number domain using a scalar value and three unit vectors along the axes of space.

[0057] V-PCC is a mechanism that efficiently codes a three dimensional (3D) object represented by a 3D cloud of points of different colors. V-PCC is a type of PCC, and thus these terms can be used interchangeably in most cases. The point cloud is captured over time and included in PCC components. The PCC components are then encoded. The position of each active point in the cloud at a certain instant is stored as a geometry map in a geometry frame. The colors and luminances are stored as patches. Specifically, the patches at a certain instant are packed into an atlas frame. The patches typically do not cover the entire atlas frame. Therefore, an occupancy frame is also generated, which indicates which parts of the atlas frame include patch data. Optionally, attributes of the points, such as transparency, can be included in an attribute frame. Thus, each PCC frame can be encoded as multiple frames that include different components, which describe the point cloud at the corresponding instant.

[0058] PCC systems decode the PCC components to reconstruct a 3D point cloud. Then, some PCC systems position a camera at a default location predefined by the decoder. The camera is used to render the 3D point cloud into a two dimensional (2D) image that can be displayed on a flat screen. The camera position defines the portion of the 3D point cloud that is depicted to the user at a corresponding time instant. These systems can require the user to control the camera and move the camera to a preferred position to render the desired content from a preferred angle. This can be problematic for some applications because the user can not know the best position to view the desired content. For example, a user can be watching a PCC video of a football game. The default camera position can be close to the center of the football field. The user can want to watch the PCC video from a position close to a quarterback or a forward, but can not know the position of the quarterback or forward on the field. This results in a time consuming search, which leads to a poor user experience.

[0059] Disclosed herein are mechanisms for indicating preferred viewing positions and angles. For example, a content creator can select one or more viewing positions and / or viewing angles at the time of encoding. Then, a user can select a preferred viewing position when watching the PCC video. To achieve this, the encoder can encode one or more camera offsets for determining a camera position and corresponding camera rotations for determining a camera viewing angle. The camera offset (e.g., camera position) includes X, Y, Z offsets relative to a predefined point of a bounding box that includes the PCC content. The camera rotation includes a viewing vector that describes a direction (e.g., viewing angle) that the camera is pointing from the position described by the camera offset. The decoder can read the camera offset and camera rotation and then render the PCC video accordingly. For example, the PCC video can be in a model matrix during decoding. The camera information can be applied as a transformation to the model matrix in order to render a 2D image from the 3D point cloud. For example, the camera offset can be used to create an offset matrix and the camera rotation can be used to create a rotation matrix. Then, the offset matrix and the rotation matrix can be applied to the model matrix to transform the PCC content into a 2D image that displays the point cloud from a predetermined position and angle. Thus, the present example increases the functionality of the encoder and decoder by creating predetermined and selectable viewing positions and angles for PCC video content. Furthermore, the present invention supports mechanisms that improve coding efficiency, thus supporting reduced processor, memory, and / or network resource usage in the encoder and / or decoder.

[0060] Figures 1 to 4 Various coding mechanisms for encoding and decoding video are described. Figure 5 To FIG. 7 describes mechanisms for converting a point cloud into a format that can be encoded and decoded by the mechanisms described in FIGs. 1 through Figure 4 To FIG. 7 describes mechanisms for converting a point cloud into a format that can be encoded and decoded by the mechanisms described in FIGs. 1 through

[0061] Figure 1 A flowchart of an example method 100 of coding a video signal. Specifically, a video signal is encoded on the encoder side. The encoding process compresses the video signal using various mechanisms to reduce the video file size. Smaller file sizes facilitate sending the compressed video file to users while reducing the associated bandwidth overhead. A decoder then decodes the compressed video file to reconstruct the original video signal for display to an end user. The decoding process generally corresponds to the encoding process to facilitate the decoder consistently reconstructing the video signal.

[0062] In step 101, a video signal is input to an encoder. For example, the video signal can be an uncompressed video file stored in memory. As another example, the video file can be captured by a video capture device, such as a video camera, and encoded to support live viewing of the video. The video file can include an audio component and a video component. The video component includes a series of image frames that, when viewed in order, produce the visual effect of motion. The frames include pixels represented in terms of light (referred to herein as luma components (or luma samples)) and color (referred to as chroma components (or chroma samples)). In some examples, the frames can also include depth values to support three-dimensional viewing.

[0063] In step 103, the video is partitioned into blocks. The partitioning includes subdividing the pixels in each frame into square and / or rectangular blocks for compression. For example, in high efficiency video coding (HEVC) (also referred to as H.265 and MPEG-H Part 2), a frame can first be divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64x64 pixels). The CTUs include luma samples and chroma samples. The CTUs can be divided into blocks using a coding tree, which recursively subdivides the blocks until a configuration is obtained that supports further encoding. For example, the luma component of a frame can be subdivided until individual blocks include relatively uniform luma values. In addition, the chroma component of a frame can be subdivided until individual blocks include relatively uniform color values. Thus, the partitioning mechanism varies depending on the content of the video frame.

[0064] In step 105, various compression mechanisms are used to compress the image patches segmented in step 103. For example, inter-frame prediction and / or intra-frame prediction can be used. Inter-frame prediction aims to take advantage of the fact that objects in common scenes tend to appear in consecutive frames. Therefore, there is no need to repeatedly describe the blocks depicting objects in the reference frame in adjacent frames. Specifically, an object (such as a table) can remain in a constant position across multiple frames. Therefore, the table is described only once, and adjacent frames can re-reference the reference frame. Pattern matching mechanisms can be used to match objects in multiple frames. Furthermore, moving objects can be represented across multiple frames due to object movement or camera movement, etc. In a specific example, video can be displayed as a moving car on the screen across multiple frames. Motion vectors can be used to describe this movement. A motion vector is a two-dimensional vector that provides the offset from the coordinates of an object in a frame to the coordinates of that object in a reference frame. Therefore, inter-frame prediction can encode image patches in the current frame into a set of motion vectors representing the offset relative to the corresponding block in the reference frame.

[0065] Intra-frame prediction encodes blocks within a common frame. Intra-frame prediction leverages the fact that luma and chroma components tend to cluster within a frame. For example, a patch of green in a section of a tree is often adjacent to several similar patches of green. Intra-frame prediction uses various directional prediction modes (e.g., 33 modes in HEVC), planar modes, and direct current (DC) modes. A directional mode indicates that the current block is similar / identical to samples of neighboring blocks in the corresponding direction. A planar mode indicates that a series of blocks in a row / column (e.g., a plane) can be interpolated based on neighboring blocks at row edges. In effect, the planar mode represents a smooth transition of light / color along a row / column using a relatively constant slope of the changing values. The DC mode is used for boundary smoothing, indicating that the block is similar / identical to the average value associated with samples of all neighboring blocks, which are associated with the angular direction of the directional prediction mode. Therefore, intra-frame predicted blocks can represent image blocks as values ​​of various relational prediction modes rather than actual values. Furthermore, inter-frame predicted blocks can represent image blocks as motion vector values ​​rather than actual values. In either case, the predicted block may not accurately represent the image block in some situations. All differences are stored in residual blocks. The residual blocks can be transformed to further compress the file.

[0066] In step 107, various filtering techniques can be applied. In HEVC, filters are applied according to an in-loop filtering scheme. The block-based prediction described above can result in the creation of a blocky image at the decoder side. In addition, the block-based prediction scheme can encode the blocks and then reconstruct the encoded blocks for later use as reference blocks. The in-loop filtering scheme iteratively applies a noise suppression filter, a deblocking filter, a sample adaptive offset (SAO) filter, and an adaptive loop filter to the blocks / frames. These filters reduce such block artifacts so that the encoded file can be accurately reconstructed. In addition, these filters reduce artifacts in the reconstructed reference blocks so that artifacts are less likely to create other artifacts in subsequent blocks encoded from the reconstructed reference blocks.

[0067] In step 109, once the video signal has been partitioned, compressed, and filtered, the resulting data is encoded into a bitstream. The bitstream includes the above-described data as well as any signaling data needed to support proper video signal reconstruction at the decoder side. For example, such data can include partitioning data, prediction data, residual blocks, and various flags that provide coding instructions to the decoder. The bitstream can be stored in memory for transmission to a decoder upon request. The bitstream can also be broadcast and / or multicast to multiple decoders. Creating the bitstream is an iterative process. Thus, steps 101, 103, 105, 107, and 109 can be performed consecutively and / or simultaneously in multiple frames and blocks. Figure 1 The order shown is presented for clarity and ease of description and is not intended to limit the video coding process to a particular order.

[0068] In step 111, the decoder receives the bitstream and begins the decoding process. Specifically, the decoder uses an entropy decoding scheme to convert the bitstream into corresponding syntax data and video data. In step 111, the decoder uses the syntax data in the bitstream to determine the partitioning of the frame. The partitioning should match the block partitioning result in step 103. Entropy encoding / decoding used in step 111 is described herein. The encoder makes a number of selections during the compression process, such as selecting a block partitioning scheme from a number of possible selections based on the spatial location of values in one or more input images. Indicating the exact selection can use a large number of bits. As used herein, a bit is a binary value that is treated as a variable (e.g., a bit value that can vary depending on context). Entropy encoding allows the encoder to discard any options that are clearly not a good fit for the particular situation, leaving a set of usable options. Then, a codeword is assigned to each of the usable options. The length of the codeword depends on the number of usable options (e.g., one bit for two options, two bits for three or four options, etc.). The encoder then encodes the codeword for the selected option. This scheme reduces the size of the codeword, as the size of the codeword is as large as is needed to uniquely represent one option from a small subset of usable options, rather than an option from a potentially large set of all possible options. The decoder then decodes the selection by determining the set of usable options in a similar manner to the encoder. By determining the set of usable options, the decoder can read the codeword and determine the selection made by the encoder.

[0069] In step 113, the decoder performs block decoding. Specifically, the decoder performs an inverse transform to generate a residual block. Then, the decoder uses the residual block and corresponding prediction blocks to reconstruct the image block according to the partitioning. The prediction blocks can include intra-prediction blocks and inter-prediction blocks generated on the encoder side in step 105. The reconstructed image block is then positioned into the frame of the reconstructed video signal according to the partitioning data determined in step 111. Syntax for step 113 can also be indicated in the bitstream through entropy encoding described above.

[0070] In step 115, the frames of the reconstructed video signal are filtered on the encoder side in a similar manner to step 107. For example, noise suppression filters, deblocking filters, adaptive loop filters, and SAO filters can be applied to the frames to remove blocking artifacts. Once the frames are filtered, in step 117, the video signal can be output to a display for viewing by an end user.

[0071] Figure 2A schematic diagram of an exemplary encoding and decoding (codec) system 200 for video coding. In particular, the codec system 200 can implement the operational method 100. The codec system 200 broadly represents the components used in an encoder and a decoder. The codec system 200 receives a video signal and partitions the video signal as described in steps 101 and 103 in the operational method 100, resulting in a partitioned video signal 201. Then, when acting as an encoder, the codec system 200 compresses the partitioned video signal 201 into an encoded bitstream as described in steps 105, 107, and 109 in the method 100. When acting as a decoder, the codec system 200 generates an output video signal from the bitstream as described in steps 111, 113, 115, and 117 in the operational method 100. The codec system 200 includes a general coder control component 211, a transform scaling and quantization component 213, an intra estimation component 215, an intra prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded picture buffer component 223, a header formatting and context adaptive binary arithmetic coding (CABAC) component 231. These components are coupled as shown. In Figure 2 In the operational method 100, the black lines represent movement of data to be encoded / decoded, and the dashed lines represent movement of control data that controls the operation of the other components. The components of the codec system 200 can all be used in an encoder. A decoder can include a subset of the components of the codec system 200. For example, a decoder can include the intra prediction component 217, the motion compensation component 219, the scaling and inverse transform component 229, the in-loop filter component 225, and the decoded picture buffer component 223. These components are described herein.

[0072] Partitioned video signal 201 is a captured video sequence that has been partitioned into blocks of pixels by a coding tree. The coding tree uses various partitioning modes to subdivide the blocks of pixels into smaller blocks of pixels. These blocks can then be further subdivided into smaller blocks. These blocks can be referred to as nodes on the coding tree. Larger parent nodes are partitioned into smaller child nodes. The number of times a node is subdivided is referred to as the depth of the node / coding tree. In some cases, a coding unit (CU) can include the partitioned blocks. For example, a CU can be a sub-portion of a CTU that includes a luma block, a red-difference chroma (Cr) block, and a blue-difference chroma (Cb) block, and corresponding syntax instructions for the CU. The partitioning modes can include binary tree (BT), triple tree (TT), and quad tree (QT) for partitioning a node into two, three, or four child nodes, respectively, of different shapes depending on the partitioning mode used. Partitioned video signal 201 is forwarded to general coder control component 211, transform scaling and quantization component 213, intra estimation component 215, filter control analysis component 227, and motion estimation component 221 for compression.

[0073] General coder control component 211 is used to make decisions related to encoding images of a video sequence into a bitstream according to application constraints. For example, general coder control component 211 manages the optimization of bit rate / bitstream size versus reconstructed quality. Such decisions can be made according to storage space / bandwidth availability and image resolution requests. General coder control component 211 also manages buffer usage according to transmission speed to alleviate buffer underflow and overflow problems. To address these problems, general coder control component 211 manages partitioning, prediction, and filtering by the other components. For example, general coder control component 211 can dynamically increase compression complexity to increase resolution and increase bandwidth usage, or decrease compression complexity to decrease resolution and bandwidth usage. Thus, general coder control component 211 controls the other components of codec system 200 to balance video signal reconstructed quality versus bit rate problems. General coder control component 211 creates control data that controls the operation of the other components. The control data is also forwarded to header formatting and CABAC component 231 for encoding into the bitstream to indicate parameters for decoding in a decoder.

[0074] The partitioned video signal 201 is also sent to a motion estimation component 221 and a motion compensation component 219 for inter-prediction. Frames or slices of the partitioned video signal 201 can be divided into video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter-predictive coding on received video blocks relative to one or more blocks in one or more reference frames to provide temporal prediction. The codec system 200 can perform a number of coding processes in order to select an appropriate coding mode for each video data block, among other things.

[0075] The motion estimation component 221 and the motion compensation component 219 can be highly integrated, but are shown separately for conceptual purposes. Motion estimation performed by the motion estimation component 221 is a process of generating motion vectors that are used to estimate the motion of video blocks. For example, a motion vector can represent a displacement of a coding object relative to a prediction block. A prediction block is a block that is found to highly match a block to be coded in terms of pixel difference. A prediction block can also be referred to as a reference block. Such pixel difference can be determined by a sum of absolute difference (SAD), a sum of square difference (SSD), or other difference metrics. HEVC uses several coding objects, including a CTU, a coding tree block (CTB), and a CU. For example, a CTU can be divided into multiple CTBs, which can then be divided into multiple CBs for inclusion in a CU. A CU can be encoded as a prediction unit (PU) that includes prediction data and / or a transform unit (TU) that includes transform residual data of the CU. The motion estimation component 221 generates motion vectors, PUs, and TUs using rate-distortion analysis as part of a rate-distortion optimization process. For example, the motion estimation component 221 can determine multiple reference blocks, multiple motion vectors, etc. for a current block / frame, and can select a reference block, a motion vector, etc. that has the best rate-distortion characteristics. The best rate-distortion characteristics balance the quality of video reconstruction (e.g., amount of data loss due to compression) and coding efficiency (e.g., size of the final encoding).

[0076] In some examples, the coding system 200 can compute values for sub-integer pixel positions of reference pictures stored in the decoded picture buffer component 223. For example, the video coding system 200 can interpolate values for quarter-pel positions, eighth-pel positions, or other fractional-pel positions of reference pictures. Thus, the motion estimation component 221 can perform a motion search with respect to both full-pel positions and fractional-pel positions and output motion vectors with fractional-pel precision. The motion estimation component 221 computes motion vectors for PUs of a video block in an inter-coded slice by comparing the location of the PUs to the location of the prediction blocks of the reference pictures. The motion estimation component 221 outputs the computed motion vectors as motion data to the header formatting and CABAC component 231 for encoding and to the motion compensation component 219.

[0077] The motion compensation performed by the motion compensation component 219 can involve fetching or generating a prediction block according to the motion vectors determined by the motion estimation component 221. Additionally, in some examples, the motion estimation component 221 and the motion compensation component 219 can be functionally integrated. After receiving a motion vector for a PU of a current video block, the motion compensation component 219 can locate the prediction block to which the motion vector points. Then, by subtracting the pixel values of the prediction block from the pixel values of the current video block being coded, pixel difference values are formed, thereby forming a residual video block. Generally, the motion estimation component 221 performs motion estimation with respect to the luma component, and the motion compensation component 219 uses the motion vectors computed from the luma component for the chroma and luma components. The prediction block and the residual block are forwarded to the transform scaling and quantization component 213.

[0078] The partitioned video signal 201 is also sent to the intra-estimation component 215 and the intra-prediction component 217. Like the motion estimation component 221 and the motion compensation component 219, the intra-estimation component 215 and the intra-prediction component 217 can be highly integrated, but are shown separately for conceptual purposes. The intra-estimation component 215 and the intra-prediction component 217 perform intra-prediction with respect to a block in the current frame to replace inter-prediction performed by the motion estimation component 221 and the motion compensation component 219 between frames as described above. Specifically, the intra-estimation component 215 determines an intra-prediction mode for encoding the current block. In some examples, the intra-estimation component 215 selects an appropriate intra-prediction mode from a plurality of tested intra-prediction modes to encode the current block. The selected intra-prediction mode is then forwarded to the header formatting and CABAC component 231 for encoding.

[0079] For example, intra-estimation component 215 computes rate-distortion values using rate-distortion analysis of various tested intra-prediction modes and selects an intra-prediction mode with the best rate-distortion characteristics among the tested modes. Rate-distortion analysis generally determines an amount of distortion (or error) between an encoded block and an original, unencoded block that was encoded to produce the encoded block, as well as a rate (e.g., number of bits) used to generate the encoded block. Intra-estimation component 215 computes a ratio from the distortion and rate of various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion values for the block. In addition, intra-estimation component 215 can be used to code depth blocks of a depth map using depth modeling modes (DMMs) according to rate-distortion optimization (RDO).

[0080] When implemented on an encoder, intra-prediction component 217 can generate a residual block from the prediction block according to the selected intra-prediction mode determined by intra-estimation component 215 or, when implemented on a decoder, read the residual block from the bitstream. The residual block includes the difference between the prediction block and the original block, represented as a matrix. The residual block is then forwarded to transform scaling and quantization component 213. Intra-estimation component 215 and intra-prediction component 217 can operate on luma and chroma components.

[0081] Transform scaling and quantization component 213 is used to further compress the residual block. Transform scaling and quantization component 213 applies a transform, such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform, to the residual block producing a video block comprising residual transform coefficient values. Wavelet transforms, integer transforms, sub-band transforms or other types of transforms could also be used. The transform can convert the residual information from a pixel value domain to a transform domain, e.g., a frequency domain. Transform scaling and quantization component 213 is also used to scale the transformed residual information, e.g., according to frequency. This scaling involves applying a scale factor to the residual information in order to quantize different frequency information at different granularities, which can affect the ultimate visual quality of the reconstructed video. Transform scaling and quantization component 213 is also used to quantize the transform coefficients to further reduce the rate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting a quantization parameter. In some examples, transform scaling and quantization component 213 can then scan the matrix including the quantized transform coefficients. The quantized transform coefficients are forwarded to header formatting and CABAC component 231 to be encoded into the bitstream.

[0082] The scaling and inverse transform component 229 applies the inverse operations of the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 applies inverse scaling, inverse transform, and / or inverse quantization to reconstruct a residual block in the pixel domain, e.g., for later use as a reference block, which can become a prediction block for another current block. The motion estimation component 221 and / or the motion compensation component 219 can calculate a reference block by adding the residual block back to the corresponding prediction block for use in motion estimation of a subsequent block / frame. Filters are used to reconstruct the reference block to reduce artifacts created during scaling, quantization, and transform. Such artifacts can cause inaccurate prediction (and create other artifacts) when predicting a subsequent block.

[0083] The filter control analysis component 227 and the in-loop filter component 225 use filters for residual blocks and / or reconstructed image blocks. For example, the transform residual block of the scaling and inverse transform component 229 can be merged with the corresponding prediction block of the intra-prediction component 217 and / or the motion compensation component 219 to reconstruct an original image block. Filters can then be used to reconstruct the image block. In some examples, filters can in turn be used for residual blocks. As with other components in FIG. 2, the filter control analysis component 227 and the in-loop filter component 225 are highly integrated, can be implemented together, but are shown separately for conceptual purposes. Figure 2

[0084] When acting as an encoder, the filtered reconstructed image blocks, residual blocks, and / or prediction blocks are stored in the decoded picture buffer component 223 for later use in motion estimation, as described above. When acting as a decoder, the decoded picture buffer component 223 stores reconstructed and filtered blocks and forwards them as part of the output video signal to a display. The decoded picture buffer component 223 can be any storage device capable of storing prediction blocks, residual blocks, and / or reconstructed image blocks.

[0085] ​The header format and CABAC component 231 receive data from various components of the encoding / decoding system 200 and encode this data into an encoded bitstream for transmission to the decoder. Specifically, the header format and CABAC component 231 generate various headers to encode control data (such as general control data and filter control data). Furthermore, prediction data (including intra-frame prediction) and motion data, as well as residual data in the form of quantization transform coefficient data, are encoded in the bitstream. The final bitstream contains all the information required by the decoder to reconstruct the original segmented video signal 201. This information may also include an intra-frame prediction mode index table (also called a codeword map), definitions of the coding context of various blocks, indications of the most likely intra-frame prediction modes, indications of segmentation information, etc. This data can be encoded using entropy coding techniques. For example, information can be encoded using context adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding techniques. After entropy coding, the encoded bitstream can be sent to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0086] Figure 3 This is a block diagram of an exemplary video encoder 300. The video encoder 300 can be used to implement the encoding function of the encoding / decoding system 200 and / or implement steps 101, 103, 105, 107, and / or 109 of the operation method 100. The encoder 300 segments the input video signal to generate a segmented video signal 301, wherein the segmented video signal 301 is substantially similar to the segmented video signal 201. The segmented video signal 301 is then compressed by components of the encoder 300 and encoded into a bitstream.

[0087] In particular, the partitioned video signal 301 is forwarded to an intra prediction component 317 for intra prediction. The intra prediction component 317 can be substantially similar to the intra estimation component 215 and the intra prediction component 217. The partitioned video signal 301 is also forwarded to a motion compensation component 321 for inter prediction from reference blocks in a decoded picture buffer component 323. The motion compensation component 321 can be substantially similar to the motion estimation component 221 and the motion compensation component 219. The predicted and residual blocks of the intra prediction component 317 and the motion compensation component 321 are forwarded to a transform and quantization component 313 for transform and quantization of the residual blocks. The transform and quantization component 313 can be substantially similar to the transform scaling and quantization component 213. The transformed and quantized residual blocks and corresponding predicted blocks (along with associated control data) are forwarded to an entropy encoding component 331 for encoding into the bitstream. The entropy encoding component 331 can be substantially similar to the header formatting and CABAC component 231.

[0088] The transformed and quantized residual blocks and / or corresponding predicted blocks are also forwarded from the transform and quantization component 313 to an inverse transform and dequantization component 329 for reconstruction into reference blocks for use by the motion compensation component 321. The inverse transform and dequantization component 329 can be substantially similar to the scaling and inverse transform component 229. According to an example, in-loop filters in an in-loop filter component 325 are also used for the residual blocks and / or the reconstructed reference blocks. The in-loop filter component 325 can be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. As described with respect to the in-loop filter component 225, the in-loop filter component 325 can include multiple filters. The filtered blocks are then stored in the decoded picture buffer component 323 for use by the motion compensation component 321 as reference blocks. The decoded picture buffer component 323 can be substantially similar to the decoded picture buffer component 223.

[0089] Figure 4 A block diagram of an example video decoder 400. The video decoder 400 can be used to implement the decoding functionality of the codec system 200 and / or implement steps 111, 113, 115, and / or 117 of the operating method 100. For example, the decoder 400 receives a bitstream from the encoder 300 and generates a reconstructed output video signal from the bitstream for display to an end user.

[0090] The bitstream is received by an entropy decoding component 433. The entropy decoding component 433 is configured to implement an entropy decoding scheme, such as CAVLC, CABAC, SBAC, PIPE coding, or other entropy coding techniques. For example, the entropy decoding component 433 can use header information to provide contexts to interpret other data encoded as codewords in the bitstream. The decoded information includes any information needed to decode the video signal, such as general control data, filter control data, partitioning information, motion data, prediction data, and quantized transform coefficients in residual blocks. The quantized transform coefficients are forwarded to the inverse transform and dequantization component 429 to be reconstructed into residual blocks. The inverse transform and dequantization component 429 can be similar to the inverse transform and dequantization component 329.

[0091] The reconstructed residual blocks and / or predicted blocks are forwarded to an intra prediction component 417 to be reconstructed into image blocks according to intra prediction operations. The intra prediction component 417 can be similar to the intra estimation component 215 and the intra prediction component 217. Specifically, the intra prediction component 417 uses a prediction mode to locate a reference block in the frame and uses the residual block for the result to reconstruct an intra predicted image block. The reconstructed intra predicted image blocks and / or residual blocks, along with corresponding inter prediction data, are forwarded through an in-loop filter component 425 to a decoded picture buffer component 423. The decoded picture buffer component 423 and the in-loop filter component 425 can be substantially similar to the decoded picture buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image blocks, residual blocks, and / or predicted blocks, and such information is stored in the decoded picture buffer component 423. The reconstructed image blocks of the decoded picture buffer component 423 are forwarded to the motion compensation component 421 for inter prediction. The motion compensation component 421 can be substantially similar to the motion estimation component 221 and / or the motion compensation component 219. Specifically, the motion compensation component 421 uses motion vectors of reference blocks to generate predicted blocks and uses residual blocks for the result to reconstruct image blocks. The resulting reconstructed blocks can also be forwarded through the in-loop filter component 425 to the decoded picture buffer component 423. The decoded picture buffer component 423 continues to store other reconstructed image blocks, which can be reconstructed into frames through partitioning information. The frames can also be placed in a sequence. The sequence is output as a reconstructed output video signal to a display.

[0092] The above mechanisms are exemplary machines and / or processes that can be used to encode and decode 2D videos presented in square / rectangular frames. However, PCC video data represents irregular 3D objects using points. The following mechanisms are PCC-specific video coding processes. These mechanisms can be used to represent 3D point clouds in 2D square / rectangular frames so that the 2D frames can be encoded / decoded by the above mechanisms. As a general example, a 3D point cloud at a particular time instance can be described by a geometry frame that represents the position of each point. Color values and light values of the points can be represented as 2D patches projected onto the points as shown in the geometry frame. The patches at a time instance are packed into a square / rectangular atlas frame, which can be encoded / decoded by the mechanisms described above. Since the patches can not completely cover the atlas frame, a corresponding occupancy frame can be used to represent which areas of the atlas frame include patch data and which areas have no available video data. Any additional attributes of the points can be encoded in corresponding attribute frames. These mechanisms are described in detail below.

[0093] Figure 5 An example of a point cloud media 500 that can be coded according to the PCC mechanisms. Thus, when the method 100 is performed, the point cloud media 500 can be encoded by an encoder (e.g., the codec system 200 and / or the encoder 300) and reconstructed by a decoder (e.g., the codec system 200 and / or the decoder 400).

[0094] Figures 1 to 4 The mechanisms described in the background assume coding of 2D frames in general. However, the point cloud media 500 is a point cloud that changes over time. Specifically, the point cloud media 500 (which can also be referred to as a point cloud and / or a point cloud representation) is a set of points in a 3D space. These points can also be referred to as samples. Each point can be associated with multiple types of data. For example, each point can be described by a position. The position is a location in the 3D space and can be described as a set of Cartesian coordinates. In addition, each point can include a color. The color can be described by a luminance (e.g., light) and a chrominance (e.g., color). The color can be described by red (R), green (G), and blue (B) values (denoted as (R, G, B)) or luminance (Y), blue projection (U), and red projection (V) (denoted as (Y, U, V)). The points can also include other attributes. An attribute is an optional scalar or vector property that can be associated with each point in the point cloud. Attributes can include reflectance, transparency, surface normal, timestamp, material identifier (ID), etc.

[0095] Since each point in the point cloud media 500 can be associated with multiple types of data, the point cloud media 500 can be prepared for compression according to Figures 1 to 4 The mechanisms described in the background use several supporting mechanisms to prepare the point cloud media 500 for compression. For example, the point cloud media 500 can be classified into frames, where each frame includes all data related to a point cloud at a particular state or time instance. Thus,Figure 5 A single frame of the point cloud media 500 is depicted. Then, the point cloud media 500 is coded frame-by-frame. The point cloud media 500 can be enclosed by a 3D bounding box 501. The 3D bounding box 501 is a 3D rectangular prism whose size is designed to enclose all points of the point cloud media 500 of the corresponding frame. It is noted that in case the point cloud media 500 comprises disjoint sets, multiple 3D bounding boxes 501 can be employed. For example, the point cloud media 500 can depict two unconnected figures, in which case a 3D bounding box 501 is placed around each figure. The points in the 3D bounding box 501 are processed as described below.

[0096] Figure 6 An example of a tile 603 created from a point cloud 600 is shown. The point cloud 600 is a single frame of the point cloud media 500. Furthermore, the point cloud 600 is enclosed by a 3D bounding box 601 which is substantially similar to the 3D bounding box 501. Thus, the point cloud 600 can be encoded by an encoder (e.g. the codec system 200 and / or the encoder 300) and reconstructed by a decoder (e.g. the codec system 200 and / or the decoder 400) when performing the method 100.

[0097] The 3D bounding box 601 comprises six faces and thus six 2D rectangular frames 602, each of which is located on one face (e.g. top face, bottom face, left face, right face, front face and back face) of the 3D bounding box 601. By projecting the point cloud 600 onto the corresponding 2D rectangular frame 602, the point cloud 600 can be converted from 3D data to 2D data. This way, the tile 603 is created. It is noted that the 2D rectangular frame 602 can also be referred to as a 2D bounding box (e.g. of the 3D bounding box 601). The tile 603 is a 2D representation of a portion of the 3D point cloud 600, wherein the tile 603 comprises a representation of a portion of the point cloud 600 which is visible from the corresponding 2D rectangular frame 602. The tile 603 can be formally defined as a 2D rectangular region within a sprite associated with volumetric information. It is noted that the representation of the point cloud 600 from the 2D rectangular frame 602 can comprise multiple disjoint components. Thus, the 2D rectangular frame 602 can comprise multiple tiles 603. Thus, the point cloud 600 can be represented by more than six tiles 603. The tile 603 can also be referred to as a sprite, sprite data, sprite information and / or sprite component. By converting the 3D data to a 2D format, the tile 603 of the point cloud 600 can be coded according to video coding mechanisms, e.g. inter prediction and / or intra prediction.

[0098] Figures 7A to 7C A mechanism for encoding a 3D point cloud is shown, which has been converted to 2D information, as described in Figure 6 Particular, Figure 7AAn exemplary occupancy frame 710 associated with a set of patches (e.g., patch 603) is shown. Occupancy frame 710 includes occupancy values ​​713 and 715, which are decoded in binary form. Occupancy values ​​713 and 715 are values ​​indicating whether an atlas sample corresponds to an associated sample in 3D space. For example, an occupancy value of zero (described as occupancy 715) indicates that a portion of bounding box 601 is not occupied by one of patches 603. Those portions of bounding box 601 represented by zero do not participate in the reconstruction of the volumetric representation (e.g., point cloud 600). Conversely, an occupancy value of 1 (described as occupancy 713) indicates that a portion of bounding box 601 is occupied by one of patches 603. Those portions of bounding box 601 represented by one participate in the reconstruction of the volumetric representation (e.g., point cloud 600). Occupancy frame 710 can be formally defined as a set of occupancy values ​​constituting a 2D array and represents the entire occupancy information for a single atlas frame. Therefore, occupancy frame 710 can indicate whether each sample location in the atlas corresponds to a valid 3D point in the point cloud representation.

[0099] Various slices created by projecting 3D information onto a 2D plane can be packed into rectangular (or square) video frames. This approach can be advantageous because various video codecs (e.g., AVC, HEVC, and VVC) are pre-configured to decode such video frames. Therefore, the PCC codec can use other video codecs to decode the slices. Figure 7A As shown, pieces can be packed into frames. Pieces can be packed using any algorithm. For example, pieces can be packed into frames based on size. In a specific example, pieces are arranged from largest to smallest. The largest piece can be placed first in any open space, and once a size threshold is exceeded, smaller pieces fill the gaps. For example, the largest piece can be placed facing the upper left of the frame, and smaller pieces can be placed sequentially facing the lower right, with smaller pieces filling the gaps after the size threshold is exceeded. Figure 7A As shown, this packing scheme generates blank space that does not include slice data. To avoid encoding blank space, occupancy frames 710 are used. Occupancy frames 710 include all occupancy data of the point cloud at a specific time. Specifically, occupancy frames 710 include one or more occupancy values ​​(also referred to as occupancy data, occupancy information, and / or occupancy components). Occupancy 713 and / or 715 can be implemented as a 2D array corresponding to atlas 733 (slice set), where the values ​​of atlas 733 indicate whether each sample location in atlas 733 corresponds to a valid 3D point in the point cloud representation. Figure 7A As shown, occupied frame 710 includes a valid data area depicted as occupied 713. A valid data area indicates that atlas 733 / piece data exists at the corresponding position in occupied frame 710. Occupied frame 710 also includes an invalid data area depicted as occupied 715. An invalid data area indicates that atlas 733 / piece data does not exist at the corresponding position in occupied frame 710.

[0100] Figure 7B An exemplary geometry frame 720 associated with a set of patches (e.g., patches 603) is shown. The geometry frame 720, also referred to as a geometry map, provides or depicts the contours or topography of each patch 603. Specifically, the geometry frame 720 represents the distance of each point in the patch 603 from the planar surface (e.g., 2D rectangular frame 602) of the bounding box 601. The geometry frame 720 includes a geometry shape 723. The geometry shape 723 can be defined as a set of Cartesian coordinates associated with a volume frame. A volume frame is a set of 3D points represented by Cartesian coordinates and zero or more corresponding sets of attributes at a particular time instant. Thus, a volume frame is a combination of the occupancy frame 710, the geometry frame 720, the atlas frame 730, and (optionally) one or more attribute frames that describe a point cloud at a specified time instant. For example, the geometry shape 723 can describe the location of each point in the 3D bounding box at the corresponding time instant.

[0101] As described above, the geometry frame 720 includes the geometry shape 723, also referred to as a geometry map, geometry data, geometry information, and / or geometry component of a point cloud at a particular time instant. The geometry map 723 can be implemented as a 2D array created by aggregating the geometry information associated with each patch, where the geometry information / data is a set of Cartesian coordinates associated with the point cloud frame. Specifically, the patches are all projected from points in 3D space. This projection has the effect of removing 3D information from the patches. The geometry shape 723 preserves the 3D information removed from the patches. For example, each sample in a patch is taken from a point in 3D space. Thus, the geometry frame 720 can include the 3D coordinates associated with each sample in each patch. Thus, the geometry frame 720 / geometry shape 723 can be used by a decoder to map / convert the 2D patches back to 3D space to reconstruct the 3D point cloud. Specifically, the decoder can map each patch sample onto the appropriate 3D coordinates to reconstruct the point cloud.

[0102] Further, Figure 7CAn exemplary atlas frame 730 associated with a set of tiles (e.g., tile 603) is shown. The atlas frame 730 provides or depicts samples of tile 603 in the bounding box 601. The atlas frame 730 can be formally defined as a collection of two-dimensional (2D) bounding boxes (also referred to as tiles and / or atlas 733) projected to a rectangular frame, which corresponds to a volume in 3D space (referred to as a 3D bounding box) on which the volumetric data is rendered. For example, the atlas frame 730 can include color components and / or intensity components of points in tile 603. The color components can be based on an RGB color model, a YUV color model, or based on another known color model. The occupancy frame 710, the geometry frame 720, and the atlas frame 730 can be used to code the point cloud 600 and / or the point cloud media 500. Thus, when performing the method 100, the occupancy frame 710, the geometry frame 720, and the atlas frame 730 can be encoded by an encoder (e.g., the codec system 200 and / or the encoder 300) and reconstructed by a decoder (e.g., the codec system 200 and / or the decoder 400).

[0103] As described above, the atlas frame 730 includes one or more atlases 733 (also referred to as atlas data, atlas information, atlas components, and / or tiles) of the point cloud at a particular time instant. The atlases 733 are collections of 2D bounding boxes projected to a rectangular frame corresponding to a 3D bounding box in 3D space, where each 2D bounding box / tile represents a subset of the point cloud. Specifically, the atlases 733 include tiles created when the 3D point cloud is projected to 2D space as described in connection with Figure 6 Thus, the atlases 733 / tiles include image data (e.g., color and light values) associated with the point cloud at the corresponding time instant. The atlases 733 correspond to the occupancy frame 710 of Figure 7A and the geometry frame 720 of Figure 7B Specifically, the atlases 733 include data according to the valid data regions of the occupancy 713, and do not include data according to the invalid data regions of the occupancy 715. Further, the geometry shapes 723 include 3D information of samples in the atlases 733.

[0104] It is also noted that the point cloud can include attributes (also referred to as attribute data, attribute information, and / or attribute components). These attributes can be included in an attribute frame. The attributes can include all data regarding the corresponding attributes of the point cloud at a particular time instant. An example of an attribute frame is not shown because the attributes can include a wide variety of different data. Specifically, the attributes can be any scalar or vector property associated with each point in the point cloud, such as reflectance, surface normal, timestamp, material ID, etc. Further, the attributes are optional (e.g., user-defined) and can vary depending on the application. However, when used, the point cloud attributes can be included in an attribute frame in a manner similar to the atlases 733, the geometry shapes 723, and the occupancy.

[0105] Thus, the encoder can compress the point cloud frame into atlas frames 730 of atlas 733, geometry frames 720 of geometry 723, occupancy frames 710 of occupancy 713 and 715 values, and (optionally) attribute frames of attributes (collectively referred to as volume frames). For example, the atlas frames 730, geometry frames 720, occupancy frames 710, and / or attribute frames can be further compressed by different encoders for transmission to a decoder. The decoder can decompress the atlas frames 730, geometry frames 720, occupancy frames 710, and / or attribute frames. The decoder can then use the atlas frames 730, geometry frames 720, occupancy frames 710, and / or attribute frames to reconstruct the point cloud frame to determine a reconstructed point cloud for the corresponding time instance. The reconstructed point cloud frames can then be included in order to reconstruct the original point cloud sequence (e.g., for display and / or for data analysis). In a particular example, the atlas frames 730 and / or atlas 733 can be encoded and decoded using the techniques described in connection with Figures 1 to 4 the VVC, HEVC, and / or AVC codecs.

[0106] Figure 8 An example mechanism 800 to define camera position 811 and view vector 815 to render a V-PCC frame included in 3D bounding box 801, etc. In one example, a point cloud (e.g., point cloud media 500) can be reconstructed in 3D bounding box 801, which can be substantially similar to 3D bounding box 501 and / or 601. In a particular example, a set of patches 603 can be decoded from occupancy frames 710, geometry frames 720, atlas frames 730, and / or attribute frames. The patches can be projected onto points in 3D bounding box 801 to reconstruct the point cloud. A camera at camera position 811 can be used to render the reconstructed point cloud in 3D bounding box 801. Thus, mechanism 800 can be used when implementing methods 100, 1100, and / or 1200. Further, mechanism 800 can be used by codec system 200, encoder 300, decoder 400, video coding device 1000, and / or system 1300, e.g., when reconstructing a point cloud for error testing of an encoder or for display by a decoder. Further, data describing mechanism 800 can be indicated by a PCC bitstream (e.g., V-PCC bitstream 900).

[0107] As described above, the point cloud can be reconstructed in a 3D bounding box 801, which can be substantially similar to the 3D bounding boxes 501 and / or 601. In particular, geometry can be used to place the points of the point cloud in the 3D bounding box 801. Then, slices and / or attributes can be projected onto the points in the 3D bounding box 801 to reconstruct the point cloud frame. The 3D bounding box 801 can be described as including a top 801a, a bottom 801b, a right side 801c, a left side 801d, a near side 801f, and a far side 801e. These names are used to clearly discuss other components. For example, the 3D bounding box 801 can be located around an origin 802, which is a predefined coordinate. Thus, the top 801a is above the origin 802, the bottom 801b is below the origin 802, the right side 801c is to the right of the origin 802, the left side 801d is to the left of the origin 802, the near side 801f is in front of the origin 802, and the far side 801e is behind the origin 802.

[0108] The 3D bounding box 801 is a 3D object and includes 3D data. A camera 810 can be used to convert the 3D data into a form that can be displayed on a 2D screen. The camera 810 is a viewport that provides a 2D view of the 3D object. In other words, the camera 810 can provide a 2D view of the contents of the 3D bounding box 801 from a camera position 811 at an angle defined by a view vector 815. The camera position 811 is a coordinate that describes the center of the viewport of the camera 810. The view vector 815 is a vector that extends 90 degrees in front of the viewport of the camera 810, thus defining a direction pointing from the camera position 811 to the viewport. In some PCC systems, the decoder positions the camera 810 at a predefined camera position 811 and uses a predefined view vector 815 regardless of the contents included in the 3D bounding box 801. These systems can require the user to control the camera 810 and move the camera 810 to a preferred position to render the desired contents from a preferred angle. This can be problematic for certain applications because the user can not know the best position to view the desired contents in the 3D bounding box 801. For example, a user can be viewing a PCC video of a football game. The default camera position 811 can be near and / or pointing to the center of the football field in the 3D bounding box 801. The user can want to view the PCC video from a position near a quarterback or a forward, but can not know the position of the quarterback or forward on the field in the 3D bounding box 801. This results in a time-consuming search, which leads to a poor user experience.

[0109] The present disclosure supports content creators selecting and providing suggested camera positions 811 and suggested viewing vectors 815. This supports content creators indicating to users viewports that can include interesting material. Further, this enables content creators to guide the experience of users. Multiple cameras 810 can be used, so multiple camera positions 811 and viewing vectors 815 can be suggested to users for selection. For each camera 810, the camera position 811 can be represented by a camera offset 813. The camera offset 813 is a parameter representing the distance in 3D space between the camera position 811 and a predefined point of the 3D bounding box 801 including the point cloud. In some systems, the camera offset 813 is represented as a measurement from the bottom 801b vertex, the left 801d vertex, and the near 801f vertex of the 3D bounding box 801. Thus, the camera offset 813 represents the difference in the X, Y, and Z axes between the camera position 811 and the bottom 801b vertex, the left 801d vertex, and the near 801f vertex of the 3D bounding box 801. Thus, the camera offset 813 includes an X component, a Y component, and a Z component describing these differences. The camera offset 813 can be encoded in the PCC bitstream and provide enough data to place the camera 810 at the camera position 811.

[0110] The viewing vector 815 of each camera 810 can be represented by a camera rotation 817. Thus, the camera rotation 817 is a parameter representing the viewing vector 815 pointing from the camera 810 to the 3D bounding box 801 including the point cloud. In some examples, the camera rotation 817 is described in quaternion notation. Quaternion notation is a vector representation in the complex number domain using a scalar value and three unit vectors along the axes of space. Quaternion notation can be used to describe any vector in 3D space. A decoder can position the viewport of the camera 810 based on the camera offset 813 and the camera rotation 817 represented in the PCC bitstream. This viewport can then be used to render a 2D representation of the PCC content in the 3D bounding box 801 from the suggested position and angle.

[0111] In a specific implementation, matrix multiplication can be used to accomplish this rendering. For example, the PCC content can be reconstructed by initializing a model matrix representing the 3D bounding box 801. The geometry can be defined by positioning values representing points into the model matrix. The patches and attributes can also be associated with these points in the model matrix, projected onto the geometry. Then, various transformations can be applied to the model matrix. For example, the camera 810 information can be applied as a transformation to the model matrix in order to render a 2D image from the 3D point cloud. For example, the camera offset 813 can be used as an offset matrix and the camera rotation 817 can be used as a rotation matrix. The offset matrix is a matrix representation of the camera offset 813 that can be used to transform the 3D model of the points in the model matrix into a 2D representation. In one specific example, the offset matrix can be represented as follows:

[0112]

[0113] Offset[0], Offset[1], and Offset[2] include the X, Y, and Z components of the camera offset 813, respectively. A scale factor exchange to a value of 1 can also be used to scale the PCC content.

[0114] The rotation matrix is a matrix representation of the camera rotation 817 that can be used to transform a 3D model of a point in the model matrix to a 2D representation. The rotation matrix can include quaternion values of the camera rotation 817 applied to the model matrix. In one specific example, the rotation matrix can be represented as follows:

[0115]

[0116] where qX represents the X component of the camera rotation 817, qY represents the Y component of the camera rotation 817, qZ represents the Z component of the camera rotation 817, and qW represents the scalar component of the camera rotation 817. qW can be calculated as follows:

[0117] qW = Sqrt(l - (qX 2 + qY 2 + qZ 2 ))

[0118] Accordingly, the encoder can indicate the camera rotation 817 by encoding qX, qY, and qZ. The decoder can then calculate qW from qX, qY, and qZ. The decoder can then multiply the model matrix by the offset matrix and the rotation matrix to obtain a 2D rendering of the PCC content in the 3D bounding box 801 from the viewport defined by the camera position 811 and the view vector 815. As a result, a user's selection of the indicated camera 810, or a default selection, causes the user to see the content in the 3D bounding box 801 from a position and angle desired by the content creator. The indication of this data is described in detail below. Accordingly, this example increases the functionality of the encoder and the decoder by creating predetermined and selectable viewing positions and angles for PCC video content. Furthermore, this disclosure supports mechanisms to improve coding efficiency by minimizing the indication of such data, thus supporting reduced processor, memory, and / or network resource usage in the encoder and / or decoder.

[0119] Figure 9A schematic diagram of an exemplary V-PCC bitstream 900 for indicating camera parameters for rendering a point cloud. For example, the V-PCC bitstream 900 can be generated by the codec system 200, the encoder 300, and / or the video coding device 1000 for decoding by the codec system 200, the decoder 400, and / or the video coding device 1000, in accordance with the methods 100, 1100, and / or 1200. Further, the V-PCC bitstream 900 can be used to encode a set of slices 603 from the point cloud media 500 into the occupancy frames 710, the geometry frames 720, the atlas frames 730, and / or the attribute frames. Further, the V-PCC bitstream 900 can encode the camera offset 813 and / or the camera rotation 817 to support rendering from the indicated camera viewpoint in the decoder.

[0120] The V-PCC bitstream 900 includes a series of V-PCC units 910. The V-PCC units 910 are data containers sized for placement in data packets for transmission to a decoder. In particular, the V-PCC units 910 include V-PCC components and / or corresponding parameter sets related to one or more volume frames representing a point cloud at a corresponding time instance in a V-PCC video sequence. In a specific example, the V-PCC units 910 can be configured as network abstraction layer (NAL) units corresponding to the versatile video coding (VVC) standard. NAL units are data containers of a packet size designed to support network transmission. For example, the size of a single NAL unit is typically designed to support network transmission. The NAL units can include a header indicating a NAL unit type and a payload including the associated data. The V-PCC units 910 include a V-PCC unit payload 914 and a V-PCC unit header 912. The V-PCC unit payload 914 includes video data of a corresponding type related to a volume frame. The V-PCC unit header 912 is a header indicating the type of data included in the V-PCC unit payload 914.

[0121] According to an example, the V-PCC unit payload 914 can include a V-PCC parameter set 911, an atlas frame 913, an occupancy frame 915, a geometry frame 917, and / or an attribute frame 919. The V-PCC parameter set 911 is a set of parameters describing the settings for coding one or more corresponding volume frames. For example, the V-PCC parameter set 911 can include an atlas adaptation parameter set (APS) 921. The atlas APS 921 is a syntax structure including syntax elements / parameters that apply to one or more slices included in the atlas frame 913. For example, the parameters in the atlas APS 921 can apply to all one or more (e.g., or all) slices associated with the PCC frame. The V-PCC parameter set 911 can also include other parameter sets, such as an atlas frame parameter set, an atlas sequence parameter set, a video-based volumetric coding (V3C) parameter set, etc. The main difference between the parameter sets is whether the included parameters are related to V-PCC at the video level (e.g., V3C parameter set), the sequence level (e.g., atlas sequence parameter set), the frame level (e.g., atlas frame parameter set), and / or the slice level (e.g., atlas APS). Other parameter sets can also be used for specific purposes, and the above is not intended to be an exhaustive list.

[0122] The atlas frame 913 is a collection of 2D bounding boxes / slices / atlas projected to a rectangular frame corresponding to a volume in 3D space, such as the 3D bounding boxes 501, 601, and / or 801, on which the volume data is rendered. The atlas frame 913 can be substantially similar to the atlas frame 730. The occupancy frame 915 is a collection of occupancy values constituting a 2D array and represents the entire occupancy information for a single atlas frame. Occupancy is a value indicating whether an atlas sample corresponds to an associated sample in 3D space. The occupancy frame 915 can be substantially similar to the occupancy frame 710. The geometry frame 917, also referred to as a geometry map, is a frame including geometry slice information projected at a particular depth. Geometry is a set of Cartesian coordinates associated with a volume frame. The geometry frame 917 can be substantially similar to the geometry frame 720. The attribute frame 919 is a frame including scalar or vector attributes, such as color, reflectance, surface normal, timestamp, material identifier (ID), etc., optionally associated with each point in a volume frame of a point cloud. A volume frame is a set of 3D points represented by Cartesian coordinates and zero or more corresponding sets of attributes at a particular time instant. A video of a point cloud includes a series of volume frames. Thus, the V-PCC parameter set 911 includes a set of parameters indicating how to decode one or more volume frames. The atlas frame 913, the occupancy frame 915, the geometry frame 917, and the attribute frame 919 include video data for reconstructing a volume frame.

[0123] As noted above, some V-PCC systems always position the camera to render a 2D view of the 3D point cloud at a default position. In these systems, the data in the atlas frame 913, occupancy frame 915, geometry frame 917, and attribute frame 919 are used to reconstruct a volume frame. The volume frame is then rendered by the camera at the default position. This approach can not be beneficial in many cases, as the default position does not change from video to video, thus providing a view of the volume frame that is unlikely to show useful information.

[0124] The present disclosure supports content creators selecting and providing suggested camera positions and suggested viewing vectors. This supports content creators indicating to users viewports that can include interesting views of the volume frame. Furthermore, this enables content creators to guide the experience of the user. In a specific example, the atlas APS 921 can be used to code suggested camera positions and suggested viewing vectors. For example, the atlas APS 921 can include one or more atlas camera parameter (ACP) syntax structures. Each ACP syntax structure can include data describing a camera used when rendering. For example, the atlas APS 921 can encode an ACP syntax structure that codes a camera position as a camera offset, such as the camera offset 813. Furthermore, the ACP syntax structure can code a camera viewing vector, such as the camera rotation 817, as a quaternion representation.

[0125] In a specific implementation, the camera offset can be coded as an ACP offset on axis [d] 933. For example, the ACP offset on axis [d] 933 can include X, Y, and Z components of the camera offset, where d is a counter variable to represent the offset component being represented. For example, d can be set to 0, 1, or 2 to represent components X, Y, or Z, respectively. In a specific implementation, the ACP offset on axis [d] 933 can be defined as follows. The ACP offset on axis [d] 933 represents the offset value of the current camera model along the d-axis, denoted as Offset[d], with an increment of 2 -16 . The value of the ACP offset on axis [d] 933 should be in the range of -2 31 to 2 31 -1, inclusive, where d is in the range of 0 to 2, inclusive. The values of d equal to 0, 1, and 2 correspond to the X, Y, and Z axes, respectively. When not present, ACP offset on axis [d] can be inferred to be equal to 0. Offset[d] can be derived according to Offset[d] = ACP offset on axis [d] ÷ 2 16determined. The decoder can generate the offset matrix from the data in Acp_offset_on_axis[d] 933. The offset matrix can be represented as follows:

[0126]

[0127] where Offset[0], Offset[1], and Offset[2] are the X, Y, and Z displacements of the camera, and 1 can be replaced with a scaling factor as needed.

[0128] Further, the unit vector quaternion components of the camera rotation can be encoded as ACP rotation qx (Acp_rotation_qx) 935, ACP rotation qy (Acp_rotation_qy) 937, and ACP rotation qz (Acp_rotation_qz) 939. The scalar components of the camera rotation can be determined by the decoder from the unit vector components. In one particular example, Acp_rotation_qx 935, Acp_rotation_qy 937, and Acp_rotation_qz 939 can be defined as follows. Acp_rotation_qx 935 represents, in quaternion notation, the x component of the rotation of the current camera model, qX. The value of acp_rotation_qx can range from -2 14 to 2 14 (inclusive). When not present, it can be inferred that Acp_rotation_qx 939 is equal to 0. The value of qX can be determined according to qX = acp_rotation_qx ÷ 2 14 .

[0129] Further, Acp_rotation_qy 937 represents, in quaternion notation, the y component of the rotation of the current camera model, qY. The value of acp_rotation_qy 937 can range from -2 14 to 2 14 (inclusive). When not present, it can be inferred that Acp_rotation_qy 937 is equal to 0. The value of qY can be determined according to qY = acp_rotation_qy ÷ 2 14 .

[0130] Further, Acp_rotation_qz 939 represents, in quaternion notation, the z component of the rotation of the current camera model, qZ. The value of Acp_rotation_qz 939 can range from -2 14 to 2 14the range of [0, 1] (inclusive). When not present, Acp_rotation_qz 939 can be inferred to equal 0. The value of qZ can be derived from qZ = acp_rotation_qz ÷ 2 14 determined.

[0131] The fourth / scalar component qW of the rotation of the current camera model expressed in quaternion notation can be calculated as follows: qW = Sqrt(l - (qX2+ qY2+ qZ2)). The decoder can generate a rotation matrix from the data in Acp_rotation_qx 935, Acp_rotation_qy 937, and Acp_rotation_qz 939. For example, a unit quaternion can be expressed as a rotation matrix as follows:

[0132]

[0133] Thus, the encoder can code the camera offset in the V-PCC bitstream 900 as Acp_offset_on_axis[d] 933 and the camera rotation as Acp_rotation_qx 935, Acp_rotation_qy 937, and Acp_rotation_qz 939. The decoder can reconstruct the point cloud in the 3D bounding box by including the corresponding PCC video data in the model matrix. The decoder can then generate an offset matrix describing the camera offset from Acp_offset_on_axis[d] 933 and a rotation matrix describing the camera rotation from Acp_rotation_qx 935, Acp_rotation_qy 937, and Acp_rotation_qz 939. The decoder can then apply the offset matrix and the rotation matrix to the model matrix through matrix multiplication or the like to render a 2D view of the 3D point cloud from the indicated viewport. Thus, Acp_offset_on_axis[d] 933, Acp_rotation_qx 935, Acp_rotation_qy 937, and Acp_rotation_qz 939 include data that can be used as a transformation to transform the reconstructed point cloud from a 3D volume representation to a 2D representation from the indicated camera position and viewing angle, which can be displayed on a 2D screen for an end user to view. Thus, Acp_offset_on_axis[d] 933, Acp_rotation_qx 935, Acp_rotation_qy 937, and Acp_rotation_qz 939 can be used by a content creator to represent a preferred viewing position and angle, and thus can be used by an encoder to guide a user’s experience of the 3D V-PCC content coded by the V-PCC bitstream 900.

[0134] The following describes in detail the implementation of the above information. The 3D reconstruction information describes the volume and shape of the object. However, there are also some additional parameters, such as extrinsic camera parameters including the direction and position in space. In addition, the scaling parameters provide information about the size and spacing of points in the physical space. However, some PCC systems do not use any information that defines the director's default view of the immersive object. Defining a camera requires the use of two parameter sets. These parameter sets include a position in 3D space, which can be defined as a translation relative to the point cloud frame origin. The camera parameters also include a rotation relative to the coordinate system of the point cloud frame. In order to provide rotation and translation parameters for the camera, the messages in the PCC bitstream are modified to include a list of corresponding parameters.

[0135] The extrinsic camera parameters in the 3D rendering pipeline can be defined as follows. Specifically, the definition of the camera in the world coordinates includes two factors. The first is the offset parameter, which is the X, Y and Z displacement of the camera relative to the world coordinate system. The offset matrix, denoted as T, can be defined as follows:

[0136]

[0137] where X, Y and Z are the X, Y and Z displacements of the camera, and 1 can be replaced with a scaling factor as needed.

[0138] The second parameter is the direction matrix, denoted as R, which defines the camera rotation relative to the world coordinate system, an example of which is described as follows:

[0139]

[0140] where a, b, g are the rotation angles relative to the x, y and z world coordinate axes, respectively. The resulting transformation can be applied to the bounding box used in the scene graph and including the reconstructed point cloud. By using these parameters, the content creator can suggest several predefined camera positions. Such camera positions can be referred to as director views.

[0141] In one example implementation, a supplemental enhancement information (SEI) message can carry the geometric transformation parameters, and can be modified by adding the following information. For example, one or more cameras can be associated with the model to represent a predefined or director view for directing the attention of the audience to a particular action in the volumetric scene. Each camera can be represented using an offset along each axis in the global scene coordinate system. Each camera can be characterized using a specific rotation along each axis. In another example, each camera rotation can be characterized using a quaternion (x, y, z and w2=1-(x2+y2+z2)).

[0142] The syntax for the geometric transformation parameter SEI message, including the parameters described above, can be described as follows:

[0143]

[0144]

[0145] An example semantic change to the SEI message semantics of the geometric transformation parameter is as follows. `gtp_num_camera_info` represents the predefined number of camera presets. `gtp_num_camera_info` equal to zero indicates that camera information does not exist. When it does not exist, the value of `gtp_num_camera_info` is inferred to be 0. `gtp_camera_offset_on_axis[i][d]` represents the value of the i-th camera offset along the d-axis. The value of `gtp_geometry_scale_on_axis[i][d]` can be between 0 and 2. 32 The range is -1 (inclusive), where d is in the range of 0 to 2 (inclusive). Values ​​of d equal to 0, 1, and 2 correspond to the X, Y, and Z axes, respectively. When it does not exist, it can be inferred that gtp_geometry_scale_on_axis[i][d] is equal to 0. gtp_camera_orientation_on_axis[i][d] represents the d-th component Orientation[i][d] in the direction of the i-th camera. The value of gtp_camera_orientation_on_axis[i][d] can be in the range -2. 15 to 2 15 The range is -1 (inclusive). Values ​​of d equal to 0, 1, and 2 correspond to the X, Y, and Z rotation components, respectively. If it does not exist, it should be inferred that gtp_camera_orientation_on_axis[i][d] is equal to 0. The value of Orientation[i][d] can be calculated as follows:

[0146] Orientation[i][d]=gtp_camera_orientation_on_axis[i][d]÷2 15

[0147] Quaternion methods can also be used to decode camera rotation. Three vectors can be used to define the camera. These vectors include a position vector, a view vector, and an up vector. Alternatively, a first-person camera can be defined by rotating the view vector. With quaternions, vectors can be rotated about any axis. This can be achieved by converting the view vector to a quaternion. Then, a rotation quaternion can be defined. The rotation quaternion can then be applied to the view vector.Figure Four Atoms are used to achieve rotation. To create a view... Figure Four The quaternion (V) is derived from the x, y, and z values ​​of the view vector, with 0 added as a scalar component (w). Therefore, V = [0, view]. A quaternion can then be generated to represent the rotation. For this, the vector as the rotation center (A) and the rotation angle (θ) are obtained. The formula for establishing the rotation quaternion (R) is as follows: Vector A = [x, y, z]Rx = Ax * sin(θ / 2)Ry = Ay * sin(θ / 2)Rz = Az * sin(θ / 2)Rw = cos(θ / 2). The rotation quaternion R defines the rotation. After rotation, the quaternion representing the view is used, denoted as W. The rotation operation is described as W = R * V * R′, where R′ is the conjugate of R. The fourth component of the geometric rotation of the current point cloud image, represented in quaternion notation, RotationW, can be calculated as follows:

[0148] RotationX = Orientation[0]

[0149] RotationY = Orientation[1]

[0150] RotationZ = Orientation[2]

[0151] RotationW=Sqrt(1-(Orientation[0]2+Orientation[1]2+Orientation[2]2))

[0152] The above describes an exemplary V-PCC message that provides a set of camera displacement parameters relative to the origin of the point cloud frame in the scene. The camera position can be represented by axial offsets in the x, y, and z directions within the scene. The V-PCC message can also provide a set of camera orientation parameters in the scene. Camera orientation can be represented by rotation angles d, β, and γ, representing Euler rotations along the x, y, and z axes. Alternatively, camera orientation can be represented by RotationX, RotationY, r_z, and r_w, where the r_w rotation is derived, using quaternion notation.

[0153] Figure 10FIG. 10 is a schematic diagram of an example video coding device 1000. The video coding device 1000 is suitable for implementing the disclosed examples / embodiments described herein. The video coding device 1000 includes a downstream port 1020, an upstream port 1050, and / or a transceiver (Tx / Rx) 1010. The transceiver (Tx / Rx) 1010 includes a transmitter and / or a receiver for transmitting data upstream and / or downstream over a network. The video coding device 1000 also includes a processor 1030, including logic and / or a central processing unit (CPU), for processing data; a memory 1032 for storing data. The video coding device 1000 can also include electrical, optical-to-electrical (OE), electrical-to-optical (EO), and / or wireless communication components coupled to the upstream port 1050 and / or the downstream port 1020 for data transmission over electrical, optical, or wireless communication networks. The video coding device 1000 can also include an input and / or output (I / O) device 1060 for data communication with a user. The I / O device 1060 can include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O device 1060 can also include input devices such as a keyboard, a mouse, a trackball, etc., and / or corresponding interfaces for interacting with these output devices.

[0154] Processor 1030 is implemented in both hardware and software. Processor 1030 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). Processor 1030 communicates with downlink port 1020, Tx / Rx 1010, uplink port 1050, and memory 1032. Processor 1030 includes a decoding module 1014. Decoding module 1014 implements the disclosed embodiments described herein (e.g., methods 100, 1100, and 1200) that can use point cloud media 500 divided into a set of slices 603 and encoded as occupancy frames 710, geometry frames 720, and atlas frames 730 in a V-PCC bitstream 900. Decoding module 1014 can also implement mechanism 800 described herein and any other methods / mechanisms. Furthermore, the decoding module 1014 can implement the encoding / decoding system 200, encoder 300, and / or decoder 400. For example, the decoding module 1014 in the encoder can encode camera offset and / or camera rotation in the PCC bitstream. Then, the decoding module 1014 in the decoder can use the camera offset to generate an offset matrix and / or use the camera rotation to generate a rotation matrix. The decoder can then apply the offset and rotation matrices to a model matrix including the reconstructed point cloud to render a point cloud frame from the indicated position and angle. This allows the encoder to specify and the decoder to use one or more predefined camera positions and angles to display the reconstructed point cloud to the user. Therefore, the decoding module 1014 enables the video decoding device 1000 to provide additional functionality and / or improve decoding efficiency when decoding video data. Thus, the decoding module 1014 improves the functionality of the video decoding device 1000 and solves problems specific to the field of video decoding. Furthermore, the decoding module 1014 affects the transformation of the video decoding device 1000 to different states. Alternatively, the decoding module 1014 may be implemented as instructions (e.g., a computer program product stored in a non-transitory medium) stored in memory 1032 and executed by processor 1030.

[0155] Memory 1032 includes one or more memory types such as disks, tape drives, solid-state drives, read only memory (ROM), random access memory (RAM), flash, ternary content-addressable memory (TCAM), static random-access memory (SRAM), etc. Memory 1032 can be used as an overflow data store, to store programs selected for execution, and to store instructions and data read during program execution, when selected for execution.

[0156] Figure 11 is an example method 1100 of encoding a V-PCC bitstream (e.g., V-PCC bitstream 900) to indicate camera parameters for rendering a point cloud, e.g., according to mechanism 800. An encoder (e.g., codec system 200, encoder 300, and / or video coding device 1000) can use method 1100 when performing method 100. Thus, method 1100 can operate on point cloud media 500, which is divided into a set of tiles 603 and encoded into occupancy frames 710, geometry frames 720, atlas frames 730, and / or attribute frames.

[0157] Method 1100 can begin when a content creator determines to encode video as a 3D point cloud according to PCC mechanisms. For example, an encoder can begin encoding PCC content for a plurality of frames according to input from a content creator. At step 1101, the encoder encodes a plurality of 2D tiles, geometry shapes, occupancy, and / or attributes that describe a point cloud frame. The encoder can encode these items into a PCC bitstream (e.g., V-PCC bitstream 900) (e.g., as atlas frames 913, geometry frames 917, occupancy frames 915, and attribute frames 919). This data can be sufficient to reconstruct a volume / point cloud frame in a decoder. For example, occupancy can be used to remove certain compression-related coding artifacts in atlas frames that are not actual video data. Geometry shapes can be included in a model matrix to recreate a 3D point cloud in a 3D bounding box, and 2D tiles and / or attributes can be projected onto the geometry shapes.

[0158] At step 1103, the encoder can determine a camera offset that describes a position of a camera relative to the point cloud frame. The camera offset can be determined from user input. For example, a content creator can indicate one or more preferred positions of various cameras to guide a viewer to watch the video content. The camera offset represents such preferred camera positions and can be used by a decoder to support generation of an offset matrix that is used to render camera views of the point cloud frame after the point cloud is reconstructed in the decoder (e.g., from data from step 1101). For example, the camera offset can represent a first offset corresponding to an X axis, a second offset corresponding to a Y axis, and a third offset corresponding to a Z axis. In a specific implementation, the camera offset can be included in the ACP syntax structure in the atlas APS 921 in Acp_offset_on_axis[d] 933. This provides sufficient information for the decoder to generate an offset matrix that is used to render 2D views of the 3D point cloud. In one example, the offset matrix includes four rows and four columns. The first offset (e.g., X component) can be located in the first row fourth column, the second offset (e.g., Y component) can be located in the second row fourth column, and the third offset (e.g., Z component) can be located in the third row fourth column. In one example, such an offset matrix can be represented as follows:

[0159]

[0160] At step 1105, the encoder can determine a camera rotation that describes a viewing vector of a camera relative to the point cloud frame. The camera rotation can be determined from user input. For example, a content creator can indicate one or more preferred viewing angles of various cameras to guide a viewer to watch the video content. The camera rotation represents such preferred camera views and can be used by a decoder to support generation of a rotation matrix that is used to render camera views of the point cloud frame after the point cloud is reconstructed in the decoder (e.g., from data from step 1101). As described above, a viewing angle can be described by a viewing vector that extends at a 90 degree angle from a plane that describes a viewport associated with the camera. Thus, the camera position and viewing angle describe a location and direction of a viewport that is selected by a user. The viewing vector can be described as a camera rotation in quaternion representation. For example, the camera rotation represents an X component of a rotation of the camera in quaternion representation, a Y component of a rotation of the camera in quaternion representation, and a Z component of a rotation of the camera in quaternion representation. The camera rotation also includes a W component that can be computed from the X, Y, and Z components. For example, the W component can be computed according to qW = Sqrt(l-(qX2+qY2+qZ2)), where qW, qX, qY, and qZ represent the W component, the X component, the Y component, and the Z component, respectively, where Sqrt is a square root function.

[0161] In a particular implementation, the qX, qY, and qZ components of the camera rotation can be included in the acp_rotation_qx 935, acp_rotation_qy 937, and acp_rotation_qz 939 in the ACP syntax structure in the atlas APS 921, respectively. qW can not be coded, but can be computed in the decoder from the qX, qY, and qZ components as described above. This provides the decoder with sufficient information to generate a rotation matrix for rendering a 2D view of the 3D point cloud. In one example, the rotation matrix includes four rows and four columns, and can include various combinations of the qW, qX, qY, and qZ components as follows:

[0162]

[0163] Thus, the rotation matrix includes an X component, a Y component, a Z component, and a W component of the rotation of the camera expressed in quaternion notation.

[0164] At step 1107, the encoder encodes the camera offsets and the camera rotation into the PCC bitstream. At step 1109, the PCC bitstream can also be stored for transmission to a decoder. The PCC bitstream can be forwarded to a content server for long-term storage and / or transmission to a decoder / end-user upon request. Thus, a content creator can select one or more camera positions and viewing angles in the V-PCC bitstream and encode them for directing the viewpoint of a viewer when watching the reconstructed V-PCC content on a decoder. Thus, the method 1100 increases the functionality of the encoder and the decoder by creating predetermined and selectable viewing positions and angles for PCC video content. Moreover, the method 1100 supports mechanisms for improving coding efficiency, such as by omitting qW from the V-PCC bitstream. Thus, the method 1100 supports reducing the usage of processor, memory, and / or network resources in the encoder and / or the decoder.

[0165] Figure 12 is a flowchart of an exemplary method of decoding a V-PCC bitstream (e.g., the V-PCC bitstream 900) to render a point cloud, for example, according to the mechanisms 800 and the indicated camera parameters. The decoder (e.g., the codec system 200, the decoder 400, and / or the video coding device 1000) can use the method 1200 when performing the method 100. Thus, the method 1200 can operate on the point cloud media 500, which is divided into the set of slices 603 and encoded into the occupancy frames 710, the geometry frames 720, the atlas frames 730, and / or the attribute frames.

[0166] For example, after the method 1100 ends, the method 1200 can begin with a decoder beginning to receive a V-PCC bitstream representing coded data of a point cloud video sequence. At step 1201, the decoder receives a PCC bitstream including PCC components. Specifically, the bitstream can include a plurality of 2D patches in an atlas frame, geometry shapes in a geometry frame, occupancy in an occupancy frame, and / or attributes in an attribute frame. The bitstream can also include one or more camera offsets and one or more camera rotations of one or more cameras. The camera offsets and camera rotations define a viewport for viewing the V-PCC content as indicated by a content creator.

[0167] At step 1203, the decoder can initialize a model matrix representing a 3D bounding box and include the geometry shapes in the model matrix. The decoder can then decode the patches and convert the patches to a 3D patch coordinate system by projecting the patches onto points in the 3D bounding box / model matrix to obtain a point cloud frame of the point cloud content. The attributes, if present, can also be projected onto the geometry shapes to create a reconstructed point cloud frame / volume frame.

[0168] At step 1205, the decoder can determine an offset matrix from the camera offsets of the camera. For example, the camera offsets can represent a first offset corresponding to an X-axis, a second offset corresponding to a Y-axis, and a third offset corresponding to a Z-axis. These offsets represent a displacement between a camera position and a predetermined point in the 3D bounding box (e.g., a bottom vertex, a left vertex, a near vertex of the 3D bounding box) on each axis. In a specific implementation, the camera offsets can be included in Acp_offset_on_axis[d] 933 in an ACP syntax structure in the atlas APS 921. This provides the decoder with sufficient information to generate / determine an offset matrix for rendering a 2D view of the 3D point cloud. In one example, the offset matrix includes four rows and four columns. The first offset (e.g., X-component) can be located at the first row fourth column, the second offset (e.g., Y-component) can be located at the second row fourth column, and the third offset (e.g., Z-component) can be located at the third row fourth column. In one example, such an offset matrix can be represented as follows:

[0169]

[0170] At step 1207, the decoder can determine a rotation matrix from the camera rotation of the camera. As described above, the viewing angle of the camera can be described by a viewing vector extending at a 90-degree angle from a plane describing the viewport associated with the camera. Thus, the camera position and viewing angle describe the position and orientation of the viewport selected by the content creator for the user. The viewing vector can be described as a camera rotation in quaternion representation. For example, the camera rotation can include an X component of the rotation of the camera in quaternion representation, a Y component of the rotation of the camera in quaternion representation, and a Z component of the rotation of the camera in quaternion representation. The camera rotation also includes a W component, which can be computed from the X, Y, and Z components. For example, the W component can be computed according to qW = Sqrt(l-(qX2+qY2+qZ2)), where qW, qX, qY, and qZ represent the W component, the X component, the Y component, and the Z component, respectively, where Sqrt is a square root function.

[0171] In a specific implementation, the qX, qY, and qZ components of the camera rotation can be included in acp_rotation_qx 935, acp_rotation_qy 937, and acp_rotation_qz 939 in the ACP syntax structure in atlas APS 921, respectively. qW can not be coded, but can be computed in the decoder from qX, qY, and qZ components as described above. The decoder can obtain qX, qY, and qZ components from the V-PCC bitstream and compute qW component from qX, qY, and qZ components. Then, at step 1207, the decoder can generate the rotation matrix from qW, qX, qY, and qZ components. In one example, the rotation matrix includes four rows and four columns, and can include various combinations of qW, qX, qY, and qZ components, as follows:

[0172]

[0173] Thus, the rotation matrix includes an X component, a Y component, a Z component, and a W component of the rotation of the camera in quaternion representation.

[0174] The decoder can then render a 2D view of the 3D point cloud using the camera defined viewport. At step 1209, the decoder can apply the offset matrix and the rotation matrix to transform the points / data in the model matrix by matrix multiplication, etc. Thus, the decoder renders a 2D view of the 3D point cloud by applying the offset matrix and the rotation matrix to the point cloud content. This approach transforms the data from a 3D coordinate system to a 2D coordinate system that can be displayed on a flat screen. Thus, this approach supports viewing the 3D content from the perspective of the viewport defined by the content creator. The rendered point cloud content can then be forwarded to a display at step 1211. Thus, the method 1200 increases the functionality of the encoder and the decoder by creating predetermined and selectable viewing positions and angles for PCC video content. Furthermore, the method 1200 supports mechanisms to improve coding efficiency, such as by omitting qW from the V-PCC bitstream. Thus, the method 1200 supports reducing the usage of processor, memory, and / or network resources in the encoder and / or the decoder.

[0175] Figure 13 is a schematic diagram of an example system for indicating camera parameters, e.g., in the V-PCC bitstream 900, to render a point cloud, e.g., according to the mechanism 800. The system 1300 can be implemented by an encoder and a decoder, such as the codec system 200, the encoder 300, the decoder 400, and / or the video coding device 1000. Thus, the system 1300 can operate on the point cloud media 500, which is divided into a set of tiles 603 and encoded into occupancy frames 710, geometry frames 720, atlas frames 730, and / or attribute frames. Furthermore, the system 1300 can be used when implementing the methods 100, 1100, and / or 1200.

[0176] The system 1300 includes a video encoder 1302. The video encoder 1302 includes a determining module 1303 for determining a camera offset describing a position of a camera relative to a point cloud frame to support generating an offset matrix for rendering a camera view of the point cloud frame. The video encoder 1302 further includes an encoding module 1305 for encoding a plurality of 2D tiles and geometry shapes describing the point cloud frame into a PCC bitstream. The encoding module 1305 is further for encoding the camera offset into the PCC bitstream. The video encoder 1302 further includes a storing module 1306 for storing the bitstream for transmission to a decoder. The video encoder 1302 further includes a transmitting module 1307 for transmitting the bitstream to a video decoder 1310. The video encoder 1302 can be further for performing any of the steps in the method 1100.

[0177] The system 1300 further includes a video decoder 1310. The video decoder 1310 includes a receiving module 1311 configured to receive a bitstream including a plurality of 2D patches in a atlas frame and camera offsets of a camera. The video decoder 1310 further includes a decoding module 1313 configured to decode the patches. The video decoder 1310 further includes a converting module 1315 configured to convert the patches to a 3D patch coordinate system by projection in a 3D bounding box to obtain a point cloud frame. The video decoder 1310 further includes a determining module 1317 configured to determine an offset matrix according to the camera offsets. The video decoder 1310 further includes an applying module 1318 configured to apply the offset matrix to the point cloud frame. The video decoder 1310 further includes a forwarding module 1319 configured to forward the point cloud frame for display. The video decoder 1310 can be further configured to perform any of the steps in the method 1200.

[0178] A first component is directly coupled to a second component when there are no intermediate components between the first component and the second component other than a wire, a trace, or other dielectric. A first component is indirectly coupled to a second component when there are one or more intermediate components between the first component and the second component in addition to the wire, the trace, or other dielectric. The term "coupled" and its synonyms include both direct and indirect couplings. Unless otherwise stated, the term "about" means a range of ±10% of the number that follows.

[0179] It is also to be understood that the steps of the exemplary methods set forth herein are not necessarily required to be performed in the order described, and the order of the steps of these methods should be understood as merely an example. Likewise, additional steps can be included or existing steps can be removed, or one or more steps can be added to or performed before or after existing steps without departing from the scope of the various embodiments of the present invention.

[0180] While several embodiments have been provided in the present disclosure, it is to be understood that the disclosed systems and methods can be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to limit the disclosure to the details given herein. For example, the various elements or components can be combined or integrated in another system or certain features can be omitted, or not implemented.

[0181] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate can be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and can be made without departing from the spirit and scope of the disclosure.

Claims

1. A method for implementing a decoder, characterized in that, The method includes: The decoder's receiver receives a bitstream, which includes multiple slices in the atlas frame. The bitstream also includes camera offset parameters and rotation parameters. The offset parameters are used to determine the position of the camera, and the rotation parameters are used to determine the viewing angle of the camera. The decoder's processor decodes the slices in the atlas frame; The processor converts the slice into a three-dimensional (3D) slice coordinate system to obtain point cloud content; The processor determines an offset matrix based on the offset parameter and a rotation matrix based on the rotation parameter. The offset matrix represents the displacement of the camera relative to the world coordinate system, and the rotation matrix represents the rotation of the camera relative to the world coordinate system. The processor applies the offset matrix and the rotation matrix to the point cloud content.

2. The method according to claim 1, characterized in that, The offset parameters represent the first offset corresponding to the X-axis, the second offset corresponding to the Y-axis, and the third offset corresponding to the Z-axis.

3. The method according to claim 2, characterized in that, The offset matrix comprises four rows and four columns, with the first offset located in the first row and fourth column, the second offset located in the second row and fourth column, and the third offset located in the third row and fourth column.

4. The method according to claim 1, characterized in that, The rotation parameters are represented by quaternions for the X component of the camera's rotation, the Y component of the camera's rotation, and the Z component of the camera's rotation.

5. The method according to claim 4, characterized in that, The rotation matrix includes the X component, Y component, Z component, and W component of the camera's rotation, represented in quaternion notation.

6. The method according to claim 5, characterized in that, The W component is calculated according to the following formula: qW=Sqrt(1) (qX 2 +qY 2 +qZ 2 In this context, qW, qX, qY, and qZ represent the W component, the X component, the Y component, and the Z component, respectively, and Sqrt is the square root function.

7. The method according to any one of claims 1 to 6, characterized in that, The slices in the atlas frame are two-dimensional (2D) slices.

8. A method for implementing an encoder, characterized in that, The method includes: The processor encodes multiple two-dimensional (2D) patches and geometries describing the point cloud frame into a bitstream; The processor determines offset parameters describing the position of the camera relative to the point cloud frame to support the generation of an offset matrix for rendering the camera view of the point cloud frame. The offset parameters are used to determine the position of the camera, and the offset matrix is ​​used to represent the displacement of the camera relative to the world coordinate system. The processor determines rotation parameters describing the viewing vector of the camera relative to the point cloud frame to support the generation of a rotation matrix for rendering the camera view of the point cloud frame. The rotation parameters are used to determine the viewing angle of the camera, and the rotation matrix is ​​used to represent the rotation of the camera relative to the world coordinate system. The processor encodes the offset parameter and the rotation parameter into the bitstream.

9. The method according to claim 8, characterized in that, The method further includes: The bitstream is stored in the encoder's memory.

10. The method according to claim 8, characterized in that, The offset parameters represent the first offset corresponding to the X-axis, the second offset corresponding to the Y-axis, and the third offset corresponding to the Z-axis.

11. The method according to claim 10, characterized in that, The offset matrix comprises four rows and four columns, with the first offset located in the first row and fourth column, the second offset located in the second row and fourth column, and the third offset located in the third row and fourth column.

12. The method according to any one of claims 8 to 11, characterized in that, The rotation parameters are represented by quaternions for the X component of the camera's rotation, the Y component of the camera's rotation, and the Z component of the camera's rotation.

13. The method according to claim 12, characterized in that, The rotation matrix includes the X component, Y component, Z component, and W component of the camera's rotation, represented in quaternion notation.

14. The method according to claim 13, characterized in that, The W component is calculated according to the following formula: qW=Sqrt(1) (qX 2 +qY 2 +qZ 2 In this context, qW, qX, qY, and qZ represent the W component, the X component, the Y component, and the Z component, respectively, and Sqrt is the square root function.

15. A video decoding device, characterized in that, include: A processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, the receiver, the memory, and the transmitter are configured to perform the method according to any one of claims 1 to 14.

16. A non-transitory computer-readable medium, characterized in that, The invention includes a computer program product for use by a video decoding device, wherein the computer program product includes computer-executable instructions stored in the non-transitory computer-readable medium, which, when executed by a processor, cause the video decoding device to perform the method according to any one of claims 1 to 14.

17. A decoder, characterized in that, include: The receiving module is used to receive a bitstream comprising multiple two-dimensional (2D) slices in an atlas frame. The bitstream also includes camera offset parameters and rotation parameters. The offset parameters are used to determine the position of the camera, and the rotation parameters are used to determine the viewing angle of the camera. A decoding module is used to decode the chip; A conversion module is used to convert the slice into a three-dimensional (3D) slice coordinate system to obtain point cloud frames; A determining module is used to determine an offset matrix based on the offset parameters and a rotation matrix based on the rotation parameters, wherein the offset matrix represents the displacement of the camera relative to the world coordinate system and the rotation matrix represents the rotation of the camera relative to the world coordinate system. An application module is used to apply the offset matrix and the rotation matrix to the point cloud frame.

18. The decoder according to claim 17, characterized in that, The decoder is also used to perform the method according to any one of claims 1 to 7.

19. An encoder, characterized in that, include: A determination module is used to determine offset parameters describing the position of the camera relative to a point cloud frame, so as to support the generation of an offset matrix for rendering the camera view of the point cloud frame. The offset parameters are used to determine the position of the camera, and the offset matrix is ​​used to represent the displacement of the camera relative to the world coordinate system. The determining module is further configured to determine rotation parameters describing the viewing vector of the camera relative to the point cloud frame, so as to support the generation of a rotation matrix for rendering the camera view of the point cloud frame, wherein the rotation parameters are used to determine the viewing angle of the camera, and the rotation matrix is ​​used to represent the rotation of the camera relative to the world coordinate system. Encoding module, used for: Encode multiple two-dimensional (2D) patches and geometries describing point cloud frames into the bitstream; The offset parameter and the rotation parameter are encoded into the bitstream.

20. The encoder according to claim 19, characterized in that, The encoder is also used to perform the method according to any one of claims 8 to 14.

21. A computer-readable storage medium, characterized in that, The storage contains a bitstream generated by the method according to any one of claims 8 to 14.

Images