Video-based point cloud compression model to world signaling information

Abstract

An apparatus, method and computer program for implementing video-based point cloud compression model-to-world signaling are disclosed. The example apparatus includes: a component for providing first signaling information (807), wherein the first signaling information includes information related to a world domain (811), wherein the world domain (811) is a point cloud frame represented by a plurality of points in a first volume coordinate system; and a component for providing second signaling information (807), wherein the second signaling information includes information related to a transformation of a model domain (802) to a world domain (811), wherein the model domain (802) represents the point cloud frame by a plurality of points in a second volume coordinate system.

Description

Technical Field

[0001] The exemplary and non-limiting embodiments relate generally to multimedia and software, and more particularly to video-based cloud compression model to world signaling information. Background Art

[0002] Performing video encoding and decoding is known. Summary of the invention

[0003] The following summary is intended as an example only. This summary is not intended to limit the scope of the claims.

[0004] According to one aspect, an apparatus includes: a component for providing first signaling information, wherein the first signaling information includes information related to a world domain, wherein the world domain is a point cloud frame represented by multiple points in a first volume coordinate system; and a component for providing second signaling information, wherein the second signaling information includes information related to a transformation from a model domain to a world domain, wherein the model domain represents the point cloud frame by multiple points in a second volume coordinate system.

[0005] According to one aspect, an apparatus comprises: at least one processor; and at least one non-transitory memory comprising computer program code; wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least perform: providing first signaling information, the first signaling information comprising information related to a world domain, wherein the world domain is a point cloud frame represented by a plurality of points in a first volume coordinate system; and providing second signaling information, the second signaling information comprising information related to a transformation from a model domain to a world domain, wherein the model domain represents the point cloud frame by a plurality of points in a second volume coordinate system.

[0006] According to one aspect, a method includes: providing first signaling information, the first signaling information including information related to a world domain, wherein the world domain is a point cloud frame represented by multiple points in a first volume coordinate system; and providing second signaling information, the second signaling information including information related to a transformation from a model domain to a world domain, wherein the model domain represents the point cloud frame by multiple points in a second volume coordinate system.

[0007] According to one aspect, a machine-readable, non-transitory program storage device is provided, tangibly embodying a program of instructions executable by the machine for performing operations, the operations comprising: providing first signaling information, the first signaling information comprising information related to a world domain, wherein the world domain is a point cloud frame represented by a plurality of points in a first volume coordinate system; and providing second signaling information, the second signaling information comprising information related to a transformation of a model domain to a world domain, wherein the model domain represents the point cloud frame by a plurality of points in a second volume coordinate system. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The foregoing aspects and other features are explained in the following description in conjunction with the accompanying drawings, in which:

[0009] Figure 1 is a block diagram depicting an overview of the video-based point cloud compression (V-PCC) process.

[0010] Figure 2 is a block diagram depicting an overview of the V-PCC decompression process.

[0011] Figure 3 A sample image (1 frame) of the point cloud depicting the example.

[0012] Figure 4 is an illustration depicting an example point cloud projected onto a “bounding box” plane.

[0013] Figure 5A An example depicting projected geometry.

[0014] Figure 5B Example of depicting a texture image.

[0015] Figure 6 An example of a diagrammatic layer projection structure.

[0016] Figure 7 is a diagram showing the separation of stacked surface patches.

[0017] Figure 8 An example V-PCC encoder operation is shown.

[0018] Fig. 9 An example V-PCC decoder operation is shown.

[0019] Fig.10 An example apparatus configured to implement video-based cloud compression model-to-world signaling based on the examples described herein.

[0020] Fig.11 An example method for implementing video-based cloud compression model-to-world signaling based on the examples described in this article. DETAILED DESCRIPTION

[0021] The following abbreviations and acronyms that may be found in the specification and / or drawings are defined as follows:

[0022] 2D or 2d two-dimensional

[0023] 3D or 3d three-dimensional

[0024] 6DOF Six Degrees of Freedom

[0025] AR Augmented Reality

[0026] AVC Advanced Video Coding

[0027] CD Context Description

[0028] Codec

[0029] CGI Computer Generated Imagery

[0030] GI Geometry Information

[0031] HEVC High Efficiency Video Coding

[0032] HEVCRExt HEVC format range extension

[0033] HEVC-SCC HEVC Screen Content Coding

[0034] HM HEVC Test Model

[0035] IEC International Electrotechnical Commission

[0036] ISO International Organization for Standardization

[0037] ISOBMFF ISO Base Media File Format

[0038] KD k-dimensional

[0039] MPEG Moving Picture Experts Group

[0040] MPEG-I MPEG Immersive

[0041] MR Mixed Reality

[0042] MSE Mean Square Error

[0043] N# International Organization for Standardization document number

[0044] OM Occupancy Map

[0045] PCC Point Cloud Coding / Compression

[0046] Pcm Pulse Code Modulation

[0047] Psd power spectral density

[0048] RGB Red, Green, Blue

[0049] SEI Supplemental Enhancement Information

[0050] Sps sequence parameter set

[0051] TMC2 Text Model Category 2

[0052] TMC2v0 Text Model Category 2 Version 0

[0053] V-PCC Video-based point cloud coding / compression

[0054] VR Virtual Reality

[0055] WxH width times height

[0056] Examples described herein relate to encoding and decoding of digital volumetric video.

[0057] Volumetric video data

[0058] Volumetric video data represents a three-dimensional scene or object and can be used as input for AR, VR, and MR applications. This data describes the geometry (shape, size, position in 3D space) and corresponding attributes (e.g., color, opacity, reflectivity, ...), as well as any possible temporal changes in the geometry and attributes at a given time instance (such as a frame in a 2D video). Stereoscopic video is either generated from a 3D model (i.e., CGI) or captured from a real-world scene using a variety of capture solutions (e.g., a combination of multiple cameras, laser scanning, video, and dedicated depth sensors, etc.). In addition, a combination of CGI and real-world data is also possible. Typical representation formats for this volumetric data are triangular meshes, point clouds, or voxels. Temporal information about the scene can be included in the form of individual capture instances (i.e., "frames" in 2D video) or in other ways (e.g., the position of an object as a function of time).

[0059] Because volumetric video describes a 3D scene (or object), such data can be viewed from any viewpoint. Therefore, volumetric video is an important format for any AR, VR or MR application, especially for providing 6DOF viewing capabilities.

[0060] The increase in computing resources and the advancement of 3D data acquisition equipment have enabled the reconstruction of highly detailed volumetric video representations of natural scenes. Infrared, laser, time of flight and structured light are all examples of devices that can be used to construct 3D video data. The representation of 3D data depends on how the 3D data is used. Dense voxel arrays have been used to represent volumetric medical data. In 3D graphics, polygonal meshes are widely used. On the other hand, point clouds are very suitable for capturing applications such as real-world 3D scenes where the topology is not necessarily a 2D manifold. Another method of representing 3D data is to encode this 3D data into a set of textures and depth maps, as in the case of multi-view plus depth. Closely related to the technology used in multi-view plus depth is the use of height maps and multi-level surface maps.

[0061] Compression of volumetric video data

[0062] In dense point clouds or voxel arrays, the reconstructed 3D scene can contain tens of millions or even hundreds of millions of points. If such a representation is to be stored or exchanged between entities, efficient compression becomes critical. Standard volumetric video representation formats (such as point clouds, grids, voxels) have the problem of poor temporal compression performance. Identifying motion-compensated correspondences in 3D space is an ill-defined problem because both the geometry and the corresponding attributes can change. For example, temporally consecutive "frames" may not have the same number of grids, points, or voxels. Therefore, the compression efficiency of dynamic 3D scenes is very low. 2D video-based methods (i.e., multi-view + depth) for compressing volumetric data have much better compression efficiency, but rarely cover the entire scene. Therefore, they only provide limited 6DOF capabilities.

[0063] Instead of the above-mentioned methods, a 3D scene represented as a grid, point and / or voxel can be projected onto one or more geometric structures. These geometric structures are "unfolded" onto 2D planes (two planes per geometric structure: one for texture and one for depth), which are then encoded using standard 2D video compression techniques. The relevant projected geometry information is sent to the decoder along with the encoded video file. The decoder decodes the video and performs inverse projection to regenerate the 3D scene in any desired representation format (not necessarily the starting format).

[0064] Projecting the volume model onto a 2D plane allows the use of standard 2D video coding tools with efficient time compression. Therefore, the coding efficiency is greatly improved. Using geometric projections instead of the prior art 2D video-based methods (i.e., multi-view + depth) allows for better coverage of the scene (or objects). Therefore, the 6DOF capability is improved. Using several geometric structures for individual objects further improves the coverage of the scene. In addition, standard video coding hardware can be used for real-time compression / decompression of the projection plane. The complexity of the projection and back-projection steps is very low.

[0065] MPEG Video-based Point Cloud Coding (V-PCC) Test Model (MPEG N18017)

[0066] For the sake of clarity, some selected excerpts from MPEG document N18017 are provided. Figure 1 and Figure 2 Provides an overview of the compression / decompression process implemented in V-PCCTMC2. In particular, Figure 1 is a block diagram 100 depicting an overview of a video-based point cloud compression (V-PCC) compression process, and Figure 2 is a block diagram 200 depicting an overview of the V-PCC decompression process.

[0067] In the encoding phase, the input point cloud frame is processed in the following way: First, the volumetric 3D data is represented as a set of 3D projections in different components. In the separation phase, the image is decomposed into the far and near components of the geometric structure and the corresponding attribute components. In addition, the occupancy Figure 2 d image to indicate the portion of the image that can be used. The 2D projection consists of independent patches (104) based on the geometric characteristics of the input point cloud frame (102). After the patches (104) have been generated (108) and the 2D frame for video encoding has been created, the occupancy map (106), geometric information (110) and auxiliary information can be compressed (112, 114, 116, 118, 120). At the end of the process, the separate bitstreams are multiplexed (122) into an output compressed binary file (124).

[0068] The decoding process begins by demultiplexing (202) the input compressed into a binary file (124) into geometry (206), attributes (204), occupancy map (208) and side information (210) streams. The side information stream (210) is entropy coded (a detailed description of coding methods for side information compression is provided in MPEG N18017). The occupancy map can be compressed using an entropy coding method or a video compression method depending on the level selected. The geometry stream is decoded (212) and combined with the occupancy map and side information, whereby smoothing (214, 216) is applied to reconstruct the point cloud geometry (218). Based on the decoded attribute video stream (220) and the reconstructed information for the smoothed geometry (212 and 214), the occupancy map and side information attributes (218) of the point cloud can be reconstructed. After the attribute reconstruction stage, an additional attribute smoothing method is used for point cloud refinement.

[0069] therefore, Figure 1 The structural block diagram 100 shown in FIG. 1 is used for encoding, and Figure 2 The structural block diagram 200 is used for decoding.

[0070] Point cloud representation in V-PCC

[0071] Each point cloud frame represents a data set of points with unique coordinates and attributes within a 3D volume space. Figure 3 An example of a point cloud frame 300 is shown in FIG.

[0072] Patch Generation

[0073] The patch generation process decomposes the point cloud frame by converting 3D samples into 2D samples on a given projection plane using a strategy that provides the best compression. In TMC2v0, the patch generation process aims to decompose the point cloud into the minimum number of patches with smooth boundaries while also minimizing the reconstruction error.

[0074] In the initial stage, the normal per each point is estimated. Based on the nearest neighbors m of the point within a predefined search distance, a tangent plane and its corresponding normal are defined for each point. A KD tree is used to separate the data and find the normal at point p. i nearby neighbors, and the centroid of that point set to define the normal. The center of gravity c is calculated as follows:

[0075]

[0076] The normal is estimated from the eigen decomposition for the defined point cloud as:

[0077]

[0078] Based on this information, each point is associated with a corresponding plane of the point cloud bounding box. Each plane is represented by a corresponding normal with the following values definition:

[0079] -(1.0,0.0,0.0)

[0080] -(0.0,1.0,0.0)

[0081] -(0.0,0.0,1.0)

[0082] -(-1.0,0.0,0.0)

[0083] -(0.0,-1.0,0.0)

[0084] -(0.0,0.0,-1.0)

[0085] More precisely, each point is associated with the plane with the closest normal (i.e., the plane that maximizes the point normal and the plane normal dot product of ).

[0086]

[0087] The sign of the normal is defined by the position of the point relative to the "center". Figure 4 The projection estimation description is demonstrated in . Figure 4 is a diagram 400 depicting an example point cloud projected onto a “bounding box” plane.

[0088] In turn, the initial clusters are refined by iteratively updating the cluster index associated with each point based on the point's normal and the cluster index of the point's closest neighbors.The next step involves extracting patches by applying a connected component extraction process.

[0089] Patch Packaging

[0090] The patch packing process tries to generate geometry and texture maps by properly taking into account the generated patches and by trying to efficiently place the geometry or texture data corresponding to each patch onto a 2D grid of size WxH. This placement also takes into account the use of a defined minimum size block TxT, which specifies the minimum distance between different patches as placed on this 2D grid. The corresponding value (T) is encoded and sent to the decoder.

[0091] Figure 5A An example 500 of a projection geometry is depicted, and Figure 5B An example 550 of a texture image is depicted.

[0092] The packing method in TMC2 uses an exhaustive search algorithm as follows: Initially, patches are placed on a 2D grid in a way that ensures non-overlapping insertion. Samples belonging to patches (rounded to values ​​that are multiples of T) are considered to occupy blocks. In addition, the guard between adjacent patches is enforced to be a distance of at least one block (a multiple of T). The patches are processed in an ordered manner based on a patch index list. Each patch from the list is iteratively placed on the grid. The grid resolution depends on the size of the original point cloud, and its width (W) and height (H) are sent to the decoder. In the absence of empty space available for the next patch, the height value of the grid is initially doubled and the insertion of the patch is evaluated again. If the insertion of all patches is successful, the height is trimmed to the minimum required value. However, it is not allowed to set this value lower than the value originally specified in the encoder. The final values ​​of W and H correspond to the frame resolution used to encode texture and geometry video signals using a suitable video codec.

[0093] Image Generation

[0094] The image generation process utilizes the 3D to 2D mapping computed during the packing process to store the geometry and texture of the point cloud as an image. To better handle the case where multiple points are projected to the same sample, each patch is projected onto two images called layers. More precisely, let H(u,v) be the set of points of the current patch that are projected to the same sample (u,v).

[0095] Figure 6 An example of a layer projection structure 600 is illustrated. The first layer (also referred to as the near layer) 602 stores the points in H(u,v) with the lowest depth D0. The second layer (referred to as the far layer) 604 captures the points in H(u,v) with the highest depth in the interval [D0, D0+Δ], where Δ is a user-defined parameter describing the thickness of the surface.

[0096] When there is a stack of multiple different surfaces in the connected component, a surface separation method is applied to prevent the mixing of different surfaces in the connected component. One of the methods used to separate the surfaces is to use the difference in the MSE values ​​of the points in the RGB color space: if

[0097] MSE(R1-R2,G1-G2,B1-B2)>threshold;threshold=20

[0098] Then separate the patch.

[0099] Among them, R1, G1, and B1 are attribute values ​​belonging to T0, and R2, G2, and B2 are attribute values ​​belonging to T1.

[0100] Figure 7 is a diagram showing stacked surface patch separation 700. Note that this example geometry video is monochrome.

[0101] filling

[0102] The padding process aims to fill the empty spaces between patches in order to produce a piecewise smooth image suitable for video compression. The following method is used:

[0103] • Each block of TxT (eg, 16x16) pixels is processed independently.

[0104] If a block is empty (ie, all its pixels belong to empty space), fill the block with pixels by copying the last row or column of the previous TxT block in raster order.

[0105] If the block is full (ie, no empty pixels), do nothing.

[0106] • If a block has both empty and filled pixels, the empty pixels are iteratively filled with the average of their non-empty neighbors.

[0107] Video Compression

[0108] The generated images / layers are stored as video frames and compressed using the HM 16.16 video codec according to the HM configuration provided as a parameter.

[0109] Auxiliary patch information

[0110] For each patch, the following metadata is encoded / decoded:

[0111] Index of the projection plane

[0112] ο Index 0 for the planes (1.0, 0.0, 0.0) and (-1.0, 0.0, 0.0)

[0113] ο Index 1 for the planes (0.0,1.0,0.0) and (0.0,-1.0,0.0)

[0114] ο Index 2 for the planes (0.0, 0.0, 1.0) and (0.0, 0.0, -1.0)

[0115] 2D bounding box (u0,v0,u1,v1)

[0116] The 3D position (x0, y0, z0) of the patch in terms of depth δ0, tangential offset s0 and bi-tangential offset r0. Depending on the chosen projection plane, (δ0, s0, r0) is calculated as follows:

[0117] ο Index 0, δ0 = x0, s0 = z0, r0 = y0

[0118] ο Index 1, δ0=y0, s0=z0, r0=x0

[0119] ο Index 2, δ0=z0, s0=x0, r0=y0

[0120] Additionally, mapping information providing for each TxT block its associated patch index is encoded as follows:

[0121] For each TxT block, let L be an ordered list of indices of patches such that their 2D bounding boxes contain the block. The order in this list is the same as the order used to encode the 2D bounding boxes. L is called the candidate patch list.

[0122] • The empty space between patches is considered a patch and is assigned a special index of 0, which is added to the candidate patch list for all blocks.

[0123] Let I be the index of the patch to which the current TxT block belongs, and let J be the position of I in L. Instead of explicitly encoding the index I, its position J is arithmetically encoded,

[0124] This results in better compression efficiency.

[0125] Occupancy Map (OM) Coding – Lossy Conditions

[0126] A two-dimensional binary image of resolution (Width / B0)x(Height / B1), where Width and Height are the width and height of the geometry and texture images to be compressed. Samples equal to 1 mean that one or more corresponding / co-located samples in the geometry and texture images can be treated as point cloud points when decoding, while samples equal to 0 can be ignored (usually including padding information). The resolution of the occupancy map does not have to be the same as the resolution of the geometry and texture images, but the occupancy map can be encoded with a precision of B0xB1 blocks. In order to achieve lossless coding, B0 and B1 are chosen to be equal to 1. In practice, B0=B1=2 or B0=B1=4 can produce visually acceptable results while significantly reducing the number of bits required to encode the occupancy map. The generated binary image covers only a single color plane. However, given the prevalence of 4:2:0 codecs, it may be desirable to extend the image with a "neutral" or fixed-value chrominance plane (e.g., adding a chrominance plane with all sample values ​​equal to 0 or 128, assuming an 8-bit codec is used).

[0127] The obtained video frames are compressed by using a video codec (eg, AVC, HEVC RExt, HEVC-SCC) with lossless coding tool support.

[0128] The occupancy map is simplified by detecting empty blocks and non-empty blocks of resolution TxT in the occupancy map, and for non-empty blocks, their patch indices are encoded as follows:

[0129] • Create a list of candidate patches for each TxT block by considering all patches that contain the block.

[0130] The candidate list is sorted in reverse order of patches.

[0131] For each block,

[0132] 1. If the candidate list has one index, no encoding is done.

[0133] 2. Otherwise, arithmetic encode the index of the patch in this list.

[0134] Geometry reconstruction

[0135] The point cloud geometry reconstruction process utilizes the occupancy map information in order to detect non-empty pixels in the geometry / texture image / layer. The 3D positions of the points associated with these pixels are calculated using the auxiliary patch information and the geometry image. More precisely, let P be the point associated with the pixel (u,v), let (δ0,s0,r0) be the 3D position of the patch to which it belongs, and let (u0,v0,u1,v1) be its 2D bounding box. P can be expressed in terms of depth δ(u,v), tangential offset s(u,v) and bi-tangential offset r(u,v) as follows:

[0136] δ(u,v)=δ0+g(u,v)

[0137] s(u,v)=s0–u0+u

[0138] r(u,v)=r0–v0+v

[0139] Among them, g(u,v) is the brightness component of the geometric image.

[0140] smooth

[0141] The smoothing process aims to mitigate potential discontinuities that may appear at patch boundaries due to compression artifacts. The implemented method moves boundary points to the centroids of their closest neighbors.

[0142] Texture reconstruction

[0143] Texture values ​​are read directly from the texture image.

[0144] V-PCC Ancillary Information Bitstream Syntax (MPEG N18180)

[0145] The following bitstream syntax is taken from ISO / IEC 23090-5:2018 (E) CD study, available as MPEG document N18180. Only elements relevant to the examples described herein are selected. The semantics of each syntax element are provided in N18180, however, they are not necessary for understanding the examples described herein.

[0146] 7.3.3.4 Geometry Information Syntax

[0147]

[0148]

[0149] The syntax and semantics for the SEI message payload may be specified. SEI messages facilitate processes related to decoding, reconstruction, display, or other purposes. However, the decoding process does not require SEI messages. Conformance decoders are not required to process this information to conform the output order to a specific specification. Some SEI message information is required to check bitstream consistency and for output timing decoder consistency.

[0150] Regarding SEI payload syntax, the general SEI message syntax is as follows.

[0151]

[0152]

[0153] The input to the V-PCC encoder is a point cloud frame represented by a plurality of points in a first coordinate domain (the model domain). The V-PCC encoder converts the point cloud frame into another integer coordinate domain (the video-based point cloud compression domain, the vpcc domain). The conversion is done by forming the boundaries of the object represented by the point cloud frame with a volume (the quantization volume). In turn, the object is quantized based on the selected bit depth, and the bit depth is signaled in the gi_geometry_3d_coordinates_bitdepth_minus1 field in the VPCC bitstream. For example, this can be 10 bits, i.e., 1024x 1024x 1024 points.

[0154] The first problem: the VPCC bitstream does not provide the decoder with information on how to convert from the VPCC domain (video-based point cloud compression domain) back to the model domain.

[0155] Objects or groups of objects represented by point cloud frames in the model domain can be placed relative to yet another coordinate domain (the world domain). For example, animation effects can be achieved when the model domain origin changes frame by frame in the world domain.

[0156] In the current V-PCC specification, when there are multiple point cloud frames constituting a point cloud sequence, the origin of the model domain is the same as that of the world domain for all frames. In other words, the quantization volume is the same for all frames and is static in the world domain. The V-PCC encoder can calculate the quantization volume based on the first frame of the sequence or based on the number or frame of the sequence. In the former, the resolution of the point cloud frame may be the highest, but in some point cloud frames in the model domain, it may happen that the object is outside the quantization volume. In the latter, the V-PCC encoder can ensure that the object represented by the point cloud frame in the real domain is always represented by the point cloud frame in the model domain at the expense of the resolution of the object. In order to solve the above problems, the V-PCC encoder should adapt the origin of the model domain relative to the world domain on a frame-by-frame basis.

[0157] Second problem: V-PCC does not provide the possibility to adapt the quantization volume on a frame-by-frame basis.

[0158] After decoding the point cloud frame, the application may place it in a scene (AR or VR). The application's rendering engine may need to transform the point cloud frame based on external information (for example, the camera's viewpoint changes due to the movement of the VR headset). In order to perform this operation, the renderer needs to know which point (pivot point) to transform relative to.

[0159] The third problem: the V-PCC specification does not provide information about the pivot point of an object or group of objects represented by a point cloud frame.

[0160] To ensure an immersive experience in AR applications, the placement of the decoded point cloud frames in the scene should maintain the real-world size of the represented objects.

[0161] Fourth problem: The V-PCC specification does not provide information on how to convert the decoded point cloud frames in the encoder domain back to the real-world domain, so that the scale of the objects is preserved.

[0162] The example scaling related information described in this article solves the above mentioned problems.

[0163] A similar but simplified version of the signaling described herein was proposed by the inventors of the examples described herein in their contribution “Model scale and offset signaling for PCC” (m42470, April 2018, San Diego, USA).

[0164] In the examples described herein, signaling information about the vpcc to model to world domain conversion is introduced to improve encoding performance and maintain the highest possible fidelity of the decoded point cloud. Additionally, this signaling allows content creators to include information in the VPCC bitstream that ensures that the bitstream is consistent with the content creator's intent. Figure 1 The decoded object is reproduced consistently and can be reused in multiple scenes. Therefore, the examples described in this article achieve the following goals.

[0165] 1. Define signaling information about model to world conversion.

[0166] a. Signaling the pivot point of an object or group of objects represented by a point cloud frame in the video-based point cloud compression domain (vpcc domain).

[0167] b. Signaling scaling parameters for scaling an object or group of objects represented by a point cloud frame from the vpcc domain to the model domain.

[0168] c. Signaling rotation parameters that rotate the object or group of objects represented by the point cloud frame around the signaled pivot point so that it is correctly oriented in the world domain.

[0169] d. Signaling translation parameters that translate the object or group of objects represented by the point cloud frame relative to the signaled pivot point so that they are correctly placed in the world domain.

[0170] 2. Define signaling information about the world domain.

[0171] e. Length units in the signaling world domain.

[0172] f. The coordinate system of the signaling world domain.

[0173] g. Signaling the size of the bounding box around which an object or group of objects represented by a sequence of point cloud frames in the world domain is placed.

[0174] h. Signaling the pivot point of the bounding box in the world domain.

[0175] 3. Introducing new V-PCC encoder and decoder operations based on the proposed signaling.

[0176] benefit

[0177] Examples described herein provide several benefits and technical effects, including: i) encoders implementing pre-processing algorithms that improve compression efficiency, ii) content creators providing information about scaling and true size, iii) world coordinate domain dimensions of objects can be preserved, and iv) multiple point cloud models can be merged into the same scene.

[0178] In the described example, a new signaling structure is introduced into the V-PCCSEI signaling. The high-level block diagrams of the encoder and decoder of the described example are respectively shown in Figure 8 and Fig. 9 Depicted in.

[0179] Figure 8An example V-PCC encoder operation 850 of an example V-PCC encoder 800 is shown. The operation of the V-PCC encoder 800 can be as follows: a point cloud frame (811) in the world domain is provided as an input to the V-PCC encoder (800) together with side information (810). The side information module (810) can provide (including but not limited to) information about the length unit of the world domain, the bounding box of the object or group of objects represented by the sequence of point cloud frames in the world domain, the pivot point of the bounding box, and the world domain coordinate system axis. The conversion (801) performs a conversion of the point cloud frame (811) in the world domain into a point cloud frame (802) in the model domain and generates SEI messages (807, 808). The conversion module (801) can use the information provided by the side information (810), or it can perform calculations to obtain the necessary information, including but not limited to translation, rotation, and pivot point. The conversion module (801) operates on a frame-by-frame basis and it can find the most appropriate representation of an object or group of objects represented by a point cloud frame (802) in the model domain. The point cloud frame (802) in the model domain is provided to a quantization module (803), which converts the point cloud frame (802) in the model domain into a point cloud frame (804) in a video-based point cloud compression domain (vpcc domain). The quantization module (803) provides information on how to scale back from the vpcc domain to the model domain. This information is included in the SEI message model to world (808). The point cloud frame (804) in the vpcc domain is input to the patch data, occupancy map, geometry, attribute generation / encoding module (805). The generation / encoding module (805) operates according to the selected profile, tier, level parameters of the V-PCC specification and the VPCC bitstream (806). The output of the V-PCC encoder (800) is a VPCC bitstream (806) containing SEI messages (eg, 807, 808, or other one or more SEI messages).

[0180] Fig. 9An example V-PCC decoder operation 950 is shown. The operation of the V-PCC decoder may be as follows: A VPCC bitstream (901) containing SEI messages (907, 908) is provided as an input to a V-PCC decoder (902). The V-PCC decoder (902) operates in accordance with the V-PCC specification and generates a point cloud frame (903) in the vpcc domain. A conversion module (904) converts the point cloud frame (903) in the vpcc domain into a point cloud frame (905) in the world domain based on information signaled by the SEI message (908). The conversion module (904) may have an intermediate state, wherein the point cloud frame (903) in the vpcc domain is first transformed into a point cloud frame in the model domain, and then the point cloud frame in the model domain is transformed into a point cloud frame (905) in the world domain. The point cloud frame (905) in the world domain is provided to a rendering module (906) together with information signaled by the SEI message (907). The rendering module (906) may render the point cloud frame (905) in the world domain with appropriate scaling, orientation and translation. In actual implementation, some of the blocks (904, 905, 906) may reside in the application, where the PCC decoder only provides decoded video and metadata.

[0181] The SEI message (808, 908) structure contains information that allows the conversion (904) of a point cloud (804, 903) in the vpcc domain to a point cloud (811, 905) in the world domain.

[0182]

[0183]

[0184] This SEI message (808 or 908) provides information about how to transform the reconstructed point cloud frame from the decoder integer domain to the world domain starting from the current access unit. If applicable, the transformation can be performed in the following order: scaling, rotation, and translation. The rotation and translation can be performed relative to the indicated pivot point.

[0185] The message remains valid until the next SEI message of the same type or the end of the vpcc sequence, whichever is earlier in decoding order.

[0186] precision_minus3 indicates the value used to calculate the number of bits used for the translation (x, y, z), rotation (x, y, z), scale, and axis (x, y, z) fields.

[0187] The translation_flag indicates whether translation information is signaled. A translation_flag equal to 1 indicates, for example, that translation information is signaled. A translation_flag equal to 0 indicates, for example, that no translation information is signaled.

[0188] rotation_flag indicates whether rotation information is signaled. A rotation_flag equal to 1 indicates, for example, that rotation information is signaled. A rotation_flag equal to 0 indicates, for example, that rotation information is not signaled.

[0189] scale_flag indicates whether scaling information is signaled. scale_flag equal to 1 indicates, for example, that scaling information is signaled. scale_flag equal to 0 indicates, for example, that scaling information is not signaled. pivot indicates the position of the pivot point.

[0190] Table Y axis regulations

[0191] Axis describe 0 Center of bounding box (quantized volume) 1 The origin of the point cloud coordinate system 2 reserve 3 Explicit signaling of the bounding box's origin

[0192] translation_x, translation_y, translation_z indicate the translation along the x-axis, y-axis, and z-axis, respectively. The values ​​of translation_x, translation_y, and translation_z are stored as fixed-point values ​​(see Table X). The number of bits used to represent translation_x, translation_y, translation_z is equal to 2 precision_minus3+3 .

[0193] rotation_x, rotation_y, rotation_z indicate a geometric rotation quaternion. The fourth component of the quaternion can be calculated by the following equation:

[0194] rotation_x 2 +rotation_y 2 +rotation_z 2 +rotation_w 2 =1.0

[0195] The values ​​of rotation_x, rotation_y and rotation_z are stored as fixed point values ​​(see Table X). The number of bits used to represent rotation_x, rotation_y and rotation_z is equal to 2 precision_minus3+3 .

[0196] scale indicates the uniform scaling of the object. The value of scale is stored as a fixed point value (see Table X). The number of bits used to represent the scale is equal to 2 precision_minus3+3 .

[0197] pivot_x, pivot_y, pivot_z indicate the pivot position relative to the origin of the coordinate system. The values ​​of pivot_x, pivot_y, and pivot_z are stored as fixed point values ​​(see Table X). The number of bits used to represent pivot_x, pivot_y, pivot_z is equal to 2 precision_minus3+3 .

[0198] Table X provides example values ​​for the aforementioned fixed point values.

[0199] Table X: Fixed point representation based on precision_minus3 values

[0200]

[0201] A fixed-point number is a signed or unsigned value obtained by dividing an integer by an appropriate power of 2. For example, the fixed-point number 30.2 is obtained by dividing a 32-bit integer by 2. 2 The fixed point number 16.16 is formed by dividing the 32-bit integer by 2 16 The fixed point number 2.6 is formed by dividing the 8-bit integer by 2 6 Formed.

[0202] The SEI messages (eg, 807, 907) signaling information about the world region may be used by the rendering module (906).

[0203]

[0204]

[0205] This SEI message (eg, 807, 907) provides information about the world region. The message remains valid until the next SEI message of the same type or the end of the vpcc sequence (whichever is earlier in decoding order).

[0206] The world's pivot point may be the center of the volume when volume_present_flag is equal to 1. Otherwise, the origin of the coordinate system may be used as the pivot point.

[0207] unit_of_length_flag indicates the type of length unit defined. unit_of_length_flag equal to 1 indicates that the length unit is meter. unit_of_length_flag equal to 0 indicates that the length unit is arbitrary.

[0208] world_orientation indicates the world orientation used. world_orientation equal to 0 indicates that the orientation is as defined in ISO / IEC 23090-2. world_orientation equal to 1 indicates that the orientation is specified by the unit upvector (up_x, up_y, up_z) and the unit front vector (front_x, front_y, front_z). The coordinate system specified in ISO / IEC 23090-2 is the same as in ISO / IEC 23008-3.

[0209] volume_present_flag indicates whether volume information is signaled.

[0210] up_x, up_y, up_z indicate unit vectors describing the up direction in a right-handed coordinate system. The values ​​of up_x, up_y, and up_z are stored as 32.32 fixed point values.

[0211] front_x, front_y, front_z indicate unit vectors describing the front direction in a right-handed coordinate system. The values ​​of front_x, front_y, and front_z are stored as 32.32 fixed point values.

[0212] volume_x, volume_y, volume_z indicate the volume dimensions along the x-axis, y-axis, and z-axis, respectively. The values ​​of volume_x, volume_y, and volume_z are stored as 32.32 fixed point values.

[0213] pivot indicates the location of the pivot point in the world.

[0214] Table Y axis regulations

[0215] Axis describe 0 The origin of the point cloud coordinate system 1 Explicit signaling of the bounding box's origin

[0216] pivot_x, pivot_y, pivot_z indicate the pivot position relative to the origin of the coordinate system. The pivot_x, pivot_y, and pivot_z values ​​are stored as 64.64 fixed point values.

[0217] Additional Embodiments: In another embodiment, the information of the SEI messages (807, 808, 907, 908) is not delivered within the VPCC bitstream (806, 901), but is equivalently conveyed by other means (e.g., as timing metadata associated with the VPCC track within ISOBMFF).

[0218] Additional Embodiments: In another embodiment, the information of the SEI message (807, 808, 907, 908) is delivered through other structures within the VPCC bitstream (806, 901) (e.g., as part of a sequence parameter set or a frame parameter set).

[0219] Additional Embodiments: In another embodiment, the conversion module (801) is not a part of the VPCC encoder (800), but is an independent pre-processing module that provides equivalent information to the VPCC encoder (800).

[0220] Additional embodiment: In another embodiment, the length unit may be signaled according to the value of unit_of_length_flag. For example, if unit_of_length == 0, a fixed point scaling value may be followed that explicitly gives a length of one unit, or a signed power value may be used to derive the length unit.

[0221] Additional Embodiments: In another embodiment, the Pivot Center field may have more bits dedicated to another indicating predefined field (eg, the center of the floor plane of the bounding box).

[0222] The examples described herein may contribute to standardization, such as in MPEG-I 23090 Part 5 - Video-based Point Cloud Compression.

[0223] The structures and concepts described in this document may be included in a standard as normative text, and therefore any product claiming to comply with the requirements may be considered an adopter.

[0224] Fig.10An example device 1000 that can be implemented in hardware is configured to implement model-to-world signaling for video-based cloud compression based on the examples described herein. The device 1000 includes a processor 1002, at least one non-transitory memory 1004 including computer program code 1005, wherein the at least one memory 1004 and the computer program code 1005 are configured to, together with at least one processor 1002, cause the device to implement circuits, processes, components, modules, or functions (collectively referred to as 1006) to implement signaling as described herein. The device 1000 optionally includes a display and / or I / O interface 1008, which can be used to display various aspects or states of the methods described herein (e.g., while the method is being executed or at a subsequent time). The display and / or I / O interface 1008 can also be configured to receive input such as user input. The device 1000 also optionally includes one or more network (NW) interfaces (I / F) 1010. NW I / F 1010 may be wired and / or wireless and communicate over the Internet / other network via any communication technology. NW I / F 1010 may include one or more transmitters and one or more receivers. Device 1000 may be configured as a server or client based on the various signaling aspects described herein (e.g., device 1000 may be a remote, virtual, or cloud device).

[0225] References to "computers", "processors", etc. should be understood to cover not only computers with different architectures such as single / multiple processor architectures and serial (von Neumann) / parallel architectures, but also special-purpose circuits such as field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), signal processing devices, and other processing circuits. References to computer programs, instructions, code, etc. should be understood to cover software for a programmable processor, or firmware, such as programmable content of a hardware device, which may include instructions for a processor, or configuration settings for a fixed-function device, gate array, or programmable logic device, etc.

[0226] The memory 1004 may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The memory 1004 may include a database for storing data.

[0227] As used in this application, the term "circuitry" refers to all of the following: (a) hardware circuit implementations only (such as implementations of analog and / or digital circuits only); (b) combinations of circuits and software (and / or firmware), such as (if applicable): (i) a combination of processors; or (ii) portions of processors / software, including digital signal processors, software, and memory, which work together to enable the device to perform various functions; and (c) circuits, such as a microprocessor or a portion of a microprocessor, that require software or firmware to operate, even if the software or firmware is not physically present. This definition of "circuitry" applies to all uses of the term in this application. As another example, as used in this application, the term "circuitry" also covers an implementation of only a processor (or multiple processors) or a portion of a processor and its accompanying software and / or firmware. The term "circuitry" also covers (for example and if applicable to the specifically required element) a baseband integrated circuit or application processor integrated circuit for a mobile phone, or a similar integrated circuit in a server, cellular network device, or another network device.

[0228] Fig.11 An example method 1100 for implementing six-degree-of-freedom spatial layout signaling based on the examples described herein is provided. At 1102, the method includes providing first signaling information including information related to a world domain, wherein the world domain is a point cloud frame represented by a plurality of points in a first volume coordinate system. At 1104, the method includes providing second signaling information including information related to a transformation from a model domain to a world domain, wherein the model domain represents the point cloud frame by a plurality of points in a second volume coordinate system.

[0229] An example apparatus includes: a component for providing first signaling information, wherein the first signaling information includes information related to a world domain, wherein the world domain is a point cloud frame represented by multiple points in a first volume coordinate system; and a component for providing second signaling information, wherein the second signaling information includes information related to a transformation of a model domain to a world domain, wherein the model domain represents the point cloud frame by multiple points in a second volume coordinate system.

[0230] The apparatus may further include: a component for signaling one or more camera parameters in an adaptation parameter set of the extended point cloud frame.

[0231] The apparatus may further include means for signaling the presence or absence of one or more camera parameters.

[0232] The apparatus may further include: wherein the one or more camera parameters are related to at least one of scale, offset, or rotation and are used for reconstruction after the decoding operation.

[0233] The apparatus may further include means for signaling one or more volume availability parameters in the sequence parameter set.

[0234] The apparatus may further include: wherein the volume availability parameters include one or more of: an indication of the one or more volume availability parameters expressed in meters; an indication of the presence or absence of display frame information; or an indication of the presence or absence of an anchor point.

[0235] The apparatus may further include: means for performing adaptation of the generated point set in reconstruction of the point cloud frame.

[0236] The apparatus may further include: wherein the adaptation uses as input: a number of points of a point cloud frame; an array of positions of points in the point cloud frame; an anchor point; and at least one of an offset matrix, a rotation matrix, or a scaling matrix.

[0237] The apparatus may further include: means for performing at least one of an operation of a video-based point cloud coding encoder or an operation of a video-based point cloud coding decoder based on the provided first signaling and the provided second signaling.

[0238] The apparatus may further include: a component for encoding a point cloud frame in the world domain into a video-based point cloud bitstream using a video-based point cloud code encoder; or a component for decoding a video-based point cloud bitstream using a video-based point cloud code decoder to render a point cloud frame in the world domain.

[0239] The apparatus may further include: wherein the first signaling information and the second signaling information are generated as a result of conversion of the point cloud frame in the world domain to the point cloud frame in the model domain.

[0240] The apparatus may further include: wherein the conversion of the point cloud frame in the world domain to the point cloud frame in the model domain determines a representation of the object or group of objects represented by the point cloud frame in the model domain.

[0241] The apparatus may further include: wherein rendering of the point cloud frame in the world domain is based on first signaling information.

[0242] The apparatus may further include: wherein conversion of a point cloud frame in a video-based point cloud compression domain to a point cloud frame in a world domain is based on second signaling information.

[0243] The device may further include: wherein the conversion of point cloud frames in the video-based point cloud compression domain to point cloud frames in the world domain includes an intermediate state, wherein the point cloud frames in the video-based point cloud compression domain are first transformed into point cloud frames in the model domain, and then the point cloud frames in the model domain are transformed into point cloud frames in the world domain.

[0244] The apparatus may further include: wherein the first signaling information signaling including information related to the world domain transmits one or more of: a signal length unit in the world domain; a coordinate system of the world domain; a size of a bounding box around which an object or group of objects represented by a sequence of multiple point cloud frames in the world domain is placed; or a pivot point of the bounding box in the world domain.

[0245] The apparatus may further include: wherein second signaling information including information related to the conversion from the model domain to the world domain signals one or more of: a pivot point of an object or group of objects represented by a point cloud frame in a video-based point cloud compression domain; scaling parameters for scaling the object or group of objects represented by the point cloud frame from the video-based point cloud compression domain to the world domain; rotation parameters for rotating the object or group of objects represented by the point cloud frame around the signaled pivot point so that they are correctly oriented in the world domain; or translation parameters for translating the object or group of objects represented by the point cloud frame relative to the signaled pivot point so that they are correctly placed in the world domain.

[0246] An example apparatus comprises: at least one processor; and at least one non-transitory memory comprising computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform: providing first signaling information comprising information related to a world domain, wherein the world domain is a point cloud frame represented by a plurality of points in a first volume coordinate system; and providing second signaling information comprising information related to a transformation of a model domain to a world domain, wherein the model domain represents the point cloud frame by a plurality of points in a second volume coordinate system.

[0247] The apparatus may further include: wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least perform: signaling one or more camera parameters in an adaptation parameter set of the extended point cloud frame.

[0248] The apparatus may further comprise: wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform: signaling the presence or absence of one or more camera parameters.

[0249] The apparatus may further include: wherein the one or more camera parameters are related to at least one of scale, offset, or rotation and are used for reconstruction after the decoding operation.

[0250] The apparatus may further include: wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least perform: signaling one or more volume availability parameters in the sequence parameter set.

[0251] The apparatus may further include: wherein the volume availability parameters include one or more of: an indication of the one or more volume availability parameters expressed in meters; an indication of the presence or absence of display frame information; or an indication of the presence or absence of an anchor point.

[0252] The apparatus may further include: wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least perform: performing adaptation of the generated point set in reconstruction of the point cloud frame.

[0253] The apparatus may further include: wherein the adaptation uses as input: a number of points of a point cloud frame; an array of positions of points in the point cloud frame; an anchor point; and at least one of an offset matrix, a rotation matrix, or a scaling matrix.

[0254] The device may further include: wherein the at least one memory and the computer program code are configured to, together with at least one processor, cause the device to at least perform: based on the provided first signaling and the provided second signaling, perform at least one of the operations of a video-based point cloud coding encoder or the operations of a video-based point cloud coding decoder.

[0255] The apparatus may further include: wherein the at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to at least perform: using a video-based point cloud code encoder to encode a point cloud frame in a world domain into a video-based point cloud bitstream; or using a video-based point cloud code decoder to decode a video-based point cloud bitstream to render a point cloud frame in a world domain.

[0256] The apparatus may further include: wherein the first signaling information and the second signaling information are generated as a result of conversion of the point cloud frame in the world domain to the point cloud frame in the model domain.

[0257] The apparatus may further include: wherein the conversion of the point cloud frame in the world domain to the point cloud frame in the model domain determines a representation of the object or group of objects represented by the point cloud frame in the model domain.

[0258] The apparatus may further include: wherein rendering of the point cloud frame in the world domain is based on first signaling information.

[0259] The apparatus may further include: wherein conversion of a point cloud frame in a video-based point cloud compression domain to a point cloud frame in a world domain is based on second signaling information.

[0260] The device may further include: wherein the conversion of point cloud frames in the video-based point cloud compression domain to point cloud frames in the world domain includes an intermediate state, wherein the point cloud frames in the video-based point cloud compression domain are first transformed into point cloud frames in the model domain, and then the point cloud frames in the model domain are transformed into point cloud frames in the world domain.

[0261] The apparatus may further include: wherein the first signaling information signaling including information related to the world domain transmits one or more of: a signal length unit in the world domain; a coordinate system of the world domain; a size of a bounding box around which an object or group of objects represented by a sequence of multiple point cloud frames in the world domain is placed; or a pivot point of the bounding box in the world domain.

[0262] The apparatus may further include: wherein second signaling information including information related to the conversion from the model domain to the world domain signals one or more of: a pivot point of an object or group of objects represented by a point cloud frame in a video-based point cloud compression domain; scaling parameters for scaling the object or group of objects represented by the point cloud frame from the video-based point cloud compression domain to the world domain; rotation parameters for rotating the object or group of objects represented by the point cloud frame around the signaled pivot point so that they are correctly oriented in the world domain; or translation parameters for translating the object or group of objects represented by the point cloud frame relative to the signaled pivot point so that they are correctly placed in the world domain.

[0263] An example method includes providing first signaling information including information related to a world domain, wherein the world domain is a point cloud frame represented by a plurality of points in a first volume coordinate system; and providing second signaling information including information related to a transformation of a model domain to a world domain, wherein the model domain represents the point cloud frame by a plurality of points in a second volume coordinate system.

[0264] The method may further include: signaling one or more camera parameters in an adaptation parameter set of the extended point cloud frame.

[0265] The method may further include signaling the presence or absence of one or more camera parameters.

[0266] The method may further include: wherein the one or more camera parameters are related to at least one of scale, offset, or rotation and are used for reconstruction after the decoding operation.

[0267] The method may further include signaling one or more volume availability parameters in the sequence parameter set.

[0268] The method may further include: wherein the volume availability parameters include one or more of: an indication of the one or more volume availability parameters in meters; an indication of the presence or absence of display frame information; or an indication of the presence or absence of an anchor point.

[0269] The method may further comprise performing adaptation of the generated point set in the reconstruction of the point cloud frame.

[0270] The method may further include: wherein the adaptation uses as input: the number of points of the point cloud frame; an array of positions of the points in the point cloud frame; an anchor point; and at least one of an offset matrix, a rotation matrix, or a scaling matrix.

[0271] The method may further include: performing at least one of the operations of a video-based point cloud coding encoder or the operations of a video-based point cloud coding decoder based on the provided first signaling and the provided second signaling.

[0272] The method may further include: using a video-based point cloud code encoder to encode the point cloud frame in the world domain into a video-based point cloud bitstream; or using a video-based point cloud code decoder to decode the video-based point cloud bitstream to render the point cloud frame in the world domain.

[0273] The method may further include: wherein the first signaling information and the second signaling information are generated as a result of conversion of the point cloud frame in the world domain to the point cloud frame in the model domain.

[0274] The method may further include: wherein the conversion of the point cloud frame in the world domain to the point cloud frame in the model domain determines a representation of the object or group of objects represented by the point cloud frame in the model domain.

[0275] The method may further include: wherein rendering of the point cloud frame in the world domain is based on first signaling information.

[0276] The method may further include: wherein the conversion of the point cloud frame in the video-based point cloud compression domain to the point cloud frame in the world domain is based on second signaling information.

[0277] The method may further include: wherein, the conversion of point cloud frames in the video-based point cloud compression domain to point cloud frames in the world domain includes an intermediate state, wherein the point cloud frames in the video-based point cloud compression domain are first transformed into point cloud frames in the model domain, and then the point cloud frames in the model domain are transformed into point cloud frames in the world domain.

[0278] The method may further include: wherein the first signaling information including information related to the world domain signals one or more of: a signal length unit in the world domain; a coordinate system of the world domain; a size of a bounding box around which an object or group of objects represented by a sequence of multiple point cloud frames in the world domain is placed; or a pivot point of the bounding box in the world domain.

[0279] The method may further include: wherein, second signaling information including information related to the conversion of the model domain to the world domain signals one or more of the following: a pivot point of an object or a group of objects represented by a point cloud frame in a video-based point cloud compression domain; a scaling parameter for scaling the object or the group of objects represented by the point cloud frame from the video-based point cloud compression domain to the world domain; a rotation parameter for rotating the object or the group of objects represented by the point cloud frame around the signaled pivot point so that they are correctly oriented in the world domain; or a translation parameter for translating the object or the group of objects represented by the point cloud frame relative to the signaled pivot point so that they are correctly placed in the world domain.

[0280] A non-transitory program storage device readable by a machine may be provided, tangibly embodying a program of instructions executable by the machine for performing operations, the operations comprising: providing first signaling information, the first signaling information comprising information related to a world domain, wherein the world domain is a point cloud frame represented by a plurality of points in a first volume coordinate system; and providing second signaling information, the second signaling information comprising information related to a transformation of a model domain to a world domain, wherein the model domain represents the point cloud frame by a plurality of points in a second volume coordinate system.

[0281] The operations of the non-transitory program storage device may further include signaling one or more camera parameters in an adaptation parameter set of the extended point cloud frame.

[0282] Operation of the non-transitory program storage device may further include signaling the presence or absence of one or more camera parameters.

[0283] The non-transitory program storage device may further include: wherein the one or more camera parameters are related to at least one of scaling, offset, or rotation and are used for reconstruction after the decoding operation.

[0284] Operations of the non-transitory program storage device may further include signaling one or more volume availability parameters in the sequence parameter set.

[0285] The non-transitory program storage device may further include: wherein the volume availability parameters include one or more of: an indication of one or more volume availability parameters expressed in meters; an indication of the presence or absence of display frame information; or an indication of the presence or absence of an anchor point.

[0286] The operations of the non-transitory program storage device may further include performing adaptation of the generated point set in reconstruction of the point cloud frame.

[0287] The non-transitory program storage device may further include: wherein the adaptation uses as input: a number of points of a point cloud frame; an array of positions of points in the point cloud frame; an anchor point; and at least one of an offset matrix, a rotation matrix, or a scaling matrix.

[0288] The operation of the non-transitory program storage device may further include: performing at least one of the operation of a video-based point cloud coding encoder or the operation of a video-based point cloud coding decoder based on the provided first signaling and the provided second signaling.

[0289] The operations of the non-transitory program storage device may further include: using a video-based point cloud code encoder to encode the point cloud frame in the world domain into a video-based point cloud bitstream; or using a video-based point cloud code decoder to decode the video-based point cloud bitstream to render the point cloud frame in the world domain.

[0290] The non-transitory program storage device may further include: wherein the first signaling information and the second signaling information are generated as a result of conversion of the point cloud frame in the world domain to the point cloud frame in the model domain.

[0291] The non-transitory program storage device may further include: wherein the conversion of the point cloud frame in the world domain to the point cloud frame in the model domain determines a representation of the object or group of objects represented by the point cloud frame in the model domain.

[0292] The non-transitory program storage device may further include: wherein the rendering of the point cloud frame in the world domain is based on the first signaling information.

[0293] The non-transitory program storage device may further include: wherein the conversion of the point cloud frame in the video-based point cloud compression domain to the point cloud frame in the world domain is based on second signaling information.

[0294] The non-transitory program storage device may further include: wherein, the conversion of a point cloud frame in a video-based point cloud compression domain to a point cloud frame in a world domain includes an intermediate state, wherein the point cloud frame in the video-based point cloud compression domain is first transformed into a point cloud frame in a model domain, and then the point cloud frame in the model domain is transformed into a point cloud frame in the world domain.

[0295] The non-transitory program storage device may further include: wherein the first signaling information signaling including information related to the world domain transmits one or more of the following: a signal length unit in the world domain; a coordinate system of the world domain; the size of a bounding box around which an object or group of objects represented by a sequence of multiple point cloud frames in the world domain is placed; or a pivot point of the bounding box in the world domain.

[0296] The non-transitory program storage device may further include: wherein second signaling information including information related to the conversion of the model domain to the world domain signals one or more of the following: a pivot point of an object or group of objects represented by a point cloud frame in a video-based point cloud compression domain; a scaling parameter for scaling the object or group of objects represented by the point cloud frame from the video-based point cloud compression domain to the world domain; a rotation parameter for rotating the object or group of objects represented by the point cloud frame around the signaled pivot point so that it is correctly oriented in the world domain; or a translation parameter for translating the object or group of objects represented by the point cloud frame relative to the signaled pivot point so that it is correctly placed in the world domain.

[0297] An example apparatus may include one or more circuits configured to perform the method of any one of the signaling examples described herein.

[0298] It should be understood that the foregoing description is illustrative only. Various alternatives and modifications may be devised by those skilled in the art. For example, the features described in the various dependent claims may be combined with each other in any suitable combination. In addition, the features from the different embodiments described above may be selectively combined into new embodiments. Therefore, the above description is intended to cover all such alternatives, modifications and variations that fall within the scope of the appended claims.

Claims

1. A device comprising: A means for providing first signaling information, wherein the first signaling information comprises information related to a world domain, wherein the world domain is a point cloud frame represented by a plurality of points in a first volume coordinate system, wherein the first signaling information comprises information related to the world domain: - signal length unit in said world domain; - the coordinate system of the world domain; - the size of a bounding box around which an object or group of objects represented by a sequence of a plurality of point cloud frames in the world domain is placed; and Means for providing second signaling information, wherein the second signaling information comprises information related to a transformation of a model domain to the world domain, wherein the model domain represents the point cloud frame by a plurality of points in a second volume coordinate system.

2. The device according to claim 1, further comprising: Means for signaling one or more camera parameters in an adaptation parameter set that extends the point cloud frame.

3. The apparatus according to claim 2, further comprising: Means for signaling whether said one or more camera parameters exist.

4. The device according to any one of claims 2 to 3, wherein: The one or more camera parameters are related to at least one of scaling, offset, or rotation and are used for reconstruction after the decoding operation.

5. The device according to any one of claims 2 to 4, further comprising: Means for signaling one or more volume availability parameters in a sequence parameter set.

6. The device according to claim 5, wherein: The volume availability parameters include one or more of the following: an indication of the one or more volumetric availability parameters expressed in meters; an indication of whether box information exists; or An indication of whether an anchor exists.

7. The device according to any one of claims 1 to 6, further comprising: Means for performing adaptation of a generated point set in reconstruction of said point cloud frame.

8. The device according to claim 7, wherein: The adaptation uses the following as input: The number of points in the point cloud frame; An array of positions of points in the point cloud frame; Anchor point; as well as At least one of an offset matrix, a rotation matrix, or a scaling matrix.

9. The device according to any one of claims 1 to 8, further comprising: A component for performing at least one of the operations of a video-based point cloud coding encoder or the operations of a video-based point cloud coding decoder based on the provided first signaling and the provided second signaling.

10. The apparatus according to claim 9, further comprising: means for encoding the point cloud frames in the world domain into a video-based point cloud bitstream using the video-based point cloud encoding encoder; or Means for decoding the video-based point cloud bitstream using the video-based point cloud codec to render the point cloud frame in the world domain.

11. The device according to any one of claims 1 to 10, wherein: The first signaling information and the second signaling information are generated as a result of conversion of the point cloud frame in the world domain to the point cloud frame in the model domain.

12. The device according to any one of claims 1 to 11, wherein: The first signaling information includes information related to the world domain: The pivot point of the bounding box in the world region.

13. The device according to any one of claims 1 to 12, wherein: The second signaling information including information related to the conversion of the model domain to the world domain signals one or more of the following: a pivot point of an object or group of objects represented by said point cloud frame in a video-based point cloud compression domain; a scaling parameter for scaling an object or group of objects represented by the point cloud frame from the video-based point cloud compression domain to the world domain; rotation parameters for rotating an object or group of objects represented by the point cloud frame about a signaled pivot point to be correctly oriented in the world domain; or A translation parameter for translating an object or group of objects represented by the point cloud frame relative to the signaled pivot point to be correctly placed in the world domain.

14. A method comprising: Providing first signaling information, the first signaling information including information related to a world domain, wherein the world domain is a point cloud frame represented by a plurality of points in a first volume coordinate system, wherein the first signaling information includes information related to the world domain: - signal length unit in said world domain; - the coordinate system of the world domain; - the size of a bounding box around which an object or group of objects represented by a sequence of a plurality of point cloud frames in the world domain is placed; and Second signaling information is provided, the second signaling information comprising information related to a transformation of a model domain to the world domain, wherein the model domain represents the point cloud frame by a plurality of points in a second volume coordinate system.

15. A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine for performing operations comprising: Providing first signaling information, the first signaling information including information related to a world domain, wherein the world domain is a point cloud frame represented by a plurality of points in a first volume coordinate system, wherein the first signaling information includes information related to the world domain: - signal length unit in said world domain; - the coordinate system of the world domain; - the size of a bounding box around which an object or group of objects represented by a sequence of a plurality of point cloud frames in the world domain is placed; and Second signaling information is provided, the second signaling information comprising information related to a transformation of a model domain to the world domain, wherein the model domain represents the point cloud frame by a plurality of points in a second volume coordinate system.

Images