Point cloud encoding device, point cloud decoding device, point cloud encoding method, point cloud decoding method, and program

Abstract

To improve the encoding efficiency for attribute information in point cloud information.SOLUTION: A point cloud encoding device includes: an interpolation unit that performs interpolation processing on a reconstructed point cloud of encoded frame regarding attribute information of point cloud information to generate a reference frame with decimal precision; a motion estimation unit that generates motion information by estimating motion between the decimal precision reference frame and the integer precision frame; a prediction unit that generates a predicted value on the basis of the motion information; and an entropy encoding unit that entropy encodes the difference between the point group of the frame and the predicted value.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a point cloud encoding device, a point cloud decoding device, a point cloud encoding method, a point cloud decoding method, and a program. [Background technology]

[0002] Conventionally, a point cloud inter-prediction technique such as that shown in Non-Patent Document 1 is known. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] “Graph-based compression of dynamic 3D point cloud sequences,” IEEE Transactions on Image Processing, vol. 25, no. 4, pp. 1765--1778, 2016 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional method described above, the motion vectors have integer voxel precision, and there is a problem in that the coding efficiency cannot be sufficiently improved.

[0005] The present invention has been made in consideration of the above circumstances, and its object is to improve the coding efficiency of attribute information in point group information. [Means for solving the problem]

[0006] (1) One aspect of the present invention is a point cloud encoding device comprising: an interpolation unit that performs an interpolation process on a reconstructed point cloud of an encoded frame for attribute information in the point cloud information to generate a reference frame with decimal precision; a motion estimation unit that performs motion estimation between the reference frame with decimal precision and the frame with integer precision to generate motion information; a prediction unit that generates a predicted value based on the motion information; and an entropy encoding unit that entropy encodes the difference between the point cloud of the frame and the predicted value. (2) In one aspect of the present invention, in the above-mentioned point cloud encoding device, the interpolation unit generates an interpolated value by performing an interpolation process on attribute values ​​based on the attribute values ​​of the two points that are closest to each other for each decimal precision position. (3) In one aspect of the present invention, in the above-mentioned point cloud encoding device, when there are two or more interpolated values ​​for each decimal precision position, the interpolation unit uses the average value of those values ​​as the final interpolated value. (4) In one aspect of the present invention, in the above-mentioned point cloud coding device, the prediction unit sets the attribute value of the closest point in the integer precision frame among the points in the decimal precision reference frame after motion compensation based on the motion information as the predicted value. (5) In one aspect of the present invention, in the point cloud encoding device described above, the decimal precision is 1 / 2 voxel precision. (6) One aspect of the present invention is a point cloud decoding device comprising: an interpolation unit that performs interpolation processing on a reconstructed point cloud of a decoded frame for attribute information in the point cloud information to generate a reference frame with decimal precision; an entropy decoding unit that decodes motion information and a prediction residual from a bitstream; a prediction unit that generates a predicted value based on the motion information and the reference frame with decimal precision; and an attribute information decoding unit that sets the sum of the prediction residual and the predicted value as the decoded value of the attribute value. (7) In one aspect of the present invention, in the above-mentioned point cloud decoding device, the interpolation unit generates an interpolated value by performing an interpolation process on attribute values ​​based on the attribute values ​​of the two points that are closest to each other for each decimal precision position. (8) In one aspect of the present invention, in the point cloud decoding device described above, when there are two or more interpolated values ​​for each decimal precision position, the interpolation unit uses the average value of those values ​​as the final interpolated value. (9) In one aspect of the present invention, in the above-mentioned point cloud decoding device, the prediction unit sets the attribute value of the closest point in the integer precision frame among the points in the decimal precision reference frame after motion compensation based on the motion information as the predicted value. (10) In one aspect of the present invention, in the point cloud decoding device described above, the decimal precision is 1 / 2 voxel precision. (11) One aspect of the present invention is a point cloud encoding method, which includes: performing an interpolation process on attribute information of point cloud information for a reconstructed point cloud of an encoded frame to generate a reference frame with decimal precision; performing motion estimation between the reference frame with decimal precision and the frame with integer precision to generate motion information; generating a predicted value based on the motion information; and entropy encoding the difference between the point cloud of the frame and the predicted value. (12) One aspect of the present invention is a program for causing a computer provided in a point cloud encoding device to perform an interpolation process on attribute information in the point cloud information for a reconstructed point cloud of an encoded frame to generate a reference frame with decimal precision, perform motion estimation between the reference frame with decimal precision and the frame with integer precision to generate motion information, generate a predicted value based on the motion information, and entropy encode the difference between the point cloud of the frame and the predicted value. (13) One aspect of the present invention is a point cloud decoding method, which includes: performing an interpolation process on a reconstructed point cloud of a decoded frame for attribute information in point cloud information to generate a reference frame with decimal precision; decoding motion information and a prediction residual from a bitstream; generating a predicted value based on the motion information and the reference frame with decimal precision; and using the sum of the prediction residual and the predicted value as a decoded value of an attribute value. (14) One aspect of the present invention is point group decoding. Devicethe program causing a computer equipped with the program to perform an interpolation process on the reconstructed point group of a decoded frame for attribute information in point group information to generate a reference frame with decimal precision, decode motion information and a prediction residual from the bit stream, generate a predicted value based on the motion information and the reference frame with decimal precision, and set the sum of the prediction residual and the predicted value as the decoded value of the attribute value. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the coding efficiency of attribute information included in point group information. [Brief explanation of the drawings]

[0008] [Figure 1A] FIG. 10 is a diagram showing all candidates for integer voxel and half voxel positions in this embodiment. [Figure 1B] FIG. 2 is a diagram showing a first example of a fractional voxel obtained from adjacent integer voxels in this embodiment. [Figure 1C] FIG. 10 is a diagram showing a second example of a fractional voxel obtained from adjacent integer voxels in this embodiment. [Figure 1D] FIG. 10 is a diagram showing a third example of a fractional voxel obtained from adjacent integer voxels in this embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of a comparison between an original block and a super-resolution reference block. [Figure 3] FIG. 2 is a diagram illustrating an example of motion compensation prediction according to the present embodiment. [Figure 4A] FIG. 2 shows an example of a rate-distortion curve for a first data set (longdress). [Figure 4B] FIG. 10 shows an example of a rate-distortion curve for a second data set (soldier). [Figure 4C] FIG. 10 shows an example of a rate-distortion curve for a third data set (red and black). [Figure 4D]FIG. 10 shows an example of a rate-distortion curve for a fourth data set (loot). [Figure 5] FIG. 10 is a diagram showing the BD rate performance relative to the baseline of this embodiment. [Figure 6] FIG. 1 is a diagram illustrating a point cloud processing system according to an embodiment of the present invention. [Figure 7] FIG. 2 is a diagram illustrating an example of functional blocks of the point group encoding device according to the present embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of functional blocks of an attribute information encoding unit according to the present embodiment. [Figure 9] FIG. 2 is a diagram illustrating an example of functional blocks of a motion estimation unit. [Figure 10] FIG. 2 is a diagram illustrating an example of functional blocks of a point group decoding device according to the present embodiment. [Figure 11] FIG. 2 is a diagram illustrating an example of functional blocks of a point attribute information decoding unit according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] [overview] The present invention will be outlined below with reference to the drawings. Recent advances in 3D measurement and reconstruction technology have made capturing 3D scenes ubiquitous. In dynamic point clouds, each frame consists of a list of data points with 3D coordinates and RGB color values. Because the raw format of point clouds requires significant bandwidth, there has been considerable interest in point cloud compression techniques, and MPEG standardization efforts are underway that consider both video-based point cloud compression (VPCC) and geometry-based point cloud compression (G-PCC).

[0010] To achieve efficient compression of dynamic point clouds, several inter-frame (temporal) prediction methods have been proposed. These methods can be broadly classified into three types: Voxel-based methods estimate the motion vector (MV) for each voxel, select a few points as anchors in both the predicted and reference frames, and establish correspondences through spectral matching, leading to a sparse set of motion vectors. A smoothness constraint can then be used to obtain a dense set of motion vectors from the sparse set, providing the motion for all remaining points. In patch-based techniques, motion estimation (ME) is regarded as unsupervised 3D point registration, and motion vectors are estimated by iterative closest point (ICP) for each patch generated by K-means clustering. In this embodiment, we focus on a block-based approach, which divides the frame to be predicted into multiple non-overlapping frames. The division is into three-dimensional blocks. For each block, the best matched block in the reference frame is selected according to a specific matching criterion. This criterion can be purely geometric, e.g., an ICP-based approach that generates a rigid transformation, or it can combine geometric and color attribute information. Recent research has also focused on accelerating block-based motion search, including both efficient search pattern design and search window reduction.

[0011] The present embodiment is motivated by the observation that sub-pixel accurate motion estimation is an essential tool in modern video coding. On the other hand, all of the aforementioned motion estimation methods for dynamic point clouds are based on integer voxel displacements. There are two main reasons why it is difficult to directly extend video-based fractional motion estimation to 3D data.

[0012] First, the point cloud is irregularly distributed within each frame, and only voxels corresponding to the surfaces of objects in the scene contain attribute information. Therefore, attribute interpolation at new voxel locations can be based on traditional techniques, but there is an additional challenge of selecting only new voxel locations that coincide with object surfaces (even though those surfaces are not explicitly known). For example, creating additional, less accurate voxels inside objects should be avoided. Second, voxels are not necessarily consistent across frames, i.e., both the number of voxels and their spatial distribution vary from frame to frame. Thus, two matching blocks in consecutive frames generally have different numbers of voxels containing attribute information, necessitating the development of alternative methods to the one-to-one pixel (or sub-pixel) matching commonly used in traditional video. In this embodiment, attention is focused on fractional voxel motion estimation (FvME) on the assumption that integer voxel motion vectors (IvMV) have already been obtained by the existing integer voxel motion estimation (IvME) method. Specifically, we use pre-computed IvMVs from a public database. Our method first creates fractional voxels between pairs of adjacent occupied integer voxels. The neighboring voxels are used to maintain consistency with the object surface, eliminating the need for explicit surface estimation. A higher-resolution point cloud can then be obtained by interpolating the attributes of each fractional voxel from the values ​​of neighboring integer voxels. FvME is implemented by searching for fractional voxel motion vectors (FvMVs) around the location given by IvMV and selecting the fractional displacement that results in the lowest motion-compensated prediction error. Motion-compensated prediction is achieved by directly copying the attributes of the closest voxel in the matched block in the reference frame as the attributes of a voxel in a block in the current frame. Our FvME method outperforms transform-based approaches without inter- or intra-prediction, and significantly outperforms temporal prediction methods based on IvMVs from a database.

[0013] (Sub-voxel motion estimation and correction) Real-world scenes and objects are captured from various viewing angles by multiple calibrated and synchronized RGB or RGB-D camera clusters. After stitching and voxelization, a dynamic point cloud is generated on an integer grid. While 3D voxel coordinates are obtained as integer approximations to the true position of objects in 3D space, it is important to note that the optimal displacement between frames is unlikely to be exactly integer. Therefore, fractional values ​​of voxel coordinate displacements may be better than integer values, and high-resolution motion vectors can potentially provide more accurate motion and, therefore, more accurate motion compensation (MC) predictions. Furthermore, distortions due to lossy coding in previously reconstructed point cloud frames can increase prediction errors, and camera noise, lighting changes in the capture environment, and object movement can also add noise to color attributes, leading to imperfect matching during motion estimation. Thus, similar to traditional video, where fractional motion compensation is well known to contribute to noise reduction, the process of generating higher-resolution point clouds and attributes at fractional voxel locations can contribute to noise reduction and potentially improve the quality of reference frames.

[0014] (occupied fractional voxels) Here, we will explain the definition of a fractional voxel and the interpolation method of this embodiment. Based on the same design concept as images and videos, fractional voxels are created at selected intermediate positions between voxels on an integer resolution grid. Here, a voxel located midway between two adjacent integer voxels is defined as a fractional voxel with half the resolution (1 / 2-voxel). As described above, not all of the three-dimensional space is occupied by integer voxels. The areas occupied by integer voxels usually correspond to the surface of an object. Therefore, in this embodiment, new fractional voxels are created only at positions in the three-dimensional space that (approximately) coincide with the surface that represents the positions of the occupied integer voxels, and attributes are interpolated only in these newly created fractional voxels. Two integer voxels with coordinates vj and vk are said to be adjacent if their distance is less than or equal to a threshold ρ. Then, fractional voxels are created only between adjacent vj and vk (assuming they are close enough that they likely belong to the same surface), and the corresponding interpolated color attributes are calculated as follows:

[0015]

number

[0016] where vi is a voxel in the fractional voxel set Vf with color signal C(vi), and vj and vk are voxels in the integer voxel set Vi with color signals C(vj) and C(vk), respectively. L(-) represents the voxel coordinate, ρ is a distance threshold, and dist(vj,vk) measures the Euclidean distance between the coordinates of vj and vk. Note that different pairs of integer voxels may generate the same fractional voxel. Therefore, to eliminate repeated fractional voxels after interpolation, attributes belonging to the same fractional voxel but interpolated from different pairs of neighboring voxels are merged by averaging. Figure 1b shows some examples of possible fractional voxel locations, and it can be seen that interpolation based on neighboring integer voxels tends to favor higher voxel resolution on the (implied) surface on which the voxel is located.

[0017] FIG. 1 is a diagram showing an example of integer voxels and fractional voxels according to this embodiment. FIG. 1A is a diagram showing all possible integer and half voxel positions in this embodiment. FIG. 1B is a diagram showing a first example of fractional voxels obtained from neighboring integer voxels in this embodiment. FIG. 1C is a diagram showing a second example of fractional voxels obtained from neighboring integer voxels in this embodiment. FIG. 1D illustrates a third fractional voxel derived from neighboring integer voxels in this embodiment. FIG. Note that the interpolated fractional voxels shown in Figures 1A-1D are likely to belong to the same surface as the neighboring integer voxels from which they were derived.

[0018] (Motion estimation with fractional voxel accuracy) Due to the discrepancy in voxel distribution in consecutive frames, it is difficult to establish an exact one-to-one correspondence between voxels in two matching blocks. To generalize motion-compensated (MC) prediction to fractional motion in three-dimensional space, we start by super-resolving a reference frame as described above.

[0019] Figure 2 shows an example of a comparison between an original block and a super-resolution reference block. As can be seen, the continuity between voxels and their corresponding attributes significantly increases the base plane, which provides a better predictor when high-resolution motion is available. The low-pass filtering used in the interpolation also contributes to removing attribute noise.

[0020] Next, motion vectors for motion compensation (MC) are estimated with fractional precision. The entire motion estimation process, including IvME and FvME, is a coarse-to-fine procedure. Each estimated motion vector is obtained as the sum of the displacements of IvMV and FvMV. Given an IvMV MVi, the optimal FvMV MVoptf is searched from a set of candidate fractional displacements. To super-resolve the reference frame with half-voxel precision (i.e., fractional precision), each coordinate of the fractional displacement MVf takes the value {-1 / 2, 0, 1 / 2}, resulting in 27 possible displacements. 3 is a diagram showing an example of motion compensation prediction according to this embodiment. As shown in the figure, for a given fractional displacement MVf, each attribute of the current block is predicted from the nearest voxel of the super-resolution transformed reference block. Then, the displacement with the smallest prediction error is selected.

[0021]

number

[0022] where Bsr and BsrMC are the super-resolution reference blocks before and after translation by the motion vector, respectively, and vi and vj are the voxels with color signals C(vi) and C(vj) of blocks Bp and BsrMC, respectively. Epred(-,-) is a function for measuring the prediction error. Lb(-) represents the coordinates of the block, and V(-) represents the set of voxels within the block.

[0023] (Motion compensated (MC) prediction with fractional voxel accuracy) The obtained fractional motion vector is used to perform motion-compensated (MC) prediction. Specifically, as shown in Equation (2), the voxels of the reference block are translated by the integer motion vector Mvi, and then further translated by the obtained optimal fractional displacement MVoptf. Next, a temporal correspondence between the voxels of the prediction block Bp and the nearest voxels of the super-resolved reference block BsrMC is established, and motion-compensated prediction is performed. The attributes of each voxel in the prediction block are predicted by copying the attributes of the corresponding voxel in the reference frame.

[0024]

number

[0025] (Example) Here, we evaluate the color attribute compression by the FvME method of this embodiment from a dataset (four sequences: longdress, redandblack, loot, and soldier). Each sequence contains 300 frames. Here, we assume that IvMV is given and used to estimate FvMV.

[0026] We use a publicly available 3D motion vector database. IvMV is chosen to minimize a hybrid distance metric: δ = δg + 0.35δc, which combines δg, the average Euclidean distance between voxels, and δc, the average color distance between voxels in the Y channel. We consider motion only for blocks of 16x16x16 size, and implement a conventional inter-coding method that uses previously decoded frames as references.

[0027] In accordance with the MPEG Call for Proposal (CfP) for Point Cloud Compression, the block-based FvME method (ProposedFvME) of this embodiment was evaluated on a group of 32 frames, with the first frame configured in intra mode and the remaining frames configured in inter prediction. The threshold distance between integer voxels for interpolating fractional voxels is set as ρ = √3 in equation (1) above. Color is converted from RGB to YUV space, and the Y, U, and V channels are processed independently. When searching for the optimal FvMV candidate in equation (2), squared distance is used as a measure of prediction error. In intra-coded frames, all blocks undergo a region-adaptive graph Fourier transform (RA-GFT), and in inter-coded frames, all blocks undergo motion compensation. After motion-compensated (MC) prediction, the residual is transformed using the graph Fourier transform (GFT). To compute the GFT, a threshold graph is constructed within each block, connecting voxels whose Euclidean distance is less than or equal to √3. If the graph for a block is not connected after thresholding, a complete graph is constructed instead, so that the transform coefficients of each block consist of a single approximation coefficient (DC) and multiple detail coefficients (AC). The DC coefficients of all blocks are concatenated and coded. Then, the AC coefficients are coded block by block. This method is equivalent to single-level RA-GFT. All transforms are uniformly quantized, and the coefficients are entropy coded using the adaptive run-length Golomb-Rice algorithm (RLGR). As for the FvMV overhead, since there are a total of 27 FvMVs, 8 bits are used for each FvMV signal. For IvMV, 4 bits are used to indicate the value of each axis and 1 bit is used to indicate the sign, for a total of 15 bits to represent IvMV. The FvMV and IvMV overhead are entropy coded using the Lempel-Ziv-Markov chain algorithm.

[0028] As a baseline, we considered the following method: IvME, which uses database motion (DM) for motion-compensated (MC) prediction and adds integer local refinement (DM+RF) to motion-compensated (MC) prediction. Local refinement uses a different criterion aimed at selecting matching blocks with similar geometry (δg) and color attributes (δc). There is no guarantee that this metric, especially the relative weighting between distances (0.35), is optimal in terms of coding efficiency. Therefore, motion-compensated experiments show that these IvMVs may underperform coding methods without motion compensation. In such cases, performance can be improved by locally refining IvMVs from the database and minimizing only the color error, rather than the hybrid error used in the database. The database motion is refined by an additional local search with integer precision to improve the matching accuracy over the original one. The local search range for each axis is set to [-1, 1], which completely encompasses the fractional positions searched by the FvME method of this embodiment.

[0029] To evaluate the benefits of high-resolution reference and FvMV, we propose two inter-encoding schemes using super-resolution reference blocks. These schemes use fractional motion vectors for compensation prediction and do not. First, to evaluate super-resolution methods, we implement a scheme that uses integer local fine DM and super-resolution reference blocks and considers IvME for prediction, denoted as "DM+RF+SR." The difference between DM+RF and DM+RF+SR is the resolution of the reference block. Therefore, to evaluate the benefits of FvMV, we implement a scheme that uses fractional resolution for both the reference block and motion vectors, denoted as "FvME of this embodiment." In order to make a fair comparison between the encoding methods, all other test conditions were the same.

[0030] Furthermore, to make the performance evaluation more complete, we decided to include two state-of-the-art (all-intra) anchor solutions: RA-GFT and Region-Adaptive Haar Transform (RAHT). In RA-GFT, a block size of 16 is used. The residual is entropy coded by RLGR.

[0031] (Evaluation indicators) The evaluation metrics are the number of bits per voxel (bpv) and the average peak signal-to-noise ratio of the Y component (PSNR-Y).

[0032]

number

[0033] where Yt and Y^t represent the original and reconstructed signals on the same voxel in the t-th frame, respectively, T is the total number of frames, bt is the number of bits required to encode the t-th YUV component (including IvMV and FvMV overhead if necessary), and Nt is the total number of occupied voxels in the t-th frame. We also list the Bjontegaard-Delta (BD-rate) results for bitrate.

[0034] (Experimental results and analysis) FIG. 4 is a diagram illustrating an example of a rate-distortion curve. FIG. 4A is a diagram showing an example of a rate-distortion curve for a first data set (longdress). FIG. 4B shows an example of a rate-distortion curve for the second data set (soldier). FIG. 4C shows an example of a rate-distortion curve for the third data set (red and black). FIG. 4D shows an example of a rate-distortion curve for the fourth data set (loot). We note that using only the original IvME from the database results in suboptimal performance compared to RAHT and RA-GFT. This is due to the criteria used in the database to select the optimal motion vector based on shape and color information. After local refinement at integer precision, the performance of IvME(DM+RF) is significantly improved over IvME(DM).

[0035] FIG. 5 is a diagram showing the BD rate performance relative to the baseline of this embodiment.

[0036] After super-resolution of the reference blocks, the performance of the proposed IvME (DM+RF+SR) is further improved over DM+RF without increasing the motion vector resolution. The DM+RF+SR scheme has the advantage of lower complexity than our FvME and can sometimes achieve better results than intra-coding schemes. Finally, increasing the motion vector resolution to 1 / 2 voxel results in further coding gains, outperforming the intra-coding baselines RA-GFT and RAHT by an average of 2.8 dB and 4.6 dB, respectively. This method consistently outperforms DM+RF+SR, but at the cost of higher complexity due to the additional motion search.The results show that both interpolated fractional voxels and high-resolution motion vectors lead to high coding gains, outperforming both IvME and non-predictive transform-based schemes.

[0037] This figure summarizes the performance of the proposed method for IvME(DM+RF), RA-GFT, and RAGT in terms of BD-rate. FvME of this embodiment can achieve an average bitrate reduction of 57% compared to IvME(DM+RF). Compared with the prior art, the proposed method can achieve an average bitrate reduction of 61% and 43% for RAHT and RA-GFT, respectively.

[0038] [Embodiment] Specific examples of embodiments of the present invention will be described below with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, and various variations, including combinations with other existing components, are possible. Therefore, the following description of the embodiments does not limit the scope of the invention as defined in the claims.

[0039] (First embodiment) Hereinafter, a point cloud processing system 10 according to a first embodiment of the present invention will be described with reference to FIGS. 6 is a diagram showing a point cloud processing system 10 according to the embodiment. As shown in FIG. 6, the point cloud processing system 10 includes a point cloud encoding device 100 and a point cloud decoding device 200.

[0040] The point cloud encoding device 100 is configured to generate encoded data (bitstream) by encoding an input point cloud signal, and the point cloud decoding device 200 is configured to generate an output point cloud signal by decoding the bitstream.

[0041] The input point cloud signal and the output point cloud signal are composed of position information and attribute information of each point in the point cloud, such as color information and reflectance of each point. Here, the bit stream may be transmitted from the point group encoding device 100 to the point group decoding device 200 via a transmission path. Alternatively, the bit stream may be stored in a storage medium and then provided from the point group encoding device 100 to the point group decoding device 200.

[0042] (Point cloud encoding device 100) Hereinafter, the point group encoding device 100 according to this embodiment will be described with reference to FIG. 7 is a diagram showing an example of functional blocks of the point cloud encoding device 100 according to this embodiment. As shown in FIG. 7, the point cloud encoding device 100 includes a geometric information encoding unit 101, an attribute information encoding unit 102, a local decoding unit 103, a frame buffer 104, and a bitstream integration unit 105.

[0043] The geometric information encoding unit 101 receives a point group to be encoded, performs a geometric information encoding process, and outputs a bit stream of the geometric information to the bit stream integration unit 105. Here, the geometric information encoding method can be realized by a known method such as G-PCC, so details are omitted. The unit also performs a local decoding process of the geometric information to generate a reconstructed point group based on the geometric information obtained on the decoding side. At this time, if the geometric information between the point group to be encoded and the reconstructed point group does not completely match, attribute information (hereinafter, RGB color information will be used as an example) of each point in the reconstructed point group is generated based on the color information of the point group to be encoded. In other words, the attribute information is information indicating the attribute (e.g., color) of each point in the point group. For example, for each point in the reconstructed point group, the point in the point group to be encoded that has the smallest Euclidean distance may be identified, and the color information of that point may be used as is. Alternatively, K nearest points in the point group to be encoded may be identified, and the color information may be calculated by interpolation from the nearest points. The reconstructed point group generated in this way is output to the attribute information encoding unit 102.

[0044] The attribute information encoding unit 102 receives as input the reconstructed point cloud of the frame output from the geometric information encoding unit 101 and the reconstructed point cloud of the encoded frame stored in the frame buffer 104, encodes the attribute information of the frame, and outputs a bit stream of the attribute information to the bit stream integration unit 105. It also outputs data necessary for local decoding to the local decoding unit 103. The data necessary for local decoding is, for example, the geometric information of the reconstructed point cloud output from the geometric information encoding unit, and the bit stream generated by the attribute information encoding unit 102, or a set of values ​​of encoded symbols (syntax elements) immediately before being converted into a bit stream by entropy encoding. Details of the processing by the attribute information encoding unit 102 will be described later.

[0045] The local decoding unit 103 receives as input the data required for local decoding output from the attribute information encoding unit 102, locally decodes the frame, generates a reconstructed point cloud for the frame, and outputs it to the frame buffer 104. The reconstructed point cloud for the frame obtained here is completely identical to the reconstructed point cloud obtained by the point cloud decoding device 200.

[0046] The frame buffer 104 stores the reconstructed point group of the frame output from the local decoding unit 103, and outputs the stored reconstructed point group to the attribute information encoding unit 102 when encoding a frame that will be encoded after the frame in question.

[0047] (Attribute information encoding unit 102) The attribute information encoding unit 102 according to this embodiment will be described below with reference to FIGS. 8 is a diagram showing an example of functional blocks of the attribute information encoding unit 102 according to this embodiment. As shown in FIG. 8, the attribute information encoding unit 102 includes a motion estimation unit 1021, a prediction unit 1022, an interpolation unit 1023, and an entropy encoding unit 1024.

[0048] The motion estimation unit 1021 receives as input the reconstructed point group of the frame output from the geometric information encoding unit 101 and the reconstructed point group of the reference frame at decimal precision positions interpolated by the interpolation unit 1023, performs motion compensation, and outputs motion information. Here, the reference frame is an already encoded frame stored in the frame buffer 104. The motion information is, for example, a motion vector calculated for each region of a predetermined size.

[0049] An example of the processing of the motion estimation unit 1021 will be described below with reference to FIG. FIG. 9 is a diagram showing an example of functional blocks of the motion estimation unit 1021.

[0050] The integer precision motion estimation unit 10211 receives as input the reconstructed point cloud of the current frame output from the geometric information encoding unit 101 and the reconstructed point cloud of the reference frame at decimal precision positions interpolated by the interpolation unit 1023, performs motion estimation with integer precision, and calculates motion information. This motion information is, for example, a motion vector with integer precision calculated for each region of a predetermined size (for example, 16×16×16). The specific motion compensation method can be realized by a known method, and details thereof will be omitted. In the following description, the decimal precision position value calculated by the interpolation operation is also referred to as an interpolated value.

[0051] The decimal precision motion estimator 10212 receives as input the reconstructed point cloud of the current frame output from the geometric information encoder 101, the reconstructed point cloud of the reference frame at decimal precision positions interpolated by the interpolation unit 1023, and the integer precision motion vector derived by the integer precision motion estimator 10211, and performs decimal precision motion compensation to generate decimal precision motion information. This decimal precision motion information may be, for example, a correction vector with decimal precision based on the integer precision motion vector. Specifically, for example, a half precision correction vector may be one in which the x, y, and z coordinate values ​​are limited to only take on one of the values ​​{½, 0, −½}. An example of a method of decimal precision motion compensation will be described with reference to FIG. 3 mentioned above. In Figure 3, Bp represents the area to be coded in the frame (for example, an area of ​​16 x 16 x 16 size), Brs represents the reference area identified by the motion vector (= integer precision motion vector + decimal precision correction vector) among the group of reconstructed points of the reference frame at the decimal precision position interpolated by the interpolation unit 1023, and BrMCs represents the motion-compensated reference area.

[0052] (Step S1) First, Brs is motion compensated with a temporary motion vector to generate BrMCs. The temporary motion vector is obtained by adding a temporary decimal precision correction vector (for example, with all x, y, and z coordinates halved) to the integer precision motion vector derived by the integer precision motion estimation unit 10211. (Step S2) Next, for each point in Bp, the point in BrMCs to which the distance is smallest is identified. (Step S3) The attribute values ​​of the points in BrMCs identified in step S3 are used as predicted values ​​of the attribute values ​​of the points in Bp. (Step S4) The difference (prediction residual) between the attribute value of the point in Bp and the predicted value is calculated. (Step S5) A cost value is calculated by integrating the prediction residuals at all points within Bp. For example, the cost value may be the sum of absolute errors of the residuals. As another example, the cost value may be the sum of squared errors of the residuals. (Step S6) The above steps (Step S1) to (Step S5) are executed for all possible combinations of coordinate values ​​for the provisional decimal precision correction vector (when the x, y, and z coordinate values ​​each take only one of the values ​​{1 / 2, 0, -1 / 2} (3 x 3 = 27 combinations)), and the decimal precision correction vector that results in the smallest cost value is adopted.

[0053] The prediction unit 1022 receives as input the motion information generated by the motion estimation unit 1021, the reconstructed point group of the frame in question output from the geometric information encoding unit 101, and the reconstructed point group of the reference frame at decimal precision positions interpolated by the interpolation unit 1023, and generates a predicted value of the attribute information of each point in the reconstructed point group of the frame in question. The predicted value can be generated, for example, based on the motion information determined by the motion estimation unit 1021, using the same procedures as those in (Step S1) to (Step S3) above.

[0054] The interpolation unit 1023 receives as input a reconstructed point cloud of an already encoded frame stored in the frame buffer 104, interpolates points at decimal-precision positions (i.e., performs so-called super-resolution processing), and generates a reconstructed point cloud of a reference frame at decimal-precision positions. A specific example of the interpolation method will be described with reference to FIG. 1A. In FIG. 1A, the open points are points at integer-precision positions (where all coordinate values ​​are integers), and the shaded points are points at decimal-precision positions. Here, an example will be described in which a point at a half-precision position (where at least one of the x, y, and z coordinates is an integer + 0.5) is interpolated from a point at an integer-precision position. As shown in FIGS. 1B to 1D, two integer-precision points closest to each half-precision point are identified, and if those two points exist, the attribute information of the point at the half-precision position is interpolated by calculating the average of the attribute values ​​of the two points. Also, if there are three or more points at the nearest integer position, that is, if there are multiple combinations of "two points at the nearest integer position," the average value of the attribute information for each combination is first calculated, and then the average value of those points is further taken to obtain the final attribute value at the 1 / 2 precision position.

[0055] The entropy coding unit 1024 receives as input the difference (prediction residual) between the reconstructed point group of the frame output from the geometric information coding unit 101 and the predicted value calculated by the prediction unit 1022, and the motion information generated by the motion estimation unit 1021 (e.g., a motion vector with integer precision and a correction vector with decimal precision), and performs entropy coding to generate a bit stream of attribute information.

[0056] (Point Cloud Decoding Device 200) Hereinafter, the point group decoding device 200 according to this embodiment will be described with reference to FIG. 10 is a diagram showing an example of functional blocks of a point cloud decoding device 200 according to this embodiment. As shown in FIG. 10, the point cloud decoding device 200 includes a bitstream dividing unit 201, a geometric information decoding unit 202, an attribute information decoding unit 203, and a frame buffer 204.

[0057] The bit stream dividing unit 201 receives the bit stream output from the point cloud encoding device 100 and divides the bit stream into a bit stream of geometric information and a bit stream of attribute information. The bit stream of geometric information is output to a geometric information decoding unit 202, and the bit stream of attribute information is output to an attribute information decoding unit 203.

[0058] The geometric information decoding unit 202 receives the bit stream of geometric information output from the bit stream dividing unit 201, decodes the geometric information, and generates a reconstructed point cloud. At this point, each point in the reconstructed point cloud contains only geometric information. The specific decoding method can be realized by a known method such as G-PCC, so details will be omitted. The reconstructed point cloud containing only geometric information is output to the attribute information decoding unit 203. In the following description, the value obtained by the decoding operation is also referred to as a decoded value.

[0059] The attribute information decoding unit 203 receives as input the bit stream of attribute information output from the bit stream splitting unit 201, the reconstructed point group consisting of only geometric information output from the geometric information decoding unit 202, and the reconstructed point group of the decoded frame stored in the frame buffer 204, decodes the attribute information (e.g., color information) of each point in the reconstructed point group from the bit stream of attribute information, and outputs the reconstructed point group of the frame.

[0060] The frame buffer 204 stores the reconstructed point group of the frame output from the attribute information decoding unit 203, and outputs the stored reconstructed point group to the attribute information decoding unit 203 when decoding a frame that will be decoded after the frame in question.

[0061] (Attribute information decoding unit 203) The attribute information decoding unit 203 according to this embodiment will be described below with reference to FIG. 11 is a diagram showing an example of functional blocks of the attribute information decoding unit 203 according to this embodiment. As shown in FIG. 11, the attribute information decoding unit 203 includes an entropy decoding unit 2031, a prediction unit 2032, and an interpolation unit 2033.

[0062] The entropy decoding unit receives the bitstream of attribute information generated by the bitstream dividing unit 201 as input, and decodes the prediction residual and motion information. The prediction unit 2032 receives the reconstructed point group consisting of only geometric information generated by the geometric information decoding unit 202 and the motion information decoded by the entropy decoding unit, and derives a predicted value of the attribute value of each point in the reconstructed point group consisting of only geometric information. The predicted value can be derived using, for example, the same method as that used by the prediction unit 1022.

[0063] The interpolation unit 2033 receives the reconstructed point cloud of an already encoded frame stored in the frame buffer 204, interpolates points at decimal precision positions (in other words, performs so-called super-resolution processing), and generates the reconstructed point cloud of the reference frame at decimal precision positions. Specific processing can be realized by processing similar to that of the interpolation unit 1023.

[0064] The predicted value generated by the prediction unit 2032 and the prediction residual decoded by the entropy decoding unit 2031 are added together to generate a group of reconstructed points for the frame.

[0065] The above-described point group encoding device 100 and point group decoding device 200 may be realized as a program that causes a computer to execute each function (each step). In each of the above embodiments, the present invention has been described using the application of the point cloud encoding device 100 and the point cloud decoding device 200 as an example, but the present invention is not limited to such an example and can be similarly applied to a point cloud encoding / decoding system having the functions of the point cloud encoding device 100 and the point cloud decoding device 200.

[0066] According to the point cloud encoding device 100 or the point cloud decoding device 200 of this embodiment, motion compensation is performed with decimal voxel precision, thereby improving prediction performance and encoding efficiency.

[0067] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like are also included within the scope that does not deviate from the gist of the present invention.

[0068] In addition, a computer program for realizing the functions of each of the above-described devices may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read and executed by a computer system. Note that the "computer system" here may also include hardware such as an OS and peripheral devices. In addition, "computer-readable recording medium" refers to writable non-volatile memory such as a flexible disk, optical magnetic disk, ROM, or flash memory, portable media such as a DVD (Digital Versatile Disc), or a storage device such as a hard disk built into a computer system.

[0069] Furthermore, the term "computer-readable recording medium" also includes those that retain a program for a certain period of time, such as volatile memory (e.g., DRAM (Dynamic Random Access Memory)) within a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may also be for realizing part of the above-mentioned functions. Furthermore, the above-mentioned functions may be realized in combination with a program already recorded in the computer system, that is, a so-called differential file (differential program). [Explanation of symbols]

[0070] 10...point cloud processing system, 100...point cloud encoding device, 101...geometric information encoding unit, 102...attribute information encoding unit, 103...local decoding unit, 104...frame buffer, 105...bitstream integration unit, 200...point cloud decoding device, 201...bitstream division unit, 202...geometric information decoding unit, 203...attribute information decoding unit, 204...frame buffer, 1021...motion estimation unit, 1022...prediction unit, 1023...interpolation unit, 1024...entropy encoding unit, 2031...entropy decoding unit, 2032...prediction unit, 2033...interpolation unit

Claims

1. A point cloud encoding device, an interpolation unit that performs an interpolation process on the reconstructed point cloud of the encoded frame with respect to attribute information indicating attributes of the point cloud to generate a reference frame with decimal precision; a motion estimation unit that performs motion estimation between the reference frame with decimal precision and the frame with integer precision to generate motion information; a prediction unit that generates a prediction value based on the motion information; an entropy coding unit that entropy-codes the difference between the point group of the frame and the predicted value; A point cloud encoding device comprising:

2. the interpolation unit performs an interpolation process on attribute values ​​based on attribute values ​​of two points that are closest to each other for each decimal precision position to generate an interpolated value; The point cloud encoding device according to claim 1 .

3. When two or more interpolated values ​​exist for each decimal precision position, the interpolation unit determines the average value of the interpolated values ​​as the final interpolated value. The point cloud encoding device according to claim 2 .

4. the prediction unit sets, as a predicted value, an attribute value of a point that is closest to each point in the frame with integer precision among points in the reference frame with decimal precision after motion compensation based on the motion information; The point cloud encoding device according to claim 1 .

5. The point cloud encoding device according to claim 1 , wherein the decimal precision is half voxel precision.

6. A point cloud decoding device, comprising: an interpolation unit that performs an interpolation process on the reconstructed point cloud of the decoded frame with respect to attribute information of the point cloud information to generate a reference frame with decimal precision; an entropy decoding unit that decodes motion information and prediction residuals from the bitstream; a prediction unit that generates a predicted value based on the motion information and the decimal precision reference frame; an attribute information decoding unit that sets the sum of the prediction residual and the predicted value as a decoded value of the attribute value; A point cloud decoding device comprising:

7. the interpolation unit performs an interpolation process on attribute values ​​based on attribute values ​​of two points that are closest to each other for each decimal precision position to generate an interpolated value; The point cloud decoding device according to claim 6 .

8. When two or more interpolated values ​​exist for each decimal precision position, the interpolation unit determines the average value of the interpolated values ​​as the final interpolated value. The point cloud decoding device according to claim 7 .

9. the prediction unit sets, as a predicted value, an attribute value of a point that is closest to each point in the frame with integer precision among points in the reference frame with decimal precision after motion compensation based on the motion information; The point cloud decoding device according to claim 6 .

10. The point cloud decoding device according to claim 6 , wherein the decimal precision is half voxel precision.

11. A point cloud encoding method, comprising: performing an interpolation process on the reconstructed point cloud of the encoded frame for attribute information of the point cloud information to generate a reference frame with decimal precision; generating motion information by performing motion estimation between the reference frame with decimal precision and the frame with integer precision; generating a prediction value based on the motion information; entropy coding the difference between the point cloud of the frame and the predicted value; A point cloud encoding method comprising:

12. The computer included in the point cloud encoding device performing an interpolation process on the reconstructed point cloud of the encoded frame for attribute information of the point cloud information to generate a reference frame with decimal precision; generating motion information by performing motion estimation between the reference frame with decimal precision and the frame with integer precision; generating a prediction value based on the motion information; entropy coding the difference between the point cloud of the frame and the predicted value; A program to execute.

13. A point cloud decoding method, comprising: performing an interpolation process on the reconstructed point cloud of the decoded frame for attribute information of the point cloud information to generate a reference frame with decimal precision; decoding motion information and prediction residuals from the bitstream; generating a prediction value based on the motion information and the decimal precision reference frame; a sum of the prediction residual and the predicted value as a decoded value of the attribute value; A point cloud decoding method having:

14. A computer included in the point cloud decoding device includes: performing an interpolation process on the reconstructed point cloud of the decoded frame for attribute information of the point cloud information to generate a reference frame with decimal precision; decoding motion information and prediction residuals from the bitstream; generating a prediction value based on the motion information and the decimal precision reference frame; a sum of the prediction residual and the predicted value as a decoded value of the attribute value; A program to execute.

Images