A learning-based point cloud geometry and color joint compression method

Abstract

This invention discloses a learning-based method for joint geometric and color compression of point clouds. The method specifically includes: preprocessing the initial training point cloud input, inputting it into an encoding module to obtain a feature matrix and a mask matrix, inputting the feature matrix and mask matrix into a decoding module to obtain an initial reconstructed training point cloud, calculating the geometric reconstruction training loss, color reconstruction training loss, and information entropy loss to obtain the point cloud training compression loss result, and then passing the initial inference point cloud through an optimized target learnable conditional entropy model to obtain the learning-based joint geometric and color compression result. This invention fully considers the mutual influence between geometric and color reconstruction losses by weighted summation of the geometric reconstruction loss, color reconstruction loss, and information entropy loss; the proposed learnable conditional entropy model enables more flexible conditional distribution modeling and can be widely applied in the field of point cloud processing technology.

Description

Technical Field

[0001] This invention relates to the field of point cloud processing technology, and in particular to a learning-based method for joint compression of point cloud geometry and color. Background Technology

[0002] Point clouds are collections of points with three-dimensional geometric positions and other attributes (such as color, reflectivity, etc.), which can be used to flexibly represent stereoscopic visual data such as 3D scenes and objects. Efficient point cloud compression methods can effectively promote the development of emerging fields such as virtual reality and autonomous driving.

[0003] Currently, representative traditional point cloud compression methods internationally include geometry-based point cloud compression (G-PCC) and video-based point cloud compression (V-PCC). G-PCC models and encodes point cloud information using appropriate data structures (such as octrees and triangular meshes) based on the coordinate information of the point cloud. V-PCC maps the 3D point cloud to a 2D plane and then encodes the point cloud using mature video coding methods. In recent years, with the rapid development of computer vision, some deep learning-based point cloud compression methods have emerged. These methods are based on 3D vision network architectures such as 3D convolution and multilayer perceptrons, combined with distribution estimation methods such as entropy models, to construct end-to-end trainable point cloud compression models. In terms of geometric information encoding of point clouds, some deep learning-based end-to-end compression models have achieved better rate-distortion performance than traditional methods. However, research on color information encoding of point clouds is still in a relatively early stage. Only in the case of lossless color encoding, some studies have achieved performance exceeding traditional point cloud color encoding by constructing models with strong context prediction networks. Taking into account the existing methods, the following areas for improvement exist:

[0004] 1. Mainstream point cloud encoding schemes process geometric and color information of point clouds separately, typically encoding geometric information first and then encoding color information based on the reconstructed geometric information. Therefore, when encoding geometric information, the model cannot fully consider its impact on color encoding, which is detrimental to the model's ability to balance geometric and color encoding performance during compression.

[0005] 2. Existing learning-based point cloud compression methods are limited to encoding only geometry or only color, and no scheme has yet emerged that simultaneously employs lossy encoding of both geometry and color. However, joint lossy encoding of geometry and color is of great significance for both theoretical research and practical applications.

[0006] 3. Existing learning-based point cloud compression methods typically use assumed conditional prior distributions to model latent variables, such as Gaussian, Gaussian mixture, and ogistic distributions. These distributions offer relatively limited flexibility in modeling. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a flexible and computationally inefficient learning-based method for joint compression of point cloud geometry and color.

[0008] On one hand, embodiments of the present invention provide a learning-based point cloud geometry-color joint compression method, including:

[0009] The initial training point cloud is preprocessed to obtain the first training point cloud.

[0010] The first training point cloud is input into the encoding module to perform the first operation, thereby obtaining the first training feature matrix and the first training mask matrix;

[0011] The first training feature matrix and the first training mask matrix are input into the decoding module to obtain the initial reconstructed training point cloud.

[0012] The information entropy loss is obtained by estimating the distribution of the first training feature matrix and the first training mask matrix using an initial learnable conditional entropy model.

[0013] Based on the coordinates of the initial training point cloud and the coordinates of the initial reconstructed training point cloud, the geometric reconstruction training loss is obtained through a first calculation.

[0014] Based on the geometric information of the initial training point cloud and the initial reconstructed training point cloud, the color reconstruction training loss is obtained through a second calculation.

[0015] The point cloud training compression loss is obtained by weighted summation of the geometric reconstruction training loss, the color reconstruction training loss, and the information entropy loss.

[0016] The initial learnable conditional entropy model is optimized based on the point cloud training compression loss results to obtain the target learnable conditional entropy model.

[0017] The initial inference point cloud is inferred based on the target learnable conditional entropy model to obtain the learning-based point cloud geometry and color joint compression result.

[0018] Optionally, the step of inputting the first training feature matrix and the first training mask matrix into the decoding module to obtain the initial reconstructed training point cloud includes:

[0019] The distribution estimate of the first-scale training feature matrix is ​​obtained by using global prior estimation of the first training feature matrix;

[0020] Multiple point cloud upsampling operations are performed on the first-scale training feature matrix to obtain the distribution estimates of the multi-scale training feature matrix and the multi-scale training mask matrix.

[0021] Based on the distribution estimation of the multi-scale training feature matrix and the distribution estimation of the multi-scale training mask matrix, the initial reconstructed training point cloud is obtained.

[0022] Optionally, in the step of estimating the distribution of the first training feature matrix and the first training mask matrix using an initial learnable conditional entropy model to obtain the information entropy loss, the formula for the initial learnable conditional entropy model is:

[0023]

[0024]

[0025]

[0026] Where y is the training feature matrix or training mask matrix, z is the training feature matrix reconstructed by the decoding module, and p y (y|z) is the conditional distribution of y given z. For unit uniform noise, Q represents the addition of uniform unit noise, f K f1 is the depth decomposition entropy model, θ i h represents the distribution parameters of the deep decomposition entropy model. p This is a parameter mapping model that contains learnable parameters. The weight parameters are for the deep decomposition entropy model. These are the bias parameters for the deep decomposition entropy model. L is the amplitude adjustment parameter for the deep decomposition entropy model. 信息熵 For the information entropy loss, y i Let i be the i-th training feature matrix or the i-th training mask matrix.

[0027] Optionally, in the step of obtaining the geometric reconstruction training loss through a first calculation based on the coordinates of the initial training point cloud and the coordinates of the initial reconstructed training point cloud, the calculation formula for the first calculation is:

[0028]

[0029] Among them, L 几何 The training loss is defined as the training geometry reconstruction loss, where N is the total number of points in the initial reconstructed training point cloud, and y is the training loss. i p(y) represents the occupancy status of the i-th point in the initial reconstructed training point cloud; i ) is the y estimated by the decoding module i The probability of being occupied.

[0030] Optionally, the step of obtaining the color reconstruction training loss through a second calculation based on the geometric information of the initial training point cloud and the initial reconstructed training point cloud includes:

[0031] Based on the color values ​​of the initial training point cloud and the initial reconstructed training point cloud, the initial reconstructed training point cloud is recolored to obtain the target reconstructed training point cloud;

[0032] The color reconstruction training loss is obtained from the color values ​​of the target reconstructed training point cloud and the initial reconstructed training point cloud.

[0033] Optionally, optimizing the initial learnable conditional entropy model based on the point cloud training compression loss result to obtain the target learnable conditional entropy model includes:

[0034] Based on the point cloud training compression loss results, backpropagation is performed on the initial learnable conditional entropy model;

[0035] The conditional distribution in the initial learnable conditional entropy model after backpropagation is pre-calculated, and the calculation result is used as the model parameters of the target learnable conditional entropy model.

[0036] The parameter mapping model and the deep decomposition entropy model are removed from the initial learnable conditional entropy model to obtain the target learnable conditional entropy model.

[0037] Optionally, the step of reasoning the initial inference point cloud based on the target learnable conditional entropy model to obtain a learning-based point cloud geometry-color joint compression result includes:

[0038] The initial inference point cloud is subjected to a second preprocessing step to obtain sub-point cloud blocks;

[0039] The sub-point cloud blocks are subtracted from the mean and normalized to obtain the RGB mean value, and the RGB mean value is written into the sub-bitstream;

[0040] The sub-point cloud blocks are input into the encoding module to obtain the sub-feature matrix and the sub-mask matrix;

[0041] The sub-feature matrix and the sub-mask matrix are input into the decoding module to obtain the distribution estimates of the sub-feature matrix and the sub-mask matrix;

[0042] Based on the distribution estimation of the sub-feature matrix and the distribution estimation of the sub-mask matrix, the feature matrix and the mask matrix are arithmetic encoded by an arithmetic encoder to obtain the arithmetic encoding result, and the arithmetic encoding result is written into the sub-bitstream;

[0043] The sub-bitstreams of all the aforementioned sub-point cloud blocks are integrated into a total bitstream;

[0044] The total bitstream is processed through a second operation to obtain the target lossy reconstructed sub-point cloud;

[0045] The lossy reconstructed sub-point clouds of the target are integrated and reconstructed to obtain the result of learning-based joint compression of point cloud geometry and color.

[0046] On the other hand, embodiments of the present invention also provide a learning-based point cloud geometric color joint compression device, comprising:

[0047] The first module is used to perform a first preprocessing on the input initial training point cloud to obtain the first training point cloud;

[0048] The second module is used to input the first training point cloud into the encoding module to perform the first operation, thereby obtaining the first training feature matrix and the first training mask matrix.

[0049] The third module is used to input the first training feature matrix and the first training mask matrix into the decoding module to obtain the initial reconstructed training point cloud.

[0050] The fourth module is used to estimate the distribution of the first training feature matrix and the first training mask matrix using an initial learnable conditional entropy model, and obtain the information entropy loss.

[0051] The fifth module is used to obtain the geometric reconstruction training loss based on the coordinates of the initial training point cloud and the coordinates of the initial reconstructed training point cloud through a first calculation.

[0052] The sixth module is used to obtain the color reconstruction training loss through a second calculation based on the geometric information of the initial training point cloud and the initial reconstructed training point cloud.

[0053] The seventh module is used to perform a weighted summation of the geometric reconstruction training loss, the color reconstruction training loss, and the information entropy loss to obtain the point cloud training compression loss result.

[0054] The eighth module is used to optimize the initial learnable conditional entropy model based on the point cloud training compression loss results to obtain the target learnable conditional entropy model.

[0055] The ninth module is used to infer the initial inference point cloud based on the target learnable conditional entropy model, and obtain the learning-based point cloud geometry and color joint compression result.

[0056] On the other hand, embodiments of the present invention also provide an electronic device, including a processor and a memory; the memory is used to store a program; the processor executes the program to implement the aforementioned learning-based point cloud geometric color joint compression method.

[0057] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a program that is executed by a processor to implement the aforementioned learning-based point cloud geometry and color joint compression method.

[0058] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0059] The embodiments of the present invention include at least the following beneficial results: The present invention obtains the point cloud training compression loss result by weighted summation of geometric reconstruction training loss, color reconstruction training loss, and information entropy loss, fully balancing geometric reconstruction loss, color reconstruction loss, and corresponding information entropy loss, and fully considering the mutual influence between geometric and color reconstruction losses, which helps to generate reconstructed point clouds with higher overall quality; The learnable conditional entropy model of the present invention avoids introducing distribution assumptions about the feature matrix, enabling more flexible conditional distribution modeling, which helps to reduce the size of the transmission bitstream, while not increasing the computational complexity of the inference stage. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 A flowchart of a learning-based point cloud geometry-color joint compression method provided in an embodiment of the present invention;

[0062] Figure 2 This is a block diagram of the learning-based point cloud geometry and color joint compression device provided in an embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] On one hand, embodiments of the present invention provide a learning-based point cloud geometric color joint compression method, referring to... Figure 1 The method includes steps 100 to 900:

[0065] Step 100: Perform the first preprocessing on the input initial training point cloud to obtain the first training point cloud.

[0066] Optionally, the first preprocessing involves first performing data augmentation on the initial training point cloud, and then performing mean subtraction and normalization operations on the RGB channels of the augmented point cloud to obtain the first training point cloud. The data augmentation operation first prunes the initial training point cloud, controlling the number of points in a single point cloud to not exceed 200,000. Then, it randomly flips the point cloud along the x, y, and z axes of the coordinate system, and then randomly swaps the RGB channels to achieve data augmentation. The purpose of the data augmentation operation is to make the amplified training data as close as possible to the real distribution of data, thereby improving detection accuracy. The augmented point cloud is then represented using a sparse matrix, and the mean subtraction and normalization operations on the RGB channels of the point cloud are performed to obtain the first training point cloud. The formulas for calculating the mean subtraction and normalization are as follows:

[0067]

[0068] Among them, R i This represents the value of one of the three RGB channels of the i-th point, and N represents the number of points in the point cloud.

[0069] Step 200: Input the first training point cloud into the encoding module to perform the first operation to obtain the first training feature matrix and the first training mask matrix.

[0070] Optionally, the first training point cloud is input into the encoding module to downsample the point cloud, and then processed by [round(log2(max)] i x i After [1] downsampling iterations, the first training feature matrix and the first training mask matrix are obtained, which are the minimum-scale training feature matrix and the minimum-scale training mask matrix. The encoding module is used to extract the geometric and feature information of the point cloud and generate compact feature representations at multiple scales in a lossy manner. These compact feature representations are feature matrices containing feature information at multiple scales and mask matrices containing geometric information at multiple scales. The aim is to minimize the amount of information used for storage and transmission while preserving the main semantic information of the point cloud. Point cloud downsampling involves resampling the point cloud according to certain sampling rules. The aim is to reduce the density of the point cloud while ensuring that the overall geometric features of the point cloud remain unchanged, thereby reducing the amount of data processed and the complexity of the algorithm. Common downsampling methods include voxel downsampling, random downsampling, and uniform downsampling. This invention uses a point cloud downsampling module and an Insight-Resistant module (composed of three-dimensional sparse convolution and ReLU activation function) to downsample the first training point cloud to obtain the first training feature matrix and the first training mask matrix.

[0071] Step 300: Input the first training feature matrix and the first training mask matrix into the decoding module to obtain the initial reconstructed training point cloud.

[0072] Optionally, the first training feature matrix and the first training mask matrix are input into the decoding module to obtain the initial reconstructed training point cloud. The decoding module is used to reconstruct the geometric and feature information of the input initial training point cloud based on the compact feature representation generated by the encoding module. The decoding module first estimates the minimum-scale feature matrix through a global prior to obtain a distribution estimate of the minimum-scale feature matrix, and then upsamples the minimum-scale feature matrix to obtain a distribution estimate of the feature matrix and the mask matrix at a higher scale. Specifically, the minimum-scale feature matrix is ​​obtained by performing a 2x downsampling convolution on the initial training point cloud [round(log2(max...]. i x i The resulting 3D matrix is ​​obtained after downsampling by 1] times; the feature matrix at a higher scale is obtained by upsampling the current scale feature matrix once using a 2x upsampling convolution; further, based on the feature matrix at a higher scale, the distribution of the feature matrix and the mask matrix at higher scales are estimated. The decoding module completes round(log2(max)) i x i After one lossless point cloud upsampling, a second lossy point cloud upsampling is performed to obtain a lossy reconstructed point cloud, where x i Let be the coordinates of the i-th point in the initial training point cloud, and 'round()' represent the rounding function. The lossy reconstructed point cloud contains the geometric and color information of the initial training point cloud. Further, the initial reconstructed training point cloud is obtained by performing mean and denormalization operations on the RGB channels of the lossy reconstructed point cloud, respectively. The formulas for the mean and denormalization operations are as follows:

[0073]

[0074] Among them, R i R′ represents the RGB channel value of the i-th point in the initial training point cloud, N represents the number of points in the initial training point cloud, and R′ represents the value of the i-th point. j This represents the RGB channel value of the j-th point in the initial reconstructed training point cloud obtained by the decoding module.

[0075] Step 400: Perform distribution estimation on the first training feature matrix and the first training mask matrix using the initial learnable conditional entropy model to obtain the information entropy loss.

[0076] Optionally, a distribution estimation result is obtained by performing distribution estimation on the first training feature matrix and the first training mask matrix using an initial learnable conditional entropy model. The distribution estimation result p... y The information entropy loss is calculated using (y|z), and the formula for the initial learnable conditional entropy model is:

[0077]

[0078]

[0079]

[0080] Where y is the first training feature matrix or the first training mask matrix; z is the lower-scale feature matrix reconstructed by the decoding module; p y (y|z) is the conditional distribution estimate of y given z, i.e., the information entropy of the feature matrix or mask matrix; The noise is uniform; Q represents the addition of uniform unit noise during training and the quantization cutoff function during inference; f K f1 represents the depth decomposition entropy model; K represents the number of parameters in the depth decomposition entropy model; θ i h represents the distribution parameters of the deep decomposition entropy model. p This is a parameter mapping model containing learnable parameters, whose function is to map the feature matrix z at a lower scale to distributed parameters that can be used to construct a deep decomposition entropy model; The weight parameters are for the deep decomposition entropy model. These are the bias parameters for the deep decomposition entropy model. L is the amplitude adjustment parameter for the deep decomposition entropy model. 信息熵 For the information entropy loss, y i Let i be the i-th training feature matrix or the i-th training mask matrix.

[0081] g k (y i )=y i +tanh(a k )tanh(y i )

[0082] f k (y i ) = g k (softplus(H k )y i +b k ), 1≤k≤K

[0083] f K (y i ) = sigmoid(softplus(H K )y i +b K )

[0084] Where tanh is the hyperbolic tangent function, g k (y i y is the nonlinear activation function of the deep entropy decomposition model. ILet a be the variable whose distribution is to be estimated. K The magnitude adjustment parameters for the deep decomposition entropy model are softplus and sigmoid, which are activation functions. These activation functions ensure that the function modeled by the model satisfies the non-negativity and monotonicity properties of the probability distribution function. The definitions of the two activation functions are as follows:

[0085] softplus(x) = log(1 + e) X )

[0086]

[0087] The aforementioned initial learnable conditional entropy model avoids explicitly assuming the distribution function of the conditional distribution by dynamically constructing the distribution parameters of the deep decomposition entropy model, and can more flexibly construct the distribution of the feature matrix and the mask matrix.

[0088] Step 500: Based on the coordinates of the initial training point cloud and the coordinates of the initial reconstructed training point cloud, the geometric reconstruction training loss is obtained through a first calculation.

[0089] Optionally, the coordinates of the initial training point cloud and the coordinates of the initial reconstructed training point cloud are mapped one-to-one, and the cross-entropy loss between the two is obtained using the first calculation formula, which is used as the geometric reconstruction training loss. The first calculation formula is as follows:

[0090]

[0091] Among them, L 几何 The training loss is for geometric reconstruction; N is the number of points in the initial reconstructed training point cloud; y I This indicates the occupancy status of the i-th point in the initial reconstructed training point cloud; 1 indicates occupancy, and 0 indicates non-occupancy; p(y I ) is the y estimated by the decoding module i The probability of being occupied.

[0092] Step 600: Based on the geometric information of the initial training point cloud and the initial reconstructed training point cloud, the color reconstruction training loss is obtained through a second calculation.

[0093] Optionally, the initial reconstructed training point cloud is first recolored based on the geometric information of the initial training point cloud and the initial reconstructed training point cloud to obtain the target reconstructed training point cloud. Then, the color reconstruction training loss is obtained from the color value of the target reconstructed training point cloud and the initial reconstructed training point cloud.

[0094] Recoloring is implemented based on a distance-weighted inverse. First, let the coordinates of the initial training point cloud be... Color value The initial coordinates of the reconstructed training point cloud are: The recoloring process requires calculating the color values ​​of the initial reconstructed training point cloud. Where N is the number of points in the initial training point cloud, and M is the number of points in the initial reconstructed training point cloud. For all initial training point clouds x... i In the set Find its nearest neighbor in the set x′ and form a set x′. i The expression for x′i is:

[0095]

[0096] Among them, y K Let x be the point i The k-th nearest neighbor, d(x) i ,y j x represents the initial training point cloud. i Compared with the initial reconstruction training point cloud y J nearest neighbor distance, x I For the initial training point cloud x′ i The set of nearest neighbors in the initial reconstructed training point cloud.

[0097] For each initial reconstructed training point cloud y J Find the coordinates x of all initial training point clouds that take it as their nearest neighbor. i , forming set y i Find the color value 'a' of all initial training point clouds that use it as their nearest neighbor. i , forming set y i y i and y i The expression is:

[0098]

[0099]

[0100] Among them, y j The coordinates of the initial training point cloud, y, are the nearest neighbors of the initial reconstructed training point cloud. j For initial reconstruction training points

[0101] The initial training point of the nearest neighbor of the cloud, the color of the cloud, z i For the initial set of training point clouds to be reconstructed, y jk For y j The k-th nearest

[0102] The ratio of the color value of a neighboring point to its distance value, y jk For y j The reciprocal of the distance value of the k-th nearest neighbor, where N is the number of points in the initial training point cloud and M is the number of points in the initial reconstructed training point cloud.

[0103] Calculate the recoloring color value b of the initial reconstructed training point cloud. j The calculation formula is:

[0104]

[0105] Among them, b j The color value for recoloring, y jk For y j The ratio of the color value to the distance value of the k-th nearest neighbor, y JK for

[0106] y J The reciprocal of the distance value to the k-th nearest neighbor.

[0107] After recoloring, the formula for calculating the color reconstruction loss is as follows:

[0108]

[0109] Among them, c J b represents the color value of the j-th point in the initial training point cloud. j The recolor value is the color of the j-th point.

[0110] Step 700: The geometric reconstruction training loss, the color reconstruction training loss, and the information entropy loss are weighted and summed to obtain the point cloud training compression loss result.

[0111] Alternatively, the formula for calculating the weighted sum is:

[0112] L = 0.01·L 几何 +2.0L 颜色 +L 信息熵

[0113] Where L is the point cloud training compression loss result, L 几何 For geometric reconstruction training loss, L 颜色 For color reconstruction training loss, L 信息熵 This represents the loss of information entropy.

[0114] Step 800: Optimize the initial learnable conditional entropy model based on the point cloud training compression loss results to obtain the target learnable conditional entropy model.

[0115] Optionally, optimization begins by backpropagating and optimizing the initial learnable conditional entropy model using the automatic differentiation function of the deep learning framework PyTorch, based on the point cloud training compression loss results. Then, optimization is achieved by continuously adjusting the weight of the information entropy loss; initially, the weight is lowered to 0.01, then linearly increased during training, iterating to 1 after approximately 5000 iterations. Finally, the conditional distribution p in the initial learnable conditional entropy model is optimized. y (y|z) is pre-calculated, and the results are saved as model parameters for the target learnable conditional entropy model. For example, let z = {0, 1, ..., 1023} and y = {-9, ..., 9, 10}, and calculate p as defined in the formula of the initial learnable conditional entropy model. y (y|z), to obtain the two-dimensional matrix P Y ∈R 1204 *20 The two-dimensional matrix P y For the pre-calculated conditional distribution, the aforementioned value range can be adjusted according to the actual range; finally, the parameter mapping model h in the initial learnable conditional entropy model is removed. P We use a deep decomposition entropy model to simplify the reasoning logic and reduce the number of model parameters, thus obtaining a target learnable conditional entropy model.

[0116] Step 900: Perform inference on the initial inference point cloud according to the target learnable conditional entropy model to obtain the learning-based point cloud geometry and color joint compression result.

[0117] Optionally, the step of reasoning from the initial point cloud based on the target learnable conditional entropy model to obtain the learning-based point cloud geometric-color joint compression result includes:

[0118] Step 901: Perform a second preprocessing on the initial inference point cloud to obtain sub-point cloud blocks.

[0119] Optionally, a KD-tree can be used to divide the input point cloud into blocks, resulting in sub-point cloud blocks of no more than 200,000 pixels in size, so that each sub-point cloud can be compressed separately. A KD-tree, or k-dimensional tree for short, is a spatial partitioning data structure often used for searches in high-dimensional spaces, such as range searches and nearest neighbor searches. Since the number of point clouds is generally large, using a KD-tree for retrieval and block division can significantly reduce time consumption and ensure that the finding and registration of associated points in the point cloud are in real time.

[0120] Step 902: Subtract the mean and normalize the sub-point cloud block to obtain the RGB mean, and write the RGB mean into the sub-bitstream.

[0121] Optionally, the sub-point cloud blocks are subtracted from their mean and normalized to obtain the RGB mean, and then the RGB mean is written into the sub-bitstream in 32-bit floating-point form. The calculation formulas for subtracting the mean and normalizing are as follows:

[0122]

[0123] Among them, R i This represents the RGB channel value of the i-th point, and N represents the number of points in the sub-point cloud block.

[0124] Step 903: Input the sub-point cloud block into the encoding module to obtain the sub-feature matrix and the sub-mask matrix.

[0125] Optionally, the sub-point cloud block is input into the encoding module for downsampling, and then rounded (log2(max)). i x i After several downsampling operations, sub-feature matrices and sub-mask matrices are obtained.

[0126] Step 904: Input the sub-feature matrix and the sub-mask matrix into the decoding module to obtain the distribution estimate of the sub-feature matrix and the distribution estimate of the sub-mask matrix.

[0127] Optionally, the decoding module first uses a global prior to estimate the minimum-scale sub-feature matrix, obtaining a distribution estimate of the minimum-scale sub-feature matrix. Specifically, the minimum-scale sub-feature matrix is ​​obtained by rounding (log2(max)) the sub-point cloud blocks using a 2x downsampling convolution. i x i After )) downsampling, the resulting three-dimensional matrix is ​​obtained, where x i Let be the coordinates of the i-th point in the sub-point cloud block, and `round()` be the rounding function. Then, the decoding module upsamples the minimum-scale sub-feature matrix to obtain the distribution estimates of the feature matrix and mask matrix at a higher scale. Further, based on the higher-scale feature matrix, the decoding module performs distribution estimates on the even higher-scale feature matrix and mask matrix. After completing `round(log2(max)`, the decoding module... i x i Lossless point cloud upsampling is performed to obtain the distribution estimates of the sub-feature matrix and the sub-mask matrix at each scale of the encoding module, where x i Let be the coordinates of the i-th point in the sub-point cloud block, and round() is the rounding function.

[0128] Step 905: Based on the distribution estimation of the sub-feature matrix and the distribution estimation of the sub-mask matrix, perform arithmetic encoding on the feature matrix and the mask matrix using an arithmetic encoder to obtain the arithmetic encoding result, and write the arithmetic encoding result into the sub-bitstream.

[0129] Optionally, based on the distribution estimation of the sub-feature matrix and the distribution estimation of the sub-mask matrix, an arithmetic encoder rANS is used to perform arithmetic encoding on the sub-feature matrix and the sub-mask matrix output by the encoding module to obtain the arithmetic encoding result, and the arithmetic encoding result is written into the bitstream; wherein, rANS is a non-mean entropy encoding for non-binary characters.

[0130] Step 906: Integrate the sub-bitstreams of all the sub-point cloud blocks into a total bitstream.

[0131] Optionally, a header is constructed, the length of each sub-stream is written into the header, and then the header and each sub-stream are integrated into a total bitstream, which is used for storage and transmission.

[0132] Step 907: Obtain the target lossy reconstructed sub-point cloud by processing the total bitstream through the second operation.

[0133] Optionally, the second operation is as follows: First, after the decoding module receives the total bitstream output by the encoding module, it splits the bitstream according to the length of each sub-bitstream recorded in the header information to obtain sub-bitstreams, and decodes each sub-bitstream separately; then, for each sub-bitstream, it splits it according to the header information to obtain the bitstream of the sub-feature matrix and the bitstream of the sub-mask matrix at each scale; then, it uses global prior to perform arithmetic decoding on the sub-feature matrix at the smallest scale to obtain the sub-feature matrix at the smallest scale, inputs it into the decoding module, performs upsampling of the point cloud, and obtains the distribution estimate of the sub-feature matrix and the distribution estimate of the sub-mask matrix at higher scales; further, it performs arithmetic decoding on the sub-feature matrix and the sub-mask matrix at higher scales to achieve lossless reconstruction at that scale. After completing lossless upsampling at all scales except the highest scale, it performs lossy upsampling at the highest scale for the last time to obtain the target sub-lossy reconstructed point cloud at the same scale as the sub-point cloud block.

[0134] Step 908: Integrate and reconstruct the target lossy sub-point cloud to obtain the result of learning-based point cloud geometric color joint compression.

[0135] Optionally, the RGB values ​​of the target lossy reconstructed sub-point cloud are first averaged using the RGB mean value stored in the bitstream. Then, the RGB channels of the target lossy reconstructed sub-point cloud are normalized and rounded to obtain discrete RGB prediction values ​​in the range of 0 to 255. The coordinates and colors of all the target lossy reconstructed sub-point clouds that have undergone normalization and rounding are written into a single point cloud file in any order. The point cloud file can be saved using common point cloud formats such as ply and off. This point cloud file is the result of joint compression of the learned point cloud geometry and color. The calculation formula for the normalization and rounding operations is:

[0136]

[0137] Among them, R′ i R represents the RGB channel value of the i-th point in the sub-point cloud block, N represents the number of points in the sub-point cloud block, and R represents the value of the i-th point. j This represents the RGB channel value of the j-th point in the lossy reconstructed sub-point cloud of the decoding module, where round represents the rounding function.

[0138] The following example illustrates the application of the learning-based point cloud geometry and color joint compression method provided in the embodiments of the present invention.

[0139] 1. First, the initial training point cloud is preprocessed to obtain the first training point cloud. The first preprocessing first performs data augmentation on the initial training point cloud, and then performs mean subtraction and normalization operations on the three RGB channels of the augmented point cloud to obtain the first training point cloud.

[0140] 2. Then, the first training point cloud is input into the encoding module for downsampling to obtain the first training feature matrix and the first training mask matrix.

[0141] 3. Then input the first training feature matrix and the first training mask matrix into the decoding module to obtain the initial reconstructed training point cloud.

[0142] 4. Information entropy is obtained by performing distribution estimation on the first training feature matrix and the first training mask matrix using an initial learnable conditional entropy model. The information entropy is used to calculate the information entropy loss. Geometric reconstruction training loss is obtained through a first calculation based on the coordinates of the initial training point cloud and the coordinates of the initial reconstructed training point cloud. Color reconstruction training loss is obtained through a second calculation based on the geometric information of the initial training point cloud and the initial reconstructed training point cloud. The geometric reconstruction training loss, the color reconstruction training loss, and the information entropy loss are weighted and summed to obtain the point cloud training compression loss result.

[0143] 5. Optimize the initial learnable conditional entropy model to obtain the target learnable conditional entropy model.

[0144] 6. Based on the target learnable conditional entropy model, infer the initial inference point cloud to obtain the learning-based point cloud geometry and color joint compression result.

[0145] In summary, the learning-based point cloud geometry and color joint compression method of this invention has the following advantages:

[0146] 1. In this embodiment of the invention, the geometric reconstruction training loss, color reconstruction training loss and information entropy loss are weighted and summed to obtain the point cloud training compression loss result. The geometric reconstruction loss, color reconstruction loss and corresponding information entropy loss are fully balanced, and the mutual influence between geometric and color reconstruction losses is fully considered, which helps to generate a reconstructed point cloud with higher overall quality.

[0147] 2. The learnable conditional entropy model in this embodiment of the invention avoids introducing distribution assumptions about the feature matrix, enabling more flexible conditional distribution modeling, which helps to reduce the size of the transmitted bitstream, while not increasing the computational complexity of the inference stage.

[0148] 3. In the inference process, the embodiments of the present invention perform joint information extraction on the geometry and color of the point cloud, avoiding the problem of redundant information extraction caused by separate compression. Compared with the existing technology that processes geometry and color information separately, the present invention can effectively reduce computational complexity and improve practicality.

[0149] Reference Figure 2 This invention also provides a learning-based point cloud geometry and color joint compression device, comprising:

[0150] The first module 201 is used to perform a first preprocessing on the input initial training point cloud to obtain the first training point cloud.

[0151] The second module 202 is used to input the first training point cloud into the encoding module to perform a first operation to obtain a first training feature matrix and a first training mask matrix.

[0152] The third module 203 is used to input the first training feature matrix and the first training mask matrix into the decoding module to obtain the initial reconstructed training point cloud.

[0153] The fourth module 204 is used to perform distribution estimation on the first training feature matrix and the first training mask matrix through an initial learnable conditional entropy model to obtain the information entropy loss.

[0154] The fifth module 205 is used to obtain the geometric reconstruction training loss through a first calculation based on the coordinates of the initial training point cloud and the coordinates of the initial reconstructed training point cloud.

[0155] The sixth module 206 is used to obtain the color reconstruction training loss through a second calculation based on the geometric information of the initial training point cloud and the initial reconstructed training point cloud.

[0156] The seventh module 207 is used to perform a weighted summation of the geometric reconstruction training loss, the color reconstruction training loss, and the information entropy loss to obtain the point cloud training compression loss result.

[0157] The eighth module 208 is used to optimize the initial learnable conditional entropy model based on the point cloud training compression loss result to obtain the target learnable conditional entropy model.

[0158] The ninth module 209 is used to infer the initial inference point cloud based on the target learnable conditional entropy model to obtain a learning-based point cloud geometry and color joint compression result.

[0159] This invention also provides an electronic device, including a processor and a memory, wherein the memory is used to store a program, and the processor executes the program to implement the aforementioned learning-based point cloud geometry and color joint compression method.

[0160] This invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the aforementioned learning-based point cloud geometry and color joint compression method.

[0161] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0162] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0163] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0164] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0166] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0167] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0168] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0169] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0170] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A learning-based method for joint compression of point cloud geometry and color, characterized in that, include: The initial training point cloud is preprocessed to obtain the first training point cloud. The first training point cloud is input into the encoding module to perform the first operation, thereby obtaining the first training feature matrix and the first training mask matrix; The first training feature matrix and the first training mask matrix are input into the decoding module to obtain the initial reconstructed training point cloud. The information entropy loss is obtained by estimating the distribution of the first training feature matrix and the first training mask matrix using an initial learnable conditional entropy model. Based on the coordinates of the initial training point cloud and the coordinates of the initial reconstructed training point cloud, the geometric reconstruction training loss is obtained through a first calculation. Based on the geometric information of the initial training point cloud and the initial reconstructed training point cloud, the color reconstruction training loss is obtained through a second calculation. The point cloud training compression loss is obtained by weighted summation of the geometric reconstruction training loss, the color reconstruction training loss, and the information entropy loss. The initial learnable conditional entropy model is optimized based on the point cloud training compression loss results to obtain the target learnable conditional entropy model. The initial inference point cloud is inferred based on the target learnable conditional entropy model to obtain the learning-based point cloud geometry and color joint compression result.

2. The learning-based point cloud geometry and color joint compression method according to claim 1, characterized in that, The step of inputting the first training feature matrix and the first training mask matrix into the decoding module to obtain the initial reconstructed training point cloud includes: The distribution estimate of the first-scale training feature matrix is ​​obtained by using global prior estimation of the first training feature matrix; Multiple point cloud upsampling operations are performed on the first-scale training feature matrix to obtain the distribution estimates of the multi-scale training feature matrix and the multi-scale training mask matrix. Based on the distribution estimation of the multi-scale training feature matrix and the distribution estimation of the multi-scale training mask matrix, the initial reconstructed training point cloud is obtained.

3. The learning-based point cloud geometry and color joint compression method according to claim 1, characterized in that, In the step of estimating the distribution of the first training feature matrix and the first training mask matrix using an initial learnable conditional entropy model to obtain the information entropy loss, the formula for the initial learnable conditional entropy model is: Where y is the training feature matrix or training mask matrix, and z is the training feature matrix reconstructed by the decoding module. For a given hour Conditional distribution, For unit uniform noise, Q represents the addition of unit uniform noise. For deep decomposition entropy model, The distribution parameters of the deep decomposition entropy model, This is a parameter mapping model that contains learnable parameters. These are the weight parameters for the deep decomposition entropy model. These are the bias parameters for the deep decomposition entropy model. For adjusting the magnitude of the deep decomposition entropy model, For the information entropy loss, Let i be the i-th training feature matrix or the i-th training mask matrix.

4. The learning-based point cloud geometry and color joint compression method according to claim 1, characterized in that, In the step of obtaining the geometric reconstruction training loss through a first calculation based on the coordinates of the initial training point cloud and the coordinates of the initial reconstructed training point cloud, the calculation formula for the first calculation is as follows: ) in, The training loss is defined as N, where N is the total number of points in the initial reconstructed training point cloud. The occupancy status of the i-th point in the initial reconstruction training point cloud; Estimated for the decoding module The probability of being occupied.

5. The learning-based point cloud geometry and color joint compression method according to claim 1, characterized in that, The color reconstruction training loss is obtained through a second calculation based on the geometric information of the initial training point cloud and the initial reconstructed training point cloud, including: Based on the color values ​​of the initial training point cloud and the initial reconstructed training point cloud, the initial reconstructed training point cloud is recolored to obtain the target reconstructed training point cloud; The color reconstruction training loss is obtained from the color values ​​of the target reconstructed training point cloud and the initial reconstructed training point cloud.

6. The learning-based point cloud geometry and color joint compression method according to claim 3, characterized in that, The step of optimizing the initial learnable conditional entropy model based on the point cloud training compression loss result to obtain the target learnable conditional entropy model includes: Based on the point cloud training compression loss results, backpropagation is performed on the initial learnable conditional entropy model; The conditional distribution in the initial learnable conditional entropy model after backpropagation is pre-calculated, and the calculation result is used as the model parameters of the target learnable conditional entropy model. The parameter mapping model and the deep decomposition entropy model are removed from the initial learnable conditional entropy model to obtain the target learnable conditional entropy model.

7. The learning-based point cloud geometry and color joint compression method according to claim 1, characterized in that, The step of reasoning on the initial inference point cloud based on the target learnable conditional entropy model to obtain a learning-based point cloud geometry-color joint compression result includes: The initial inference point cloud is subjected to a second preprocessing step to obtain sub-point cloud blocks; The sub-point cloud blocks are subtracted from the mean and normalized to obtain the RGB mean value, and the RGB mean value is written into the sub-bitstream; The sub-point cloud blocks are input into the encoding module to obtain the sub-feature matrix and the sub-mask matrix; The sub-feature matrix and the sub-mask matrix are input into the decoding module to obtain the distribution estimates of the sub-feature matrix and the sub-mask matrix; Based on the distribution estimation of the sub-feature matrix and the distribution estimation of the sub-mask matrix, the feature matrix and the mask matrix are arithmetic encoded by an arithmetic encoder to obtain the arithmetic encoding result, and the arithmetic encoding result is written into the sub-bitstream; The sub-bitstreams of all the aforementioned sub-point cloud blocks are integrated into a total bitstream; The total bitstream is processed through a second operation to obtain the target lossy reconstructed sub-point cloud; The lossy reconstructed sub-point clouds of the target are integrated and reconstructed to obtain the result of learning-based joint compression of point cloud geometry and color.

8. A learning-based point cloud geometric color joint compression device, characterized in that, include: The first module is used to perform a first preprocessing on the input initial training point cloud to obtain the first training point cloud; The second module is used to input the first training point cloud into the encoding module to perform the first operation, thereby obtaining the first training feature matrix and the first training mask matrix. The third module is used to input the first training feature matrix and the first training mask matrix into the decoding module to obtain the initial reconstructed training point cloud. The fourth module is used to estimate the distribution of the first training feature matrix and the first training mask matrix using an initial learnable conditional entropy model, and obtain the information entropy loss. The fifth module is used to obtain the geometric reconstruction training loss based on the coordinates of the initial training point cloud and the coordinates of the initial reconstructed training point cloud through a first calculation. The sixth module is used to obtain the color reconstruction training loss through a second calculation based on the geometric information of the initial training point cloud and the initial reconstructed training point cloud. The seventh module is used to perform a weighted summation of the geometric reconstruction training loss, the color reconstruction training loss, and the information entropy loss to obtain the point cloud training compression loss result. The eighth module is used to optimize the initial learnable conditional entropy model based on the point cloud training compression loss results to obtain the target learnable conditional entropy model. The ninth module is used to infer the initial inference point cloud based on the target learnable conditional entropy model, and obtain the learning-based point cloud geometry and color joint compression result.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 7.

Images