A placeholder-guided artifact removal method for wide-bitrate dynamic point cloud coding

Abstract

The application discloses a kind of placeholder guide artifact removal methods for wide code rate dynamic point cloud coding, the method includes: extracting placeholder from video-based dynamic point cloud coding compression block, and the size of the placeholder is equal to the size of geometry compression block after being up-sampled;The geometry compression block and the up-sampled placeholder are cascaded, and then input into the Uformer variant, the effective features of the geometry compression block are extracted, and the recovery block is obtained;According to the recovery block and the geometry compression block, the corresponding logarithmic error is calculated, and it is used as error loss;The learning rate is updated using the SGD function, the model parameters are iterated using the adaptive gradient optimizer, the error loss is balanced, the model is trained, and the final optimal model is obtained;The geometry block and the corresponding up-sampled placeholder are cascaded, input into the optimal model, and the geometry video frame after removing artifacts is obtained.

Description

Technical Field

[0001] This invention relates to a placeholder-guided artifact removal method for wide bitrate dynamic point cloud encoding, belonging to the interdisciplinary technical field of image and video processing and machine learning. Background Technology

[0002] In recent years, point clouds have been widely used to describe 3D objects. To better facilitate the transmission and storage of point clouds in 3D applications, MPEG initiated the development of point cloud compression standards. Video-based point cloud compression is one of the standards developed for dynamic point cloud compression.

[0003] In video-based dynamic point cloud coding methods, a block projection method is proposed to fully utilize temporal correlation. This method projects the geometric and attribute information of the point cloud onto a 2D video, and then compresses the 2D video using video coding techniques such as HEVC and VVC. Specifically, within a video frame, adjacent points with similar normal vectors are first grouped into a geometric projection block, which is then packed into a 2D mesh to construct the video. However, the shapes of geometric projection blocks are generally irregular, with gaps between blocks, resulting in a large number of empty pixels in the generated video. To facilitate encoding and decoding, empty pixels are filled with adjacent non-empty pixels, and a placeholder map (OM) needs to be provided to the decoder to indicate which pixels are empty.

[0004] Lossy video compression in video-based dynamic point cloud coding methods introduces compression artifacts, leading to a decline in point cloud reconstruction quality. Traditional compression artifact removal methods mainly utilize local prior knowledge or non-local similarity of images, such as autoregressive models and sparse representations. However, these methods are often limited by the inefficiency of manual features and the strong assumptions of prior models.

[0005] In recent years, due to their powerful ability to learn representations, deep learning-based models, such as convolutional neural networks (CNNs), have been widely used for compression artifact removal and image denoising. The main idea behind CNN-based noise denoising is to predict noise by extracting local contextual features. The training objective is to minimize the error between the noise signal and the original signal, such as mean squared error (MSE) and mean absolute error (MAE). CNN-based models have achieved excellent performance in compression artifact removal due to their advantage in extracting local contextual features of images.

[0006] However, the videos generated in video-based dynamic point cloud coding methods differ from natural image signals, thus applying CNNs to video-based dynamic point cloud coding artifact removal still presents some challenges. The main problems are as follows: First, filling empty pixels with adjacent non-empty pixels fails to accurately reflect local contextual information; second, due to an imbalance in the iterative update of learnable parameters, the training cost based on MSE (Mean Sequence of Effects) causes the trained model to perform better at low bit rates than at high bit rates. To address these issues, this invention proposes a placeholder-map-guided geometric compression artifact removal method for dynamic point cloud coding applicable across a wide bit rate range.

[0007] However, the numerous empty pixels filled with adjacent non-empty pixels in the video fail to accurately reflect local context information, reducing the noise prediction accuracy in compression artifact removal. Furthermore, due to the imbalance in the iterative updates of the model's learnable parameters, the mean squared error (MSE)-based training model performs better at low bit rates than at high bit rates. Therefore, this invention proposes a learning-based wide-bit-rate dynamic point cloud coding geometric compression artifact removal method, mainly comprising a placeholder map-based contextual feature extraction scheme and an incremental peak signal-to-noise ratio (PSNR)-based training method. Summary of the Invention

[0008] The purpose of this invention is to provide a placeholder-guided artifact removal method for wide bitrate dynamic point cloud encoding, so as to solve the defects of existing technologies that cannot truly reflect local context information and the imbalance of iterative updates of learnable parameters of the model.

[0009] A placeholder-guided artifact removal method for wide-bitrate dynamic point cloud coding, the method comprising:

[0010] Placeholder images are extracted from video-based dynamic point cloud coded compression blocks, and the placeholder images are upsampled to make their size equal to that of the geometric compression blocks.

[0011] The geometrically compressed block and the upsampled placeholder map are concatenated and then input into the Uformer variant to extract the effective features of the geometrically compressed block and obtain the recovered block.

[0012] The corresponding logarithmic error is calculated based on the recovery block and the geometric compression block, and is used as the error loss.

[0013] The learning rate is updated using the SGD function, and the model parameters are iterated and the error loss is balanced using the adaptive gradient optimizer to train the model and obtain the final optimal model.

[0014] The geometric blocks and their corresponding upsampled placeholder images are concatenated and input into the optimal model to obtain geometric video frames after artifact removal.

[0015] Furthermore, the input for training the optimal model includes far-layer blocks, corresponding near-layer blocks, and placeholder maps extracted from video-based dynamic point cloud coded compressed blocks.

[0016] Furthermore, the placeholder image is upsampled and then max-pooled, and the max-pooled placeholder image is input into the Uformer variant.

[0017] Furthermore, methods for calculating error loss include:

[0018] The difference between the recovered block and the original block is calculated, and then multiplied by the oversampled placeholder image to filter out interference from the difference calculation of empty pixels to non-empty pixels;

[0019] The logarithmic error is calculated point by point for each non-empty pixel, and the summation yields the final error loss.

[0020] Furthermore, the formula for calculating the error loss is as follows:

[0021] In the formula, and These represent the recovery blocks of the distant layer and the corresponding near layer, respectively; "and" " represents the corresponding original block, while " om "" indicates a placeholder image for upsampling.

[0022] Furthermore, the formula for calculating the error loss is as follows:

[0023] Furthermore, the optimal model is trained based on incremental peak signal-to-noise ratio (PSNR), and the incremental PSNR method includes:

[0024] The training objective is to obtain the minimum MSE between the recovered block and the original block.

[0025] ,

[0026] in, m The number of training samples, Indicates the first i One original block;

[0027] To obtain the optimal model, gradient descent is typically used during training to update the parameters, specifically...

[0028]

[0029] in, η The learning rate specifies the size of the parameter updates. This is for noise reduction error;

[0030] For each iteration, the objective is to maximize the sum of incremental PSNR, which is expressed as:

[0031]

[0032] Where, ∆ PSNR i For the first i The incremental PSNR for each sample, which is the PSNR gain after artifact removal, is obtained from the following formula:

[0033]

[0034] in, n Number of signal bits; symbol and These represent the recovery block and the input compressed block, respectively. MSE Error, the equation can be rewritten as

[0035]

[0036] Then, we get

[0037]

[0038] The equation can be rewritten as follows

[0039]

[0040] For each batch of samples, the error before artifact removal is constant; for each iteration, If it is a constant, then it simplifies to:

[0041]

[0042] Set the final objective function as

[0043]

[0044] The logarithmic operation in the final objective function adjusts the impact of MSE on total cost.

[0045] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0046] The proposed deep learning-based geometric compression artifact removal method for video-based dynamic point cloud coding across a wide bitrate range effectively improves the quality of reconstructed point clouds. The main innovations of this invention are: first, the proposed OMCFE method, which improves the accuracy of extracted contextual features by utilizing OM (omic markers) for empty pixels; and second, the proposed IPT method, which significantly improves the artifact removal performance of video-based dynamic point cloud coding geometric compression blocks, particularly in the high bitrate range, by balancing the error differences between high and low bitrate compression blocks. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the geometric compression artifact removal method based on video dynamic point cloud coding with a wide bit rate range according to the present invention;

[0048] Figure 2 This is an example diagram of the geometric frame (left) and corresponding placeholder image (right) generated by the video-based dynamic point cloud encoding of this invention;

[0049] Figure 3 This presents the statistical results of denoising error of the training model of the present invention on different quantization parameter encoding blocks at different epochs. Detailed Implementation

[0050] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0051] This invention discloses a placeholder-guided artifact removal method for wide-rate dynamic point cloud coding, such as... Figure 1 The steps shown are as follows:

[0052] Step 1:

[0053] Step 1: Extract the placeholder image from the video-based dynamic point cloud encoding compression block and upsample it to make its size equal to that of the geometric compression block, then proceed to Step 2;

[0054] Step 2: Concatenate the geometrically compressed block and the upsampled placeholder map, then input them into the Uformer variant to extract the effective features of the geometrically compressed block and obtain the recovered block.

[0055] Step Two:

[0056] Step 3: Using the recovery block obtained in Step 2 and the corresponding geometric block before compression, the corresponding logarithmic error is calculated by formula (10) under the condition that only the actual non-empty pixels are considered, and it is used as the error loss to execute Step 4.

[0057] Step 4: Update the learning rate using the SGD function, iterate the model parameters using the Adamw (adaptive gradient) optimizer, balance the error loss, train the model, and obtain the final optimal model parameters.

[0058] Step 3:

[0059] The geometric compression block and the corresponding upsampled placeholder map are concatenated and input into the model with the optimal parameters to achieve the final removal of geometric artifacts based on dynamic point cloud coding in video, resulting in the final geometric video frame after artifact removal.

[0060] The contextual feature extraction based on the placeholder image is as follows:

[0061] Video-based dynamic point cloud coding (DPLC) produces videos that differ from natural images due to the padding of empty pixels. In DPLC, empty pixels are padded with adjacent non-empty pixels for easier encoding. For geometric videos, frames are divided into multiple non-overlapping, independently padded square blocks. If all pixels in a block are empty, the block is padded by copying the last column or row of the previous block. If a block contains both empty and non-empty pixels, the empty pixels are iteratively padded with the average of its non-empty neighboring pixels. Figure 2 The image shows an example of a filled geometric frame and its corresponding placeholder image.

[0062] Simple padding schemes cannot accurately reflect local contextual information. Therefore, local contextual features extracted using deep learning methods such as convolutional networks are often inaccurate, which affects the accuracy of noise prediction. To reconstruct point clouds, dynamic point clouds based on videos encode placeholder maps to indicate which pixels are empty and which are not. From another perspective, placeholder maps indicate which regions in the image provide genuine contextual information and which regions provide false contextual information. Therefore, this invention proposes a placeholder map-based Contextual Feature Extraction (OMCFE) to accurately utilize contextual information for noise prediction.

[0063] The framework of the proposed method is as follows: Figure 1 As shown. The video generated in video-based dynamic point cloud encoding is segmented into non-overlapping blocks. Artifact removal is then performed on each block. A variant of the Uformer, a state-of-the-art denoising model, is used as the backbone of this method. The difference lies in the new input and output of the model. (See diagram.) Figure 1As shown, both far-layer and corresponding near-layer geometry blocks are used as input. More importantly, the corresponding placeholder maps are also input into the model. The input can be considered to have three channels: far-layer, near-layer, and placeholder map. Furthermore, in each encoding / decoding stage of the Uformer variant, a placeholder map of the corresponding size is also input into the Transformer Blocks (TB). Note that the placeholder map is downsampled during encoding. To ensure that the sizes of the first three input channels are equal, the placeholder map is upsampled accordingly; and before the placeholder map is input into each encoding / decoding stage, it needs to be max-pooled accordingly. Finally, the output of the Uformer ontology is the recovered blocks of the far and near layers.

[0064] 2) Training scheme based on incremental PSNR (Peak Signal-to-Noise Ratio)

[0065] Learning-based artifact removal primarily involves training a parameter set to... θ model F For a given noisy image patch x The model can predict a recovery block, i.e. x’ = F ( x ; θ Generally, a large number of image patches are needed to train the model to achieve the training objective of minimizing the MSE between the recovered block and the original block, i.e.:

[0066] , (1)

[0067] in, m The number of training samples, Indicates the first i The original blocks. To obtain the optimal model, gradient descent is typically used during training to update the parameters, specifically...

[0068] (2)

[0069] in, η The learning rate specifies the size of the parameter updates. It has been observed that the denoising error is: This also directly affects the magnitude of model parameter updates.

[0070] The development of video-based dynamic point cloud encoding (point cloud compression tool) aims to compress point clouds over a wide bit rate range. Therefore, artifact removal should also be adapted to a wide bit rate range. As shown in (2), the magnitude of parameter updates is directly affected by the instantaneous denoising error. However, in practical applications, the noise in high-bit-rate compressed blocks is usually smaller than that in low-bit-rate compressed blocks, therefore the denoising error of high-bit-rate compressed blocks is also smaller than that of low-bit-rate compressed blocks, such as... Figure 3 As shown, we observed the denoising error of the trained model at different epochs on blocks compressed with different quantization parameters. The results also verify the above statement. Therefore, parameter updates are more affected by low bit rate blocks, and the trained model performs better on low bit rate blocks. To ensure the applicability of the model over a wide bit rate range, we propose an incremental PSNR-based training (IPT) scheme by balancing the impact of high and low bit rate blocks on parameter updates.

[0071] Specifically, for each iteration, the goal is to maximize the sum of incremental PSNRs, which is expressed as:

[0072] (3)

[0073] Where, ∆ PSNR i For the first i The incremental PSNR for each sample is the PSNR gain after artifact removal, which is obtained by the following formula.

[0074] (4)

[0075] in, n This represents the number of signal bits. (Symbol) and These represent the recovery block and the input compressed block, respectively. MSE Error. The equation above can be rewritten as:

[0076] (5)

[0077] Then we can get

[0078] (6)

[0079] The above equation can be rewritten as follows:

[0080] (7)

[0081] For each batch of samples, the error before artifact removal is constant. That is, for each iteration, It is a constant. Therefore, equation (7) simplifies to:

[0082] (8)

[0083] We can set the final objective function as

[0084] (9)

[0085] The logarithmic operation in the final objective function adjusts the impact of the Mean Squared Estimate (MSE) on the total cost. Specifically, it reduces the impact of large MSEs on the objective function, while enhancing the impact of small MSEs.

[0086] Note that only the error of non-empty pixels is considered during training. The error for each sample is calculated as follows:

[0087] (10)

[0088] In the formula, and These represent the recovery blocks from the distant and corresponding near layers, respectively. The symbol " "and" " represents the corresponding original block, while " om "" indicates a placeholder image for upsampling.

[0089] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A placeholder-guided artifact removal method for wide bitrate dynamic point cloud coding, characterized in that, The method includes: Placeholder images are extracted from video-based dynamic point cloud coded compression blocks, and the placeholder images are upsampled to make their size equal to that of the geometric compression blocks. The geometrically compressed block and the upsampled placeholder map are concatenated and then input into the Uformer variant to extract the effective features of the geometrically compressed block and obtain the recovered block. The corresponding logarithmic error is calculated based on the recovery block and the geometric compression block, and is used as the error loss. The learning rate is updated using the SGD function, and the model parameters are iterated and the error loss is balanced using the adaptive gradient optimizer to train the model and obtain the final optimal model. The geometric blocks and their corresponding upsampled placeholder images are concatenated and input into the optimal model to obtain geometric video frames after artifact removal.

2. The placeholder-guided artifact removal method for wide bitrate dynamic point cloud coding according to claim 1, characterized in that, The input for training the optimal model includes far-layer blocks, corresponding near-layer blocks, and placeholder maps extracted from video-based dynamic point cloud coded compressed blocks.

3. The placeholder-guided artifact removal method for wide bitrate dynamic point cloud coding according to claim 1, characterized in that, The placeholder image is upsampled and then max-pooled. The max-pooled placeholder image is then input into the Uformer variant.

4. The placeholder-guided artifact removal method for wide bitrate dynamic point cloud coding according to claim 1, characterized in that, Methods for calculating error loss include: The difference between the recovered block and the original block is calculated, and then multiplied by the oversampled placeholder image to filter out interference from the difference calculation of empty pixels to non-empty pixels; The logarithmic error is calculated point by point for each non-empty pixel, and the summation yields the final error loss.

5. The placeholder-guided artifact removal method for wide bitrate dynamic point cloud coding according to claim 4, characterized in that, The formula for calculating the error loss is as follows: In the formula, and These represent the recovery blocks of the distant layer and the corresponding near layer, respectively. "and" " represents the corresponding original block, while" om "" indicates a placeholder image for upsampling.

6. The placeholder-guided artifact removal method for wide bitrate dynamic point cloud coding according to claim 1, characterized in that, The optimal model is trained based on incremental peak signal-to-noise ratio (PSNR), and the incremental PSNR training method includes: The training objective is to obtain the minimum MSE between the recovered block and the original block. ;in, m The number of training samples, Indicates the first i One original block; To obtain the optimal model, gradient descent is typically used to update parameters during training, specifically: ； in, η The learning rate specifies the size of the parameter updates. This is for noise reduction error; For each iteration, the goal is to maximize the sum of incremental PSNR, which is expressed as: ; Where, ∆ PSNR i For the first i The incremental PSNR for each sample, which is the PSNR gain after artifact removal, is obtained by the following formula: ;in, n Number of signal bits; symbol and These represent the recovery block and the input compressed block, respectively. MSE Error, the equation can be rewritten as: Then, we get The equation can be rewritten as follows For each batch of samples, the error before artifact removal is constant; for each iteration, If it is a constant, then it simplifies to: Let the final objective function be set as: The logarithmic operation in the final objective function adjusts the impact of MSE on total cost.

Images