VVC-based efficient point cloud projection video coding method

Abstract

The invention relates to the field of digital video signal coding, in particular to a VVC-based efficient point cloud projection video coding method, which comprises the following steps of: inputting a point cloud projection sequence into a coder to obtain division mode distribution of a geometric video CU and an attribute video CU; constructing a dedicated data set for each distribution mode based on the division mode distribution; performing fine adjustment and multi-modal feature extraction on the special data set, and calculating a division probability and a mode probability; the division probability and the mode probability are judged, and candidate pool direct control is carried out; judging the confidence coefficient of the division mode, and selectively writing the mode into a candidate pool according to a judgment result to complete coding; according to the video coding method designed by the invention, a self-adaptive division decision mechanism is established by combining the correlation between the useful pixel distribution characteristics of the CU and the multi-source video, the coding complexity is reduced and the compression efficiency is improved on the premise of keeping or basically keeping the coding quality, and the video coding method is suitable for a point cloud compression scene with a relatively large data scale.

Description

Technical Field

[0001] This invention relates to the field of digital video signal coding, and more specifically to an efficient point cloud projection video coding method based on VVC. Background Technology

[0002] To compress point cloud data, the Moving Pictures Experts Group (MPEG) Immersive Media Working Group released a video-based point cloud compression scheme (V-PCC) in 2017, and provided a standard compression encoder, TMC2. Point cloud video is typically mapped to a two-dimensional plane in the form of geometric video and attribute video, and then compressed using hybrid codec frameworks such as HEVC and VVC (Versatile Video Coding).

[0003] In existing technologies, the V-PCC fast coding method based on HEVC has been extensively studied and has achieved significant progress. Its main idea is to improve coding efficiency by optimizing the coding unit (CU) partitioning process. H.266 / VVC can save approximately 50% of the bitrate compared to H.265 / HEVC. This standard not only integrates the advantages of previous coding technologies but also introduces a more flexible block partitioning structure and provides stronger compression capabilities for diverse video formats and content. Compared to HEVC, VVC is clearly more suitable for handling point cloud compression tasks with large amounts of data. However, geometric and textured videos generated from point cloud projection contain a large number of blank areas introduced by the filling algorithm, and the distribution pattern of video content also differs significantly from natural videos. Based on these factors, the VVC video coding algorithm is difficult to directly apply to point cloud projection videos.

[0004] Therefore, there is an urgent need for a fast VVC encoding method that combines the unique properties of point cloud projection images. Summary of the Invention

[0005] In view of this, this application discloses an efficient point cloud projection video coding method based on VVC to solve the above problems; including:

[0006] S1. Input the point cloud projection sequence into the encoder to obtain the partitioning pattern distribution of geometric video CU and attribute video CU;

[0007] S2. Based on the partitioning pattern distribution, construct a dedicated dataset for each distribution pattern;

[0008] S3. Fine-tune the dedicated dataset and extract multimodal features, and calculate the partition probability and pattern probability;

[0009] S4. Set a threshold for the division probability, distinguish between the division probability and the pattern probability, and control the candidate pool directly based on the discrimination result;

[0010] S5. Determine the confidence level of the partitioning pattern, and selectively write the pattern into the candidate pool according to the determination result to complete the encoding.

[0011] This application presents a high-efficiency point cloud projection video coding method based on VVC. By combining the useful pixel distribution characteristics of the CU with the correlation between multiple source videos, an adaptive partitioning decision mechanism is established. This reduces coding complexity and improves compression efficiency while maintaining or basically maintaining coding quality, making it suitable for point cloud compression scenarios with large data scales. By combining the scale difference between finer near layers and larger far layers, different thresholds are set, and modes are selectively written into the candidate pool according to confidence level. By designing the master head decision, candidate mask, and hierarchical threshold, inefficient distortion evaluation is reduced while improving video coding quality. This avoids a large number of invalid RDcost comparisons and also reduces the time overhead of pushing modes for each CU. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the efficient point cloud projection video coding method based on VVC in this application.

[0013] Figure 2 This is a schematic diagram of the neural networks used for geometric video scenes and attribute video scenes respectively when performing differential feature extraction in the embodiments of this application;

[0014] Figure 3 This is a schematic diagram of near-layer full CU occupancy for attribute video in an embodiment of this application;

[0015] Figure 4 This is a schematic diagram showing the CU occupied by the near-layer portion of the attribute video in an embodiment of this application;

[0016] Figure 5 This is a schematic diagram of attribute video near-layer not occupying CU in an embodiment of this application;

[0017] Figure 6 These are the performance test results of the efficient point cloud projection video encoding method in the embodiments of this application;

[0018] Figure 7 This is an overview diagram of the V-PCC encoding structure and model embedding in the embodiments of this application;

[0019] Figure 8 This is a flowchart illustrating the efficient point cloud projection video encoding method based on VVC in Embodiment 3 of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, features, and advantages of this application clearer and to enable those skilled in the art to better understand the technical solutions of this application, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments.

[0021] Example 1:

[0022] This embodiment includes an efficient point cloud projection video encoding method based on VVC, such as... Figure 1 As shown, it includes:

[0023] S1. Input the point cloud projection sequence into the encoder TMC to obtain the partitioning pattern distribution of geometric video CU and attribute video CU.

[0024] Specifically, the encoder TMC encodes 2D point clouds to generate 2D projected videos: occupancy video, geometric video, and attribute video. The partitioning patterns of the geometric video CU and attribute video CU are obtained by statistically analyzing the frequency and conditional probability of each partitioning type based on the occupancy type and CU size.

[0025] The occupancy types include: fully occupied, partially occupied, and unoccupied; the partitioning types include: QT, BT_H, BT_V, TT_H, and TT_V, representing quadtree partitioning, horizontal binary tree partitioning, vertical binary tree partitioning, horizontal ternary tree partitioning, and vertical ternary tree partitioning, respectively. For example... Figure 3 , Figure 4 , Figure 5 The figures shown are visualizations of the distribution patterns of fully occupied CUs, partially occupied CUs, and unoccupied CUs, respectively.

[0026] Furthermore, differential feature extraction is performed on geometric videos and attribute videos:

[0027] For geometric video scenes, MobileNetV3_small_with_feat is used. The brightness of the geometric image is input, and the same size occupancy map is cropped as the input of the model channel. The occupancy map can effectively reflect the useful pixel distribution of the geometric video. In addition, according to the point cloud projection characteristics, the near and far layers of the geometric video use the same backbone network but different feature selections. Texture statistics are performed on the near layer, and optical flow features are additionally used on the basis of texture statistics in the far layer.

[0028] For attribute video scenarios, an enhanced MobileNetV3_large_with_feat is used. The attribute image is input, and occupancy maps and geometric maps of the same size are cropped to assist in attribute map prediction. Simultaneously, when constructing the dataset for attribute videos, geometric images and their features are added to assist in attribute CU partitioning. The pixel variance of the geometric video and the partitioning of attribute CUs are deeply correlated. The near and far layers of the attribute video use the same backbone network but different feature selections. Texture statistics are performed on the near layers, while optical flow features are additionally used on the far layers in addition to texture statistics.

[0029] The enhanced MobileNetV3_large_with_feat adds an SE module and an FPN branch to the standard MobileNetV3_large_with_feat model to enhance multi-scale representation. The structural diagrams of the neural network graphs MobileNetV3_small_with_feat for geometric videos and MobileNetV3_large_with_feat for attribute videos are shown below. Figure 2 As shown.

[0030] S2. Based on the partitioning pattern distribution, construct a dedicated dataset for each distribution pattern.

[0031] Specifically, the construction process is carried out according to video type × level × partition distribution similarity × CU size category; the CU size category includes: narrow block, square block, and multi-size block; before constructing the hierarchical dataset, narrow blocks are augmented with one or more of the following: proportional scaling, directional filling, and low-intensity affine; square blocks are augmented with one or more of the following: cropping, rotation, scaling, and lighting perturbation; the same mini-batch maintains the same shape to reduce interpolation distortion.

[0032] S3. Fine-tune the dedicated dataset and extract multimodal features, and calculate the partition probability and pattern probability.

[0033] Specifically, the fine-tuning and multimodal feature extraction mentioned above apply to the geometric video CU-specific dataset and the attribute video CU-specific dataset.

[0034] For geometric video CUs, this includes: constructing a two-channel patch x for near-layer geometric video CUs. img =[Y g ,O]∈R {2×H×W} Extract the scalar feature x of the two-channel patch. feat Among them, Y g The luminance component represents the geometric video, and O represents the occupancy map, which is a binarized (or grayscale) auxiliary image used in the V-PCC coding framework to mark whether point cloud data exists on the two-dimensional projection plane.

[0035] For attribute video CUs, this includes: constructing a three-channel patch x for near-layer attribute video CUs. img =[Y a ,Y g [,O], extract the scalar features of the three-channel patch. Where, Y a This represents the luminance component of the attribute video; fine-tuning for MobileNetV3-large includes alignment and channel-independent normalization. Near-layer layers use texture features from the attribute video. Since the geometric video extracts variance features and pixel gradient variance to assist the network in learning attribute CUs, it helps the network better extract and learn features. Far-layer layers use near-layer pixels as a reference, so optical flow features are used to help the network learn CU partitioning. This results in a three-channel image patch that fuses texture, depth, and occupancy state, which is then fed into the network to improve the reliability of CU partitioning decisions.

[0036] The calculation of the partition probability and pattern probability of the dataset includes: for channel patch x img Global average pooling is used to obtain x, and scalar features x are then processed. feat Linear projection is performed to obtain f, and the two are concatenated to obtain the multimodal concatenation vector c=[x;f]. In this embodiment, global average pooling is implemented based on the MobileNetV3 (inverse residual + ReLU / HS) model. Specifically, the geometric video CU uses MobileNetV3-small, and the attribute video CU uses MobileNetV3_large. A lightweight dual-head network is used to perform inference and gating on the multimodal concatenation vector c. The main head outputs the splitting probability p_split, indicating whether to continue splitting, and the sub-head outputs the mode probability p_mode for the specific mode.

[0037] S4. Set a threshold for the division probability, distinguish between the division probability and the pattern probability, and control the candidate pool directly based on the discrimination result.

[0038] Specifically, the determination is based on thresholds set according to the CU size and layer. For CU size: if the current geometric video CU is located in an even-numbered layer and min(w,h)≤8, then the baseline process is rolled back, where w and h represent the width and height of the CU, respectively; if it is located in an odd-numbered layer and min(w,h)<32, the process is also rolled back. The baseline process means adding all partitioning modes and entering the candidate mode control pool. Odd-numbered layers correspond to far layers, and even-numbered layers correspond to near layers.

[0039] For hierarchical threshold setting: set a main threshold τ_main, a secondary threshold τ_sub, and a minimum threshold τ_0. Classify the output probability of the video CU mode and determine whether the partition probability is higher than τ_sub. If so, send the corresponding partition mode to the candidate mode pool; if there is no corresponding mode, send the mode with partition probability between τ_sub and τ_main to the mode candidate pool; if there is still no corresponding mode, send the mode with partition probability between τ_0 and τ_sub to the mode candidate pool; if there is still no corresponding mode, that is, all mode probabilities are lower than the minimum threshold τ_0, send all partition modes to the mode candidate pool, just like the original normal encoding.

[0040] Furthermore, the confidence levels of geometric videos and attribute videos differ. The probability thresholds for geometric videos are set as τ_main_geo, τ_sub_geo, and τ_0_geo, respectively, while the probability thresholds for attribute videos are set as τ_main_attr, τ_sub_attr, and τ_0_attr, respectively. This step avoids calculating and comparing the rate-distortion cost (RD cost) for each mode individually and avoids the "push-then-clear" approach. Because the confidence levels of geometric videos and attribute videos differ, the thresholds for embedding them into the encoder also differ.

[0041] Furthermore, if the width or height of the partitioned blocks CU in the far layer is almost all greater than or equal to 32, the threshold and τ_main are adaptively increased to enhance the early stopping effect.

[0042] S5. Determine the confidence level of the partitioning pattern, and selectively write the pattern into the candidate pool according to the determination result to complete the encoding.

[0043] Specifically, for undetermined partitioning patterns, the push of partitioning patterns is controlled by flag bits (flag_att / flag0-flag4). After pushing BT_V / BT_H, the DID_VERT_SPLIT / DID_HORZ_SPLIT status flags are maintained synchronously to ensure compatibility with subsequent processes. For determined partitioning patterns, after CU encoding is completed, the compressed bit stream is output.

[0044] The confidence level of the partitioning pattern is determined, and the determination logic is as follows:

[0045] Set the primary confidence threshold TH_MAIN and the secondary confidence threshold TH_SUB;

[0046] High confidence no split: If 1-p_split≥TH_MAIN, set flag_att=false and all sub-flags are false, and do not push any split candidates.

[0047] High-confidence splitting: If p_split≥TH_MAIN and max(subProb)≥TH_SUB, only the corresponding single splitting mode is pushed; max(subProb) represents the probability of the mode with the highest probability among the five splitting modes. If it is greater than the threshold, the mode is pushed; Since the learning rates of the five modes are different, some may have high accuracy and some may have low accuracy. Therefore, the five splitting modes have corresponding thresholds to ensure the encoding quality.

[0048] Low confidence rollback: If the conditions for no partitioning or partitioning with high confidence are not met, all partitioning modes remain valid, the flags are not changed, and the baseline process is entered.

[0049] Furthermore, the performance of the efficient point cloud projection video coding method based on VVC in this embodiment was tested. The test conditions were full-frame configuration, and the point cloud sequence dataset used for the test is shown in [link to test]. Figure 5 In the picture, The formula for saving encoding time is:

[0050]

[0051] in, Indicates the video type, including geometric video. With attribute video , This indicates that the design method of this application is in coding. The encoding time of video-like content This indicates that the original encoder TMC2-v18.0 is paired with... The time required for video encoding.

[0052] The coding loss is measured using the Bjontegaard Delta-Rate (BDBR), a commonly used metric in video coding. The test results are as follows: Figure 5 As shown in the graph, the time complexity change and encoding loss after the point cloud fast encoding unit partitioning decision can be seen from the graph. The method designed in this application can effectively shorten the encoding time, reduce the encoding complexity, and has almost no encoding loss. The effect of geometric video is actually increased, while attribute video has almost no loss.

[0053] Example 2:

[0054] This embodiment includes an efficient point cloud projection video encoding method based on VVC, such as... Figure 7As shown, the process includes: packaging patches generated from the original point cloud, and converting the point cloud into multimodal data such as geometry, attributes, and occupancy patch information. Geometric and attribute images are generated, and the data is smoothed and optimized, followed by video compression and auxiliary patch information compression. In the image inpainting stage, the "model embedding location" in the diagram represents the encoder position, completing the logic of point cloud projection sequence → pattern distribution division → dedicated dataset → probability calculation → candidate pool control designed in this application. In the encoding and transmission stage, the multimodal data processed by the model is compressed and encoded into a video stream, then integrated into a compressed bitstream by a multiplexer. The compressed bitstream is split by a reverse multiplexer, and the video stream, occupancy map, and auxiliary patch information are decompressed separately. The decompressed data undergoes geometric reconstruction, smoothing, and attribute reconstruction sequentially, ultimately outputting decompressed point cloud data.

[0055] The method designed in this application (partitioning mode, dataset construction, probability calculation, etc.) corresponds to the process from "model embedding" to "image filling" in the flowchart as the core logic of the encoder, which is responsible for converting the point cloud projection sequence into a partitioning / compression mode of multimodal data.

[0056] Example 3:

[0057] This embodiment includes an efficient point cloud projection video encoding method based on VVC, such as... Figure 8 As shown, the difference from Embodiment 1 is that, in this embodiment, an early termination mechanism is also included between S1 and S2. After the point cloud projection sequence is input into the encoder, it is determined whether the geometric video CU and the attribute video CU are blank or CUs of a specific size. Blank or CUs of a specific size are terminated early, and the standard encoding process of V-PCC is directly entered. For geometric video CUs, sizes of 32×16, 16×32, and 16×16 require early termination; for attribute video CUs, sizes other than 32×32, 128×128, and 64×64 require early termination.

[0058] Finally, it should be noted that the above description only depicts some embodiments of this application. For those skilled in the art, various changes, modifications, substitutions, and variations can be conceived of these embodiments without departing from the principles and spirit of this application. The scope of protection of this application is defined by the appended claims and their equivalents, and all the above-mentioned behaviors should be covered within the scope of protection of this application.

Claims

1. A method for efficient point cloud projection video coding based on VVC, characterized in that, The method comprises the following steps: S1, inputting a point cloud projection sequence into an encoder to obtain a division mode distribution of a geometric video CU and an attribute video CU; S2, constructing a special data set for each distribution mode based on the division mode distribution; S3, fine-tuning and multi-modal feature extraction on the special data set to calculate a division probability and a mode probability; S4, setting a division probability threshold to distinguish the division probability and the mode probability, and directly controlling a candidate pool according to the distinguishing result; S5, distinguishing the confidence of the division mode, and selectively writing the mode into the candidate pool according to the distinguishing result to complete encoding.

2. The VVC-based efficient point cloud projection video coding method of claim 1, wherein, The division mode distribution of the geometric video CU and the attribute video CU is obtained according to the occupancy type and the CU size, and the frequency and the conditional probability of each division type are obtained by statistics; The occupancy type includes full occupancy, partial occupancy, and non-occupancy; the division type includes QT, BT_H, BT_V, TT_H, and TT_V, which respectively represent quad-tree division, horizontal binary-tree division, vertical binary-tree division, horizontal ternary-tree division, and vertical ternary-tree division.

3. The VVC-based efficient point cloud projection video coding method of claim 1, wherein, The division mode distribution of the geometric video CU and the attribute video CU further comprises differential feature extraction on the geometric video and the attribute video; For the geometric video scene, MobileNetV3_small_with_feat is adopted to input the luminance of the geometric picture and cut the same size of the occupancy graph as the model channel input; the near layer and the far layer of the geometric video adopt the same backbone network but different feature selection, the near layer is subjected to texture statistics, and the far layer is additionally subjected to optical flow features on the basis of the texture statistics; For the attribute video scene, an enhanced MobileNetV3_large_with_feat is adopted to input the attribute picture and cut the same size of the occupancy graph and the geometric graph as the auxiliary attribute graph prediction input; the near layer and the far layer of the attribute video adopt the same backbone network but different feature selection, the near layer is subjected to texture statistics, and the far layer is additionally subjected to optical flow features on the basis of the texture statistics; the enhanced MobileNetV3_large_with_feat adds an SE module and an FPN branch to the standard MobileNetV3_large_with_feat model to enhance the multi-scale representation.

4. The VVC-based efficient point cloud projection video coding method of claim 1, wherein, The special data set for each distribution mode is constructed in the process of video type x level x division distribution similarity x CU size category; The CU size category includes narrow blocks, square blocks, and multi-size blocks; before constructing the hierarchical data set, the narrow blocks are subjected to one or more of equal scaling, directional padding, and low-intensity affine for data enhancement, and the square blocks are subjected to one or more of cutting, rotating, scaling, and illumination disturbance for enhancement; the same mini-batch maintains the same shape to reduce interpolation distortion.

5. The VVC-based efficient point cloud projection video coding method of claim 1, wherein, The fine-tuning and multi-modal feature extraction are for the geometric video CU special data set and the attribute video CU special data set; For a geometry video CU, comprising: constructing a two-channel patch x img =[Y g , O] of the geometry video near-layer CU, extracting scalar features x feat of the two-channel patch; wherein Y g represents a luma component of the geometry video, and O represents an occupancy map, i.e., a binary auxiliary image used to mark whether there is point cloud data on a two-dimensional projection plane in a V-PCC coding framework; For attribute video CU, comprising: constructing a three-channel patch x of an attribute video near-layer CU img =[Y a ,Y g ,O] and extracting scalar features of the three-channel patch; wherein Y a represents a luminance component of the attribute video; and the fine-tuning of the MobileNetV3-large comprises: alignment and channel-independent normalization.

6. The VVC-based efficient point cloud projection video coding method of claim 1, wherein, The calculating the split probability and the mode probability of the data set comprises: performing channel patching on x img Global average pooling is performed to obtain x, and a scalar feature x feat Linear projection is performed to obtain f, and the two are spliced to obtain a multi-modal splicing vector c=[x;f]; a lightweight double-head network is used to infer and gate the multi-modal splicing vector c, the main head outputs the split probability p_split of whether to continue to split, and the sub-head outputs the mode probability p_mode of the specific mode.

7. The VVC-based efficient point cloud projection video coding method of claim 1, wherein, The direct control of the candidate pool according to the distinguishing result is distinguished according to the CU size and the level threshold; For CU size: if the current geometry video CU is located in even layer and min(w, h)≤8, fallback to the baseline process; If located in odd layer and min(w, h)<32, also fallback to the baseline process; The baseline process means adding all the partition modes and entering the candidate mode control pool; Odd layer corresponds to the far layer, and even layer corresponds to the near layer; For layer threshold setting: set the main threshold τ_main, the auxiliary threshold τ_sub and the minimum threshold τ_0, and classify the output probability of the video CU mode, judge whether the partition probability is higher than τ_sub, if yes, send the corresponding partition mode into the candidate mode pool; if there is no corresponding mode, send the mode with partition probability between τ_sub and τ_main into the mode candidate pool; if there is still no corresponding mode, send the mode with partition probability between τ_0 and τ_sub into the mode candidate pool; if all mode probabilities are lower than the minimum threshold τ_0, send all partition modes into the mode candidate pool; the confidence of geometry video and attribute video is different.

8. The VVC-based efficient point cloud projection video coding method of claim 7, wherein, If the width or height of the partition block CU of the far layer is all greater than or equal to 32, adaptively increase τ_main, τ_sub and τ_0.

9. The VVC-based efficient point cloud projection video coding method of claim 1, wherein, The confidence of the partition mode is judged, and the judgment logic is as follows: Set the confidence main threshold TH_MAIN and the confidence auxiliary threshold TH_SUB; High confidence non-partition: if 1-p_split≥TH_MAIN, set the flag bit flag_att=false and each sub-flag is false, do not push into any partition candidate, p_split represents the partition probability; High confidence partition: if p_split≥TH_MAIN and max(subProb)≥TH_SUB, only push the corresponding single partition mode; max(subProb) represents the probability of the mode with the maximum probability of the five partition modes, if it is greater than the threshold, push the mode; Low confidence fallback: if the high confidence non-partition or high confidence partition condition is not met, keep each partition mode valid, do not change the flag bit flags, and enter the baseline process.

10. The VVC-based efficient point cloud projection video coding method of claim 1, wherein, An early termination mechanism is also included between S1 and S2: after the point cloud projection sequence is input into the encoder, judge whether the geometry video near layer and the attribute video near layer are blank or specific size CU, early terminate the blank or specific size CU, and directly enter the standard encoding process of V-PCC; The specific size, for geometry video CU, 32×16, 16×32, 16×16 size needs early termination; for attribute video CU, non-32×32, 128×128, 64×64 size needs early termination.

Images