Lightweight multi-source information combined V-PCC inter-frame mode rapid selection method

Abstract

The invention relates to a lightweight multi-source information combined V-PCC inter-frame mode rapid selection method, which comprises the following steps that: geometric videos and attribute videos generated by a V-PCC encoder for a target point cloud frame are obtained, each type of video comprises a near layer and a far layer, and four encoding layers are formed in total; respectively constructing a binary classification model and a multi-classification model for coding units CU of different scales in each coding layer; predicting the probability that a coding unit CU in a coding layer belongs to the MERGESKIP mode by using a binary classification model, skipping other mode detection if the probability value is greater than a first set threshold, and taking the MERGESKIP mode as a final mode of the coding unit CU; otherwise, utilizing a multi-classification model to carry out probability prediction on other inter-frame modes of a coding unit CU in the coding layer, rejecting the inter-frame modes of which the probabilities are lower than a second set threshold value according to a probability prediction result, and reserving the inter-frame modes of which the probabilities are higher than the second set threshold value to construct a candidate inter-frame mode set; and submitting the candidate inter-frame mode set to a V-PCC encoder for mode detection, and determining a final mode of a coding unit (CU) in the candidate inter-frame mode set by the V-PCC encoder. According to the method, the rationality of mode selection can be effectively guaranteed, the coding quality loss caused by rapid decision is avoided, the balance of efficiency and quality is realized in V-PCC inter-frame coding, and the requirements of scenes such as immersive multimedia and three-dimensional vision on efficient and high-quality V-PCC coding are met.

Description

Technical Field

[0001] This invention belongs to the field of video coding technology, and in particular relates to a lightweight method for fast selection of V-PCC inter-frame modes based on the combination of multi-source information. Background Technology

[0002] With the rapid development of immersive multimedia (such as virtual reality and augmented reality) and 3D vision applications (such as 3D reconstruction and autonomous driving environmental perception), the efficient transmission and storage of large-scale 3D point cloud data has become a key industry requirement. Video-based Point Cloud Compression (V-PCC), as a core technology to meet this requirement, has become an important support for achieving efficient point cloud data processing. Existing V-PCC standards (such as the MPEG-G-PCC video base scheme) are usually based on the HEVC / VVC coding framework. They compress 3D point clouds by projecting them into 2D geometric video and attribute video to achieve a balance between compression efficiency and reconstructed visual quality. Among these, the inter-frame prediction mode selection of the Coding Unit (CU) directly determines the compression performance of V-PCC—the rationality of the mode selection not only affects the compression rate of point cloud data, but also the computational efficiency and real-time performance of the encoding process.

[0003] Currently, optimization research on CU inter-frame prediction mode selection in the V-PCC field mainly falls into two categories: one is rule-based and heuristic methods, which are mainly designed for intra-frame prediction scenarios. For example, they can extract local features, analyze the mode selection rules of adjacent blocks, optimize decision logic by combining edge information and texture complexity, or use adaptive mechanisms to dynamically adjust mode priorities. However, these methods rely too much on local statistical features and preset rules, making it difficult to capture the complex spatiotemporal dependencies of 2D video frames after 3D point cloud projection. They also lack robustness in dynamically changing point cloud scenarios (such as moving target point clouds), which can easily lead to mode selection bias. The second approach is based on machine learning. This type of method learns mode selection strategies through neural networks. For example, it uses convolutional neural networks (CNNs) to optimize intra-frame mode decisions, employs random forest algorithms to achieve early termination of inter-frame prediction, optimizes inter-frame prediction filters through lightweight neural networks, adjusts quantization parameters (QP) based on learning mechanisms, or introduces sparse convolution and multi-scale 3D feature learning to achieve dynamic point cloud geometric compression. However, existing machine learning methods are mostly designed for traditional two-dimensional video coding scenarios and are not fully adapted to the three-dimensional point cloud projection characteristics of V-PCC. Especially in the combination of multi-scale CUs (such as CUs of different sizes in geometric near / far layers and attribute near / far layers) and complex inter-frame modes (such as INTER_ME, AFFINE, MERGE_GEO, INTRA, MERGE_SKIP), the computational complexity is too high and the real-time performance is insufficient, making it difficult to meet the needs of immersive multimedia and other applications for efficient real-time V-PCC coding.

[0004] In summary, existing V-PCC inter-frame mode selection techniques have significant limitations: neither rule-based nor machine learning-based methods are specifically optimized for the inherent characteristics of V-PCC's 3D point cloud projection and the differentiated requirements of multiple coding layers (geometric near layer, geometric far layer, attribute near layer, attribute far layer). This results in an inability to balance compression efficiency, visual quality, and real-time coding performance in multi-scale CU processing and complex mode decisions, creating a technical bottleneck for the efficient implementation of real-time V-PCC applications. Therefore, a novel fast inter-frame mode selection method combining deep learning, threshold selection mechanisms, and cross-layer information fusion logic is urgently needed to reduce the computational complexity of V-PCC inter-frame coding and significantly improve coding speed while ensuring point cloud reconstruction quality. Summary of the Invention

[0005] To address the problems existing in the background art, this invention provides a lightweight, multi-source information-based V-PCC inter-frame mode fast selection method, comprising:

[0006] S1: Obtain the geometric video and attribute video generated by the V-PCC encoder on the target point cloud frame. Each type of video includes near layer and far layer, forming a total of four coding layers.

[0007] S2: Construct binary classification and multi-class classification models for coding units (CUs) of different scales in each coding layer;

[0008] S3: Use a binary classification model to predict the probability that the coding unit CU in the coding layer belongs to the MERGE_SKIP mode. If the probability value is greater than the first set threshold, skip the detection of other modes and take the MERGE_SKIP mode as the final mode of the coding unit CU; otherwise, proceed to step S4.

[0009] S4: Use a multi-classification model to predict the probability of other inter-frame modes of coding units (CUs) in the coding layer. Based on the probability prediction results, remove inter-frame modes with probabilities lower than the second set threshold and retain inter-frame modes with probabilities higher than the second set threshold to construct a candidate inter-frame mode set.

[0010] S5: Submit the candidate inter-frame mode set to the V-PCC encoder for mode detection, and the V-PCC encoder determines the final mode of the coding unit (CU) from the candidate inter-frame mode set.

[0011] The present invention has at least the following beneficial effects

[0012] This invention significantly reduces unnecessary mode detection calculations in V-PCC inter-frame coding by prioritizing high-probability mainstream modes and accurately filtering low-probability invalid modes, thereby significantly improving coding efficiency and alleviating the real-time bottleneck of traditional methods in multi-scale CUs and complex modes. On the other hand, relying on differentiated model adaptation for the four coding layers and combined with the encoder's final mode confirmation mechanism, it can effectively ensure the rationality of mode selection and avoid coding quality loss caused by rapid decision-making. Thus, it achieves a balance between efficiency and quality in V-PCC inter-frame coding, meeting the needs of immersive multimedia, 3D vision and other scenarios for efficient and high-quality V-PCC coding. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0014] Figure 2 This is a statistical diagram of the prediction modes of different coding units in the geometric near layer of different point cloud sequences in the V-PCC encoder in an embodiment of the present invention.

[0015] Figure 3 This is a statistical diagram of the prediction modes of different coding units in the attribute near layer of different point cloud sequences in the V-PCC encoder in an embodiment of the present invention.

[0016] Figure 4 This is a statistical diagram of the prediction modes of different coding units in the geometric far layer of different point cloud sequences in the V-PCC encoder in an embodiment of the present invention.

[0017] Figure 5 This is a statistical diagram of the prediction modes of different coding units in the attribute far layer of different point cloud sequences in the V-PCC encoder in an embodiment of the present invention.

[0018] Figure 6 This is a schematic diagram of the binary classification model and multi-class classification model structure constructed for encoding units (CUs) with sizes of 32×32, 64×64, 128×128, 16×16 and 8×8 in the geometric near layer and attribute near layer of this invention.

[0019] Figure 7 This is a schematic diagram of the binary classification model and multi-class classification model structure constructed for the coding units (CUs) with sizes of 32×32, 64×64 and 128×128 in the geometric far layer and attribute far layer of this invention.

[0020] Figure 8 This is a schematic diagram of the structure of the convRes module in an embodiment of the present invention. Detailed Implementation

[0021] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0022] In this embodiment, to better study the fast inter-frame mode decision-making in V-PCC, this invention uses the reference software TMC2-v18.0 of the V-PCC point cloud coding standard to statistically analyze the distribution characteristics of prediction modes during the inter-frame coding stage. The test sequences are selected from seven dynamic point cloud sequences defined in the V-PCC Common Test Conditions (CTC), and coding experiments are conducted under a low-delay configuration. The quantization parameters (QP) used during coding are consistent with the standard settings, and all tests are performed on the inter-frame prediction stage. V-PCC inter-frame coding mainly includes five prediction modes: INTER_ME, AFFINE, MERGE_GEO, INTRA, and MERGE_SKIP. To obtain the statistical characteristics of mode distribution, this invention encodes 10 frames for each sequence and calculates the proportion of each prediction mode in the coding unit. The results are plotted as a mode distribution diagram, as shown below. Figures 2-5As shown, the geometric video and attribute video generated after projection processing by the V-PCC encoder are shown. Each type of video includes near-layer and far-layer. Figures 2-5 These figures represent the inter-frame mode prediction proportions of each coding unit (CU) in the geometrically near layer, attribute-near layer, geometrically far layer, and attribute-far layer, respectively. Figures 2-5 It can be observed that during inter-frame coding, the coding unit (CU) tends to choose the MERGE_SKIP coding mode in most cases, which dominates the overall distribution. This phenomenon indicates that point cloud videos possess strong spatial structural stability, leading to the frequent occurrence of the MERGE_SKIP pattern during encoding. Furthermore, we observed that in the V-PCC encoder, the proportion of coding units (CUs) of different sizes varies across different coding layers. In the geometrically near and attribute-near layers, the majority of CUs are combinations of 8×8, 16×16, 32×32, 64×64, and 128×128 sizes. For example, an 8×16 CU can be considered as a combination of two 8×8 CUs. Smaller CUs, such as 4×4 or 4×8, are less numerous. In the geometrically far and attribute-far layers, the majority of CUs are combinations of 32×32, 64×64, and 128×128 sizes, while other sizes are less common. Most CUs across all four coding layers are predicted as the MERGE_SKIP pattern. Therefore, this embodiment, considering the size distribution of CUs in different coding layers, employs a pattern prediction method to prioritize determining whether an image patch belongs to the "MERGE_SKIP" category. Specifically, the model first performs a binary classification prediction on the "MERGE_SKIP" pattern. If the prediction confidence is higher than a set threshold, the encoding process for the current block is terminated directly to avoid unnecessary redundant calculations. If the prediction result indicates that encoding should continue, the model proceeds to the next stage of pattern classification prediction to further guide subsequent encoding decision-making processes.

[0023] Please see Figure 1 This invention provides a lightweight, multi-source information-based fast V-PCC inter-frame mode selection method, comprising:

[0024] S1: Obtain the geometric video and attribute video generated by the V-PCC encoder on the target point cloud frame. Each type of video includes near layer and far layer, forming a total of four coding layers.

[0025] In this embodiment, after receiving a 3D point cloud frame, the V-PCC encoder first separates the two types of core information contained in the point cloud: one type is "geometric information" describing the spatial location and structural morphology of the point cloud, and the other type is "attribute information" characterizing the surface features of the point cloud (such as color and texture), laying the foundation for subsequent processing. Then, based on the V-PCC projection mechanism (following the HEVC / VVC framework), the separated geometric information and attribute information are projected onto a two-dimensional plane, respectively, and transformed into "geometric video" (carrying only geometric information) and "attribute video" (carrying only attribute information) that can be processed according to two-dimensional video coding logic, realizing the dimensionality reduction conversion from 3D point cloud to 2D video. Finally, for the geometric video and attribute video generated after projection, the encoder performs layering according to the "point cloud projection depth": the region that is close to the two-dimensional plane during projection, with dense point cloud data and rich details is divided into the "near layer", and the region that is far from the projection distance and has relatively sparse point cloud data is divided into the "far layer". Thus, the geometric video generates "geometric near layer" and "geometric far layer", and the attribute video generates "attribute near layer" and "attribute far layer", ultimately forming 4 coding layers.

[0026] Please see Figure 6 , Figure 7 and Figure 8 S2: Construct binary classification and multi-class classification models for coding units (CUs) of different scales in each coding layer;

[0027] Preferably, the binary classification model and the multi-class classification model are constructed using a lightweight convolutional neural network.

[0028] In this embodiment, in the geometric near-layer coding layer and the attribute near-layer coding layer, binary classification models and multi-classification models are constructed for coding units (CUs) with sizes of 8×8, 16×16, 32×32, 64×64, and 128×128, respectively. The binary and multi-classification models for coding units (CUs) with sizes of 32×32, 64×64, and 128×128 include: a first convRes module, a first max-pooling layer, a second convRes module, a second max-pooling layer, a Flatten layer, a fully connected neural network, and a softmax activation function, all cascaded sequentially. The binary and multi-classification models for coding units (CUs) with sizes of 8×8 and 16×16 include: a first convRes module, a second convRes module, a Flatten layer, a fully connected neural network, and a softmax activation function, all cascaded sequentially.

[0029] In this embodiment, binary classification models and multi-class classification models are constructed for coding units (CUs) with sizes of 32×32, 64×64, and 128×128 in the geometric far-end coding layer and the attribute far-end coding layer, respectively. The binary classification model and multi-class classification model for coding units (CUs) with sizes of 32×32, 64×64, and 128×128 include: a first convRes module, a first max pooling layer, a second convRes module, a second max pooling layer, a Flatten layer, a fully connected neural network, and a softmax activation function, which are cascaded in sequence.

[0030] In this embodiment, binary and multi-class classification models based on lightweight convolutional neural networks are constructed for coding units (CUs) of different scales in the four coding layers of V-PCC: geometric near layer, geometric far layer, attribute near layer, and attribute far layer (e.g., 8×8, 16×16, 32×32, 64×64, and 128×128 for geometric near layer and attribute near layer, and 32×32, 64×64, and 128×128 for geometric far layer and attribute far layer). The model structure is optimized according to the differences in CU size. This achieves accurate adaptation of the model to the characteristics of each coding layer and the distribution law of CUs at different scales. At the same time, the lightweight design ensures the accuracy of feature capture while controlling the computational complexity. This provides reliable model support for the subsequent steps of quickly predicting the MERGE_SKIP mode and efficiently screening candidate inter-frame modes, laying the foundation for reducing the computational load of V-PCC inter-frame coding from the core model level.

[0031] In this embodiment, in the geometric near-layer coding layer and the attribute near-layer coding layer, for coding units (CUs) other than those with sizes of 8×8, 16×16, 32×32, 64×64, and 128×128, they are divided into multiple CUs with sizes of 8×8, 16×16, 32×32, 64×64, or 128×128. The binary classification model and multi-class classification model of the corresponding size are used to predict the pattern of the multiple CUs after division. The prediction result of the coding unit CU is obtained by averaging the prediction results of all CUs after division. For coding units CUs other than those with sizes of 8×8, 16×16, 32×32, 64×64, and 128×128, if they cannot be formed by combining sizes of 8×8, 16×16, 32×32, 64×64, or 128×128, the coding unit CU is directly sent to the V-PCC encoder for pattern detection.

[0032] In this embodiment, for CUs of non-standard sizes such as 8×8 and 16×16, flexible processing—either dividing them into standard-sized CUs for prediction and averaging or directly sending them to the encoder—ensures coverage of CUs of all scales. This not only guarantees the integrity of the prediction logic but also lays an efficient foundation for the accurate construction of subsequent candidate mode sets through the core logic of "prioritizing high-probability modes," achieving a balance between coding efficiency and prediction reliability. For example, for an 8×16 coding unit CU, it can be divided into two 8×8 CUs for probability prediction, and then the average of the two CUs is calculated to obtain the probability prediction result of the coding unit CU. For CUs that cannot be formed by combining basic sizes such as 8×8 and 16×16 (such as 4×4, 4×8, etc.), which account for a small proportion, to avoid increasing model complexity and training costs, and to efficiently handle special cases while controlling overall computational redundancy, this embodiment allows the V-PCC encoder to process them directly according to the conventional coding process.

[0033] In this embodiment, in the geometric far-layer coding layer and the attribute far-layer coding layer, for coding units (CUs) other than those with sizes of 32×32, 64×64, and 128×128, they are divided into multiple CUs with sizes of 32×32, 64×64, or 128×128. The binary classification model and multi-class classification model of the corresponding size are used to predict the pattern of the multiple CUs after division. The prediction result of the coding unit CU is obtained by averaging the prediction results of all CUs after division. For coding units (CUs) other than those with sizes of 32×32, 64×64, and 128×128, if they cannot be formed by combining sizes of 32×32, 64×64, or 128×128, the coding unit CU is directly sent to the V-PCC encoder for pattern detection.

[0034] In this embodiment, for CUs of non-standard sizes such as 32×32 and 64×64, flexible processing—either dividing them into standard-sized CUs for prediction and averaging or directly sending them to the encoder—ensures coverage of CUs of all scales. This not only guarantees the integrity of the prediction logic but also lays an efficient foundation for the accurate construction of the subsequent candidate mode set through the core logic of "prioritizing high-probability modes," achieving a balance between coding efficiency and prediction reliability. For example, for a 32×64 coding unit CU, it can be divided into two 32×32 CUs for probability prediction, and then the average of the two CUs is calculated to obtain the probability prediction result of the coding unit CU. For CUs that cannot be formed by combining basic sizes such as 32×32 and 64×64 (such as 8×32, 4×64, etc.), which account for a small proportion, to avoid increasing model complexity and training costs, and to efficiently handle special cases while controlling overall computational redundancy, this embodiment allows the V-PCC encoder to process them directly according to the conventional coding process.

[0035] In this embodiment, the convRes module includes multiple cascaded convRes units, each convRes unit including: a first convolutional layer, a first batch normalization (BN) layer, a first Swish activation function, a second convolutional layer, a second BN layer, a third BN layer, and a second Swish activation function, which are cascaded in sequence; wherein, the input features of the convRes module and the input features of the third BN layer are connected by residuals through 1×1 convolution or directly.

[0036] In this embodiment, the binary classification model and the multi-class classification model play a key role in the V-PCC inter-frame pattern prediction scenario through a structural design of "multi-level cascaded convRes units + residual connections": On the one hand, the convolutional layers, BN layers, and Swish activation functions cascaded sequentially within each convRes unit can efficiently capture the multi-dimensional features of the coding unit (CU) (such as pixels, residuals, optical flow, etc.), the BN layer can accelerate model training and stabilize parameter updates, and the Swish activation function enhances the fitting ability to nonlinear features, achieving accurate extraction from basic features to high-level semantic features; on the other hand... The module effectively alleviates the gradient vanishing problem in deep network training through 1×1 convolution or direct connection residual mechanisms, ensuring that key information is not lost during feature transmission. It is particularly adaptable to the feature differences of different coding layers (geometric near / far layers, attribute near / far layers) and different scale CUs (8×8 to 128×128). This not only ensures the depth and feature extraction accuracy of the lightweight model, but also avoids the computational redundancy caused by complex structures. It provides high-quality feature support for subsequent binary classification prediction of MERGE_SKIP mode and multi-class prediction of other inter-frame modes, laying the foundation for the model to accurately distinguish inter-frame modes.

[0037] In this embodiment, the loss functions used during the training of the binary classification model and the multi-class classification model include:

[0038]

[0039] in, Let C represent the cathodic function, and C represent the number of categories. Indicates the weighting parameter; This represents the probability that the predicted result belongs to category c; This indicates the actual label.

[0040] In this embodiment, the binary classification model and the multi-class classification model are trained using a constructed loss function, and the weight parameters are used to perform the training. To balance the class distribution differences among different inter-frame modes, and address the class imbalance issue where the MERGE_SKIP mode has a significantly higher proportion than other modes in inter-frame coding, weight parameters are used. Higher weights can be assigned to minority class pattern samples to avoid model bias due to majority class samples dominating training, thus ensuring the balance of predictions for various patterns; by leveraging The adjustment factor reduces the loss contribution of easily distinguishable samples, focuses the training on difficult samples, improves the model's discrimination accuracy of boundary samples, and reduces the situation of "non-MERGE_SKIP being misclassified as MERGE_SKIP" in binary classification or "pattern confusion" in multi-class classification; thirdly, by combining the log loss term with the above adjustment mechanism, the model's predicted probability and the fitting degree of the true label are precisely optimized, ensuring that the trained binary classification model can reliably output the MERGE_SKIP pattern probability and the multi-class model can accurately distinguish other inter-frame patterns.

[0041] S3: Use a binary classification model to predict the probability that the coding unit CU in the coding layer belongs to the MERGE_SKIP mode. If the probability value is greater than the first set threshold, skip the detection of other modes and take the MERGE_SKIP mode as the final mode of the coding unit CU; otherwise, proceed to step S4.

[0042] In this embodiment, for the binary classification model corresponding to the geometrically near-layer coding layer, its input features include: the pixels of the coding unit CU, the residual of the coding unit CU, the occupancy map of the coding unit CU, and the optical flow information of the coding unit CU; in the geometrically near-layer coding layer, the pixels of the coding unit CU, the residual of the coding unit CU, the occupancy map of the coding unit CU, and the optical flow information of the coding unit CU are concatenated as the input features of the binary classification model.

[0043] For the binary classification model corresponding to the geometrically far-reaching coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, pixels of corresponding coding unit CU in the geometrically near-reaching coding layer, and MERGE_SKIP pattern probability prediction results of corresponding coding unit CU in the geometrically near-reaching coding layer. Specifically, in the binary classification model, the MERGE_SKIP pattern probability prediction results of corresponding coding unit CU in the geometrically near-reaching coding layer are concatenated with the output features of the Flatten layer and used as the input features of the fully connected neural network. In the geometrically far-reaching coding layer, the pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, and pixels of corresponding coding unit CU in the geometrically near-reaching coding layer are concatenated to obtain the input features of the binary classification model. The flattened output features of the binary classification model, concatenated with the MERGE_SKIP pattern probability prediction results of corresponding coding unit CU in the geometrically near-reaching coding layer, are then input into the fully connected neural network of the binary classification model for subsequent inference.

[0044] For a binary classification model corresponding to an attribute-near coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, pixels of corresponding coding unit CU in the geometrically near coding layer, and MERGE_SKIP pattern probability prediction results of coding unit CU corresponding to the geometrically near coding layer; wherein, in the binary classification model, the MERGE_SKIP pattern probability prediction results of coding unit CU corresponding to the geometrically near coding layer are concatenated with the output features of the Flatten layer and used as the input features of the fully connected neural network;

[0045] In the attribute near-layer coding layer, the pixels of coding unit CU, the residuals of coding unit CU, the occupancy map of coding unit CU, the optical flow information of coding unit CU, and the pixels of the corresponding coding unit CU in the geometric near-layer coding layer are concatenated to obtain the input features of the binary classification model. The output features of the flattened layer of the binary classification model are concatenated with the MERGE_SKIP pattern probability prediction results of the corresponding coding unit CU in the geometric near-layer coding layer and then input into the fully connected neural network of the binary classification model for subsequent inference.

[0046] For the binary classification model corresponding to the attribute far-layer coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, pixels of corresponding coding unit CU in the geometric far-layer coding layer, pixels of corresponding coding unit CU in the attribute near-layer coding layer, and the MERGE_SKIP pattern probability prediction result of the coding unit CU corresponding to the attribute near-layer coding layer; wherein, in the binary classification model, the MERGE_SKIP pattern probability prediction result of the coding unit CU corresponding to the attribute near-layer coding layer is concatenated with the output features of the Flatten layer. The features are then used as input features to the fully connected neural network. In the attribute far-layer coding layer, the pixels of the coding unit CU, the residuals of the coding unit CU, the occupancy map of the coding unit CU, the optical flow information of the coding unit CU, the pixels of the corresponding coding unit CU in the geometric far-layer coding layer, and the pixels of the corresponding coding unit CU in the attribute near-layer coding layer are concatenated to obtain the input features of the binary classification model. The output features of the flattened layer of the binary classification model are concatenated with the MERGE_SKIP pattern probability prediction results of the corresponding coding unit CU in the attribute near-layer coding layer and then input into the fully connected neural network of the binary classification model for subsequent inference.

[0047] In this embodiment, by adapting and constructing a binary classification model for each coding layer, the probability of the coding unit CU belonging to the dominant MERGE_SKIP pattern is prioritized for prediction. If the probability exceeds a first set threshold, the pattern is directly locked and other redundant detections are skipped, which significantly reduces unnecessary pattern judgment computation and improves coding efficiency from the source. At the same time, differentiated multi-source input features are designed for binary classification models of different coding layers (such as integrating pixels from geometrically distant layers with MERGE_SKIP prediction results, and combining attribute layer with geometric layer information), which strengthens feature correlation to improve prediction accuracy and avoids coding quality loss caused by misjudgment.

[0048] S4: Use a multi-classification model to predict the probability of other inter-frame modes of coding units (CUs) in the coding layer. Based on the probability prediction results, remove inter-frame modes with probabilities lower than the second set threshold and retain inter-frame modes with probabilities higher than the second set threshold to construct a candidate inter-frame mode set.

[0049] For the multi-class classification model corresponding to the geometrically near-layer coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, and optical flow information of coding unit CU. In the geometrically near-layer coding layer, the pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, and optical flow information of coding unit CU are concatenated as input features of the multi-class classification model.

[0050] For the multi-class classification model corresponding to the geometrically far-reaching coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, pixels of corresponding coding unit CU in the geometrically near-reaching coding layer, and probability prediction results of other optimal modes of coding unit CU in the geometrically near-reaching coding layer. Specifically, in the multi-class classification model, the probability prediction results of other optimal modes of coding unit CU in the geometrically near-reaching coding layer are concatenated with the output features of the Flatten layer and used as the input features of the fully connected neural network. In the geometrically far-reaching coding layer, pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, and pixels of corresponding coding unit CU in the geometrically near-reaching coding layer are concatenated and used as the input features of the multi-class classification model. After the output features of the Flatten layer of the multi-class classification model are flattened, they are concatenated with the probability prediction results of other optimal modes of coding unit CU in the geometrically near-reaching coding layer and then input into the fully connected neural network of the binary classification model for subsequent inference. The probability prediction results of other optimal modes include: the other modes with the highest probability and their corresponding probabilities.

[0051] For a multi-class classification model corresponding to an attribute-near coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, pixels of corresponding coding unit CU in the geometrically near coding layer, and other optimal mode probability prediction results of coding unit CU corresponding to the geometrically near coding layer; wherein, in the multi-class classification model, the other optimal mode probability prediction results of coding unit CU corresponding to the geometrically near coding layer are concatenated with the output features of the Flatten layer and used as the input features of the fully connected neural network;

[0052] In the attribute-near coding layer, the pixels of the coding unit CU, the residuals of the coding unit CU, the occupancy map of the coding unit CU, the optical flow information of the coding unit CU, and the pixels of the corresponding coding unit CU in the geometrically near coding layer are concatenated as the input features of the multi-classification model. After the output features of the flattened layer of the multi-classification model are concatenated with the probability prediction results of the other optimal modes of the corresponding coding unit CU in the geometrically near coding layer, they are input into the fully connected neural network of the binary classification model for subsequent inference. The probability prediction results of the other optimal modes include: the other modes with the highest probability and their corresponding probabilities.

[0053] For a multi-class classification model corresponding to an attribute far-layer coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, pixels of corresponding coding unit CU in the geometric far-layer coding layer, pixels of corresponding coding unit CU in the attribute near-layer coding layer, and other optimal pattern probability prediction results of coding unit CU corresponding to the attribute near-layer coding layer; among them, in the multi-class classification model, the other optimal pattern probability prediction results of coding unit CU corresponding to the attribute near-layer coding layer are concatenated with the output features of the Flatten layer and used as the input features of the fully connected neural network;

[0054] In the attribute far-layer coding layer, the pixels of coding unit CU, the residuals of coding unit CU, the occupancy map of coding unit CU, the optical flow information of coding unit CU, the pixels of the corresponding coding unit CU in the geometric far-layer coding layer, and the pixels of the corresponding coding unit CU in the attribute near-layer coding layer are concatenated as the input features of the multi-classification model. The output features of the flattened multi-classification model are concatenated with the probability prediction results of the remaining optimal modes of the corresponding coding unit CU in the attribute near-layer coding layer and then input into the fully connected neural network of the binary classification model for subsequent inference. The probability prediction results of the remaining optimal modes include the remaining modes with the highest probability and their corresponding probabilities.

[0055] In this embodiment, differentiated multi-source input features (including pixel, residual, occupancy map, and optical flow information of the coding unit (CU) itself) are designed for the multi-classification models of geometrically near layer, geometrically far layer, attribute near layer, and attribute far layer. These features are combined with cross-layer correlation information (the geometrically far layer incorporates the pixel of the corresponding CU of the geometrically near layer and the probability prediction results of other optimal modes, and the attribute layer incorporates the pixel of the corresponding CU of the geometric layer or attribute near layer and the probability prediction results of other optimal modes). Furthermore, the model input is optimized by concatenating the output features of the Flatten layer with the cross-layer prediction results. This achieves multiple beneficial effects: it allows the multi-classification model of each coding layer to fully capture its own features and cross-layer correlation features, strengthening the ability to distinguish complex inter-frame modes and reducing mode confusion; it provides a reliable basis for the selection of candidate inter-frame mode sets through accurate probability prediction, avoiding low-probability invalid modes from occupying coding resources; and it adapts to the characteristic differences of different coding layers, ensuring the prediction accuracy of multi-scale CUs in non-MERGE_SKIP mode scenarios. Thus, while improving the V-PCC inter-frame coding efficiency, it effectively maintains the stability of coding quality.

[0056] It should be noted that:

[0057] The pixels of a CU originate from the projection processing of the target point cloud frame by the V-PCC encoder. The encoder first projects the geometric and attribute information of the 3D point cloud into 2D geometric video and attribute video, respectively. Then, the two types of video frames are divided into several CUs according to the division rules of coding units (CUs) (such as 8×8, 16×16, etc.). The pixels of each CU correspond to the original pixel values ​​of the region divided in the geometric video / attribute video frame (geometric video pixels reflect spatial location-related features, and attribute video pixels reflect surface features such as color and texture).

[0058] The residual of the CU is calculated and generated by the inter-frame prediction step of the V-PCC encoder. The encoder first predicts the current CU based on the reference frame (adjacent encoded frames) to obtain the "predicted pixel". Then, the original pixel value of the current CU is subtracted from the "predicted pixel", and the difference is the residual of the CU, which is used to characterize the difference between the current CU and the corresponding area of ​​the reference frame.

[0059] The occupancy map of the CU is a binary image synchronously generated by the V-PCC encoder during the point cloud projection stage. During projection, the encoder determines whether each pixel position of the 2D video frame corresponds to a valid point in the 3D point cloud (i.e., whether it is "occupied" by the point cloud) and generates a global occupancy map. Then, according to the CU division rules, the occupancy map information of the corresponding region of each CU is extracted (identifying which pixel positions in the CU have valid point cloud data and which do not), which serves as the spatial validity feature of the CU.

[0060] Optical flow information of CU: generated by RAFT-based optical flow calculation method - In order to utilize the inter-frame spatiotemporal correlation, we estimate the pixel motion trend between the current frame and the reference frame through the RAFT algorithm to obtain a global optical flow field that reflects the direction and amplitude of pixel motion; then for each CU, we extract the set of optical flow vectors in its region as features characterizing the pixel motion state within the CU.

[0061] S5: Submit the candidate inter-frame mode set to the V-PCC encoder for mode detection, and the V-PCC encoder determines the final mode of the coding unit (CU) from the candidate inter-frame mode set.

[0062] Experimental simulation:

[0063] All algorithms proposed in this paper are fast algorithms for V-PCC inter-frame coding. Therefore, all subsequent data operations are performed in the low-latency mode configured in V-PCC. To analyze the feasibility of the algorithms, this paper uses seven dynamic point cloud sequences under the V-PCC Common Test Conditions (CTC) for testing, as shown in Tables 1 and 2:

[0064] Table 1 Comparison of BDBR and encoding time between the method proposed in this invention and the V-PCC benchmark.

[0065]

[0066] Table 1 compares the coding performance of the pattern synthesis algorithm on different point cloud video sequences, with evaluation metrics including BDBR variation and coding time variation. Overall, the BDBR of the geometrically near layer increases by approximately 1.73%, the BDBR of the attribute near layer increases by approximately 1.97%, the BDBR of the geometrically far layer decreases by approximately 2.75%, and the BDBR of the attribute far layer increases by approximately 1.24%. Simultaneously, the coding time of each layer decreases significantly, with the average coding times of the geometrically near layer, attribute near layer, geometrically far layer, and attribute far layer decreasing by 35.05%, 39.11%, 49.29%, and 58.38%, respectively. Specifically, in the loot sequence, the geometrically near layer BDBR increases by only 2.26%, but the coding time decreases by 46.89%; in the soldier sequence, the attribute far layer BDBR decreases by 6.16%, while the coding time decreases by 65.71%.

[0067] Table 2 compares the performance of the proposed method with the V-PCC benchmark.

[0068]

[0069] Table 2 shows the BD-rate performance comparison between the proposed point cloud coding method and the V-PCC benchmark method in geometric and attribute coding. BD-rate is a key indicator for evaluating coding performance, reflecting the code rate change under the same distortion conditions. Overall, the results show that the proposed method has excellent coding efficiency in both geometric and attribute coding. In geometric coding, the average BD-rates of D1 and D2 are −0.05% and −0.03%, respectively, indicating that the geometric code rate can be reduced by an average of approximately 0.03%–0.05% while maintaining geometric reconstruction accuracy. Most test sequences (such as loot, soldier, etc.) achieve significant code rate savings, while a few sequences (such as redandblack, longdress) show only slight increases (<0.15%). In attribute coding, the average BD-rates of Luma, Cb, and Cr components are −0.08%, 0.00%, and −0.01%, respectively. Luma and Cr components show stable code rate savings, while the Cb component is close to the benchmark method, indicating stable overall performance. Most sequences showed a negative BD-rate across all three components, with some sequences showing an increase of no more than 0.26%.

[0070] In summary, by leveraging the differentiated characteristics of the four coding layers (geometric near layer, geometric far layer, attribute near layer, and attribute far layer) of V-PCC, we constructed adaptive lightweight convolutional neural network binary and multi-class classification models for coding units (CUs) of different scales (e.g., 8×8 to 128×128 for geometric near layer / attribute near layer, and 32×32 to 128×128 for geometric far layer / attribute far layer). The binary classification model prioritizes high-probability MERGE_SKIP patterns to reduce redundant detection, while the multi-class model accurately constructs a set of candidate inter-frame patterns. Combined with flexible handling of CUs of special sizes (partitioning for prediction or direct input to the encoder), this significantly improves coding efficiency while ensuring stable coding quality. Experimental data show that the average coding time for each coding layer is significantly reduced. The reduction ranges from 35.05% to 58.38% (e.g., an average reduction of 58.38% for attribute far layers, and a reduction of 65.71% for attribute far layer encoding time in Soldier sequences). The overall BDBR is kept within a reasonable range (an average reduction of 2.75% for geometric far layers, a reduction of 6.16% for attribute far layers in Soldier sequences, and small fluctuations in BDBR for most sequences). Furthermore, by relying on Focal loss to balance pattern category differences, the convRes module to enhance feature extraction, and cross-layer information fusion to improve model adaptability, it effectively solves the problem that existing methods cannot balance V-PCC compression efficiency, visual quality, and real-time performance, meeting the needs of immersive multimedia, 3D vision, and other scenarios for efficient V-PCC encoding.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A lightweight method for fast inter-frame mode selection in V-PCC based on multi-source information fusion, characterized in that, include: S1: Obtain the geometric video and attribute video generated by the V-PCC encoder on the target point cloud frame. Each type of video includes near layer and far layer, forming a total of four coding layers. S2: Construct binary classification and multi-class classification models for coding units (CUs) of different scales in each coding layer; S3: Use a binary classification model to predict the probability that the coding unit CU in the coding layer belongs to the MERGE_SKIP mode. If the probability value is greater than the first set threshold, skip the detection of other modes and take the MERGE_SKIP mode as the final mode of the coding unit CU; otherwise, proceed to step S4. S4: Use a multi-classification model to predict the probability of other inter-frame modes of coding units (CUs) in the coding layer. Based on the probability prediction results, remove inter-frame modes with probabilities lower than the second set threshold and retain inter-frame modes with probabilities higher than the second set threshold to construct a candidate inter-frame mode set. S5: Submit the candidate inter-frame mode set to the V-PCC encoder for mode detection, and the V-PCC encoder determines the final mode of the coding unit (CU) from the candidate inter-frame mode set.

2. The lightweight multi-source information combined V-PCC inter-frame mode fast selection method according to claim 1, characterized in that, The binary classification model and the multi-class classification model are constructed using a lightweight convolutional neural network.

3. The lightweight multi-source information combined V-PCC inter-frame mode fast selection method according to claim 2, characterized in that, In the geometric near-layer encoding layer and the attribute near-layer encoding layer, binary classification models and multi-class classification models are constructed for encoding units (CUs) with sizes of 8×8, 16×16, 32×32, 64×64, and 128×128, respectively. The binary and multi-class classification models for encoding units (CUs) with sizes of 32×32, 64×64, and 128×128 include: a first convRes module, a first max-pooling layer, a second convRes module, a second max-pooling layer, a Flatten layer, a fully connected neural network, and a softmax activation function, all cascaded sequentially. The binary and multi-class classification models for encoding units (CUs) with sizes of 8×8 and 16×16 include: a first convRes module, a second convRes module, a Flatten layer, a fully connected neural network, and a softmax activation function, all cascaded sequentially.

4. The lightweight multi-source information combined V-PCC inter-frame mode fast selection method according to claim 2, characterized in that, In the geometric far-end encoding layer and the attribute far-end encoding layer, binary classification models and multi-class classification models are constructed for encoding units (CUs) with sizes of 32×32, 64×64, and 128×128, respectively. The binary classification model and multi-class classification model for encoding units (CUs) with sizes of 32×32, 64×64, and 128×128 include: a first convRes module, a first max pooling layer, a second convRes module, a second max pooling layer, a Flatten layer, a fully connected neural network, and a softmax activation function, which are sequentially cascaded.

5. A lightweight, multi-source information-based V-PCC inter-frame mode fast selection method according to claim 3 or 4, characterized in that, The convRes module includes multiple cascaded convRes units. Each convRes unit includes: a first convolutional layer, a first batch normalization (BN) layer, a first Swish activation function, a second convolutional layer, a second BN layer, a third BN layer, and a second Swish activation function, all cascaded in sequence. The input features of the convRes module and the input features of the third BN layer are connected by residuals through a 1×1 convolution or directly.

6. The lightweight multi-source information combined V-PCC inter-frame mode fast selection method according to claim 3, characterized in that, In the geometric near-layer coding layer and the attribute near-layer coding layer, for coding units (CUs) other than those with sizes of 8×8, 16×16, 32×32, 64×64, and 128×128, they are divided into multiple CUs with sizes of 8×8, 16×16, 32×32, 64×64, or 128×128. The binary classification model and multi-class classification model of the corresponding size are used to predict the pattern of the multiple CUs after division. The prediction result of the coding unit CU is obtained by averaging the prediction results of all CUs after division. For coding units CUs other than those with sizes of 8×8, 16×16, 32×32, 64×64, and 128×128, if they cannot be formed by combining sizes of 8×8, 16×16, 32×32, 64×64, or 128×128, the coding unit CU is directly sent to the V-PCC encoder for pattern detection.

7. The lightweight multi-source information combined V-PCC inter-frame mode fast selection method according to claim 4, characterized in that, In the geometric far-layer coding layer and the attribute far-layer coding layer, for coding units (CUs) other than those with sizes of 32×32, 64×64, and 128×128, they are divided into multiple CUs with sizes of 32×32, 64×64, or 128×128. The binary classification model and multi-class classification model of the corresponding size are used to predict the pattern of the multiple CUs after division. The prediction result of the coding unit CU is obtained by averaging the prediction results of all CUs after division. For coding units (CUs) other than those with sizes of 32×32, 64×64, and 128×128, if they cannot be formed by combining sizes of 32×32, 64×64, or 128×128, the coding unit CU is directly sent to the V-PCC encoder for pattern detection.

8. The lightweight multi-source information combined V-PCC inter-frame mode fast selection method according to claim 2, characterized in that, The loss functions used during the training of the binary classification model and the multi-class classification model include: in, Let C represent the cathodic function, and C represent the number of categories. Indicates the weighting parameter; This represents the probability that the predicted result belongs to category c; This indicates the actual label.

9. The lightweight multi-source information combined V-PCC inter-frame mode fast selection method according to claim 2, characterized in that, For the binary classification model corresponding to the geometrically near-layer coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, and optical flow information of coding unit CU; For the binary classification model corresponding to the geometrically far coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, pixels of corresponding coding unit CU in the geometrically near coding layer, and MERGE_SKIP pattern probability prediction results of corresponding coding unit CU in the geometrically near coding layer; wherein, in the binary classification model, the MERGE_SKIP pattern probability prediction results of corresponding coding unit CU in the geometrically near coding layer are concatenated with the output features of the Flatten layer and used as the input features of the fully connected neural network; For a binary classification model corresponding to an attribute-near coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, pixels of corresponding coding unit CU in the geometrically near coding layer, and MERGE_SKIP pattern probability prediction results of coding unit CU corresponding to the geometrically near coding layer; wherein, in the binary classification model, the MERGE_SKIP pattern probability prediction results of coding unit CU corresponding to the geometrically near coding layer are concatenated with the output features of the Flatten layer and used as the input features of the fully connected neural network; For the binary classification model corresponding to the attribute far-layer coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, pixels of corresponding coding unit CU in the geometric far-layer coding layer, pixels of corresponding coding unit CU in the attribute near-layer coding layer, and MERGE_SKIP pattern probability prediction results of coding unit CU corresponding to the attribute near-layer coding layer; among them, in the binary classification model, the MERGE_SKIP pattern probability prediction results of coding unit CU corresponding to the attribute near-layer coding layer are concatenated with the output features of the Flatten layer and used as the input features of the fully connected neural network.

10. The lightweight multi-source information combined V-PCC inter-frame mode fast selection method according to claim 2, characterized in that, For the multi-classification model corresponding to the geometrically near-layer coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, and optical flow information of coding unit CU; For the multi-class classification model corresponding to the geometrically far coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, pixels of corresponding coding unit CU in the geometrically near coding layer, and other optimal mode probability prediction results of coding unit CU corresponding to the geometrically near coding layer; wherein, in the multi-class classification model, the other optimal mode probability prediction results of coding unit CU corresponding to the geometrically near coding layer are concatenated with the output features of the Flatten layer and used as the input features of the fully connected neural network; For a multi-class classification model corresponding to an attribute-near coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, pixels of corresponding coding unit CU in the geometrically near coding layer, and other optimal mode probability prediction results of coding unit CU corresponding to the geometrically near coding layer; wherein, in the multi-class classification model, the other optimal mode probability prediction results of coding unit CU corresponding to the geometrically near coding layer are concatenated with the output features of the Flatten layer and used as the input features of the fully connected neural network; For a multi-class classification model corresponding to an attribute far-layer coding layer, its input features include: pixels of coding unit CU, residuals of coding unit CU, occupancy map of coding unit CU, optical flow information of coding unit CU, pixels of corresponding coding unit CU in the geometric far-layer coding layer, pixels of corresponding coding unit CU in the attribute near-layer coding layer, and other optimal pattern probability prediction results of coding unit CU corresponding to the attribute near-layer coding layer; among them, in the multi-class classification model, the other optimal pattern probability prediction results of coding unit CU corresponding to the attribute near-layer coding layer are concatenated with the output features of the Flatten layer and used as the input features of the fully connected neural network.

Images