Point cloud quality assessment method based on depth ordinal regression

Abstract

The invention relates to a point cloud quality assessment method based on depth ordinal regression. The method comprises a feature fusion step based on a symmetric cross-modal attention mechanism and an ordered regression step of point cloud quality assessment. In the feature fusion step, point cloud features are extracted through farthest point sampling and a K nearest neighbor algorithm, point cloud projection is performed to generate an image, image features are extracted, the two features are fused by using a cross-modal symmetric attention mechanism, and global feature representation is generated; in the step of ordered regression, a depth ordinal number loss function is designed to optimize network parameters, the prediction probability is converted into a continuous point cloud quality score by using ordered information of the quality score, a step effect is prevented, and an evaluation result is closer to a true value. Through feature fusion and ordered regression, the accuracy and stability of point cloud quality evaluation are effectively improved, and the evaluation result is closer to a true value.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a point cloud quality assessment method based on deep ordinal regression. Background Art

[0002] Compared with traditional 2D images, 3D sensors can often provide the geometry, shape and scale information captured in the real world in a more comprehensive and intuitive way. Point cloud is a common format in 3D data. It refers to a collection of a large number of three-dimensional points. It can save the most original information in three-dimensional space without any changes, ensuring realism and three-dimensionality.

[0003] With the rapid development of 3D acquisition technology, point clouds have been widely used in fields such as autonomous driving, construction and mapping, virtual reality and augmented reality, and medical imaging. Since each dense point cloud contains tens of thousands of points, its media content will be compressed during data storage and transmission, which will inevitably cause distortion of varying degrees and types. Low-quality point clouds may cause errors or performance issues in the calculation process. We need to evaluate the degree of distortion, find defects in the point cloud (such as sparse and missing points), check the geometric accuracy and noise level of the point cloud, ensure that it meets the requirements, optimize the data acquisition or processing process, and improve reliability and accuracy. Therefore, point cloud quality assessment is an important part of ensuring the accuracy, integrity and reliability of three-dimensional data.

[0004] At present, point cloud quality assessment still faces great challenges. Since point clouds are irregular data sets in three-dimensional space, the data dimensions are high and the number is huge, and there is a lack of topological structure, making it difficult to directly apply traditional image quality assessment methods, and it is more difficult to directly evaluate than two-dimensional images. Traditional point cloud quality assessment methods are divided into three categories according to whether the original point cloud is available: full reference, semi-reference, and no reference. In the early stages of development, MPEG proposed some simple point-based full reference methods, such as p2point and p2plane. Due to the difficulty in collecting original point clouds, thanks to the rapid development of deep learning, no-reference point cloud quality assessment methods have been significantly improved. For example, Chetouani et al. extracted hand-crafted features block by block and used a classic CNN model for quality regression; Liu et al. used multi-view projection for feature extraction; Zhang et al. used several statistical distributions to estimate quality perception parameters from the distribution of geometric and color attributes; Fan et al. inferred the visual quality of point clouds from captured video sequences; Liu et al. used end-to-end sparse CNN for quality prediction; Yang et al. further transferred quality information from natural images to understand the quality of point cloud rendered images through domain adaptation.

[0005] Most of the existing deep learning-based no-reference point cloud quality assessment models are formulated as regression problems and trained by minimizing the mean square error. However, the error measurement does not take into account the relative order between different ratings on the quality scale, thus affecting the effectiveness of the model. To solve this problem, the present invention reformulates the no-reference point cloud quality assessment learning as an ordered regression problem, combines multimodal feature extraction, converts the predicted probability into a continuous variable of image quality through deep ordered loss and uses soft ordered inference to obtain the final point cloud quality score. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a point cloud quality assessment method based on deep ordinal regression, which can effectively evaluate the quality of the input point cloud in point cloud quality assessment, and the evaluation result has high accuracy and is less time-consuming.

[0007] The specific plan is as follows:

[0008] A point cloud quality assessment method based on deep ordinal regression, comprising:

[0009] Based on the feature fusion step of the symmetric cross-modal attention mechanism, the two modalities of point cloud and image are combined to extract the point cloud visual quality features and image modal features through the feature encoder; the extracted point cloud visual quality features and image modal features are integrated to analyze the interaction between the modalities and strengthen the feature representation to generate the final feature vector;

[0010] Point cloud quality assessment is expressed as an ordered regression step. Based on the final feature vector, the prediction probability is converted into a continuous point cloud quality score by simplifying multiple classification problems into a series of binary classification problems. An adaptive factor is introduced to make the prediction score adapt to the sparsity of the discretization interval to solve the step effect that is easy to occur when using hard threshold inference.

[0011] Furthermore, the feature fusion step based on the symmetric cross-modal attention mechanism specifically includes:

[0012] Point cloud modal feature extraction step: divide the input point cloud into local areas and extract multiple patches, input the point cloud feature encoder and extract the local features of the point cloud;

[0013] Image modality feature extraction step: project the point cloud to a fixed viewpoint to generate a projection image, and input it into the image encoder to extract image features;

[0014] The steps of generating the fused global feature representation are as follows: Based on the cross-modal symmetric attention mechanism, the extracted point cloud features and image features are fused to generate the fused global feature representation.

[0015] Furthermore, the base point cloud modal feature extraction step specifically includes:

[0016] Given a normalized point cloud Use the farthest point sampling to obtain N δ Anchor point For each anchor point, the K nearest neighbor algorithm is used to find N s Adjacent points are used to form the corresponding point cloud sub-model. The calculation formula is as follows:

[0017]

[0018] Where S is the set of sub-models, KNN(·) represents the KNN operation, and N δ Indicates the number of anchor points; using the point cloud feature encoder θ P (·) Map the obtained sub-model into the quality-aware embedding space, and the calculation formula is as follows:

[0019]

[0020] in, Represents the I-th sub-model S l The quality-aware embedding of C P Represents the point cloud encoder θ P (·) the number of output channels, is the merged result after average fusion.

[0021] Furthermore, the image modality feature extraction step specifically includes:

[0022] by The interval defines a circular path that rotates around the distorted color point cloud for rendering, capturing the projection N while maintaining the consistency of the texture. I , the calculation formula is as follows:

[0023]

[0024] Among them, X and Y represent the horizontal and vertical coordinates of each point in the point cloud on the projection plane, Z represents the vertical coordinate of each point in the point cloud, the path is centered on the geometric center of the point cloud, and R represents a fixed observation distance; Open3D is used for rendering, and the rendered 2D image is embedded into the quality perception space using a 2D image encoder. The calculation formula is as follows:

[0025]

[0026] in, represents the t-th image projection I t The quality-aware embedding of C I Denotes the image encoder θ I (·) the number of output channels, is the merged result after average fusion.

[0027] Furthermore, the fused global feature representation generation step specifically includes:

[0028] Given features from point cloud and image modalities Use a linear layer to unify them to the same size:

[0029]

[0030] in, is the adjusted feature, W P and W I is a learnable linear mapping, C′ is the adjusted number of channels, and in order to further distinguish the components between the modalities, a multi-head attention module is used:

[0031]

[0032] where Γ(·) represents the multi-head attention operation, β(·) represents the attention function, and j μ represents the μth head, W, W Q , W K , W V represents a learnable linear mapping, and the final quality embedding is cascaded through the intra-modal features and the guided multimodal features obtained by the symmetric cross-modal attention module, and the calculation formula is as follows:

[0033]

[0034] in, represents the cascade operation, Ψ(·) represents the symmetric cross-modal attention operation, Represents the final quality-aware characteristics.

[0035] Furthermore, the point cloud quality assessment is expressed as an ordered regression step, specifically including:

[0036] Ordinal loss optimization step: Design a deep ordinal loss function to learn network parameters, and use the ordered information in the discrete labels of quality scores to make the network more sensitive to predictions with inconsistent quality score order attributes;

[0037] Soft ordinal reasoning step: The predicted probability is converted into a continuous point cloud quality score through soft ordinal reasoning. The probability or confidence predicted by the network is used to prevent the step effect that is easily produced by ordinal operations, so that the quality score is more adapted to the sparsity of the discretization interval, thereby achieving an effect closer to the ground truth.

[0038] Furthermore, the ordinal loss optimization step specifically includes:

[0039] The entire range of the existing image quality score database is discretized into K sub-intervals of equal size. The calculation formula is as follows:

[0040]

[0041] in, represents the base function, d * ∈{0, 1, ..., k-1} represents the discrete label of the ground truth, s * Represents the original continuous value of the image quality score, s min 、s max denote the minimum and maximum scores in a given database, respectively;

[0042] The continuous quality scores are uniformly discretized into multiple bins in the logarithmic space, and the bins are marked according to the range of quality score values ​​covered by each bin, and the ordinal threshold c is obtained. k ∈{0, 1, ..., c k-1 The calculation formula is as follows:

[0043]

[0044] For each instance c k ∈{0, 1, ..., c k-1}, apply the binary classifier to predict whether the ordinal value of the sample is greater than c k , and predict the hidden sample ordinal value based on the classification results of K-1 binary classifiers, and define the output of the model as Where Y is a 2K-dimensional vector, the deep ordinal loss calculation formula is as follows:

[0045]

[0046] Where d represents the predicted discrete label, represents the ordinal probability when d is greater than k.

[0047] Furthermore, the soft ordinal number reasoning step specifically includes:

[0048] The output probability is converted into a continuous variable of image quality, and the calculation formula is as follows:

[0049]

[0050] in Represents the introduced adaptive factor, d is between 0 and 1, indicating the degree to which the predicted category is close to d+1.

[0051] The present invention adopts the above technical solution and has the following beneficial effects:

[0052] (1) The present invention adopts a symmetric cross-modal attention mechanism to force the model to associate and synthesize quality-aware patterns, mobilize the complementary information between image and point cloud modalities, and jointly utilize intra-modal and cross-modal features to optimize quality representation;

[0053] (2) The present invention proposes a deep ordinal loss function for regularization, which imposes a larger loss on the prediction of inconsistent ordinal attributes of quality labels, so that the confidence of the learned quality category is more accurately represented;

[0054] (3) Compared with similar methods (quality assessment method without reference point cloud), the present invention takes into account the relative order of quality ratings, makes the quality score prediction probability more accurate, and improves the overall assessment effect; compared with similar methods (quality assessment method with full reference point cloud, quality assessment method with semi-reference point cloud), it can perform a simpler, less time-consuming, and more accurate quality assessment without using the original point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of the framework of the point cloud quality assessment method based on deep ordinal regression of an example of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0057] like Figure 1 As shown, the present invention provides a point cloud quality assessment method based on deep ordinal regression, which includes the following steps.

[0058] S101, the feature fusion step based on the symmetric cross-modal attention mechanism, combines the two modalities of point cloud and image, extracts the point cloud visual quality features and image modal features through the feature encoder; integrates the extracted point cloud visual quality features and image modal features, analyzes the interaction between modalities and strengthens the feature representation, generates the final feature vector, and solves the problem of insufficient coverage of single modality information.

[0059] Specifically, given a colored point cloud in represents geometric coordinates, It represents the additional RGB color information. The corresponding point cloud modality is obtained by normalizing the original point cloud coordinates. The image modality I is obtained by rendering the colored point cloud P into a 2D projection (the projection does not contain color information). In order to make the segmented point cloud sub-model retain important local geometric structure patterns, the segmented point cloud needs to be repaired.

[0060] Specifically, the method for repairing the segmented point cloud is the feature fusion step based on the symmetric cross-modal attention mechanism, which specifically includes:

[0061] S1011, point cloud modal feature extraction step: divide the input point cloud into local areas and extract multiple patches, input the point cloud feature encoder and extract point cloud local features;

[0062] Given a normalized point cloud Use the farthest point sampling (FPS) to obtain N δ Anchor point For each anchor point, the K nearest neighbor algorithm (KNN) is used to find N s Adjacent points are used to form the corresponding point cloud sub-model:

[0063]

[0064] Where S is the set of sub-models, and KNN(·) represents the KNN operation. The local sub-model maintains the corresponding mode and uses the point cloud feature encoder θ P (·) Map the obtained sub-model into the quality-aware embedding space:

[0065]

[0066] in, Represents the I-th sub-model S I The quality-aware embedding of C P Represents the point cloud encoder θ P (·) the number of output channels, It is the merged result after average fusion, and the final point cloud modal features are obtained.

[0067] S1012, image modality feature extraction step: projecting the point cloud to a fixed viewpoint to generate a projection image, and inputting the projection image into an image encoder to extract image features;

[0068] Camera The rendering is performed along a defined circular path at a fixed viewing distance around the distorted color point cloud, capturing the projection N while maintaining the consistency of the texture. I .

[0069]

[0070] The path is centered at the geometric center of the point cloud, R represents a fixed viewing distance, and Open3D is used for rendering. The rendered 2D image is then embedded into the quality-aware space using a 2D image encoder:

[0071]

[0072] in, represents the t-th image projection I t The quality-aware embedding of CI Denotes the image encoder θ I (·) the number of output channels, It is the combined result after average fusion, and the final image modality feature is obtained.

[0073] S1013, a step of generating a fused global feature representation: based on a cross-modal symmetric attention mechanism, the extracted point cloud features and image features are fused to generate a fused global feature representation;

[0074] Given features from point cloud and image modalities Use a linear layer to unify them to the same size:

[0075]

[0076] in, is the adjusted feature, W P and W I is a learnable linear mapping, and C′ is the adjusted number of channels. In order to further distinguish the components between modalities, a multi-head attention module is used:

[0077]

[0078] where Γ(·) represents the multi-head attention operation, β(·) represents the attention function, and h μ is the μth head, W, W Q , W K , W V is a learnable linear mapping, and the final quality embedding can be cascaded through the intra-modal features and the guided multi-modal features obtained by the symmetric cross-modal attention module:

[0079]

[0080] S102, point cloud quality assessment is expressed as an ordered regression step. Based on the final feature vector, the prediction probability is converted into a continuous point cloud quality score by simplifying multiple classification problems into a series of binary classification problems. An adaptive factor is introduced to make the prediction score adapt to the sparsity of the discretization interval to solve the step effect that is easy to occur when using hard threshold inference.

[0081] Based on the overall perception of a given image space, the image is placed into one of the quality categories (e.g., the interval representing "poor" to "excellent" in a continuous range from 0 to 10), and then the quality score is refined (i.e., producing a decimal number) by comparing the current image with other images belonging to this local perceptual quality interval, formulating the model as an ordinal regression problem, making it more consistent with human preference levels.

[0082] Specifically, the point cloud quality assessment is expressed as an ordered regression step, specifically including:

[0083] S1021 Ordinal loss optimization step: Design a deep ordinal loss function to learn network parameters, and use the ordered information in the discrete labels of quality scores to make the network more sensitive to predictions with inconsistent quality score order attributes;

[0084] First, the entire range of the existing image quality score database is discretized into K sub-intervals of equal size:

[0085]

[0086] in, represents the base function, d * ∈{0, 1, ..., k-1} is the discrete label of the ground truth, s * is the raw continuous value of the image quality score, s min 、s max denote the minimum and maximum scores in a given database, respectively; the continuous quality scores are discretized uniformly into multiple bins in logarithmic space; each bin covers a range of quality score values, and the bins are labeled according to the range (i.e., the label index of the feature indicates its distance); the ordinal threshold c is obtained k ∈{0, 1, ..., c k-1 The calculation formula is as follows,

[0087]

[0088] The typical multi-class classification loss ignores the ordered information between discrete labels, while the quality scores have strong ordered correlation because they form an ordered set; the quality assessment problem can be converted into an ordered regression problem, and a deep ordered loss (DO-loss) is designed for the network to learn the network parameters, converting the multi-class classification problem into a set of binary classification sub-problems: for each instance c k ∈{0, 1, ..., c k-1}, apply the binary classifier to predict whether the ordinal value of the sample is greater than c k ; and predict the hidden sample ordinal value based on the classification results of K-1 binary classifiers, and define the output of the model as Where Y is a 2K-dimensional vector, and the calculation formula for the deep ordinal loss (DO-loss) is as follows:

[0089]

[0090]

[0091] Where d represents the predicted discrete label, is the ordinal probability when d is greater than k; compared with the conventional use of cross entropy loss to train classification networks, deep ordinal loss (DO-loss) can update network parameters more effectively. Due to the inherent sequential attributes contained in quality ratings, the sequential loss is more sensitive to predictions that are inconsistent with the sequential attributes of the ground truth.

[0092] Based on the output probabilities of K binary classification instances, the calculation formula for predicting the image quality score s is as follows:

[0093]

[0094] where η(·) denotes the indicator function, where η(true) = 1 and η(false) = 0. The rounding operation of hard inference ignores the probability (or confidence) of the network prediction, which may lead to predictions in transition regions that are difficult for the network to distinguish.

[0095] S1022 Soft ordinal reasoning step: The predicted probability is converted into a continuous point cloud quality score through soft ordinal reasoning. The probability or confidence of network prediction is used to prevent the step effect that is easy to be produced by ordinal operation, so that the quality score is more adapted to the sparsity of the discretization interval, so as to achieve an effect closer to the ground truth.

[0096] Accurately using the probability (or confidence) predicted by the network to prevent the step effect that is easy to be produced by ordinal operations makes the quality score more adaptable to the sparsity of the discretized interval, so as to achieve an effect closer to the ground truth; classification instead of regression for quality assessment can naturally obtain the confidence of the quality distribution; the elements in the confidence map of each class only focus on a specific quality score interval, which simplifies network learning, but it introduces a discretized mean square error, which is very sensitive to the number of quality score intervals; the reasoning strategy based on hard thresholds ignores the obtained probability distribution, which will lead to step effects in the feature map; and the naive hard reasoning is extended to a soft version, called soft ordinal reasoning, to solve the above problems; soft ordinal reasoning makes full use of the predicted confidence, has a strong classification ability for the transition area between objects, and can convert the output probability into a continuous variable of image quality. The specific calculation formula is as follows:

[0097]

[0098] in d is between 0 and 1, indicating the degree to which the predicted category is close to d+1. In this soft reasoning, due to the introduction of the adaptive factor The predicted scores will adapt to the sparsity of the discretization intervals, which makes the inferred scores closer to the ground truth.

[0099] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A point cloud quality assessment method based on deep ordinal regression, characterized in that: include: Based on the feature fusion step of the symmetric cross-modal attention mechanism, the two modalities of point cloud and image are combined to extract the point cloud visual quality features and image modal features through the feature encoder; the extracted point cloud visual quality features and image modal features are integrated to analyze the interaction between the modalities and strengthen the feature representation to generate the final feature vector; Point cloud quality assessment is expressed as an ordered regression step. Based on the final feature vector, the prediction probability is converted into a continuous point cloud quality score by simplifying multiple classification problems into a series of binary classification problems. An adaptive factor is introduced to make the prediction score adapt to the sparsity of the discretization interval to solve the step effect that is easy to occur when using hard threshold inference.

2. The point cloud quality assessment method based on deep ordinal regression according to claim 1 is characterized in that: The feature fusion step based on the symmetric cross-modal attention mechanism specifically includes: Point cloud modal feature extraction step: divide the input point cloud into local areas and extract multiple patches, input the point cloud feature encoder and extract the local features of the point cloud; Image modality feature extraction step: project the point cloud to a fixed viewpoint to generate a projection image, and input it into the image encoder to extract image features; The steps of generating the fused global feature representation are as follows: Based on the cross-modal symmetric attention mechanism, the extracted point cloud features and image features are fused to generate the fused global feature representation.

3. The point cloud quality assessment method based on deep ordinal regression according to claim 2 is characterized in that: The base point cloud modal feature extraction step specifically includes: Given a normalized point cloud Use the farthest point sampling to obtain N δ Anchor point For each anchor point, the K nearest neighbor algorithm is used to find N s Adjacent points are used to form the corresponding point cloud sub-model. The calculation formula is as follows: Where S is the set of sub-models, KNN(·) represents the KNN operation, and N δ Indicates the number of anchor points; using the point cloud feature encoder θ P (·) Map the obtained sub-model into the quality-aware embedding space, and the calculation formula is as follows: in, Represents the I-th sub-model S l The quality-aware embedding of C P Represents the point cloud encoder θ P (·) the number of output channels, is the merged result after average fusion.

4. The point cloud quality assessment method based on deep ordinal regression according to claim 2 is characterized in that: The image modality feature extraction step specifically includes: by The interval defines a circular path that rotates around the distorted color point cloud for rendering, capturing the projection N while maintaining the consistency of the texture. I , the calculation formula is as follows: Among them, X and Y represent the horizontal and vertical coordinates of each point in the point cloud on the projection plane, Z represents the vertical coordinate of each point in the point cloud, the path is centered on the geometric center of the point cloud, and R represents a fixed observation distance; Open3D is used for rendering, and the rendered 2D image is embedded into the quality perception space using a 2D image encoder. The calculation formula is as follows: in, represents the t-th image projection I t The quality-aware embedding of N δ represents the number of anchor points, C I Denotes the image encoder θ I (·) the number of output channels, is the merged result after average fusion.

5. The point cloud quality assessment method based on deep ordinal regression according to claim 2, characterized in that: The fused global feature representation generation step specifically includes: Given features from point cloud and image modalities and Use a linear layer to unify them to the same size: in, and is the adjusted feature, W P and W I is a learnable linear mapping, C′ is the adjusted number of channels; through the multi-head attention module: where Γ(·) denotes a multi-head attention operation, β(·) denotes an attention function, Q denotes a query, K denotes a key, V denotes a value, and h μ represents the μth head, W, W Q , W K , W V Represents a learnable linear mapping, which is finally weighted and summed to obtain the output of the attention head. The final quality embedding is cascaded with the guided multimodal features obtained by the intra-modal features and the symmetric cross-modal attention module. The calculation formula is as follows: in, represents the cascade operation, Ψ(·) represents the symmetric cross-modal attention operation, Represents the final quality-aware characteristics.

6. The point cloud quality assessment method based on deep ordinal regression according to claim 1, characterized in that: The point cloud quality assessment is expressed as an ordered regression step, specifically including: Ordinal loss optimization step: Design a deep ordinal loss function to learn network parameters, and use the ordered information in the discrete labels of quality scores to make the network more sensitive to predictions with inconsistent quality score order attributes; Soft ordinal reasoning step: The predicted probability is converted into a continuous point cloud quality score through soft ordinal reasoning. The probability or confidence of network prediction is used to prevent the step effect that is easy to be produced by ordinal operation, so that the quality score is more adapted to the sparsity of the discretization interval.

7. The point cloud quality assessment method based on deep ordinal regression according to claim 6 is characterized in that: The ordinal loss optimization step specifically includes: The entire range of the existing image quality score database is discretized into K sub-intervals of equal size. The calculation formula is as follows: in, represents the base function, d * ∈{0, 1, ..., k-1} represents the discrete label of the ground truth, s * Represents the original continuous value of the image quality score, s min 、s max denote the minimum and maximum scores in a given database, respectively; The continuous quality scores are uniformly discretized into multiple bins in the logarithmic space, and the bins are marked according to the range of quality score values ​​covered by each bin, and the ordinal threshold c is obtained. k ∈{0, 1, ..., c k-1 The calculation formula is as follows: For each instance c k ∈{0, 1, ..., c k-1 }, apply the binary classifier to predict whether the ordinal value of the sample is greater than c k , and predict the hidden sample ordinal value based on the classification results of K-1 binary classifiers, and define the output of the model as Where Y is a 2K-dimensional vector, the deep ordinal loss calculation formula is as follows: Where d represents the predicted discrete label, represents the ordinal probability when d is greater than k.

8. The point cloud quality assessment method based on deep ordinal regression according to claim 6, characterized in that: The soft ordinal number reasoning step specifically includes: The output probability is converted into a continuous variable of image quality, and the calculation formula is as follows: in Represents the introduced adaptive factor, d is between 0 and 1, indicating the degree to which the predicted category is close to d+1.

Images