No-reference-point cloud quality evaluation algorithm based on projection
By projecting point cloud data into a two-dimensional plane and performing multi-scale feature fusion, combined with visual geometry group network or residual network, the problem of insufficient semantic information capture by point cloud quality evaluation algorithm in the existing technology is solved, and a higher level is achieved rating accuracy and reliability.
Patent Information
- Application Number
- CN202410956340.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-15
AI Technical Summary
The existing point cloud quality evaluation algorithm has shortcomings in capturing semantic information, resulting in low accuracy of evaluation results. The traditional method only relies on geometric information and manually extracts single-scale features, and its performance is not ideal.
Using a projection-based reference-free point cloud quality evaluation algorithm, by projecting point cloud data to a two-dimensional plane, extracting different scale features and performing combination pooling, combining visual geometric group networks or residual networks for multi-scale feature fusion, and finally Regression prediction is performed using a fully connected layer to simulate multi-view and multi-scale perception of human vision systems.
It improves the accuracy and objectivity of point cloud quality evaluation, can more comprehensively capture different manifestations of point cloud distortion, and provides more reliable quality evaluation results.
Smart Images

Figure CN118967579A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology and can be applied to quality evaluation scenarios of distorted point clouds. Specifically, the present invention is a projection-based reference-free point cloud quality evaluation algorithm. Background Art
[0002] Point cloud quality evaluation technology is widely used in virtual reality, augmented reality, autonomous driving, medical imaging and other fields. Due to the limitations of actual application scenarios and specific sensors, point cloud data is easily interfered by various factors during collection, processing and transmission, such as downsampling, Gaussian filtering, compression, etc., which leads to the degradation of point cloud quality and further affects the perception of human eyes. Therefore, an objective point cloud quality evaluation model is needed to evaluate whether the point cloud meets the application requirements.
[0003] The point cloud quality evaluation algorithms commonly used at present are the quality evaluation algorithm based on three-dimensional points and the evaluation algorithm based on two-dimensional projection. The evaluation algorithm based on three-dimensional points mainly evaluates the quality by comparing the similarity of corresponding points or corresponding areas in three-dimensional areas, that is, extracting some geometry or features from each point cloud in a global or local manner, thereby evaluating the point cloud quality. Since this method only considers the geometric information of the point cloud and ignores the semantic information, the accuracy of the point cloud quality evaluation result is low. The evaluation algorithm based on two-dimensional projection mainly projects the three-dimensional point cloud onto a two-dimensional plane and uses a relatively mature two-dimensional image quality evaluation method to evaluate the quality. These methods mainly manually extract single-scale features of the point cloud projection image, and the evaluation performance is not ideal. Therefore, it is necessary to design a point cloud quality evaluation algorithm that is more in line with human perception and has more accurate prediction performance. Summary of the invention
[0004] In view of the shortcomings of the prior art, the present invention proposes a projection-based reference-free point cloud quality evaluation algorithm. The present invention first projects the point cloud data onto a two-dimensional plane, then extracts different scale features of the point cloud projection image, and fuses the different scale features after combined pooling. Finally, a fully connected layer is used to regress the fused features, and the average prediction score after regression of all scale features is used as the objective score of the point cloud quality.
[0005] Taking into account the different perceived qualities of images of different scales by the human visual system, the present invention first projects the distorted point cloud from six perspectives, and stacks the projections into image blocks as input. Based on the concept of multi-scale feature fusion of the feature pyramid framework, in the feature extraction part, the Visual Geometry Group Network (VGG) or the Residual Network (ResNet) is used as the backbone of the feature pyramid to extract multi-scale features of the image, and each scale feature is subjected to two pooling processes. The features processed at the same scale are fused, and the fused features are regressed using a fully connected layer. Finally, the mean of the prediction values of all layers is taken as the objective score of the point cloud quality.
[0006] The present invention uses multi-view projection and multi-scale feature fusion to better simulate the perception of the human visual system and improve the accuracy of point cloud quality evaluation. Compared with traditional methods, the present invention can more comprehensively capture different manifestations of point cloud distortion and provide more objective and reliable quality evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 It is a flow chart of the point cloud quality evaluation algorithm of the present invention. DETAILED DESCRIPTION
[0008] The following is combined with Figure 1 The point cloud quality evaluation algorithm of the present invention is further explained.
[0009] like Figure 1 As shown, the point cloud quality evaluation algorithm of the present invention includes four steps: step 1 point cloud projection processing; step 2 multi-scale feature extraction, step 3 feature fusion, and step 4 quality score prediction.
[0010] Step 1: Point cloud projection
[0011] Projecting the distorted point cloud means using a preset rotation matrix R and different rotation angles to project the distorted point cloud in six viewing angles to obtain two-dimensional projection images of six different viewing angles (up, down, left, right, front, and back). Assume that the coordinates of the distorted point cloud are (O x ,O y ,O z ), the coordinates after the rotation matrix R and the rotation angle are (P x ,P y ,P z ), then the rotation operation is expressed as:
[0012] R=R top R bottom R left R right Rfront R back
[0013]
[0014] Among them, formulas (1)(2), (3)(4), (5)(6) respectively represent the top view, bottom view, left view, right view, front view and back view obtained by counterclockwise rotation around the X-axis, Y-axis and Z-axis, and formula (7) represents the distorted point cloud after the rotation matrix.
[0015] The obtained two-dimensional projection images are stacked into three-dimensional image blocks as the input of the feature extraction unit, and multi-scale features are extracted using the feature pyramid network. Assuming that the size of the two-dimensional projection image is M×N, the stacking operation is expressed as:
[0016]
[0017] in, V represents the pixel value located in the i-th row and j-th column of the k-th perspective projection image. ijk Represents the pixel value at row i, column j, and layer k in a three-dimensional block, where i∈(1,2,3,...,M) and j∈(1,2,3,...,N).
[0018] Step 2: Multi-scale feature extraction
[0019] The three-dimensional image block is represented as I, and the features of the image block are upsampled to obtain I k , the top-down feature construction process is expressed as:
[0020]
[0021] in, It represents the value of the feature of the kth layer in the top-down path at position (i, j), and Upsample represents the upsampling operation.
[0022] The bottom-up feature construction process is expressed as:
[0023]
[0024] in, It represents the value of the feature of the k-1th layer in the bottom-up path at position (i, j), and Downsample represents the downsampling operation.
[0025] Multi-scale feature fusion is expressed as:
[0026]
[0027] Step 3: Feature Fusion
[0028] The feature P of each pyramid level k Two pooling operations are performed as input. Assuming that the size of each feature is H×W×C, the window size of the pooling operation is m×m, where H represents the height, W represents the width, C represents the number of channels, and k represents the pyramid level.
[0029] The mean pooling operation is expressed as:
[0030]
[0031] The maximum pooling operation is expressed as:
[0032]
[0033] Among them, P k [j:j+m,i:i+m] represents feature P k The m×m region with position (i, j) as the upper left corner.
[0034] The features obtained after mean and maximum pooling operations Perform feature fusion to obtain fusion features The calculation formula is as follows:
[0035]
[0036] Among them, β and γ are weight parameters, β>0, γ>0, β+γ=1, which are used to control the importance of features under different pooling methods.
[0037] Step 4: Quality score prediction
[0038] The fusion features As input, it is flattened and converted into a one-dimensional vector. Assuming that the fusion feature The size of is N×H×W×C, where N represents the number of projection surfaces, H represents the height, W represents the width, and C represents the number of channels.
[0039] The flattening operation is expressed as:
[0040]
[0041] where p k Represents the flattened feature vector, whose size is NHWC×1, and Flatten represents the flattening operation.
[0042] The p k The regression operation is performed through the fully connected layer (i.e., linear transformation). Assuming that the output dimension of the fully connected layer is 1, it means predicting a point cloud quality value.
[0043] The regression operation is expressed as:
[0044] z k =W·p k +b (16)
[0045] Among them, p k Represents the input feature z through the fully connected layer k The output after linear transformation, W represents the weight matrix of the fully connected layer, and b represents the bias vector of the fully connected layer.
[0046] After passing through the fully connected layer, the ReLU activation function is used to obtain the predicted value Q of the kth level k .
[0047] The specific operation is expressed as:
[0048] Q k =ReLU(z k ) (17)
[0049] Finally, the predicted values of all levels n are averaged as the objective score Q of the entire point cloud.
[0050] The resulting comprehensive score is expressed as:
[0051]
[0052] The performance of the algorithm was verified on the SJTU-PCQA and WPC databases. The prediction accuracy and monotonicity of the proposed model were measured using four evaluation criteria: Pearson linear correlation coefficient (PLCC), Spearman Rank-order Correlation Coefficient (SROCC), Kendall Rank-order Correlation Coefficient (KROCC), and Root Mean Squared Error (RMSE). The value range of PLCC, SROCC, and KROCC is [-1, 1]. The closer the absolute value is to 1, the stronger the correlation and the higher the consistency. The range of RMSE is [0, ∞]. The closer it is to 0, the better the prediction performance.
[0053] Table 1 Performance comparison with other advanced methods on SJTU-PCQA database
[0054]
[0055] Table 2 Performance comparison with other advanced methods on the WPC database
[0056]
[0057] It can be seen from Table 1 that for the monotonicity evaluation indicators SROCC and KROCC, this method is the highest on the SJTU-PCQA database; for the prediction accuracy indicators PLCC and RMSE, this method is the highest on the SJTU-PCQA database.
[0058] As can be seen from Table 2, for the monotonicity evaluation indicators SROCC and KROCC, this method is the highest on the WPC database; for the prediction accuracy indicators PLCC and RMSE, the PLCC value of this method is the highest and the RMSE value is the lowest. In short, the proposed method has achieved good no-reference point cloud quality evaluation results, which can maintain good consistency with the subjective evaluation of human vision.
[0059] For those skilled in the art, the above embodiments are illustrative rather than restrictive, and the scope of the present invention is defined by the rights and claims rather than the above embodiments, and therefore, it is intended that all changes in the meaning and scope of the equivalent elements of the claims are included in the present invention. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural.
Claims
1. A projection-based no-reference point cloud quality assessment algorithm, characterized in that: include: Step 1: Project the distorted point cloud to obtain two-dimensional projection images of the distorted point cloud at six different viewing angles, and stack the projection images at different viewing angles into image blocks; Step 2: According to the projection graph in step 1, build a neural network and extract its features; The visual geometry group network is used as the backbone of the feature pyramid to extract multi-scale features of the image. For the top-down feature construction part, the spatial resolution is upsampled by 2 times, and then the upsampled image is added and merged element by element with the corresponding bottom-up image. Step 3: Each scale feature obtained in step 2 is processed by two different pooling methods, and the features processed at the same scale are fused to obtain the final fused features; Step 4: The fused features in step 3 are regressed through the fully connected layer to predict the quality scores of all scales, and the mean of the predicted values of all scales is used as the objective score of the point cloud quality.
2. The point cloud quality evaluation algorithm according to claim 1, characterized in that: The projection processing of the distorted point cloud in step 1 refers to using a preset rotation matrix R and different rotation angles to perform six-view projection processing on the distorted point cloud to obtain six two-dimensional projection images of the distorted point cloud at different view angles. Assuming that the coordinates of the distorted point cloud are (0 x ,O y ,O z ), the coordinates after the rotation matrix R and the rotation angle are (P x ,P y ,P z ), then the rotation operation is expressed as: R=R top R bottom R left R right R front R back Among them, formulas (1)(2), (3)(4), (5)(6) respectively represent the top view, bottom view, left view, right view, front view and back view obtained by counterclockwise rotation around the X-axis, Y-axis and Z-axis, and formula (7) represents the distorted point cloud after the rotation matrix.
3. The point cloud quality evaluation algorithm according to claim 1 or 2, characterized in that: In step 2, the two-dimensional projection images of the six perspectives are stacked into a three-dimensional image block as the input of the feature extraction unit, and multi-scale features are extracted using a feature pyramid network. Assuming that the size of the two-dimensional projection image is M×N, the stacking operation is expressed as: Among them, M and N represent the length and width of the projection image. V represents the pixel value located in the i-th row and j-th column of the k-th perspective projection image. ijk Represents the pixel value at row i, column j, and layer k in a three-dimensional block, where i∈(1,2,3,...,M) and j∈(1,2,3,...,N).
4. The point cloud quality evaluation algorithm according to claim 1 or 2, characterized in that: In step 2, the three-dimensional image block is represented as I, and the features of the image block are upsampled to obtain I k , the top-down feature construction process is expressed as: in, Represents the value of the feature of the kth layer at position (i, j) in the top-down path, and Upsample represents the upsampling operation; The bottom-up feature construction process is expressed as: in, It represents the value of the feature of the k-1th layer in the bottom-up path at position (i, j), and Downsample represents the downsampling operation; Multi-scale feature fusion is expressed as:
5. The point cloud quality evaluation algorithm according to claim 1 or 2, characterized in that: In step 3, the feature P of each pyramid level is k As input, two pooling operations are performed and the pooled features are fused to obtain the fused features. Assume that the size of each feature is H×W×C, and the window size of the pooling operation is m×m, where H represents the height, W represents the width, C represents the number of channels, and k represents the pyramid level; The mean pooling operation is expressed as: The maximum pooling operation is expressed as: P k max =max i,j (P k [j:j+m,i:i+m]) (13) Among them, P k [j:j+m,i:i+m] represents feature P k The area of size m×m with position (i,j) as the upper left corner; After the mean and maximum pooling operations, the feature P is obtained k avg , P k max Perform feature fusion to obtain the fused feature P k fusion , and its calculation formula is as follows: P k fusion =β·P k avg +γ·P k max (14) Among them, β and γ are weight parameters, β>0, γ>0, β+γ=1, which are used to control the importance of features under different pooling methods.
6. The point cloud quality evaluation algorithm according to claim 1 or 2, characterized in that: In step 4, the fusion feature As input, it is flattened to convert it into a one-dimensional vector. The flattening operation is expressed as: where p k Represents the flattened feature vector, and Flatten represents the flattening operation; The p k The regression operation is performed through the fully connected layer. Assuming that the output dimension of the fully connected layer is 1, it means predicting a point cloud quality value. The regression operation is expressed as: z k =W·p k +b (16) Among them, p k Represents the input features through the fully connected layer, z k The output after linear transformation, W represents the weight matrix of the fully connected layer, and b represents the bias vector of the fully connected layer; After passing through the fully connected layer, the ReLU activation function is used to obtain the predicted value Q of the kth level k ; The specific operation is expressed as: Q k =ReLU(from k ) (17) Finally, the predicted values of all levels n are averaged as the objective score Q of the entire point cloud; The resulting comprehensive score is expressed as:
