A point cloud quality assessment method based on optimal view selection and multimodal fusion

Abstract

The present invention discloses a point cloud quality assessment method based on optimal view selection and multimodal fusion, and belongs to the technical field of three-dimensional point cloud quality assessment. The method predicts four optimal view angles of the point cloud through an optimal view selection network, and projects along the four optimal view angles to obtain a 2D projection image of the point cloud; then the point cloud is divided into multiple patches; next, the 2D projection image and the patch are feature extracted through an image encoder and a point encoder; then the channel attention mechanism is used to model and fuse multimodal quality perception information, and finally the final score of the point cloud is obtained through a quality regression network. The present invention takes into account the 2D structural features of the point cloud, as well as the 3D spatial information, and is innovative and feasible.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud quality assessment, and in particular, relates to a point cloud quality assessment method based on optimal view angle selection and multimodal fusion. Background Art

[0002] As one of the mainstream formats for representing 3D models or scenes, point cloud has been widely used in augmented reality and virtual reality, interactive multimedia, autonomous driving, games and other fields. However, point cloud is prone to distortion and deformation during the acquisition, compression and transmission process. These distortions not only lead to data errors, but also affect the user's visual perception in related applications. Therefore, point cloud quality assessment (PCQA) is crucial to ensure the quality of user experience.

[0003] According to the extent of using reference point cloud in the quality assessment process, point cloud objective quality assessment can be divided into full-reference quality assessment, semi-reference quality assessment and no-reference quality assessment. However, in practical applications, all or part of the information of the original point cloud is usually unavailable, so no-reference quality assessment methods that only use distorted point cloud have been widely studied. No-reference quality assessment methods can be further divided into point-based methods and projection-based methods.

[0004] Point-based methods directly extract point-by-point features from 3D models, but since point clouds are usually very complex and consist of a large number of points, it is difficult to develop point-based PCQA methods and it is difficult to effectively extract quality-aware features. Projection-based methods first project point clouds into 2D space from multiple viewpoints, and then apply traditional image quality assessment (IQA) methods to the generated projected images to obtain the final quality score. 2D images contain rich texture and semantic information, but are highly dependent on viewpoints; 3D point clouds are more sensitive to geometric deformations and are viewpoint invariant. However, existing reference-free point cloud quality assessment methods do not fully combine the information of 3D point clouds and 2D projections, resulting in unsatisfactory performance.

[0005] Therefore, how to use multimodal information to evaluate point cloud quality is a key issue that needs to be solved urgently.

[0006] The channel attention mechanism is to dynamically adjust the importance of each channel by weighting different channels of the feature map, thereby improving the feature expression ability. In the process of point cloud and image fusion, different channels may contain different types of useful information, which can be effectively captured and strengthened through the channel attention mechanism. Therefore, multimodal fusion learning is feasible.

[0007] The optimal perspective selection network provides a 2D point cloud representation for point cloud multimodal fusion learning, providing the network with richer geometric information and texture details. Therefore, the optimal perspective helps to improve the accuracy and reliability of point cloud quality assessment. Summary of the invention

[0008] The purpose of the present invention is to provide a point cloud quality assessment method based on optimal view selection and multimodal fusion. The method predicts the four best view angles of the point cloud through the optimal view selection network, and projects along the four best view angles to obtain the 2D projection image of the point cloud; then the point cloud is divided into multiple patches; next, the 2D projection image and the patch are feature extracted through the image encoder and the point encoder; then the channel attention mechanism is used to model and fuse the multimodal quality perception information, and finally the final score of the point cloud is obtained through the quality regression network.

[0009] To achieve the above object, the present invention provides a point cloud quality assessment method based on optimal view selection and multimodal fusion, comprising the following steps:

[0010] S1: Sample 1000 views of the reference point cloud, calculate the four view quality indicators for each view, and select the view with the highest view quality indicator as the ground truth;

[0011] S2: The distorted point cloud passes through the optimal view selection network to obtain the four optimal viewpoints of the distorted point cloud;

[0012] S3: Obtain 2D projection images of the point clouds corresponding to the four optimal viewing angles;

[0013] S4: Divide the distorted point cloud into multiple patches;

[0014] S5: extract features from the point cloud 2D projection image and the patch through their respective feature extraction branch networks to obtain the point cloud 2D projection image and patch features;

[0015] S6: Perform linear transformation on the point cloud 2D projection image and patch features;

[0016] S7: Use the channel attention mechanism to fuse the features after linear transformation;

[0017] S8: The fused features are passed through the quality regression network to obtain the score of the distorted point cloud.

[0018] Further, in step S1, the specific steps are as follows:

[0019] S11: Use the Fibonacci spherical sampling method to sample the reference point cloud P′ on the unit sphere S 2 Uniformly sample 1000 views

[0020] S12: For each sampled view, after rendering the point cloud into a 2D image, four view quality indicators are calculated for each view: viewpoint entropy (VE), visibility ratio (VR), view KL divergence (VKL), and view mutual information (VMI). Then, the view quality indicators are normalized to the range of [0,1], and the view with the highest view quality indicator is selected as the groundtruth.

[0021] Further, in step S2, the specific steps are as follows:

[0022] S21: Monte Carlo convolutional neural network (MCCNN) is used to extract features from the distorted point cloud. The feature extraction network has four convolutional layers, and the convolution radius of each layer is 0.05, 0.2, 0.3 and Finally, a 2048-dimensional feature representation is obtained;

[0023] S22: Through four parallel multi-layer perceptrons (MLP), each perceptron has three layers, the size of which is 1024, 256 and 3 respectively. Each perceptron outputs a view vector

[0024] S23: Calculate the cosine similarity loss between the view predicted by each MLP network and the true view.

[0025] Further, in step S3, the Open3D package is used to obtain the four viewing vectors obtained in step S22. The point cloud is rendered and the corresponding point cloud 2D projection images (I1, I2, I3, I4) are generated.

[0026] Further, in step S4, the distorted point cloud P is sampled by the farthest point to obtain N δ For each anchor point, use the nearest neighbor algorithm to find N s neighboring points and use them to construct a local patch. δ =6, N s =2048.

[0027] Further, in step S5, the specific steps are as follows:

[0028] S51: The 2D projection image of the point cloud obtained in S3 is passed through ResNet50 for feature extraction to obtain the 2D feature representation of the point cloud

[0029] S52: The patch obtained in S4 is passed through PointNet++ for feature extraction to obtain the 3D feature representation of the point cloud

[0030] Further, in step S6, the specific steps are as follows:

[0031] S61: 2D feature representation of point cloud The eigenvectors are obtained by averaging each element At the same time, the point cloud 3D feature representation The eigenvector is obtained by averaging each element

[0032] S62: The feature vector and Perform linear projection operation and project to the same dimension, which are recorded as and

[0033] Further, in step S7, the channel attention mechanism is used to fuse the and Integrate multi-modal quality perception information to generate a fused feature vector. The specific steps are as follows:

[0034] S71: and Input Squeeze and Concat (SPC) module to obtain the multi-scale feature map F in the channel direction SPC ;

[0035] S72: Use the SEWeight module to extract the attention of feature maps of different scales and obtain the channel-wise attention vector W SE ;

[0036] S73: Recalibrate the attention vector in the channel direction through the Softmax function to obtain the recalibrated weight W′ of the multi-scale channel SE ;

[0037] S74: Multiply the recalibrated weights by the corresponding feature maps element by element to obtain multi-scale feature information F refined .

[0038] Furthermore, in step S8, the fused feature vector obtained in S7 is input into the quality regression network to obtain the final quality score. The specific steps are as follows:

[0039] S81: Use two fully connected layers to transform the quality feature F refined Regression to predict quality score;

[0040] S82: For the quality assessment task, we not only focus on the accuracy of the predicted quality value, but also attach importance to the quality ranking. Therefore, the adopted loss function includes two parts: mean square error (MSE) and rank error.

[0041] The advantages of the present invention compared with the prior art are:

[0042] (1) This paper proposes a reference-free point cloud quality assessment method based on multimodal feature learning, which takes into account both the 2D structural features of the point cloud and the 3D spatial information.

[0043] (2) The present invention obtains a novel method for realizing 2D representation of point clouds through the optimal view selection method. By selecting the view with the highest view quality index, the three-dimensional information of the point cloud can be effectively converted into a two-dimensional representation. This representation not only retains the geometric characteristics of the point cloud, but also enhances the visual interpretability.

[0044] (3) The present invention combines the channel attention mechanism to weight different channels of the feature map during the multimodal feature fusion process and dynamically adjust the importance of each channel. This improves the feature expression capability by effectively capturing and enhancing the useful information of point clouds and images.

[0045] To some extent, other advantages and objectives of the present invention will be described in the following description. And through the following examination and study, the advantages of the present invention will be obvious to those skilled in the art. The objectives and other advantages of the present invention can be achieved and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the overall technical roadmap of the point cloud quality assessment method based on optimal view selection and multimodal fusion of the present invention;

[0047] Figure 2 Schematic diagram of the process of sampling the point cloud perspective in the present invention;

[0048] Figure 3 Schematic diagram of the channel attention mechanism module in the present invention;

[0049] Figure 4 Schematic diagram of the mass regression network in the present invention. DETAILED DESCRIPTION

[0050] The present invention is further described in detail below with reference to the accompanying drawings and examples.

[0051] S1: Sample 1000 views of the reference point cloud P′, calculate the four view quality indicators for each view, and select the view with the highest view quality indicator as the ground truth;

[0052] S11: For the reference point cloud P′, use the Fibonacci spherical sampling method to sample the point cloud P′ on the unit sphere S 2Uniformly sample 1000 views like Figure 2 As shown;

[0053] S12: For each sampling angle v i ∈V, after rendering the point cloud into a 2D image using the Open3D package, four view quality indicators are calculated for each view, namely, viewpoint entropy (VE), visibility ratio (VR), viewpoint KL divergence (VKL), and viewpoint mutual information (VMI).

[0054] When calculating the view quality index, a reference view v0 is introduced. v0 is an arbitrarily selected view from the uniformly sampled view set V, which is used to calculate the view KL divergence (VKL) and view mutual information (VMI). The calculation formulas for the four view quality indicators are as follows:

[0055]

[0056] Among them, p ij is the viewing angle v i The probability distribution of the jth pixel, M is the total number of pixels.

[0057]

[0058] Among them, N vis (v i ) represents the viewing angle v i The number of visible point cloud points, N total Represents the number of points in the point cloud.

[0059]

[0060] Among them, p ij is the viewing angle v i The probability distribution at the jth pixel, p 0j is the probability distribution of the jth pixel under the reference view angle.

[0061]

[0062] Among them, p ij is the viewing angle v i The probability distribution at the jth pixel, q j is the average probability distribution of the j-th pixel under the reference view angle.

[0063] Then, the view quality index is normalized to the range of [0,1], where 0 and 1 refer to the view quality of the worst and best views, respectively.

[0064]

[0065] Where Q(v) represents the visual quality index under the viewing angle, including VE′(v), VR′(v), VKL′(v), VMI′(v). min(Q) and max(Q) are the minimum and maximum values ​​of all visual quality indicators, respectively.

[0066] Select the view with the highest view quality index as the ground truth:

[0067]

[0068] in, Represents the groundtruth with four indicators: VE, VR, VKL, and VMI.

[0069] S2: The distorted point cloud P passes through the optimal perspective selection network to obtain the four optimal perspectives of the distorted point cloud

[0070] S21: Use Monte Carlo convolutional neural network (MCCNN) to extract features from the distorted point cloud P. The feature extraction network has four convolutional layers, and the convolution radius of each layer is 0.05, 0.2, 0.3 and Finally, we get a 2048-dimensional feature representation F v ;

[0071] S22: Through four parallel multi-layer perceptrons (MLP), each perceptron has three layers, the size of which is 1024, 256 and 3 respectively. Each perceptron outputs a view vector

[0072] S23: Calculate the cosine similarity loss between the perspective predicted by each MLP network and the true perspective. The cosine similarity loss is defined as:

[0073]

[0074] in, is the predicted view vector, v * is the real view vector.

[0075] For four different view quality indicators, the losses are:

[0076]

[0077] The losses of the four MLP networks are weighted summed as the total loss of the network:

[0078]

[0079] Among them, ω VE ,ω VR ,ω VKL,ω VMI is the weight coefficient.

[0080] S3: Use the Open3D package to follow the four view vectors obtained in S22 The point cloud is rendered and the corresponding point cloud 2D projection images (I1, I2, I3, I4) are generated.

[0081] S4: For the distorted point cloud P, the farthest point sampling is used to obtain N δ For each anchor point, use the nearest neighbor algorithm to find N s neighboring points and use them to construct a local patch.

[0082]

[0083] Where S is a patch, KNN(·) represents the KNN operation, and δ m represents the mth farthest sampling point, N δ =6, N s =2048.

[0084] S5: extract features from the point cloud 2D projection image and the patch through their respective feature extraction branch networks to obtain the point cloud 2D projection image and patch features;

[0085] S51: The 2D projection image of the point cloud obtained in S3 is passed through ResNet50 for feature extraction to obtain the 2D feature representation of the point cloud

[0086]

[0087] Among them, θ I (·) represents image feature encoder: ResNet50.

[0088] S52: The patch obtained in S4 is passed through PointNet++ for feature extraction to obtain the 3D feature representation of the point cloud

[0089]

[0090] Among them, θ P (·)Point cloud encoder θ P (·): PointNet++.

[0091] S6: Perform linear transformation on the point cloud 2D projection image and patch features;

[0092] S61: 2D feature representation of point cloud The eigenvectors are obtained by averaging each element

[0093]

[0094] in, represents the t-th image projection I t The quality-aware embedding of C I Denotes the 2D image encoder θ I (·) the number of output channels, is the result after average fusion.

[0095] At the same time, the point cloud 3D feature representation The eigenvector is obtained by averaging each element

[0096]

[0097] in, represents the lth patch S l The quality-aware embedding of C P Represents the point cloud encoder θ P (·) the number of output channels, The average fusion result.

[0098] S62: The feature vector and Perform linear projection operation and project to the same dimension, which are recorded as and

[0099]

[0100] in, and is the adjusted feature, W P and W I is a learnable linear mapping, and C′ is the number of adjusted channels.

[0101] S7: Use the channel attention mechanism to fuse the features after linear transformation, such as Figure 3 As shown;

[0102] Use channel attention mechanism to fuse S62 and In order to integrate multimodal quality perception information, a fused feature vector is generated.

[0103] S71: First and Enter the Squeeze and Concat (SPC) module to obtain a multi-scale feature map in the channel direction:

[0104]

[0105] S72: Use the SEWeight module to extract the attention of feature maps of different scales and obtain the attention vector in the channel direction:

[0106] W SE =σ(W2δ(W1F SPC )

[0107] Among them, σ represents the sigmoid function, δ represents the ReLU function, and W1 and W2 are learnable weight matrices.

[0108] S73: Recalibrate the attention vector in the channel direction through the Softmax function to obtain the recalibrated weights of the multi-scale channels:

[0109] W′ SE =Softmax(W SE )

[0110] S74: Multiply the recalibrated weights by the corresponding feature maps element by element to obtain multi-scale feature information:

[0111] F +efined =W′ SE ⊙F SPC

[0112] Among them, F refined Represents the feature vector after channel attention learning.

[0113] S8: The fused features are passed through the quality regression network to obtain the score of the distorted point cloud, such as Figure 4 As shown;

[0114] The fused feature vector obtained in S7 is input into the quality regression network to obtain the final quality score. The specific steps are as follows:

[0115] S81: Use two fully connected layers to transform the quality feature F refined Regression to predict quality score:

[0116] Q η =FC2(FC1(F refined ))

[0117] Among them, Q η is the predicted quality score, FC1 and FC2 represent the first and second fully connected layers, respectively.

[0118] S82: For the quality assessment task, we not only focus on the accuracy of the predicted quality value, but also attach importance to the quality ranking. Therefore, the adopted loss function includes two parts: mean square error (MSE) and rank error.

[0119] MSE is used to make the predicted value close to the quality label, which can be derived as:

[0120]

[0121] Among them, Q η is the predicted quality score, Q′ η is the quality label of the point cloud and n is the size of the mini-batch.

[0122] When point clouds have close quality labels, Rank loss can better help the model distinguish quality differences. To this end, a differentiable ranking function is used to approximate Rank loss:

[0123]

[0124] Among them, i and j are the corresponding indices of the two point clouds in the mini-batch, and the Rank loss can be expressed as:

[0125]

[0126] The loss function is the weighted sum of MSE loss and Rank loss:

[0127] Loss = λ1L MSE +λ2L rank

[0128] Among them, λ1 and λ2 are used to adjust the proportion of MSE loss and Rank loss in the loss function.

Claims

1. A point cloud quality assessment method based on optimal view selection and multimodal fusion, characterized in that The steps are as follows: S1: Use the Fibonacci spherical sampling method to sample the reference point cloud P′ on the unit sphere S 2 Uniformly sample 1000 viewing angles V = {v1,v2,v3,v4,…,v 1000 } ; For each sampled view, after rendering the point cloud into a 2D image, four view quality indicators are calculated for each view: viewpoint entropy, visibility ratio, view KL divergence, and view mutual information; Then, the view quality index is normalized to the range of [0,1], and the view with the highest view quality index is selected as the groundtruth; S2: Monte Carlo convolutional neural network is used to extract features from the distorted point cloud. The feature extraction network has four convolutional layers, and the convolution radius of each layer is 0.05, 0.2, 0.3 and Finally, a 2048-dimensional feature representation is obtained; through four parallel multi-layer perceptrons, each perceptron has three layers, the sizes are 1024, 256 and 3 respectively, and each perceptron outputs a view vector i=1,2,3,4; calculate the cosine similarity loss between the perspective predicted by each MLP network and the true perspective, and obtain the four optimal perspectives of the distorted point cloud; S3: Use the Open3D package to follow the four viewing vectors obtained in step S2 Render the point cloud and generate the corresponding point cloud 2D projection images (I1, I2, I3, I4); S4: For the distorted point cloud P, use the farthest point sampling to obtain N δ For each anchor point, use the nearest neighbor algorithm to find N s neighboring points, and use them to construct a local patch; where N δ =6, N s =2048; S5: The 2D projection image of the point cloud obtained in S3 is passed through ResNet50 for feature extraction to obtain the 2D feature representation of the point cloud The patches obtained in S4 are passed through PointNet++ for feature extraction to obtain the 3D feature representation of the point cloud. S6: 2D feature representation of point cloud The eigenvectors are obtained by averaging each element At the same time, the point cloud 3D feature representation The eigenvector is obtained by averaging each element The feature vector and Perform linear projection operation and project to the same dimension, which are recorded as and S7: and Input Squeeze and Concat module to obtain the multi-scale feature map F in the channel direction SPC ; Use the SEWeight module to extract the attention of feature maps of different scales and obtain the attention vector W in the channel direction SE ; The attention vector in the channel direction is recalibrated through the Softmax function to obtain the recalibrated weight W′ of the multi-scale channel SE ; Multiply the recalibrated weights by the corresponding feature maps element by element to obtain the multi-scale feature information F refined ; S8: Use two fully connected layers to transform the quality feature F refined Regression is to predict the quality score; for the quality assessment task, we not only focus on the accuracy of the predicted quality value, but also attach importance to the quality ranking, so the loss function used includes two parts: mean square error and rank error.

Images