A multi-task learning point cloud quality no-reference evaluation method

Abstract

The application provides a multi-task learning point cloud quality no-reference evaluation method, and belongs to the field of three-dimensional point cloud quality evaluation. The scheme comprises a data preprocessing step, a multi-scale feature extraction step, a feature fusion step and a multi-task learning step. First, for a given point cloud, a six-way orthogonal projection method is used to map it to six standard orthogonal viewing angles, and three types of 2D images are generated for each viewing angle: texture maps, depth maps and placeholder maps. Then, the texture maps, depth maps and placeholder maps under the six viewing angles are sent to a multi-scale feature extraction unit, and a feature pyramid network is used to extract cross-scale features. Then, the features of different scales are fused. Finally, the fused features are sent to a multi-task learning network, with quality score regression prediction as the main task and distortion type classification as the auxiliary task. Through joint optimization for multi-task training, the score of the point cloud quality is finally output.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of 3D data processing and computer vision, specifically, to a multi-task learning-based point cloud quality evaluation method without reference. Background Technology

[0002] With the continuous advancement of 3D technology, computer vision, and autonomous driving technology, point clouds, due to their high-fidelity representation of the shape features and 3D positioning of target objects, are widely used in industries such as augmented reality, autonomous navigation, intelligent manufacturing quality monitoring, and digital preservation of historical buildings. However, problems such as decreased signal-to-noise ratio, data loss, morphological distortion, and coordinate offset during the acquisition, transmission, encoding, and analysis of point cloud data often directly lead to deviations in downstream applications. For example, autonomous driving systems may create safety hazards due to incorrect obstacle recognition, and industrial quality inspection may result in errors in determining component specifications. Therefore, the quality of point clouds has become a key prerequisite for ensuring the reliability of related technology applications, making point cloud quality evaluation research of great significance.

[0003] Currently, there are two main methods for point cloud quality assessment: full-reference assessment and no-reference assessment. No-reference assessment methods do not require any reference point cloud information; they directly analyze the characteristics of the distorted point cloud itself for quality assessment, and are therefore widely used in practical scenarios. Current no-reference point cloud quality assessment methods are mainly divided into two categories: 3D point cloud-based methods and projection-based methods. 3D-based methods extract geometric features of the point cloud, such as local curvature, point density distribution, and normal vector changes, and then find the correspondence between these features and the degree of distortion. Projection-based methods convert the 3D point cloud into a 2D image for processing, leveraging mature 2D domain technologies to avoid the complexity of directly analyzing 3D point clouds. Simultaneously, they comprehensively preserve key point cloud information through multi-view and multi-type projections, thereby achieving accurate quality assessment.

[0004] Current point cloud quality evaluation methods rely solely on texture maps as a single projection when projecting 3D point clouds into 2D images. This approach is prone to insufficient feature representation due to viewpoint occlusion or information loss. In contrast, multi-type, multi-view projection methods employ a multi-type projection strategy of "depth map + texture map + placeholder map" combined with six-view transformation. This approach can fully preserve the 3D geometric attributes and surface texture features of point clouds, thereby improving the comprehensiveness and robustness of feature representation.

[0005] Existing point cloud quality assessment methods focus solely on the task of "quality score regression," neglecting the relationship between quality score and distortion type classification. Since the impact of distortion type classification on quality score is non-linear, single-task learning struggles to effectively capture this information. For example, missing points can cause structural damage, leading to a sharp drop in quality score; while slight geometric noise may have a smaller impact. Therefore, identifying distortion types helps in more accurately estimating quality scores. Single-task learning cannot effectively utilize this correlation information to optimize the model, resulting in insufficient generalization ability in complex distortion scenarios and unstable evaluation results.

[0006] In summary, existing no-reference point cloud quality assessment methods have shortcomings in feature extraction, multi-task learning, and multi-view information utilization, making it difficult to meet the requirements of assessment accuracy, generalization ability, and real-time performance. Therefore, a method is needed that can fully extract multi-scale and multi-type features and improve point cloud quality assessment performance through multi-task learning. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing no-reference point cloud quality assessment methods, such as insufficient feature extraction and inadequate single-task learning, and to provide a multi-task learning-based no-reference point cloud quality assessment method. This method projects the point cloud into six viewpoints, along with texture maps, depth maps, and placeholder maps for each viewpoint. It then utilizes a feature pyramid network to extract multi-scale features and combines this with collaborative learning of the quality score regression main task and the distortion type classification auxiliary task to achieve quality assessment of no-reference point clouds.

[0008] This invention provides a referenceless point cloud quality evaluation method for multi-task learning, comprising: Step 1: The distorted point cloud is subjected to a six-term orthogonal projection method to obtain two-dimensional projection images of the distorted point cloud from six different perspectives. Each perspective contains three different types of images: texture image, depth image and placeholder image. The projection images from different perspectives are stacked into image blocks. Step 2: Based on the projection map in Step 1, construct a neural network and extract its features; use the visual geometry network as the backbone network of the feature pyramid to extract multi-scale features of the image; in the top-down feature construction stage, first upsample the spatial resolution to twice the original, and then add and fuse the upsampled feature map with the corresponding bottom-up feature map element by element. Step 3: The features obtained in Step 2 are processed by average pooling and max pooling in sequence to fuse the features at different scales and obtain the final fused features. Step 4: The fused features from Step 3 are fed into a multi-task learning network, where they undergo quality score regression through a double fully connected layer and distortion type classification through a fully connected layer.

[0009] The point cloud quality evaluation method without reference proposed in this invention belongs to the field of 3D point cloud quality evaluation. The scheme includes a data preprocessing step, a multi-scale feature extraction step, a feature fusion step, and a multi-task learning step. First, for a given point cloud, a six-way orthogonal projection method is used to map it to six standard orthogonal viewpoints, generating three types of 2D images for each viewpoint: a texture map, a depth map, and a placeholder map. Then, the texture maps, depth maps, and placeholder maps from the six viewpoints are fed into a multi-scale feature extraction unit, where a feature pyramid network is used to extract cross-scale features. Next, the features from different scales are fused. Finally, the fused features are fed into a multi-task learning network, with quality score regression prediction as the primary task and distortion type classification as an auxiliary task. Multi-task training is performed through joint optimization, ultimately outputting a point cloud quality score. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the point cloud quality method of the present invention. Detailed Implementation

[0011] The point cloud quality evaluation algorithm of the present invention will be further explained below with reference to the accompanying drawings.

[0012] As shown in the figure, the image quality assessment algorithm of the present invention includes four steps: step 1 data preprocessing; step 2 multi-scale feature extraction; step 3 feature fusion; and step 4 multi-task learning.

[0013] Step 1, Data Preprocessing The feature is that, given a point cloud: (1), in This represents the coordinates of the nth point in the spatial point cloud. This represents the texture feature corresponding to that point. Projecting the distorted point cloud involves using a preset rotation matrix and different rotation angles to project the distorted point cloud from six different perspectives, obtaining 2D projected images from six different viewpoints. These six perspectives are respectively... axis, axis, The top, bottom, left, right, front, and back views are obtained by rotating the axis counterclockwise. From one perspective, the point is first transformed from the world coordinate system to the camera coordinate system using camera extrinsics. The transformation process can be represented as follows: (2), in and The first The rotation matrix and translation vector corresponding to each viewpoint This represents the point in the camera coordinate system. Then, an orthogonal photogrammetry model is used to project the 3D point onto the 2D image plane, obtaining the projected coordinates. At the same time, the camera coordinate system Quantity This serves as the corresponding depth information. Based on the projection results described above, points are rasterized on the image plane for any pixel location. Select several projection points within its neighborhood and calculate the first projection point. Distance from each candidate point to the pixel center: (3), Based on this distance, a predefined attenuation function is applied. Calculate the coverage intensity of a point to a pixel. ,in The weights are obtained by normalizing the coverage intensity of all candidate points: (4), Using the aforementioned weights, the point attributes at each pixel are weighted and fused to generate a texture map and a depth map: (5), in Indicates the first The color value at pixel (u,v) from each viewpoint.

[0014] (6), in This represents the weighted depth information corresponding to the pixel, used to characterize the spatial structure of the point set. Simultaneously, a placeholder map is constructed based on whether the pixel is covered by any other point, obtained by thresholding the alpha value at the pixel. (7), in For indicator functions, This is a preset threshold. When a pixel is covered by at least one point, the placeholder image is set to 1; otherwise, it is set to 0.

[0015] Finally, the texture map and depth map are linearly normalized to map their numerical range to [0,1], thereby obtaining the texture map of the point set X at each viewpoint. Depth map and placeholder images Three multi-viewpoint two-dimensional representations. Ultimately, the total set of six viewpoints and the three image types within each viewpoint is: (8), Through the above steps, 6 views × 3 types = 18 2D images can be generated for each point cloud to be evaluated, which fully covers the geometric, texture and density information of the point cloud, providing a rich data source for subsequent feature extraction.

[0016] Step 2, Feature Extraction The six-view images of three types generated by data preprocessing are stitched together and fed into a multi-scale feature extraction unit. Each extraction unit is built based on a Feature Pyramid Network (FPN) to extract features at different scales.

[0017] Assume the size of the 2D projection is By stacking three types of images from six different perspectives, a three-dimensional image block representation can be formed as follows: (9), in, Represents the position of the 3D image block located at the th line, number Column, No. Layer pixel values.

[0018] The three-dimensional image block As input, a multi-scale feature representation is constructed through a feature pyramid network. Let the FPN be... The features of a layer are represented as .

[0019] (10), in, Indicates the first in the top-to-bottom path The features of the layer are in the location The value at that location, This represents an upsampling operation; the bottom-up feature construction process is represented as: (11), in, Indicates the first in the bottom-up path The feature map of the layer at location The value at that location, This indicates a downsampling operation.

[0020] Multi-scale feature fusion is represented as: (12) in This is the multi-scale representation after fusion.

[0021] Step 3, Feature Fusion The feature fusion section will combine the features extracted in the previous step As input, the features are further aggregated through mean pooling and max pooling operations. Let the scale of the input features be... The window size for pooling operations is ,in Indicates altitude, Indicates width, Indicates the number of channels.

[0022] Mean pooling is represented as: (13) The max pooling operation is represented as: (14) in, Representation of feature map China-Israel position The size of the top left corner is The area; The pooling results are then fused together using the mean and max pooling methods to obtain the final pooling feature: (15) in, Weight parameters, , This is used to control the importance of features under different pooling methods.

[0023] Step 4, Multi-task learning The multi-task feature regression part uses "quality score regression" as the main task and "distortion type classification" as the auxiliary task to construct a multi-task learning model. The evaluation accuracy of the main task is improved through collaborative optimization among tasks. The specific process is as follows: Fusion features Perform a flattening operation to convert it into a one-dimensional vector. (16) For the quality score regression task, the feature vector Input to a fully connected regression layer, output the predicted point cloud quality score: (17) in, and These are the weight matrix and bias vector for the regression task, respectively. Indicates the first The quality score predicted for each sample.

[0024] For distortion type classification tasks, the same applies to feature vectors. As input, it is fed into a fully connected classification layer, and after passing through a Softmax activation function, the distribution of each distortion type is output: (18), in, Indicates the first The predicted vectors for each sample belong to various distortion types, and the total number of distortion types is... .

[0025] For quality score regression, the SmoothL1Loss loss function is used, which is defined as: (19), in, This represents the number of samples for the quality score regression task. It is the first The true value of the quality score of each sample. SmoothL1Loss can approach L2 loss when the error is small and L1 loss when the error is large. It is more robust to outliers and is suitable for loss calculation of samples with different error magnitudes in regression tasks.

[0026] For distortion type classification tasks, the FocalLoss loss function is used. Its formula is: (20), here, Indicates the first Each sample has its true distortion category The predicted probability. It is a class balancing factor, used to balance the uneven number of samples in different classes, giving more attention to the minority class; It is a modulation factor used to reduce the loss weight of easily classified samples, making the model pay more attention to difficult-to-classify samples.

[0027] Ultimately, the total loss function for multiple tasks The weighted sum of the two tasks is expressed as follows: (twenty one), in, The loss weight coefficients for the quality score regression task are used to balance the contribution ratios of the main task and auxiliary task in the overall training process.

[0028] The algorithm's performance was validated using the SJTU-PCQA and WPC databases. Four evaluation criteria were used to measure the prediction accuracy and monotonicity of the proposed model: Pearson linear correlation coefficient (PLCC), Spearman rank-order correlation coefficient (SROCC), Kendall rank-order correlation coefficient (KROCC), and root mean squared error (RMSE). PLCC, SROCC, and KROCC range from [-1, 1], with values ​​closer to 1 indicating stronger correlation and higher consistency. RMSE ranges from [0, 1]. The closer the value is to 0, the better the prediction performance.

[0029] Table 1. Performance comparison with other advanced methods on the SJTU-PCQA database. Table 2. Performance comparison with other advanced methods on the WPC database. As shown in Table 1, our method achieves the highest scores on the SJTU-PCQA database for both monotonicity evaluation metrics SROCC and KROCC, and also the highest scores on the SJTU-PCQA database for both prediction accuracy metrics PLCC and RMSE. Table 2 shows that our method achieves the highest scores on the WPC database for both monotonicity evaluation metrics SROCC and KROCC, and for both prediction accuracy metrics PLCC and RMSE, our method's PLCC value is closest to 1, while its RESS value is relatively low. In summary, our proposed method achieves significant results in point cloud quality evaluation and maintains good consistency with subjective human evaluation.

Claims

1. A point cloud quality evaluation method without reference for multi-task learning, characterized in that, include: Step 1: The distorted point cloud is subjected to a six-term orthogonal projection method to obtain two-dimensional projection images of the distorted point cloud from six different perspectives. Each perspective contains three different types of images: texture image, depth image and placeholder image. The projection images from different perspectives are stacked into image blocks. Step 2: Based on the projection map in Step 1, construct a neural network and extract its features; use the visual geometry network as the backbone network of the feature pyramid to extract multi-scale features of the image; in the top-down feature construction stage, first upsample the spatial resolution to twice the original, and then add and fuse the upsampled feature map with the corresponding bottom-up feature map element by element. Step 3: The features obtained in Step 2 are processed by average pooling and max pooling in sequence to fuse the features at different scales and obtain the final fused features. Step 4: The fused features from Step 3 are fed into a multi-task learning network, where they undergo quality score regression through a double fully connected layer and distortion type classification through a fully connected layer.

2. The point cloud quality evaluation method according to claim 1, characterized in that, Given a point cloud: （1）， in This represents the coordinates of the nth point in the spatial point cloud. This represents the texture feature corresponding to that point. Projecting the distorted point cloud involves using a preset rotation matrix and different rotation angles to project the distorted point cloud from six different perspectives, obtaining 2D projected images from six different viewpoints. These six perspectives are respectively... axis, axis, The top view, bottom view, left view, right view, front view, and back view obtained by rotating the axis counterclockwise, are shown in the figure. From one perspective, the point is first transformed from the world coordinate system to the camera coordinate system using camera extrinsics. The transformation process can be represented as follows: （2）， in and The first The rotation matrix and translation vector corresponding to each viewpoint To represent the point in the camera coordinate system, an orthogonal photogrammetry model is then used to project the 3D point onto the 2D image plane, obtaining the projected coordinates. At the same time, the camera coordinate system Quantity As corresponding depth information, based on the above projection results, points are rasterized on the image plane. For any pixel position... Select several projection points within its neighborhood and calculate the first projection point. Distance from each candidate point to the pixel center: （3）， Based on this distance, a predefined attenuation function is applied. Calculate the coverage intensity of a point to a pixel. ,in The weights are obtained by normalizing the coverage intensity of all candidate points: （4） ， Using the aforementioned weights, the point attributes at each pixel are weighted and fused to generate a texture map and a depth map: （5）， in Indicates the first Pixels from a single perspective The color value at that location, (6), in This represents the weighted depth information corresponding to the pixel, used to characterize the spatial structure of the point set. Simultaneously, a placeholder map is constructed based on whether the pixel is covered by any other point, obtained by thresholding the alpha value at the pixel. （7）， in For indicator functions, As a preset threshold, the placeholder image is set to 1 when a pixel is covered by at least one point, and 0 otherwise. Finally, the texture map and depth map are linearly normalized to map their numerical range to [0,1], thereby obtaining the texture map of the point set X at each viewpoint. Depth map and placeholder images Three multi-viewpoint two-dimensional representations are used, resulting in a total set of six viewpoints and three types of images under each viewpoint: （8）， Through the above steps, 6 views × 3 types = 18 2D images can be generated for each point cloud to be evaluated, which fully covers the geometric, texture and density information of the point cloud, providing a rich data source for subsequent feature extraction.

3. The point cloud quality evaluation method according to claim 2, characterized in that, The six viewpoint images of three types generated by data preprocessing are stitched together and fed into a multi-scale feature extraction unit. Each extraction unit is built based on a feature pyramid network to extract features at different scales. Assuming the size of the 2D projection image is... By stacking three types of images from six different perspectives, a 3D image block is formed, which is represented as follows: （9）， in, Represents the position of the 3D image block located at the th line, number Column, No. Layer pixel values.

4. The point cloud quality evaluation method according to claim 3, characterized in that, The three-dimensional image block As input, a multi-scale feature representation is constructed through a feature pyramid network. Let the i-th feature in the FPN be... The features of a layer are represented as , （10） ， in, Indicates the first in the top-to-bottom path The features of the layer are in the location The value at that location, This represents an upsampling operation; the bottom-up feature construction process is represented as: （11）， in, Indicates the first in the bottom-up path The feature map of the layer at location The value at that location, This indicates a downsampling operation. Multi-scale feature fusion is represented as: （12）， in This is the multi-scale representation after fusion.

5. The point cloud quality evaluation method according to claim 4, characterized in that, The features extracted in the previous step As input, the features are further aggregated through mean pooling and max pooling operations. Let the scale of the input features be... The window size for pooling operations is ,in Indicates altitude, Indicates width, Indicates the number of channels. Mean pooling is represented as: （13）， The max pooling operation is represented as: （14）， in, Representation of feature map China-Israel position The size of the top left corner is The area; The pooling results are then fused together using the mean and max pooling methods to obtain the final pooling feature: （15）， in, Weight parameters, , This is used to control the importance of features under different pooling methods.

6. The point cloud quality evaluation method according to claim 5, characterized in that, The multi-task feature regression part uses "quality score regression" as the main task and "distortion type classification" as the auxiliary task to construct a multi-task learning model. The evaluation accuracy of the main task is improved through collaborative optimization among tasks. The specific process is as follows: Fusion features Perform a flattening operation to convert it into a one-dimensional vector. （16）， For the quality score regression task, the feature vector Input to a fully connected regression layer, output the predicted point cloud quality score: (17) in, and These are the weight matrix and bias vector for the regression task, respectively. Indicates the first The quality score predicted for each sample For distortion type classification tasks, the same applies to feature vectors. As input, it is fed into a fully connected classification layer, and after passing through a Softmax activation function, the distribution of each distortion type is output: (18)， in, Indicates the first The predicted vectors for each sample belong to various distortion types, and the total number of distortion types is... , For quality score regression, the SmoothL1Loss loss function is used, which is defined as: （19)， in, This represents the number of samples for the quality score regression task. It is the first The true value of the quality score of each sample; For distortion type classification tasks, the FocalLoss loss function is used. Its formula is: (20)， here, Indicates the first Each sample has its true distortion category The predicted probability, It is a class balancing factor used to balance situations where the number of samples from different classes is uneven, giving more attention to the minority class. It is a modulation factor used to reduce the loss weight of easily classified samples, making the model pay more attention to difficult-to-classify samples. Ultimately, the total loss function for multiple tasks The weighted sum of the two tasks is expressed as follows: (21)， in, The loss weight coefficients for the quality score regression task are used to balance the contribution ratios of the main task and auxiliary task in the overall training process.

Images