Point cloud quality evaluation method and device based on three-dimensional and two-dimensional spatial complementary features

Abstract

The present application relates to the field of point cloud processing, and particularly relates to a point cloud quality evaluation method and device based on three-dimensional and two-dimensional space complementary features. The method and device comprise: extracting perception features in a point neighborhood; using the complementary characteristics of features in three-dimensional space and two-dimensional plane, mapping the point cloud from three-dimensional space to two-dimensional plane, extracting the spatial domain structural similarity features and wavelet domain quality perception features of the projection image; and obtaining the quality score of the point cloud through a regression model after fusing the features in different spaces. The present application reflects the perception quality of the point cloud through the complementary characteristics between the three-dimensional point features and the two-dimensional plane features, and can avoid the inconsistency between subjective and objective evaluation objects to a certain extent, and combines the advantages of three-dimensional point features and two-dimensional plane features to propose a relatively comprehensive point cloud objective quality evaluation method.

Description

Technical Field

[0001] This invention relates to the field of point cloud processing, and more specifically, to a method and apparatus for evaluating point cloud quality based on complementary features of three-dimensional and two-dimensional spaces. Background Technology

[0002] With the development of multimedia technology, 3D immersive media is becoming increasingly popular due to its ability to provide users with immersive visual experiences, depth perception, and interactivity. Point clouds are collections of unstructured points in three-dimensional space, where each point is represented by its geometric coordinates and physical properties, such as color, normal, and reflectance. Dynamic point clouds, being time-series representations of point clouds, have become one of the mainstream representations of 3D objects and scenes. They hold immense potential in many advanced 3D vision applications, such as virtual reality (VR), augmented reality (AR), autonomous driving, and medical-assisted diagnosis. However, distortions can occur during point cloud processing in these applications, including the generation and acquisition of 3D content, resampling, compression, transmission, and rendering, leading to a degradation in the perceived quality of point clouds by the human eye. Therefore, a Point Cloud Quality Assessment (PCQA) method is needed to simulate the human visual system, evaluate the visual quality of processed point clouds, and then use the perceived quality score as a key objective for optimizing point cloud processing. When developing encoding technologies, it is necessary to establish an effective quality assessment model to quantify distortion, thereby optimizing point cloud processing algorithms and promoting the design of more efficient codec schemes.

[0003] Xu et al. proposed a point cloud quality assessment model based on elastic potential energy similarity. First, the point cloud is represented as a set of spatial points. Then, a set of origins is extracted from the reference point cloud, and the elastic potential energy from the origins to their neighboring points is calculated. By comparing the elastic potential energy of the reference point cloud and the distorted point cloud, the degree of distortion of the point cloud can be quantified. Yang et al. proposed a graph similarity-based quality assessment method, GraphSIM, which uses graph signal theory to model the overall perception of the Human Visual System (HVS). GraphSIM uses keypoints after resampling the reference point cloud as the center of the local graph. Then, features are extracted from the local graphs of the reference point cloud and the distorted point cloud, and the graph similarity between the two is calculated to obtain the quality score of the point cloud. Zhang et al., based on GraphSIM, considered the multi-scale features of human perception and proposed a multi-scale objective quality assessment model for point clouds, MS-GraphSIM. Multi-scale representations are constructed from the local graphs extracted from the reference point cloud and the distorted point cloud. Then, the quality scores at different scales are fused to obtain the overall quality score of the point cloud.

[0004] In summary, the existing technology has the following drawbacks:

[0005] 1. Point-based evaluation methods do not consider the characteristics of the human visual system (such as brightness sensitivity), therefore, these methods typically perform poorly for certain types of point cloud distortion (such as color noise, downsampling, etc.). These methods enhance their ability to represent point cloud distortion by extracting features or feature sets; their performance depends on the extracted features, and they exhibit poor predictive performance for video-based point cloud compression (V-PCC).

[0006] 2. The evaluation method based on hexahedral projection transforms a 3D point cloud onto a 2D plane. This projection method suffers from content occlusion, which inevitably introduces some distortion during the projection process. This results in the loss of local visual information in subsequent steps, leading to inconsistency between subjective and objective evaluation objects. Summary of the Invention

[0007] This invention provides a point cloud quality assessment method and apparatus based on complementary features of three-dimensional and two-dimensional space, in order to at least solve the technical problem of low accuracy in existing point cloud quality assessments.

[0008] According to an embodiment of the present invention, a point cloud quality assessment method based on complementary features in three-dimensional and two-dimensional space is provided, comprising the following steps:

[0009] S101: Extract perceptual features from the point neighborhood;

[0010] S102: Utilizing the complementary properties of features in three-dimensional space and two-dimensional plane, point clouds are mapped from three-dimensional space to two-dimensional plane, and spatial domain structural similarity features and wavelet domain quality perception features of the projected image are extracted.

[0011] S103: After fusing features from different spaces, the quality score of the point cloud is obtained through a regression model.

[0012] Furthermore, step S101 specifically includes:

[0013] Convert the point cloud from the original RGB color space to the YUV color space;

[0014] Establish the point correspondence between the reference point cloud and the distorted point cloud;

[0015] The curvature contrast features of the reference and distorted point clouds are calculated separately as 3D geometric perception features;

[0016] By calculating the kurtosis features of local regions to capture differences in color distribution, and taking the differences in brightness as an important visual feature, we extract hue features to capture changes in color saturation in the local neighborhood of point clouds.

[0017] Further, in step S101, curvature contrast, kurtosis contrast, luminance difference and chromaticity contrast features are extracted from the three-dimensional point cloud.

[0018] Further, in step S102, spatial structure similarity features and wavelet domain contrast features are extracted from the projected image of the point cloud.

[0019] Furthermore, in step S103, the three-dimensional point features and two-dimensional planar features are used to obtain a visual perception model through multiple linear regression to evaluate the visual quality of the point cloud.

[0020] Furthermore, the brightness difference characteristics are calculated using the Euclidean distance between the brightness values ​​of the reference point and the distortion point.

[0021] Furthermore, a patch matching strategy is used to ensure consistency in the order and position of patches between the distorted point cloud and the reference point cloud.

[0022] Further, in step S101, the information content weighted SSIM method is used to extract features from the projected image.

[0023] Furthermore, in step S102, the image is first transformed from the spatial domain to the wavelet domain, and then fine-grained, medium-grained, and coarse-grained image features are extracted through different wavelet transform coefficients. The detailed information of the image at different frequencies is captured by dividing it from coarse to fine.

[0024] According to another embodiment of the present invention, a point cloud quality assessment device based on complementary features of three-dimensional and two-dimensional space is provided, comprising:

[0025] The first feature extraction module is used to extract perceptual features in the neighborhood of a point.

[0026] The second feature extraction module is used to map the point cloud from three-dimensional space to two-dimensional plane by utilizing the complementary properties of features in three-dimensional space and two-dimensional plane, and to extract the spatial domain structural similarity features and wavelet domain quality perception features of the projected image.

[0027] The quality prediction module is used to fuse features from different spaces and then obtain the quality score of the point cloud through a regression model.

[0028] A storage medium storing program files capable of implementing any of the above-mentioned point cloud quality assessment methods based on complementary features of three-dimensional and two-dimensional spaces.

[0029] A processor for running a program, wherein the program executes any of the above-mentioned point cloud quality assessment methods based on complementary features in three-dimensional and two-dimensional spaces.

[0030] The point cloud quality assessment method and apparatus based on complementary features of three-dimensional and two-dimensional space in this invention reflects the perceptual quality of point clouds through the complementary characteristics between three-dimensional point features and two-dimensional planar features, and can avoid inconsistencies between subjective and objective evaluation objects to a certain extent. By combining the visual mechanism of users watching VR content while wearing head-mounted display devices and traditional discrete statistics to construct a more accurate feature set, a relatively comprehensive objective quality assessment method for point clouds is proposed by combining the advantages of three-dimensional point features and two-dimensional planar features. Attached Figure Description

[0031] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0032] Figure 1 The flowchart shows the point cloud quality assessment method based on complementary features of three-dimensional and two-dimensional space according to the present invention.

[0033] Figure 2 This is a flowchart illustrating the point cloud quality assessment method based on complementary features of three-dimensional and two-dimensional space according to the present invention.

[0034] Figure 3 This is a block diagram of the point cloud quality assessment device based on complementary features of three-dimensional and two-dimensional space according to the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Example 1

[0038] According to an embodiment of the present invention, a point cloud quality assessment method based on complementary features in three-dimensional and two-dimensional space is provided. (See also...) Figure 1 This includes the following steps:

[0039] S101: Extract perceptual features from the point neighborhood;

[0040] S102: Utilizing the complementary properties of features in three-dimensional space and two-dimensional plane, point clouds are mapped from three-dimensional space to two-dimensional plane, and spatial domain structural similarity features and wavelet domain quality perception features of the projected image are extracted.

[0041] S103: After fusing features from different spaces, the quality score of the point cloud is obtained through a regression model.

[0042] The point cloud quality assessment method based on complementary features of three-dimensional and two-dimensional space in this embodiment of the invention reflects the perceptual quality of point clouds through the complementary characteristics between three-dimensional point features and two-dimensional planar features, and can avoid inconsistency between subjective and objective evaluation objects to a certain extent. By combining the visual mechanism of users watching VR content while wearing head-mounted display devices and traditional discrete statistics to construct a more accurate feature set, a relatively comprehensive objective quality assessment method for point clouds is proposed by combining the advantages of three-dimensional point features and two-dimensional planar features.

[0043] In this invention, to leverage the complementary properties of 3D point features and 2D planar features, a point cloud quality assessment model based on complementary features in 3D and 2D space (CF-PCQA) is proposed. This model mainly includes two modules: feature extraction and quality prediction. The feature extraction module is further divided into 3D spatial feature extraction and 2D planar feature extraction, enabling more efficient perception of the visual quality of point clouds. First, this invention extracts four perceptual features in the point neighborhood, including geometric (curvature contrast) and color-related features (Euclidean distance, chromaticity, and kurtosis of the luminance channel), to better capture geometric and color distortions. Second, this invention utilizes the complementary properties of features in 3D space and 2D plane to map the point cloud from 3D space to the 2D plane, extracting spatial domain structural similarity features of the projected image and wavelet domain quality-perceptual features, resulting in more accurate prediction performance and better robustness. Finally, the features from different spaces are fused and a regression model is used to obtain the point cloud quality score.

[0044] The purpose of this invention is to reflect the perceptual quality of point clouds by leveraging the complementary characteristics between three-dimensional point features and two-dimensional planar features, and to a certain extent avoid the inconsistency between subjective and objective evaluation objects. By combining the visual mechanism of users watching VR content while wearing head-mounted displays with traditional discrete statistics to construct a more accurate feature set, a relatively comprehensive objective quality evaluation method for point clouds is proposed by combining the advantages of three-dimensional point features and two-dimensional planar features.

[0045] To better reflect the brightness sensitivity of the human eye, this invention first converts the point cloud from the original RGB color space to the YUV color space. Then, it establishes a point correspondence between the reference point cloud and the distorted point cloud. Next, it calculates the curvature contrast features of the reference and distorted point clouds as three-dimensional geometric perception features. Then, it captures the differences in color distribution by calculating the kurtosis features of local regions. Since the human eye is sensitive to changes in brightness, this invention uses brightness differences as an important visual feature. In addition to brightness, this invention also extracts hue features to capture changes in color saturation in the local neighborhood of the point cloud. Finally, the extracted feature set is mapped to the quality score of the point cloud using multiple linear regression.

[0046] This invention mainly includes the following contents:

[0047] 1. Convert the point cloud from the original RGB color space to the YUV color space;

[0048] 2. Extract curvature contrast, kurtosis contrast, brightness difference, and chromaticity contrast features from 3D point clouds;

[0049] 3. Extract spatial structure similarity features and wavelet domain contrast features from the projected image of the point cloud;

[0050] 4. Obtain a visual perception model by combining 3D point features and 2D planar features through multiple linear regression, and evaluate the visual quality of the point cloud.

[0051] The technical solution of the present invention is described in detail below:

[0052] This invention proposes a point cloud quality assessment model based on complementary features in three-dimensional and two-dimensional spaces, mainly comprising two modules: a feature extraction module and a quality prediction module, as shown in the appendix. Figure 2 As shown, the feature extraction module is divided into two parts: three-dimensional spatial feature extraction and two-dimensional planar feature extraction. This fully utilizes the complementary characteristics of point clouds under different representations, thereby efficiently representing various distortion types of point clouds. In the quality prediction module, this invention first uses statistical tests to screen features to avoid redundancy. Then, it aggregates feature sets from different dimensional spaces and uses a regression model to map them to point cloud quality scores.

[0053] 1) First, calculate the quality features based on three-dimensional points.

[0054] ①. Convert the point cloud from RGB color space to YUV color space using the following formula.

[0055]

[0056] Y represents the luminance channel in the YUV color space, while U and V represent two chrominance channels used to describe the vividness and saturation of colors.

[0057] ②. Extract curvature contrast features from the reference point cloud and the distorted point cloud. First, establish the point correspondence between the reference and distorted point clouds. Project the i-th point pi in the reference point cloud Pr onto the distorted point cloud Pd, and find its nearest neighbor point pj as the corresponding point. Calculate the curvature contrast features according to the following formula:

[0058]

[0059] Where σ Pi and σ Pj represents the standard deviation of the curvature of the reference point cloud and the distorted point cloud, respectively, N represents the number of points in the reference point cloud, and k1 = 0.001 to avoid the denominator being 0.

[0060] ③. For point pi in the point cloud, its kurtosis characteristic of brightness is calculated as:

[0061]

[0062] Lj represents the brightness value of the j-th point in the neighborhood of point pi. Let be the average brightness within the neighborhood, and k be the number of points in the neighborhood. The kurtosis-contrast characteristic between the reference and distorted point clouds can be calculated using the following formula:

[0063]

[0064] N is the number of points in the reference point cloud, K Pi and K Pj represents the kurtosis values ​​of the reference point cloud and the distorted point cloud, respectively, and k2 is a very small constant to avoid the denominator being zero.

[0065] ④. To perceive the differences in brightness information across different areas, the Euclidean distance between the brightness values ​​of the reference point and the distortion point is used to calculate the brightness difference characteristics.

[0066]

[0067] L i,j , These represent the brightness values ​​of the j-th point in the neighborhood of the i-th point in the reference point cloud and the distorted point cloud, respectively. In addition to brightness characteristics, to further quantify color distortion, this invention uses hue features to capture changes in neighborhood saturation. The global hue comparison feature is calculated according to the following formula:

[0068]

[0069] It is the Gaussian weighted average of the color channels, and k3 is a constant whose value is consistent with that in [6].

[0070] 2) Extract the spatial structure similarity features and wavelet domain contrast features of the projected image after projecting the 3D point cloud through patch.

[0071] ① In addition to the 3D visual features of the point cloud, this invention also utilizes the 2D visual features of the point cloud as a supplement to further improve the model's ability to capture distortion. This invention uses a patch matching strategy to ensure the consistency of the patch order and position between the distorted point cloud and the reference point cloud. The process of projecting the 3D point cloud into a depth and texture image can be represented as:

[0072]

[0073] λ∈{r,d} represents the reference and distorted images, and Θ(·) represents the patch projection algorithm. and These represent the depth projection image and the texture projection image, respectively. After obtaining the projection images, two-dimensional spatial domain features and wavelet domain features can be further extracted.

[0074] ②. Although brightness and kurtosis features in three-dimensional space can represent attribute distortions in point clouds, they are calculated between neighboring points, making it difficult to capture displacement distortions. Therefore, this invention further extracts features from the projected image to more accurately perceive attribute distortions. To evaluate the visual quality of the two-dimensional projected image, this invention uses the Information Content Weighted SSIM (IW-SSIM) method. Therefore, the spatial domain structural similarity between the reference point cloud and the distorted point cloud can be calculated as follows:

[0075]

[0076] and Let represent the reference and distorted projected images, respectively, and Ψ(·) be the IW-SSIM algorithm. α and β represent the weights of the geometric and texture images, respectively, and α + β = 1.

[0077] ③. To further extract frequency domain features from the texture image, this invention first transforms the image from the spatial domain to the wavelet domain, and then extracts three layers (fine-grained, medium-grained, and coarse-grained) of image features using different wavelet transform coefficients. This coarse-to-fine division effectively captures detailed information of the image at different frequencies. The similarity between reference image a and distorted image b is calculated as follows:

[0078]

[0079] k4 is a non-negative constant, set to 0.001 to avoid instability. The wavelet domain features can then be expressed as:

[0080]

[0081] ω L ω C and ω H The perceptual weights for different color channels are set to 6:1:1, E L E C and E H These represent the similarity of each channel.

[0082] 3) For high-dimensional features, machine learning-based quality prediction can more effectively learn feature weights from the data. Linear regression models can quantitatively describe statistical relationships for predictive analysis and effectively represent the dependency between independent and dependent variables. Therefore, this invention uses a multiple linear regression model to map the feature set to quality scores, specifically as follows:

[0083] Q = ω1f1 + ω2f2 + ... + ω6f6

[0084] Q represents the predicted point cloud quality score. f1 to f6 represent the feature set, and ω1 to ω6 are the weights of each feature.

[0085] The key points and areas to be protected in this invention are:

[0086] 1. First, calculate the 3D geometric and color features of the point cloud. Based on the 3D point cloud data input: the original point cloud Pr and the distorted point cloud Pd, extract the four-dimensional features of geometric curvature contrast, color kurtosis, brightness contrast, and chromaticity contrast.

[0087] 2. Perform patch projection operations on the original point cloud and the distorted point cloud respectively to obtain geometric projection image and texture projection image. Extract the spatial structure similarity features and wavelet domain edge similarity features of the original projection image and the distorted projection image respectively.

[0088] 3. The features based on three-dimensional points and the structural similarity quality features based on two-dimensional projected images are fused together, and the final quality score is obtained through multiple linear regression.

[0089] 4. When extracting 3D features, the characteristics of human vision should be considered. First, the RGB color space needs to be converted to the YUV color space. Then, the kurtosis, brightness contrast and chromaticity contrast features are extracted in the neighborhood of the 3D points respectively, according to the following formula.

[0090]

[0091]

[0092]

[0093] f2, f3, and f4 represent the kurtosis, brightness contrast, and chromaticity contrast features of a local neighborhood of a point cloud, respectively.

[0094] 5. When extracting features from two-dimensional images, the saliency of human vision and edge sensitivity are considered. The spatial structure similarity features and wavelet domain edge similarity features of the original projection image and the distorted projection image are extracted, and obtained according to the following formula.

[0095]

[0096]

[0097] f5 and f6 represent the spatial domain structure similarity feature and the wavelet domain edge similarity feature, respectively.

[0098] 6. When performing feature fusion and regression, the contrast features based on three-dimensional points and the saliency features based on two-dimensional projected images are fused using a multiple linear regression model according to the following formula to obtain the final quality score.

[0099] Q = ω1f1 + ω2f2 + ... + ω6f6

[0100] Q represents the predicted point cloud quality score. f1 to f6 represent the feature set, and ω1 to ω6 are the weights of each feature.

[0101] Compared with the prior art, the advantages of the present invention are as follows:

[0102] 1. By combining the complementary characteristics of 3D point data and 2D image data, the robustness of the point cloud quality assessment model is improved. Compared with traditional algorithms based on 3D point features or 2D image features, it can not only capture the data differences of 3D points, but also reflect the visual characteristics of the human eye through 2D image features.

[0103] 2. Compared with traditional quality assessment based on three-dimensional point features, this method considers the brightness and color sensitivity characteristics of the human eye; at the same time, it introduces two-dimensional image features based on visual saliency, combining the advantages of two different dimensional features. It is more sensitive to distortions such as missing, shifted, and blurred local areas of three-dimensional points, and therefore can more accurately predict the perceptual quality of distorted point clouds.

[0104] 3. This invention is applicable to any type of point cloud distortion, and can demonstrate more accurate prediction performance for point cloud datasets with multiple distortion types.

[0105] This quality assessment framework was tested on four publicly available colored point cloud databases. The objective quality scores predicted by the algorithm were compared with subjective scores, using commonly used evaluation metrics such as Pearson correlation coefficient (PLCC), Spearman rank correlation coefficient (SROCC), Kendall rank correlation coefficient (KROCC), and root mean square error (RMSE). When calculating PLCC and RMSE, a nonlinear regression operation using a five-fold cross function was first applied to the predicted scores. The best performance was achieved on the SIAT-PCQD and WPC2.0 databases, with PLCC and SROCC values ​​of 0.882 and 0.827, and 0.882 and 0.879, respectively. The second-best performance was achieved on the ICIP2020 and SJTU-PCQD databases, with PLCC and SROCC values ​​of 0.968 and 0.963, and 0.878 and 0.856, respectively. The performance of this invention was compared with other mainstream quality assessment algorithms, including Po2Po-MSE, Po2Pl-MSE; full-reference image quality assessment algorithms P2D, PointSSIM, PCQM, GraphSIM, MS-GraphSIM, SIAT-PCQA, and MPED; and a partial-reference point cloud quality assessment algorithm, PCMRR. The comparison results are shown in Tables 1 and 2, where the best result in each column is indicated in bold. Experimental results show that the assessment method disclosed in this invention can effectively predict point cloud quality compared to other methods and has a high consistency with subjective human ratings.

[0106] Table 1. Performance comparison of the proposed method and 12 other quality assessment methods on the SIAT-PCQD and WPC2.0 databases.

[0107]

[0108] Table 2. Performance comparison of the proposed method and 12 other quality assessment methods on the ICIP2020 and SJTU-PCQD databases.

[0109]

[0110] The modified design or alternative solution of the present invention is as follows:

[0111] In this invention, the extraction of two-dimensional projection image features can be replaced by other image feature extraction algorithms, such as robust algorithms like the DISTS algorithm.

[0112] The regression model in this invention uses a multiple linear regression model, but it can also be replaced with other regression models, such as a regression model based on support vector machines.

[0113] Example 2

[0114] According to another embodiment of the present invention, a point cloud quality assessment device based on complementary features of three-dimensional and two-dimensional space is provided, see [link to previous document]. Figure 3 ,include:

[0115] The first feature extraction module 201 is used to extract perceptual features in the point neighborhood;

[0116] The second feature extraction module 202 is used to map the point cloud from three-dimensional space to two-dimensional plane by utilizing the complementary characteristics of features in three-dimensional space and two-dimensional plane, and extract the spatial domain structural similarity features and wavelet domain quality perception features of the projected image.

[0117] The quality prediction module 203 is used to fuse features from different spaces and obtain the quality score of the point cloud through a regression model.

[0118] The point cloud quality assessment device based on complementary features of three-dimensional and two-dimensional space in this embodiment of the invention reflects the perceptual quality of point clouds through the complementary characteristics between three-dimensional point features and two-dimensional planar features, and can avoid inconsistencies between subjective and objective evaluation objects to a certain extent. By combining the visual mechanism of users watching VR content while wearing head-mounted display devices and traditional discrete statistics to construct a more accurate feature set, a relatively comprehensive objective quality assessment method for point clouds is proposed by combining the advantages of three-dimensional point features and two-dimensional planar features.

[0119] In this invention, to leverage the complementary properties of 3D point features and 2D planar features, a point cloud quality assessment model based on complementary features in 3D and 2D space (CF-PCQA) is proposed. This model mainly includes two modules: feature extraction and quality prediction. The feature extraction module is further divided into 3D spatial feature extraction and 2D planar feature extraction, enabling more efficient perception of the visual quality of point clouds. First, this invention extracts four perceptual features in the point neighborhood, including geometric (curvature contrast) and color-related features (Euclidean distance, chromaticity, and kurtosis of the luminance channel), to better capture geometric and color distortions. Second, this invention utilizes the complementary properties of features in 3D space and 2D plane to map the point cloud from 3D space to the 2D plane, extracting spatial domain structural similarity features of the projected image and wavelet domain quality-perceptual features, resulting in more accurate prediction performance and better robustness. Finally, the features from different spaces are fused and a regression model is used to obtain the point cloud quality score.

[0120] The purpose of this invention is to reflect the perceptual quality of point clouds by leveraging the complementary characteristics between three-dimensional point features and two-dimensional planar features, and to a certain extent avoid the inconsistency between subjective and objective evaluation objects. By combining the visual mechanism of users watching VR content while wearing head-mounted displays with traditional discrete statistics to construct a more accurate feature set, a relatively comprehensive objective quality evaluation method for point clouds is proposed by combining the advantages of three-dimensional point features and two-dimensional planar features.

[0121] To better reflect the brightness sensitivity of the human eye, this invention first converts the point cloud from the original RGB color space to the YUV color space. Then, it establishes a point correspondence between the reference point cloud and the distorted point cloud. Next, it calculates the curvature contrast features of the reference and distorted point clouds as three-dimensional geometric perception features. Then, it captures the differences in color distribution by calculating the kurtosis features of local regions. Since the human eye is sensitive to changes in brightness, this invention uses brightness differences as an important visual feature. In addition to brightness, this invention also extracts hue features to capture changes in color saturation in the local neighborhood of the point cloud. Finally, the extracted feature set is mapped to the quality score of the point cloud using multiple linear regression.

[0122] This invention mainly includes the following contents:

[0123] 1. Convert the point cloud from the original RGB color space to the YUV color space;

[0124] 2. Extract curvature contrast, kurtosis contrast, brightness difference, and chromaticity contrast features from 3D point clouds;

[0125] 3. Extract spatial structure similarity features and wavelet domain contrast features from the projected image of the point cloud;

[0126] 4. Obtain a visual perception model by combining 3D point features and 2D planar features through multiple linear regression, and evaluate the visual quality of the point cloud.

[0127] The technical solution of the present invention is described in detail below:

[0128] This invention proposes a point cloud quality assessment model based on complementary features in three-dimensional and two-dimensional spaces, mainly comprising two modules: a feature extraction module and a quality prediction module, as shown in the appendix. Figure 2 As shown, the feature extraction module is divided into two parts: three-dimensional spatial feature extraction and two-dimensional planar feature extraction. This fully utilizes the complementary characteristics of point clouds under different representations, thereby efficiently representing various distortion types of point clouds. In the quality prediction module, this invention first uses statistical tests to screen features to avoid redundancy. Then, it aggregates feature sets from different dimensional spaces and uses a regression model to map them to point cloud quality scores.

[0129] 1) First, calculate the quality features based on three-dimensional points.

[0130] ①. Convert the point cloud from RGB color space to YUV color space using the following formula.

[0131]

[0132] Y represents the luminance channel in the YUV color space, while U and V represent two chrominance channels used to describe the vividness and saturation of colors.

[0133] ②. Extract curvature contrast features from the reference point cloud and the distorted point cloud. First, establish the point correspondence between the reference and distorted point clouds. Project the i-th point pi in the reference point cloud Pr onto the distorted point cloud Pd, and find its nearest neighbor point pj as the corresponding point. Calculate the curvature contrast features according to the following formula:

[0134]

[0135] Where σ Pi and σ Pj represents the standard deviation of the curvature of the reference point cloud and the distorted point cloud, respectively, N represents the number of points in the reference point cloud, and k1 = 0.001 to avoid the denominator being 0.

[0136] ③. For point pi in the point cloud, its kurtosis characteristic of brightness is calculated as:

[0137]

[0138] Lj represents the brightness value of the j-th point in the neighborhood of point pi. Let be the average brightness within the neighborhood, and k be the number of points in the neighborhood. The kurtosis-contrast characteristic between the reference and distorted point clouds can be calculated using the following formula:

[0139]

[0140] N is the number of points in the reference point cloud, K Pi and K Pj represents the kurtosis values ​​of the reference point cloud and the distorted point cloud, respectively, and k2 is a very small constant to avoid the denominator being zero.

[0141] ④. To perceive the differences in brightness information across different areas, the Euclidean distance between the brightness values ​​of the reference point and the distortion point is used to calculate the brightness difference characteristics.

[0142]

[0143] L i,j , These represent the brightness values ​​of the j-th point in the neighborhood of the i-th point in the reference point cloud and the distorted point cloud, respectively. In addition to brightness characteristics, to further quantify color distortion, this invention uses hue features to capture changes in neighborhood saturation. The global hue comparison feature is calculated according to the following formula:

[0144]

[0145] It is the Gaussian weighted average of the color channels, and k3 is a constant whose value is consistent with that in [6].

[0146] 2) Extract the spatial structure similarity features and wavelet domain contrast features of the projected image after projecting the 3D point cloud through patch.

[0147] ① In addition to the 3D visual features of the point cloud, this invention also utilizes the 2D visual features of the point cloud as a supplement to further improve the model's ability to capture distortion. This invention uses a patch matching strategy to ensure the consistency of the patch order and position between the distorted point cloud and the reference point cloud. The process of projecting the 3D point cloud into a depth and texture image can be represented as:

[0148]

[0149] λ∈{r,d} represents the reference and distorted images, and Θ(·) represents the patch projection algorithm. and These represent the depth projection image and the texture projection image, respectively. After obtaining the projection images, two-dimensional spatial domain features and wavelet domain features can be further extracted.

[0150] ②. Although brightness and kurtosis features in three-dimensional space can represent attribute distortions in point clouds, they are calculated between neighboring points, making it difficult to capture displacement distortions. Therefore, this invention further extracts features from the projected image to more accurately perceive attribute distortions. To evaluate the visual quality of the two-dimensional projected image, this invention uses the Information Content Weighted SSIM (IW-SSIM) method. Therefore, the spatial domain structural similarity between the reference point cloud and the distorted point cloud can be calculated as follows:

[0151]

[0152] and Let represent the reference and distorted projected images, respectively, and Ψ(·) be the IW-SSIM algorithm. α and β represent the weights of the geometric and texture images, respectively, and α + β = 1.

[0153] ③. To further extract frequency domain features from the texture image, this invention first transforms the image from the spatial domain to the wavelet domain, and then extracts three layers (fine-grained, medium-grained, and coarse-grained) of image features using different wavelet transform coefficients. This coarse-to-fine division effectively captures detailed information of the image at different frequencies. The similarity between reference image a and distorted image b is calculated as follows:

[0154]

[0155] k4 is a non-negative constant, set to 0.001 to avoid instability. The wavelet domain features can then be expressed as:

[0156]

[0157] ω L ω C and ω H The perceptual weights for different color channels are set to 6:1:1, E L E C and E H These represent the similarity of each channel.

[0158] 3) For high-dimensional features, machine learning-based quality prediction can more effectively learn feature weights from the data. Linear regression models can quantitatively describe statistical relationships for predictive analysis and effectively represent the dependency between independent and dependent variables. Therefore, this invention uses a multiple linear regression model to map the feature set to quality scores, specifically as follows:

[0159] Q = ω1f1 + ω2f2 + ... + ω6f6

[0160] Q represents the predicted point cloud quality score. f1 to f6 represent the feature set, and ω1 to ω6 are the weights of each feature.

[0161] The key points and areas to be protected in this invention are:

[0162] 1. First, calculate the 3D geometric and color features of the point cloud. Based on the 3D point cloud data input: the original point cloud Pr and the distorted point cloud Pd, extract the four-dimensional features of geometric curvature contrast, color kurtosis, brightness contrast, and chromaticity contrast.

[0163] 2. Perform patch projection operations on the original point cloud and the distorted point cloud respectively to obtain geometric projection image and texture projection image. Extract the spatial structure similarity features and wavelet domain edge similarity features of the original projection image and the distorted projection image respectively.

[0164] 3. The features based on three-dimensional points and the structural similarity quality features based on two-dimensional projected images are fused together, and the final quality score is obtained through multiple linear regression.

[0165] 4. When extracting 3D features, the characteristics of human vision should be considered. First, the RGB color space needs to be converted to the YUV color space. Then, the kurtosis, brightness contrast and chromaticity contrast features are extracted in the neighborhood of the 3D points respectively, according to the following formula.

[0166]

[0167]

[0168]

[0169] f2, f3, and f4 represent the kurtosis, brightness contrast, and chromaticity contrast features of a local neighborhood of a point cloud, respectively.

[0170] 5. When extracting features from two-dimensional images, the saliency of human vision and edge sensitivity are considered. The spatial structure similarity features and wavelet domain edge similarity features of the original projection image and the distorted projection image are extracted, and obtained according to the following formula.

[0171]

[0172]

[0173] f5 and f6 represent the spatial domain structure similarity feature and the wavelet domain edge similarity feature, respectively.

[0174] 6. When performing feature fusion and regression, the contrast features based on three-dimensional points and the saliency features based on two-dimensional projected images are fused using a multiple linear regression model according to the following formula to obtain the final quality score.

[0175] Q = ω1f1 + ω2f2 + ... + ω6f6

[0176] Q represents the predicted point cloud quality score. f1 to f6 represent the feature set, and ω1 to ω6 are the weights of each feature.

[0177] Compared with the prior art, the advantages of the present invention are as follows:

[0178] 1. By combining the complementary characteristics of 3D point data and 2D image data, the robustness of the point cloud quality assessment model is improved. Compared with traditional algorithms based on 3D point features or 2D image features, it can not only capture the data differences of 3D points, but also reflect the visual characteristics of the human eye through 2D image features.

[0179] 2. Compared with traditional quality assessment based on three-dimensional point features, this method considers the brightness and color sensitivity characteristics of the human eye; at the same time, it introduces two-dimensional image features based on visual saliency, combining the advantages of two different dimensional features. It is more sensitive to distortions such as missing, shifted, and blurred local areas of three-dimensional points, and therefore can more accurately predict the perceptual quality of distorted point clouds.

[0180] 3. This invention is applicable to any type of point cloud distortion, and can demonstrate more accurate prediction performance for point cloud datasets with multiple distortion types.

[0181] This quality assessment framework was tested on four publicly available colored point cloud databases. The objective quality scores predicted by the algorithm were compared with subjective scores, using commonly used evaluation metrics such as Pearson correlation coefficient (PLCC), Spearman rank correlation coefficient (SROCC), Kendall rank correlation coefficient (KROCC), and root mean square error (RMSE). When calculating PLCC and RMSE, a nonlinear regression operation using a five-fold cross function was first applied to the predicted scores. The best performance was achieved on the SIAT-PCQD and WPC2.0 databases, with PLCC and SROCC values ​​of 0.882 and 0.827, and 0.882 and 0.879, respectively. The second-best performance was achieved on the ICIP2020 and SJTU-PCQD databases, with PLCC and SROCC values ​​of 0.968 and 0.963, and 0.878 and 0.856, respectively. The performance of this invention was compared with other mainstream quality assessment algorithms, including Po2Po-MSE, Po2Pl-MSE; full-reference image quality assessment algorithms P2D, PointSSIM, PCQM, GraphSIM, MS-GraphSIM, SIAT-PCQA, and MPED; and a partial-reference point cloud quality assessment algorithm, PCMRR. The comparison results are shown in Tables 1 and 2, where the best result in each column is indicated in bold. Experimental results show that the assessment method disclosed in this invention can effectively predict point cloud quality compared to other methods and has a high consistency with subjective human ratings.

[0182] Table 3. Performance comparison of the proposed method and 12 other quality assessment methods on the SIAT-PCQD and WPC2.0 databases.

[0183]

[0184]

[0185] Table 4. Performance comparison of the proposed method with 12 other quality assessment methods on the ICIP2020 and SJTU-PCQD databases.

[0186]

[0187] The modified design or alternative solution of the present invention is as follows:

[0188] In this invention, the extraction of two-dimensional projection image features can be replaced by other image feature extraction algorithms, such as robust algorithms like the DISTS algorithm.

[0189] The regression model in this invention uses a multiple linear regression model, but it can also be replaced with other regression models, such as a regression model based on support vector machines.

[0190] Example 3

[0191] A storage medium storing program files capable of implementing any of the above-mentioned point cloud quality assessment methods based on complementary features of three-dimensional and two-dimensional spaces.

[0192] Example 4

[0193] A processor for running a program, wherein the program executes any of the above-mentioned point cloud quality assessment methods based on complementary features in three-dimensional and two-dimensional spaces.

[0194] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0195] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The system embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0197] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0198] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0199] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0200] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A point cloud quality assessment method based on complementary features of three-dimensional and two-dimensional space, characterized in that, Includes the following steps: S101: Extract perceptual features from the point neighborhood; In step S101, curvature contrast, kurtosis contrast, luminance difference and chromaticity contrast features are extracted from the three-dimensional point cloud. S102: Utilizing the complementary properties of features in three-dimensional space and two-dimensional plane, point clouds are mapped from three-dimensional space to two-dimensional plane, and spatial domain structural similarity features and wavelet domain quality perception features of the projected image are extracted. S103: After fusing features from different spaces, the quality score of the point cloud is obtained through a regression model; Specifically, step S101 includes: Convert the point cloud from the original RGB color space to the YUV color space; Establish the point correspondence between the reference point cloud and the distorted point cloud; The curvature contrast features of the reference point cloud and the distorted point cloud are calculated as 3D geometric perception features. By calculating the kurtosis features of local regions to capture the differences in color distribution, and taking the differences in brightness as an important visual feature, we extract hue features to capture the changes in color saturation in the local neighborhood of point clouds. In step S103, the three-dimensional point features and two-dimensional planar features are used to obtain a visual perception model through multiple linear regression to evaluate the visual quality of the point cloud. In step S102, the image is first transformed from the spatial domain to the wavelet domain, and then fine-grained, medium-grained, and coarse-grained image features are extracted through different wavelet transform coefficients. The detailed information of the image at different frequencies is captured by dividing it from coarse to fine. The wavelet domain quality perception feature is represented as follows: , and The perceptual weights for different color channels are set to 6:1:

1. , and These represent the similarity of each channel between the reference image and the distorted image; where Pr is the reference point cloud and Pd is the distorted point cloud.

2. The point cloud quality assessment method based on complementary features of three-dimensional and two-dimensional space according to claim 1, characterized in that, The brightness difference characteristics are calculated using the Euclidean distance between the brightness values ​​of the reference point and the distortion point.

3. The point cloud quality assessment method based on complementary features of three-dimensional and two-dimensional space according to claim 1, characterized in that, A patch matching strategy is used to ensure consistency in the order and position of patches between the distorted point cloud and the reference point cloud.

4. The point cloud quality assessment method based on complementary features in three-dimensional and two-dimensional space according to claim 1, characterized in that, In step S102, the spatial domain structure similarity features of the projected image are extracted using the information content weighted SSIM method.

5. A point cloud quality assessment device based on complementary features of three-dimensional and two-dimensional space, characterized in that, The point cloud quality assessment method based on complementary features in three-dimensional and two-dimensional space as described in any one of claims 1-4 is implemented; the point cloud quality assessment device includes: The first feature extraction module is used to extract perceptual features in the neighborhood of a point. The second feature extraction module is used to map the point cloud from three-dimensional space to two-dimensional plane by utilizing the complementary properties of features in three-dimensional space and two-dimensional plane, and to extract the spatial domain structural similarity features and wavelet domain quality perception features of the projected image. The quality prediction module is used to fuse features from different spaces and then obtain the quality score of the point cloud through a regression model.

Images