A graph convolution-based referenceless point cloud quality assessment method

Abstract

The application provides a no-reference point cloud quality evaluation method based on graph convolution, and belongs to the field of three-dimensional point cloud quality evaluation. The scheme comprises data preprocessing, feature extraction, attention enhancement, construction of a single-modal graph structure, cross-modal feature fusion, and finally output of an objective score of point cloud quality. Specifically, for a given point cloud, an orthogonal projection method is used to map it to six standard orthogonal viewing angles, and three types of two-dimensional images are generated for each viewing angle: texture maps, depth maps and placeholder maps; then the three types of images are input into a residual neural network to extract features; then a double attention mechanism is introduced to adaptively weight the extracted features to highlight key features; then a dynamic graph structure is constructed by similarity, and a single-modal global fusion feature is aggregated by graph convolution; then a unified feature integrator is used for cross-modal fusion to fully integrate the mutual information of multiple modalities; and finally, a full connection layer is used to output the final quality score of the point cloud in a multi-view scene.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of 3D data processing and computer vision, specifically, to a referenceless point cloud quality evaluation method based on graph convolution. Background Technology

[0002] Due to the rapid development of technologies such as 3D scanning, virtual reality (VR), and autonomous driving, 3D point clouds have become a core visual data format because they can accurately represent the geometric and texture information of 3D scenes and objects. 3D point clouds not only provide an efficient means in object detection but also play an indispensable role in scene understanding, environmental perception modules in autonomous driving systems, and creating immersive experiences in virtual reality technology. However, point cloud data is massive (a single scene often contains millions of points), requiring compression for storage and transmission. This process easily introduces geometric distortion (such as point cloud sparsity caused by undersampling and shape distortion caused by compression) and texture distortion (such as color deviation caused by color quantization), directly affecting the downstream application experience. For example, distorted point clouds in VR devices can cause visual dizziness, and geometric deformation in autonomous driving environmental perception may lead to target misidentification. Therefore, establishing a high-performance point cloud quality evaluation method is crucial. Point cloud quality evaluation can not only provide a unified objective quality judgment standard but also quantify various distortions in point clouds, thus providing reference indicators for different point cloud acquisition and processing technologies. Through efficient and automated point cloud quality assessment methods, researchers and engineers can quickly identify data defects and optimize algorithm performance.

[0003] Currently, there are two main methods for point cloud quality assessment: full-reference assessment and no-reference assessment. No-reference assessment methods do not require any information from the reference point cloud; they directly analyze the characteristics of the distorted point cloud itself for quality assessment. Therefore, they are widely used in practical scenarios. Traditional no-reference methods also face challenges. Some methods only focus on the distortion of point cloud geometry, ignoring the collaborative effects of multi-dimensional information such as texture and depth. Some methods rely on 2D projected image feature extraction but fail to effectively model the 3D structural relationships of point clouds, resulting in insufficient perception of global distortion. Point clouds are typical non-Euclidean data (disordered and sparsely distributed in 3D space). The "pixel-by-pixel convolution" of traditional Convolutional Neural Networks (CNNs) is not only inefficient (involving a large number of redundant background pixels) but also cannot capture the 3D structural relationships of point clouds (such as point offsets and local clustering breaks). Relying on a fixed grid, the convolution kernel of CNNs needs to slide on a regular pixel grid, but point clouds have no grid and cannot directly apply convolution operations. They are sensitive to order. The input order of CNNs will affect the feature extraction results (such as changes in the pixel order of an image will lead to recognition errors), but the disorder of point clouds will make the features extracted by CNNs unstable. There is serious redundant computation. The sparsity of point clouds will cause CNNs to perform useless convolutions in "blank areas without effective points", wasting computing power and easily extracting invalid features (such as background noise), resulting in insufficient accuracy of quality assessment.

[0004] In summary, quality assessment of multi-dimensional data such as point clouds often employs simple stitching or independent processing of different modal features such as texture, depth, and occupancy, without deeply modeling the spatial relationships and semantic dependencies between different modal distortions (such as the synergistic relationship between texture blurring and depth discontinuity). Therefore, to address these issues, our research focuses on improving and innovating graph convolutional networks. Graph convolutional networks are naturally adapted to non-Euclidean data and can establish spatial relationships and semantic dependencies between different modal distortions. Thus, researching a quality assessment method based on graph convolutional networks can better help people evaluate the distortion level of point clouds and the performance of compression algorithms. Summary of the Invention

[0005] This invention provides a referenceless point cloud quality assessment method based on graph convolution, comprising: Step 1: Project the distorted point cloud to obtain three types of two-dimensional projection images corresponding to the six different viewpoints of the distorted point cloud, namely texture image, depth image, and occupancy image. Step 2: Based on the projected image in Step 1, in each modality feature extraction sub-network, a deep residual convolutional neural network is used as the feature extraction backbone network to perform layer-by-layer convolutional feature extraction on the input image from each viewpoint, outputting high-dimensional semantic feature mapping, and outputting texture features, depth features, and occupancy features respectively. Step 3: Input the three types of high-dimensional semantic features obtained in Step 2 into the channel-space attention enhancement module. First, the channel dimension is weighted by response, and then the spatial dimension is weighted by saliency to highlight key semantic channels and important spatial regions, and obtain the enhanced texture feature map, depth feature map and occupancy feature map. Step 4: The enhanced trimodal feature maps obtained in Step 3 are subjected to global average pooling to compress the two-dimensional spatial features into one-dimensional node feature vectors of the corresponding views, thus forming a multi-view node feature set; then, the node features are linearly mapped, and the correlation score between any two view nodes is calculated by scaling the dot product, and then the adaptive adjacency matrix is ​​obtained by softmax normalization, thereby dynamically representing the connection strength and relationship weight between different views, and finally constructing the texture modality map structure, depth modality map structure and occupancy modality map structure respectively; Step 5: Input the node feature sets and their corresponding adjacency matrices of each modality generated in Step 4 into a multi-layer graph convolutional network. Use graph convolution operations to propagate and aggregate features between adjacent nodes. The adjacency matrix is ​​self-connected and symmetrically normalized before graph convolution to balance the impact of differences in degree between different nodes. The graph convolutional network adopts a multi-layer cascaded structure, gradually mapping node features from high-dimensional space to low-dimensional discriminative feature space. Batch normalization and non-linear activation functions are combined in each layer to improve representation ability. Then, the output of the third layer of the graph convolutional network is taken as the structure-aware node representation, and global average aggregation is performed on each view node in the view dimension to obtain global feature vectors for texture modality, depth modality, and occupancy modality, respectively. Step 6: Concatenate the global feature vectors of texture mode, depth mode, and occupancy mode obtained in Step 5 to form a joint representation vector. Then, project the joint representation to a unified latent space through a fully connected mapping to achieve linear coupling and alignment of information from different modes. Further, input the projected features into the feature enhancement unit based on the Transformer encoder. Through residual normalization and feedforward network structure, perform nonlinear modeling and robustness enhancement on the joint representation. Finally, output a quality score through a regression layer to predict the overall quality of the input sample.

[0006] The proposed graph convolution-based no-reference point cloud quality assessment method belongs to the field of 3D point cloud quality assessment. The scheme includes data preprocessing, feature extraction, attention enhancement, single-modal graph structure modeling, cross-modal feature fusion, and quality prediction. First, for a given point cloud, a six-way orthogonal projection method is used to map it to six standard orthogonal viewpoints, generating three types of 2D images for each viewpoint: texture map, depth map, and placeholder map. Then, the texture map, depth map, and placeholder map from the six viewpoints are used to extract features through an adapted residual neural network, ensuring the integrity of each modality's features. Next, a dual attention mechanism is used to highlight key features, followed by the construction of a dynamic graph structure through similarity, and graph convolution aggregation to form a single-modal global fusion feature. A unified feature integrator then performs cross-modal fusion, and finally, a fully connected layer outputs the final quality score for the multi-view scene. Attached Figure Description

[0007] Appendix Figure 1 This is a schematic diagram of the point cloud quality evaluation method of the present invention. Detailed Implementation

[0008] The point cloud quality evaluation algorithm of the present invention will be further explained below with reference to the accompanying drawings.

[0009] As shown in the attached figure, the image quality assessment algorithm of the present invention includes four steps: step 1 data preprocessing; step 2 feature extraction; step 3 attention enhancement; step 4 single-modal graph structure construction; step 5 cross-modal feature fusion; and step 6 quality score prediction.

[0010] Step 1, Data Preprocessing Its characteristic is that, given a point cloud (1) in Indicates the first The three-dimensional coordinates of a point in a spatial point cloud. This indicates the color attribute of that point. The index of a point in the point cloud. This represents the total number of points in the point cloud. To eliminate differences in spatial location and scale within the point cloud, it is first normalized. This specifically includes: calculating the geometric center of the point cloud. (2) in This represents the average value of the point cloud in three-dimensional space, used to characterize the overall spatial distribution center of the point cloud. All points are translated to a coordinate system with the geometric center as the origin, and scaled according to the maximum Euclidean distance from each point in the point cloud to the geometric center, resulting in the normalized point coordinates: (3) in Represents the Euclidean norm. For the first The three-dimensional spatial coordinates of the points For the normalized first The three-dimensional spatial coordinates of each point are used. Through the above processing, the point cloud is constrained within a unit sphere space, thereby eliminating differences in scale and position between different point cloud samples and improving the consistency and stability of subsequent multi-view processing. To comprehensively characterize the spatial structure information of the point cloud from multiple directions, six fixed viewpoints are preset, corresponding to the front view, rear view, left view, right view, top view, and bottom view obtained by rotating the point cloud around the three-dimensional coordinate axes. For each viewpoint... Set the corresponding camera extrinsic parameters, including the rotation matrix. Translation vector To determine the camera's spatial pose from that viewpoint. In the... From one perspective, the normalized point cloud is transformed from the world coordinate system to the camera coordinate system using camera extrinsic parameters. The transformation relationship is expressed as follows: (4) Among them and The first The rotation matrix and translation vector corresponding to each viewpoint For the first From the perspective of the first The points are represented in the camera coordinate system. Then, an orthogonal photogrammetry model is used to project the 3D points onto the 2D image plane, obtaining the projected coordinates. At the same time, the camera coordinate system Quantity This serves as the corresponding depth information. Based on the projection results described above, points are rasterized on the image plane for any pixel location. Select several projection points within its neighborhood and calculate the first projection point. Distance from each candidate point to the pixel center: (5) in, Used to measure the spatial proximity between a candidate point and the current pixel. For the first The point at the th Projected from the first perspective, the first The coordinates of the candidate points in the camera coordinate system. pixel coordinates of a two-dimensional image plane In the horizontal direction, In the vertical direction. Based on this distance, a predefined attenuation function is applied. Calculate the coverage intensity of a point to a pixel. ,in The weights are obtained by normalizing the coverage intensity of all candidate points: (6) in Indicates the first From the perspective of the first Candidate point pairs of pixels Normalization process, The candidate point number is... This is a temporary index used when traversing all candidate points. Using the aforementioned weights, the point attributes at each pixel are weighted and fused to generate the first... Texture maps from various perspectives: (7) in Indicates the first Pixels from a single perspective The color value at that location, Represents texture modality, For the first From the perspective of the first The color attribute of the candidate point. Simultaneously, the depth information of the candidate points is weighted and fused using the same weighting coefficient to generate the first... Depth map from multiple perspectives: (8) in It represents the weighted depth information corresponding to the pixel, used to characterize the spatial structural relationship of the point set. Indicates the first The depth information of each candidate point in the camera coordinate system is obtained. Simultaneously, a placeholder map is constructed based on whether a pixel is covered by any point, and the alpha value at each pixel is thresholded to obtain the following: (9) in For indicator functions, This is a preset threshold. Indicates the first From the perspective of the first Candidate point pairs of pixels The normalization process is performed. When a pixel is covered by at least one point, the placeholder map is set to 1; otherwise, it is set to 0. Finally, the texture map and depth map are linearly normalized to map their values ​​to [0,1], thus obtaining the point set for each viewpoint. texture map Depth map and placeholder images Three multi-viewpoint two-dimensional representations. Ultimately, the total set of six viewpoints and the three image types within each viewpoint is: (10) Through the above steps, 6 viewpoints × 3 types = 18 2D images can be generated for each point cloud to be evaluated. While ensuring the integrity of geometric structure and attribute information, the data consistency and representation stability are improved, providing high-quality input for subsequent point cloud analysis tasks.

[0011] Step 2, Feature Extraction The six viewpoint images of three types generated by data preprocessing are input in the form of [image type name missing]. For each modality, an independent ResNet101 feature extraction branch is constructed. Initial feature extraction is performed first through a stem module consisting of 7×7 convolutions, batch normalization, ReLU activation, and max pooling. Subsequently, four residual stages—Layer 1, Layer 2, Layer 3, and Layer 4—are passed sequentially. Each stage consists of multiple Bottleneck residual blocks, with numbers of 3, 4, 23, and 3 respectively. Each Bottleneck block contains a 1×1 convolution for channel dimensionality reduction, a 3×3 convolution for spatial feature extraction, and a 1×1 convolution for channel dimensionality enhancement. Finally, residual connections and residual fusion are used to achieve feature extraction. The texture modality has 3 input channels and directly uses ImageNet pre-trained weights. The depth-occupancy modality has 1 input channel, performing channel adaptation on the first layer of convolutions and inheriting pre-trained priors. The residual network extracts high-level semantic features layer by layer in the following manner: (11) in This represents the ResNet101 feature encoder for the corresponding modality. Indicates the first From one perspective, Output images for each modality. The size of the output feature map is: (12) in For batch size, For the number of viewpoints, For the number of channels, The height of the image. The width of the image. The input image first undergoes a large receptive field and pooling operation: (13) in With a kernel size of 7×7 and a stride of s=2, the output channels are 64. For normalization operations, For activation function processing, the output size is Then, the max pooling operation is mainly used to extract low-level visual features such as edges and orientation textures. , After max pooling, the features enter the core unit of ResNet-101, the Bottleneck residual block, whose mathematical form is: (14) The main branch transformation Corresponding to three convolutional layers, The convolutional kernel is 1×1, and its function is channel-wise dimensionality reduction. The convolutional kernel is 3×3, and its function is spatial feature extraction. The convolution kernel is 1×1, and its function is to increase the dimensionality of the channels. This is the residual branch. ResNet-101 consists of 4 stages with a total of 33 bottlenecks. Each bottleneck has three convolutional layers. The image enters stage 1-Layer 1, which outputs 256 channels. This layer has 3 blocks without downsampling operations. , Learning about local edge combinations and simple structural shapes, stage2-Layer2 uses downsampling with stride=2. , This layer learns more complex local patterns and texture structures; stage3 (Layer 3) learns the geometric composition structure of middle-layer semantic parts. , The highest semantic feature layer, stage4-Layer4, ultimately yields a high-dimensional feature map. , Overall feature extraction process summary formula: (15) The final high-dimensional feature map By downsampling, the original pixel-level representation is transformed into high-level semantic features with a larger receptive field, thereby significantly reducing the complexity of subsequent attention calculations while maintaining semantic expressiveness.

[0012] Step 3, Attention Enhancement The fourth-level feature map obtained from the fourth stage output of the backbone network ResNet101. The input is fed into the attention module, and to facilitate uniform convolution operations on features from each viewpoint, the batch dimension and viewpoint dimension are merged. Performing global average pooling in the spatial dimension yields: (16) in This refers to global average pooling. It refers to High-dimensional feature maps in different modalities This represents the summation over all pixels in the feature map's spatial dimension (height × width). Global average pooling is used to aggregate the response intensity of each channel globally into a scalar, forming a channel-level global statistic for subsequent estimation of "which channels are more important". After pooling, channel compression and expansion are performed using two 1×1 convolutions to achieve channel dimensionality reduction and restoration. (17) in , Compression ratio (r=16), Represents the ReLU activation function. This represents the channel weight tensor output by the Sigmoid activation function. This step primarily works by reducing dimensionality, learning the dependencies between channels, and then restoring dimensionality to generate weight coefficients for each channel, thereby enhancing discriminative channels and suppressing redundant channels. Channel weighting is then performed. Where ⊙ represents element-wise multiplication. Important channels are amplified, and unimportant channels are suppressed. The output... Calculate the channel average chart and the channel maximum chart separately: , The mean plot reflects the overall activation intensity. This refers to the pixel coordinates in the feature map space. The feature map is a channel-weighted map, and the maximum value map reflects the most significant response. Combining the two can more stably indicate which locations in space are more critical. Spatial attention feature map Then, pass it through a 7×7 convolution and apply Sigmoid activation: (18) in For the convolution kernel used to extract spatial weights, The spatial attention weight map is then spatially weighted to obtain the channel-space dual-attention weighted feature map: Then restore the perspective dimension: (19) After completing the channel and spatial attention weighting, this invention further performs a global average pooling operation on the enhanced feature map, and aggregates the responses in the spatial dimension to obtain a high-dimensional semantic feature vector corresponding to each viewpoint.

[0013] (20) in ; Concatenate all the feature vectors to obtain the feature matrix of the node: (twenty one) This operation can effectively eliminate spatial redundancy while preserving the key regional information emphasized by the attention mechanism, and transform the feature representation from a two-dimensional spatial form to a view-level node representation, providing unified and stable input features for subsequent graph-based relation modeling.

[0014] Step 4, Construction of the single-modal diagram structure The viewpoint node feature matrix for a specific modality, after enhancement by the attention module and global average pooling, is as follows: (twenty two) in For batch size, For the number of viewpoints, The node feature dimension is 2048. Indicates the first The first sample The feature vectors of each viewpoint are used as input to the graph structure generation module. The goal of the graph structure generation module is to automatically construct a graph structure of relationships between viewpoints based on the feature similarity between nodes from different viewpoints, obtaining the adjacency matrix. The graph structure is represented as in The node feature matrix, This is the adjacency matrix. To improve the stability of graph structure learning, node features are first mapped to a low-dimensional embedding space: (twenty three) (twenty four) in The learnable parameter matrix; For projection dimensions; These refer to calculating the similarity between any two viewpoint nodes by using low-dimensional embedded features after different matrix mappings: (25) in ; Indicates the first With the The degree of correlation between perspectives This is the scaling factor. Then, the Softmax function is used for normalization to obtain the adjacency matrix: (26) in This constructs a graph structure. The node set is composed of the node feature matrix. Represented by the adjacency matrix; edge weights are determined by the adjacency matrix. express; It is a natural exponential function, and the graph structure is a dynamically constructed graph.

[0015] Step 5, Cross-modal feature fusion The adjacency matrix obtained in the previous step is normalized by first adding the identity matrix. To preserve node information during propagation and prevent node features from being "completely overwritten" by neighbor information, the degree matrix is ​​then calculated. After performing stability processing, a normalized adjacency matrix is ​​constructed: (27) Its main function is to normalize the influence of nodes with different degrees, preventing "nodes with more connections" from dominating other nodes during propagation. This is done during graph convolution propagation. (28) in These are the node features of the l-th layer; These are the learnable weights of the l-th layer; This means that each node sums the features of its neighbors according to the edge weights. It's handled by the Softplus activation function. ; ; ; ; Used to output the score for each node.

[0016] (29) in It is the first Features of each viewpoint node after the third convolutional layer; mean convergence will... Each node is integrated into a fixed-dimensional global modality vector; the third layer is used because it retains richer semantic representation (32 dimensions), while the fourth layer is more like a "scalar scoring" with less information, ultimately outputting three unimodal global features. .

[0017] In the graph convolution processing stage, texture modality vector, depth modality vector, and occupancy modality vector are obtained respectively, and the three are concatenated in the feature dimension to form a joint representation vector. Then, the joint representation is projected to a unified latent space through a fully connected mapping to achieve linear coupling and alignment of different modal information. Further, the projected features are input into the feature enhancement unit based on the Transformer encoder, and the joint representation is nonlinearly modeled and robustly enhanced through residual normalization and feedforward network structure. Finally, the quality score is output through the regression layer.

[0018] The global features of the three modalities are fused, and the above steps are output. As input, feature concatenation is performed first: (30) in This is a channel-level splicing operation. Using the sample sequence number, the representations of the three modalities are concatenated along the channel direction to form a joint feature vector containing texture, depth, and occupancy information. This provides the input basis for subsequent unified spatial projection and cross-modal semantic modeling. The concatenated vector has a dimension of 96. The concatenated low-dimensional vector is then mapped to the first fully connected layer fc_in of the higher-dimensional fusion space. (31) in It is 256. For the first The cross-modal feature vector of each sample after linear mapping The input layer is a learnable weight matrix. As the input layer's learnable bias vector, a linear mapping layer projects the concatenated cross-modal joint vector onto a unified hidden feature space, aligning and blending high-level semantic information from different modalities on the same feature basis. This layer achieves cross-modal linear fusion. Then, a Transformer encoder is used to perform high-dimensional nonlinear reconstruction and semantic enhancement on the concatenated cross-modal joint features. (32) in For the first The sequence-form features of each sample after dimensionality reshaping The dimension expansion operation transforms the sequence [B,1,256] into the encoder: (33) (34) in The only exception is the compression operation. This is the globally fused feature vector after dimensionality compression. The fused features output by the encoder are reconstructed to a shape of [B, 256]. This encoder can be expanded to multiple tokens (e.g., split into 3 tokens according to modality). This embodiment uses a single token to achieve equivalent nonlinear fusion. Then, quality prediction is performed. (35) in It is a learnable matrix; For learnable bias vectors; the fused ones Mapped to a scalar. For the first For each sample, the model predicts an objective score for the point cloud quality. During the training phase, a joint loss function consisting of ranking loss and regression loss is constructed to simultaneously constrain the relative ranking relationship and absolute scoring error of the prediction results. The regression loss can be either absolute error loss or mean squared error loss; in some implementations, when only the scoring regression accuracy needs to be optimized, the ranking loss weight can be set to zero. (36) in To regress the loss, or , For regression loss weighting coefficients, For sorting loss, The weighting coefficients for the ranking loss are used when the SJTU database is... The method adopted is Regression loss, when using WPC database The method adopted is loss; It is responsible for sorting consistency; through joint optimization, it achieves a synergistic improvement in sorting performance and numerical accuracy.

[0019] The algorithm's performance was validated using the SJTU-PCQA and WPC databases. Four evaluation criteria were used to measure the model's prediction accuracy and predictive monotonicity: Pearson linear correlation coefficient (PLCC), Spearman rank-order correlation coefficient (SROCC), Kendall rank-order correlation coefficient (KROCC), and root mean squared error (RMSE). PLCC, SROCC, and KROCC values ​​range from -1 to 1; the closer the absolute value is to 1, the stronger the correlation and the higher the consistency. The range of RMSE is... The closer the value is to 0, the better the prediction performance.

[0020] Table 1. Performance comparison with other advanced methods on the SJTU-PCQA database. Table 2. Performance comparison with other advanced methods on the WPC database. As can be seen from Table 1, for the monotonicity evaluation indices SROCC and KROCC, our method has the highest score in the SJTU-PCQA database; for the prediction accuracy index PLCC, our method has the highest score in the SJTU-PCQA database.

[0021] As shown in Table 2, our method achieves the highest scores on the WPC database for both the monotonicity evaluation metrics SROCC and KROCC; and for the prediction accuracy metrics PLCC and RMSE, our method has the highest PLCC value. In summary, our proposed method achieves significant results in point cloud quality evaluation and maintains good consistency with subjective human evaluation.

Claims

1. A no-reference point cloud quality assessment method based on graph convolution, characterized in that, include: Step 1: Project the distorted point cloud to obtain three types of two-dimensional projection images corresponding to the six different viewpoints of the distorted point cloud, namely texture image, depth image, and occupancy image. Step 2: Based on the projected image in Step 1, in each modality feature extraction sub-network, a deep residual convolutional neural network is used as the feature extraction backbone network to perform layer-by-layer convolutional feature extraction on the input image from each viewpoint, outputting high-dimensional semantic feature mapping, and outputting texture features, depth features, and occupancy features respectively. Step 3: Input the three types of high-dimensional semantic features obtained in Step 2 into the channel-space attention enhancement module. First, the channel dimension is weighted by response, and then the spatial dimension is weighted by saliency to highlight key semantic channels and important spatial regions, and obtain the enhanced texture feature map, depth feature map and occupancy feature map. Step 4: The enhanced trimodal feature maps obtained in Step 3 are subjected to global average pooling to compress the two-dimensional spatial features into one-dimensional node feature vectors of the corresponding views, thus forming a multi-view node feature set; then, the node features are linearly mapped, and the correlation score between any two view nodes is calculated by scaling the dot product, and then the adaptive adjacency matrix is ​​obtained by softmax normalization, thereby dynamically representing the connection strength and relationship weight between different views, and finally constructing the texture modality map structure, depth modality map structure and occupancy modality map structure respectively; Step 5: Input the node feature sets and their corresponding adjacency matrices of each modality generated in Step 4 into a multi-layer graph convolutional network. Use graph convolution operations to propagate and aggregate features between adjacent nodes. The adjacency matrix is ​​self-connected and symmetrically normalized before graph convolution to balance the impact of differences in degree between different nodes. The graph convolutional network adopts a multi-layer cascaded structure, gradually mapping node features from high-dimensional space to low-dimensional discriminative feature space. Batch normalization and non-linear activation functions are combined in each layer to improve representation ability. Then, the output of the third layer of the graph convolutional network is taken as the structure-aware node representation, and global average aggregation is performed on each view node in the view dimension to obtain global feature vectors for texture modality, depth modality, and occupancy modality, respectively. Step 6: Concatenate the global feature vectors of texture mode, depth mode, and occupancy mode obtained in Step 5 to form a joint representation vector. Then, project the joint representation to a unified latent space through a fully connected mapping to achieve linear coupling and alignment of information from different modes. Further, input the projected features into the feature enhancement unit based on the Transformer encoder. Through residual normalization and feedforward network structure, perform nonlinear modeling and robustness enhancement on the joint representation. Finally, output a quality score through a regression layer to predict the overall quality of the input sample.

2. The point cloud quality evaluation method according to claim 1, characterized in that, Given a point cloud: (1)， in Indicates the first The three-dimensional coordinates of a point in a spatial point cloud. This indicates the color attribute of that point. The index of a point in the point cloud. This represents the total number of points in the point cloud. To eliminate differences in spatial location and scale, the point cloud is first normalized; this specifically includes: calculating the geometric center of the point cloud. (2)， in This represents the average value of the point cloud in three-dimensional space, used to characterize the overall spatial distribution center of the point cloud. All points are translated to a coordinate system with the geometric center as the origin, and scaled according to the maximum Euclidean distance from each point in the point cloud to the geometric center, resulting in the normalized point coordinates: (3)， in Represents the Euclidean norm. For the first The three-dimensional spatial coordinates of the points For the normalized first The three-dimensional spatial coordinates of each point are used to constrain the point cloud within a unit sphere space, thereby eliminating differences in scale and position between different point cloud samples and improving the consistency and stability of subsequent multi-view processing. To comprehensively characterize the spatial structure information of the point cloud from multiple directions, six fixed viewpoints are preset, corresponding to the front view, rear view, left view, right view, top view, and bottom view obtained by rotating the point cloud around the three-dimensional coordinate axes. For each viewpoint... Set the corresponding camera extrinsic parameters, including the rotation matrix. Translation vector To determine the camera's spatial pose from that perspective; in the... From one perspective, the normalized point cloud is transformed from the world coordinate system to the camera coordinate system using camera extrinsic parameters. The transformation relationship is expressed as follows: (4) ， in and The first The rotation matrix and translation vector corresponding to each viewpoint For the first From the perspective of the first The points are represented in the camera coordinate system; then, an orthogonal photogrammetry model is used to project the 3D points onto the 2D image plane to obtain the projected coordinates. At the same time, the camera coordinate system Quantity As the corresponding depth information; based on the above projection results, the points are rasterized on the image plane, for any pixel position Select several projection points within its neighborhood and calculate the first projection point. Distance from each candidate point to the pixel center: （5）， in, Used to measure the spatial proximity between a candidate point and the current pixel; For the first The point at the th Projected from the first perspective, the first The coordinates of each candidate point in the camera coordinate system; pixel coordinates of a two-dimensional image plane In the horizontal direction, The vertical direction; based on this distance, a predefined attenuation function is applied. Calculate the coverage intensity of a point to a pixel. ,in The weights are obtained by normalizing the coverage intensity of all candidate points: (6)， in Indicates the first From the perspective of the first Candidate point pairs of pixels Normalization process, The candidate point number. This is a temporary index used when traversing all candidate points; using the aforementioned weights, the point attributes at the pixel are weighted and fused to generate the first... Texture maps from various perspectives: (7)， in Indicates the first Pixels from a single perspective The color value at that location, Represents texture modality, For the first From the perspective of the first The color attribute of the candidate point; simultaneously, the depth information of the candidate points is weighted and fused using the same weighting coefficient to generate the first... Depth map from multiple perspectives: (8)， in This represents the weighted depth information corresponding to the pixel, used to characterize the spatial structural relationship of the point set; Indicates the first The depth information of each candidate point in the camera coordinate system is obtained; simultaneously, a placeholder map is constructed based on whether a pixel is covered by any point, and the result is obtained by thresholding the alpha value at the pixel. (9)， in For indicator functions, This is a preset threshold. Indicates the first From the perspective of the first Candidate point pairs of pixels The normalization process is performed; when a pixel is covered by at least one point, the placeholder map is set to 1, otherwise it is set to 0; finally, the texture map and depth map are linearly normalized to map their numerical range to [0,1], thereby obtaining the point set in each viewpoint. texture map Depth map and placeholder images Three multi-view two-dimensional representations; ultimately, the total set of 6 views and 3 types of images under each view is: (10)， Through the above steps, 6 viewpoints × 3 types = 18 2D images can be generated for each point cloud to be evaluated. While ensuring the integrity of geometric structure and attribute information, the data consistency and representation stability are improved, providing high-quality input for subsequent point cloud analysis tasks.

3. The point cloud quality evaluation method according to claim 2, characterized in that, The three types of six-view images generated by data preprocessing are used to... For each modality, a separate ResNet101 feature extraction branch is constructed. Initial feature extraction is performed using a stem module consisting of 7×7 convolutions, batch normalization, ReLU activation, and max pooling. Subsequently, four residual stages—Layer 1, Layer 2, Layer 3, and Layer 4—are passed sequentially. Each stage consists of multiple Bottleneck residual blocks, with numbers of 3, 4, 23, and 3 respectively. Each Bottleneck block includes a 1×1 convolution for channel dimensionality reduction, a 3×3 convolution for spatial feature extraction, and a 1×1 convolution for channel dimensionality enhancement. Finally, residual connections and residual fusion are used to achieve the final feature extraction. The texture modality has 3 input channels and directly uses ImageNet pre-trained weights. The depth and occupancy modality have 1 input channel, and channel adaptation is performed on the first convolutional layer, inheriting pre-trained priors. The residual network extracts high-level semantic features layer by layer in the following manner: (11)， in This represents the ResNet101 feature encoder for the corresponding modality. Indicates the first From one perspective, Output images for each modality. The size of the output feature map is: (12)， in For batch size, For the number of viewpoints, For the number of channels, The height of the image. The width of the image; The input image first undergoes a large receptive field and pooling operation: (13)， in With a kernel size of 7×7 and a stride of s=2, the output channels are 64. For normalization operations, For activation function processing, the output size is ; , After max pooling, the features enter the core unit of ResNet-101, the Bottleneck residual block, whose mathematical form is: (14)， The main branch transformation Corresponding to three convolutional layers, The convolutional kernel is 1×1, and its function is channel-wise dimensionality reduction. The convolutional kernel is 3×3, and its function is spatial feature extraction. The convolution kernel is 1×1, and its function is to increase the dimensionality of the channels. It is the residual branch; ResNet-101 consists of 4 stages with a total of 33 bottlenecks. Each bottleneck has three convolutional layers. The image enters stage 1-Layer 1 and outputs 256 channels. This layer has 3 blocks without downsampling operations. , Learning about local edge combinations and simple structural shapes, stage2-Layer2 uses downsampling with stride=2. , This layer learns more complex local patterns and texture structures; stage3 (Layer 3) learns the geometric composition structure of middle-layer semantic parts. , The highest semantic feature layer, stage4-Layer4, ultimately yields a high-dimensional feature map. , Overall feature extraction process summary formula: （15）， The final high-dimensional feature map ; By downsampling, the original pixel-level representation is transformed into high-level semantic features with a larger receptive field, thereby significantly reducing the complexity of subsequent attention calculations while ensuring semantic expressiveness.

4. The point cloud quality evaluation method according to claim 3, characterized in that, The fourth-level feature map obtained from the fourth stage output of the backbone network ResNet101. The input is fed into the attention module. To facilitate unified convolution operations on features from each viewpoint, the batch dimension and viewpoint dimension are merged. Performing global average pooling in the spatial dimension yields: = （16）， in This refers to global average pooling. It refers to High-dimensional feature maps in different modalities This represents the summation over all pixels in the feature map's spatial dimension (height × width). Global average pooling is used to aggregate the response intensity of each channel globally into a scalar, forming a channel-level global statistic for subsequent estimation of "which channels are more important". After pooling, channel compression and expansion are performed using two 1×1 convolutions to achieve channel dimensionality reduction and restoration. (17)， in , For compression ratio (r=16), Represents the ReLU activation function. This represents the channel weight tensor output by the Sigmoid activation function. This step primarily works by reducing dimensionality, learning the dependencies between channels, and then restoring the dimensionality to generate weight coefficients for each channel, thereby enhancing discriminative channels and suppressing redundant channels; then, channel weighting is performed. Where ⊙ represents element-wise multiplication; important channels are amplified, and unimportant channels are suppressed; the output... Calculate the channel average chart and the channel maximum chart separately: , The mean plot reflects the overall activation intensity, indicating These are the pixel coordinates in the feature map space. The feature map is a channel-weighted map, and the maximum value map reflects the most significant response. Combining the two can more stably indicate which locations are more critical in space. Spatial attention feature map Then, pass it through a 7×7 convolution and apply Sigmoid activation: （18）， in For the convolution kernel used to extract spatial weights, The spatial attention weight map is then spatially weighted to obtain the channel-space dual-attention weighted feature map: Then restore the perspective dimension: (19)， After completing channel and spatial attention weighting, a global average pooling operation is further performed on the enhanced feature map to aggregate the responses in the spatial dimension and obtain the high-dimensional semantic feature vector corresponding to each viewpoint. (20)， in =1,...., ; =1,....., Concatenate all the feature vectors to obtain the feature matrix of the node: (21)， This operation can effectively eliminate spatial redundancy while preserving the key regional information emphasized by the attention mechanism, and transform the feature representation from a two-dimensional spatial form to a view-level node representation, providing a unified and stable input feature for subsequent graph-based relation modeling.

5. The point cloud quality evaluation method according to claim 4, characterized in that, The viewpoint node feature matrix for a specific modality, after enhancement by the attention module and global average pooling, is as follows: (22)， in For batch size, For the number of viewpoints, The node feature dimension is 2048. Indicates the first The first sample The feature vectors of each viewpoint; this tensor serves as the input to the graph structure generation module; the goal of the graph structure generation module is to automatically construct a graph structure of relationships between viewpoints based on the feature similarity between nodes from different viewpoints, and obtain the adjacency matrix: The graph structure is represented as in The node feature matrix, To improve the stability of graph structure learning, the node features are first mapped to a low-dimensional embedding space, using the adjacency matrix: (23)， (24)， in The learnable parameter matrix; For projection dimensions; These refer to calculating the similarity between any two viewpoint nodes by using low-dimensional embedded features after different matrix mappings: （25）， in ; Indicates the first With the The degree of correlation between perspectives Using the scaling factor, the adjacency matrix is ​​then normalized using the Softmax function. （26）， in This constructs a graph structure. The node set is composed of the node feature matrix. Represented by the adjacency matrix; edge weights are determined by the adjacency matrix. express; It is a natural exponential function, and the graph structure is a dynamically constructed graph.

6. The point cloud quality evaluation method according to claim 5, characterized in that... The adjacency matrix obtained in the previous step is normalized by first adding the identity matrix. To preserve node information during propagation and prevent node features from being "completely overwritten" by neighbor information, the degree matrix is ​​then calculated. After performing stability processing, a normalized adjacency matrix is ​​constructed: （27）， Its main function is to normalize the influence of nodes with different degrees, preventing "nodes with more connections" from dominating other nodes during propagation; this is crucial for graph convolution propagation. （28）， in These are the node features of the l-th layer; These are the learnable weights of the l-th layer; This means that each node sums the features of its neighbors according to the edge weights. It is handled by the Softplus activation function; ; ; ; ; Used to output the score for each node: (29)， in It is the first Features of each viewpoint node after the third convolutional layer; mean convergence will... Each node is integrated into a fixed-dimensional global modality vector; the third layer is used because it retains richer semantic representation (32 dimensions), while the fourth layer is more like a "scalar scoring" with less information, ultimately outputting three unimodal global features. .

7. The point cloud quality evaluation method according to claim 6, characterized in that... The global features of the three modalities are fused, and the above steps are output. As input, feature concatenation is performed first: (30)， in This is a channel-level splicing operation. Using the sample sequence number, the representations of the three modalities are concatenated along the channel direction to form a joint feature vector containing texture, depth, and occupancy information. This provides the input basis for subsequent unified spatial projection and cross-modal semantic modeling. The concatenated vector has a dimension of 96. The concatenated low-dimensional vector is then mapped to the first fully connected layer fc_in of the higher-dimensional fusion space. (31)， in It is 256. For the first The cross-modal feature vector of each sample after linear mapping The input layer is a learnable weight matrix. As the learnable bias vector for the input layer, a linear mapping layer projects the concatenated cross-modal joint vector onto a unified hidden feature space, aligning and blending high-level semantic information from different modalities on the same feature basis. This layer achieves cross-modal linear fusion. Then, a Transformer encoder is used to perform high-dimensional nonlinear reconstruction and semantic enhancement on the concatenated cross-modal joint features. (32)， in For the first The sequence-form features of each sample after dimensionality reshaping The dimension expansion operation transforms the sequence [B,1,256] into the encoder: (33)， (34)， in The only exception is the compression operation. This is the globally fused feature vector after dimensionality compression. The fused features output by the encoder are reconstructed to a shape of [B, 256]. This encoder can be expanded to multiple tokens before quality prediction is performed. (35)， in It is a learnable matrix; For learnable bias vectors; the fused ones Mapped to a scalar; For the first For each sample, the model predicts an objective score for the point cloud quality. During the training phase, a joint loss function consisting of ranking loss and regression loss is constructed to simultaneously constrain the relative ranking relationship and absolute scoring error of the prediction results. The regression loss can be either absolute error loss or mean squared error loss. In some implementations, when only the scoring regression accuracy needs to be optimized, the ranking loss weight can be set to zero. (36)， in To regress the loss, or , For regression loss weighting coefficients, For ranking loss, The weighting coefficients for the ranking loss are used when the SJTU database is... The method adopted is Regression loss, when using WPC database The method adopted is loss; It is responsible for sorting consistency; through joint optimization, it achieves a synergistic improvement in sorting performance and numerical accuracy.

Images