Multi-modal point cloud quality evaluation method based on adaptive block projection

Abstract

The invention relates to the field of three-dimensional data processing, and discloses a multi-modal point cloud quality evaluation method based on adaptive block projection, comprising the following steps: preprocessing point cloud data, dividing the point cloud data into a plurality of local clusters, each local cluster comprising a central point and neighborhood points thereof; generating a two-dimensional cluster projection drawing corresponding to the spatial position of each local cluster based on the local cluster; extracting features of the two-dimensional modal projection drawing and the three-dimensional local cluster data, and respectively obtaining two-dimensional modal features and three-dimensional modal features; decoupling the two-dimensional modal features and the three-dimensional modal features to generate modal-independent quality features and modal-related features; the modal-independent quality features are mapped to a shared unified feature representation space; fusing the features of the two-dimensional mode and the three-dimensional mode to generate a joint feature; and based on the joint features, predicting the quality score of the point cloud through a regression network. According to the invention, the accuracy, robustness and calculation efficiency of point cloud quality evaluation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional data processing, and in particular to a multi-modal point cloud quality assessment method based on adaptive block projection. Background Art

[0002] In recent years, with the rapid development of 3D acquisition and processing technology, point cloud, as an important form of 3D data representation, has been widely used in fields such as autonomous driving, virtual reality, and computer graphics. Point cloud consists of a large number of unordered points with geometric coordinates and attribute information (such as color, normal vector, etc.), providing highly refined information for a variety of application scenarios. However, the high precision and large-scale characteristics of point cloud data place high demands on storage and transmission. To this end, the International Organization for Standardization MPEG has proposed compression schemes for static point clouds and dynamic point clouds, such as geometry-based point cloud compression (G-PCC) and video-based point cloud compression (V-PCC). However, these compression methods are usually lossy and may introduce distortions such as structural damage and texture blur. In addition, environmental interference during the acquisition process and channel errors during the transmission process can also lead to a decrease in the quality of point cloud data, which will have an adverse effect on the perceptual experience of end users.

[0003] In the field of point cloud quality assessment, although subjective evaluation methods can directly reflect the perceived quality of users, they are costly, time-consuming, and difficult to adapt to the processing requirements of large-scale data. Therefore, objective point cloud quality assessment (PCQA) methods have gradually become a research hotspot. Existing objective methods are mainly divided into full reference (FR), semi-reference (RR), and no-reference (NR) methods. Among them, the full-reference method requires a complete reference point cloud as a comparison basis, and evaluates the quality by directly calculating the geometric or attribute differences between the distorted point cloud and the reference point cloud. However, the applicability of such methods is limited by the difficulty of obtaining reference point clouds in actual scenes. Semi-reference methods reduce dependence on complete reference point clouds by transmitting partial reference information, such as statistical features, but their practical applications are still limited. In contrast, no-reference methods do not need to rely on reference point clouds and have broader application prospects, but how to accurately evaluate the quality of point clouds under no reference conditions is still a challenging problem.

[0004] Existing methods for quality assessment of reference-free point clouds mainly include point-based methods, projection-based methods, and multimodal methods. Point-based methods directly extract geometric and attribute information from three-dimensional point clouds, but their computational efficiency for large-scale point clouds is low, and it is difficult to fully capture the perceptual features of point clouds. Projection-based methods facilitate point cloud quality assessment using image quality assessment technology by projecting point clouds onto two-dimensional planes. However, since projection loses some three-dimensional spatial information, it is less sensitive to geometric-related distortions. Multimodal methods combine the feature information of three-dimensional point clouds and two-dimensional projections to more comprehensively characterize the quality of point clouds. However, existing multimodal methods have deficiencies in the alignment and fusion of modal features. For example, features of different modalities may not be effectively fused due to differences in feature expression forms. At the same time, existing fusion methods have high computational complexity, which limits their efficiency in practical applications. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a multimodal point cloud quality assessment method based on adaptive block projection, which effectively improves the accuracy, robustness and computational efficiency of point cloud quality assessment through adaptive projection, multimodal feature alignment and fusion under reference-free conditions.

[0006] To achieve the above objectives, the present invention is implemented by the following technical solutions: a multimodal point cloud quality assessment method based on adaptive block projection, comprising the following steps: Preprocess the point cloud data and divide it into multiple local clusters, each of which contains the central point and its neighboring points; Generate a 2D cluster projection map corresponding to each local cluster's spatial position; Extract the features of the 2D modal projection image and the 3D local cluster data to obtain the 2D modal features and 3D modal features respectively; Decouple 2D modal features and 3D modal features to generate modality-independent quality features and modality-dependent features; Mapping modality-independent quality features into a shared unified feature representation space; Fusion of 2D and 3D modality features to generate joint features; Based on the joint features, the quality score of the point cloud is predicted through a regression network.

[0007] The present invention also provides a multimodal point cloud quality assessment device based on adaptive block projection, comprising: The preprocessing module is used to divide the point cloud data into blocks and generate local point cloud clusters; Adaptive cluster projection module, used to generate a 2D cluster projection map corresponding to each point cloud cluster and optimize the projection direction; The multimodal feature extraction module extracts the texture features of the 2D modal projection image and the geometric features of the 3D point cloud cluster respectively; A feature alignment module to align modality-independent quality features into a shared unified feature representation space; The regression network module is used to fuse 2D and 3D modality features and output point cloud quality scores.

[0008] The present invention provides a multimodal point cloud quality assessment method based on adaptive block projection, which has the following beneficial effects: 1. The adaptive block projection strategy proposed in this invention avoids the redundant calculation in the traditional multi-faceted projection method by learning the optimal projection direction of the local cluster, while retaining sufficient quality information.

[0009] 2. The method of the present invention makes up for the limitations of a single modality method in describing point cloud distortion by combining the texture and color features of the 2D projection modality with the geometric and structural features of the 3D point cloud modality.

[0010] 3. The feature alignment strategy designed by the present invention reduces the distribution differences between different modalities by mapping the quality features of 2D and 3D modalities to a shared unified feature representation space. This strategy not only improves the effect of multimodal feature fusion, but also better simulates the process of human subjective quality assessment, thereby improving the robustness and consistency of the overall assessment.

[0011] 4. The method of the present invention adopts a modular design, including an adaptive projection module, a feature alignment module and a regression prediction network, so that the method can flexibly adapt to different types of point cloud distortion tasks. This modular design shows excellent generalization ability on different databases and can be applied to a variety of scenarios from large-scale datasets to small-scale datasets.

[0012] 5. The total loss function of the present invention combines MOS loss, embedding loss, reconstruction loss and decoupling loss, and comprehensively improves the quality prediction ability of the model through a multi-objective optimization strategy. This design not only ensures the accuracy of point cloud quality prediction, but also further optimizes the effect of multimodal feature mapping and fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the method flow of the present invention; Figure 2 The overall framework diagram of the method of the present invention, wherein (a) is a point cloud preprocessing and adaptive cluster projection module, and (b) is a schematic diagram of a multimodal feature alignment strategy; Figure 3 is a schematic diagram of an adaptive block projection module of the present invention; Figure 4 Schematic diagram of the PCN network structure of the present invention; Figure 5 Schematic diagram of posture changes of the test set projection during the pre-training PCN process of an embodiment of the present invention; Figure 6 It is a schematic diagram of the device structure of the present invention.

[0014] Among them, 10, preprocessing module; 20, adaptive cluster projection module; 30, multimodal feature extraction module; 40, feature alignment module; 50, regression network module. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] Please refer to the attached Figure 1 -Attached Figure 5 ,The present invention provides a multi-modal point cloud quality assessment method based on ,adaptive block projection, which aims to improve the accuracy and robustness of point cloud quality ,assessment by mining and fusing the 2D and 3D modal information of point cloud data.

[0017] like Figure 1 As shown, the multimodal point cloud quality assessment method based on adaptive block projection may include the following steps: S1, preprocessing point cloud data; S2, generating a 2D cluster projection map corresponding to each local cluster's spatial position; S3, extracting features of 2D modal projection image and 3D local cluster data; S4, decoupling 2D modal features and 3D modal features; S5, mapping modality-independent quality features to a shared unified feature representation space; S6, integrating the features of the 2D modality and the 3D modality to generate a joint feature; S7. Based on the joint features, the quality score of the point cloud is predicted through the regression network.

[0018] The following is a detailed description of each step of the method.

[0019] For step S1, in this embodiment, the input point cloud data is preprocessed to divide the original point cloud into multiple local clusters to adapt to the input requirements of the subsequent feature extraction module while retaining the local spatial characteristics and quality information of the point cloud.

[0020] As an option, the input point cloud data can be represented as: in: Represents each point in the point cloud The three-dimensional coordinates of Indicates that each point Dimensions additional attribute features, such as color, normal vector, or intensity value.

[0021] It is understandable that point cloud data is usually an irregular and disordered three-dimensional point set. Directly processing large-scale point cloud data may lead to excessive computational complexity. Therefore, in a possible implementation, the point cloud data is pre-processed to divide the point cloud into multiple local clusters to adapt to the subsequent network structure and reduce the computational burden.

[0022] In this embodiment, the specific block operation includes the following steps: First, the Farthest Point Sampling (FPS) algorithm is used to extract the point cloud data. Select points as the center points of the local cluster. The selected center point set can be expressed as: It should be noted that the FPS algorithm gradually selects the point with the maximum distance from the selected point set in an iterative manner, thereby ensuring that the selected center points are evenly distributed and can better cover the entire point cloud data.

[0023] Specifically, the initial step of the FPS algorithm is to randomly select a point as the first center point. In each subsequent iteration, the point with the largest distance to the currently selected center point set is selected from the remaining point set until the selected center point set is reached. Up to the center point.

[0024] After the center point selection is completed, for each center point , using the nearest neighbor search algorithm (KNearest Neighbors, KNN) from the point cloud data Select nearest neighbor points, forming a The local point cloud cluster with The specific representation of the local point cloud cluster is: in, Represents the first point searched from the point cloud nearest neighbor points.

[0025] It should be noted that the KNN algorithm performs a neighborhood search for each point in the point cloud data based on the Euclidean distance to determine the distance from the center point. Recent This search method can ensure that each local cluster has strong local correlation in space, thus providing more meaningful local geometry and attribute information for the subsequent feature extraction step.

[0026] In a possible implementation, the entire point cloud segmentation process will eventually result in M ​​local point cloud clusters, each of which consists of a central point and its neighborhood points. The set of these point cloud clusters can be expressed as: It should be noted that the size of the local cluster is determined by the number of neighborhood points Determines the number of clusters It is determined by the number of center points sampled by FPS. It can be flexibly adjusted according to different application requirements. and The value of is used to balance the computational complexity and the completeness of the local point cloud information.

[0027] In some embodiments, the divided local clusters retain the geometric structure and additional attribute feature information of the original point cloud. For example, some point cloud data may contain color information (RGB values) or normal vector information, which will be retained during the block division process to facilitate a comprehensive evaluation of the point cloud quality features in subsequent steps.

[0028] It can be understood that through the above preprocessing steps, the point cloud data is divided into multiple local clusters, each of which contains a central point and its neighboring points. This block division method not only reduces the computational complexity of directly processing the complete point cloud, but also retains the local geometric structure and attribute characteristics of the point cloud, laying the foundation for subsequent feature extraction and quality assessment steps.

[0029] For step S2, in this embodiment, a 2D cluster projection map corresponding to the spatial position of each local cluster is generated, aiming to provide input for subsequent 2D modal feature extraction, while ensuring strict spatial correspondence between the 2D projection and the 3D cluster data.

[0030] It is understandable that the 3D spatial information of the point cloud needs to be mapped to a 2D plane through projection in order to extract features using mature image processing technology. However, the effect of projection depends largely on the choice of projection direction and the amount of information in the projection image. To this end, this embodiment uses an adaptive patch projection module (APPM) to optimize the projection generation of each local cluster.

[0031] In this embodiment, the selection of the projection direction is based on the calculation of the normal vector of the surface of the point cloud cluster. Specifically, the eigenvalue decomposition of the covariance matrix is ​​used to estimate the normal vector of each point in the point cloud cluster. The formula for calculating the covariance matrix is: in, , , , respectively, Centered at The local point set All points in , and Coordinate set, represents the calculation of the covariance between the two. and are the diagonal matrices consisting of the eigenvector matrix and the corresponding eigenvalue of the covariance matrix. In the above formula, the eigenvector corresponding to the smallest eigenvalue is the point The normal vector .

[0032] It should be noted that the estimation of the normal vector can effectively characterize the local geometric characteristics of the point cloud cluster and provide a basis for the optimization of the projection direction.

[0033] In a possible implementation, in order to maximize the amount of information in the projection image, this embodiment calculates the effective amount of information in the projection direction based on the cosine similarity between the normal vector and the normal vector of the projection plane. The specific formula is: in: Represents a local cluster In the projection direction The amount of effective information under is the normal vector of the projection plane; is the normal vector of the point in the local cluster; is the cosine similarity of two vectors, and its calculation formula is: like Figure 3 As shown, the present invention uses an adaptive block projection module (APPM) to optimize the pose of the point cloud cluster and generates a transformation matrix by learning the relationship between the normal vector and the normal vector of the projection plane. , adjust the point cloud cluster’s pose. The adjusted point cloud cluster can be expressed as: in: is a 3×3 rotation matrix; and are the coordinates and attributes of the points in the point cloud cluster respectively; is the adjusted point.

[0034] It should be noted that the transformation matrix The generation of is based on the Patch Pose Correction Network (PCN) network, such as Figure 4 As shown in the figure, a schematic diagram of the PCN network structure is shown. The network adaptively adjusts the pose of the point cloud by learning the normal vector distribution of the point cloud cluster to ensure the optimality of the projection direction.

[0035] After the pose adjustment is completed, the adjusted point cloud cluster Through the projection camera Map to a two-dimensional plane and generate the corresponding 2D cluster projection map The projection operation can be expressed as: in, Represents the projection operation of the point cloud to the 2D plane. For the The two-dimensional projection image corresponding to the local clusters.

[0036] In some embodiments, in order to reduce computational complexity, the projection operation for the local cluster only generates a single-view projection image instead of a conventional multi-view projection. This is because the spatial structure of the local cluster is relatively simple, and the single-view projection can fully retain key quality information.

[0037] It can be understood that through the above optimization steps, each local cluster generates a 2D cluster projection map that strictly corresponds to its spatial position. These projection maps can not only represent the local texture information of the point cloud cluster, but also avoid the spatial position deviation between the modalities, providing a reliable input for the subsequent 2D modal feature extraction.

[0038] For step S3, in this embodiment, by performing feature extraction on the 2D modal projection image and the 3D local cluster data, 2D modal features and 3D modal features related to the point cloud quality are obtained respectively, providing input data for subsequent feature decoupling and multimodal fusion.

[0039] Each 2D modal projection image is generated by step S2 and is represented as , each 3D local cluster is represented as .

[0040] It is understandable that the 2D modal projection image and the 3D local cluster contain different dimensional information of the point cloud. The 2D modal projection image mainly represents the texture, color and other characteristics of the point cloud, while the 3D local cluster focuses more on spatial characteristics such as geometry, structure and normal vector. Therefore, extracting the features of the two modalities separately can make full use of the advantages of the two data forms.

[0041] In a possible implementation, for a 2D modal projection image , and extract its texture, color and other related features through the 2D feature extraction network. The extraction process can be expressed as: in: Represents 2D modal features; It is a 2D feature extraction network; is the dimension of the extracted 2D modal features.

[0042] As an option, A deep model based on a convolutional neural network (CNN) may be used, such as ResNet, VGG, etc. For example, for a smaller 2D projection image, a shallower network structure may be selected to reduce computational complexity.

[0043] It should be noted that the training goal of the 2D modal feature extraction network is to extract quality-related features in the projection image, including texture clarity, color consistency, and distortion information in the projection.

[0044] For 3D local clusters , extract its geometric and spatial structure features through the 3D feature extraction network. The extraction process can be expressed as: in: Represents 3D modal features; It is a 3D feature extraction network; is the dimension of the extracted 3D modal features.

[0045] In one possible implementation, Network structures suitable for point cloud data can be used, such as PointNet, PointNet++, or sparse convolution-based networks (SparseConvNet). These networks can directly process point cloud data and extract local geometric features from the spatial distribution and attributes of points.

[0046] It should be noted that the training goal of the 3D modal feature extraction network is to extract the geometric information of local clusters of point clouds, including the integrity of the structure, the density distribution of point clouds, and the influence of noise.

[0047] As an option, for the entire point cloud data, the feature set extracted from the 2D modal projection map is expressed as: The feature set extracted by the 3D local cluster is expressed as: in, Represents the total number of local clusters.

[0048] It can be understood that these feature sets retain information about the quality of the point cloud in both 2D and 3D modalities. More sensitive to distortion in projection layers such as texture and color, while the 3D modal feature set It is more sensitive to anomalies in geometric structure and spatial distribution.

[0049] In some embodiments, the depth and parameters of the feature extraction network can be adjusted according to specific task requirements. For example, for point cloud quality assessment tasks that require higher precision, the number of layers of the feature extraction network can be increased to improve its feature expression capabilities; and for resource-constrained application scenarios, the network depth can be trimmed to reduce computational overhead.

[0050] Through the solution of this embodiment, the feature extraction operation of point cloud data in 2D and 3D modes is completed, providing the necessary input data for the subsequent multi-modal feature decoupling and fusion. The above extraction process has strong robustness and generalization ability, can accurately capture the key features of point cloud quality, and ensure the effectiveness of subsequent evaluation tasks.

[0051] For step S4, in this embodiment, the extracted 2D modal features and 3D modal features are decoupled to generate modality-independent quality features and modality-dependent features, respectively, laying the foundation for subsequent feature mapping to a unified feature representation space and fusion operations.

[0052] It is understandable that 2D modal features and 3D modal features come from different modes of point clouds, and these features may have significant differences in characteristics and distribution. If feature fusion is performed directly, modal differences may affect the joint expression ability of multimodal features. Therefore, this embodiment separates modal features into modality-independent quality features and modality-related features by feature decoupling, thereby reducing the differences between modalities.

[0053] In a possible implementation, the 2D modal features and the 3D modal features are decoupled by a quality feature encoder and a modal feature encoder, respectively. and 3D modal features , the decoupling process is expressed as: in: It represents the quality characteristics that are independent of the mode and is mainly used to reflect the overall quality information of the point cloud; Represents modality-related features, used to retain modality-specific expression information; is the quality feature encoder, is the modal feature encoder; Corresponding to and , representing 2D modal features and 3D modal features.

[0054] It should be noted that both the quality feature encoder and the modal feature encoder can adopt a multi-layer perceptron (MLP) structure to decompose the features through a nonlinear activation function.

[0055] As an option, in order to ensure the decoupling independence of the modality-independent quality features and the modality-dependent features in the feature space, this embodiment uses the distance correlation metric to constrain the decoupling process. The calculation formula of the distance correlation metric is: in: Represents two eigenvectors and The distance correlation of is the distance covariance, and its calculation formula is: and Respectively represent the central distance between samples; Indicates the sample size.

[0056] In one possible implementation, for each modal feature, a distance correlation metric is used to constrain the modal-independent quality feature. and modality-related features The corresponding constraint loss is expressed as: It should be noted that, through this constraint mechanism, the redundant information between modality-independent features and modality-dependent features can be effectively reduced, thereby improving the accuracy of feature decoupling.

[0057] In an exemplary implementation, in order to ensure that the decoupled features can fully express the original modal features, this embodiment further introduces reconstruction loss. , the modal-independent quality characteristics and modality-related features After cascading, using the decoder Reconstructed into the original modal features, the loss of the reconstruction process is expressed as: in: is the modal decoder; Represents a feature cascade operation.

[0058] In some embodiments, in order to further improve the decoupling effect, the structure of the feature encoder and decoder can be adjusted according to the actual scene. For example, for point cloud data in complex scenes, the number of encoder layers can be increased to enhance its feature expression ability; and for application scenarios with high real-time requirements, the network depth can be trimmed to reduce the computational overhead.

[0059] Through the solution of this embodiment, 2D modal features and 3D modal features are successfully decoupled into modality-independent quality features and modality-dependent features. Among them, the modality-independent quality features focus on the global information of point cloud quality, providing a unified expression basis for subsequent feature mapping and fusion; the modality-dependent features retain the modality-specific local information, which is used to improve the refinement of multimodal fusion. This decoupling mechanism effectively alleviates the impact of differences between modalities on feature expression capabilities, and provides technical support for the accuracy and robustness of subsequent point cloud quality assessment.

[0060] For step S5, in this embodiment, the decoupled modality-independent quality features are mapped to a shared unified feature representation space, aiming to align different modality features through a unified feature expression method and reduce the impact of modality differences on feature fusion.

[0061] It is understandable that the 2D modal and 3D modal features are derived from different modes of the point cloud data. Even after decoupling, the modal-independent quality features There may still be some differences in feature distribution and spatial expression. If the features are directly fused, the features between the modalities may not be fully matched. Therefore, this embodiment constructs a discrete unified feature representation space , and the modality-independent quality features are mapped to the shared feature space using the nearest neighbor search method to achieve alignment of different modal features.

[0062] In one possible implementation, the unified feature representation space It consists of a series of feature vectors (i.e., "codewords"), expressed as: in: represents the first feature vectors; The number of codewords representing the feature space; Represents the dimension of the feature vector.

[0063] It should be noted that the feature representation space Codeword It is a learnable parameter that can be adaptively adjusted during training to better fit the distribution of input features.

[0064] As an option, for each modally independent quality feature , and map it to a unified feature space by searching for the nearest neighbor The mapping process can be expressed as: in: Representing quality characteristics Mapping results in a unified feature space; represents the Euclidean distance.

[0065] In this way, each modality-independent quality feature is quantized and aligned by finding the closest feature vector. It should be noted that this mapping method can reduce the distribution differences of modal features while providing consistent quality representation in the feature space.

[0066] In an exemplary implementation, in order to optimize the feature mapping process, this embodiment jointly optimizes the encoder and the feature space. To this end, the embedding loss function is introduced The learning process for constraining the feature encoder and unifying the feature space is specifically expressed as: in: Indicates the operation of stopping gradient backpropagation, which is used to fix the optimization direction of some parameters; Represents the L2 norm, which is used to measure the distance between features.

[0067] It can be understood that by embedding the constraints of the loss function, the encoder outputs a modality-independent quality feature and the codewords in the feature space Closer, thus optimizing the feature mapping effect.

[0068] In some embodiments, in order to improve the learning efficiency of the feature space and the adaptability to the modal features, the number of codewords in the feature space can be increased. Or adjust the dimension of the feature vector To enhance the spatial representation capability. For example, in the case of point cloud data with complex distortion, a larger The value can be used to quantify the quality characteristics in a more fine-grained manner; in scenarios with high real-time requirements, or To reduce computational overhead.

[0069] Through the solution of this embodiment, each modality-independent quality feature is successfully mapped to a shared unified feature representation space. The mapped features can not only maintain the global information of point cloud quality, but also reduce the difference in feature expression between different modalities, providing a unified representation basis for subsequent multimodal feature fusion. This mapping mechanism based on a shared feature space significantly improves the robustness and consistency of point cloud quality assessment.

[0070] For step S6, in this embodiment, by fusing the features of the 2D modality and the 3D modality, a joint feature is generated to provide a unified feature expression for the quality prediction of the point cloud. The features of the 2D modality and the 3D modality are derived from two different expressions of the point cloud, which respectively reflect the texture and color information of the point cloud on the projected image, and the structure and distribution information in the three-dimensional geometric space. Through feature fusion, the information of the two modalities can be comprehensively utilized to improve the accuracy and robustness of the point cloud quality assessment.

[0071] It can be understood that after being mapped into the unified feature representation space, the 2D modal features and the 3D modal features have been aligned into the same feature space. The 2D modal feature representation at this time is , the 3D modal feature is expressed as ,in Represents the number of local clusters.

[0072] In one possible implementation, for each local cluster, its corresponding 2D modal feature and 3D modal features Perform cascade operations to generate joint features . Specifically expressed as: in: For the The joint features of local clusters; Represents the feature concatenation operation, which concatenates the feature vectors of the two modes by dimension; ,in is the dimension of 2D and 3D modality features.

[0073] It should be noted that, through the cascade operation, the joint feature not only retains the information of the two modal features, but also forms a unified representation in the feature space, providing a complete feature expression for the subsequent point cloud quality prediction.

[0074] In an exemplary implementation, in order to further improve the expressiveness of the joint features, the cascaded features can be further processed. For example, by introducing a shallow multi-layer perceptron (MLP) network, the cascaded features are nonlinearly transformed to further fuse the information of the two modalities. The specific formula for this operation is: in: For the feature fusion network, an MLP consisting of several fully connected layers and activation functions (such as ReLU or LeakyReLU) can be used; is the final joint feature, indicating The complete features of a local cluster.

[0075] It is understandable that this approach can better capture the relationship between modalities and improve the representation ability of joint features.

[0076] As an option, the joint features of all local clusters can form a feature set, expressed as: or: in, is the joint feature set after direct cascading, It is the joint feature set after being processed by the fusion network.

[0077] In some embodiments, in order to reduce the feature dimension, the cascaded features can be subjected to a dimensionality reduction operation. For example, the joint features are mapped to a lower dimensional space through a fully connected layer, which is expressed as: in: It is a dimension reduction network; .

[0078] It should be noted that the dimensionality reduction operation can effectively reduce the computational overhead while maintaining the main information of the joint features.

[0079] Through the solution of this embodiment, the 2D modality and 3D modality features are successfully fused in a unified feature representation space to generate a complete joint feature. This fusion method not only retains the feature advantages of the two modalities, but also optimizes the differences between the modalities so that the joint feature can more comprehensively reflect the overall quality information of the point cloud.

[0080] For step S7, in this embodiment, the joint features are used to predict the quality of the point cloud, aiming to quantify the perceived quality of the point cloud through a regression network. The quality prediction score of each cluster is obtained by regressing the joint features of the local clusters of the point cloud, and the scores of all local clusters are summarized to generate the overall quality score of the point cloud.

[0081] It is understandable that the quality score of the point cloud reflects the user's subjective rating (Mean Opinion Score, MOS) of the point cloud data. In order to simulate this subjective perceived quality, this embodiment introduces a regression network to perform regression prediction on the joint features to minimize the error between the predicted score and the true score.

[0082] Alternatively, the joint feature set is represented as: in: For the The joint features of local clusters; Indicates the number of local clusters for point cloud division.

[0083] In one possible implementation, the regression network uses a multi-layer perceptron (MLP) structure. Specifically, the regression network consists of several fully connected layers and nonlinear activation functions (such as ReLU or Leaky ReLU). The input of the regression network is the joint feature , the output is The prediction quality scores of local clusters The prediction process is expressed as: in: represents a regression network; For the The prediction quality scores of local clusters.

[0084] It should be noted that the training goal of the regression network is to minimize the prediction score With the true score The mean squared error (MSE) of the model is expressed as: in: is the size of mini-batch; and are the predicted score and the true score respectively.

[0085] In an exemplary implementation, in order to obtain the quality score of the entire point cloud, the prediction scores of all local clusters are weighted and summarized, which is specifically expressed as: in: is the prediction quality score of the entire point cloud; For the The weights of local clusters satisfy .

[0086] It is understandable that by introducing weights , the prediction score can be adjusted according to the contribution of each local cluster to the overall quality of the point cloud. For example, a higher weight can be given to a local cluster in a dense point area, while a lower weight can be given to a local cluster with more noise or outliers.

[0087] In order to further optimize the quality evaluation network, the total loss function of this embodiment is composed of multiple parts. Finally, the total loss function of the quality evaluation network is expressed as: in: is a hyperparameter used to control the proportion of different loss terms; is the mean square error loss, which is used to optimize the prediction of the overall quality score of the point cloud; It is the embedding loss, which is used to optimize the mapping effect of 2D and 3D modality features in a unified feature representation space; It is the reconstruction loss, which is used to ensure that the decoupled features can fully reconstruct the original modal features; is the distance correlation loss, which is used to constrain the independence of modal-independent quality features and modal-dependent features.

[0088] It should be noted that by optimizing the total loss function, the objectives of different loss items can be coordinated to further improve the prediction accuracy and feature expression ability of the network.

[0089] Through the solution of this embodiment, the joint features of the point cloud are successfully mapped to the prediction quality score, realizing the quantitative evaluation of the perceived quality of the point cloud. At the same time, by introducing multiple loss functions, the prediction accuracy, feature mapping effect, feature decoupling integrity and other aspects are comprehensively optimized. This multi-objective optimization strategy significantly improves the performance of the point cloud quality evaluation network and provides a high-precision and high-robustness solution for practical applications.

[0090] Experimental Example: This paper conducts a series of experiments on the proposed multimodal point cloud quality assessment method based on adaptive block projection to verify the effectiveness and superiority of the method. The experiments include performance evaluation on public datasets, comparative analysis with existing technologies, and module effectiveness verification. The experimental results and related analysis further illustrate the advantages of the method in the multimodal point cloud quality assessment task.

[0091] Experimental Datasets and Evaluation Metrics The performance of the method of the present invention is evaluated on the following public datasets: 1.WPC dataset: contains 20 reference point clouds and 740 distorted point clouds. Each reference point cloud corresponds to 4 different types of distortion, including downsampling (DS), Gaussian noise (GN), G-PCC encoding, and V-PCC encoding. Among the different types of distortion, 3 levels of DS, 9 levels of GN, 16 types of G-PCC compression, and 9 types of V-PCC compression are generated for each reference point cloud.

[0092] 2. SJTU-PCQA dataset: contains 9 reference point clouds and 378 distorted point clouds. Each reference point cloud corresponds to 7 different distortion types, including octree compression (OT), color noise (CN), downsampling (DS), downsampling and color noise (D+C), geometric Gaussian noise (GGN), etc. Each distortion has 7 distortion levels.

[0093] 3. CPCD2.0 dataset: contains 10 reference point clouds and 360 distorted point clouds. Each reference point cloud corresponds to 4 different types of distortion, including G-PCC-1 coding distortion, G-PCC-2 coding distortion, V-PCC coding distortion and Gaussian noise (GN). Each distortion type has 9 distortion levels.

[0094] To ensure the accuracy of the evaluation, the following data division and evaluation strategies are adopted: The WPC, SJTU-PCQA and CPCD2.0 datasets used 5-fold, 9-fold and 10-fold cross validation respectively, and the average performance of all test scenarios was calculated to obtain the final results.

[0095] The performance of the method of the present invention is evaluated using the following evaluation indicators: 1. Spearman rank correlation coefficient (SROCC): measures the monotonic relationship between the predicted score and the subjective score. The calculation formula is: in, Indicates The difference between the prediction score ranking and the MOS value ranking of the point cloud, Indicates the number of point clouds.

[0096] 2. Pearson Linear Correlation Coefficient (PLCC): Measures the linear correlation between the predicted score and the subjective score. The calculation formula is: in, and Represent the average of all MOS values ​​and predicted scores, respectively. To eliminate the influence of nonlinear factors on the predicted scores, a four-parameter logistic regression function is used to map the predicted scores to the same scale as the MOS values.

[0097] The value range of SROCC and PLCC is [-1,1]. The closer their absolute values ​​are to 1, the better the performance of the method.

[0098] Experimental details and parameter settings All experiments are completed in the same software and hardware environment. The experimental environment configuration is as follows: Operating system: Windows 10; CPU: Intel Core i7-13700k; Memory: 128GB GPU:Nvidia GeForce RTX 4090; Framework version: PyTorch 2.1.2.

[0099] The training network parameters are set as follows: The global random seed is set to 42; Batch size is set to 4; The optimizer is AdamW, and the initial learning rate is 5×10 -5 ; The learning rate reduction strategy uses an equal-interval step size reduction, and the learning rate decreases to the original 0.9 every 5 epochs; The network is trained for a total of 100 epochs.

[0100] The model parameters are set as follows: The number of point cloud clusters M = 32, and the size of each cluster R = 8192; The width and height of the projected image are H=W=224; The code table size of the unified feature representation space is L=128, and the codeword length is d 3 =1024; The loss ratio parameters of each item in the total loss are: α=1, β=0.5, γ=0.1, δ=0.01.

[0101] Experimental results and comparison The performance of the proposed method is compared with the existing state-of-the-art methods in different databases, including point-based methods (P), projection-based methods (I) and multimodal methods (M). The comparison results are shown in Table 1.

[0102] Table 1 Performance comparison of different point cloud quality evaluation methods in SJTU-PCQA, CPCD2.0 and WPC databases. The best results are shown in bold.

[0103] The experimental results show that the proposed method achieves the best performance on the WPC and CPCD2.0 databases, and also exhibits highly competitive performance on the SJTU-PCQA database, which demonstrates the advancement and universality of the proposed method in multimodal point cloud quality assessment tasks.

[0104] The specific analysis is as follows: In the WPC and CPCD2.0 databases, the NR-PCQA method that relies only on point cloud geometry information performs poorly, while the projection-based NR-PCQA method can better reflect the visual perception of dense point clouds, so its performance is superior. In the SJTU-PCQA database, the point-based method performs better than the projection-based method because the distortion types of this database are mainly related to geometric information, and the projection method is less sensitive to these distortions.

[0105] The multimodal approach combines the advantages of point and projection graphs and performs significantly better than single-modal approaches.

[0106] Module validity verification 1. Verification of the effectiveness of the adaptive projection strategy By comparing the computational complexity and performance of conventional multi-faceted projection and adaptive projection, the experimental results are shown in Table 2 and Figure 5shown.

[0107] Table 2. Inference computation and performance of different projection strategies in cluster projection on the WPC database. The best results are shown in bold.

[0108] The results show that: According to the division of training set and test set mentioned above, the posture changes of the test set projection during the pre-training PCN process are recorded, such as Figure 5 As shown. It can be seen that as the network is gradually optimized, for angles with poor projection vision, the network gradually adjusts its posture toward the projection direction predefined by the camera. For angles that are initially better, the network only makes slight adjustments. Further, only the 2D projection image modality information is used to train the single-modal model, and the performance of the quality evaluation model is tested, including single-sided projection as a baseline method, 3-sided projection as a multi-view projection representative, and the proposed adaptive projection data. It is believed that for point cloud clusters, it basically does not contain a three-dimensional structure, so the 3-sided projection (i.e., the front, side, and top) can already reflect all point cloud information. According to the results in Table 2, it can be seen that compared with the baseline method of single-sided projection, 3-sided projection can effectively improve the evaluation performance, which is in line with our intuition. But it also brings 3 times the computational consumption. In addition, compared with the baseline method, the proposed projection method not only has better performance, but also has a computational consumption of only 1.27 times that of the baseline method. The performance of the proposed projection method is even comparable to that of the 3-sided projection. Therefore, the proposed adaptive projection method can reduce the computational consumption by about 60% compared with the 3-sided positive projection method and maintain close performance.

[0109] 2. Verification of the effectiveness of multimodal feature fusion and alignment strategy The experimental results are shown in Table 3 by verifying the performance of using only single-modal features, directly cascading multi-modal features, and using the alignment strategy proposed by the present invention. Table 3. The impact of using different modal information and fusion alignment methods on the final performance in WPC and SJTU-PCQA databases. The best results are shown in bold.

[0110] The results show that: First, when the cascaded features of 2D and 3D are used together, it has a stronger quality representation ability than the model that only uses a single branch. This is because the distortion features extracted by the multimodal model can complement each other, making up for the defect that a single modality cannot represent all distortions well. Therefore, its performance surpasses the method of using only a single 3D point modality or 2D projection modality, which further proves the advanced nature of multimodal information. Secondly, the designed alignment strategy not only considers the gap between different modalities, but also considers the characteristics of human subjective evaluation. Therefore, compared with directly using cascaded features, the performance of executing the proposed feature alignment strategy before feature cascading and then cascading is better.

[0111] The multimodal point cloud quality assessment device based on adaptive block projection described below and the multimodal point cloud quality assessment method based on adaptive block projection described above can refer to each other.

[0112] Please refer to the attached Figure 6 The present invention also provides a multimodal point cloud quality assessment device based on adaptive block projection, comprising: A preprocessing module 10 is used to divide the point cloud data into blocks and generate local point cloud clusters; An adaptive cluster projection module 20, used to generate a 2D cluster projection map corresponding to each point cloud cluster and optimize the projection direction; A multimodal feature extraction module 30 extracts texture features of the 2D modal projection image and geometric features of the 3D point cloud cluster respectively; A feature alignment module 40 for aligning modality-independent quality features to a shared unified feature representation space; The regression network module 50 is used to fuse the 2D modality and 3D modality features and output a point cloud quality score.

[0113] The device of this embodiment can be used to execute the above method embodiment, and its principles and technical effects are similar, which will not be repeated here.

[0114] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal point cloud quality assessment method based on adaptive block projection, characterized in that: The following steps are involved: Preprocess the point cloud data and divide it into multiple local clusters, each of which contains the central point and its neighboring points; Generate a 2D cluster projection map corresponding to each local cluster's spatial position; Extract the features of the 2D modal projection image and the 3D local cluster data to obtain the 2D modal features and 3D modal features respectively; Decouple 2D modal features and 3D modal features to generate modality-independent quality features and modality-dependent features; Mapping modality-independent quality features into a shared unified feature representation space; Fusion of 2D and 3D modality features to generate joint features; Based on the joint features, the quality score of the point cloud is predicted through a regression network.

2. The multimodal point cloud quality assessment method based on adaptive block projection according to claim 1, characterized in that: The step of preprocessing the point cloud data comprises: Point cloud data Block, point cloud data Contains point The three-dimensional coordinates and of Dimension additional features, Additional features include color or normal vector; Using the farthest point sampling algorithm from the point cloud Select Points are used as the center points of the local cluster to generate a set of center points ; For each center point , selected by the nearest neighbor search algorithm nearest neighbor points, generate a set of neighborhood points , each local cluster Defined as the center point and the set of neighborhood points, expressed as: After completing the division of the point cloud data, a point cloud cluster set is formed .

3. The multimodal point cloud quality assessment method based on adaptive block projection according to claim 1, characterized in that: The step of generating a 2D cluster projection map corresponding to the spatial position of each local cluster comprises: For each local cluster Calculate the surface normal vector of the point in , the calculation of the normal vector is based on the eigenvalue decomposition of the covariance matrix of the local surface. The specific calculation method is: in, represents the covariance matrix, is a set of coordinates and The covariance of , and For local cluster points , and Coordinate set, the eigenvector corresponding to the minimum eigenvalue of the covariance matrix is ​​the normal vector ; The adaptive block projection module is used to optimize the projection direction of the local cluster to make the effective information content of the projection graph Maximize the effective information amount and the calculation formula is: in, is a local cluster, are the parameters of the projection camera, is the normal vector of the projection plane, For point The normal vector of is the cosine similarity function; For each local cluster Apply the posture correction operation to adjust its relative posture with the projection plane, through the projection camera The adjusted local cluster Project to a 2D plane to generate a 2D cluster projection map , its formula is expressed as: in, is the posture transformation matrix of the point cloud cluster, For point The three-dimensional coordinates of For point Additional features of Represents a projection operation; Perform the above operations on all local clusters to generate the corresponding 2D cluster projection graph set .

4. The multimodal point cloud quality assessment method based on adaptive block projection according to claim 3, characterized in that: The adaptive block projection module comprises: A normal vector estimation unit, used to estimate the surface normal vector of each point in the point cloud cluster; The pose correction network is used to generate the relative optimal pose of the point cloud cluster and the projection plane.

5. The multimodal point cloud quality assessment method based on adaptive block projection according to claim 1, characterized in that: The step of extracting the features of the 2D modal projection image and the 3D local cluster data to obtain the 2D modal features and the 3D modal features respectively comprises: 2D projection map of each local cluster , encode it through the 2D feature extraction network to obtain 2D modal features , and its calculation formula is: in, represents a 2D feature extraction network, represents the extracted 2D modal features, is the dimension of the 2D modal feature; For each local cluster , encode it through the 3D feature extraction network to obtain 3D modal features , and its calculation formula is: in, represents the 3D feature extraction network, represents the extracted 3D modal features, is the dimension of the 3D modality feature; 2D modal projection of all local clusters and 3D local clusters After extracting the features respectively, a complete 2D modal feature set and a 3D modal feature set are formed.

6. The multimodal point cloud quality assessment method based on adaptive block projection according to claim 1, characterized in that: The step of decoupling the 2D modal features and the 3D modal features to generate modality-independent quality features and modality-dependent features includes: The 2D modal features and 3D modal features Input the quality feature encoder and modal feature encoder separately and decouple them into modality-independent quality features and modality-related features , the specific calculation formula is: in, and They are quality feature encoder and modal feature encoder respectively; Using distance correlation metric to constrain quality features and modality-related features , the calculation formula of distance correlation measure is: in, is the distance covariance, and its calculation formula is: in, and is the center distance between samples, is the sample size; For the decoupling results of all modal features, add distance correlation constraint loss , defined as: Through the above decoupling operation, a modality-independent quality feature set and a modality-dependent feature set are generated for all 2D modal features and 3D modal features respectively.

7. The multimodal point cloud quality assessment method based on adaptive block projection according to claim 1, characterized in that: The step of mapping the modality-independent quality features to a shared unified feature representation space comprises: Constructing a shared unified feature representation space , which represents the space as a The discrete code table of feature vectors is expressed as: in, represents the feature vector, is the size of the code table, is the dimension of the codeword; For each mode independent quality characteristics , mapped to the feature space by nearest neighbor search The closest eigenvector in , the specific calculation formula is: in, represents the Euclidean distance, Represents the feature vector mapped to the feature space; Adding embedding loss , defined as: in, Indicates stopping gradient backpropagation, which is used to optimize the feature encoder and feature space respectively; The above mapping operation is performed on all modality-independent quality features to generate a shared representation set of all quality features.

8. The multimodal point cloud quality assessment method based on adaptive block projection according to claim 1, characterized in that: The step of fusing the features of the 2D modality and the 3D modality to generate a joint feature includes: Perform cluster-by-cluster cascading operations on the modality-independent quality features of all local clusters to generate a joint feature sequence , the specific formula is: in, represents the feature cascade operation, and Respectively The 2D modal quality features and 3D modal quality features of local clusters.

9. The multimodal point cloud quality assessment method based on adaptive block projection according to claim 1, characterized in that: The step of predicting the quality score of the point cloud through the regression network based on the joint features includes: The joint feature sequence of all local clusters is input into the regression network; The regression network is composed of a multi-layer perceptron, receives the joint features as input, processes the joint features through multi-layer linear transformation and non-linear activation function, and generates a prediction quality score for each local cluster; The predicted quality scores of the local clusters are weighted and aggregated to generate a final quality score of the point cloud, and compared with the subjective quality score corresponding to the point cloud; The regression network is trained to minimize the difference between the predicted quality score and the subjective quality score of the point cloud, and finally the prediction of the point cloud quality is completed.

10. A multimodal point cloud quality assessment device based on adaptive block projection, used to execute the method according to any one of claims 1 to 9, characterized in that: include: The preprocessing module is used to divide the point cloud data into blocks and generate local point cloud clusters; Adaptive cluster projection module, used to generate a 2D cluster projection map corresponding to each point cloud cluster and optimize the projection direction; The multimodal feature extraction module extracts the texture features of the 2D modal projection image and the geometric features of the 3D point cloud cluster respectively; A feature alignment module to align modality-independent quality features into a shared unified feature representation space; The regression network module is used to fuse 2D and 3D modality features and output point cloud quality scores.

Images