Multi-modal feature fusion reference-point-free cloud quality evaluation method

Abstract

The invention provides a multi-modal feature fusion reference-point-free cloud quality evaluation method, and belongs to the field of three-dimensional point cloud quality evaluation. The scheme comprises a data preprocessing step, a feature extraction step, a feature fusion step and a feature regression step. Firstly, point cloud is subjected to two kinds of preprocessing, one is that the point cloud is decomposed into sub-models, and the other is that a camera rotates around the point cloud to generate a dynamic video. And the obtained point cloud sub-model and the dynamic video are respectively sent to a feature extraction unit, and in the feature extraction stage, geometric distortion features are extracted from the point cloud sub-model, and spatial features and time features are extracted from the dynamic video. And then entering a feature fusion stage, fusing spatial features, time features and geometric distortion features by using a new cross-modal attention module to obtain final quality perception features, and finally, regressing the final quality features to a predicted quality score by using a dual full-connection layer.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a point cloud quality evaluation method based on multi-modality. Background Art

[0002] With the continuous development of 3D data acquisition technology, point cloud, as an immersive media, is widely used in virtual reality, augmented reality, autonomous driving and other fields. However, the point cloud acquisition, compression, processing, transmission and rendering processes can easily introduce distortion, resulting in a decrease in point cloud quality. Therefore, it is crucial to accurately evaluate the perceptual quality of point clouds.

[0003] Currently, there are two types of no-reference quality evaluation methods in point cloud quality no-reference evaluation: evaluation methods based on two-dimensional projection and evaluation methods based on 3D point clouds. The first type of two-dimensional projection-based methods projects point clouds into 2D images and feeds them into network models similar to convolutional neural networks, such as BRISQUE, NIQE, and PQA-net. The projection contains rich texture and semantic information. This projection-based point cloud quality assessment (PCQA) method has achieved competitive performance with the help of mature image quality assessment (IQA) methods, but they ignore the fact that 3D models are also perceived in a dynamic viewing mode, in which the viewpoint will constantly change according to the feedback of the rendering device. The two-dimensional projection method is highly dependent on the viewpoint and ignores the problem of viewpoint change.

[0004] In contrast, 3D point cloud-based methods directly infer the point cloud quality based on the original 3D data and use effective processing tools such as sparse convolution and graph convolution, such as ResSCNN and 3D-NSS. These methods use the fact that three-dimensional point clouds are more sensitive to geometric deformation and are invariant to viewpoint, but the point cloud data has a large amount of computation and the problem of incomplete data caused by occlusion has always existed.

[0005] The above methods usually use single-modal information for visual quality evaluation, and the mutual synergy between different modal data is lacking. Therefore, how to evaluate point cloud quality based on multi-modal information is an urgent problem to be solved. Summary of the invention

[0006] In view of the above technical problems, the present invention proposes a new reference-free multimodal point cloud quality assessment method. The point cloud is processed into two forms of data: one is to process the point cloud into a point cloud sub-model, and the other is to regard the point cloud as a mobile camera video. After feature extraction, multimodal feature fusion is performed to form a new quality perception feature for point cloud quality assessment. This method makes full use of the advantages of three-dimensional point clouds being more sensitive to geometric deformation and the richer image texture and semantic information contained in two-dimensional videos, while solving the single viewpoint problem of two-dimensional projection, thereby improving the accuracy of point cloud quality assessment.

[0007] The present invention proposes a reference-free point cloud quality evaluation method based on multimodal feature fusion. Considering that the perception of point clouds by the human visual system is based on different viewpoints, the point cloud is firstly decomposed into sub-models and processed into dynamic videos. Then, the obtained point cloud sub-models and dynamic videos are respectively sent to feature extraction units. In the feature extraction stage, geometric distortion features are extracted from the point cloud sub-models, and spatial features and temporal features are extracted from the dynamic videos. The geometric distortion features are high-level semantic features extracted from the three-dimensional point cloud itself, the spatial features are high-level semantic features extracted from the two-dimensional images, and the temporal features are high-level features extracted from the video time series. Then, in the feature fusion stage, a new cross-modal attention module is used to fuse features, and the geometric distortion features, spatial features, temporal features and corresponding core tokens are respectively sent to the cross-attention module, and the processed features are then spliced ​​to obtain the final quality perception features. Finally, two fully connected layers are used to regress the quality features to the predicted quality scores, thereby obtaining an objective evaluation of the point cloud quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a schematic diagram of the point cloud quality evaluation algorithm flow chart of the present invention. DETAILED DESCRIPTION

[0009] The following is combined with Figure 1 The point cloud quality evaluation algorithm of the present invention is further explained.

[0010] like Figure 1 As shown, the image quality evaluation algorithm of the present invention includes four steps: step 1 data preprocessing; step 2 feature extraction, step 3 multimodal feature fusion, and step 4 feature regression.

[0011] Step 1: Data preprocessing.

[0012] Point cloud collection for quality assessment where α l ∈R 1×3 represents geometric coordinates, β l ∈R 1×3Represents additional RGB color information, N represents the number of points. Unlike point clouds used for classification and segmentation, high-quality point clouds usually contain a large number of dense points, which makes it difficult to extract features directly from the source points unless downsampling methods are used. However, common sampling methods aim to retain semantic information, but inevitably destroy the geometric patterns that are critical to quality assessment. In order to avoid the geometric errors caused by downsampling and maintain the smoothness and roughness of local patterns, several local sub-models are collected from the point cloud to represent the geometric structure features. Specifically, given a normalized point cloud P, the farthest point sampling method is used to obtain a point cloud set The anchor point set is represented as For each anchor point, the K nearest neighbor algorithm is used to find the P around the anchor point. s adjacent points, these points form a sub-model,

[0013]

[0014] Where P represents the point cloud set, H t is a collection of sub-models, KNN() represents the K nearest neighbor algorithm, η m Indicates the mth farthest sampling point.

[0015] The dynamic video is generated by the camera rotating and capturing 120 frames of pictures. After capturing the current frame, the camera rotates 120 times around the center of the circular path. ° to capture the next frame. Then capture 360 / 12=30 frames of segments to cover each circular path, resulting in a point cloud video consisting of 4 segments and 4×30=120 frames. In addition, the camera returns to the same starting position after sampling frames for each path, so the captured video is continuous between segments.

[0016] Determine the geometric mean center of the point cloud P as:

[0017]

[0018] Where X λ represents the (X, Y, Z) coordinates of the mean center of the point cloud, N represents the number of point clouds, and λ n Represents the (X, Y, Z) coordinates of each point in the point cloud.

[0019] Determining the four circularly symmetric rotation paths can be expressed as:

[0020]

[0021] Where (X rot ,Y rot ,Z rot) represents the camera position coordinates, RotaA, RotaB, RotaC, RotaD represent four rotation paths, and R represents the rotation radius.

[0022] According to the above video capture process, we can get a video sequence V, which contains four video clips Each video clip consists of 30 frames, and then select the key frames in each single video clip. (j is the frame index in the clip, 0≤j<30) for spatial feature extraction, while for a single video clip, the entire video clip is used for temporal feature extraction.

[0023] Step 2: Feature extraction

[0024] Feature extraction includes 2.1 geometric distortion feature extraction based on point cloud sub-model, 2.2 temporal feature extraction and spatial feature extraction based on dynamic video

[0025] 2.1 Geometric distortion feature extraction, the local point cloud sub-model can maintain the local features, using the dynamic graph convolutional neural network (Dynamic Graph CNN, DGCNN) as the point cloud encoder, the obtained sub-model H t The local geometric distortion features, the extracted geometric distortion features e p It is expressed as:

[0026]

[0027] where e p Represents the sub-model H t The geometric distortion characteristics, D p represents the point cloud encoder DGCNN, GAP() represents the global average pooling function, Represents the geometric distortion features after average pooling.

[0028] 2.2.1 Spatial feature extraction, selecting key frames from each path in the dynamic video The keyframe index is set to 7, which means that the 7th frame of each clip is used for spatial feature extraction. The blank background of the keyframe is first removed, and then a 224×224×3 image patch is randomly cropped from the keyframe for training, and an image patch of the same resolution is cropped in the center for testing. The spatial features are calculated by taking the cropped keyframe as the input of VGG11 without the fully connected layer. The spatial features extracted from each keyframe are s It is expressed as:

[0029]

[0030] where e s Represents a key frame The spatial characteristics of D s represents the image encoder VGG11 model, GAP() represents the global average pooling function, Represents the spatial features after average pooling.

[0031] 2.2.2 Temporal feature extraction from video clips using the SlowFast R50 network The extracted fast_feature is used as the temporal feature. Video clip Extracting time features from t It is expressed as:

[0032] e t ={D s (C i )} (11)

[0033]

[0034] where e t Indicates video segment C i The time characteristics of D s represents the SlowFast R50 network model, GAP() represents the global average pooling function, Represents the temporal features after average pooling.

[0035] Step 3: Feature Fusion

[0036] A single modality may be incomplete and cannot cover enough information for quality assessment. A cross-modal attention module Transformer block is used to study the interaction between visual quality features collected from different modalities. To further explore the distinguishing components between modes, a multi-head attention module is used. Given the different intra-modal features F x 、F y As input set, we define the multi-head attention operation l() as:

[0037]

[0038]

[0039] Where l() represents the multi-head attention operation, ε() represents the attention function, represents the μth head, softmax() represents the normalization function, are the learnable weights for Q, K, and V in the μth attention head, respectively;

[0040] In order to perform separate processing and interleaved fusion of different modalities in the fusion module, we introduce core tokens to connect point cloud features and video features. The core tokens are randomly initialized and gradually updated during the training process. The specific steps of the fusion module are as follows:

[0041] Input different modal features R x , R y , use linear projection to resize them to the same dimension:

[0042]

[0043] in is the adjusted feature, W x , W y is a learnable linear mapping;

[0044] The final quality embedding can be obtained by concatenating the intra-modal features and the guided multi-modal features obtained through the cross-modal attention module;

[0045]

[0046] in represents the concatenation operation, l() represents the cross-modal multi-head attention operation, Represents the generated core token, represents the updated intra-modal features, Represents the updated core token, R o represents the fused quality-aware features;

[0047] Process the geometric distortion features first and spatial characteristics The fusion update between the two, the core token generated here is

[0048] First, the geometric distortion features extracted from the point cloud are Core Token and The self-connection is fed into the cross-attention module to obtain the updated features

[0049]

[0050] Then the spatial features Core Token and The self-connection is fed into the cross-attention module to obtain the updated features

[0051]

[0052] Core Token The update is done by adding itself The connection is fed into the cross attention module to obtain the updated features

[0053]

[0054] Core Token The update is done by adding itself The connection is fed into the cross attention module to obtain the updated features

[0055]

[0056] Final point cloud geometric distortion features and the spatial characteristics of the keyframes The fusion updated features It is expressed as:

[0057]

[0058] Similarly, spatial features and time characteristics The fusion between the update generation token is The obtained fusion updated features The feature is expressed as:

[0059]

[0060] Time characteristics and geometric distortion characteristics The fusion between the update generation token is The obtained fusion updated features The feature is expressed as:

[0061]

[0062] Finally, concatenate the updated features Get the final quality perception feature e q It is expressed as:

[0063]

[0064] Step 4: Feature regression

[0065] Using the generated quality-aware features, we use two levels of fully connected (FC) layers, consisting of 1024 and 1 neurons respectively, to perform feature regression and obtain the objective score for point cloud quality evaluation, which can be derived as:

[0066] Q i=FC(e q ) (30)

[0067] Where FC() represents the number of fully connected layers, Q i Indicates the point cloud quality score.

[0068] The loss function is used to measure the difference or error between the model's prediction results and the true label, which directly affects the model's training effect and final performance. When training the model, the goal of the loss function is to minimize the gap between the predicted value and the true label so that the model can better fit the data. The mean square error (MSE) is directly used as the loss function:

[0069]

[0070] Where n represents the number of training, L μ Denotes the subjective quality score, Q μ Represents the prediction quality score.

[0071] The quality scores predicted by different PCQA methods have different scale differences from the subjective quality scores. Before calculating the standard values, a five-parameter logistic function is used to map the predicted scores so that they can be better aligned with the quality labels and it is easier to compare the performance of different methods:

[0072]

[0073] where {β i |i=1,2,...,5} are fitting parameters, y is the prediction score, Score the mapping.

[0074] The algorithm performance was verified on SJTU-PCQA, and four common consistency evaluation criteria, Person correlation coefficient (Pearson line correlation coefficient, PLCC), root mean square error (Root Mean Squared Error, RMSE), Spearman rank-order correlation coefficient (Spearman rank-order correlation coefficient, SROCC), and Kendall rank-order correlation coefficient (KROCC) were used to judge the correlation between the predicted score and the subjective quality score. The value range of PLCC, SROCC, and KROCC is [-1, 1]. The closer the absolute value is to 1, the stronger the correlation and the higher the consistency. The range of RMSE is [0, ∞]. The closer it is to 0, the better the prediction performance. An excellent model should make the SRCC, KRCC, and PLCC values ​​as close to 1 as possible, and make the RMSE value as close to 0 as possible.

[0075] Table 1 Test results on SJTU-PCQA dataset

[0076]

[0077] According to the data presented in Table 1, it is concluded that in terms of the monotonicity evaluation indicators SOCC and KRCC, the performance of this method on the SJTU-PCQA database is closest to 1; and in terms of the prediction accuracy indicators PLCC and RESM, the PLCC value of this method is closest to 1, and the RESM value is closest to 0. In summary, the method proposed by us has achieved remarkable results in point cloud quality evaluation and can maintain good consistency with the subjective evaluation of human eyes.

Claims

1. A reference-free point cloud quality assessment method based on multimodal feature fusion, characterized by: The three-dimensional point cloud quality evaluation method comprises: Step 1 Data preprocessing: Two preprocessing steps are performed on each point cloud: one is to segment it into multiple point cloud sub-models, and the other is to convert it into a dynamic video. The segmentation is based on the K nearest neighbor algorithm and the farthest point sampling algorithm to perform sub-model segmentation to obtain a local sub-model of the point cloud; the dynamic video generation is to process the point cloud into a dynamic video by rotating the viewpoint around a circular path; Step 2 feature extraction: including geometric distortion feature extraction of the local sub-model and temporal feature extraction and spatial feature extraction of the dynamic video part, wherein the geometric distortion feature is extracted by a point cloud encoder, the spatial feature is extracted by an image encoder, and the temporal feature uses a 3D convolutional network model as a temporal feature extraction unit; Step 3: Feature fusion: Use the cross-modal attention module to process the features, and then concatenate the processed features with the original features; Step 4: Feature regression: A two-level fully connected layer consisting of 1024 and 1 neurons respectively is used to perform feature regression to obtain an objective score of the 3D point cloud quality.

2. The point cloud quality evaluation method according to claim 1, characterized in that: Given a point cloud set where α l ∈R 1×3 represents geometric coordinates, β l ∈R 1×3 Indicates additional RGB color information, N indicates the number of points; The segmentation of the sub-model uses the farthest point sampling method to obtain a point cloud set The anchor point set is represented as For each anchor point, the K nearest neighbor algorithm is used to find the P around the anchor point. S adjacent points, these points form one of the sub-models, Where P represents the point cloud set, H t is a collection of sub-models, KNN() represents the K nearest neighbor algorithm, η m represents the mth farthest sampling point; The generation of the dynamic video first establishes the geometric mean center of the point cloud, and then rotates around the geometric center of the point cloud through 4 symmetrical circular rotation paths. In order to maintain the consistency of the video, the camera rotation step between consecutive frames is set to 12. The geometric mean center of the point cloud P is defined as: Where X λ represents the (X, Y, Z) coordinates of the mean center of the point cloud, N represents the number of point clouds, and λ n Represents the (X, Y, Z) coordinates of each point in the point cloud; The four circularly symmetric rotation paths can be expressed as: Where (X rot ,Y rot ,Z rot ) represents the camera position coordinates, RotaA, RotaB, RotaC, RotaD represent four rotation paths, and R represents the rotation radius.

3. The point cloud quality evaluation method according to claim 1, characterized in that: In step 2, the point cloud encoder is used to obtain the sub-model H t The local geometric distortion feature, the extracted geometric distortion feature e p It is expressed as: where e p Represents the sub-model H t The geometric distortion characteristics, D p represents the point cloud encoder, GAP() represents the global average pooling function, Represents the geometric distortion features after average pooling; Select a keyframe from each path (j is the frame index in the clip, 0≤j<30) for spatial feature extraction. Given a key frame, after cropping the blank background border of the key frame, the deep learning model without the fully connected layer is used to extract the spatial features. The spatial features extracted from the key frame are s It is expressed as: where e s Represents a key frame The spatial characteristics of D s represents the image encoder, GAP() represents the global average pooling function, Represents the spatial features after average pooling; A pre-trained 3D convolutional network model is used as the temporal feature extractor to extract temporal features from video clips. Extracting time features from t It is expressed as: e t ={D t (C i )} (11) where e t Indicates video segment C i The time characteristics of D t represents a 3D convolutional network model, GAP() represents the global average pooling function, Represents the temporal features after average pooling.

4. The point cloud quality evaluation method according to claim 1, characterized in that: In step 3, given different modal features F x 、F y As input set, we define the multi-head attention operation l() as: Where l() represents the cross-modal multi-head attention operation, ε() represents the attention function, represents the μth head, softmax() represents the normalization function, are the learnable weights for Q, K, and V in the μth attention head, respectively; Use core tokens to connect point cloud features and video features. The core tokens are randomly initialized and updated during training. Input different modality features R x , R y , use linear projection to resize them to the same dimension: in is the adjusted feature, W x , W y is a learnable linear mapping; The final quality embedding is obtained by concatenating the intra-modal features and the guided multi-modal features obtained through the cross-modal attention module; in represents the concatenation operation, l() represents the cross-modal multi-head attention operation, Represents the generated core token, represents the updated intra-modal features, Represents the updated core token, R o represents the fused quality-aware features; Process the geometric distortion features first and spatial characteristics The fusion update between the two, the core token generated here is First, the geometric distortion features extracted from the point cloud are Core Token and The self-connection is fed into the cross-attention module to obtain the updated features Then the spatial features Generated core token and The self-connection is fed into the cross-attention module to obtain the updated features Core Token The update is done by adding itself The connection is fed into the cross attention module to obtain the updated features Core Token The update is done by adding itself The connection is fed into the cross attention module to obtain the updated features Final point cloud geometric distortion features and the spatial characteristics of the keyframes The fusion updated features It is expressed as: Spatial characteristics and time characteristics The fusion between the update generation token is The obtained fusion updated features The feature is expressed as: Time characteristics and geometric distortion characteristics The fusion between the update generation token is The obtained fusion updated features The feature is expressed as: Finally, concatenate the updated features Get the final quality perception feature e q It is expressed as:

5. The point cloud quality evaluation method according to claim 1, characterized in that: In step 4, two fully connected (FC) layers are used, which are composed of 1024 and 1 neurons respectively, to perform feature regression, which can be derived as follows: Q i =FC(e q ) (30) Where FC() represents the number of fully connected layers, Q i Represents the point cloud quality prediction score.

6. The point cloud quality evaluation method according to claim 1, characterized in that: In order to directly study the effectiveness of this method, the mean square error (MSE) is directly used as the loss function: Where n represents the number of training, L μ Denotes the subjective quality score, Q μ Represents the prediction quality score.

Images