A 3D point cloud quality prediction method based on graph convolutional neural network

Abstract

The present invention provides a three-dimensional point cloud quality prediction method based on a graph convolutional neural network, comprising: step S1, for at least one three-dimensional object, obtaining a data set of the three-dimensional object, and obtaining a training set and a test set by performing dual-path multi-view projection and preprocessing on the distorted point cloud; step S2, using a deep learning framework to build a reference-free point cloud quality evaluation network, and obtaining a predicted quality score according to the projection image group processing of the two paths; step S3, using the training set to train the reference-free point cloud quality evaluation network; step S4, according to step S3, the reference-free point cloud quality evaluation network is trained for 50 rounds to obtain a reference-free point cloud quality evaluation network model; step S5, analyzing the test set through the reference-free point cloud quality evaluation network model to obtain an evaluation result of the reference-free point cloud quality. The beneficial effect is that the present invention can effectively capture global structural information, reduce redundant calculations, and improve the accuracy of the predicted quality score.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud quality evaluation, and in particular to a three-dimensional point cloud quality prediction method based on a graph convolutional neural network. Background Art

[0002] In recent years, the development of 3D visual information acquisition technology has made point clouds easier to obtain and gradually become a popular visual data type. This 3D visual data is usually represented in the form of point clouds, voxels or grids. Since point clouds can fully and accurately describe three-dimensional objects, they are considered to be an effective three-dimensional data representation method. Point clouds are mainly used to describe a complete three-dimensional scene or object, including geometric attributes, color attributes and other attributes (normal vectors, opacity, reflectivity, time, etc.). At present, point clouds have been widely studied and applied in application scenarios such as three-dimensional reconstruction, classification and segmentation, facial expression representation, autonomous driving, virtual reality, etc. Although point clouds can realistically record three-dimensional objects through huge point sets, they also consume a lot of memory and are difficult to achieve data transmission under limited network bandwidth. This new and effective data representation method poses challenges to current hardware storage and network transmission. Therefore, in order to achieve efficient storage and transmission, point clouds need to be compressed.

[0003] However, point cloud compression may introduce artifacts, resulting in a decrease in the visual quality of point clouds. Effective point cloud quality prediction methods can not only help people predict the degree of distortion of point clouds and the performance of compression algorithms, but also help optimize the visual quality of distorted point clouds. Therefore, how to accurately evaluate the perceptual quality of point cloud data becomes a key issue. Among the existing point cloud quality evaluation methods, point-based methods and projection-based methods have achieved certain results. However, most of the current projection-based methods mainly rely on 6 projection planes for quality evaluation, without considering the multi-view perception of the human visual system on point cloud quality. This means that the model fails to fully utilize the correlation between different projection images for modeling, resulting in its insufficient ability to perceive the overall quality of point clouds, and cannot effectively capture global structural information. There are many redundant calculations and the prediction accuracy is very limited. Summary of the invention

[0004] The technical problem to be solved by the present invention is to effectively capture global structural information, reduce redundant calculations, and improve the accuracy of predicted quality scores. In order to overcome the defects of the above-mentioned prior art (or related technology), the present invention provides a three-dimensional point cloud quality prediction method based on a graph convolutional neural network.

[0005] The present invention provides a three-dimensional point cloud quality prediction method based on a graph convolutional neural network, comprising:

[0006] Step S1, for at least one three-dimensional object, obtaining a data set of the three-dimensional object, the data set including a plurality of distorted point clouds and a subjective quality score corresponding to each of the distorted point clouds, then performing dual-path multi-view projection on each of the distorted point clouds and preprocessing the projection image to obtain a corresponding preprocessed projection image; and dividing each of the preprocessed projection images and the subjective quality score corresponding to each of the preprocessed projection images into a training set and a test set;

[0007] Step S2, using a deep learning framework to build a reference-free point cloud quality assessment network, the reference-free point cloud quality assessment network includes a backbone module, a multi-level attention perception module, a multi-level conversion module and a quality prediction module, through which the backbone module receives the projection image groups of the two paths and outputs a feature map and feature map , through the multi-level attention perception module to the feature map and the characteristic graph Convolution operation, feature map multiplication operation and global average pooling operation are performed in sequence to obtain the feature map and feature map , through the multi-level conversion module to the feature map And the feature map Processing to obtain graph structure and graph structure , the graph structure is predicted by the quality prediction module And the graph structure Processing is performed to obtain a predicted quality score S;

[0008] Step S3, using the training set to train the reference-free point cloud quality assessment network to obtain the prediction quality score S corresponding to each distorted point cloud in the training set;

[0009] Step S4, training the no-reference point cloud quality assessment network for multiple rounds according to step S3 to obtain a no-reference point cloud quality assessment network model;

[0010] Step S5, testing each of the distorted point clouds in the test set through the no-reference point cloud quality evaluation network model, and outputting the predicted quality score S of each of the distorted point clouds as the evaluation result of the no-reference point cloud quality.

[0011] Compared with the prior art, the three-dimensional point cloud quality prediction method based on graph convolutional neural network in the present invention has the following advantages:

[0012] In the present invention, in step S1, the distorted point cloud is subjected to dual-path multi-view projection to obtain two groups of projection images on different projection paths, and on this basis, a highly perceptually consistent preprocessed projection image is constructed through preprocessing to characterize the mutual dependence between different preprocessed projection images, thereby effectively capturing global structural information; in step S2, a graph convolutional network is applied to the graph structure to describe the global dependency between different preprocessed projection images, and to aggregate the feature information of the multi-view projection images, effectively reducing redundant calculations and improving the global understanding ability of the three-dimensional point cloud quality prediction method; in step S3, the no-reference point cloud quality evaluation network is trained using the training set, and in step S4, the no-reference point cloud quality evaluation network is trained for multiple rounds to improve the robustness of the no-reference point cloud quality evaluation network model and effectively improve the accuracy of the prediction quality score.

[0013] In a possible implementation, in step S1, dual-path multi-view projection is performed on each distorted point cloud in the data set, and the number of projection images obtained on each projection path is N. Then, each projection image is pre-processed by resizing, cropping, and normalization to obtain an image size of The pre-processed projection image.

[0014] In a possible implementation manner, in step S1, the process of preprocessing each of the projection images includes:

[0015] Each of the projected images is resized and cropped, and the size of each of the projected images is scaled to , and then normalize each of the scaled projection images, normalize the pixel values ​​of all pixels in the R channel of the scaled projection image to a mean of 0.485 and a variance of 0.229, normalize the pixel values ​​of all pixels in the G channel to a mean of 0.456 and a variance of 0.224, and normalize the pixel values ​​of all pixels in the B channel to a mean of 0.406 and a variance of 0.225.

[0016] In a possible implementation, in step S2, the backbone module is a ResNet101 backbone network comprising a five-layer structure, the five-layer structure is connected in sequence, the input end of the first layer serves as the input end of the backbone module and simultaneously receives the projection image groups of the two paths, each of the projection image groups comprises N images of size The preprocessed projection image, the output ends of the second and fifth layers are used as the multi-level output ends of the backbone module, and the output end of the first layer outputs the feature map , the input end of the second layer receives the feature map , the output end of the second layer outputs the feature map , the input end of the third layer receives the feature map , the output feature map of the third layer , the input end of the fourth layer receives the feature map , the output feature map of the fourth layer , the input end of the fifth layer receives the feature map , the output end of the fifth layer outputs the feature map ; wherein the feature map The size is , the feature map The size is , the feature map The size is , the feature map The size is , the feature map The size is .

[0017] In a possible implementation, in step S2, the multi-level attention perception module obtains the feature map through two branches: And the feature map , wherein the execution steps of the first branch include:

[0018] The characteristic diagram As input, a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 256, and an output channel number of 1 is used to compress the channel size, and a Sigmoid activation function is applied to map the feature value range to [0,1] to obtain the attention feature map , the attention feature map The size is ; and with the characteristic graph As input, a global average pooling layer is used to perform global average pooling in the spatial dimension, and then it passes through a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 256, and an output channel number of 16, and a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 16, and an output channel number of 256, and the Sigmoid activation function is applied to map the feature value range to [0,1] to obtain the attention feature map , the attention feature map The size is ; Then the attention feature map and the attention feature map Multiply to get the mixed attention feature map , the hybrid attention feature map The size is ; and the mixed attention feature map With the characteristic diagram Multiply and connect with the feature map through residual connection Add, and then perform a global average pooling operation to obtain the feature map , the feature map The size is ;

[0019] The execution steps of the second branch include:

[0020] The characteristic diagram As input, a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 2048, and an output channel number of 1 is used to compress the channel size, and a Sigmoid activation function is applied to map the feature value range to [0,1] to obtain the attention feature map , the attention feature map The size is ; and with the characteristic graph As input, a global average pooling layer is used to perform global average pooling in the spatial dimension, and then it passes through a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 2048, and an output channel number of 128, and a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 128, and an output channel number of 2048, and the Sigmoid activation function is applied to map the eigenvalue range to [0,1] to obtain the attention feature map , the attention feature map The size is ; and the attention feature map and the attention feature map Multiply to get the mixed attention feature map , the hybrid attention feature map The size is ; and the mixed attention feature map With the characteristic diagram Multiply and connect with the feature map through residual connection Add, and then perform a global average pooling operation to obtain the feature map , the feature map The size is .

[0021] In a possible implementation manner, in step S2, the multi-level conversion module includes a multi-level feature fusion block and a graph construction block. And the feature map Perform channel splicing to obtain the feature map , the feature map The size is Then, the feature map is projected according to the dual-path multi-view projection path. Segmentation is performed to obtain the horizontal projection feature map And vertical projection feature map , the horizontal projection feature map The size is , the vertical projection feature map The size is ; The horizontal projection feature map is constructed by the map building block And the vertical projection feature map Each projection feature in is taken as a node, and the adjacency matrix is ​​constructed based on the adjacency relationship between any two nodes. and , the adjacency matrix The size is NxN, the adjacency matrix The size is NxN, and then according to the feature map , the feature map , the adjacency matrix and the adjacency matrix Forming the corresponding graph structure And the graph structure .

[0022] In a possible implementation, in step S2, the quality prediction module includes two graph convolution networks with the same structure and not shared, and a hybrid prediction block, wherein both graph convolution networks include a four-layer structure and the four-layer structures are sequentially connected, and the graph structure is received by the first graph convolution network. And process to get the feature map , feature map , feature map and feature map , receiving the graph structure through the second graph convolutional network And process to get the feature map , feature map , feature map and feature map , and then the feature map is predicted by the hybrid prediction block , the feature map , the feature map , the feature map , the feature map And the feature map Use average pooling and a fully connected layer with an output channel number of 1 to obtain the corresponding first processed feature map, and And the feature map Average pooling and fully connected layer are performed respectively to obtain the corresponding second processed feature map, and then the first processed feature map and the second processed feature map are channel-joined to obtain a mixed feature map. , the mixed feature map The size of is 10x1, and finally the mixed feature map The predicted quality score S is obtained through a fully connected layer with 10 input channels and 1 output channel.

[0023] Compared with the existing technology, after adopting the above technical scheme, the feature representation ability of the reference-free point cloud quality evaluation network model is enhanced by dynamically fusing the image features of the horizontal projection image and the vertical projection image extracted by two graph convolutional networks that do not share weights, thereby effectively improving the accuracy of the predicted quality score.

[0024] In a possible implementation, in step S2, for the first graph convolutional network, the input end of the first layer receives the graph structure as the input end of the first graph convolutional network. , the output end of the first layer outputs the feature map , the input of the second layer receives the graph structure , the output end of the second layer outputs the feature map , the input of the third layer receives the graph structure , the output feature map of the third layer , the input of the fourth layer receives the graph structure , the output feature map of the fourth layer ; wherein the feature map The size is , the feature map The size is , the feature map The size is , the feature map The size is .

[0025] In a possible implementation, in step S2, for the second graph convolutional network, the input end of the first layer receives the graph structure as the input end of the second graph convolutional network. , the output end of the first layer outputs the feature map , the input of the second layer receives the graph structure , the output end of the second layer outputs the feature map , the input of the third layer receives the graph structure , the output feature map of the third layer , the input of the fourth layer receives the graph structure , the output feature map of the fourth layer ; wherein the feature map The size is , the feature map The size is , the feature map The size is , the feature map The size is .

[0026] In a possible implementation, in step S2, the two graph convolution networks each include four graph convolution blocks connected in sequence, the input end of the first graph convolution block serves as the input end of the graph convolution network, the input end of the second graph convolution block receives the feature map output by the output end of the first graph convolution block, the input end of the third graph convolution block receives the feature map output by the output end of the second graph convolution block, the input end of the fourth graph convolution block receives the feature map output by the output end of the third graph convolution block, and the feature maps output by the output ends of the first, second, third, and fourth graph convolution blocks serve as the output end of the graph convolution network; wherein the first three graph convolution blocks have the same structure and include graph convolution layers, batch normalization layers, and Softpl layers connected in sequence. us activation function layer, the fourth graph convolution block only includes the graph convolution layer and the Softplus activation function layer connected in sequence, the input end of the graph convolution layer serves as the input end of the graph convolution block where it is located, and the output end of the Softplus activation function layer serves as the output end of the convolution block where it is located; the number of input channels of the graph convolution layer in the first graph convolution block is 2304, and the number of output channels is 512, the number of input channels of the graph convolution layer in the second graph convolution block is 512, and the number of output channels is 128, the number of input channels of the graph convolution layer in the third graph convolution block is 128, and the number of output channels is 32, and the number of input channels of the graph convolution layer in the fourth graph convolution block is 32, and the number of output channels is 1. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the steps of the present invention;

[0028] Figure 2 It is a framework diagram of the reference-free point cloud quality evaluation network of the present invention;

[0029] Figure 3 It is a framework diagram of the multi-level attention perception module of the present invention;

[0030] Figure 4 It is a framework diagram of the multi-level conversion module of the present invention;

[0031] Figure 5 It is a framework diagram of the graph convolutional network of the present invention;

[0032] Figure 6 A framework diagram of a hybrid prediction block of the present invention;

[0033] Figure 7 The first part of the scatter plot between the objective prediction scores and subjective quality scores of the three-dimensional point cloud quality prediction method of the present invention and the existing point cloud quality prediction method on the SJTU dataset;

[0034] Figure 8 The second part of the scatter plot is between the objective prediction scores and subjective quality scores of the three-dimensional point cloud quality prediction method of the present invention and the existing point cloud quality prediction method on the SJTU dataset;

[0035] Fig. 9 The first part of the scatter plot is between the objective prediction scores and subjective quality scores of the three-dimensional point cloud quality prediction method of the present invention and the existing point cloud quality evaluation method on the WPC data set;

[0036] Fig.10 The second part of the scatter plot is between the objective prediction scores and subjective quality scores of the 3D point cloud quality prediction method of the present invention and the existing point cloud quality evaluation method on the WPC dataset. DETAILED DESCRIPTION

[0037] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present invention, and are not intended to limit the protection scope of the embodiments of the present invention. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0038] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] See also Figure 1 The embodiment of the present invention discloses a three-dimensional point cloud quality prediction method based on a graph convolutional neural network, comprising:

[0040] Step S1: Get a data set, which contains various types of point cloud data. The original samples of each point cloud data are compressed and noise processed to different degrees to generate distorted point clouds of various qualities, and each distorted point cloud has its subjective quality score; then perform dual-path multi-view projection on each distorted point cloud in the data set, and the number of projected images obtained on each projection path is N; and perform preprocessing of resizing, cropping and normalization on each projected image, so that the size of the preprocessed image is ; Then all pre-processed projection images and their corresponding subjective quality scores are divided into a training set and a test set, and the point cloud type that appears in the training set will not appear again in the test set; wherein, in this embodiment, N=10, H=W=224, C=3;

[0041] Step S2: Use the deep learning framework to build a deep neural network as a reference-free point cloud quality evaluation network, such as Figure 2 As shown in Figure 1, the network mainly includes a backbone module, a multi-level attention perception module, a multi-level conversion module, and a quality prediction module;

[0042] The backbone module is a ResNet101 backbone network with five layers. The five layers in the ResNet101 backbone network are connected in sequence. The input end of the first layer serves as the input end of the backbone module and simultaneously receives the projection image groups of the two paths. Each projection image group contains N images of size The input end of the second layer of the ResNet101 backbone network receives the feature map output by the output end of the first layer of the ResNet101 backbone network, the input end of the third layer of the ResNet101 backbone network receives the feature map output by the output end of the second layer of the ResNet101 backbone network, the input end of the fourth layer of the ResNet101 backbone network receives the feature map output by the output end of the third layer of the ResNet101 backbone network, the input end of the fifth layer of the ResNet101 backbone network receives the feature map output by the output end of the fourth layer of the ResNet101 backbone network, and the output ends of the second and fifth layers serve as the multi-level output ends of the backbone module; the feature map output by the output end of the first layer of the ResNet101 backbone network is recorded as The input of the second layer receives the feature map output by the output of the first layer of the ResNet101 backbone network , the feature map output from the output end of the second layer of the ResNet101 backbone network is recorded as The input of the third layer receives the feature map output by the output of the second layer of the ResNet101 backbone network , the feature map output from the output end of the third layer of the ResNet101 backbone network is recorded as The input of the fourth layer receives the feature map output by the output of the third layer of the ResNet101 backbone network , the feature map output from the output end of the fourth layer of the ResNet101 backbone network is recorded as The input of the fifth layer receives the feature map output by the output of the fourth layer of the ResNet101 backbone network , the feature map output from the output end of the fifth layer of the ResNet101 backbone network is recorded as ; Among them, the feature map output by the output end of the first layer of the ResNet101 backbone network The size is , The size is , The size is , The size is , The size is ; The ResNet101 backbone network is an existing structural framework, and its network structure has been made public, such as: the reference K. He, X.Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.770-778, 2016. ("Image Recognition Based on Deep Residual Learning") has recorded the ResNet101 backbone network;

[0043] Step S3: Use the training set to train the no-reference point cloud quality evaluation network. After each round of training, the no-reference point cloud quality evaluation network outputs the predicted quality score of each distorted point cloud in the training set, denoted as S, and then calculates the loss of the network, denoted as L. ; where n represents the batch size, GT represents the subjective quality score of each distorted point cloud in the training set, and the initial learning rate of the Adam optimizer is , every 10 rounds, the learning rate is decayed to 0.5 times of the original value, and the batch size is 32;

[0044] Step S4: training for 50 rounds in total according to the process of step S3 to obtain a reference-free point cloud quality evaluation network model;

[0045] Step S5: Use the no-reference point cloud quality evaluation network model to test each distorted point cloud in the test set, and output the predicted quality score corresponding to each distorted point cloud in the test set based on the no-reference point cloud quality evaluation network model as the evaluation result of the no-reference point cloud quality.

[0046] Continue to see Figure 1 In step 1, the process of preprocessing a distorted point cloud is as follows: first, the distorted point cloud is projected with dual paths and multiple perspectives using the existing technology, and the number of projection images obtained on each projection path is N; then each projection image is resized and cropped to scale the size of the projection image to ; Secondly, the scaled projection image is normalized, and the pixel values ​​of all pixels in the R channel of the scaled projection image are normalized to a mean of 0.485 and a variance of 0.229, the pixel values ​​of all pixels in the G channel are normalized to a mean of 0.456 and a variance of 0.224, and the pixel values ​​of all pixels in the B channel are normalized to a mean of 0.406 and a variance of 0.225. Since the input of a deep neural network usually has a fixed form, the input projection image needs to be preprocessed. Here, the projection image is normalized to improve the training efficiency of the subsequent model.

[0047] See also Figure 3 ,The multi-level attention perception module is divided into two branches for execution, specifically:

[0048] (1) The first branch uses the feature map output from the output of the second layer of the ResNet101 backbone network For input, A convolution layer with a kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 256, and an output channel number of 1 is used to achieve channel size compression, and then the Sigmoid activation function is applied to map the feature value range to [0,1]. The resulting attention feature map is recorded as , The size is ; Then, a global average pooling layer is used to perform global average pooling in the spatial dimension, and then a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 256, and an output channel number of 16 is passed through, and a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 16, and an output channel number of 256 is passed through. Finally, the Sigmoid activation function is applied to map the eigenvalue range to [0,1]. The obtained attention feature map is recorded as , The size is ; The two attention feature maps are multiplied, and the resulting mixed attention feature map is recorded as , The size is ; With the original input Multiply, and then connect with the residual Add, and then perform a global average pooling operation, the resulting feature map is recorded as , The size is ;

[0049] (2) The second branch uses the feature map output from the output of the fifth layer of the ResNet101 backbone network For input, A convolution layer with a kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 2048, and an output channel number of 1 is used to achieve channel size compression, and then the Sigmoid activation function is applied to map the feature value range to [0,1]. The resulting attention feature map is recorded as , The size is ; Then, a global average pooling layer is used to perform global average pooling in the spatial dimension, and then a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 2048, and an output channel number of 128 is passed through, and a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 128, and an output channel number of 2048 is passed through. Finally, the Sigmoid activation function is applied to map the eigenvalue range to [0,1]. The obtained attention feature map is recorded as , The size is ; The two attention feature maps are multiplied, and the resulting mixed attention feature map is recorded as , The size is , With the original input Multiply, and then connect with the residual Add, and then perform a global average pooling operation, the resulting feature map is recorded as , The size is .

[0050] See also Figure 4 The multi-level conversion module is mainly composed of a multi-level feature fusion block and a graph construction block; the multi-level feature fusion block transforms the multi-level feature graph. , After channel splicing, the obtained feature map is recorded as , The size is ; Then follow the projection path After segmentation, the horizontal projection feature map is recorded as , the vertical projection feature map is recorded as , The size is , The size is ; The graph building block represents each projection image feature as a node in the graph, and constructs an adjacency matrix based on the adjacency relationship between any two nodes. If the rotation step between the two projection images is less than the threshold It is considered that the two projected images are connected in space, and the adjacency matrices are recorded as and , The size is NxN, The size is NxN; finally, according to the feature map and , adjacency matrix and Forming the corresponding graph structure , ; Here, the feature map is converted into a correspondence map structure according to the nonlinear relationship and global dependency between multi-view projection images from different projection paths.

[0051] Continue to see Figure 2 The quality prediction module mainly consists of two identical, non-shared graph convolutional networks and a hybrid prediction block. The four layers of the graph convolutional network are connected in sequence; the input end of the first layer of the first graph convolutional network receives the graph structure as the input end of the graph convolutional network. , the feature map output at the output of the first layer is recorded as The input of the second layer receives the graph structure output by the output of the first layer of the graph convolutional network , the feature map output at the output of the second layer is recorded as The input of the third layer receives the graph structure output by the output of the second layer of the graph convolutional network , the feature map output at the output of the third layer is recorded as The input of the fourth layer receives the graph structure output by the output of the third layer of the graph convolutional network , the feature map output at the output of the fourth layer is recorded as ; Among them, the feature map output by the output end of the first layer of the first graph convolutional network is The size is , The size is , The size is , The size is ; The input end of the first layer of the second graph convolutional network receives the graph structure as the input end of the graph convolutional network , the rest of the operations are the same as above; among them, the feature map output by the output end of the first layer of the second graph convolutional network The size is , The size is , The size is , The size is ; Hybrid prediction block pair , , , , , Use average pooling and a fully connected layer with an output channel of 1, and then and The average pooling and full connection layers are performed respectively, and then the channel splicing of these feature maps is performed. The obtained mixed feature map is recorded as , The size is 10x1; The final prediction quality score S is obtained through a fully connected layer with 10 input channels and 1 output channel.

[0052] See also Figure 5 In step 2, each graph convolution network includes four graph convolution blocks, the input end of the first graph convolution block is used as the input end of the graph convolution network, the input end of the second graph convolution block receives the feature map output by the output end of the first graph convolution block, the input end of the third graph convolution block receives the feature map output by the output end of the second graph convolution block, and the input end of the fourth graph convolution block receives the feature map output by the output end of the third graph convolution block. The feature maps output by the output ends of the first, second, third, and fourth graph convolution blocks are multi-level feature maps output by the output end of the graph convolution network; among them, the first three The structures of the graph convolution layers are the same, all of which include a graph convolution layer, a batch normalization layer, and a Softplus activation function layer connected in sequence. The fourth graph convolution block only contains a graph convolution layer and a Softplus activation function layer connected in sequence. The input end of the graph convolution layer is used as the input end of the graph convolution block where it is located, and the output end of the Softplus activation function layer is used as the output end of the convolution block where it is located; the number of input channels of the graph convolution layer in the first graph convolution block is 2304, and the number of output channels is 512. The input end of the graph convolution layer in the first graph convolution block receives And the adjacency matrix M, the feature map output at the output of the Softplus activation function layer in the first graph convolution block is recorded as , The size is , the number of input channels of the graph convolution layer in the second graph convolution block is 512, the number of output channels is 128, and the input end of the graph convolution layer in the second graph convolution block receives And the adjacency matrix M, the feature map output at the output of the Softplus activation function layer in the second graph convolution block is recorded as , The size is , the number of input channels of the graph convolution layer in the third graph convolution block is 128, the number of output channels is 32, and the input end of the graph convolution layer in the third graph convolution block receives And the adjacency matrix M, the feature map output at the output of the Softplus activation function layer in the third graph convolution block is recorded as , The size is , the number of input channels of the graph convolution layer in the fourth graph convolution block is 32, the number of output channels is 1, and the input end of the graph convolution layer in the fourth graph convolution block receives And the adjacency matrix M, the feature map output at the output of the Softplus activation function layer in the fourth graph convolution block is recorded as , The size is .

[0053] See also Figure 6 In step 2, the hybrid prediction block is mainly divided into two parts. The first part is the horizontal projection feature map , , And vertical projection feature map , , The average pooling and the fully connected layer with an output channel number of 1 are used in turn, and the output feature maps are recorded as , , , , , , , , , After feature concatenation, the obtained mixed feature map is recorded as , The size is 10x1; The final prediction quality score S is obtained through a fully connected layer with 10 input channels and 1 output channel.

[0054] Continue to see Figure 1 ,In order to further verify the feasibility and effectiveness of the method of the present invention, the method of the present invention was experimented;

[0055] In the experiment, we directly selected public point cloud quality datasets for training and testing, including SJTU and WPC. In the experiment, we adopted the k-fold cross-validation strategy. For each point cloud quality dataset, we randomly selected The distorted samples are used as the training set, and the remaining point cloud is used as the test set. The final result can be obtained by averaging the performance of k times;

[0056] Four indicators are used to evaluate and compare different point cloud quality evaluation methods, including Spearman rank correlation coefficient SRCC, Pearson linear correlation coefficient PLCC, Kendall rank correlation coefficient KRCC and root mean square error RMSE; specifically, SRCC and KRCC are mainly used to evaluate the monotonicity of the model to ensure that the model can maintain a consistent trend when the variables change; while PLCC and RMSE are used to evaluate the accuracy of the model to ensure that the prediction results are as close to the actual values ​​as possible; Figure 7 and Figure 8 It is a scatter plot between the objective prediction scores and subjective quality scores of the three-dimensional point cloud quality prediction method of the present invention and the existing point cloud quality prediction method on the SJTU dataset; Fig. 9 and Fig.10 The scatter plot is a scatter plot between the objective prediction score and the subjective quality score of the three-dimensional point cloud quality prediction method of the present invention and the existing point cloud quality evaluation method on the WPC dataset; the following Table 1 gives a quantitative comparison of the evaluation indicators of the method of the present invention and the common point cloud quality evaluation method on the SJTU and WPC datasets:

[0057] Table 1 Quantitative comparison of the evaluation indicators of the proposed method and common point cloud quality evaluation methods on the SJTU and WPC datasets

[0058] ;

[0059] Among them, higher correlation coefficient and lower RMSE represent better performance.

[0060] In the description of the present invention, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" etc. means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0061] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A three-dimensional point cloud quality prediction method based on graph convolutional neural network, characterized in that: The following steps are involved: Step S1, for at least one three-dimensional object, obtaining a data set of the three-dimensional object, the data set including a plurality of distorted point clouds and a subjective quality score corresponding to each of the distorted point clouds, then performing dual-path multi-view projection on each of the distorted point clouds and preprocessing the projection image to obtain a corresponding preprocessed projection image; and dividing each of the preprocessed projection images and the subjective quality scores corresponding to each of the preprocessed projection images into a training set and a test set; Step S2, using a deep learning framework to build a reference-free point cloud quality assessment network, the reference-free point cloud quality assessment network includes a backbone module, a multi-level attention perception module, a multi-level conversion module and a quality prediction module, through which the backbone module receives the projection image groups of the two paths and outputs a feature map and feature map , through the multi-level attention perception module to the feature map and the characteristic graph Convolution operation, feature map multiplication operation and global average pooling operation are performed in sequence to obtain the feature map and feature map , through the multi-level conversion module to the feature map And the feature map Processing to obtain graph structure and graph structure , the graph structure is predicted by the quality prediction module And the graph structure Processing is performed to obtain a predicted quality score S; Step S3, using the training set to train the reference-free point cloud quality assessment network to obtain the prediction quality score S corresponding to each distorted point cloud in the training set; Step S4, training the no-reference point cloud quality assessment network for multiple rounds according to step S3 to obtain a no-reference point cloud quality assessment network model; Step S5, testing each of the distorted point clouds in the test set by using the no-reference point cloud quality evaluation network model, and outputting the predicted quality score S of each of the distorted point clouds as an evaluation result of the no-reference point cloud quality; In step S2, the quality prediction module includes two graph convolution networks with the same structure and no sharing and a hybrid prediction block, the two graph convolution networks both include a four-layer structure and the four-layer structures are connected in sequence, and the graph structure is received by the first graph convolution network. And process to get the feature map , feature map , feature map and feature map , receiving the graph structure through the second graph convolutional network And process to get the feature map , feature map , feature map and feature map , and then the feature map is predicted by the hybrid prediction block , the feature map , the feature map , the feature map , the feature map And the feature map The corresponding first processed feature map is obtained by using average pooling and a fully connected layer with an output channel number of 1, and the feature map is And the feature map Average pooling and fully connected layer are performed respectively to obtain the corresponding second processed feature map, and then the first processed feature map and the second processed feature map are channel-joined to obtain a mixed feature map. , the mixed feature map The size of is 10x1, and finally the mixed feature map The predicted quality score S is obtained through a fully connected layer with 10 input channels and 1 output channel; In step S2, the two graph convolution networks each include four graph convolution blocks connected in sequence, the input end of the first graph convolution block serves as the input end of the graph convolution network, the input end of the second graph convolution block receives the feature map output by the output end of the first graph convolution block, the input end of the third graph convolution block receives the feature map output by the output end of the second graph convolution block, the input end of the fourth graph convolution block receives the feature map output by the output end of the third graph convolution block, and the feature maps output by the output ends of the first, second, third, and fourth graph convolution blocks serve as the output end of the graph convolution network; wherein the first three graph convolution blocks have the same structure, and all include graph convolution layers, batch normalization layers, and Softplus activation functions connected in sequence. layer, the fourth graph convolution block only includes the graph convolution layer and the Softplus activation function layer connected in sequence, the input end of the graph convolution layer serves as the input end of the graph convolution block where it is located, and the output end of the Softplus activation function layer serves as the output end of the convolution block where it is located; the number of input channels of the graph convolution layer in the first graph convolution block is 2304, and the number of output channels is 512, the number of input channels of the graph convolution layer in the second graph convolution block is 512, and the number of output channels is 128, the number of input channels of the graph convolution layer in the third graph convolution block is 128, and the number of output channels is 32, and the number of input channels of the graph convolution layer in the fourth graph convolution block is 32, and the number of output channels is 1.

2. The three-dimensional point cloud quality prediction method according to claim 1, characterized in that: In step S1, dual-path multi-view projection is performed on each distorted point cloud in the data set, and the number of projection images obtained on each projection path is N. Then, each projection image is pre-processed by resizing, cropping and normalization to obtain an image size of The pre-processed projection image.

3. The three-dimensional point cloud quality prediction method according to claim 2, characterized in that: In step S1, the process of preprocessing each of the projection images includes: Each of the projected images is resized and cropped, and the size of each of the projected images is scaled to , and then normalize each of the scaled projection images, normalize the pixel values ​​of all pixels in the R channel of the scaled projection image to a mean of 0.485 and a variance of 0.229, normalize the pixel values ​​of all pixels in the G channel to a mean of 0.456 and a variance of 0.224, and normalize the pixel values ​​of all pixels in the B channel to a mean of 0.406 and a variance of 0.

225.

4. The three-dimensional point cloud quality prediction method according to claim 1, characterized in that: In step S2, the backbone module is a ResNet101 backbone network comprising a five-layer structure, wherein the five-layer structure is connected in sequence, and the input end of the first layer serves as the input end of the backbone module and simultaneously receives the projection image groups of the two paths, each of which comprises N images of size The preprocessed projection image, the output ends of the second and fifth layers are used as the multi-level output ends of the backbone module, and the output end of the first layer outputs the feature map , the input end of the second layer receives the feature map , the output end of the second layer outputs the feature map , the input end of the third layer receives the feature map , the output feature map of the third layer , the input end of the fourth layer receives the feature map , the output feature map of the fourth layer , the input end of the fifth layer receives the feature map , the output end of the fifth layer outputs the feature map ; wherein the feature map The size is , the feature map The size is , the feature map The size is , the feature map The size is , the feature map The size is .

5. The three-dimensional point cloud quality prediction method according to claim 1, characterized in that: In step S2, the multi-level attention perception module obtains the feature map through two branches. and the characteristic graph , wherein the execution steps of the first branch include: The characteristic diagram As input, a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 256, and an output channel number of 1 is used to compress the channel size, and a Sigmoid activation function is applied to map the feature value range to [0,1] to obtain the attention feature map , the attention feature map The size is ; and with the characteristic graph As input, a global average pooling layer is used to perform global average pooling in the spatial dimension, and then it passes through a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 256, and an output channel number of 16, and a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 16, and an output channel number of 256, and the Sigmoid activation function is applied to map the feature value range to [0,1] to obtain the attention feature map , the attention feature map The size is ; Then the attention feature map and the attention feature map Multiply to get the mixed attention feature map , the hybrid attention feature map The size is ; and the mixed attention feature map With the characteristic diagram Multiply and connect with the feature map through residual connection Add, and then perform a global average pooling operation to obtain the feature map , the feature map The size is ; The execution steps of the second branch include: The characteristic diagram As input, a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 2048, and an output channel number of 1 is used to compress the channel size, and a Sigmoid activation function is applied to map the feature value range to [0,1] to obtain the attention feature map , the attention feature map The size is ; and with the characteristic graph As input, a global average pooling layer is used to perform global average pooling in the spatial dimension, and then it passes through a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 2048, and an output channel number of 128, and a convolution layer with a convolution kernel size of 1, a stride size of 1, a padding of 0, an input channel number of 128, and an output channel number of 2048, and the Sigmoid activation function is applied to map the eigenvalue range to [0,1] to obtain the attention feature map , the attention feature map The size is ; and the attention feature map and the attention feature map Multiply to get the mixed attention feature map , the hybrid attention feature map The size is ; and the mixed attention feature map With the characteristic diagram Multiply and connect with the feature map through residual connection Add, and then perform a global average pooling operation to obtain the feature map , the feature map The size is .

6. The three-dimensional point cloud quality prediction method according to claim 1, characterized in that: In step S2, the multi-level conversion module includes a multi-level feature fusion block and a graph construction block. And the feature map Perform channel splicing to obtain the feature map , the feature map The size is Then, the feature map is projected according to the dual-path multi-view projection path. Segmentation is performed to obtain the horizontal projection feature map And vertical projection feature map , the horizontal projection feature map The size is , the vertical projection feature map The size is ; The horizontal projection feature map is constructed by the map building block And the vertical projection feature map Each projection feature in is taken as a node, and the adjacency matrix is ​​constructed based on the adjacency relationship between any two nodes. and , the adjacency matrix The size is NxN, the adjacency matrix The size is NxN, and then according to the feature map , the feature map , the adjacency matrix and the adjacency matrix Forming the corresponding graph structure And the graph structure .

7. The three-dimensional point cloud quality prediction method according to claim 1, characterized in that: In step S2, for the first graph convolutional network, the input end of the first layer receives the graph structure as the input end of the first graph convolutional network. , the output end of the first layer outputs the feature map , the input of the second layer receives the graph structure , the output end of the second layer outputs the feature map , the input of the third layer receives the graph structure , the output feature map of the third layer , the input of the fourth layer receives the graph structure , the output feature map of the fourth layer ; wherein the feature map The size is , the feature map The size is , the feature map The size is , the feature map The size is .

8. The three-dimensional point cloud quality prediction method according to claim 1, characterized in that: In the step S2, for the second graph convolutional network, the input end of the first layer receives the graph structure as the input end of the second graph convolutional network , the output end of the first layer outputs the feature map , the input of the second layer receives the graph structure , the output end of the second layer outputs the feature map , the input of the third layer receives the graph structure , the output feature map of the third layer , the input of the fourth layer receives the graph structure , the output feature map of the fourth layer ; wherein the feature map The size is , the feature map The size is , the feature map The size is , the feature map The size is .

Images